The unexpected effectiveness of one-shot decompilation with Claude(blog.chrislewis.au) |
The unexpected effectiveness of one-shot decompilation with Claude(blog.chrislewis.au) |
I also wasn't familiar with this terminology:
> You hand it a function; it tries to match it, and you move on.
In decompilation "matching" means you found a function block in the machine code, wrote some C, then confirmed that the C produces the exact same binary machine code once it is compiled.
The author's previous post explains this all in a bunch more detail: https://blog.chrislewis.au/using-coding-agents-to-decompile-...
I also remember a few things in the singleplayer being very difficult. The number of times I had to fight/race Dameian in his giant robot running down the mountainside... It's carved into my brain like that footrace against Wizpig in DKR or the Donkey Kong arcade game for the Rareware coin in DK64.
The battle items in Snowboard Kids were clever and memorable. The parachute missile that would launch racers up in the air and then deploy the parachute so they slowly float back down was such a frustrating item to be hit with. The pans that would hit all opponents was iconic and it was hilarious that you could somehow doge it with invisibility. Even the basic rock dropped on the course was somehow memorable.
Great game. It's heartwarming to know that others still remember it and care about it.
The interesting part: the model consistently underestimates its own speed. We built a complete bug bounty submission pipeline - target research, vulnerability scanning, POC development - in hours when it estimated days. The '10 attempts' heuristic resonates - there's definitely a point where iteration stops being productive.
For decompilation specifically, the 1M context window helps enormously. We can feed entire codebases and ask 'trace this user input to potential sinks' which would be tedious manually. Not perfect, but genuinely useful when combined with human validation.
The key seems to be: narrow scope + clear validation criteria + iterative refinement. Same as this decompilation work.
Eventually some or many of these attempts would, of course, fail, and require programmer intervention, but I suspect we might be surprised how far it could go.
The modern approach is: feed the errors back to the LLM and have it fix them.
They had access to the same C compiler used by Nintendo in 1999? And the register allocation on a MIPS CPU is repeatable enough to get an exact match? That's impressive.
The groundwork for this kind of "matching" process is: sourcing odd versions of the obscure tooling that was used to build the target software 20 years ago, and playing with the flag combinations to find out which was used.
It helps that compilers back then were far less complex than those of today, and so was the code itself. But it's still not a perfect process.
There are cases of "flaky" code - for example, code that depends on the code around it. So you change one function, and that causes 5 other functions to no longer match, and 2 functions to go from not matching to matching instead.
Figuring out and resolving those strange dependencies is not at all trivial, so a lot of decompliation efforts end up wrapping it up at some "100% functional, 99%+ matching".
> Snowboard Kids 2 was written in C and compiled to MIPS machine code. The compiler was likely GCC 2.7.2 based on the instruction patterns [3]
The footnote is interesting: https://blog.chrislewis.au/using-coding-agents-to-decompile-...
> This is mostly just guesswork and trying different variations of compiler versions and configuration options. But it isn’t as bad as it sounds since the time period limits which compilers were plausibly used. The compiler arguments used in other, similar, games also provide a useful reference.
What LLMs are (still?) not good at is one-shot reverse engineering for understanding by a non-expert. If that's your goal, don't blindly use an LLM. People already know that you getting an LLM to write prose or code is bad, but it's worth remembering that doing this for decompilation is even harder :)
Not what I would have expected from a 'one-shot'. Maybe self-supervised would be a more suitable term?
See also, "zero-shot" / "few-shot" etc.
1. Getting an LLM to do something based on a single example
2. Getting an LLM to achieve a goal from a single prompt with no follow-ups
I think both are equally valid.
We're essentially trying to map 'traditional' ML terminology to LLMs, it's natural that it'll take some time to get settled. I just thought that one-shot isn't an ideal name for something that might go off into an arbitrarily long loop.
It doesn't do it in one-shot on the GPU either. It feeds outputs back into inputs over and over. By the time you see tokens as an end-user, the clanker has already made a bunch of iterations.
I stayed away from decompilation and reverse engineering, for legal reasons.
Claude is amazing. It can sometimes get stuck in a reason loop but will break away, reassess, and continue on until it finds its way.
Claude was murdered in a dark instance dungeon when it managed to defeat the dragon but ran out of lamp oil and torches to find its way out. Because of the light system it kept getting “You can’t seem to see anything in the darkness” and randomly walked into a skeleton lair.
Super fun to watch from an observer. Super terrifying that this will replace us at the office.
I hope that others find this similarly useful.
It's good at cleaning up decompiled code, at figuring out what functions do, at uncovering weird assembly tricks and more.
Anyway, we're reaching the point where documentation can be generated by LLMs and this is great news for developers.
I then pasted this to another CC instance running the FE app, and it made the counter part.
Yes, I could have CC running against both repos and sometimes do, but I often run separate instances when tasks are complex.
Although of course if you don't vibe document but instead just use them as a tool, with significant human input, then yes go ahead.
The hardest form of code obfuscation is called homomorphic computing, which is code transformed to act on encrypted data isomorphically to regular code on regular data. The homomorphic code is hard obfuscated by this transformation.
Now create a homomorphic virtual machine, that operates on encrypted code over encrypted data. Very hard to understand.
Now add data encryption/decryption algorithms, both homomorphically encrypted to be run by the virtual machine, to prepare and recover inputs, outputs or effects of any data or event information, for the homomorphic application code. Now that all data within the system is encrypted by means which are hard obfuscated, running on code which is hard obfuscated, the entire system becomes hard^2 (not a formal measure) opaque.
This isn't realistic in practice. Homomorphic implementations of even simple functions are extremely inefficient for the time being. But it is possible, and improvements in efficiency have not been exhausted.
Equivalent but different implementations of homomorphic code can obviously be made. However, given the only credible explanations for design decisions of the new code are, to exactly match the original code, this precludes any "clean room" defenses.
--
Implementing software with neural network models wouldn't stop replication, but would decompile as source that was clearly not developed independent from the original implementation.
Even distilling (training a new model on the "decompiled" model) would be dead giveaway that it was derived directly from the source, not a clean room implementation.
--
I have wondered, if quantum computing wouldn't enable an efficient version of homomorphic computing over classical data.
Just some wild thoughts.
Like, if it ever leaks, or you were planning on releasing it, literally every step you took in your crime is uploaded to the cloud ready to send you to prison.
It's what's stopped me from using hosted LLMs for DMCA-legal RE. All it takes is for a prosecutor/attorney to spin a narrative based on uploaded evidence and your ass is in court.
Have you tried asking them to simply open source the code?
Even if they were willing to (they're not) and if they still have the code (they don't), it will contain proprietary code from Nintendo and you'll never get your hands on that (legally)
I asked Opus how hard it would be to port the script extender for Baldurs Gate 3 from Windows to the native Linux Build. It outlined that it would be very difficult for someone without reverse engineering experience, and correctly pointed out they are using different compilers, so it's not a simple mapping exercise. It's recommendation was not to try unless I was a Ghrida master and had lots of time in my hands.
I've had CC build semi-complex Tauri, PyQT6, Rust and SvelteKit apps for me without me having ever touched that language. Is the code quality good? Probably not. But all those apps were local-only tools or had less than 10 users so it doesn't matter.
I have this in my CLAUDE.md now.
“We are here to do the difficult and have plenty of time and there’s no rush.”
Which is to say that probably antropic don’t have good training documents and evals to teach the model how to do that.
Well they didn’t. But now they have some.
If the author want to improve his efficiency even more, I’d suggest he starts creating tools that allow a human to create a text trace of a good run on decompilating this project.
Those traces can be hosted in a place Antropic can see and then after the next model pre-training there will be a good chance the model become even better at this task.
> An open-source license is a type of license for computer software and other products that allows the source code, blueprint or design to be used, modified or shared (with or without modification) under defined terms and conditions.
https://en.wikipedia.org/wiki/Open_source
Companies have been really abusing what open source means- claiming something is "open source" cause they share the code and then having a license that says you can't use any part of it in any way.
Similarly if you ever use that software or depending on where you downloaded it from, you might have agreed not to decompile or read the source code. Using that code is a gamble.
FOSS specifically means/meant free and open source software, the free and software words are there for a reason
so we don’t need another distinction like “source available” that people need to understand to convey an already shared concept
yes, companies abuse their community’s interest in something by blending open source legal term as a marketing term
It'll either all be in the cloud, so you never run the code...
Or it'll be on a chip, in a hermetically sealed usb drive, that you plug in to your computer.
https://hackaday.com/2014/07/31/playing-starcraft-on-an-arm/
First we get to tackle all of the small ideas and side projects we haven't had time to prioritize.
Then, we start taking ownership of all of the software systems that we interact with on a daily basis; hacking in modifications and reverse engineering protocols to suit our needs.
Finally our own interaction with software becomes entirely boutique: operating systems, firmware, user interfaces that we have directed ourselves to suit our individual tastes.
Obviously others aren’t concerned or don’t live in jurisdictions where that would be an issue.
Though, that’s only for actively developer software. I can imagine a great future where all retro games are now source available.
And it will be great for retro game preservation.
Having more integrated tools and tutorials on this would be awesome.
If you are able to monitor what happens to encrypted data being processed by an LLM, could you not match that with the same patterns created by unencrypted data?
Real simple example, let's say I have a program that sums numbers. One sends the data to an LLM or w/e unencrypted, the other encrypted.
Wouldn't the same part of the LLM/compute machine "light up" so to speak?
(Of course, the reality is much more complicated than this; you can trace things like power and in theory track individual activations with great difficulty and then do interpretability to see what the model is doing. But hopefully it illustrates that you can usually take some sort of special operation and turn it into a process that does the same operation on different data.)
John Carmack
One of the great challenges of building apps is guessing the 80/20. I think we’re actually entering the long dreamt of reusable component age.
That's very different from the decompilation projects being discussed here, which do distribute the decompiled code.
These decompilation projects do involve some creative choices, which means that the decompilation would likely be considered a derivative work, containing copyrightable elements from both the authors of the original binary and the authors of the decompilation project. This is similar to a human translation of a literary work. A derivative work does have its own copyright, but distributing a derivative work requires permission from the copyright holders of both the original and the derivative. So a decompilation project technically can set their own license, and thereby add additional restrictions, but they can't overwrite the original license. If there is no original license, the default is that you can't distribute at all.
They had gotten surprisingly close to a complete decompilation, but then they tried to request a copy of the source code from the copyright office citing that they needed it as a result of ongoing unrelated litigation with Nintendo.
Later on this killed them in court.
I expect that for games the more important piece will be the art assets - like how the Quake game engine was open source but you still needed to buy a copy of the game in order to use the textures.
For this project, it described its reasoning well, and knowing my own skillset, and surface level info on how one would start this, it had many good points that made the project not realistic for me.
See my other comment: https://news.ycombinator.com/item?id=46175760
Relevant: https://github.com/orgs/community/discussions/82431
> When you make a creative work (which includes code), the work is under exclusive copyright by default. Unless you include a license that specifies otherwise, nobody else can copy, distribute, or modify your work without being at risk of take-downs, shake-downs, or litigation. Once the work has other contributors (each a copyright holder), “nobody” starts including you.
...but it's very clearly not an open source license.
I don't want invented rationales for changes, I want to know the actual reason a developer decided that the code should work that way.
Not so much when you have a lot of code from 6 years ago, built around an obscure SDK, and you have to figure out how it works, and the documentation is both incredibly sparse and in Chinese.
If people __can__ actually read undocumented code with the help of LLMs, why do you need human-written documentation really?