The port I couldn't ship(ammil.industries) |
The port I couldn't ship(ammil.industries) |
> I don’t recall what happened next. I think I slipped into a malaise of models. 4-way split-paned worktrees, experiments with cloud agents, competing model runs and combative prompting.
You’re trying to have the LLM solve some problem that you don’t really know how to solve yourself, and then you devolve into semi-random prompting in the hope that it’ll succeed. This approach has two problems:
1. It’s not systematic. There’s no way to tell if you’re getting any closer to success. You’re just trying to get the magic to work.
2. When you eventually give up after however many hours, you haven’t succeeded, you haven’t got anything to build on, and you haven’t learned anything. Those hours were completely wasted.
Contrast this with you beginning to do the work yourself. You might give up, but you’d understand the source code base better, perhaps the relationship between Perl and Typescript, and perhaps you’d have some basics ported over that you could build on later.
This feels like the LLM-enabled version of this behavior (except that in the former case, students will quickly realize that what they’re doing is pointless and ask a peer or teacher for help; whereas maybe the LLM is a little too good at hijacking that and making its user feel like things are still on track).
The most important thing to teach is how to build an internal model of what is happening, identify which assumptions in your model are most likely to be faulty/improperly captured by the model, what experiments to carry out to test those assumptions…
In essence, what we call an “engineering mindset” and what good education should strive to teach.
That sounds like a lot of people I’ve known, except they weren’t students. More like “senior engineers”.
I'm currently torn on whether to actually release it - it's in a private GitHub repository at the moment. It's super-interesting and I think complies just fine with the MIT licenses on MicroQuickJS so I'm leaning towards yes.
Its got to 402 tests with 2 failing - the big unlock was the test suite from MicroQuickJS: https://github.com/bellard/mquickjs/tree/main/tests
Its been spitting out lines like this as it works:
I see the issue - toFixed is using
Python’s default formatting which uses
round-half-to-even rounding, but
JavaScript uses round-half-away-from-zero.idk how complete it is but it solved youtube's challenges etc for a long time
https://github.com/yt-dlp/yt-dlp/blob/6d92f87ddc40a319590976...
Here's the transcript showing how I built it: https://static.simonwillison.net/static/2025/claude-code-mic...
Though if you look in those files some of them run a ton of test functions and assertions.
My new Python library executes copies of the tests from that mquickjs repo - but those only count as 7 of the 400+ other tests.
Or why not run MicroQuickJS under Fil-C? It's ideal since it has not dependencies.
I'm generating a lot of PDFs* in claude, so it does ascii diagrams for those, and it's generally very good at it, but it likely has a lot of such diagrams in its training set. What it then doesn't do very well is aligning them under modification. It can one-shot the diagram, it can't update it very well.
The euphoric breakthrough into frustration of so-called vibe-coding is well recognised at this point. Sometimes you just have to step back and break the task down smaller. Sometimes you just have to wait a few months for an even better model which can now do what the previous one struggled at.
* Well, generating Typst mark-up, anyway.
Specially considering that the output would be essentially the same: a bunch of code that doesn't work.
Edit: I totally agree with your point about not wanting to learn a language. That's definitely a situation where LLMs can excel and almost an ideal use case for them. I just think that Perl, in particular, will be hard to work with, given the current capabilities of LLM coding tools and models. It might be necessary to actually learn the language, and even that might not be enough.
> I spent weeks casually trying to replicate what took years to build. My inability to assess the complexity of the source material was matched by the inability of the models to understand what it was generating.
When the trough of disillusionment hits, I anticipate this will become collective wisdom, and we'll tailor LLMs to the subset of uses where they can be more helpful than hurtful. Until then, we'll try to use AI to replace in weeks what took us years to build.
I don’t see a particularly good reason why LLMs wouldn’t be able to do most programming tasks, with the limitation being our ability to specify the problem sufficiently well.
> able to do most programming tasks, with the limitation being our ability to specify the problem sufficiently well
We've spent 80 years trying to figure that out. I'm not sure why anyone would think we're going to crack this one anytime in the next few years.
Such has always been the largest issue with software development projects, IMO.
"I took a long-overdue peek at the source codebase. Over 30,000 lines of battle-tested Perl across 28 modules. A* pathfinding for edge routing, hierarchical group rendering, port configurations for node connections, bidirectional edges, collapsing multi-edges. I hadn’t expected the sheer interwoven complexity."
The AI's are super capable now, but still need a lot of guiding towards the right workflow for the project. They're like a sports team, but you still need to be a good coach.
One difference between this story and the various success stories is that the latter all had comprehensive test suites as part of the source material that agents could use to gain feedback without human intervention. This doesn’t seem to exist in this case, which may simply be the deal breaker.
Perhaps, but perhaps not. The reason tests are valuable in these scenarios is they are actually a kind of system spec. LLMs can look at them to figure out how a system should (and should not) behave, and use that to guide the implementation.
I don’t see why regular specs (e.g. markdown files) could not serve the same purpose. Of course, most GitHub projects don’t include such files, but maybe that will change as time goes on.
I think because they're doomed to become outdated without something actually enforcing the spec.
It turns out that having a "trainer" to "coach" you is not a coincidence: these two words evolved together from the rail industry to the gym. Do "port" and "ship" have a similar history, evolving together from the maritime industry to software?
Is that confidence of or positivity? I hope I will find out in the future, here on HN.
I use this LLM called git clone.
I consider myself a bit of an expert vibe engineer and the challenge is alluring :D
This is easy work, made hard by the "allure" of LLMs, which go from emphatic to emetic in the blink of an eye.
If you don't know what you are doing, you should stay away from LLMs if there is anything at all at stake.
The actual goal is to faithfully replicate the functionality and solve the same use cases with a different set of base technologies.
You describing similar but different instrumental goals, which may help reaching the real goal.
Cheekiness aside, your framing is helpful!
I simply cannot come up with tasks the LLMs can't do, when running in agent mode, with a feedback loop available to them. Giving a clear goal, and giving the agent a way to measure it's progress towards that goal is incredibly powerful.
With the problem in the original article, I might have asked it to generate 100 test cases, and run them with the original Perl. Then I'd tell it, "ok, now port that to Typescript, make sure these test cases pass".
It's really easy to come up with plenty of algorithmic tasks that they can't do.
Like: implement an algorithm / data structure that takes a sequence of priority queue instructions (insert element, delete smallest element) in the comparison model, and return the elements that would be left in the priority queue at the end.
This is trivial to do in O(n log n). The challenge is doing this in linear time, or proving that it's not possible.
(Spoiler: it's possible, but it's far from trivial.)
Our team used claude to help port a bunch of python code to java for a critical service rewrite.
As a "skeptic", I found this to demonstrate both strengths and weaknesses of these tools.
It was pretty good at taking raw python functions and turning them into equivalent looking java methods. It was even able to "intuit" that a python list of strings called "active_set" was a list of functions that it should care about and discard other top level, unused functions. The functions had reasonable names and picked usable data types for every parameter, as the python code was untyped.
That is, uh, the extent of the good.
The bad: It didn't "one-shot" this task. The very first attempt, it generated everything, and then replaced the generated code with a "I'm sorry, I can't do that"! After trying a slightly different prompt it of course worked, but it silently dropped the code that caused the previous problem! There was a function that looked up some strings in the data, and the lookup map included swear words, and apparently real companies aren't allowed to write code that includes "shit" or "f you" or "drug", so claude will be no help writing swear filters!
It picked usable types but I don't think I know Java well enough to understand the ramifications of choosing Integer instead of integer as a parameter type. I'll have to look into it.
It always writes a bunch of utility functions. It refactored simple and direct conditionals into calls to utility functions, which might not make the code very easy to read. These utility functions are often unused or outright redundant. We have one file with like 5 different date parsing functions, and they were all wrong except for the one we quickly and hackily changed to try different date formats (because I suck so the calling service sometimes slightly changes the timestamp format). So now we have 4 broken date parsing functions and 1 working one and that will be a pain that we have to fix in the new year.
The functions look right at first glance but often had subtle errors. Other times the ported functions had parts where it just gave up and ignored things? These caused outright bugs for our rewrite. Enough to be annoying.
At first it didn't want to give me the file it generated? Also the code output window in the Copilot online interface doesn't always have all the code it generated!
It didn't help at all with the hard part: Actual engineering. I had about 8 hours and needed find a way to dispatch parameters to all 50ish of these functions and I needed to do it in a way that didn't involve rebuilding the entire dispatch infrastructure from the python code or the dispatch systems we had in the rest of the service already, and I did not succeed. I hand wrote manual calls to all the functions, filling in the parameters, which the autocomplete LLM in intellij kept trying to ruin. It would constantly put the wrong parameters places and get in my way, which was stupid.
Our use case was extremely laser focused. We were working from python functions that were designed to be self contained and fairly trivial, doing just a few simple conditionals and returning some value. Simple translation. To that end it worked well. However, we were only able to focus the tool into this use case because we already had the 8 years experience of the development and engineering of this service, and had already built out the engineering of the new service, building lots of "infrastructure" that these simple functions could be dropped into, and giving us easy tooling to debug the outcomes and logic bugs in the functions using tens of thousands of production requests, and that still wasn't enough to kill all errors.
All the times I turned to claude for help on a topic, it let me down. When I thought java reflection was wildly more complicated than it actually is, it provided the exact code I had already started writing, which was trivial. When I turned to it for profiling our spring boot app, it told me to write log statements everywhere. To be fair, that is how I ended up tracking down the slowdown I was experiencing, but that's because I'm an idiot and didn't intuit that hitting a database on the other side of the country takes a long time and I should probably not do that in local testing.
I would pay as much for this tool per year as I pay for Intellij. Unfortunately, last I looked, Jetbrains wasn't a trillion dollar business.
In software engineering, "ship" commonly means "distribute" (to a deliberately unspecified audience), while "port" commonly means "software manually translated to another programming language or adapted to another platform".
I'm sure the MS plan is not just asking Claude "port this code to rust: <paste>", but it's just fun to think it is :)
0: https://www.theregister.com/2025/12/24/microsoft_rust_codeba...
The etymological roots are quite interesting, though. We aren't quite sure where the word "ship" comes from — Etymonline hazards
> Watkins calls this a "Germanic noun of obscure origin." OED says "the ultimate etymology is uncertain." Traditionally since Pokorny it is derived from PIE root *skei- "to cut, split," perhaps on the notion of a tree cut out or hollowed out, but the semantic connection is unclear. Boutkan gives it "No certain IE etymology."
The word "port" goes back to the PIE root "*per-" meaning "forward", and thus as a verb "to lead". It seems to have emerged in Latin in multiple forms: the word "portus" ("harbor"), verb "portare" (to carry or bring). I was surprised to learn that the English "ferry" does not come from the other Latin verb with the sense of carrying (the irregular "ferre"), but from Germanic and Norse words... that are still linked back to "*per-".
Basically, transportation (same "port"!) has been important to civilization for a long time, and quite a bit of it was done by, well, shipping. And porting software is translating the code; the "lat" there comes from the past participle of the irregular Latin verb mentioned above, about which
> Presumably lātus was taken (by a process linguists call suppletion) from a different, pre-Latin verb. By the same process, in English, went became the past tense of go. Latin lātus is said by Watkins to be from *tlatos, from PIE root *tele- "to bear, carry" (see extol), but de Vaan says "No good etymology available."
This does not appear to be true.
Train (etymonline):
> "to discipline, teach, bring to a desired state or condition by means of instruction," 1540s, which probably is extended from the earlier sense of "draw out and manipulate in order to bring to a desired form" (Middle English trainen, attested c. 1400 as "delay, tarry" on a journey, etc.); from train (n.) For the notion of "educate" from that of "draw," compare educate.
[That train (n.) doesn't refer to the rail industry, which didn't really exist in the 1540s. It refers to a succession (as one railcar will follow another in later centuries), or to the part of your clothing that might drag on the ground behind you, or to the act of dragging anything generally. Interestingly, etymonline derives this noun from a verb train meaning to drag; given the existence of this verb, I see no reason to derive the verb train in the sense "teach" from the noun derived from the same verb in the sense "drag". The entry on the verb already noted that it isn't unexpected for "drawing" [as water from a well] to evolve into "teaching".]
Coach (wiktionary):
> The meaning "instructor/trainer" is from Oxford University slang (c. 1830) for a "tutor" who "carries" one through an exam
Coach might be a metaphor from the rail industry (or the horse-and-buggy industry), but trainer isn't.
[0]
Java's `int` is a 32-bit "machine" integer (in a virtual architecture, but still stored by value with no additional space overhead). Java's `Integer` is an object with reference semantics, like[1] every value in a Python program — but unlike Python's `int`, it still has the 32-bit range restriction. If you need arbitrary-size integer values in Java, those come from `java.math.BigInteger`.
> It always writes a bunch of utility functions. It refactored simple and direct conditionals into calls to utility functions, which might not make the code very easy to read.
Are the names good, at least? I do this sort of thing and often find it helpful. Of course, that does depend on choosing one utility function for the same task and reusing it, and being sure it actually works.
> I hand wrote manual calls to all the functions, filling in the parameters, which the autocomplete LLM in intellij kept trying to ruin. It would constantly put the wrong parameters places and get in my way, which was stupid.
Yeah, Java lacks a lot of Python's nice tricks for this. (I've had those frustrations with IDEs since long before LLMs.)
> it told me to write log statements everywhere. To be fair, that is how I ended up tracking down the slowdown I was experiencing, but that's because I'm an idiot and didn't intuit that hitting a database on the other side of the country takes a long time and I should probably not do that in local testing.
It sounds like you wanted this for immediate debugging. The word "logging" does not autocomplete "to a remote server db" in my head. Sometimes it's useful to have mental defaults oriented towards what is temporary and quick rather than what is permanent and robust.
[0] Did you consider asking the LLM? It can probably deal with this question pretty well if you ask directly, although I don't know how much it would take to get from there to actually having it fix any problems. But I might as well write a human perspective since I'm here.
[1] Unlike Python, all those "objects with reference semantics" can be NULL in Java (and you need a possibly-third-party annotation to restrict that type to be non-null). There is no "null object" analogous to Python's `None`.
Property based testing can be really useful here.
I found Google Antigravity (with the current Gemini models) to be fairly capable. If I had to guess, it seems like they set up their system to get that divide-and-conquer going. As you suggest, it's not that hard: they just have to put the instructions in their equivalent of the system prompt.
Well, when I say 'not that hard', I mean it's an engineering problem to get the system and tooling working together nicely, not really an AI problem.
That made the test pass of course, leaving the code as broken as it ever was. Guess that one was on me though, I never specified it shouldn't do that...
Incremental gains are fine. I suspect capability of models scales roughly as the logarithm of their training effort.
> (read: drinking water and energy)
Water is not much of a concern in most of the world. And you can cool without using water, if you need to. (And it doesn't have to be drinking water anyway.)
Yes, energy is a limiting factor. But the big sink is in training. And we are still getting more energy efficient. At least to reach any given capability level; of course in total we will be spending more and more energy to reach ever higher levels.
Water is a concern in huge parts of the World, as is energy consumption.
And if the big sink is “just” in training, why is there so much money being thrown at inference capacity?
I thought it was mad when I read that Bitcoin uses more energy than the country of Austria, but knowing AI inference using more energy than all the homes in the USA is so, so, so much worse given the quality of the outputs are so mediocre.
It only works with Claude Code for the web sessions at the moment but I expect I'll get it working for local sessions too.
It's a very weird and uncomfortable way of working - I've said in the past that I don't like a single line of unreviewed AI-generated code in anything beyond a prototype, and now here I am with 13,000+ lines of mostly unreviewed Python written by Claude Opus 4.5.
I'm leaving the alpha label on it until I'm a whole lot more comfortable with the codebase!
I do however know that the tests are pretty comprehensive because I had the model use TDD from the very start - write a test, watch it fail, then implement code to make it pass.
I was able to keep an eye on what it was doing on my phone while it worked and the TDD process seemed to be staying honest.
Here's one example from the full transcript, showing how it implemented closures: https://static.simonwillison.net/static/2025/claude-code-mic...
Claude Code found more edge cases to write tests for than I ever would have thought of. And I've been doing this for 20 years.
I feel we were hearing very similar claims 40 years ago, about how the next version of "Fourth Generation Languages" were going to enable business people and managers to write their own software without needing pesky programmers to do it for them. They'll "just" need to learn how to specify the problem sufficiently well.
(Where "just" is used in it's "I don't understand the problem well enough to know how complicated or difficult what I'm about to say next is" sense. "Just stop buying cigarettes, smoker!", "Just eat less and exercise more, fat person!", "Just get a better paying job, poor person!", "Just cheer up, depressed person!")
If not working at one of the big players or running your own, it appears that even the APIs these days are wrapped in layers of tooling and abstracting raw model access more than ever.
No, the APIs for these models haven't really changed all that much since 2023. The de facto standard for the field is still the chat completions API that was released in early 2023. It is almost entirely model improvements, not tooling improvements that are driving things forward. Tooling improvements are basically entirely dependent on model improvements (if you were to stick GPT-4, Sonnet 3.5, or any other pre-2025 model in today's tooling, things would suck horribly).
I disagree. This almost entirely model capability increases. I've stated this elsewhere: https://news.ycombinator.com/item?id=46362342
Improved tooling/agent scaffolds, whatever, are symptoms of improved model capabilities, not the cause of better capabilities. You put a 2023-era model such as GPT-4 or even e.g. a 2024-era model such as Sonnet 3.5 in today's tooling and they would crash and burn.
The scaffolding and tooling for these models have been tried ever since GPT-3 came out in 2020 in different forms and prototypes. The only reason they're taking off in 2025 is that models are finally capable enough to use them.
My personal opinion is that there was a threshold earlier this year where the models got basically competent enough to be used for serious programming work. But all the major on the ground improvements since then has gone from the agents, and not all agents are equal, while all sota models are effectively.
Yes definitely. But this is to be expected. Heck take the same person and put them in two different environments and they'll have very different performance!
> But cross (same tier) model in the same agent is much less stark.
Unclear what you mean by this. I do agree that the big three companies (OpenAI, Anthropic, Google DeepMind) are all more or less neck and neck in SOTA models, but every new generation has been a leap. They just keep leaping over each other.
If you compare e.g. Opus 4.1 and Opus 4.5 in the same agent harness, Opus 4.5 is way better. If you compare Gemini 3 Pro and Gemini 2.5 Pro in the same agent harness, Gemini 3 is way better. I don't do much coding or benchmarking with OpenAI's family of models, but anecdotally have heard the same thing going from GPT-5 to GPT-5.2.
The on the ground improvements have been coming primarily from model improvements, not harness improvements (the latter is unlocked by the former). Again, it's not that there were breakthroughs in agent frameworks that happened; all the ideas we're seeing now have all been tried before. Models simply weren't capable enough to actually use them. It's just that more and more (pre-tried!) frameworks are starting to make sense now. Indeed, there are certain frameworks and workflows that simply did not make sense with Q2-Q3 2025 models that now make sense with Q4 2025 models.
All things being equal I agree that the models are improving, but for many of the tasks I’m testing what has the most improvement is the agent. The agents choosing the appropriate model for the task for instance has been huge.
I do believe there is beneficial symbiosis but for my results the agent's provide much bigger variance than the model.