Scarf has moved away from Haskell(avi.press) |
Scarf has moved away from Haskell(avi.press) |
so does this mean that the LLM writes code that is so good that the compiler does not find any more errors?
or is it due to the nature of haskell that makes it hard to write bad code to begin with?
or just that because the haskell compiler catches more errors there is less broken haskell code for the AI to train on?
and what does that mean for the switch to python? if the python compiler/interpreter doesn't catch as many errors do we even know that the code is good?
or is this more like the belief if the LLM can generate good haskell code, surely it can also generate good python?
what's the solution here? speeding up the haskell compiler? if that were easy, would it not already have happened?
personally i still don't trust LLM code generation. i didn't learn haskell yet, but what i hear about it makes me more likely to trust that LLMs can generate good haskell code than python.
i believe the future in LLM code generation is code that can be proven to be correct. proving code correct has been a research topic at some point.
Refactoring spaghetti has become easier in the LLM era because it can just read all the code, and there is now a skill floor on the programmers that kicks in somewhere relatively high. The benefits of type systems might have suffered because of that.
I don't personally use Haskell for anything, but I use Lean and occasionally some other languages with expressive type systems, and like you I've found it to be a pretty great experience for working with LLMs. But I've also experienced what the author is talking about, with languages that sit at different points on the type system spectrum, regarding a languages ecosystem/infra layer becoming a bottleneck. I don't think it's ultimately about the type system but the broader ergonomics of the language/ecosystem.
So I think his criticism is less than expressive type systems are a pre-LLM concept, and more that Haskell has an individually bad "agentic coding story".
Also, for some reason Optional[T] became deprecated, just as the ecosystem finally embraced types ~3 years ago.
In fact, one my company's greenfield projects decided to use TypeScript instead of Python for the [surprisingly] more consistent tooling, and the fact that the big LLM providers all have official TypeScript SDKs anyway. Also, for agentic coding, LLMs don't seem noticeably worse at TypeScript than Python.
My experience can be summarized as:
- for some reason we need 2-3 static analysis tools just for typechecking
- no tool understands each other's comment directives
- each tool reports a different error in your codebase
- even big libraries (e.g. matplotlib) make half their functions return Any
- you'll be tempted to silence the "partially unknown type" warnings, and you'll have to do it for each tool that's running.
Optional[T] is now T | None. Means exactly the same thing but doesn't require an import. Support for the older syntax presumably won't be removed for a long while regardless.
The type safety we gave up hasn’t been noticeable in any concrete way yet, especially considering our test coverage has never been better.
One approach would be to not use LLMs.
But I do think what benefits LLMs is the speed and accuracy of feedback. Type systems cover the accuracy part, but haskell was killing them on speed. It seems like a strange choice to go so far the other way on accuracy when there's a lot of languages in between. But I'm not familiar with the project so not in a position to call it.
It's not also really about expressiveness IMO. I've found LLMs to be best with more constrained type systems: they are better at ocaml than they are at typescript.
But I'd look at people a bit oddly if they said: 'We didn't want to set up CI caching and compiled languages took 30 minutes per run so we changed our entire codebase to python'.
Maybe it makes sense for them, and caching across dynamically spawned VM's is admittedly a harder problem which most build systems aren't great at, but still. I can easily believe that getting build caching to be reliable would be a lot of work, but is it more work than a full rewrite of a significant codebase?
When the potential set of behaviors you could write a program to have is infinite, but the actual behavior you want is singular, a programming language is more importantly defined by which ones it eliminates up front than which ones it lets you write (assuming it lets you write the one you want at all, but that's almost always going to be the case for most general purpose languages). Bugs are just false positives in this framing, where the program you wrote seems like the one you wanted, but there's some divergence between what you thought you were getting and what you actually got, and catching some of those up front is a huge part of why type systems are so useful.
For some time now it’s felt clear (or at least extremely) compelling that agents need fast compile times in order to be effective, especially when you’re working in parallel. But the other thing that has felt just as obvious is that agents need strong type systems and narrow guardrails in order to constrain their outputs. These two things felt clear enough to me that, like the author, I wanted to choose a language ecosystem that maximized them. There _are_ languages that both have expressive type systems _and_ fast compile times. I wonder if the author investigated any of them, before deciding that no compilation time at all was acceptable.
In my case I landed in OCaml. I think there are other options in the space—Go if you want less typing but faster compiles; Rust if you want more types but slower compiles. My mostly vibes-based evaluation landed on OCaml, and I’ve been pretty happy with the results.
I read the second paragraph of linked article as saying close to the opposite of that, particularly,
"the model can often avoid the mistake before the compiler ever sees the code. And as the models get better, the relative value of catching every possible issue at compile time changes."
In other words, LLMs are much less likely than humans to make dumb, fat-finger mistakes, and, when they do, are able to catch and fix them more quickly, ergo the value of type checking has fallen.
Everything in the prior sentence is, obviously, highly debatable. But it felt like part of the premise.
I would also add Kotlin, Clojure and F#.
Scala not really as the compilation is not much better, and since the Scala 3 reboot, the ecosystem doesn't seem to be doing that well.
The market opportunity for Haskell on the JVM is gone, although they are doing cool stuff with capabilities.
And that is not even considering how often the agent needs to run tests to get it right.
Regarding the first, I think you're probably right, but then again, if there is a 15-minute base cost, it's hard to amortize that through fewer incremental runs of the compiler.
(Which isn't to say that I think they are doing the right thing.)
The amount of certainty random people have that LLMs have already revolutionized software development seems to be directly proportional to the media awareness of the AI companies finance unsustainability.
Slow compile times should have been a deal breaker for how they impacted human coders. LLM coding just makes the problem more stark.
As much as I respect this guy who tried to work and push an alternative ecosystem, it's hard for me to shake off the impression that, rather than due to Haskell compile time, he moved to python because it's easier to find developers for it and it's the de facto scripting language for LLMs.
No problem about that, of course. Running a company is hard enough, I think that passion and idealism for a language/platform/technology out of aesthetic appreciation can only go so far and after a certain age just making money and reaching your professional objectives count more.
Fast compile times is one of the most important qualities for developer productivity. It made Haskell a non-starter for many developers even before LLM driven development took off.
We're a Haskell shop (and have been for over 10 years now) and are finding agentic development with Haskell to work pretty damn well.
Cold compile times in Haskell are painful indeed. Our development practices don't really cause us to do that much - even with agents.
It's unclear to me if the development practices at Scarf that cause them to hit this pain often are worth it if it means giving up Haskell because the compile times are too bad. Maybe they are, but I don't think so.
Since Opus 4.6, LLMs have been pretty clever at using fancy types with libraries like Servant and Beam. The expressiveness of the types, combined with feedback from the compiler, means that agents converge quickly to something that works. I don't think I've noticed agents having to run the compiler so often that compilation speed is an issue.
I would like to add one additional observation, since we have been using both Haskell and Python in production for a long time:
Haskell excels at platform work, while Python excels at product work.
Our infrastructure teams work in Haskell (and also Rust nowadays), while our product teams work in Python. This gives us the best of both worlds (in my opinion): fast and rock-solid infrastructure on the platform side, and fast development speed and quick iteration cycles on the product side.
This setup has worked well for years for us, but it remains to be seen how and if this is going to change as well in the new AI era.
If you go from taking 2 days to write some code and 20 minutes to type check (which does seem long, don't get me wrong, but still) to 10 minutes to prompt some code and 20 minutes to type check, that percentage increase to me isn't enough to justify switching.
You're still almost 2 days ahead, and converting those 20 minutes to 20 seconds are not going to make you ship a feature appreciably faster. But those types stand strong and I don't believe they can yet be replaced by an LLM believing they're correct.
Having said that, I also think that Haskell should massively speed things up. Having strong types if nothing else should surely produce some amazing type-checking speed wins.
[0]: https://discourse.haskell.org/t/after-7-years-in-production-...
- The benefits of more "extreme" type systems are more accessible and valuable than ever. I have a fairly involved project built on Lean that I hope to open source this month, and it's been a joy to work in even for uses outside of mathematics.
- Readability, build time, infra complexity, and everything that affects your speed after finishing your implementation--these things now matter more than ever.
It's sort of a dual ergonomics problem, in some sense. And given that, the author's lament makes complete sense to me, especially:
"An AI-enabled Haskell ecosystem would ask different questions. How do we make Haskell easier for agents to use well? How do we get more high-quality Haskell examples into model training data? How can we scale reviews? How do we make library docs full of copy-pastable, realistic examples, not just beautiful types? How do we make project bootstrap fast? How do we make error messages more agent-friendly? How do we reduce cold build times? How do we make common industrial patterns obvious to a model that is trying to help?"
My intuition is that type-safe languages with fast compilers are the best option. Maybe Go? I personally prefer Java just due to my experience running it in production, but am not sure there's many arguments for it over Go in a greenfield application. The other candidate would be Rust, but I worry about token efficiency and tool performance, I suspect it's not worth it for the runtime improvements.
All that being said, in this article switching to Python seems like a wild choice. Relatively poor performance, no compile time checking at all. Python's big selling point was developer ergonomics, which seems largely irrelevant now.
These are all just thoughts at the moment, I should try to find some evidence one way or another.
> The model can often avoid the mistake before the compiler ever sees the code.
> The type safety we gave up hasn’t been noticeable in any concrete way yet [...]
> Type safety can be a huge advantage for LLM-generated code if the compiler is helping the agent converge quickly.
Well, good to have this question cleared up once and for all :-)
We need more general purpose Elm languages in the space.
- haskell exceptions and laziness are devastating for production
- too small ecosystem, had to write 10+ SDKs (now with AI that's less of an issue)
- haskell ecosystem is too fragmented due to prima donna prevalence
I find Ruby a very beautiful language, and Rails is an excellent web framework, but I need typed functions, record types and sum types.
They help not just with correctness, but also as living documentation that lets me understand AI generated code. TypeScript provides discriminated union, but not exhaustive pattern-matching, and its syntax is a bit verbose, but since I'm no longer writing most of the code myself, I can live with it.
However I can't imagine using Python or any other dynamic language going forward. There is likely good reason for you to choose it, and I'm curious to know what that is.
LLMs have made me move away more from python rather than into it. I'm very surprised by this experiences of the author. The article is all over the place as well. Going basically all in on Python because it is apparently better than Haskell for LLM use and than agreeing with someone that says Rust is the best.
Who cares about performance.
Another obvious point is that an industry that runs on code slop will stagnate in terms of language an human tooling design.
Everything sort of exists, but it's this heinous zero documentation, high pain tolerance thing: buck2 and RBE with NativeLink and hooking that up to action runners and it needs to all work in a container or on nix or in a deb and on MacOS, you hand roll the auth and the certs and where do your compilers come from, can it do NVIDIA, can it do mobile.
Problem is switching off Haskell doesn't help for long: the agents proliferate and you're back where you started with more bugs. So I've been sucking it up and getting all this shit one click and it works. This is good enough for my use, and if the Scarf folks want a solution and are willing to work with a garage band startup, I'd be open to doing a closed alpha. I have a buck2 where you write the rules in Haskell (if you even need to change the prelude, it ships with a WASM that isn't coupled to fbcode), and the Nix cache/substitutor is backed by NativeLink so it scales to anything and it speaks all the protocols correctly and with a verified supply chain.
I'm not even really sure this will become a product, I just need it, but I sort of suspect others will need it too. If there's interest I'll put up a landing page with an email sign up thing.
That is: Have a Haskell base system. Have a Python "development" version on which you iterate at lightning speed. But also, in the background, moving at whatever pace it takes, have an agent running that imports all the Python development changes into the Haskell version. Have nightly builds of the Haskell version to reap its benefits (issues caught by the type system, more efficient native code). They must have continuous or nightly processes to fix bugs in the Python code anyway, there is no way that all the things they ship "while still on the call with the customer" are always tested on the full test suite and always 100% correct.
And it doesn't have to be Python/Haskell of course. The "development" version could be a (hypothetical?) interpreted Haskell. I have no idea if ghci would be useful for this. Neither do I know if the 15-minute Haskell build time is spent in the frontend (so an interpreter would have to pay that cost too) or in code generation or linking (which the interpreter wouldn't need to care about). Anyway, these are things I would think about before I did what the OP did.
I reckon language choice matters more at the edges of economic activity where a specific language feature really does make the difference in the end product, but most activity that is leveraging LLMs now is more generic enterprise SaaS software.
Or if staying on Linux, JVM snapshots.
this is undoubtedly true.