JIT Compiling Code in 5μs(malisper.me) |
JIT Compiling Code in 5μs(malisper.me) |
> The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile.
That's only true of writing a JIT from scratch. There's no rarity of JITs, it's just that LLVM (and other frameworks) are often used. Every major interpreter has a JIT compiler. PCRE2 has a JIT compiler. There are JIT frameworks out there with much faster code-generation than LLVM: Cranelift, GNU Lightning, Mir. I doubt they could do code-generation faster than a custom copy-and-patch JIT, but they'd be much faster than LLVM.
[0] https://www.pinaraf.info/2024/03/look-ma-i-wrote-a-new-jit-c... , discussed: https://news.ycombinator.com/item?id=39742916
Template-based code generators suffer from bad code quality due to missing register allocation.
Our TPDE-based compilers compile a bit slower than template-based code generation but the generated code is much smaller and faster ([2] Fig. 2). Also for database workloads ([2] Fig. 6).
All that said, Postgres' main limitation is that it (IIRC) only compiles single expressions from operators, not pipelines. This fundamentally limits the achievable performance improvement compared to databases that perform more extensive query compilation.
[1]: https://aengelke.net/pubs/2403-cgo.pdf [2]: https://aengelke.net/pubs/2602-cgo1.pdf
PS: sorry for the promotion of my own research here, just couldn't resist.
> Not sure where the idea comes from that Cranelift is much faster than LLVM -O0
Cranelift describes itself as a fast, secure, relatively simple and innovative compiler backend. [0] Interesting that LLVM can compete there, with its optimisations dialed down.
> Postgres' main limitation is that it (IIRC) only compiles single expressions from operators, not pipelines. This fundamentally limits the achievable performance improvement compared to databases that perform more extensive query compilation.
That sounds pretty limiting. That's separate from query optimisation though, right? The query optimiser is presumably able to reason 'broadly' and not just at the level of individual expressions? High-level query-plan optimisation must be much more consequential than effective use of JIT compilation.
On reflection I wonder if I overstated the widespread use of JIT and of JIT compiler frameworks. All the 'major' well-resourced high-profile JIT-based interpreters I can think of don't use an off-the-shelf JIT framework for their backend, which makes sense as they want to carefully tune the code-generation. OpenJDK, OpenJ9, .Net, V8, SpiderMonkey, JavaScriptCore. LuaJIT and Python's new JIT don't use one either, nor does the Linux kernel's BPF engine.
The Guile Scheme interpreter uses a fork of the GNU Lightning JIT library. [0] Julia and (as mentioned) Postgres use LLVM for their JITs. I'm trying to think of other projects that use a JIT framework/library.
Similarly, I can't think of many problem domains where it makes sense to use JIT. The ones that spring to mind are interpreters (of course), regex engines, and DBMSs. JIT can also help in high-performance computing, to tailor the code to the particular problem and the particular CPU. [1] I don't think there are many other contexts where it makes sense to use JIT though.
JIT compilation brings its own drawbacks in portability (both between hardware platforms and operating systems), complexity, and perhaps cybersecurity, which might also limit its adoption, even if a good JIT framework could help with all three.
[0] https://doc.guix.gnu.org/guile/latest/en/html_node/Just_002d...
[1] https://www.intel.com/content/www/us/en/developer/articles/t...
Except that using LLVM has high latency limitting it's applicability. Postgres just disabled LLVM by default because of this[0].
[0] https://www.postgresql.org/message-id/E1w8GWU-002bSL-31%40ge...
There's a strong 'diminishing returns' effect in striking a balance between compile time and the performance of the generated code. I'd expect a more lightweight (less optimising) JIT engine to be able to produce code with pretty respectable performance while taking only a fraction of the time that LLVM takes. There's a follow-up to the blog post I linked above, which bears this out. [0] (I don't know if that JIT solution is production-ready or viable for merging into postgres, mind.)
The blog post [0] gives this performance comparison:
> So, on our stupid benchmark, doing 10 times a simple SELECT * FROM demo WHERE a = 42 on a 10 million rows table...
PostgreSQL No JIT LLVM JIT Copyjit
---------- ------ -------- -------
Average time (ms) 120 106 (-12%) 101 (-15%)
Compilation time (ms) 0 19 0.06
Instructions 13,350,766,209 10,643,820,667 (-21%) 12,769,013,536 (-5%)
Cycles 4,660,821,596 4,005,881,863 (-14%) 3,924,602,439 (-16%)
Branches 2,322,470,659 1,798,221,785 (-23%) 2,031,456,214 (-13%)
It was the limits of 8 bit home computers hardware that made the interpreter version be more widely known.
Same to Lisp, Smalltalk, and many other languages.
Fully agree with you.
It is very relevant
I might use this approach to generate the stencils for a JIT firewall I’ve been experimenting with.
It also occurs to me that this could be used to generate eBPF byte code on the fly as well
Besides run time, JIT is available also when the code is compiled or loaded for execution (i.e., do you have a compilation or loading speed-up in mind? no problem, you can also compile that speed-up into native machine code, and so ad infinitum...).
Also, on my machine, compiling the identity lambda form takes about 200 usec with (optimize (compilation-speed 3) (debug 0)). SBCL could use a faster JIT mode, perhaps at compilation-speed 3/speed 0. Perhaps there are some other internal special variables that could be tweaked to reduce compile time.
[0] https://www.sbcl.org/manual/#compiler-only-implementation
By not using LLVM, you're missing all the optimizations it does.
LLVM is a large dependency if you need to JIT. There are plenty of smaller (and much faster) alternatives which are much better fit for smaller projects. Larger projects usually roll out their own jit-pipeline because they can integrate better with the source language/interpreter and apply tricks LLVM is not well suited to (say, LLVM is not great at deoptimisation). I think only Julia is really a heavy user of LLVM JIT, also it is known for extremely slow repl from time to time.
That's not exactly blazing fast for a low level C-like language, but it's not bad. It's infinitely faster than what I've ever gotten a toy interpreter to be.
Some human, somewhere, has to describe how to turn high-level language constructs into machine code. "When you see this pattern, emit this sequence of bytes." That's just templates and stencils. There's no magic for turning source code into machine code by divining the ISA at compile time.
Anything that's taking source code and, at the time of execution, is compiling it to machine code on-the-fly is JIT compilation. Regardless how long it takes, lack of optimization, or which machine is the target (x86, ARM32/64, RISC-V, JVM, WebAssembly), it's JIT.
And yet, a good portion of software that runs today's world is written in scripting languages & executed using interpreters.
Which is okay! Imho: multiply [# of users] with [how often each user sees that software's effect] and [how much that contributes to the overall user experience], then you get a ballpark idea of how much $$/effort is worth spending on optimization.
In other words: for a one-off, don't bother. But as usercount, frequency of use by individual users, poor UX or RAM/CPU consumption goes up, progress from script -> compiled -> optimizing compiler -> (if necessary) hand-optimized assembly as needed. And of course consider high-level design, data structures, algorithms etc in that process. A change there might be more effective than a switch from interpreted -> optimizing compiler.
"Developer time" should not factor into that much (again: imho) unless users=developers.
Thoughtlessly putting every change through a (slow?) pipeline that does 'random' toolbox-of-optimizations without need, is wasteful. Apply that toolbox as needed while keeping the above in mind.
I believe this because every time I use AI for domains that I consider myself above competent, if it is anything beyond UI components or a simple CRUD endpoints, I cringe at the quality of what it generates.
This has made me to be extremely cautious of starting working in a new domain with AI if I want anything beyond throw away quick hacks or junk, shy of quick bug fixes perhaps.
Besides all the other mentioned points, I think a good remaining less-discussed point is that quality in software has always been in the eye of the beholder. You may very well see the AI output as low quality, I may not, and its not necessarily clear who is right or wrong, because there was always little precedent in objectively evaluating code quality.
This is a long standing issue, and was never resolved before AI happened, and the coming of AI has not really changed things, except that AI is under a magnifying glass obviously. How do we objectively measure the quality of code? There's some general consensus on things, but surprisingly little is true professional agreed upon consensus.
If I can make up a figure, I would guess 95% of software engineering quality rhetoric, and craftmanship advice, is just strongly held opinions.
This is not something I can prove, but if I look at the (still ongoing.....) debates on very basic ideas like clean code, and the reactions from also-great programmers like Carmack, Blow & Muratori, it is clear to me that there is little consensus on even the fundamentals of software design.
If all these people can produce excellent working software while disagreeing on these fundamentals (of quality), it means we do not yet understand what the fundamentals are.
It generally performs better if the tasks are broken done into small manageable pieces, and the person is actually reviewing and calling out problems, which usually requires the person to be a competent engineer in the problem domain to begin with.
But yes, I have personally used it to build what the OP calls a JIT. I would usually write that by hand and it would take me one week. The AI does it in an hour.
That’s not a joke. A lot went on to be programmers professionally. And judging by the quality of closed & open source code I witness daily those figures from university accurately depict people’s capabilities.
Now that said, if you can’t really code then using AI will be a godsend to said individuals.
In fairness, UI components are probably one of the hardest things to do completely correctly, even with just HTML. As soon as you start thinking about i18n, screen reader support, color contrast, keyboard controls, and all of the layout and positioning you're trying to achieve at different viewport sizes, it's extremely hard for a "just competent" engineer to do an S-tier job. Even with the most vanilla default built in components it's not easy to get this correct, and I think we all cringe at what competent engineers create by hand in this domain.
Also often the difficulty with writing code is simply knowing where to start - getting past the blank page. AI can help a lot with that. Often there's a task where I've got kind of writers block, but you can ask AI to do it and suddenly it's like "ah yeah, sort of but actually that's not quite right we should do it this way".
In a long run session Fable 5 generated a Disney principled (physically based) shading/lighting engine from scratch, both with a CPU (SIMD accelerated) backend _and_ a full GPU Vulkan backend. Exceptional performance too; the CPU backend runs almost realtime and literally looks better than some AAA games outright. Took it about ~8 hours wall time total time to achieve this.
Fun fact: even Apple themselves have JIT. JavaScriptCore on iOS has JIT, it's just that the App Store policies forbid any application submissions with JIT or trying to mmap/mprotect an executable region. There used to be apps on TrollStore that runs JIT
> - Android Runtime Just-In-Time (JIT) compilation/profiling is fully disabled and replaced with full ahead-of-time (AOT) compilation. The only JIT compilation in the base OS is the V8 JavaScript JIT which is disabled by default for the Vanadium browser with per-site exception support.
> - Dynamic code loading for both native code or Java/Kotlin classes is blocked for nearly the entire base OS. […]
> - Dynamic code loading for both native code or Java/Kotlin classes can be disabled for user installed apps via 3 exploit protection toggles: […]
The drawback to this approach is there can be some bloat because you can only mprotect at page granulariy so a jitted function that only takes say 10 bytes to represent would take up a full page in memory, but this is extreme and in practice, the overhead is unlikely to be worth worrying about.
Allowing code execution allows code execution, that's it, that's the entirety of it.
Alternatively, only allow for the execution of cryptographly signed static linked binaries, this naturally includes the interpreter above.
Mine doesn't have the SB-INTERPRETER package, so I doubt binding the variable has an effect.
One of the competing open Common Lisp implementations, CCL, has a much faster compiler (albeit one that produces worse code). This can be useful in development.
See https://github.com/marcoheisig/Petalisp#why-is-petalisp-writ...
Yes, yes, and yes. For databases, query optimization (esp. join ordering for larger queries, which heavily depends on estimates) is fundamental. Query optimization happens at the level of the query plan, JIT compilation is only relevant afterwards. A bad query plan leads to asymptotically worse performance (e.g., bad join ordering with huge intermediate results).
On query plan execution: The "classical" model as used in e.g. Postgres is a pull-based iterator model, where operators implement a next() method yielding the next tuple and in there recursively call next() on their child operators (e.g., a next() of a select operator calls next() on its child operator, then applies the predicate [what Postgres JIT-compiles], and returns the tuple if the predicate was true). This can happen one tuple at a time (Postgres) or "vectorized" where multiple tuples are processed at once (e.g. DuckDB). A query-compiling database will split the tree into pipelines and compile each pipeline as one function (e.g., a pipeline will iterate over all the tuples from a source (e.g. tablescan) and a select operator then becomes an if statement inside that loop). This results in pretty tight loops, avoids per-tuple dispatch overhead, and enables more optimizations inside the JIT-ted code (e.g., tuple values don't need to be reloaded from memory all the time). (I find the original paper on query compilation [1] to be well readable.)
the tell-tale mark of AI code is highly over-engineered local solutions to trivial problems that don't matter, or that were already solved better elsewhere and that no sane human would ever duplicate.
The JIT must also only be allowed to call into specific code, controlled by the runtime, and nothing else.
> Anyone who thinks AI is good with writing code that is hard to write for the operator, not due to lack of basic software engineering know how but complexity of the domain, either has access to models beyond what is available to the public or is completely lost.
You could read it as quality in the operational correctness sense, but just as well in the software architectural design sense. My comment indeed applies to only one of those.
However, why judge correctness as a "cringe on quality", rather than just objectively saying its producing errors. This is why my response is in the software direction.
Software is no different. So when someone claims that "you can't question the quality of AI output because quality of software is [partly] subjective", they're appealing to subjectivism. This position is generally held by the incompetent to entirely dismiss any question of craftsmanship or quality.
What if it was rejected on an objective basis? You can't know, so you have to err on the side of caution, but you didn't.
So it's not embarrassing at all. In fact, it is more embarrassing to double down without knowing more.
It's a fallacy because you can always say that the choice of an objective metric was subjective, so everything is subjective.
Absolutely, at the end of the day the only metrics that matter are performance and code validity. A lot of the "this code is awful" arguments I hear just boil down to "this code is stylistically awful" and never talk about it's performance.
JITs do not grant the ability to bypass any OS/system sandboxes. The lack of W^X doesn't do that, either. If a process opts out of W^X, such as to enable a JIT, it's voluntarily making itself less hardened, but at the end of the day this isn't any more meaningful than the program being allowed to be written in, say, C, which also voluntarily reduces the processes security hardening.
And my point about C is literally that even without a JIT, applications can still have arbitrary execution vulnerabilities.
A JIT intended to run untrusted code as part of a sandbox, like a browser, is a big risk. But that's because of the untrusted code part, not the JIT. By comparison, something like a Python or Java JIT is as near as makes no difference completely risk free. The JIT is working on exclusively "trusted" code. Same basic concept applies here with this database usage.
I used Fable on a Zephyr project with time sensitive code for LR-WPAN and it broke everything. Literally made the code worst to the point that the devices stopped connecting.
And even if everyone is using the same LLM model, quality varies on lot on who's driving it.
If I need to be a domain expert anyway, the value of the tool goes down by orders of magnitude. Same if I need to first break the task down into pieces and keep reviewing all the output. That sounds to me like >80% of the work I'd need to do anyway.
If I need to design and understand all of the code anyway, I might as well skip the whole process of repeatedly fixing the subpar-at-every-level LLM output and write it all myself.
Personally, I've found the greatest value in asking for simple tasks, like wiring up APIs, generating boilerplate, bug finding etc. Anything that requires effort to do but results in either very little or very simple output, so that I can easily verify its correctness.
But give the LLM anything remotely complex to generate and it cakes its pants.
You absolutely don't. You only need to be roughly aware of what the code needs to be doing. Similar to how a software architect historically didn't personally oversee every line of code in an org, only it's overall structure. The implementation specific details can be left to the AI.
In this example usage, one would hope that authentication already happened before the JIT processed the command. So an authorized user can attack themselves is the only realistic risk, which is hardly significant
There's plenty of scope for harm just within the process, even ignoring the possibility of escaping the process. In the case of a database server, essentially everything of value takes place within the database process (or processes). That process presumably has both access to the raw database data, and network access. We wouldn't want it sending data to an attacker's server.
> one would hope that authentication already happened before the JIT processed the command
We'd hope, yes, but SQL injection issues are still somewhat common. Also, an organisation might trust their DBMS to enforce permissions, and a JIT bug is the kind of thing that might allow non-permissioned data access. A DBMS should be hardened against malicious queries, just as a browser should be hardened against malicious JavaScript.
In web browsers, the numbers show JIT compilers are a major cause of security issues. I don't know if there are hard numbers on JIT engines causing security issues in DBMSs though.
Of course, but the process has every right to decide for itself if it wants to take that risk. Just like it decides if it wants to take the risk of a memory unsafe language, forgoing fuzzing, or going all out with formal verification.
I've also never claimed that "you can't question the quality of AI output because quality of software is [partly] subjective". I'm not sure how you got there. I am simply saying that "quality" is an ill defined term and this is why we might see so much divergence when discussing AI code "quality".
You can absolutely evaluate code quality objectively, e.g performance, bugs, crashes.
Only on operating systems that allow dynamic executable code.
To the OS those are all identical scenarios, it's irrelevant what caused the arbitrary code execution to happen.
Which at this point most companies would rather save money and forbid JIT altogether.
Note that mainframes and micros have JIT environments that aren't at the same safety level as regular desktop PCs.
For example,
https://medium.com/@dhemanthc/ibm-i-architecture-how-timi-an...
I am refuting that statement by showing that not allowing dynamic executable code is not enough. It is irrelevant to that end that you can mitigate ROP by other means.