Why is Arrays.fill 265 times slower on G1GC?(krzysztofslusarski.github.io) |
Why is Arrays.fill 265 times slower on G1GC?(krzysztofslusarski.github.io) |
Slightly less annoying summary from ChatGPT free: https://chatgpt.com/share/6a9ac7a3-15a0-83eb-8c2a-6f72cd9beb....
Caveat emptor: it makes high level sense, but I haven’t thought about it in detail.
Could it be human? Sure. But it doesn’t seem likely to me.
I'm already at the point of "the reward/effort of HN is getting pretty low", but the question is, where to leave for?
1. Upgrade your JDK for the best performance (as the article says, the slowdown is gone in JDK 26).
2. Don't try to help the GC by pooling objects. Mutating old objects can be expensive, while allocating new ones is cheap (at least for objects that don't do some exceptionally expensive initialisation).
There are also middle ground options, like pooling objects but giving the pool a lifecycle that is tied to a request.
In particular, the JDK's GCs are heavily optimised for short-lived objects with high allocation rates. What you want to avoid is temporary data finding itself in the old gen. The newer GCs may dynamically size the young generation to match your program's natural meaning of "short lived", but the longer an object lives, the higher the chances it ends up in the old gen. You want only objects that stick around for a very long time (and have a low allocation rate) to end up in old gen. If you just write naive code, chances are things will work out well. Once you start being clever, you're taking a risk.
So if you're willing to profile your program, with a workload that's representative of production workload (a microbenchmark is useless) on every runtime version and potentially change your "manual optimisation" every six months, you can try. But if not, the advice for the best performance over time is to rely on the platform and let it do its thing. The reason is that the JVM is continuously being optimised for "normal programs". If you're doing anything too clever, you may find that in a future release, your code is making things worse because the optimisations that target normal code don't help your code (or could even treat it as unusual and have it hit slow paths).
I once spoke to a company that were very proud in getting something like a 10% improvement over naive code in Java 8, thanks to some hand optimisation they worked a lot on, only to discover that it caused a 15% regression compared to doing nothing special on JDK 11.
The parallel collector is a perfectly fine collector, particularly for smaller heaps. Even the serial collector isn't bad for things like a containerized environment, yet G1 replaces it by default now [1].
It's not a bad algorithm, but especially when you start talking about sub 2G environments I've not seen a situation where the parallel and serial collectors won't handily beat G1 on pretty much every metric. Major collectors with modern CPUs just doesn't take much time for a lot of memory.
The benchmark creates an array in the old generation (by being big enough) and stores an object (allocated in the new generation). This triggers the GC barrier for every writes. Something rare in real application.
The G1 barrier before Java 26 is slow because:
- the GC barrier and some GC threads do concurrent operations on the same memory zone (the card table)
- the barrier is big (a lot of assembler instructions) so it also troubles the loop unrolling optimization performed by JITs
Parallel GC has a simple barrier and do not care about latency (no GC check inside the loop).
The barrier implementation of G1GC was changed in Java 26, so update your Java runtime version and move on.
tl;dr: In JDk 25, filling a large array of references with objects living in a different heap region is extremely slow when using G1 GC as opposed to parallel GC. Solution: Move to Java 26. Or increase G1HeapRegionSize.
Details:
They used Amazon Corretto as JDK. When Arrays.fill() was called with UseG1GC and UseParallelGC, the former was really, really slow.
Then they go on to give an ARM64 primer (because they ran it on an Apple M4 Max).
The part where they mention g1BarrierSetAssembler_x86.cpp and g1BarrierSetAssembler_aarch64.cpp is where at least some readers get lost. They are essentially saying that the logic (for both x86 and ARM64) is the same. The reason why only those two files matter for this test, I believe, comes down to JEP 304[1]. Just accept it and continue reading. (And validate it later if you want to.)
They list around twelve lines of ARM instructions that matter for this test - cheat sheet, essentially.
The three lines of actually generated ARM instructions are shown then for parallel GC (the much faster one in this case).
They explain what a write barrier is (basically GC bookkeeping). So as a result, there are three instructions, but they are neat and tight. Also, we see that eight elements are filled in each iteration of the loop which fills the arrays.
Then, they show what was generated for G1 GC. And it is indeed too long. Around 20 instructions. However, some of them shouldn't have been executed because of three exit conditions. They explain why none of them fired.
They prove that by subtracting eight from the humongous threshold size, it is allocated to an Eden/young region and the problem vanishes because one of the three exit conditions is triggered thus.
Then, they show that it is a problem that increases almost linearly (my guess, based on the numbers) with the array size- so the larger the array size is, the more the time delay!!
Finally, the solution: JDK 26 took care of this issue. They also point to JEP 522 (I didn't go through it yet). If you have an older version, use the G1HeapRegionSize flag if you can do so.
That is it, essentially. Definitely long, but well-written, based on actual testing.
Removing allocation pressure can also have effects on other parts of the system, but that is anything but trivial to measure or reason about.
I just have a problem with G1 because in my experience, the best place for it is fairly large heaps. Get something sub 2 or even 10G, especially if you have a few cores to offer, and the parallel and often even the serial collector will give G1 latency even on major collections with superior throughput and overhead.
In the modern enterprise Java world (that I've been exposed to) it's very common to have a mandate that all components deploy a minimum of N instances across X regions for resiliency. By design that almost always means you're deploying at least 2x more compute than you strictly need, so the top priority is generally minimizing per-instance overhead to minimize cloud spend.
For example, the default templates at my current company deploy something like 0.25-0.5 vCPU per instance, and therin lies the rub. ZGC performance is _catastrophically_ bad with <=1 cpus because when there's only one core, any "concurrent" GC events become full on stop the world events. We had someone pilot a change to the default JVM args for all components because they heard that ZGC would reduce latency, only to discover that basically all of our microservices immediately failed their perf tests. For the first one I spot checked, throughput was down ~90% and p95 went from ~40ms to >1s, because more time was being spent on "background" GC than actually servicing requests.
Hope that didn't come off adversarial. I just find GC fascinating, and ended up spending a bunch of time working with the team that owns those defaults to draft general recommendations. TLDR is that when in doubt don't specify/let the JVM pick for you, and don't be surprised if it picks serial :)
That throughput penalty is not very high with generational ZGC. It's not zero, but it's not very high, either. What ZGC mostly does is spread the memory management activity more evenly across the duration of the program (this does have a cost due to barriers being active more, but it's not huge). But we have some work planned to improve ZGC even further, which is why I didn't say I think it will become the default imminently, only eventually.
> ZGC performance is _catastrophically_ bad with <=1 cpus
That may well be true, but the JVM can automatically choose a different default algorithm for these circumstances. Indeed, until very recently, the default for low-CPU environments was different (Serial) than for bigger ones (G1).
> So the difference has to be in what the JIT generated, and the profiler gives us exactly that.
> That is the whole vocabulary. Let’s read some code.
> Decoding the x86 version instruction by instruction is out of scope here.
> What is not architecture specific is the logic.
Here’s a segment flagged by Pangram: https://www.pangram.com/history/87e25169-30a4-4030-a65a-dba8...
But I absolutely think it is bad for the reader, and deserves to be called out. Authors should know that it’s not good enough.
"If you are sure the link is correct, please make sure you are logged in with the account associated with this result."
The summary I shared is much more straightforward. It’s mediocre, but just barely good enough to extract the message without making me super annoyed.