Zig's Incremental Compilation Internals(mlugg.co.uk) |
Zig's Incremental Compilation Internals(mlugg.co.uk) |
> Semantic analysis is the most difficult part of the compiler to handle incrementally. Perhaps unsurprisingly then, this is where language design starts to matter a lot: while I am pretty confident that most modern languages could support incremental compilation similar to how we do, certain design decisions can make that much more difficult. Zig has had its design tweaked over the years (sometimes controversially) specifically so that it is easier to support fast incremental compilation.
This is something I wish that we had done with Rust. It is impossible to do all of the things at once, though, and we already had a tremendous amount of things to do. This is also part of the "when do you ship 1.0" tradeoff; for our goals with the language, 2015 was the right moment to launch, but if had a few more years to bake things, maybe we could have made compile times way faster. Software engineering is hard.
I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them is 100%, all of them give up something in exchange for what they offer, and different programmers have different preferences for the compromise they prefer, and even that preference is context-dependent. Which of those less-than-100% memory safety compromises is the table stakes? And given that all of these compromises require something that could be quite substantial, depending on the circumstance, in exchange, and consequently programmers with the highest level of knowledge and expertise choose every one of those four in different situations, to me it seems pretty obvious that none of these is "table stakes".
I don't think I agree with this framing. The question to me isn't "what can you do while being memory safe", it's "can you accidentally do something memory unsafe without noticing?" Rust and Java are the same here; you need to explicitly opt into using the language's mechanism for relaxing restrictions (Rust's `unsafe` blocks, Java's `Unsafe` class APIs), whereas from what I understand, neither Zig or C offers anything strict in that way.
Zig has an identical memory safety profile to C. It has facilities to make it easier to stay memory safe, but those facilities are basically equivalent to what you have in C++ and that's equivalent memory safety profile as C.
> So among these four languages we already have four levels of memory safety, none of them is 100%
No, you've pretended like there's four when really it's Java / Rust which are safe by default and Zig/C/C++ which are unsafe by default.
One effective metric to evaluate is memory safety per LoC. Rust is ~0.2 vulnerabilities per MLoC. Java is effectively 0. C and C++ both seem to be about 1,000 vulnerabilities per MLoC. Zig is too new and hasn't had any analysis done on it, but generously it's likely at least 10-100.
So the table stakes could be defined as 1 memory safety vulnerability per MLoC.
https://github.com/rust-lang/rust/issues/2369#issuecomment-1...
This would already help a lot.
I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that is relatively low.
Of course there's always desire for more:
GHC currently needs to parse+typecheck+codegen a file before it can process other files that import it. Codegen is slow. Thus, there's currently demand split compilation into "stages", so that the next file can be typechecked after its imports have been just typechecked (not codegenned).
I would also enjoy if recompilation avoidance were to happen at the function level, not the file level.
Macro systems are a key language feature that can destroy incremental recompilation. In theory, Haskell is well set up for that, as its macro system (TemplateHaskell) is fully AST based and _theoretically_ could distinguish "fully pure" macros from side-effectful macros (such as splicing the current git commit in as a string literal). But the recompilation avoidance system does not currently exploit such differences.
An example that's being funded right now: https://rust-lang.github.io/rust-project-goals/2026/expansio...
We can always compile with full optimization just before shipping?
Zig doesn't have the same safety guarantees, it's on the dev to use safe coding patterns, so the tradeoff for safety is discipline or experience.
In particular, Rust made several good design decisions around this stuff that keeps those checks fast, like keeping checks local rather than being global.
Maybe at some point in the future zig could add a rust compilation target ( like with `-ofmt=c` )...
I understand that for a release mode a single giant binary may be desirable, but I am struggling to understand this design for debug builds. Moreover, while reading this article, I found myself wondering what happens if the main binary becomes corrupted. Maybe the user cancels compilation with ctrl+c while it is patching the binary. Even if they have a story for avoiding and/or detecting corruption, it is simpler to not patch in the first place and always generate a new main binary. Again, this is reasonable because the new binary is mainly just a list of shared libraries to link which will not take up much space and can be written to disk quickly. Moreover, this process can be done recursively, e.g. at the subdirectory level, so that during incremental linking a few quite small shared libraries may be produced rather than patching in the new code and writing cascading relocations.
(100 shared libraries take 6ms/process, which is a lot better (0.9ms for 1 library, for reference), but, especially with in-place patching, it has a good shot at being better than the overhead from dynamic linking)
Incremental compilation generally already depends on its stored intermediate data not getting corrupted, and the final binary need not be any differently handled in that aspect.
Using this native (written in Zig) C compiler to translate C source into Zig source as a part of the build, would presumably lend itself trivially to all the incremental logic in TFA, as updating C would update the generated Zig, and the incremental logic would detect differences just like it detects differences made by a human in an editor. Maybe there are aspects of the generated Zig that would complicate that somewhat, but I don't know -- just a warning about my ignorance.
This is part of plans to remove the hard LLVM dependency. AFAIK, the LLVM dependency will still be a variant many will use for the convenience of Zig as a much better clang, but removing the hard dependency is part of enabling all these great features like incremental compilation.
TL;DR: only debug builds for now, could extend to release builds once we have our own optimisation passes one day, but some optimisations will still be inapplicable.
Getting incremental linker to work with llvm is kinda hard atm. Maybe the zig team has a plan dunno.
That's not planned AFAIK (see https://github.com/ziglang/zig/issues/16269). `translate-c` is really only intended for header translation, not C source code.
See https://github.com/ziglang/zig/issues/20875 for the (not fully fleshed out yet) plans around C compilation.
The main issue is that so far such tools haven't been a priority for Rust.
There's also stuff around name resolution.
Proc macros are just an inherently very slow way to do what they do.
Because Rust commits to the traditional compilation model and pipleine (which I think is overall a good thing, or at least, a good thing to support), it does a lot of work that will eventually be thrown away. Consider this example: I have a library with a function foo that returns a simple 42. I have a binary which calls foo from that library and prints the result. Now imagine the library is a hundred thousand lines of unrelated code to what the binary needs, but is useful for other people. Because compilation works in the "produce libraries, produce binary, link them all together" style model, you have to compile the entire library with all of that code, when all you need is really one function. That intermediate work is useful, and I'm picking an example that's deliberately extreme, of course.
There's a bunch of stuff like this, and I do not have time to really say more than that right now. But yeah, monomorphized generics also produce a lot of compile time pressure too, in various ways.
Anyway I just also want to reiterate a few things: first of all, all of these decisions were made for good reasons, and there are pros to what Rust does and why. It's just that compile times suffer because of it. What I wish was that we had taken compile times into more consideration when deciding what to do and why in a more serious way. The same decisions might have been made, but at least it would have been known, rather than the situation now, where there's just a tremendous amount of work to try to optimize what exists, rather than having the freedom to maybe tweak some things to make that job way easier.
I feel extremely validated reading this! I've had a pet theory for a while that compile times would be drastically reduced if every single item in a crate were implicitly under a cargo feature flag and then they only got enabled if they were imported (and used in non-dead code). I know that making something like that work isn't anywhere close to as simple as I'm describing it, and there are probably a million edge cases, but I've long felt that the ergonomics of Cargo features basically making it too annoying to expose everything conditionally (and then transitively expose all of the features from all of the direct dependencies as well so that things depending on your library could also only conditionally enable them) is secretly the reason that people think Rust compile times are slow, and seeing someone who has way more direct knowledge than me of how all of it works under the hood give a similar take makes me more confident that I might have been on to something all along.
Now, that's ridiculous, but something not too different happens to me with Rust. I reach for a low-level language when I want to do low-level things in a more convenient way than in Java, but the very things that would make me reach for a low-level language in the first place are unsafe in Rust. So in ~100% of the programs I want to write in a low-level language, Rust and Zig offer the same level of memory safety (but I need to pay a higher price for Rust). That Rust reminds me that what I want to do is unsafe doesn't help me.
Of course, other people may want to reach for a low-level language in other situations and their perspective could be different, but if I pay the price and get little in return I can't see how that would be "table stakes". Table stakes imply some universality that is obviously not here.
Honestly, that's exactly how I feel in reverse. The framing you gave is more intellectually interesting, but it doesn't help explain the actual real-world outcomes where in practice, Rust and Java both don't have much problem with unsafety, whereas C does, and at least from what I've heard, Zig does as well.
> I reach for a low-level language when I want to do low-level things in a more convenient way than in Java, but the very things that would make me reach for a low-level language in the first place are unsafe in Rust. So in ~100% of the programs I want to write in a low-level language, Rust and Zig offer the same level of memory safety (but I need to pay a higher price for Rust). That Rust reminds me that what I want to do is unsafe doesn't help me.
I mean, sure, if you want to do things that are fundamentally not possible to validate because you think you're smart enough not to screw up, that's going to make Rust a tough sell. My issue with it is that history has shown that the best C and C++ programmers in the world still write code where memory safety rears its head, so I'm distrustful of the claim that being smart and diligent is enough to prevent the sort of bugs that we're still dealing with after half a century of us learning how not to write C. You need to have an excess of either talent or hubris to consider that a reasonably safe path, and given that the amount of talent needed is a lot higher than the amount of hubris, it seems way more likely that it's the latter.
The alternative is just learning how to write code that doesn't require expressing things in a way that can't be validated. While there are some things that fundamentally are not possible to, I'm dubious that it's anywhere close to as high as you seem to expect if your experience is that you literally can't reduce the amount of unsafe code you need in Rust below "literally my entire program is unsafe".
I'm not interested in the definition so much as I am in calling it "table stakes", and so the fact that these languages satisfy their promises is uninteresting in isolation. What matters is the value of their promises. The majority of Rust programs I see, I wouldn't have written in a low-level language, so the fact that it offers memory safety for the things I don't need it to do does nothing for me.
Now, clearly, Rust's originators didn't consider what Java offers (or at least what it offered 20 years ago when Rust was first conceived) to be table stakes or they wouldn't have wanted Rust. Java exacted some price in exchange for its memory safety that was unacceptable to Rust's originators and trumped its memory safety. But the same thing happens with Rust vs Zig. Rust exacts a heavy price for its memory safety, that - just as in Rust's case vs Java - is sometimes unacceptable. So I can't see how any of these could be "table stakes".
> I mean, sure, if you want to do things that are fundamentally not possible to validate because you think you're smart enough not to screw up, that's going to make Rust a tough sell.
What Rust can validate and what can fundamentally be validated are two very, very different things. Compared to what ATS can validate, what Rust can validate is almost indistinguishable from C. In Rust you have to do lots and lots of things that require you to be "smart enough not to screw up" that you could prove in ATS, and still no one (including Rust programmers) would say that what ATS offers is "table stakes" because, obviously, it comes at a high price that the people who choose Rust don't want to pay.
So clearly different languages offer different capabilities and charge a price for them. Sometimes the price is worth it and sometimes it isn't.
> so I'm distrustful of the claim that being smart and diligent is enough to prevent the sort of bugs that we're still dealing with after half a century of us learning how not to write C
But Java or Rust programs still suffer from a lot of bugs that ATS could eliminate, if you're willing to pay the price, and you're clearly unwilling. ATS programmers could say about Rust programmers what you say about C++ programmers. Clearly there's no universal table stakes here.
They really don't. Look at how many basic data structures (in the standard library or outside it) require unsafe features in Java vs Rust.
> Zig has an identical memory safety profile to C
It really doesn't. Zig gives you the same spatial memory safety as Rust and very much not like C (and violations of spatial memory safety are a bigger cause of vulnerabilities than violations of temporal memory safety).
This is a fundamental misunderstanding of how "unsafe" code relates to a platform's trusted computing base. Rust could move all of those unsafe data structures out of the standard library and into the compiler itself, thereby reducing the amount of occurrences of the string "unsafe" in the source, code, but this would do nothing to reduce the size of the trusted computing base that Rust presents. In fact, it would decrease our confidence in that code, because Rust libraries have a robust ecosystem of tools for validating their correctness, unlike whatever IR the Rust compiler itself is emitting. Java's own data structures are implemented by a runtime written in C++, which forms their own trusted computing base.
You have just described six orders of magnitude in your attempt to rebut pron pointing out the four languages have four levels of memory safety.
They aren't necessarily, though. Supposing that Zig were "10 issues per MLoC" (with just as much handwaving as the original poster), it would be equidistant from Rust and C. Java may also be more than one order of magnitude away from Rust; we say ~0 but is it 0.01, 0.001, 0.0001...? And why is "1 issue per MLoC" the acceptable metric that delineates what constitutes table-stakes memory-safe language? Because it's a nice, round-sounding number? I think 0 is a nicer, rounder number than 1, so let's call only Java table stakes and condemn all other languages to the garbage bin, tradeoffs be damned. Or would you say your arbitrary delineation point worth more than mine?
What I do not like, primarily comes down to how the project is talked about and marketed. First, because it promotes an "us vs them" mindset, instead of a "we're all trying to improve memory safety" mindset, and second, because in doing so, it also overstates its case.
These things are sort of intertwined. Let's talk about the overstatement first. Fil-c has its own definition of memory safety that is slightly different than others. For example, I saw this recently:
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
struct User {
char name[8];
int is_root;
};
int main(int argc, char*argv[]) {
struct User* user = malloc(sizeof(struct User));
strcpy(user->name, argv[1]);
if (user->is_root) {
printf("I am root!\n");
} else {
printf("I am not root :(\n");
}
return 0;
}
But beyond corners like this, fil-c's author will go on about "Rust has unsafe as a hatch, fil-c does not" while if you control-f for "zunsafe_" on https://fil-c.org/stdfil you get ... escape hatches.
The author regularly erases the difference between "traps at runtime" and "is prevented at compile time", which are legitimate tradeoffs where one or the other may be better depending on what you're doing. But they're presented as either equivalent, or one is superior, and I find this muddles the discourse. The performance issues also tie into this, "add a GC" is absolutely a valid way to handle these sorts of issues, but it is not the same thing as what Rust does. And that's okay! But presenting it as purely superior means that it's just hard to talk about.
Speaking of muddling the discourse, the author regularly trolls on X, providing tons of bad faith arguments and generally trying to rile up a "fil-c vs Rust" war that I think reduces our ability to talk about these differences in a calm, engineering focused context.
Finally, due to its design, fil-c is effectively Linux only. That's great for Linux, but many people also use other systems, and so it is not a meaningful option for them.
Anyway, after saying all that: I still think that it is a good project, and that it should exist and continue to be worked on. I just wish that the heat was turned down, and people could talk about the various approaches and their tradeoffs without turning it into a culture war.
But you did the same thing when, on the spectrum that ranges from C to ATS, with Zig, Rust, and Java somewhere in the middle (though all closer to C than to ATS), you declared the exact compromise that Rust makes "table stakes"! [1]
Zig improves on C's memory safety when it comes to spatial safety, possibly the more impactful kind, so it, too, could be part of the "we're all trying to improve memory safety", yet you exclude it.
You're trying to draw some hard line that passes exactly between Rust and Zig on the C to ATS spectrum, and I'm trying to say that that line isn't there (your attempt at a definition of delineating safe and unsafe code also applies to C). Obviously, C, Zig, Rust, Java, and ATS all make very different tradeoffs, all of which may be more or less attractive to different people and in different circumstances, but there is no sharp line, at least not one that is meaningful enough to be "table stakes". Your personal inclinations place a premium on the things Rust offers and Zig doesn't while mine are the opposite, but I make no claim to universality.
I'm happy to accept that not everyone shares my aesthetics and can understand why some people prefer Rust, but those claims to or hints at universality annoy me (as they did when they were made by Haskellers, and I actually find Haskell's aesthetics quite pleasing), as they are simply unsupported. I've spent a lot of time studying formal methods and software correctness in general (https://pron.github.io) and if there's one thing we know in that field is that things are never that simple (and, bringing this back to this posts topic, even something like incremental compilation can contribute to program correctness).
(Now, you may argue that you're only talking about "memory safety" and not correctness in general, but what gives memory safety value is that violations are causes of many dangerous vulnerabilities; but once, say, Java eliminates all of them, 100% of bugs/vulnerability - which are still numerous - will be caused by other problems, all potentially avoidable with ATS, so why isn't ATS table stakes? Of course, the answer is cost, but all the languages on the spectrum differ in their costs.)
[1]: I assume that you meant Rust's compromise, because you implied that Zig doesn't pass that bar but Rust does.
I do not go around posting "omg Rust is SO MUCH BETTER than zig or fil-c, which are TRASH." I talk about engineering tradeoffs, and what matters to me personally. I do not say "if you use Zig, you are a bad person." I am not saying that any comparison is bad. I am saying that the way that the comparison is presented is bad. That is different.
> you declared the exact compromise that Rust makes "table stakes"
Table stakes for me.
> Zig improves on C's memory safety when it comes to spatial safety,
I agree that it's an improvement on C!
> yet you excluded it.
I said that it is not pursuing a design that I personally find compelling enough to use to write software. That doesn't mean that I think it's worthless. This whole thing started off with me talking about how much I respect the Zig project! Yet you're trying to turn this into something where I'm talking shit. I presented a specific technical tradeoff that is important to me. That is very different.
> You're trying to draw some hard line on a spectrum that passes exactly between Rust and Zig, and I'm trying to say that that line isn't there (your attempt at a definition of delineating safe and unsafe code also applies to C)
I don't believe you've shown that. And my "attempt" does apply to C: it fails the bar, because it does not delineate between a safe subset and an unsafe superset.
> you may argue that you're only talking about "memory safety" and not correctness in general,
I am in fact talking about "memory safety" and have been this whole time, yes.
> what gives memory safety value is that violations are causes of many dangerous vulnerabilities; but once, say, Java eliminates all of them, 100% of bugs/vulnerability - which are still numerous - will be caused by other problems, all potentially avoidable with ATS, so why isn't ATS table stakes?
This is just an entirely different question. Yes, there are other forms of safety that are important too. That's just not what we're talking about here.
It is absolutely possible that one language might actually have an objectively better approach to memory safety than another, and in such cases it is usually possible to argue for this using sound technical or empirical arguments. But the way the author of Fil-C presents their arguments it often comes across in a kind of antagonistic manner, like he has a chip on his shoulder.
Incidentally, you might want to look at gc-sections, which Rust already does. As well as https://rust-lang.github.io/rust-project-goals/2025h2/relink... which is kinda related.
The sort of key here is understanding that "produce a library" means that every public item is "used" in the sense of "do we need to compile it." The real trick is to do demand-driven compilation starting from the actual final program's needs, and this is inherently at odds with the idea of producing standalone libraries and combining them into the final artifact.
It is true that for various reasons, Rust can't take as much advantage as say, C can.
> I'm guessing this is related to what you mean by potentially being able to do something differently if there were more time before 1.0?
I mean more traditional language design things, but sure, this too.
this is not so far off from hint-mostly-unused, no?