Undefined Behavior in C and C++ (2024)(russellw.github.io) |
Undefined Behavior in C and C++ (2024)(russellw.github.io) |
Fixing the document is worthwhile, and certainly a reminder that WG21's equivalent effort needs to make the list before it can even begin that process on its even longer document, but practical C programmers don't read the document and since this UB was a "ghost" they weren't tripped by it. Removing items from the list this way does not translate to the meaningful safety improvement you might imagine.
There's not a whole lot of movement there towards actually fixing the problem. Maybe it will come later?
I would strongly suspect that C compiler implementers very much do read the document, though. Which, as far as I can see, means "ghosts" could easily become actual UB (and worse, sneaky UB that you wouldn't expect.)
But the original article also complains about the number of trivial UB.
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p14...
i.e., the UB already existed, but it was not explicit had to be inferred from the whole text and the boundaries were fuzzy. Remember that anything not explicitly defined by the standard, is implicitly undefined.
Also remember, just because you can legally construct a pointer it doesn't mean it is safe to dereference.
Including a header that is not in the program, and not in ISO C, is undefined behavior. So is calling a function that is not in ISO C and not in the program. (If the function is not anywhere, the program won't link. But if it is somewhere, then ISO C has nothing to say about its behavior.)
Correct, portable POSIX C programs have undefined behavior in ISO C; only if we interpret them via IEEE 1003 are they defined by that document.
If you invent a new platform with a C compiler, you can have it such that #include <windows.h> reformats all the attached storage devices. ISO C allows this because it doesn't specify what happens if #include <windows.h> successfully resolves to a file and includes its contents. Those contents could be anything, including some compile-time instruction to do harm.
Even if a compiler's documentationd doesn't grant that a certain instance of undefined behavior is a documented extension, the existence of a de facto extension can be inferred empirically through numerous experiments: compiling test code and reverse engineering the object code.
Moreover, the source code for a compiler may be available; the behavior of something can be inferred from studying the code. The code could change in the next version. But so could the documentation; documentation can take away a documented extension the same way as a compiler code change can take away a de facto extension.
Speaking of object code: if you follow a programming paradigm of verifying the object code, then undefined behavior becomes moot, to an extent. You don't trust the compiler anyway. If the machine code has the behavior which implements the requirements that your project expects of the source code, then the necessary thing has been somehow obtained.
i) "Fil-C is a fanatically compatible memory-safe implementation of C and C++. Lots of software compiles and runs with Fil-C with zero or minimal changes. All memory safety errors are caught as Fil-C panics." "Fil-C only works on Linux/X86_64."
ii) "scpptool is a command line tool to help enforce a memory and data race safe subset of C++. It's designed to work with the SaferCPlusPlus library. It analyzes the specified C++ file(s) and reports places in the code that it cannot verify to be safe. By design, the tool and the library should be able to fully ensure "lifetime", bounds and data race safety." "This tool also has some ability to convert C source files to the memory safe subset of C++ it enforces"
The resulting language doesn't make sense for commercial purposes but there's no reason it couldn't be popular with hobbyists.
Run your test suite and some other workloads under Fil-C for a while, fix any problems report, and if it doesn't report any problems after a while, compile the whole thing with GCC afterwards for your release version.
They at least fixed this in c++26. No longer UB, but "erroneous behavior". Still some random garbage value (so an uninitialized pointer will likely lead to disastrous results still), but the compiler isn't allowed to fuck up your code, it has to generate code as if it had some value.
In effect if you don't opt out your value will always be initialized but not to a useful value you chose. You can think of this as similar to the (current, defanged and deprecated as well as unsafe) Rust std::mem::uninitialized()
There were earlier attempts to make this value zero, or rather, as many 0x00 bytes as needed, because on most platforms that's markedly cheaper to do, but unfortunately some C++ would actually have worse bugs if the "forgot to initialize" case was reliably zero instead.
Access to an uninitialized object defined in automatic storage, whose address is not taken, is UB.
Access to any uninitialized object whose bit pattern is a non-value, likewise.
Otherwise, it's good: the value implied by the bit pattern is obtained and computation goes on its merry way.
We have zig, Hare, Odin, V too.
Because it never achieved mainstream success?
And Zig for example is very much not memory safe. Which a cursory search for ”segfault” in the Bun repo quickly tells you.
https://github.com/oven-sh/bun/issues?q=is%3Aissue%20state%3...
And with this attitude it never will. With Rust's hype, it would.
Ada would rather be a nice choice, but most hackers love their curly brackets.
(The B language was implemented for the PDP-7 before the PDP-11, which are rather different machines. It’s sometimes suggested that the increment and decrement operators in C, which were inherited from B, are due to the instruction set architecture of the PDP-11, but this could not have been the case. Per Dennis Ritchie:¹
> Thompson went a step further by inventing the ++ and -- operators, which increment or decrement; their prefix or postfix position determines whether the alteration occurs before or after noting the value of the operand. They were not in the earliest versions of B, but appeared along the way. People often guess that they were created to use the auto-increment and auto-decrement address modes provided by the DEC PDP-11 on which C and Unix first became popular. This is historically impossible, since there was no PDP-11 when B was developed. The PDP-7, however, did have a few “auto-increment” memory cells, with the property that an indirect memory reference through them incremented the cell. This feature probably suggested such operators to Thompson; the generalization to make them both prefix and postfix was his own.
Another person puts it this way:²
> It's a myth to suggest C’s design is based on the PDP-11. People often quote, for example, the increment and decrement operators because they have an analogue in the PDP-11 instruction set. This is, however, a coincidence. Those operators were invented before the language [i.e. B] was ported to the PDP-11.
In any case, the PDP-11 usually gets all the love, but I want to make sure the other PDPs get some too!)
- it is an automatic variable whose address has not been taken; or
- the uninitialized object' bits are such that it takes on a non-value representation.
And I especially don’t buy that UB is there for register allocation.
First of all, that argument only explains UB of OOB memory accesses at best.
Second, you could define the meaning of OOB by just saying “pointers are integers” and then further state that nonescaping locals don’t get addresses. Many ways you could specify that, if you cared badly enough. My favorite way to do it involves saying that pointers to locals are lazy thunks that create addresses on demand.
The spec even says:
> behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements
No motivation is given that I could find, so the actual difference between undefined and implementation defined behaviour seems to be based on whether the behaviour needs to be documented.
Also the C spec has always been a pragmatic afterthought, created and maintained to establish at least a minimal common feature set expected of C compilers.
The really interesting stuff still only exists outside the spec in vendor language extensions.
The two instances where UB allows for optimisation are as follows:
1. The 'signed overflow' UB allows for faster array indexing. By ignoring potential overflow, the compiler can generate code that doesn't check for accidental overflow (which would require masking the array index, recomputing the address on each loop iteration). I believe the better solution here would be to introduce a specific type for iterating over arrays that will never overflow; size_t would do fine, and making signed overflow at least implementation defined, if not outright fully defined, after a suitable period during which compilers warn if you use a too-small type for array indexing.
2. The 'aliasing' UB does away with the need to read/write values to/from memory each time they're used, and is extremely important to performance optimisation.
But the rest? Most of it does precisely nothing for performance. At 'best', the compiler uses detected UB to silently eliminate code branches, but that's something to be feared, not celebrated. It isn't an optimisation if it removes vital program logic, because the compiler could 'demonstrate' that it could not possibly take the removed branch, on account of it containing UB.
The claim in the linked article ("what every C programmer should know") that use of uninitialized variables allows for additional optimisation is incorrect. What it does instead is this: if the compiler see you declare a variable, and then reading from it before writing to it, it has detected UB, and since the rule is that "the compiler is allowed to assume UB does not occur", use that as 'evidence' that that code branch will never occur and can be eliminated. It does not make things go faster; it makes them go _wrong_.
Undefined behaviour, ultimately, exists for many reasons: because the standards committee forgot a case, because the underlying platforms differ too wildly, because you cannot predict in advance what the result of a bug may be, to grandfather in broken old compilers, etc. It does not, in any way, shape, or form, exist _in order to_ enable optimisation. It _allows_ it in some cases, but that is, and never was, not the goal.
Moreover, the phrasing of "the compiler is allowed to assume that UB does not occur" was originally only meant to indicate that the compiler was allowed to emit code as if all was well, without introducing additional tests (for example, to see if overflow occurred or if a pointer was valid) - clearly that would be very expensive or downright infeasible. Unfortunately, over time this has enabled a toxic attitude to grow that turns minor bugs into major disasters, all in the name of 'performance'.
The two bullet points towards the end of the article are both true: the compiler SHOULD NOT behave like an adversary, and the compiler DOES NEED license to optimize. The mistake is thinking that UB is a necessary component of such license. If that were true, a language with more UB would automatically be faster than one with less. In reality, C++ and Rust are roughly identical in performance.
The dustbin of programming languages is jam packed with elegant, technically terrific, languages that never went anywhere.
Chill the fuck out.
True, most compilers have sane defaults in many cases for things that are technically undefined (like take sizeof(void) or do pointer arithmetic on something other than a char). But not all of these cases can be saved by sane defaults.
Undefined behavior means the compiler can replace the code with whatever. So if you e.g. compile optimizing for size, the compiler will rip out the offending code, as replacing it with nothing yields the greatest size optimization.
See also John Regehr's collection of UB-Canaries: https://github.com/regehr/ub-canaries
Snippets of software exhibiting undefined behavior, executing e.g. both the true and the false branch of an if-statement or none etc. UB should not be taken lightly IMO...
Or replacing all you mp3s with a Rick Roll. Technically legal.
(Some old version of GHC had a hilarious bug where it would delete any source code with a compiler error in it. Something like this would technically legal for most compiler errors a C compiler could spot.)
The code change might come in something as innocent as a bug fix to the compiler.
I for one am glad that compilers can assume that things that can't happen according to the language do in fact not happen and don't bloat my programs with code to handle them.
What is this supposed to mean? I can't think of any interpretation that makes sense.
I think ISO C defines the executable program to be something like the compiled translation units linked together. But header files do not have to have any particular correspondence to translation units. For example, a header might declare functions whose definitions are spread across multiple translation units, or define things that don't need any definitions in particular translation units (e.g. enum or struct definitions). It could even play macro tricks which means it declares or defines different things each time you include it.
Maybe you mean it's undefined behaviour to include a header file that declares functions that are not defined in any translation unit. I'm not sure even that is true, so long as you don't use those functions. It's definitely not true in C++, where it's only a problem (not sure if it's undefined exactly) if you ODR-rule use a function that has been declared but not defined anywhere. (Examples of ODR-rule use are calling or taking the address of the function, but not, for example, using sizeof on an expression that includes it.)
Start with a concrete example. A header that is not in our program, or described in ISO C. How about:
#include <winkle.h>
> But header files do not have to have any particular correspondence to translation units.
A header inclusion is just a mechanism that brings preprocessor tokens into a translation unit. So, what does the standard tell us about the tokens coming from #include <winkle.h> into whatever translation unit we put it into?
Say we have a single file program and we made that the first line. Without that include, it's a standard-conforming Hello World.
Not only you are faced with creating your own wrappers, if no one else has done it already.
The tooling, for IDEs and graphical debuggers, assumes either C or C++, so it won't be there for Rust.
Ideally the day will come where those ecosystems might also embrace Rust, but that is still decades away maybe.
IMHO you can today deal with UB just fine in C if you want to by following best practices, and the reasons given when those are not followed would also rule out use of most other safer languages.
C is portable in the least interesting way, namely that compilers exist for all architectures. But that's where it stops.
> IMHO you can today deal with UB just fine in C if you want to by following best practices
In the other words, short compilation time has been traded off with wetware brainwashing... well, adjustment time, which makes the supposed advantage much less desirable. It is still an advantage, I reckon though.
C is a different kind of animal that encourages terseness and economy of expression. When you know what you are doing with C pointers, the compiler just doesn't get in the way.
> When you know what you are doing with C pointers, the compiler just doesn't get in the way.
Alas, it doesn't get in the way of you shooting your own foot off, too.
Rust allows unsafe and other shenanigans, if you want that.
Tell me you use -fno-strict-aliasing without telling me.
Fwiw, I agree with you and we're in good[citation needed] company: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...
C and C++ force you to code in the C and C++ ways. It may that that's what you want, but they certainly dont let me code how I want to code!
Beyond that, recent C++ versions have much more expressive metaprogramming capability. The ability to do extensive codegen and code verification within C++ at compile-time reduces lines of code and increases safety in a significant way.
Rusts tooling is hands down better than C/++ which aids to a more streamlined and efficient development experience
The smallest binary rustc has produced is like ~145 bytes.
Well, anything were your people have more experience in the other language or the libraries are a lot better.
Same thing with e.g. strict aliasing or the various UB that exists in the standard library. For instance, it's UB to pass a null pointer to strlen. Of course, you can make that perfectly defined by adding an `if` to strlen that just returns 0. But then you're adding a branch to every strlen, and C is simply not willing to do that for performance reasons, so they say "this is UB" instead.
Pretty much instance of UB in standard C or C++ is because making it defined would either hamper the optimizer, or it would make standard library functions slower. They don't just make things UB for fun.
For example the reason why 2s complement took so long is because of some machine that ran C that still existed that was 1s complement.
> The reason is that if you compile with flags that make it defined, you lose a few percentage points of performance (primarily from preventing loop unrolling and auto-vectorization).
I certainly don’t lose any perf on any workload of mine if I set -fwrapv
If your claim is that implementers use optimization as the excuse for wanting UB, then I can agree with that.
I don’t agree that it’s a valid argument though. The performance wins from UB are unconvincing, except maybe on BS benchmarks that C compilers overtune for marketing reasons.
It explains many loop-unroll and integer overflow as well.
inlining, interprocedural optimizations.
For example, something as an trivial accessor member function would be hard to optimize.
This means losing a lot of optimisations, so in fact when you say you "don't buy" this argument you only mean that you don't care about optimisation. Which is fine, but this does mean the "improved" C isn't very useful in a lot of applications, might as well choose Java.
You won’t lose “a lot” of optimizations and you certainly won’t lose enough for it to make a noticeable difference in any workload that isn’t SPEC
Bad guys find the corner case and they can now authenticate as user 0x4C6F6769 which doesn't exist and so that's useless. But - when we upgrade to C++ 26 with the hypothetical zero "fix" now they're root instead!
You are absolutely, 100% correct though: I've never seen a case where accidental overflow doesn't start causing bugs anyway. Like, the Pac-Man kill screen is caused by a byte overflowing (it happens on level 256), and the game goes insane. Pac-Man was written in assembly where overflow is defined behavior, but that doesn't matter at all, the game is still broken. If signed overflow is essentially always a bug anyway, why not make it UB and optimize around it? Especially since it is super-valuable in being able to unroll loops.
People always bring up signed integer overflow as an argument for why UB is scary, and it always seemed like such a bad argument to me. Like, I can understand why people think UB has gone too far in C/C++, but signed overflow is such a bad example. It's one of the most sensible bits of UB in the entire standard, IMHO.
In the most basic cases, yes. It can be used as a more polished switch statement.
It's the whole paradigm of "define an ad-hoc Enum here and there", encoding rigid semantic assumptions about a function's behaviour with ADTs, and pattern matching for control-flow. This feels like a very academic approach and modifying such code to alter its opinionated assumptions isn't funny.
You're misunderstanding me: as of C++20, there is no other representation in C++ for signed integers other than two's complement (no signed ones' complement, no signed magnitude, nothing else), but signed overflow is still UB. It's not because of obscure machines or hardware, such hardware is not relevant for C++20 and later. The reason for it is performance. From the accepted paper [1]:
> The following polls were taken, and corresponding modifications made to the paper. The main change between [P0907r0] and the subsequent revision is to maintain undefined behavior when signed integer overflow occurs, instead of defining wrapping behavior. This direction was motivated by:
> * Performance concerns, whereby defining the behavior prevents optimizers from assuming that overflow never occurs
You may disagree, you may think they're wrong, but their motivation is performance, that's why this is UB. It's right there in black and white. This was C++, not C, but it's not at all unthinkable that the C standard will also mandate two's complement at some point, and if they do, they almost certainly keep signed overflow undefined for exactly the same reason.
It's not hard to write code that optimizes much better when you use signed loop variables. One of my favorite examples is this function [2] to turn a 3D mesh inside out by flipping the edges of each triangle in a triangle mesh. The godbolt link has two versions of the same function, one with a signed loop variable, one with an unsigned one. The signed one auto-vectorizes and optimizes much better because it can assume that the loop variable never overflows (this version is C++, it's trivial to rewrite it in C and get the same results).
This is why signed overflow is UB.
[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p09...
I know that this is misguided based on my own perf tests and others’ perf tests.
Also, it’s wrong to say flat out that UB on signed ints is somehow necessary for perf when even a simple perf test shows that it just doesn’t matter, and the optimization it enables is quite obscure.
Rust is nicer for async and MT than c++ in every way. I am pretty sure.
But it's still mid. If you use Rust async aggressively you will struggle with the borrow checker and the architecture results of channel hell.
If you follow the "one control thread that does everything and never blocks" you can get far, but the language does not give you much help in doing that style neatly.
I have never used Go. I love a lot of Go projects like Forgejo and SyncThing. Maybe Go solved async. Rust did not. C++ did not even add good tagged unions yet.
Doing anything concurrent in Go is also really annoying (be that async or with threads), because everything is mutable. Not just by default but always. So anything shared is very dangerous.
Not sure where this is coming from.
Async rust is amazing as long as you only mix in one more hard concept. Be it traits, generics or whatever. You can confidently write and refactor heavily multithreaded code without being deathly afraid of race conditions etc. and it is extremely empowering.
The problem comes when trying to write async generic traits in a multithreaded environment.
Then just throwing stuff at the wall and hoping something sticks will quickly lead you into despair.
And I believe that provenance is an issue in unsafe rust.
https://doc.rust-lang.org/std/ptr/index.html#provenance
The definition isn't deemed complete because of aliasing. AIUI The definition we have is adequate if you're OK with treating all edge cases for "Is this an alias?" as "Yes" but eventually Rust will also need to carefully nail down all those edge cases so that you can tread closer without falling off.
Just to be clear, this isn't a recent development, it has been this way for many years at this point.
But the point is also to dispel the conventional wisdom that C/C++ is necessarily intrinsically unsafe. It's a tradeoff between safety, performance and flexibility/compatibility. And you don't necessarily need to jump to a completely different language to get a different tradeoff.
Fil-C sacrifices some performance for safety and compatibility. The traditional compilers sacrifice some safety for performance and flexibility/compatibility. And scpptool aims to provide the option of sacrificing some flexibility for safety and performance. (Along with the other two tradeoffs available in the same program). The claim is that C++ turns out to be expressive enough to accommodate the various tradeoffs. (Though I'm not saying it's always gonna be pretty :)
To get an example that's easy to understand: before the introduction of the 'const' keyword, you just couldn't express that some variable should never be changed. And no amount of UB sanitisers would have fixed this for you: you just couldn't express the concept. There's lots of other areas of these languages that are still in a similar state.
Eg there's no way to express that a function should be pure, ie not have side effects (but is allowed to use mutation internally).
I believe gcc actually does support `__attribute__ ((pure))` to indicate function purity. (I assume it doesn't actually enforce it, but presumably it theoretically could at some point.)
It's like one of those tricky self-referential parlor box statements. "The statement on this box is not true"? Thanks I guess. But that's a game, the puzzles are supposed to be like that, whereas the mission of the ISO document was not to confuse people, so it's good that it is being improved.
One huge thing they have on their side is that their implementation is concrete. Whatever it is that, say, GCC does is de facto actually a thing a compiler can do. The standards bodies (and WG21 has been worse by some margin, but they're both guilty) may standardize anything, but concretely the compiler can only implement some things. "Just do X" where X isn't practical works fine on paper but is not implementable. This was the fate of the Consume ordering. Consume/ Release works fine on paper, you "just" need to have whole program analysis to implement it. Well of course that's not practical so it's not implemented.
I mean to say that best practices do help much but learning those best practices take much time as well. So short compilation time is easily offseted by learning time, and C was not even designed to optimize compilation time anyway (C headers can take a lot to parse and discard even when unused!). Your other points do make much more sense and it's unfortunate that first points are destructively interfering each other, hence my comment.
It's also very often not the best way to identify objects, for many reasons, including performance (spatial locality is a big deal).
These problems go away almost completely by simply using `EntityID` and going through `&mut World` for modifications, rather than passing around `EntityPtr`. This pattern gives you a lot of interesting things for free.
Pretty much nobody writing games in C++ uses raw pointers in entities to hold references to other related entities, because entities can be destroyed at any time and there's no simple way for a referring entity to know when a referenced entity is destroyed.
Using some sort of entity ID or entity handle is very common in C++, the problem is that when implementing this sort of system in Rust, developers often end up having to effectively "work around" the borrow checker, and they end up not really gaining anything in terms of correctness over C++, ultimately defeating the purpose of using Rust in the first place, at least for that particular system.
The benefits seem pretty massive, at least on the surface. For example, you can run any system that only takes `&World` (i.e., immutable access) in parallel without breaking a sweat.
This is especially true for C which supports almost nothing (it doesn't even have a sensible array type!). But is also true for C++: while it supports a lot, it doesn't support everything.
Methods in C, just have function pointers as members. Common in many codebases.
Guaranteed tail calls, all the compilers guarantee that function calls that are a return expression are tail calls.
Tagged union in C++, it's trivial as a library, see std::variant for a bad example of it, and all the various monadic/pattern-matching variants (pun intended) people have written. C is at a disadvantage here due to lack of lambdas, but I'm sure people have built stuff using some GCC extensions.
As shitty as C++ is from today's PoV, the entire gaming industry switched over within around 3 years towards the end of the 90s. 6..7 years is a long time, and a single engine (especially when it's more or less just a runtime without editor and robust asset pipeline) won't change the bigger picture that Rust is a pretty poor choice for gamedev.
Did they? What's your evidence? Are you including consoles?
Btw, the alternatives in the 1990s were worse than they are now, so the bar to clear for eg C or C++ were lower.
From what I've seen, around the late mid-90's, C++ usage was still rare, right before 2000 it was already common and most middleware didn't even offer C APIs anymore.
Of course a couple of years later Unity arrived and made the gamedev language choice more complicated again.
If we exclude AAA games, probably the vast majority of the games nowadays don't need manual memory management for the game core (C# was a popular choice, it seems). I guess that if one really needs manual memory management, languages with moderate memory safety would be a more appropriate choice (support libraries/frameworks being equal, which certainly aren't).
I've used Bevy, and ECS is not an appopriate choice for every game (I wouldn't actually advise it unless there is a specific need). It requires very careful design over the whole lifecycle (ECS-based games very easily tend to get a mess), which is exactly the opposite of one wants for rapid prototyping.
Making a nontrivial game with them is a wholly different story.
Would you expand on this? What was your C tooling/workflow that was inferior to your new Rust experience?
As for the language tooling itself, static and runtime analyzers in C and C++ (and these are table stakes at this point) do not come close to the level of accuracy of the Rust compiler. If you care about writing unsafe code, Miri is orders of magnitude better at detecting UB than any runtime analyzer I've seen for C and C++.
C and C++ have their strengths, but rapid prototyping is generally not seen to be amongst them.
This shouldn't be any more controversial than saying that pure Python is generally slow.
inline void add(int* to, int what) { *to += what; }
void foo();
void bar() {
int x = 0;
add(&x, 1);
foo();
return x;
}
The right definition is probably something like:
- pointers that come out of the outside world (syscalls) are escaped. They are just integers.
- pointers to locals have provenance. They point to an abstract location. It is up to the implementation to decide when the location gets an integer value (is in an actual address) and what that value is. The implementation must do this no later than when the pointer to the local escapes.
- pointer values passed to the outside world (syscalls) escape.
- pointer values stored in escaped memory also escape, transitively
That’s one possible definition that turns the UB into implementation defined behavior. I’m sure there are others
Again, I'm not a compiler writer, but my understanding is that non escaping variables can be optimized in SSA form, escaped variables are otherwise treated as memory and the compiler must be significantly more conservative.
In any case, whether a pointer escapes or not depends purely on the compiler capabilities and optimization level, so it would not be sane making a code well defined or UB depending on the compiler or optimization level.
edit: to be more concrete, do you think that in my example the constant folding of the return into return 1 should be allowed? And if so, which variant of this code would prevent the optimization and why?
I'm guessing you mean that every cross-platform C codebase ends up being plastered in cascading preprocessor code to deal with OS and architecture differences. Sure that's true, you still have to do some porting work regardless of the language you chose.
But honestly, is there any language more portable than C? I struggle to come up with one.
If someone told me "I need a performant language that targets all major architectures and operating systems, but also maybe I want to run it on DOS, S390X, an old Amiga I have in my closet, and any mystery-meat microcontroller I can find." then really wouldn't have a better answer for them than C89.
If C isn't portable then nothing is.
If you are targeting any recent platform, both Rust and Zig do what you want.
A decade earlier I also used Xenix and DG/UX.
That is a nice way to learn how "portable" C happens to be, even between UNIX systems, its birthplace.
You cannot write portable code without platform-specific and even environment-specific adaptations, like handling the presence of certain headers (looking at you, stdint.h and stddef.h), and let's not even start about interacting with the OS in any way.
I am a compiler writer.
The definition I gave in my post is general enough to cover all possible compilers (ones that have LTO, ones that are inside a DBT, etc).
Yes the constant folding should be allowed because the pointer to the local never escaped.
N3220: 5.1.1.1 Program Structure
A C program is not required to be translated in its entirety at the same time. The text of the program is kept in units called source files, (or preprocessing files) in this document. A source file together with all the headers and source files included via the preprocessing directive #include is known as a preprocessing translation unit. After preprocessing, a preprocessing translation unit is called a translation unit. Previously translated translation units may be preserved individually or in libraries. The separate translation units of a program communicate by (for example) calls to functions whose identifiers have external linkage, manipulation of objects whose identifiers have external linkage, or manipulation of data files. Translation units may be separately translated and then later linked to produce an executable program.
> Couldn't it be the set {main.c, winkle.h}
No; in this discussion it is important that <winkle.h> is understood not to be part of the program; no such header is among the files presented for translation, linking and execution. Thus, if the implementation doesn't resolve #include <winkle.h> we get the uninteresting situation that a constraint is violated.
Let's focus on the situation where it so happens that #include <winkle.h> does resolve to something in the implementation.
Your question about an include file that isn't part of the program just doesn't make any sense.
(Technically it says that those files together make up the "program text". As my other comment says, "program" is the binary output.)
I just checked, and if you attempt to include a file that cannot be found (in the include path, though it doesn't use that exact term) then that's a constraint violation and the compiler is required to stop compilation and issue a diagnostic. Not undefined behaviour.
What are the requirements then?
By that metric assembly is the best prototyping language.
That would be 2000, until then Sega, Nintendo and Playstion only had C and Assembly SDKs, even the Playstation Yaroze for hobbists did get released only with C and Assembly support.
PC was naturally another matter, especialy with Watcom C/C++.
You were at most in one place. My question was rather, which corners of the industry are you counting?
However you are right that one of the killer features of C++ was that it provided a pretty simple upgrade path from C to (bad) C++.
It's not just API calls. You can call C APIs from most languages just fine.
id Software was kinda famous for being the last big C holdout, having only switched to C++ with Doom 3, and development of Doom 3 started in late 2000.
You can do the same in C by wrapping your array in a struct.
Correct; but processing doesn't stop there.
> There might be undefined behaviour in the overall translation unit
But what does that mean; how do you infer that there might be undefined behavior?
Does ISO C define the behavior, or does it not?
ISO C has nothing to say about what is in #include <winkle.h> if such a header is found and didn't come from the program.
Without having anything to say about what is in it, if it is found at all, ISO C cannot be giving a definition of behavior of the tokens that are substituted for that #include.
Yes, unthinkable happenstances like addition on fixed-width integers overflowing! According to the language, signed integers can't overflow, so code like the following:
int new_offset = current_offset + 16;
if (new_offset < current_offset)
return -1; // Addition overflowed, something's wrong
int new_offset = current_offset + 16;
What would the "sky falling" look like to you? If you're expecting dramatic movie scenes like something out of Mr Robot, I'm afraid the reality is more mundane, just a never-ending series of basic programming errors that turn into remote code execution exploits because of language and compiler choices by people who don't pay the costs.
Unfortunely too many folks still pretend C is some kind of magic portable Assembly language that no other language on Earth is able to achieve the same.
Also if I care enough about ultimate performace, like anyone that actually cares about performance, I dust off my Assembly programming skills, alongside algorithms, datastructures and computer organisation.
The header model is one of the parts that makes compiling C slower than it could be. This doesn't mean that it is slow, but it's fast in spite of headers, not because of them.
> In C you can split interface and implementation cleanly between header and c-file and this enables efficient incremental builds.
That's not what does, it is the ability to produce individual translation units as intermediary files.
> Rust also does not seem to have proper separate compilation.
Rust does separate compilation, and also has efficient incremental builds. Header files are not a hard requirement for this.
I am not sure how it works in Rust as you need to monomorphize a lot of things, which come from other crates. It seems this would inevitably entangle the compilations.
It's that textual inclusion is just a terrible model. You end up reprocessing the same thing over and over again, everywhere it is used. If you #include<foo.h> 100 times, the compiler has to reparse those contents 100 times. Nested headers end up amplifying this effect. It's also at a file-level granularity, if you change a header, every single .c that imports it must be recompiled, even if it didn't use the thing that was changed. etc etc. These issues are widely known.
> I do not understand what you mean by your second point. What separation of interface and implementation allows you to do is updating the implementation without having to recompile other TUs.
Sure, but you don't need to have header files to do this. Due to issues like the above, they cause more things to be recompiled than necessary, not less.
> You can achieve this is also in different ways, but in C this works by in this way.
Right, my point is, those other ways are better.
> I am not sure how it works in Rust as you need to monomorphize a lot of things, which come from other crates. It seems this would inevitably entangle the compilations.
The fact that there are "other crates" is because Rust supports separate compilation: each crate is compiled independently, on its own.
The rlib contains the information that, when you link two crates together, the compiler can use for monomorphization. And it's true that monomorphization can cause a lot of rebuilding.
But to be clear, I am not arguing that Rust compilation is fast. I'm arguing that C could be even faster if it didn't have the preprocessor.
But then, they also sometimes misread the standard in ways I can't really understand. This often can be seen when the "interpretation" changes over time. Earlier compilers (or even earlier parts of the same compiler) implement the standard as written, some new optimization pass has some creative interpretation.
The compiler bug I'm most closely following, and which I suspect you have your eye on too is: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=119472 aka https://github.com/rust-lang/rust/issues/107975 https://github.com/llvm/llvm-project/issues/45725
But it seems like it's just that everybody fucked this up in similar ways, that's two different major compiler backends! I wouldn't be surprised if Microsoft (whose code we can't see) find that they don't get this quite right either.
Vulnerabilities to ransomware (and other forms of malware) can be perpetrated without a single bad pointer being dereferenced.
For instance, a memory-safe e-mail program can automatically open an attachment, and the memory-safe application which handles the attachment can blindly run code embedded in the document in a leaky sandbox.
There is an incredible amount of infrastructure out there that depends on C. Embedded devices, mobile devices, desktops, servers. Network stacks, telephony stacks, storage, you name it. Encryption, codecs, ...
Sky is falling would mean all of it would be falling down so badly that, for instance, you would have about a 50% chance of connecting a server that is more than four hops away.
One could certainly store the interfaces in some binary format, but is it really worth it? This would also work with headers by using a cache, but nobody does it for C because there is not much to gain. Parsing is fast anyhow, and compilers are smart enough not to look at headers multiple times when protected by include guards. According to some quick measurements, you could save a couple of percent at most.
The advantages of headers is that they are simple, transparent, discoverable, and work with outside tools in a modular way. This goes against the trend of building frameworks that tie everything together in a tightly integrated way. But I prefer the former. I do not think it is a terrible model, quite the opposite. I think it is a much better and nicer model.
Again, this isn't undefined behavior to fuck you over and compilers don't use it for optimizations because they hate you. It's because it makes a real difference for performance which is the primary reason low level languages are used.
If you for some reason want less efficient C++ then compilers even provide you flags to make this particular operation defined. There is no reason the rest of us have to suffer worse performance because of you.
Personally I would prefer if unsigned ints had the same undefined behavior by default with explicit functions for wrapping overflow. That would make developer intent much clearer and give tools a chance to diagnose unwanted overflow.
Rust does it right: The "+"-operator is wrapping in release builds (and throws an exception when debug-assertions are enabled), and there are explicit functions for checked/overflowing/saturating/strict/unchecked/wrapping addition [1]. The "unchecked" variant exploits UB-on-overflow and is marked as unsafe. These functions exist both for signed and unsigned integer types. Which once again shows that it's very well possible to design a sane, low-level language which is just as fast as C.
So what I mean is that no file matching <winkle.h> has been presented as part of the external file set given to the implementation for processsing.
I agree that if such a file is found by the implementation it becomes part of the program, as makes sese and as that word is defined by ISO C, so it is not right terminology to say that the file is not part of the program, yet may be found.
If the inclusion is successful, though, the content of that portion of that program is not defined by ISO C.
> So what I mean is that no file matching <winkle.h> has been presented as part of the external file set given to the implementation for processsing.
The thing is, there is no "external file set" that includes header files, so this sentence makes no sense.
Note that when the preprocessor is run, the only inputs are the file being preprocessed (i.e., the .c file) and the list of directories to find include files (called the include path). That's not really part of the ISO standard, but it's almost universal in practice. Then the output of the preprocessor is passed to the compiler, and now it's all one flat file so there isn't even a concept of included files at this point. The object files from compilation are then passed to the linker, which again doesn't care about headers (or indeed the top-level source files). There are more details in practice (especially with libraries) but that's the essence.
I wonder if your confusion is based on seeing header files in some sort of project-like structure in an IDE (like Visual Studio). But those are just there for ease of editing - the compiler (/preprocessor) doesn't know or care which header files are in your IDE's project, it only cares about the directories in the include path. The same applies to CMake targets: you can add include files with target_sources(), but that's just to make them show up in any generated IDE projects; it has no effect on compilation.
Or are you just maybe saying that the developer's file system isn't part of the ISO C standard, so this whole textual inclusion process is by some meaning not defined by the standard? If so, I don't think that matches the conventional meaning of undefined behaviour.
If it's neither of those, could you clarify what exactly you mean by "the external file set given to the implementation for processing"?
The "project" is all the files going into a program supplied other than by the implementation.
C programs can contain #include directives. Those #include directives can be satisfied in one of three ways: they can reference a standard header which is specified by ISO C and hence effectively built into the hosted language, such as <stdio.h>.
C programs can #include a file from the project. For instance someone's "stack.c" includes "stack.h". So yes, there is an external file set (the project) which can have header files.
C programs can also #include something which is neither of the above. That something might be not found (constraint violation). Or it might be found (the implementation provides it). For instance <sys/mmap.h>: not in your project, not in ISO C.
My fictitious <winkle.h> falls into this category. (It deliberately doesn't look like a common platform-specific header coming from any well-known implementation---but that doesn't matter to the point).
> Or are you just maybe saying that the developer's file system isn't part of the ISO C standard, so this whole textual inclusion process is by some meaning not defined by the standard?
Of course, it isn't, no I'm not saying that. The C standard gives requirements as to how a program (project part and other) is processed by the implementation, including all the translation phases that include preprocessing.
To understand what the requirements are, we must consider the content of the program. We know what the content is of the project parts: that's in our files. We (usually indirectly) know the content of the standard headers, from the standard; we ensure that we have met the rules regarding their correct use and what we may or may not rely on coming form them.
We don't know the content of successfully included headers that don't come from our project or from ISO C; or, rather, we don't know that content just from knowing ISO C and our project. In ISO C, we can't find any requirements as to what is supposed to be there, and we can't find it in our project either.
If we peek into the implementation to see what #include <winkle.h> is doing (and such a peeking is usually possible), we are effectively looking at a document, and then if we infer from that document what the behavior will be, it is a documented extension --- standing in the same place as what ISO C calls undefined behavior. Alternatively, we could look to actual documentation. E.g. POSIX tells us what is in <fcntl.h> without us having to look for the file and analyze the tokens. When we use it we have "POSIX-defined" behavior.
#include <winkle.h> is in the same category of thing as __asm__ __volatile__ or __int128_t or what have you.
#include <winkle.h> could contain the token __wipe_current_directory_at_compile_time which the accompanying compiler understands and executes as soon as it parses the token. Or __make_demons_fly_out_of_nose. :)
Do you see the point? When you include a nonstandard header that is not coming from your project, and the include succeeds, anything can happen. ISO C no longer dictates the requirements as to what the behavior will be. Something unexpected can happen, still at translation time.
Now headers like <windows.h> or <unistd.h> are exactly like <winkle.h>: same undefined behavior.
I have an execution environment, Wasm, where doing this is pretty well defined, in fact. So if I want to read the memory at address 12345, which is within bounds of the linear memory (and there's a builtin to make sure), why should it be undefined behavior?
And regarding pointer provenance, why should going through a pointer-to-integer and integer-to-pointer conversions try to preserve provenance at all, and be undefined behavior in situations where that provenance is ambiguous?
The reason I'm using integer (rather than pointer) arithmetic is precisely so I don't have to be bound by pointer arithmetic rules. What good purpose does it serve for this to be undefined (rather than implementation defined) beyond preventing certain programs to be meaningfully written at all?
I'm genuinely curious.
How would you define it? Especially in a way that is consistent with the rest of the language and allows common optimizations (remember that C supports variables, which may or may not be stored in memory)?
It's the only way to interpret *(uint64_t*)(12345) when the standard says that a integer-to-pointer conversion is "intended to be consistent with the addressing structure of the execution environment".
There exists an instruction to do that load in Wasm, there's a builtin to check that 12345 points to addressable memory, the load is valid at the assembly level, the standard says the implementation should define this to be consistent with the addressing structure of the execution environment, why the heck are we playing games and allowing the compiler to say, "nope, that's not valid, so your entire program is invalid, and we can do what ever we want, no diagnostic required"?
The performance of compilers is frequently measured using the SPEC series of CPU benchmarks, and one of the main constraints of the series SPEC series of tests is that the source code of the benchmark cannot be changed. It is static.
As a result, compiler authors have to find increasingly convoluted ways to make it possible for various new compiler optimizations to be applied to the legacy code used in SPEC. Take 403.gcc: it's based on gcc version 3.2 which was released on August 14th 2002 -- nearly 23 years ago.
By making certain code patterns undefined behaviour, compiler developers are able to relax the constraints and allow various optimizations to be applied to legacy code in places which would not otherwise be possible. I believe the gcc optimization to eliminate NULL pointer checks when the pointer is dereferenced was motivated by such a scenario.
In the real world code tends to get updated when compilers are updated, or when performance optimizations are made, so there is no need for excessive compiler "heroics" to weasel its way into making optimizations apply via undefined behaviour. So long as SPEC is used to measure compiler performance using static and unchanging legacy code, we will continue to see compiler developers committing undefined behaviour madness.
The only way around this is for non-compiler developer folks to force language standards to prevent compilers from using undefined behaviour to do that which normal software developers considers to be utterly insane code transformations.
https://dl.acm.org/doi/10.5555/11616.11617
https://jacobfilipp.com/DrDobbs/articles/DDJ/1991/9108/9108h...
But converting an integer to a pointer creates a problem if you allow that pointer to point to anything--it breaks all of the optimizations that assumed they could trace all of the uses of an address. So you need something like provenance to say that certain back-conversions are illegal. The most permissive model is a no-address-taken model (you can't forge a pointer to a variable whose address was never taken). But most compilers opt instead for a data-dependency-based model: essentially, even integer-based arithmetic of addresses aren't allowed to violate out-of-bounds at the point of dereference. Or at least, they claim to--the documentation for both gcc and llvm have this claim, but both have miscompilation bugs because they don't actually allow this.
The proposal for pointer provenance in C essentially looks at how compilers generally implement things and suggests a model that's closer to their actual implementation: pointer-to-integer exposes the address such that any integer-to-pointer can point to it. Note this is more permissive than the claimed models of compilers today--you're explicitly able to violate out-of-bounds rules here, so long as both objects have had their addresses exposed. There's some resistance to this because adhering to this model also breaks other optimizations (for example, (void*)(uintptr_t)x is not the same as x).
As a practical matter, pointer provenance isn't that big of a deal. It's not hard to come up with examples that illustrate behaviors that cause miscompilation or are undefined specifically because of pointer provenance. But I'm not aware of any application code that was actually miscompiled because the compiler implemented its provenance model incorrectly. The issue gets trickier as you move into systems code that exists somewhat outside the C object model, but even then, most of the relevant code can ignore their living outside the object model since resulting miscompiles are prevented by inherent optimization barriers anyways (note that to get a miscompile, you generally have to simultaneously forge the object's address, have the object's address be known to the compiler already, and have the compiler think the object's address wasn't exposed by other means).
It's not something that exists in the hardware. It's also not a good idea, though trying to steer people away from it proved beyond my politics.
The essential idea of pointer provenance is that it is somehow possible to enumerate all of the uses of a memory location (in a potentially very limited scope). By the time you need to introduce something like "volatile" to indicate to the compiler that there are unknown uses of a variable, you have to concede the point that the compiler needs to be able to track all the known uses within a compiler--and that process, of figuring out known uses, is pointer provenance.
As for optimizations, the primary optimization impacted by pointer provenance is... moving variables from stack memory to registers. It's basically a prerequisite for doing any optimization.
The thing is that traditionally, the pointer provenance model of compilers is generally a hand-wavey "trace dataflow back to the object address's source", which breaks down in that optimizers haven't maintained source-level data dependency for a few decades now. This hasn't been much of a problem in practice, because breaking data dependencies largely requires you to have pointers that have the same address, and you don't really run into a situation where you have two objects at the same address and you're playing around with pointers to their objects in a way that might cause the compiler to break the dependency, at least outside of contrived examples.
When the hardware behaviour and the pointer provenance model disagree, one should change the model, not change the behavior of the program.
This is sort of on the one hand not a meaningful claim, and then on the other hand not even really true if you squint anyway?
Firstly the hardware does not have pointers. It has addresses, and those really are integers. Rust's addr() method on pointers gets you just an address, for whatever that's worth to you, you could write it to a log maybe if you like ?
But the Morello hardware demonstrates CHERI, an ARM feature in which a pointer has some associated information that's not the address, a sort of hardware provenance.
Say you're working with assembly as your medium, on a von neumann machine. Writing to parts of the code section is expected behaviour. What can you optimise in such a world? Whatever cannot be observed. Which might mean replacing instructions with sequences of the same length, or it might mean you can't work out anything at all.
C is much more restricted. The "function code" isn't there, forging pointers to the middle of a function is not a thing, nor is writing to one to change the function. Thus the dataflow is much easier, be a little careful with addresses of starts of functions and you're good.
Likewise the stack pointer is hidden - you can't index into the caller's frame - so the compiler is free to choose where to put things. You can't even index into your own frame so any variable whose address is not taken can go into a register with no further thought.
That's the point of higher level languages, broadly. You rule out forms of introspection, which allows more stuff to change.
C++ has taken this too far with the object model in my opinion but the committee disagrees.
If it only exists in some hardware, how should the standard deal with that?
It seems hard to not have this without imputing a freeze semantic which would be expensive on today's systems. Maybe I don't understand what you mean ? Rust considered insisting on the freeze semantic and was brought up short by the cost as I understand it.
But maybe you're talking about something else entirely?
Most of my most recent comment is addressing the possibility that you meant this.
As I said, there is no such concept to the compiler. It isn't passed any list of files that could be included with #includr, only the .c files actually being compiled, and the directories containing includable files.
The fact that your IDE shows project files is an illusion. Any header files shown there are not treated differently by the compiler/preprocessor to any others. They can't be, because it's not told about them!
It's even possible to add header files to your IDE's project that are not in the include path, and then they wouldn't be picked up by #include. That's how irrelevant project files are to #include.
It doesn't seem as if you're engaging with the standard-based point I've been making, in spite of detailed elaboration.
> Any header files shown there are not treated differently by the compiler/preprocessor to any others.
This is absolutely false. Headers which are part of the implementation, such as standard-defined headers like <stdlib.h> need not be implemented as files. When the implementation processes #include <stdlib.h>, it just has to flip an internal switch which makes certain identifiers appear in their respective scopes as required.
For that reason, if an implementation provides <winkle.h>, there need not be such a file anywhere in its installation.
Instead, the standard says that the include directive:
> searches a sequence of implementation-defined places for a header ... and causes the replacement of that directive by the entire contents of the header.
(Note that it talks about simply substituting in text, not anything more magical, but that's digressing.)
It's careful to say "places" rather than "directories" to avoid the requirement that there's an actual file system, but the idea is the same. You don't pass the implementation every individual file that might need to be included, you pass in the places that hold them and a way to search them with a name.
Maybe you were confused by that part of the standard you quoted in an earlier comment.
One part of that says "The text of the program is kept in units called source files, (or preprocessing files) in this document." But the "source files" aren't really relevant to the include directive – those are the top-level files being compiled (what you've called "base files").
The next sentence you quoted says "A source file together with all the headers and source files included via the preprocessing directive #include is known as a preprocessing translation unit." But "all the headers" here is just referring to files that have been found by the search mechanism referred to above, not some explicit list.
Generally seems to me the C standard makes things like that UB. Signed integer overflow, for example. Implemented as wrapping two's-complement on modern architectures, defined as such in many modern languages, but UB in C due to ongoing support for niche architectures.
The issues around pointer provenance are inherent to the C abstract machine. It's a much more immediate show-stopper on architectures that don't have a flat address space, and the C abstract machine doesn't assume a flat address space because it supports architecture where that's not true. My understanding is that reflects some oddball historical architectures that aren't relevant anymore, nowadays that includes CHERI.
For what it’s worth there’s a reason you’re supposed to do this kind of access through memcpy, not by dereferencing made up pointers.
> There exists an instruction to do that load in Wasm, there's a builtin to check that 12345 points to addressable memory, the load is valid at the assembly level, the standard says the implementation should define this to be consistent with the addressing structure of the execution environment, why the heck are we playing games and allowing the compiler to say, "nope, that's not valid, so your entire program is invalid, and we can do what ever we want, no diagnostic required"?
Because the language standard is defined to target a virtual machine as output, not any given implementation. That virtual machine is then implemented on various platforms, but the capabilities of the underlying system aren’t directly accessible - they are only there to implement the C virtual machine. That’s why C can target so many different target machines.
Any value would be OK. There are aliasing rules to follow, and it's OK if those crater performance when you start using integer-to-pointer conversions a lot. Is that a problem? But in this instance, assume I don't even care.
> For what it’s worth there’s a reason you’re supposed to do this kind of access through memcpy, not by dereferencing made up pointers.
Then why allow integers to be converted to pointers at all, say it's implementation defined, and meant to represent the addressing structure of the environment?
> Because the language standard is defined to target a virtual machine as output, not any given implementation …
Again, not taking about this being portable. The standard says it's implementation defined, and meant to match the addressing structure of the platform. I offered a specific platform where all of this has a specific meaning, that's all.
What's the point of specifying this, if you're then going to say _actually_ because of aliasing it's “undefined” and as soon as that magic word appears, a smart compiler that can prove it at compile time decides this code can't possibly be reached, and deletes the entire function?
What good does this bring us if it means clang can't be used to target platforms where direct memory access is a thing?
I don’t know about this specific instance, but generally disallowing aliasing enables huge performance gains. That’s something Rust does to a great amount and one of the reasons real world code bases are faster in C than C++ (the other being a much better standard library with better defaults for containers and whatnot)
And I'm taking about both things.
Integer arithmetic that produces pointers that are just out of bounds of an object. Why can't this work? Why can't the compiler assume that, since I explicitly converted a pointer to an integer, the pointed-to object can't be put into a register, or made to go out or scope early?
Second, fabricating pointers. If I have a pointer to mmap/sbrk memory, shouldn't I be allowed to “fabricate” arbitrary pointers from integers that point into that area? If not, why not?
Finally Wasm. The linear memory is addressable from address 0 to __builtin_wasm_memory_size * PAGESIZE. Given this, and except maybe the address at zero, why should it be undefined behavior to dereference any other address?
What's the actual advantage to making these undefined behavior? What to we gain in return?
Formally it is undefined, as there is no way to give it sane semantics and it will definitely trip sanitizers and similar memory safety tools.
Fabricating pointers in an implementation-defined way is obviously ok. This would cover mmap / sbrk or mapped I/O. Note also that historically, the C standard left things UB exactly so that implementations can use it for extensions (e.g. mmap). The idea that UB == invalid program is fairly recent misinformation, but we have to react to it and make things at least implementation-defined (which also meant a bit something else before).
I like pointer tagging because I like dynamic language implementations. That tends to look like "summon a pointer from arithmetic", which will have unknown to the compiler provenance, which is where the deref without provenance is UB demon strikes.
The failure mode of pointer provenance is converting an integer to a pointer to an object that was never converted to an integer. Tricks like packing integers into unknown bits or packing pointers into floating-point NaNs don't violate pointer provenance--it's really no different from passing a pointer to an external function call and getting it back from a different external function call.
OK, now suppose that the implementation resolves #include <winkle.h> and replaces it with tokens.
The subsequent processing is what my position is concerned with.
My position is that since the standard doesn't define what those tokens are, the behavior is not defined.
In other words, a conforming implementation can respond #include <winkle.h> with any behavior whatsoever.
- It can diagnose it as not being found.
- It can replace it with the token __wipe_current_directory which that same implementation then understands as a compile-time instruction to wipe the current direcctory.
- Or any other possibility at all.
This all has to do with the fact that the header is not required to exist, but may exist, and if it does exit, it may have any contents whatsoever, including non-portable contents which that implementation understands which do weird things.
It is not required to document any of it, but if it does, that constitutes a documented extension.
A conforming implementation can support a program like this:
#include <pascal.h>
program HelloWorld;
begin
WriteLn('Hello, World!');
end.
I'll just finish by saying: yes please. And thank you for bearing with me.
You get a bunch of memory from mmap. There are no “objects” (in C terminology) in there, and a single “provenance” (if at all, you got it from a syscall, which is not part of the language).
If arbitrary integer/pointer math inside that buffer is not OK, how do to get heterogeneous allocations from the arena to work? When do slices of it become “objects” and gain any other ”provenance”?
Is C supposed to be the language you can't write an arena allocator in (or a conservative GC, or…)?
Famously you cannot implement an allocator in C into static named storage; I understand that on anonymous memory (like that returned by sbrk or mmap, or an upstream allocator) it should work fine, but, as a C++ programmer, I'm not familiar with the specific details of the C lifetime model that allow it. I understand that stores into anonymous memory can change the dynamic type (or whatever is the C equivalent) of the object.
In any case the issue is around object lifetimes and aliasing instead of pointer provenance: you can treat the anonymous memory is just a char array and you can safely form pointers into it and will carry the correct provenance information.
Several things in C, if you want to stay within the guarantees of ISO C for portable code, have to be written in straight Assembly, not even inline, as the standard only defines a asm keyword must exist, leaving all other details to the implementation.
> access via a pointer value with empty provenance is undefined behaviour
I'm annoyed that casting an aligned array of bytes to a pointer to a network packet type is forbidden, and that a pointer to float can't be cast to a pointer to a simd vector of float, and that malloc cant be written in C, but perhaps those aren't provenance either.
That's an old document. In particular, it's largely arguing for a PVI provenance model (i.e., integers carry provenance information), whereas the current TS is relying on a PNVI provenance model (i.e., integers do not carry provenance information). https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2577.pdf is the last draft pre-TS-ification (i.e., has all the background information to understand it).
> I'm annoyed that casting an aligned array of bytes to a pointer to a network packet type is forbidden, and that a pointer to float can't be cast to a pointer to a simd vector of float, and that malloc cant be written in C, but perhaps those aren't provenance either.
That's all strict aliasing rules, not pointer provenance rules. (Well, malloc has issues with living in the penumbra of the C object model). The big thing that provenance prevents you from doing is writing memcpy in C (since char access of a pointer counts as exposing the pointer, whereas the PNVI model makes memcpy a non-exposing operation).
Regarding provenance, consider this:
void bar();
int foo() {
int * ptr = malloc(sizeof(int));
*ptr = 10;
bar();
int result = *ptr;
free(ptr);
return result;
}
int foo() {
bar();
return 10;
}
Yes. It absolutely can. What are you even talking about?
C is not the Windows Start Menu. This habit of thinking it needs to do what it thinks I might expect instead of what I told it is deeply psychotic.
If you don't know the extents of the object pointed to by the char*, using an aligned vector load can reach outside the bounds of the object. Keeping provenance makes that undefined behavior.
Using integer arithmetic, and pointer-to-integer/integer-to-pointer conversions would make this implementation defined, and well defined in all of the hardware platforms where an aligned vector load can never possibly fail.
So you can't do some optimizations to functions where this happens? Great. Do it. What else?
As for why you'd want to do this. C makes strings null-terminated, and you can't know their extents without strlen first. So how do you implement strlen? Similarly your example. Seems great until you're the one implementing malloc.
But I'm sure "let's create undefined behavior for a libc implemented in C" is a fine goal.
I think your example would fall foul of reading beyond the end of an object in addition to pointer provenance. In your case the oob read is harmless as you do not expect any meaningful values for the extra bytes, but generally the compiler would not be able to give any guarantees about the content of the additional memory (or that the memory exists in the first place).
This specific use case could be addressed by the standard, but vectors are already out of the standard, so in practice you use whatever extension you have to use and abide to whatever additional rule the compiler requires (of course this is often underspecified). For example, on GCC simd primitives already have carve-outs for TBAA.
FWIW, a libc implementation in practice already must rely on compiler specific, beyond the standard behaviour anyway.
As an off-topic aside here that might help anybody who is wondering: HN deliberately doesn't provide "Reply" for very recent comments to try to dissuade you from having the sort of urgent back-and-forth you might reasonably do in a real time chat system, and less reasonably attempt (and likely regret) on platforms like Twitter.
A brief window to think about the thing you just read might cause you to write something more thoughtful, and even to realise that it wasn't saying what you had thought in the first place.
My favourite example was an example where somebody said a feature means "less typing" and another comment insisted it did not, and I was outraged until I realised all that's happening is that one person thinks "Typing" means "You know, pressing keys on your keyboard" and the other person thinks "Typing" means "You know, why an integer is different from a float in C" and so they're actually not even disagreeing the conflict is purely syntax!