Ways to break your systems code using volatile (2010)(blog.regehr.org) |
Ways to break your systems code using volatile (2010)(blog.regehr.org) |
The more commonly used language specs, IME, are C99 and C89!
That's 20 and 30 years of 'stability'.
Yuck.
Just because it works on x86, doesn't mean it works on ARM, MIPS, POWER or RISC-V. CPUs other than x86 can reorder stores with other stores and loads with other loads. It can cause the CPU to do the store that starts DMA before the stores that set up length and address are done!
Or just use C11 or C++11 memory model. Although those are still not available in too many cases, curse of having to use an ancient compiler...
Atomics may be implemented with locks, which makes them unsuitable for signal handlers. The only guaranteed lock-free type is `std::atomic_flag` which is not very useful.
`volatile sig_atomic_t` still seems like the better choice for signals.
It should not be overused, because as the article mentions it makes for slower and more confusing code, but it's not quite something to be afraid of either.
It is slower to use volatile, and bad form
https://github.com/torvalds/linux/blob/master/Documentation/...
https://www.mjmwired.net/kernel/Documentation/volatile-consi...
https://www.kernel.org/doc/html/latest/process/volatile-cons...
int* p; // pointer to int
int volatile* p_to_vol; // pointer to volatile int
int* volatile vol_p; // volatile pointer to int
int volatile* volatile vol_p_to_vol; // volatile pointer to volatile int
> "Side note: although at first glance this code looks like it fails to account for the case where TCNT1 overflows from 65535 to 0 during the timing run, it actually works properly for all durations between 0 and 65535 ticks."
E.g.:
timeDelta = timeStampNow - timeStampLast = 0 - 65535 = 1For issue #5, a possible solution not mentioned could be to write inline assembly, no? It would keep the array non-volatile and should be portable.
C++ atomics are no good here, because they are not guaranteed to be lock free or address free.
In POSIX' case, it's up to POSIX operating systems to define reasonable semantics on the memory, using constructs like PTHREAD_PROCESS_SHARED and "robust" pthread mutexes.
That's not right; you can still use std::memory_order to get the memory barriers generated that are required. These are going to obviously be lock free, they deal with memory ordering—what you tried to deal with volatile, but in general case.
See: https://en.cppreference.com/w/cpp/atomic/atomic/store
Effectively std::atomic stores and loads generate volatile accesses plus the required memory barriers to get the desired behavior.
I think a good way of summarizing volatile is this slide from my parallel architectures class [1]:
> Class exercise: describe everything that might occur during the
> execution of this statement
> volatile int x = 10
>
> 1. Write to memory
>
> Now describe everything that might occur during the execution of
> this statement
> int x = 10
>
> 1. Virtual address to physical address conversion (TLB lookup)
> 2. TLB miss
> 3. TLB update (might involve OS)
> 4. OS may need to swap in page to get the appropriate page
> table (load from disk to physical address)
> 5. Cache lookup (tag check)
> 6. Determine line not in cache (need to generate BusRdX)
> 7. Arbitrate for bus
> 8. Win bus, place address, command on bus
> 9. All caches perform snoop (e.g., invalidate their local
> copies of the relevant line)
> 10. Another cache or memory decides it must respond (let’s
> assume it’s memory)
> 11. Memory request sent to memory controller
> 12. Memory controller is itself a scheduler
> 13. Memory controller checks active row in DRAM row buffer.
> (May > need to activate new DRAM row. Let’s assume it does.)
> 14. DRAM reads values into row buffer
> 15. Memory arbitrates for data bus
> 16. Memory wins bus
> 17. Memory puts data on bus
> 18. Requesting cache grabs data, updates cache line and tags,
> moves line into exclusive state
> 19. Processor is notified data exists
> 20. Instruction proceeds
> * This list is certainly not complete, it’s just
> what I came up with off the top of my head.
I think a lot of the confusion stems from the illusion that a uniprocessor + in-order execution model implies to programmers who have never dealt with system-level code. I think in the future, performant software will require a bit more understanding of the underlying hardware on the part of your average software developer -- especially when you care about any sort of parallelism. It doesn't help that almost all common CS curriculum ignores parallelism until the 3rd year or more.
[1] http://www.cs.cmu.edu/~418/lectures/12_snoopimpl.pdf - the last 2 slides
That has many benefits, among them the ability to store its value in registers.
If you want lock-free what do you suggest we use instead of volatile?
> which is usually much worse than holding a lock mutex in registers
How can you hold a mutex in a register? That doesn't make any sense.
Lock free in Java is usually worse that what the JVM can pull off with lock elison
For a trivial example, see this code:
int f() {
int sum = 0;
for (int i = 0; i < 10; i++) sum += i;
return sum;
}
If you make "sum" volatile, the compiler is forced to do the loop[2].
And yes, that's not stupid. It's actually faster than all the register "optimizations" for practical use cases in fast VM's. Register saving across calls and at the GC is much more expensive. mem2reg is an antipattern mostly
Looking at the formatting on the actual slides, I think the 1st is meant to be a question, and the 2nd is the answer. That the first contains the word "volatile" and the second doesn't looks to me like an editing error; they probably both said "volatile" at one time (or didn't) and the proof failed to update one when updating the other.
Isn't it sobering to think that a university slide could have a minor error like that, someone could read it and internalise it as being very important, and then go off and ask interview questions about it (as suggested on the slide!!!!) for the rest of their career!
(Not the fault of the student in this thread, of course.)
The slide is admittedly a bit vague, the point is mostly to convey "lots of complicated things that you probably haven't considered are going on in the background to speed up memory accesses in a uniprocessor model." Keep in mind the class is exploring parallel architectures, and that lecture is about snooping-based cache coherence.
if you want to force actually to ram then perhaps you'd need a memory barrier.
This is not my area though. Wrong? Right?
volatile int x;
int y;
int z;
x = 10;
x = 20;
y = x;
z = x;
the constant 10 is written to x
the constant 20 is written to x
the contents of x is read and written into y
the contents of x is read and written into z
int x;
int y;
int z;
x = 10;
x = 20;
y = x;
z = x;
the constant 20 is written to x
the constant 20 is written to y
the constant 20 is written to z
All volatile does it tell the compiler "all writes must happen, and no caching of reads".
fun fact: either swap+loopback devices+FUSE/network filesystems or userfaultfd means arbitrary userspace code execution including IO to remote machines might occur.
You're saying volatile is acting as a kind of memory barrier instruction for the compiler - got it. But I'm saying I understand that at the CPU level, just considering x86 instructions, writes don't have to be forced to RAM, despite a common assumption that they are; they can remain in caches. See johntb86's reply confirming this.
Thank you for clarifying.
There are at least 5 different sets of rules for ordering on x86, due to memory types. It's in the Intel manual, along with a table that shows how they interact with each other.
This is not true.
If you are still in the class I'd love to hear a clarification - maybe I'm wrong!
I don't understand this. If your lock variable is in a CPU register how do other CPUs acquire the lock?
> Lock free in Java is usually worse that what the JVM can pull off with lock elision
I don't understand this either. Java's lock elision is only going to make a concurrent object 'lock-free' in the case where the object does not escape the compilation unit. In which case again how would another thread use it? Java will also combine adjacent critical sections created by monitors even if they escape, but it won't make them lock-free in that case.
Volatile just makes sure the compiler bothers. Otherwise, a pair of writes to the same memory location could be optimized by eliminating the first write. Volatile makes the compiler do that. Of course, the CPU itself may then do this optimization, so volatile is thus not good enough for IO.
To be as charitable as I can possibly be, the only part that could theoretically make sense is that the compiler could emit non-temporal store instructions to bypass the cache. I know compilers currently don't do that for volatile, but I don't know why.
Two reasons:
First, using nontemporal accesses would break mixed volatile and non-volatile accesses to the same memory, something which is not defined by the C standard but which some programs rely on anyway.
Second, more importantly: why would they?
- If the address you’re accessing points to hardware registers, the page table entry should be marked non-cacheable, which makes nontemporal accesses unnecessary. And if for some reason it’s not marked properly, nontemporal accesses wouldn’t be sufficient to guarantee that things work anyway, because nontemporal is just a hint which the hardware may not respect. In any case, at least on x86, AFAIK the only nontemporal instructions access 128+ bits of memory at a time, which wouldn’t even work for hardware registers (which generally require you to use a specific access size).
- If the address you’re using points to regular memory, on the other hand, volatile is probably being used to implement atomics, in which case bypassing the cache is unnecessary and also slow. In theory, compilers could compile volatile into accesses surrounded by memory barrier instructions, which would enforce a stronger memory ordering (while being faster than bypassing the cache entirely), especially useful on architectures with weaker memory models than x86. In fact, that’s what volatile does in Java. But in C, it’s pretty long-established that volatile accesses should just compile to regular load/store instructions, and any necessary barriers must be inserted manually. People writing high-performance code wouldn’t be happy if the compiler started inserting unnecessary barrier instructions for them… In any case, usage of volatile for atomics is deprecated in favor of C/C++11 atomics, which do insert barriers for you.
For instance the processor I use has a controller that enforces consistency on IO memory operations. So volatile works 'fine'. I know that. The compiler is targeting a core not an implementation. So it has no idea.