Bugs You'll Probably Only Have in Rust(gankro.github.io) |
Bugs You'll Probably Only Have in Rust(gankro.github.io) |
[1]: https://github.com/laumann/compiletest-rs
[2]: http://erickt.github.io/blog/2015/09/22/if-you-use-unsafe/
It would be neat if we could decompose unsafe like so "unsafe[this_feature,that_feature] {}". The unqualified "unsafe" could still refer to a global "free reign", but you could opt-in to "only let me violate these specific rules." It would be a hint to maintainers and might help make the std lib and other core libraries be/remain defect-free.
Another interesting "oh shoot" w/unsafe that I'm curious about: when I intentionally/unintentionally alias two variables in my unsafe block, this will invalidate assumptions made elsewhere in safe code. This is my unsafe block's bug, but it seems like something that could take a good while debugging to attribute back to my unsafe block. I don't think there's a good resolution to this one other than perhaps documentation/best practices.
Note that we sort of made a "new" kind of unsafe with the UnwindSafe trait: https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html
That's probably how we intend to solve these kinds of problem in the future.
Re: aliasing -- if it's a serious enough problem, one of two things will happen:
* Someone will develop a version of asan/ubsan for Rust.
* The Rust devs will be forced to reduce the extent to which they apply alias analysis by default (possibly with a flag to opt into it). At least temporarily.
The rust devs have backed off optimizations in the past when they break stuff in the ecosystem (struct layout optimization). But they also work with the affected devs to fix those bugs so they can turn the optimization on.
This already happened (japaric [1]). But ASan won't save you from a bug due to optimization-because-I-assumed-these-locations-dont-alias (maybe TSan might?).
[1] https://users.rust-lang.org/t/howto-sanitize-your-rust-code/...
You are correct: it's possible to write nefarious code inside an 'unsafe' block then only suffer its effects outside of it, and Rust has to document that fact. The Nomicon, mentioned in the blog post a whole bunch, points this out early on:
> 'unsafe' does more than pollute a whole function: it pollutes a whole module. Generally, the only bullet-proof way to limit the scope of unsafe code is at the module boundary with privacy.
[ https://doc.rust-lang.org/nightly/nomicon/working-with-unsaf... ]
I sometimes feel the same way, but remember that the `unsafe` keyword only unlocks four additional features:
1. Dereferencing a raw pointer
2. Calling an unsafe function or method
3. Accessing or modifying a mutable static variable (and this might conceivably even be removed entirely someday)
4. Implementing an unsafe trait
It's unclear to me how to make this any more fine-grained such that annotating the "kind" of unsafe you're using would be useful and enforceable by the compiler (which is crucial, because otherwise why not just use a comment?).
In practice I think really the only "distinction" in unsafe Rust that I want is the ability to distinguish unsafe blocks that exist only to call external C code.
Mutable static variables removed or the unsafety of accessing them? Didn't Rust, at one early point, not allow mutable global variables?
As part of the unsafe code guidelines effort, there's been talk of adding an 'unsafe checker' mode to rustc, analogous to valgrind or Clang's AddressSanitizer, which would alter code generation to add checks that could catch many classes of incorrect behavior at runtime. (This would have a high performance cost and would be intended as a debugging tool.) One of the things it would probably do is keep a global map of all live references, and complain if references are created that break the rules, e.g. a mutable reference is created to something that already has a (mutable or immutable) reference somewhere else in the program. Thus it could catch the kind of bug you mentioned.
Of course, you would have to remember to run the checker, and as a dynamic rather than static analysis it would only catch errors that are actually exhibited at runtime (so it probably wouldn't catch the MutexGuard example from the original blog post, unless there was some real code that raced on a MutexGuard). Still, in practice it should help a lot with ensuring that unsafe code doesn't break the rules.
Edit: Niko talked about this in a blog post in February. He proposes a somewhat more complex tracking system than the global list of references I mentioned:
http://smallcultfollowing.com/babysteps/blog/2017/02/01/unsa...
if implemented it would probably outperform haskell (given that monad transformers in haskell have runtime overhead, unfortunately).
edit: Go also has zero-sized types (struct{}), so I wonder if this is also possible? Probably not, I don't think, since the compiler doesn't see through interfaces.
[Talks will be recorded, but also tickets are on sale right now if you want to be there in person!]
No. It specifically uses Rust's generics system, and the fact that generics are monomorphized at compile time, whereas Go interfaces are not.
C++ templates can be used in similar ways.
In a lot of ways it makes me trust Rust even more, because there is a deeper understanding of exactly how these guarantees are made.
IIUC, technically, the bug was a missing implementation of a trait and the result was a data race (which I (weirdly, maybe) don't think of as memory safety).
In other words, TL;DR: magic is neat, except that sometimes it really sucks.
I may have misunderstood Ralf's bug. Is it really the case that MutexGuard<T> was seen as Sync if T was Send, rather that Sync? Wouldn't that be a bigger problem than just the case of MutexGuard?
One framing of the MutexGuard problem is that the type wasn't declared in a way that reflected its semantics best, although it is clearly unfortunate that doing this is more complicated than the incorrect way.
So T: Sync if &T: Send. MutexGuard internally contains a &Mutex<T> (and Poison, but that's irrelevant here). T was Cell<i32>. If you follow the rabbit hole, you'll net out that T was Send, and therefore MutexGuard was Sync.
You could imagine an alternate world where MutexGuard is Send, to allow transfer of ownership of a lock to a different thread while keeping the mutex locked. But that would mean &MutexGuard is Sync, WTF?
You understood the bug correctly, but its not a bigger problem. You probably are lacking context on auto traits, but this blog post contains the context you need if you read it again.
"This means that the compiler considers a type like MutexGuard<T> to be Sync if all its fields are Sync."
Is that true in general? Is a type thread safe if all its fields are thread safe individually?
2D Second Life clone, with full programming capability with built-in database - 2 megabytes. Solid ASM. Rust can't even come close, and never will.
And take 100x longer to develop :)
Additionally, not having the facility for low-level/unchecked code just means that things like optimised data structures/memory management/hardware interaction get implemented either in the compiler or in other languages. The former is much harder to reason about and to modify: one is essentially writing code that generates compiler IR, which is more annoying and error prone that both just writing the code directly and just writing the IR directly (one way to think about this is the compiler is one big `unsafe` block). The latter is unfortunate because it results in impedence mismatches when doing the FFI calls both semantically and with performance, and it also means that code doesn't get to benefit from the usual Rust safe checks and high level features (like ADTs) that are all still available inside `unsafe` blocks.
let p = 0xb8000 as *mut u8;
In order to demonstrate this is safe (okay so unsafe isn't in this example, creating p is safe, but writing to/reading from it is not), a language would have to know:
1. That your code is running in kernel mode, that is the entire concept of ring 0 vs ring 3.
2. That the VGA spec specifies that location in memory.
Yeah, in _theory_, you could have a language that does this, but that'd tie your language so, so, so deeply to each platform, that it's not feasible.
This can be extrapolated to all kinds of other low-level things.
That need not be the case though. You could have a kernel side allocator that sets up the MMU to map that memory to a pointer that you return which lives in the space of the process. The MMU would take care of the required arithmetic to access the memory at its actual location using an offset.
That way you can map resources from real addresses into arbitrary addresses on the user side.
I think the correct term for this mechanism is 'system address translation'.
Like, when you compile for x86 there are a bunch of rules that aren't generally safe, but on that platform they are.
* All unsafe operations don't exist.
* All unsafe operations exist, but the literal unsafe keyword and its machinery doesn't exist
The latter is how most ostensibly safe languages work. See Haskell's UnsafePerformIO, Swift's UnsafePointer, and Java's JNI for 3 examples off the top of my head.
The former is just a really gimped language that would have been a pain in the neck to implement libraries for (see other replies for examples).
A lot of built-in constructs uses unsafe under the hood. Vector (Rust's dynamic arrays) does memory allocation / resizing under the hood for example, and there's no safe way to do it, unless some safer array allocation primitive is exposed.
Also things like mem::swap(x, y) cannot be implemented at all with safe rust. in order to perform swap, you need a temporary variable. That temporary variable would be uninitialized, which Rust does not allow.
Note that in c++ it invokes copy constructor - http://www.cplusplus.com/reference/algorithm/swap/ - but Rust's mem::swap works for types that does not implement the Copy trait (Rust's equivalent to copy constructor).
Same can be said about slice splitting functions, which is primarily used to work around Rust's borrow checker.
You are correct to be confused about (&Cell<i32>) possibly being Sync, because the assumptions that were implied were wrong: when Sync talks about sharing a T, that can be entirely thought of as transferring a &T to another thread (aka Sending the &T). In this sense, sharing a &T between threads (as in, (&T): Sync) is the same as transferring a &&T to another thread, but the inner &T can be copied out so the original &T was also transferred (not just shared) between threads; that is to say, (&T): Sync is 100% equivalent to T: Sync.
Anyway, back to the example here, Cell<i32> is not Sync, so neither is &Cell<i32>, but M = Mutex<Cell<i32>> is Sync (this is a major reason Mutex exists in that form: allowing threadsafe shared mutation/manipulation of types that do not automatically support it), and thus &M is Sync too. Since MutexGuard<Cell<i32>> contains &M, it was thus automatically, incorrectly Sync.
For your second confusion, it is okay for &MutexGuard to be Sync, if MutexGuard itself is. The problem here was MutexGuard was Sync incorrectly in some cases. (MutexGuard is semantically just a fancy wrapper around a &mut T, and so should behave the same as that for traits like Send and Sync.)
The compiler does make the author pause: dangerous types like `*mut T and `UnsafeCell<T>` are not Sync, so types containing them are also not automatically Sync.
In any case, Rust only guarantees no memory unsafety (requiring no data races). It tries to help with other things, like cleaning up resources via destructors, but these are "best-effort" rather than guarantees. The only way to get a data race and/or memory unsafety with an automatically implemented Sync is with `unsafe` code, and any `unsafe` code near a data type "infects" it and so means the whole type require great care.
Tooling like asan and tsan and, hopefully, Rust-specific sanitizers and static analyzers will make this easier to get right, but fundamentally as soon as `unsafe` comes into the equation the programmer has to be paranoid. Of course, as the MutexGuard problem indicates, humans getting it right error-prone, which is why the aforementioned tooling and formal proofs---like the one that found that problem---are important, as is building and using appropriate abstractions (e.g. MutexGuard is semantically designed to be a &mut T, so maybe it could indicate this by using PhantomData, or even just storing that directly: this does require manual work, but pushing conventions like that might bridge help the gap to having great `unsafe` tooling in future).
It is definitely an important missing piece to bridge the gap to ASan/TSan, but it's still just a proposal/work in progress, not least because, AIUI, the precise rules it needs to enforce aren't yet entirely clear.
Just calling a function taking T: Sync doesn't need to do any of this, since that call happens all on a single thread. The function might do it internally if it needs to, but that is its own explicit implementation decision.
That does bring up the question, though, whether it's correct to say that a Sync type doesn't permit data races. In the example I gave above, publishing a Sync struct incorrectly can exhibit data race like symptoms on the receiving thread. So even though the type itself is Sync, that's not enough of a guarantee in the face of "unsafe" publication.
I guess "Sync types don't have data races" is the abbreviated version of "Sync types don't have data races in any safe code, no matter how weird and wonderful". That said, this qualification doesn't seem very interesting to me: something equivalent is required about pretty much any statement about any guarantee in any language with unsafe code or FFI (e.g. in Python, something along the lines of "pointers don't dangle in any code that doesn't use `ctypes`"), and thus is elided in a lot of discussions about programming languages.
If you consider `unsafe` Rust, then failing to fence correctly is just one way to get a data race.
1. It is "auto derived" when all of its constituent types are Sync.
2. It is explicitly implemented using `unsafe impl Sync for T {}`. Note the use of the `unsafe` keyword.One is by sharing the data with the thread when it is spawned via a closure. Spawning will fence. No problem there.
The second is to use a good 'ol Sender/Receiver channel pair. These are effectively a shared ring buffer that you can push to and pop from. They also have a fence somewhere.
Finally, you can stick your data into a mutex shared between threads (and let the other thread wait and read it). This will IIRC fence, or do something equivalent.
You can of course build your own ways to do this, but they will need unsafe code to be built (the three APIs above are also built with unsafe code). It is up to you to ensure you handle the fences right when doing this.
The responsibility here is on the publishing mechanism. Most folks use one of the three ways above using primitives from the stdlib depending on the use case.
The safety of things in Rust is built on abstraction. If abstraction gets something wrong in its `unsafe` details, then there is a bug there. In other words, the asterisk you're mentioning is "You can trust that safe Rust is free of memory safety, unless there are bugs." I feel like that's discussed and acknowledged quite a bit.
> Unsafe code is usually discussed in light of raw pointers and more generally raw memory ops, but I rarely see this aspect mentioned.
I guess I don't see a difference. The safeness of Rust code depends on the correct use of `unsafe`, and this applies to everything, not just Sync.
This idea of `unsafe` being "trusted code" is one of the first things that the Rustonomicon covers: https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html
At a high and general level, yeah, it's all "unsafe". But, most conversations about unsafe don't talk about this aspect. So while what you say is true, I'm merely pointing out that this concurrency aspect doesn't seem to be mentioned much. And while it's implied at a high level, I think it's worth mentioning.
Basically, there's no issue - I just think this should be called out more when concurrency is discussed.
Concurrency is not special here. There are all kinds of invariants unsafe code might be required to uphold. So yeah, we could mention concurrency, but then we could also mention UTF8, noalias, initialization, the vector length invariant, the HashMap robin hood invariant, various BTreeMap invariants, etc etc. "Make sure you have fences" is just another semi-specific invariant.
I disagree that "most conversations about unsafe don't talk about this aspect", compartmentalizing unsafe invariants is a major part of these discussions (it's like the first chapter of the nomicon, even)
Unsafe discussions that I've seen rarely talk about fences - they tend to focus on raw pointer ops, ffi, transmutes, unbound lifetimes, and in general, are single thread focused.
Concurrency is particularly complex, perhaps. I think one of the reasons you don't see that much discussion of this is that in general folks in Rust don't write that many internally-unsafe concurrent abstractions. There are a bunch of great safe building blocks out there (stdlib ones, rayon, crossbeam) which folks use for concurrency; it's very rare to build your own. So that might be it.
At least with the stuff I work on like 50% of the unsafe Rust discussions have been around thread safety and ordering and fences, but we're in that relatively rare situation where we need to build those abstractions, so perhaps it's just me who sees these discussions happening.
-----
It's also probably just that discussions introducing unsafe will deal with problems people are used to -- and memory safety is a far more "normal" problem than thread safety.
Given there's no official memory model, similar to say Java Memory Model, there's not much to go by (correct me if I'm wrong). The JMM, for instance, spells out what needs to happen for happens-before edges to be created. It also talks about racy publication of values. Granted there's no close analog to Rust's unsafe in Java. But saying "T: Sync means it's free of data races in safe code", while correct, is a slightly vacuous statement since that T interacts with other components. And yes, those components will likely involve unsafe code, and unsafe code has its own caveats, but still, I don't think it hurts to make this more pronounced. Concurrency is a special beast, rife with its own hard-to-debug hazards. Being a bit more verbose and possibly repetitive about the hazards won't hurt :).
It is vacuous at a big picture level where you're trying to understand the complete thread safety story, but that was never what that statement was trying to convey.
----
I think documentation here can be improved, though. When I get more time one of the things I want to do is do a major revamp of the concurrency docs, including paragraphs on how the memory ordering stuff works. I'll include filling out the Sync docs in this, thanks for the feedback!
Sequential consistency is irrelevant to the example I gave of a single valued struct. I wasn't talking about any ordering with respect other loads/stores.
In any case, the only reason to think about publishing is how loads and stores are ordered, so it isn't irrelevant. The single valued struct can still have the partial initialisation problem, it just appears as the single field being completely uninitialised.
I know one can look at this as being unimportant because only unsafe code that works with Sync types carries the burden of upholding the invariants. But, I still contend that memory ordering ought to be discussed, somewhere, while there's no official memory model documentation (not even sure that's in the plans, although I feel like I may have come across a github issue for that).
Anyway, I don't think I'm saying anything new in this reply over my others. If you feel existing docs are adequate in this regard, that's fine - we can agree to disagree :).