A guide to closures in Rust(hashrust.com) |
A guide to closures in Rust(hashrust.com) |
I like the article overall, but I have a bit of a pedantic note about this opening line.
Using "like" suggests that there is some similarity that can be used to understand one thing from the other, but in truth these are two distinct (though related) technical terms, and I think phrasing this relationship as a simile may actually obstruct understanding rather than enable it.
An anonymous function is just a function without a name. There's nothing particularly exciting about that, except that they only really make sense as a concept within the context of a language with first-class functions (i.e., a language where functions are values that can be passed around as arguments and such).
A closure is a function paired with an environment. Technically (meaning "in the academic literature"), this can be any function at all --- named functions can be used to create closures. But in Rust this is not the case, since they only use the term in connection with anonymous functions. This is actually addressed in the first couple of lines of the Rust Book's entry on closures [1].
Saying "a closure is like an anonymous function" would be like me saying "a car is like a set of four wheels". While it is true that the four wheels are integral to the car's identity, it is not the case that these things are directly similar in the way that the word "like" suggests they are.
Many programmers-in-training find it confusing as to why whether or not a function has a name seems to be tied to whether or not it has an environment (not that they phrase it that way but that's what it boils down to), and my usual answer is to explain something much like your post; the resolution is, there is no such connection. I have fielded this question in one form or another many times.
"Anonymous function" should not be used as a synonym for "closure". Generally I will follow along with the dominant language terminology if I am clearly in the context of a very particular language, but may include why I don't like the language community's terminology if it helps my point.
Some other conflations I see frequently in the PL space:
- "argument" vs "parameter"
- "type coercion" vs "type casting"
- "static typing" vs "strong typing"
- "function" vs "method" (and we can throw in "procedure" and "subroutine")
Personally I think the term closure should be banned, because it only adds confusion.
There are just functions, some named some anonymous, some capturing the environment, some not, some being disallowed by some constraints from the host language. The term closure cuts through this space in a useless and rather ad-hoc way.
In some pieces of art all quadrilaterals are squares, but this doesn't mean the two terms are somehow equivalent and therefore one of the terms should be done away with.
There is a consistent technical distinction between "functions" and "closures".
> There are just functions, some named some anonymous, some capturing the environment, some not
No. Functions do not capture environments. Functions don't know what environments are. If you're talking about functions and environments together, you're talking about closures. That's really all there is to it.
Woah woah, how did it not occur to me that I could just…construct a closure that directly in Rust? This feels like one of those “incredibly obvious and reasonable in hindsight” things. I don’t know why I was labouring under the assumption they could only be invoked in very special and specific places, but it’s good to know I was wrong.
That does not take a lot of trust, after a few rounds with that compiler. At least error messages are good with the Rust compiler. I've got very limited experience with Rust, but it does seem like a language with a massive threshold for beginners.
https://github.com/juspay/hyperswitch/blob/16cd32513bc6528e0...
They're working hard on it though. https://blog.rust-lang.org/inside-rust/2023/05/03/stabilizin...
We'll have Async traits in stable Rust after the summer (normally) and in 2024 they'll start working on supporting async closures :)
So if all is well, you're experience should start to become more pleasant next year. For now your closures will either have to return a future (without use of `async` keyword) or you better avoid it for now. Then again, hadn't really had the need for async closures, but I did had a big need for Async traits, as implementing Traits with manual futures was so far a big pain in the ass, and sometimes still not optimal. So I for one, welcome async traits with open arms.
It’s one of the worst corners of the ecosystem currently, and I’d very much recommend avoiding it.
This has reminded me of https://news.ycombinator.com/item?id=35925821 recently reposted, which boils down to:
> if you’re preparing your code for others to read—whether on screen or on paper—skip the ligatures.
fn fn_that_takes_closure (f: fn (i32, 3i2) -> i32) { ... }
let x = |a, b| a + b;
fn_that_takes_closure(x);
fn fn_that_takes_closure<F> (f: F)
where
F: Fn(i32, i32) -> i32It does compile.
> because `x` does not have the type `fn (i32, i32) -> i32` (that's reserved for "normal" functions).
Closures can cast to function pointers just fine as long as they don't capture anything.
Function pointers. Rust's functions all have unique anonymous types too.
In C or C++ the functions isupper and islower (which are predicates that decide whether a char is an upper or lowercase letter respectively) have the same type.
In Rust such functions always have their own unique anonymous type. char::is_uppercase and char::is_lowercase don't have the same type.
This has a consequence when we want to use a functional-like specialisation. For example "Walter White".starts_with(char::is_uppercase) isn't just passing a function pointer - that can't work, what happens instead is there's an implementation of Pattern on things which match a certain FnMut trait, these two function types match that trait, therefore Pattern is implemented for each type, so there's a monomorphization step, baking a custom implementation of this function for this specific predicate, if you use a different predicate you get a different monomorphization.
Closures in Rust are different from named functions and that can trip you up.
Closures in Rust also close over lifetime information in ways that named functions cannot do. This confused me for quite a while.
That Rust can do closures at all, do them in a sufficiently useful way that they are practical, and still maintain its safety guarantees without garbage collection is, in my opinion, in the top five accomplishments of the language. Prior to it actually happening I think it's fair to say most programming language developers would have said it's not possible, that you'll either need GC or the requisite type system will be impractically complex. At least their quirkiness generally fits the rest of the language and isn't much "new" quirkiness.
Fundamentally closures are easy in e.g. Go because you don't have to think about lifetimes, at all. As soon as you capture a variable, the GC guarantees it won't be dropped from underneath your feet. With non-GC'd languages that responsibility moves from the GC to the programmer. The trickyness of using closures in Rust seems largely to be the trickyness of managing the lifetimes of captured variables.
With those in mind, Rust's complexity does indeed look accidental here. It has its other benefits, but it does make closures a bit more difficult.
Rust is a modern, functional programming language where the programmer works in a straightjacket, and this is most painfully evident the moment you want to use closures like you're programming in Haskell.
In Haskell you don't have to think about stack vs. heap allocation. But w/o a way to make it easier for the compiler to choose stack allocation where that would be safe, Haskell ends up being heap-heavy. Rust takes the opposite tack and is stack-heavy, but unlike Haskell Rust forces you to be explicit about the alternative.
It's not like Rust doesn't have a GC. Arc is a GC after all. It's just that Rust makes it hard to write natural code using the heap and GC.
Coming from a functional background, when I first started programming Rust I really tried to leverage FP concepts as much as possible. However, I slowly began to realize that the FP version of the code I wanted to write was more complicated, more lines of code, and slower than the naive imperative version.
Clean FP without GC seems really difficult, and I think Rust team did as good a job as they could given the current state of research.
Rust does not have a massive threshold. Memory management has a massive threshold. Rust just has a lot of safety rails you might keep bumping into if you struggle with memory management. The fact that other languages let you drive off the cliff doesn't mean they have a smaller threshold.
Personally I have a background in C/C++ and moved towards C#, Go, JS etc. With an interest in many more, but the above were my main languages for a decennia.
9 years ago I picked up Go, and have used it the most as my main language since then. 7 Years ago I picked up Rust, but only since 6 months ago really intensive (including publishing the guide that came out of my experience at https://rust-lang.guide/).
Over those 7 years I've only done a production project around the start of that journey and now since 6 months ago I'm writing production code in it again and haven't stopped since. But I've learned the language like 3 different times in that journey.
It's not an easy one to get into. A lot of things to wrap your head around, especially if your goal is to really master it, rather then just "woohoo I can compile my program". I still not master it, but I do feel now fluent in it and can express my ideas well. There are also no more fights with the language or its tooling.
Compared to that, Go(lang) is super easy to get into. Ridiculously easy. Perhaps too easy. You can give it to a programmer of any level, and they probably can ship their first feature that same day. The difference is however, that. First of all with Golang it's super easy to ship code with nil pointer exceptions in it or data races. I've seen it in the best code bases. And sure, plenty of people will tell if you do it "right" you'll have no issues. I've hear similar comments from ex-colleagues still in the C++ world as well. Secondly, Golang is very opinionated and if you derive in ideas or needs a bit from what they want you to do, you are in a bit of a problem. This has been slightly improved, but it's still very limited and still a lot of magic that only built-in features can perform. And thirdly, the language is very minmal (Golang). So codebases look very verbose very quickly. Also harder to express ideas (as error support is limited, and sum types are not a thing, neither are other things like pattern matchings). It's also a lot less "functional".
This is not to compare Golang vs Rust too much, even though I did. My point was more, and I probably butcher my own comment not making that point well, is that I think it's fair to have a massive threshold for beginners to get into a programming language. As long as in return it means you get a very powerful tool in rerturn to do some of your work. And Rust is def. such a tool. Many people boast about how peformant it is. And while that is true, it is for me for most projects more of an extra benefit rather than the things I really like it for. What I like with Rust is that it gives me really high confidence in my code (comparable to projects I've written in Elm and Haskell in the past). When my project compiles I really am pretty confident it will not give me runtime surprises beyond mistakes elsewhere (e.g. infrastructure choices). It's a very expressive language and pleasant to use in that way.
And then there is of course the fact that it is truly FOSS, has a very nice community, great documentation and a lot of excellent learning resources.
I am not a fanboy of much, neither of Rust, but I do really appreciate its existence and I am happy to use it where it suits. Yes it is a massive threshold, but one worth to pay.
My answer to that is always that if you are able to do it right in C or C++ (and maybe golang) you should not run into any issues with Rust. Especially you should not ever have to fight to borrow checker because it does the job you should be doing as a C/C++ programmer in your head anyway.
A couple weeks ago I tried to get started with Go and the hello world code example wouldn't work with `go run` with some obscure module error that's meaningless to a noob, but sure it's "ridiculously easy"
Go is inspired a lot by CSP, and allows you to apply CSP calculus when using goroutines and channels.
It doesn't make it impossible to make mistakes, but it does give you tools needed to reason about your concurrent program, which is more than can be said of most other languages' concurrency support.
Programming with manual memory management should be (comparatively) hard. It's not where beginners should start. C makes it far too easy to write code with objectively bad effects on the system, as evidenced by the countless vulnerabilities discovered in critical real-world systems over the last half-century.
> after a few rounds with that compiler.
I find this phrasing curious. Do you feel that you are fighting the compiler?
Let me give you my opinion who just started learning 3 weeks ago. And I needed around 1 week until I could write a simple parser by hand and around 2 more until I am now being quite productive in this new language.
It is actually very simple go get into, depending on your experience. What do mean by 'threshold for beginners', what are beginners for you. Do you mean someone learning their first programming language or someone already proficient with one or multiple trying to learn Rust now? Because a lot of the features of Rust are present in other language and your mental models will be similar enough for them in Rust accelerating your learning curve. Of course someone new to programming will have problems as the features that are unique to rust (borrow, move) in addition to everything else leads to a lot of overhead sure.
My experience is with Java, Python and Javascript/Typescript as I know them sufficiently well to create programs and I also dabbled with many other languages just trying out how they feel (Scheme, Racket, Nim, Go, Haskell, C, C++, Ruby, Smalltalk). When starting with Rust it took very little time to get comfortable. The language feels very well designed and Option, Result type are very logical to use for return types. Many languages have these so depending with what you are familiar with this might be new to you but it is surely not more complicated than explaining try/catch blocks. Of course dealing with Option/Result is a bit uncomfortable in the beginning but If you just browse a bit through the standard library and learn about the `?` operator you see many patterns how to deal with them in an efficient manner (map, map_or_else, or, or_else, flatten, unwrap, unwrap_or,...). Rust has closures but now you find them in almost any language so it is not really surprising (maybe only nuances that come up with the borrow checker). Most of the standard traits and enums are quite straightforward such as `Default`, `From`, `FromStr`, `Iterator`, `Clone`, `PartialEq`, all the operator traits.
Also the borrow checker and some lifetime compiler errors are for sure a thing that can lead to thresholds and for my time spent with the language I would say there are many things I will need to learn to fully be able to grok the language. However, most of the time you can circumvent these problems by just not using lifetimes in your own code and by cloning variables. So for people that want to write really efficient code you can do this without cloning variables if you have a better understanding of them but as a beginner you have a simple escape hatch most of the time and allows you to gradually gain a better model of lifetimes/borrowing rules.
I think the only really threshold is the trait object problem. If you want to build any kind of nested structure or if you want some kind of "polymorphism" which is a natural reaction depending which programming language you have used. You will need to deal with understanding them and this might take some time.
You can insist on correct terminology or adapt to your surroundings and try to speak the language of your target audience. I assume somebody who calls closures "anonymous functions" is speaking as a JavaScript developer to JavaScript developers. Even if not, it's not unreasonable to adopt the terminology of such a popular language.
It is true that a lot of language communities conflate terms, and this is especially prevalent among the people who are hobbyists and amateurs in the space --- an important community, to be sure! But not a group of people who are already knowledgeable.
When people from the lay community of Language X choose to get involved in the community of Language Y or, more importantly, the community of programming languages research, they often encounter friction because suddenly these terms that they previously believed to be synonymous actually have very distinct technical meanings. So that is why I try to address these conflations when I see them.
That said, I do make a point to state up-front that my notes are pedantic, because I know not everybody cares to learn about such things, and that's perfectly fine! But I will continue writing corrections when I see them, though I try hard to make my corrections kind in nature.
They might not do that in Rust, they do in Haskell, JavaScript and plenty of others.
A "function", in the academic literature, is a form that has two parts: a set of variables that it binds (called the parameters) and a bit of encapsulated functionality (called the body). The function itself has no knowledge of the outside world. This is why the notion of free variables is important, or even relevant. A free variable is any variable that occurs unbound within the body of a function. The function doesn't know anything about the free variables; they simply exist within it.
A "closure" is a pairing of a function with an environment. If you are talking about functions and "their" environments, you are talking about closures. In most languages, a function is only syntactically valid if all of its free variables are bound within the surrounding environment. (However, if you were to implement a language with dynamic scope instead of the now-standard lexical scope, you wouldn't even check such a thing statically.)
---
I'd like to try to make my point by drawing an analogy to another pair of PL terms that are often similarly conflated: "parameters" and "arguments".
A parameter is a variable that occurs in the binding context of a function definition, and is therefore considered to be bound within the body of the function. Parameters are not free variables.
An argument is a value that is passed to a function during a function call. Arguments are bound to parameters during the set-up of the function call.
Let's imagine I want to talk about an anonymous function `(λ (x) (add1 x))`. This is a function that has one parameter, `x`. It then returns the result of incrementing the value of `x` by 1 (we are assuming the value is a number).
If I had instead said that this function "takes one argument `x`", it would technically have been incorrect. When we're talking about the function, we know nothing about the arguments it will eventually take; we only have information about the parameters.
It is exceedingly common for people to use these terms interchangeably, but they are actually distinct in the academic literature, as they each refer to entirely separate (though related) things. The function/closure distinction is similar. Many people use the terms interchangeably, but they are actually meant to be separate. You are conflating them, and my comments here have been meant to educate people about the actual distinction.
There’s a reason many projects ban the go keyword in favour of a wrapper that fixes some of this.
Go requires constant vigilance, sadly.
But saying it "requires constant vigilance" is an overstatement. As long as you put a little bit of thought in the concurrent code you're writing, it's very easy to do things right. Data races are easy to avoid if you only transfer ownership by sending pointers through channels.
The reason Go annoys people is the unforced errors. Sure, it gets a lot right. But what it gets wrong had solutions long before Go existed and those solutions were wilfully ignored.
In practice, where I work we disallow the go keyword in review and require use of a wrapper that takes in a closure and also is a wait group. It’s much harder to get wrong.
The reason I feel this stuff is important to mention is because it is not uncommon to find discussions where the conflation does cause people headaches. The problem is exacerbated by the prevalence of people who claim that the difference is "irrelevant", up until the point that the difference actually matters, at which point those people are simply not around. There are a ton of small sets of terms of this nature that, when speaking to laypeople, you would think are just synonyms, except that it turns out that sometimes the technical distinction is important. It means people who actively seek out spaces to educate those who want to learn (like myself) have to do extra work to undo the faulty learning and then start over with correct definitions. Like you said: for most people it's not a big enough deal to worry about, but the distinction is present and there are times when it does matter.
I like informing people of things, and I happen to find interest in minute details, definitions of terms, and so on. I know that's not for everyone. To solve this "problem" (if I can call it that), I try to start my discussions by being honest about my intentions: I explicitly state that the point is minor or pedantic or something to that effect. That way I get to write about the things I want to write about while also giving people plenty of room to just choose not to worry about it if they don't care, and everybody comes out the other side happy. Except you, I guess. Sorry about that.
The difference is meaningful here. You have to allocate a closure (and deal with its lifetime and the lifetimes of the variables it references) but the anonymous function is just a pointer to static code in the binary. That's the entire difficulty with closures in non-GC languages.
This distinction matters less in GC languages where you're not thinking about lifetimes either way.
Also, your arguments only partly apply in Rust. Rust doesn't heap-allocate closures. And you also often don't have to deal with lifetimes - a closure that captures variables by move or copy is perfectly self-contained
The difference between a closure that captures and a closure that doesn't is like the difference between `(T)` and `()` - same kind of thing, so it adheres to the same terms and behaviors
That's incorrect, a closure in Rust compiles down to a static function that takes its environment as an argument. None of that requires a heap allocation in the above code.
According to the Rust reference:
> Closure types > > A closure expression produces a closure value with a unique, anonymous type that cannot be written out.
https://doc.rust-lang.org/reference/types/closure.html
Even a non-capturing closure is written using a closure expression and produces a unique, anonymous type.
> Unlike a plain function, a closure allows the function to access those captured variables through the closure's copies of their values or references, even when the function is invoked outside their scope.
> The term closure is often used as a synonym for anonymous function, though strictly, an anonymous function is a function literal without a name, while a closure is an instance of a function, a value, whose non-local variables have been bound either to values or to storage locations (depending on the language; see the lexical environment section below).
My understanding is that your use of 'capture' matches up with the the use of 'bound' in the description above.
https://en.wikipedia.org/wiki/Closure_(computer_programming)
As a parting gift, I'll give you one guess what the title of the Wikipedia article on thread stacks is.
Since 2017 it certainly admits quite a few more programs, but it's still pretty easy to find things that require a different approach than they would in C or C++.
let mut animal = "fox".to_string();
let mut capture_animal = || {
animal.push_str("es");
};
//ERROR:cannot borrow `animal` as immutable because it is also borrowed as mutable
println!("Outside closure: {animal}");
capture_animal();It adds mental overhead.
I don't even like the "const" qualifier for functions in C++. It is so hard to get a design right upfront when thinking about non trivial 'const'-chains.
The borrow checker removes the mental overhead of borrow-checking. You don't have to do it by yourself, you can lean on the borrow-checker to do that for you. In C++ you're all by yourself. You may pretend you don't need to do it, but you will run into troubles pretty fast.
To me `const` removes mental overhead -- if I pass a const object somewhere, I can be sure it doesn't get changed. I don't need to inspect the code to make sure it doesn't call any methods that modify the object or trust a probably outdated comment or documentation.
False. Too many false positives
It does so better than you can in that it doesn't have false negatives anywhere you're sufficiently playing by its rules.
You can also do so better than it can, in that you can see reasons things are okay that it can't understand.
I agree that there are too many false positives to assert that a competent C or C++ programmer doesn't have learning to do to appease (and leverage) the borrow checker.
Which would interfere with Go's philosophy that zero should be a valid (usable) state for types. What would be the default value of a reference type without nil?
> separate goroutine address spaces
This is not trivial to implement. The only way I can think of is to start each goroutine in a new process, which brings along other downsides - everything must be copied, cannot pass file/socket handlers between goroutines, etc. Doesn't seem to be worth it.
> crash the process on unhandled panic in any goroutine
Go does crash the process on unhandled panic in any goroutine:
# main.go
package main
func main() {
go func() {
panic("hehe")
}()
for {
}
}
# go run main.go
panic: hehe
goroutine 5 [running]:
main.main.func1()
/[...]/main.go:5 +0x27
created by main.main
/[...]/main.go:4 +0x25
exit status 2
This is the criticism I've also had, and raised an issue on GitHub. Unfortunately, it's impossible to fix this without breaking backwards compatibility.
> But what it gets wrong had solutions long before Go existed and those solutions were wilfully ignored.
They were wilfully ignored, but only because the tradeoffs didn't seem worth it at the moment. You make it sound bad, as if Go devs were simply high on crack and knew what they doing was wrong, but did it anyways, when in reality it's more of a case of those "solutions" not being compatible with the goals of the language, or simply nobody coming up with an elegant way to incorporate it.
... but nil is not actually a usable reference. You can't do anything with it that you can do with a real reference.
The whole zero values for every type might seem neat philosophically, but never struck me as sensible.
Separate address spaces are trivial when you have a runtime, just look at Erlang.
Go could’ve also copied Erlang’s supervision trees to ensure safe and bounded goroutine lifetime.
For all their impressive achievements, Go’s creators suffer from extreme NIH syndrome. Just look at Plan 9.
Erlang's "address space isolation" is not real address space isolation, only semantic. Erlang "processes" are all still in the same address space, but the language semantics doesn't allow them to interfere with one another.
You can implement bounded goroutine lifetime by using sync.WaitGroup and other constructs for coordinating goroutines. Go just doesn't force you to.
> Go’s creators suffer from extreme NIH syndrome. Just look at Plan 9.
This is just bad faith arguing. Plan 9 was revolutionary in many ways, and brushing it off as just NIH makes me think you don't understand or recognize its achievements.
Go even shipped with segmented stacks! Also separating goroutine heaps would've been cheap by comparison. It would've also made it easier to achieve good garbage collector throughput, which Go still struggles with to this day.
Go could've also had the `go` keyword work differently, by requiring/returning some kind of handle that must be waited to potentially panic, or explicitly ignored. It would've made it impossible to incorrectly use WaitGroups, without any additional runtime overhead.
Don't get me wrong, Go is still a useful language. It's just frustrating that it failed to be great language for no good reason.
"cascading constness change"
Which is good because invoking code may rely on function being read-only.
The problem with nil for any pointer or interface is that you can compile code dereferencing it without first checking if it's nil.
Yes, Go doesn't stop you from making mistakes. I think we've already rehashed that a couple of times...