In Go, pointers (mostly) don't go with slices in practice(utcc.utoronto.ca) |
In Go, pointers (mostly) don't go with slices in practice(utcc.utoronto.ca) |
In Go, you get a stable version of the old data, and the garbage collector tracks that you still have a reference to it. This is safe, but confuses some people.
In Rust, the borrow checker won't let you modify the array while you have a reference to a slice of it. So you can't do this at all.
In C++, you get a mention on the Department of Homeland Security's US-CERT site.
Rust has created a weird perception that memory safety equals safety. Language is a tool and it should work with me: it is extremely important that my understanding of what the program should do aligns with what it actually does.
The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container". Taking a pointer/reference usually means quite an opposite. So it is "safe" in a sense that the pointer points to valid data, but is "incorrect" in a sense that it does wrong thing without warning.
Sure, one could argue that value-returning modification functions are a giveaway of invalidated data. But this is not C, go has reference counting and instead of "forcing" underlying array to maintain the same address it just keeps original pointer pointing to dereferenceable, but wrong data.
A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data.
Typically slices are passed around by value but you can take their address and modify a "shared" slice.
The built-in append() returns a new slice by value.
What happens is simply that when appending data to a slice and there is no room in the backing array, a new backing array is allocated that the returned slice points into. The old "input" slice to append is still intact and if some code has access to it, it will look at data stored in the old backing array.
I've constructed similar utility types in C and find them quite convenient. It's very convenient to have the distinction between the backing memory (array) and a slice viewing a portion of it instead of just a dynamic array.
I think it's a bit more than that. They're also riding on the static typing trend that's happening right now, so type-safety is also part of the equation. From the website:
> A language empowering everyone to build reliable and efficient software.
> Reliability: Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — enabling you to eliminate many classes of bugs at compile-time.
That means that people like me that still have a hard time with C and C++ can build efficient software using the same workflow as I'm used to in my usual "web languages" (Python and JS mostly).
Of course, since append neither guarantees nor prevents a copy, the semantics of modifying a value through a pointer to a slice element after an append are unspecified, so it is not a useful construct.
No, there is no mention of a "snapshot". You get a reference to the current backing array, which may or may not continue being used by the slice (depending on reallocations). You're pointing to the live slice backing array, and the values in it may change if someone else is manipulating the slice, up to the point where the slice backing array must be reallocated, at which point you'll continue pointing to the old backing array and be keeping it from getting GC'd.
It's not without warning, it's a well documented behavior.
Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.
Mutating them in any way is actually a special case that you do only for performance reason (i.e. you can pre-allocate and fill if you know the size ahead of time) but - as always - you try to keep those abstracted away and to the minimum.
There’s no need for Go to copy anything in the circumstance the OP described. It just doesn’t shrink the underlying array.
slice = append(slice, item)
struct intSlice {
int* addr;
int len;
int cap;
};
This can be confusing if you're used to C++'s std::vector (which owns the memory) or Python's slices. Go's slices are a shallow pointer/length system exactly like is used in C all the time. For example:
void sort(int* addr, int len);
func sort(a []int)
When you take a pointer to a slice, you get a pointer to the current version of this tuple of information for the slice. This pointer may or may not refer to a slice that anyone else is using; for instance:
ps := &s
s = append(s, 50)
At this point, '*ps' may or may not be the same thing as 's', and so it might or might not have the new '50' element at the end.
The author seems to have failed to distinguish the operation of copying a fat pointer (which opens the possibility of divergence) and the operation of the taking the address of a fat pointer (which involves no copying, so divergence is not possible - where would the divergent version be stored?).
See this code snippet: https://play.golang.org/p/tdb-O8a6hDN
s is a local variable (or global, doesn't matter). ps simply points to that local variable. You can modify the local variable all day long and ps will still point to it, not some old version of it.
To be clear the author is incorrect.
In C++, what happens is that the "iterators are invalidated" when you add something to an array. This is CONSTANTLY a source of bugs and frustration for new programmers. In C++, it may yield results or crash your program, and you are never sure quite which will happen. The best you can do (as a senior engineer) is design your software to avoid ever creating this situation in the first place and throw address sanitizer at things to try and catch them when they arise. The difference with Go is that in Go this will never result in a memory error.
Strictly speaking, the situation in Go is way better. I will take "incorrect behavior, but not a memory error" over "memory error" any day of the week.
We may forget what it's like for new programmers, but for those of us who hang out on Discord channels, Stack Overflow, and Reddit giving people help with programming, simple things like iterator invalidation are a major pain point.
"You have a memory error in your program", I say to someone. "Now that you know that you have a memory error, it is probably your highest priority to find and fix this error." And now you start walking someone through the steps of finding and fixing a memory error, which is nontrivial. You'll tell them about Address Sanitizer, GDB, and Valgrind, and you'll wish them luck.
Incidentally, this is equally true of creating additional pointers or slices pointing into an growable array. They aren’t safe after the next append(). If you grow an array then you need to refer to its elements using array indices.
But if you have a fixed-length array, or between appends, you can use both pointers and slices to point to parts of it, and it will work fine,
This all works the same as C if you think of a slice as a glorified pointer. If you’re thinking of a slice as a JavaScript array then you’ll have trouble.
The problem of Go is that it has you uses slices as that as well as actual slices, there is no vector type. So the confusion is very much understandable and to be expected.
I think it works in the same way as a pointer to std::span in C++. (Or pointer to std::string_view with the exception that std::string_view doesn't allow modification of the elements.)
I guess the difference is that std::span doesn't let you append to the backing array through the std::span directly. So with C++ you have to write more code which makes it clearer what's happening.
Go is unusual in not having a equivalent of vector.
With both slice and std::span, = does a shallow copy of just 2 or 3 pointers.
With std::vector = does a deep copy of every element, you have 2 distinct backing arrays.
Go forces you to use copy() and append() and rely on the garbage collector to make a slice fill the roll of std::vector. IMO it leads to some confusing code.
So if you got a pointer to a vector element, it now points to garbage.
Ah, I would beg to differ!
You should never be taking pointers to a dynamically resizable array, in any language. (Well, caveat, its fine if you do it only for a time period where you know the array won't be growing.) The whole point of a dynamically resizable array is that its addresses can change!
If you did this in C++, you'd get undefined behavior. In Go you get "safe" but probably-not-what-you-wanted behavior. In Rust it simply wouldn't be possible (w/o unsafe), and you'd have to use indices (which is the correct thing to do, in any language).
I kinda know enough now to avoid this, but I have to be careful and remind myself it's a possibility.
I'd love some built-in method to be able to tell whether altering a value in slice A will also alter the value in slice B (i.e. whether A and B are referring to the same backing array). As far as I'm aware there's no easy way of doing this in Go.
> To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea
Write Go in Go, don’t write C in Go. (Which applies to every language, tbh.)
The GC only operates on “dead” allocations (afaik it remains non-moving) so it’s not a concern for now.
Go’s map is (AFAIK) a pointer to the hmap stucture where everything happens, so it does behave like a reference type (all updates in the callee will be visible to the caller) but even then it can be useful to have a pointer to one so you can reset it without mutating it in-place. While that can be a bit weirder, it is also much safer. Especially given Go’s maps are not thread-safe (and in fact not memory-safe under concurrent updates).
Pointers to "fat pointer" types are sometimes needed, just like you sometimes need pointers-to-pointers.
One pitfall is when getting a slice by value in a function. You cannot be sure that someone is not going to pass you a slice into a buffer that they themselves use, so you have to be careful when appending - someone might be using that buffer and you’ll be writing over it.
Obviously not given Go was never simple in the first place. Go was built to be easy — for a certain value of easy.
Simple tools are often not easy, and simple programming languages are definitely not easy: they tend to be built out of a small set of very powerful concepts which are directly exposed to the language user, said language user has close to the power of the language designer in building abstractions. Lisps, and Smalltalks and Forths are simple, which means they are mind-bending and not only can you build what you want out of them (hello turing equivalence) you can build how you want.
And of course the simplest of languages (the turing tarpits) are barely usable at all.
A simple language, e.g. C or lisp, simple in that their grammar is simple, are definitely less easy for the programmer, than say Go. But C is not simple as an experience, since it forces the dev to mentally complect so many concepts in order to get things done: macros, memory management, etc. Lisp is complex in a different way: metaprogamming, DSLs, and deep abstractions are the norm. So simplicity/complexity tends to be something of a whack-a-mole. It's a lower bound, much like the uncertainty principle; you can always add complexity.
Go makes a lot of choices that try to really optimize the user_complexity * language_complexity product.
- If you don't know where a slice came from (such as getting it passed as an argument), treat it as read-only. Any exceptions should be documented.
- If you're going to mutate a slice, make it a private member of a struct and don't give out direct access to it.
I read this immediately as "create a new copy of the original slice with one additional element", so I presumed that was the case. It would actually be shocking the opposite, if I could end up modifying the original one (before the append) with a pointer to the new s, which seems to be the case!
Big gotcha there: treat slices as stateful at all time.
Since it has an assignment operation, it must be creating something new, otherwise it would have been a method of the slice itself
EDIT: I just realized the gotcha is not there at all, Go would consider the first slice to be of N length and the second slice of length N+1. Comparing the two slices would give an error at some point because one is shorter than the other, so the fact that the address changes or not is irrelevant. However I can see this becoming problematic with pointers, which proves the point of the article.
Edit: we would both learn something if you offered a counter example instead of downvoted.
In C++ this is UB, which is bad, but in keeping with the rest of the language.
In Rust, the compiler will not allow you to do any operation that would re-allocate the backing store whilst there are outstanding references into it.
In most other languages (eg. Java, C#, python, etc.), you can't get a pointer/reference to an array index, only a pointer/reference to the item at that index at the time you looked.
Go's decision here is especially weird given that this same thing is seemingly prevented for maps (why the inconsistency?).
Given the three goals of memory-safety, "simplicity" and performance, it's true there are not many other options Go could have chosen, but personally I think Go's interpretation of "simplicity" is incredibly warped: it's a kind of superficial simplicity that leads to programs that are much more complicated to reason about.
C and C++ are "It is very easy to make this mistake which leads to undefined behaviour and a maybe incorrect program."
Go is "It is very easy to make this mistake which leads to a maybe incorrect program."
Yes, better! But the problem is still there. I much prefer the Rust solution where there is no common mistake.
"I have this code, you see... and it takes a couple mutable references into an array... how do I translate this into Rust?"
There is no one-size-fits-all answer to that question. The code may be correct in C or C++, but the Rust type system may give you one hell of a hard time proving that it is correct to the Rust type system's satisfaction... so you refactor your code completely, or you use integer indexes into arrays rather than references, or you use unsafe code...
I've written some amount of Rust code at this point. About half of the time, when I write a project in Rust, there comes a point at which I'm fighting with the type system. I feel like this should stop happening, at some point.
They are just so obscure, I forgot about them and no one uses them. No users, no problems
They are not part of the normal type system. You cannot declare a variable of a type slice. Nor a field. But when a parameter of a function is an (open) array, you can call the function with a slice of an existing array
That avoids most problems
The backing array exists when the function is called, and the function cannot store the slice, so the slice cannot outlive the array. It is like the function borrows the array. Only problem is if the function gets another reference to the array, through a global variable or something, and resizes it
And if you want to ensure that multiple things have access to the elements, then you have to avoid reallocations or you get silent data loss.
No matter what you’re doing, a pointer to an element of an array or slice is usually the wrong thing in Go. The language would be better off without them.
Or that you’re way off in UB (UAF) land.
Ah yes, the usual excuse for it being fine that C APIs are completely broken and half of them can not be used correctly.
> Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.
Except when they don't because `append` itself amortises allocations, which means if you treat slices as immutable and pass by value you will end up with slices sharing a backing array with leftover capacity and stomping on one another's data.
> Mutating them in any way is actually a special case that you do only for performance reason
Mutating them is literally what the normal Go API usage has you do. If you want to avoid mutating slices you need to write this abortion:
s1 := append(append([]int(nil), s0...), item)
s0 := append([]string{}, "zoo")
sa := append(s0, "foo")
sb := append(s0, "bar") // overwrites sa[1]
I think the "static typing trend" is a product of Rust and Go showing people that static typing doesn't have to be cumbersome like it was in 90s-00s Java, C++, and C#. Indeed, I suspect that the quality of life improvements that Java, C#, and C++ made also improved the stock price of static typing (and building on that foundation, things like TypeScript are exposing JavaScript developers to the utility of types). Which is to say, static typing isn't an empty trend or fad (no idea if that's your intended meaning) but rather people were previously averse to static typing because the mainstream statically typed languages weren't ergonomic and people assumed that the bad ergonomics was caused by static typing--now we have many mainstream languages that show that this isn't the case.
However, I called static typing a "trend", and I'll try to explain why. I think attempts to type Python, JS, Ruby and the popularity of Go and Rust are the natural consequences of people departing from the Java/C++ ecosystem 10-20 years earlier (for good reasons). Now they are rediscovering the good parts of this ecosystem (ease of deployment with binaries/fat jars, static typing, performance). Since Twitter, Github, Youtube, Shopify, Instagram, etc have all that code around, they are going to either improve it, or try to migrate from it. For example, Shopify is working on a compiler to native for Ruby based on LLVM https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp.... Instagram is working on a performance-oriented CPython fork https://github.com/facebookincubator/cinder. Twitter, from what I understand, went back to Java, going through Scala first (which is another example of "better type system"). KhanAcademy is migrating services from a Django monolith to Go services https://blog.khanacademy.org/half-a-million-lines-of-go/. Whatsapp even had a project to do a statically typed "Erlang 2".
The "trend" here is that some companies that use "new" dynamic languages in the 00s are now very large companies that have enough money to invest in language, tooling and things like that.
But Go doesn't have that distinction, and has actual pointers you can use directly. A map is just a heap-allocated structure sitting behind a pointer.
If you create a type which is a pointer to a struct, sure you can say you've built a reference type if you want, but that doesn't actually say much to anyone, because that's not a distinction the language makes, unlike Java or C#.
> A map is just a heap-allocated structure sitting behind a pointer.
And the difference is?.. Because maps in Go behave exactly as if they were un-referenceable pointers to the hidden, heap-allocated hashtables.
A variable in Go always maintains its address after it is allocated. Assignments to that variable copy the assigned value to the original address.
Somewhat unhelpfully, this rule is even true for iteration variable - when you write 'for i,v := range arr {...}', i and v get allocated a memory address, and they get successively assigned the indices and values in arr. This implies that each element in arr is copied into the value of v, and that doing &v inside the loop gives you a completely different pointer than &arr[i]. In fact, &v will always point to the last element of arr after the loop is over.
https://play.golang.org/p/iSjoqGTg20_O
var s []int
s = append(s, 10, 20, 30)
pe := &s[0]
ps := &s
s = append(s, 50)
s[0] = 100
pe2 := &s[0]
fmt.Println("s: ", s, ", ps: ", ps, ", pe: ", pe, ", pe2: ", pe2)
// s: [100 20 30 50] , ps: &[100 20 30 50] , pe: 0xc0000be000 , pe2: 0xc0000b8030Is that still true in an app that has considerable memory pressure and has GC running now and then?
In other words, ps is a pointer to a pointer to array data. Append may change the inner pointer's value but that's about it.
In this example the backing array didn't get extended in-place. The first backing array starts at 0xc0000be000. After the append() call, there's a new backing array, which starts at 0xc0000b8030.
Specifically, what happens in
s = append(s, "something")
In Go, allocation is very important and it is always explicit, except this explicitness is sometimes masked by the fact that the := operator is not a simple "allocate everything on the left using the types of the thing on the right" operator, but "allocate at least one thing on the left using the types on the right (it is a compile-time error if there isn't at least one new value) but use normal equality for everything already allocated", which is very convenient to use, but can make developing the proper mental model for how Go works trickier. Arguably, there's some sort of design mistake in Go here, though the correct solution doesn't immediately leap to mind. (It's easy to come up with more abstractly correct options but they have poor usability compared to the current operator.)
Similarly, it can be easy to miss that Go, like C, cares deeply about the size of structures, and every = statement has, at compile time, full awareness by the compiler of exactly how much memory is going to be involved in that equality statement. Interfaces may make it seem like maybe I can have an "io.Reader" value, and first I set it to some struct that implements it with a small amount of RAM, then maybe later I can set it to a struct that uses a large amount of RAM, but the interface value itself is actually a two-word structure with two pointers in it that is all you are ever changing, and, again, those two words are given a specific location in RAM (possibly virtually, if you never use it they could conceivably never been out of a register, but the Go compiler and runtime will transparently make it live in RAM if you ever need the address for any reason) and any setting of the value of the variable that has an interface value will set only those two values, with no other RAM changing as a result. You can use the same io.Reader variable through its interface implementation as a "handle" on a wide variety of differently-sized values under the hood, even in the same function (I do this all the time when progressively "decorating" an interface value within a function), but the in-memory size of the handle itself never changes no matter what value you ask it to handle.
This is not intended as criticism, praise, defense, attack, or anything else on Go itself; it is descriptive of what it is.
The confusion stems from the fact that a slice object contains a pointer in itself, pointing to a backing array, thus adding an additional layer of indirection. Append will return a new slice that may point to a different backing array. This doesn't matter, because you put the new slice in the memory location pointed at by ps, 1000.
This is not so different from pointer pointers. If you have `int i = 0; int pi = &i; int *ppi = π` you can change `pi` (analogous to the slice) to your heart's content and `ppi` will reflect the changes.
Of course that's convenient, the issue of Go's slices is that they act as both a dynamic array and a slice viewing a portion of one. The two uses conflict with one another, and the interactions are full of traps.
package main
import "fmt"
func main() {
a := []int{0}
for i := 0; i < 40; i++ {
b := append(a, 0)
a[i] = 1
fmt.Printf("%d ", b[i])
a = b
}
}
I don't think there are many reasons, if any at all, to keep a pointer to an array element of a slice around in Go. Usually I only get the address of an array element only when passing it to some C code or doing some low level manipulation, but then I don't keep the pointer around. In Go code you usually just keep the slice object - which contains the necessary pointer anyway.
Since slice API is pass-by-value, in theory ANY method will invalidate the pointer. In practice only resizing methods actually NEED to reallocate the underlying array, but magic can happen. However, refcounting will make sure that a previously underlying array having pointers to it will remain allocated. This means that 1. pointers to single elements will always dereference 2. slice structure modification can leave pointers pointing to stale data
Mostly off-topic but FWIW "mutable reference into an array" is typically very easy in Rust you just accept a &mut [T]. This also nicely enforces that you don't try to append because that would be wrong 99% of the time.
I had specifically worded it as "a couple mutable references into an array". How do I take a small number of references into an array, say two or three?
Yeah you gotta write it out per number of times. Luckily this rarely comes up in this specific form.
No, what most readers intuit from that is that `append` performs no mutation and that this is fine:
s2 := append(s1, item)
(def s2 (conj s1 item))
Except it's absolutely not fine.
That's why other language separate slices and vectors and avoid confusing two objects which have different behaviours and uses even if their representation is very similar.
s2 := append(s1, item)
> other language separate slices and vectors
Go has a separate type for arrays, which is a value type
var arr [10]intNo, it is not.
> It's only not fine if you some how assume that s2 and s1 will always refer to the same data.
It's not fine if you assume anything about the interaction of s1 and s2. It's not fine if you assume they do alias, it's not fine if you assume they don't.
There is, fundamentally, no situation in which that construct is anything other than a footgun. `append` should only ever be used with the same slice on the LHS and RHS[0], or a brand new slice object constructed for the occasion on the RHS.
> Go has a separate type for arrays, which is a value type
An array is neither a vector nor a slice.
The issue is that Go's slices serve as both a vector and an actual slice, and the union of these interfaces creates footguns which don't exist in either.
[0] IFF that RHS is a either a non-parameter local, or a pointer to a slice
Example: https://hackage.haskell.org/package/bytestring-0.11.1.0/docs...
C++:
std::vector<int> v {1, 2, 3};
void foo(std::vector<int> *ref) {
ref.push_back(4);
}
foo(&v);
//v[3] == 4 is true here
v := []int{1, 2, 3}
func foo(ref *[]int) {
append(ref, 4)
}
foo(&v)
//v[3] == 4 may or may not be true here.
void foo(std::vector<int> *ref) {
ref->push_back(4);
}
foo(&v);
void foo(std::vector<int> &ref) {
ref.push_back(4);
}
foo(v);append(*ref, 4) // error: append(*ref, 4) evaluated but not used
this is the correct version and it also removes the incertitude:
*ref = append(*ref, 4)
A more interesting example showing that resizing can be observed (getting rid of the pointer to the slice, since it's not useful anyway):
This happens regardless of resizing, since append() at best modifies the array that *ref/v points to, but it does not modify *ref/v itself; and slices in Go have a pointer to an underlying storage AND a start and end index into that storage (multiple slices can point to different parts of the same storage).
Created here an example that shows how this interacts with resizing:
Note that C++ std::vector has an operator overload so this is actually just calling the method named operator[] on the object v and that method returns you a reference to the object in the backing array if in fact it is a suitable size (no checks are made). In particular if foo doesn't for any reason extend the vector then our program now has Undefined Behaviour.
This is not merely the C syntactic sugar array subscript operation v[3] == *(v+3) as the vector is not necessarily just a backing array pointer and some magic.
In contrast Go is really offering array syntax for this slice, that's the built-in array subscript operation and it cares whether v[3] exists when you try to compare it to 4.
Sure, std::vector::operator[]() is not just syntax sugar for *(v+i), but I don't think any of this is relevant for the discussion at hand, unless I'm missing something.
> Luckily this rarely comes up in this specific form.
Well, yeah—it would come up rarely because it only solves the problem under very specific circumstances!
It is so rare that I don’t think I’ve ever seen it in any rust codebase I’ve ever looked at.