Go Execution Modes

111 points by Rooki 11 years ago | 51 comments

personZ 11 years ago |

The dogma about preventing the sharing of pointers with C is a bit concerning. I've made use of that mechanism, to fantastic effect, albeit understanding the risks and consequences. The notion that it must be prevented because bad things can happen if you aren't careful isn't a starter, as there are essentially zero people doing that who've had problems. Instead a lot of people have enjoyed fantastic productivity because of the similarities of the implementations, and the synergy that allows.

But the team wants move to a compacting GC. Fine, add pin/unpin idioms. This ground has been well covered by other languages. Don't completely destroy a very productive mechanism because in some oddball cases it might not work.

ominous_prime 11 years ago | |

The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating. You may have used this successfully in the past, I know I have, and it works with the GC toolchain, but it is still technically incorrect to do so.

I think there's another design doc being worked on that will document the situation, and possibly have proposals to make sharing memory between Go and C more convenient. There's are many projects that make heavy use of this, where copying is slow or infeasible, and malloc is inconvenient.

personZ 11 years ago | | |

The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating.

Where is this stated? Where is it said that it's technically incorrect?

Just to be clear, I'm not passing it and then running a thread in C that just runs with the pointer to an object that no longer exists in Go. Instead I'm calling a C function synchronously -- there is no possible way the element is going to be collected during that run. It could be compacted (though Go up until now has explicitly had a non-compacting GC, so not a current risk), of course, which is what brought up this issue, but that's the reason such interop languages have pinning (flagging that something shouldn't be moved).

It wasn't by chance -- it was by design, and it was purposefully understanding the lifetime of objects.

crawshaw 11 years ago | |

You can do your own pinning, which is what I do in gobind[1]. The basic idea is to keep a map in Go of the pointers:

    var ptrs = make(map[uintptr]unsafe.Pointer)

When passing a pointer from Go to C:

    var p unsafe.Pointer = ...
    id := uintptr(p)
    ptrs[id] = p
    C.Fn(id)

When C returns the pointer to Go to be used, run it through the ptrs map again

    //export GoFn
    func GoFn(id uintptr) {
        p := ptrs[id]
        // ... use p
    }

Don't forget a cleanup function where you delete(ptrs, id). If you want to pass the same pointer multiple times to C and keep it comparable, you'll need a second map.

All of this is requires being very careful, but the hard part is the notion of holding references to memory outside the realm of the GC. I don't believe a runtime-assisted pinning API can do any better than what you can do yourself with a map.

[1] https://godoc.org/code.google.com/p/go.mobile/cmd/gobind

niemeyer 11 years ago | | |

That's a fine model when you want to hand a reference to a Go value, just to be used in Go itself, but it's not actual pinning. The data can still move, which means you cannot use the data in C, for example to share a buffer without copying into another temporary buffer managed by the C allocator.

ominous_prime 11 years ago | | |

I don't think this would really handle the pinning aspect if a compacting GC is implemented, would it? It does keep the pointer alive, but it technically doesn't do anything to prevent the GC from changing the pointer.

pjmlp 11 years ago | | |

This does not work if the GC updates the pointer concurrently.

pjmlp 11 years ago | |

I agree with the decision to forbid pointer sharing with C code.

This is the type of issues that nicely lead to security bugs, in the hands of the typical average enterprise developer.

Having a pin/unpin idiom might be good enough, though.

personZ 11 years ago | | |

This is the type of issues that nicely lead to security bugs

It's also the type of functionality that enables the efficient use of high performance C libraries and existing code (I use it to great effect with the Intel compiler for extremely high performance SIMD code).

The "someone might not use it right, therefore it's anathema" notion is misguided, and is orthogonal with the philosophy of Go. Even C# and Java -- the pinnacle of "enterprise" languages -- understand and respect the notion of pointer sharing, and early on built in mechanisms to support it. Go is a league above, though, given that it follows C struct layout rules.

But I see where the person proposing banning it got their notion from -- it's a classic hubris of "if it isn't important to me, it isn't important to anyone". A compacting GC has benefits, but the lazy notion of wholesale blocking broad and powerful uses because one doesn't personally benefit from them is how languages die.

pauldix 11 years ago |

This would be awesome to have. +1!

With InfluxDB we want to give users the ability to define custom functions and it looks like this could be used to let people do that in Go or any language really. Whereas right now we're limited to doing something like using LuaJIT.

axaxs 11 years ago |

I would love to see the gc compiler support building dynamically linked executables against a libgo runtime Ala gccgo. Unless I'm misreading, this case doesn't seem covered. I love the standalone binary deployment ease as of now, but on a machine with tens of go applications, the binary size redundancy gets old. Or did I miss something?

FooBarWidget 11 years ago | |

We're in an age where everybody is wasting gigabytes of memory through virtualization, and nobody cares. Would those few megabytes of memory saved by dynamically linking against libgo really be worth it? For past 5 years or so, it seems people value ease of deployment much more than efficiency.

ColinCera 11 years ago | | |

You have a point, but on mobile and embedded devices, nobody is "wasting gigabytes of memory through virtualization" and not caring.

davidw 11 years ago | | |

The people who value their memory usage just aren't using that kind of system, so there's some selection bias there.

mrweasel 11 years ago | |

How large are the binaries you deploy? Unless it's hundreds of MB per binary I does see the dynamically linked executables being worth the trouble.

Of cause options are nice, I just worry that in the end I don't get to choose.

ColinCera 11 years ago | | |

I think one of the motivations is that they want Go to be a language suitable for Android development and other use cases. In mobile and embedded devices, binary size still matters.

ianlancetaylor 11 years ago | |

I believe that use case is covered via -buildmode=shared and -linkshared. Just run -buildmode=shared when building the standard library.

Rapzid 11 years ago |

I think the more relevant news in there for most is the support for dynamic/runtime loading. The title here makes it sound like it's just for dynamic linking but I read the article to article to be sure and was pleasantly surprised :)

Rooki 11 years ago | |

Good point, I updated the title to include the actual title of the doc but left the rest to avoid confusion.

On topic: I'm really excited to see this, even though it's made clear this is just a plan and not a promise it's still great news for things like Android development.

lobster_johnson 11 years ago |

> Building Go packages as a shared library

Yes, please! We have a few potential Go projects which are blocked by the inability to separate Go code into plugins.

I have toyed with the idea of forking child processes and implementing plugins via pipe I/O, but that's a very unfriendly API interface, and I don't feel very inspired. (Go isn't that great at POSIX stuff like forking, either.)

_stephan 11 years ago | |

Why do the plugins have to be dynamically loaded? Would a source-based plugin/package system be an alternative?

lobster_johnson 11 years ago | | |

Consider a data processing app. It calls little data-processing modules written in Go. You want to open-source it because it's general-purpose and reusable; but you don't want to open-source all your in-house, proprietary data-processing components, too; they're not of use to anyone but yourself, or they may be something you can't open-source for other reasons.

That means the core project cannot list its plugins as dependencies. You will instead want to be able to load plugins from a configuration file.

(The problem is actually equally awkward in languages that have package systems, such as Node and Ruby. In Ruby, for example, a program can only see gems declared in the Gemfile; same problem with npm-shrinkwrap.json.)

I suppose you could set up a special build system where plugins are expected to be installed into a certain folder so that they're statically built in, with some kind of hook so that they're registered into the core. That's the Nginx approach. I'm not a big fan, especially since this method of building things need to be reinvented for each app that needs a plugin system.

You could, of course, invert the relationship and make this system a library: To set it up with your plugins, you write a small program that runs the system's main entrypoint with a list of plugins. But that just makes it harder to configure, build, use and deploy.

bkeroack 11 years ago |

I realize it's just an early draft, but it's interesting (alarming?) that the proposed plugin API has no facilities for plugin discovery or separate namespaces. Nor any apparent ability for the plugin consumer to introspect into plugins.

I suppose the namespace issue can be mitigated by a strict naming convention (something like "[application_name].[plugin_name]"), but the others are kind of a bummer.

I don't have a whole lot of positive things to say about the Python setuptools API (perhaps better called "lack-of-API internal interface"), but at least it's possible to dynamically load python plugins from arbitrary locations in the filesystem and programmatically discover their contents.

ianlancetaylor 11 years ago | |

The plugin API uses file names, so discovery would be at a higher level. Go packages impose their own namespace, so there shouldn't be any namespace issues. Introspection is useful, but would be a matter of examining the files using existing packages like debug/elf; it would not be part of the plugin API.

Basically the plugin API is isomorphic to dlopen/dlclose. Everything else is built on top of that.

bkeroack 11 years ago | | |

OK, if true that makes sense. I was interpreting the Name parameter to Open() as being something like a global plugin name rather than a filesystem location.

That combined with the Plugin struct (which hopefully would have something like an iterable Map of exported symbols) resolves the issues I was concerned about. Nice.

nteon 11 years ago | |

I'm not sure I see the benefit of plugin discovery or separate namespaces. Loading a plugin gives you a Plugin struct, which is a namespace to lookup symbols (functions + globals) from that plugin.

Most plugins would I think by necessity be application specific, so I'm not sure how much gain there in in some sort of plugin discovery. But the process of loading a plugin would also allow the plugin to call into your code to register handlers & data structures in the main application through init() functions.

From a security perspective, executing arbitrary code in your address space in order to discover the contents of a plugin sounds ill-advised.

To me, it looks like they're building the low level building blocks that allow a number of different higher level APIs to be built, which is pretty neat.

easytiger 11 years ago |

Is there a mirror? For those of us in the corporate wasteland this is not accesible

JuiceSSH 11 years ago |

Interesting that it will allow compilation of PIE executables. This is a mandatory requirement for any binaries run on newer versions of Android (L+).

dmm 11 years ago | |

Is it for security reasons? I know that OpenBSD uses PIEs so that the runtime can randomized address space locations.

frik 11 years ago |

The doc doesn't mention a timeline or Go version when we may see the new planned features.

enneff 11 years ago | |

That's unknown. Work will likely begin in December after Go 1.4 is released, so the earliest we'll see this work is in Go 1.5.