Tracing Goroutines in Realtime with eBPF

Tracing Goroutines in Realtime with eBPF(sazak.io)

80 points by darccio 45 days ago | 9 comments

Interesting to see an ebpf approach to this idea, some time ago I did the same with Dtrace: https://gaultier.github.io/blog/detecting_goroutine_leaks_wi...

However I did not think of observing the ‘castgstatus’ runtime function.

Always nice to be able to compare the DTrace and Ebpf approaches, they both have different strong points.

trueno 42 days ago |

that is actually sick.

how common is it for go devs to experience leaking goroutines ? id like to think go is a lot less shoot yourself in the foot here since they provide a framework for concurrency/parallelism rather than you working with the tiny pieces of it and building out the architecture yourself, but ive only needed to use goroutines once and it was a pretty problem-free experience.

broken_broken_ 42 days ago | |

It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.

usrnm 42 days ago | |

> how common is it for go devs to experience leaking goroutines

About as often as leaking memory in C++

trueno 42 days ago | | |

F. again i have minimal experience actually ever needing it in go, but guessing this is just generally the exercise of managing the lifecycle of a goroutine well in your code? proper handling so things dont get orphaned in buffer, fire and forget woopsies, etc.

early on i do feel like go kinda advertised batteries included concurrency but i kinda wished they advertised the foot-shooting-mechanisms and gaps in the abstraction a little more. overall i prefer to have enough control to choose how to manage the lifecycle. mem leaks bum me out and kill my steam, at least from my experience with c/cpp.

tsuzu 42 days ago |

So cool! Does it support multiple Go versions?

flakes 42 days ago | |

I would say that this is limited to a targeted Go version and architecture. For example, the filter checks for the goroutine pointer on `r28` which is correct for arm64 but not universally true.

Any changes to the struct layouts, stack, or heap layouts would also cause failures in these lookups. E.g., in Go 1.17, many functions now use direct register mappings for arguments rather than always placing arguments in the stack.

Would need to thoroughly vet compatability with each new Go version before using something like this in production.

xtoilette 42 days ago |

Cool! is there something similar for async rust ?

fch42 42 days ago | |

https://docs.rs/tokio-console/latest/tokio_console/ comes to mind.

I'm not aware of eBPF-based dynamic instrumentation for either tokio or other async runtimes, but I'm also not (in need to be) debugging this on a daily basis.