Show HN: Benchi – A benchmarking tool written in Go

Show HN: Benchi – A benchmarking tool written in Go(github.com)

58 points by lmazgon 1 year ago | 7 comments

Benchi is a CLI tool for running benchmarks and collecting metrics. It's using Docker Compose to orchestrate the infrastructure and tools being benchmarked, making it repeatable and runnable on different machines. It allows you to run the same benchmark for different tools and compare the collected results.

The repository contains a simple example. For a more elaborate example see how we use Benchi to compare data pipelines running on Conduit and Kafka Connect, two data streaming tools (still work in progress): https://github.com/ConduitIO/streaming-benchmarks

mixologic 1 year ago |

How do you control for changes in the host os environment that would affect performance of the system being observed?

spockz 1 year ago | |

We solve that partially by running the base state and the delta version through concurrently. This way most types of impact will impact both at the same time. This gives us the relative delta between versions.

Other than that, just like always, run benchmark on a stable dedicated set of hardware.

edoceo 1 year ago | | |

And run a bunch of iteration, and mash the results together maybe?

meling 1 year ago |

What’s the overhead of running in docker vs bare metal?

lmazgon 1 year ago | |

I can't say. It's definitely not the same as running on bare metal. Then again, any benchmark in a test environment is just an approximation of the real thing. To see how anything will act in the real environment, you have to run it there.

In our use case we wanted a way to quickly and easily set up benchmarks that would allow us to compare our software to our competitors under the same conditions. Given that Benchi can run the same benchmark scenario for different tools, the results are comparable with each other. We also run all benchmarks on an empty AWS EC2 instance, to minimize any other factors. But does that mean that the collected results show the absolute limit of what the tools can handle? Probably not. Under different conditions, results can change, but that's just the nature of benchmarks.

tracker1 1 year ago | |

Depends on what you are testing and how... Virtual network adapters, routing, file system redirection, etc. all have some overhead and will impact things differently. IME U haven't seen too many things affect performance more than 3-5% on native Linux. If you're using Windows or Mac as a host OS, then volume mounts to the host in particular are really going to affect the results.