Ask HN: Anyone using Bazel at startups? or migrated to/away from Bazel? Anyone using bazel at startups? Or is it thought off as too heavyweight? Context: I'm ex-Google and I think blaze/bazel brings enough to the table to be worth the complexity even for startups, but wondering I'm underestimating the actual complexity. |
Bazel gets in the way of this use case. It will help your small team not break each other's builds. But you'll burn a salary on someone as the "Bazel person," they're going to write a bash script or make file for everyone else to use anyway, and when you inevitably rewrite everything from scratch the third time all their work is going to be for nothing and they'll spend days getting it back.
Just write a bash script or python script that does what you need for today, and plan to deal with it later. The only thing startups can afford is tech debt.
Another way of thinking about this is that your product is the software and not the software to build the software. Don't waste time thinking about the tools.
- It copies your files into a temporary directory to run them and that directory is named an obnoxiously long hash digest, so it effectively takes a giant shit in all your stack traces. When you're looking at an error message, half of your field of vision is taken up by line noise of the same random hash over and over again and since it runs your Python the first few frames are just helpfully telling you "this code was run through bazel". I even wrote a script that scanned my clipboard for Python stack traces, parsed them and removed this noise to make it easier on my brain to search for the actual line with the error.
- Every import you write you have to write in two files or three if you have to export it. It's double the typing work and finding the target is a pain because there's no jump to definition of the Bazel target so it's even more typing to find it.
- I wanted to use pytest and it was much more work, I don't think I ever got it working. Normally there's no setup you just run "pytest" and it discovers all the tests and runs them.
- Your code doesn't start running as quickly.
- You have to type way more characters to start your code. I had aliases for every endpoint.
- There was something with tests where if the error log was too long it would just say "there was an error, please find it in this randomly-named file" and when you're debugging a test whose error log is too long it's going to be too long every time you re-run it too. So running and seeing the error is now a two step operation, double the work.
Maybe some things have changed or I misremembered something, but overall it was the part of our tech stack I really hated. I just wrote software, I didn't do any ops or any sort of deployment or touched live systems at all really, so I honestly have no clue what it actually did for us it was just pure downside for me. I would never use it.
Others are good points too. Thanks!
1. Bazel requires non obvious, continued maintenance. Bazel is a theoretically hermetic build system with universal caching and near constant build times. The sticking point is that it only has those properties if someone sets it up correctly and maintains its infrastructure. At our company, no one set this up, and as a result build times ballooned and local development became a mess of bash and docker. And even if someone had set this up, who would maintain it? We would have had to expand the infra team, and even a couple of people extra can be a lot for a startup.
2. Bazel also has support for every language, which is theoretically true. In practice, this support is often incomplete, out of date, poorly documented, badly supported, or some unfortunate combination, because the (language)+bazel communities that support the docs, tooling, etc are often orders of magnitude smaller than teams that work with (language). So teams that use bazel are forced to choose between two bad options: integrate things the bazel way and rely on the relatively small (language)+bazel open source communities and tools of questionable quality, or use the much much larger (language) open source communities and hack the tooling in. We did the former, and it was hell, because we a) didn't have feature complete community tooling and b) upgrading the bazel version required every team at our company that was using bazel to stop working and simultaneously check trial bazel versions for breakages.
3. Bazel makes hiring and onboarding harder. In order to fix the above problems, you need people that know bazel. Hiring someone that knows, say, python is hard enough. But python and bazel? The hiring pipeline immediately gets reduced, which means you either have to spend more time searching for qualified candidates or more time onboarding candidates who don't quite fit all your criteria.
All in all, I would recommend against bazel unless you: 1. Have or can hire a preexisting community of knowledgeable bazel developers who will make up a sizeable portion of the company. 2. Have full executive support for multiple years while the infrastructure is being created, and executive support for maintenance teams thereafter 3. Actual, company-wide problems with builds that justify the former two criteria.
For nix may be applies 3. But nix is more diverse and flex, so 1 ans 2 not really.
For 3,chances somebody knows nix because he setup his darwin, os or home with it.
I tried Bazel when it was first released as open-source but migrated away from it pretty quickly. It was just too complex for what it brought to the table. In a startup you should only be doing things that directly validate your business hypothesis, and futzing around with your build system is not really one of them.
I found it much more productive to use the native package managers and build systems that go with your language, eg. npm for Node, pip for Python, Gradle for Java or Kotlin.
Follow up: I was wondering if bazel's hardcore declarativeness + opinionatedness-about-reproducibility might help increase confidence in some situations, e.g. an organization making binary distributions of their software to external partners.
We use Bazel at work and it’s sufficiently complex to write new rules. I’ve seen NX touted as a more approachable equivalent for smaller teams, maybe check it out?
Bazel only works if you have a dedicated bazel team to handle it. In general it's shit. Avoid, I hate bazel.
All those things that were so great and wonderful at bigcorp... pretty often turns out they fixed issues that are a decade old and the state of the world has often made them irrelevant. Certainly at startup scale.
Briefly looked into Bazel, but settled on pantsbuild since Google seems to have just thrown all the language support over the wall to the community.
Being able to trivially setup CI on Github has been very helpful and important to me.
Being able to easily add targets and package things that require multiple microservices has been nice.
People say you can use the native tooling, but what even is the native tooling for python? It doesn't have a standardized packaging tool unless you count docker. pants' support for pex was important for me until we eventually adopted containers.
Folks can (and did) get the native packaging wrong, particularly when they were not super familiar with the language they needed to pick up on the fly.
Having to deal with an abstraction layer over the tooling has introduced some of its own headaches (eg for docker), but mostly for advanced features.
In the end, I'm pretty happy, I see it as an investment in CI, and the CI has caught a bunch of real bugs people pushed.
I think starting early here has been good, since the lift to get onto it has been small.
Not to say that it's bad, just that I don't buy that it's meaningfully standard. Though it seems like poetry sees zipapp generation as out of scope, but at least pdm has a plugin.
I briefly tried a bunch of toolchains including poetry to see if anything could handle the horrific pytorch packaging situation, but they basically all had problems with it, so I didn't press it and just left the training code wrapped in docker duct tape.
I am generally quite happy with pants after dealing with all this mess.
I do not have notes handy so I don’t remember exactly what happened but my impressions after a week or two invested trying to get it running were a) that our projects had too many unsupported things (dependencies, flows) that would require a huge effort to adapt; and b) that it would take way too much effort for our small team to convert the 500+ repos we were building to release our software.
I have always dreamed of a time in which I’m able to fin a tool that fills this gap. Man, not having to build the same code that hasn’t changed over and over locally and in pipelines sounds so appealing.
Can't you use something like Maven build cache ( https://maven.apache.org/extensions/maven-build-cache-extens... ) for that?
https://earthly.dev/blog/bazel-build/
My thoughts, as someone who has done a lot of build engineering, but that has only a small amount of experience with bazel:
If you are working with languages that are well supported by existing, maintained bazel rules, maybe. Otherwise, the answer is likely "no". You do not want to take on the burden of maintaining significant rules, or figuring out how to mate your ecosystem of choice to an opinionated build system that nobody else in your ecosystem uses.
If you can manage with something easier to setup, it probably makes sense to do that instead. My general experience in startups is that you have to "pick your battles" constantly. You don't have the luxury of infinite time and resources. Work on the build system has a high opportunity cost.
Most developers and other people who aren't build nerds care about the answer to two questions in regard to builds:
1. Is the build fast enough?
2. Is the build working?
That's it. This is frustrating when they don't understand the other important aspects of build system quality, but it is also, admittedly, somewhat of a pragmatic perspective.
There is much leeway inherent in the speed question. If the build is slow, people can wait. Do they have any other choice? There is plenty of other work to do while they wait.
If the build doesn't work, you, THE build guy, are going to feel the pressure to get it working again quickly. This often turns to bodging, and eventually to something resembling life support. However, nobody will care how much of a brittle, knotted mess the build code is if it still manages to mostly work.
To put it bluntly: A startup company can probably survive with a shit build system. And that might even be the right answer depending on the context.
I would personally be perfectly happy to get paid to set up a beautiful, apex example of a build system for a startup, whether it makes business sense or not. My professional integrity just won't allow me to recommend that path unless I think it's justified.
I might recommend to start with a script, or a makefile, or a ninja generator, and see how that works out first. After all, that is how Google did it for a long time before they decided to create Blaze.
A lot of the isolation and visibility control in bazel is totally useless outside of big companies.
3-6 mo of one engineer to migrate to something so hardcore as Bazel seems very good actually, especially since it seems they Did It Right. The flows in 200-engineer size org can be quite complicated.
On the good side:
- bazel forces you to have an acyclic architecture, still, this can feel annoying because every module needs to specify its dependencies.
- bazel does concurrent builds pretty well, optimizing the build times.
On the bad side:
- It took a considerable time to set everything up, we thought to be done but no, there was a weird stuff coming every week and most of the team ended up waiting for the bazel guy to fix it.
- IDE support was poor, in theory, we could just use intellij but most of the times it required help from the bazel-guy, in my case, I got frustrated about the needed maintenance and just accepted that an IDE wouldn't work. - Given the lack of IDE, it means that we couldn't easily execute a specific test from the UI, having to keep notes for the cli.
- In theory, you could use a shared cloud cache so that the builds are very effective but we could never set this up. - talking to engineers from other companies, they admitted that the bazel setup was hard and took a considerable amount of resources, they ended up writing many internal tools to get a effective usage.
Given that you plan to create a startup, do you plan to hire engineers? then, be prepared to the bazel-guy.
Also, as a startup, I bet your code will be small enough that the build times won't be very different with normal build tools.
I'd personally not use bazel unless my company offers a dedicated engineer just to maintain this integration, I think its overkill for small companies.
If it’s a bunch of Python with a C++ extension (or mostly JVM or whatever), heavy build stuff is probably overkill.
If you’re in a polyglot environment with mountains of vendored native code targeting multiple cross-compilation targets and need the flags tuned just so?
Then Pants, Bazel, and buck2 all have pros and cons.
When it works you almost forget about all the pain... But then once I took a peek at some of the custom build rules that I was blindly copy pasting and was blown away how complicated it all was. Thousands of lines of build code...
Can't imagine this layer of overengineered complexity can boost a startup, but can easily picture how it slow it down.
Your thinking is obviously distorted by some sort of google mentality.
Put your foot back to the real world please.
The world where you have competition, world where people need to get job done quickly.
Not in most technically sofisticated or mathematically cool way to brag about, but quickly and with as little overhead as possible.