How I stopped worrying and loved Makefiles(gagor.pro) |
How I stopped worrying and loved Makefiles(gagor.pro) |
I've found Earthly [0] to be the _perfect_ tool to replace Make. It's a familiar syntax (combination of Dockerfiles + Makefiles). Every target is run in an isolated Docker container, and each target can copy files from other targets. This allows Earthly to perform caching and parallelization for free, and in addition you get lots of safety with containerization. I've been using Earthly for a couple of years now and I love it.
Some things I've built with it:
* At work [1], we use it to build Docker images for E2E testing. This includes building a Go project, our mkdocs documentation, our Vue UI, and a ton of little scripts all over the place for generating documentation, release notes, dependency information (like the licenses of our deps), etc.
* I used it to create my macOS cross compiler project [2].
* A project for playing a collaborative game of Pokemon on Discord [3]
IMO Makefiles are great if you have a few small targets. If you're looking at more than >50 lines, if your project uses many languages, or you need to run targets in a Docker container, then Earthly is a great choice.
[0]: https://earthly.dev/
[1]: https://p3m.dev/
[2]: https://github.com/shepherdjerred/macos-cross-compiler
[3]: https://github.com/shepherdjerred/discord-plays-pokemon
For anything needing light-to-moderate software configuration management, cmake is readily available and simple.
If you're running a Google then you're using a build and DVCS that make use of extensive caching, synthetic filesystems, and tools that wrap other tools.
I won't say what not to use. ;)
I don't believe that.
make: ** No rule to make target `love'. Stop.
I much prefer Make to alternatives like Just or Taskfile. Besides the fact that more people know Make, Make actually has incredibly useful functionality that alternatives remove 'for simplicity', but then later you realize you want that functionality and go back to Make. Sometimes old tricks are the best tricks.
Like? None of the mainstream languages have any sort of builtin for dependency resolution.
Make certainly has some obscure variables, but of all the basic knowledge of Make you need to learn, $@ is near the top of the list (it's "target". an @ sign looks kind of like a bullseye. If you want to see it as visiting a dependency graph, it's the dependency you're currently "at").
Some system have GNU, some BSD.
> at least trivial to install
Then same for `just`, package manager have it.
I should specifically mention their docs. The docs are easily approachable with plenty of clear and concise examples. I even have a pdf copy of the doc book for quick reference.
https://marmelab.com/blog/2016/02/29/auto-documented-makefil...
help:
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
# Help idea derived from https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
# Prints the help-text from `target: ## help-text`, slightly reformatted and sorted
.PHONY: help
help: ## Write this help
awk 'BEGIN {FS = ":.*#+"}; /^[a-zA-Z_*.-]+:.*## .*$$/ {printf "%-30s %s\n", $$1, $$2}' $(MAKEFILE_LIST) | sort
sed -rn 's/^([^:]+):.*[ ]##[ ](.+)/\1:\2/p' $(MAKEFILE_LIST) | column -ts: -l2 | sort @awk '/^[a-zA-Z_-]+:[^#]*## .*$$/ {match($$0, /## (.*)$$/, arr); printf "\033[36m%-30s\033[0m %s\n", $$1, substr($$0, RSTART+3)}' $(MAKEFILE_LIST)It appears to give the benefits of autotools with the simplicity of make.
Its a code smell to me when your build system starts to become too complicated and not fit on a screen.
The main issue I have is that it goes from dead simple to pit of what the hell is happening with various things interacting with each others.
When `make` is used as a glorified front-end to `bash` scriptlets, why not use `bash` directly instead of having two-level of scripting?
See: https://blog.aloni.org/posts/bash-functional-command-relay/
This (ab)use of `make` runs multiple times a day, every day, and works perfectly every time.
The inspiration of this was an (ab)use of `make` as a way to paralellize Linux system startup: https://web.archive.org/web/20110606144530/http://www.ibm.co...
https://github.com/matheusmoreira/.files/blob/master/GNUmake...
I'm surprised at how well this thing works every time I use it. I even blogged about it.
https://www.matheusmoreira.com/articles/managing-dotfiles-wi...
Recently made a tool that processes the make database and prints the phony targets and their dependencies:
https://github.com/matheusmoreira/.files/blob/master/~/.loca...
I use it as a sort of makefile help command. Works surprisingly well too.
A task runner's tasks could be arbitrarily complicated, pulling in all sorts of dependencies of their own. This is less true for the traditional compile targets make was designed for.
Because the things we do in a Makefile are pretty much always project local and don't get reused, it limits how much heavy lifting these tasks are likely to do for us. Whereas if you built your our CLI in Python with Click or something, you would be able to make it a development dependency of your project. You can afford to invest in those tasks more because they'll be reused.
The Just command runner has the same problem, but at least it's designed to be a task runner.
I think that's called CMake.
I've never used eval.
Make's use of Bash can lead to hairy quoting/escaping. I use Make as a "dumb high-level runner" and put any sort of intelligence, like conditionals or loops or networking, in lower-level scripts or programs.
Make is an orchestrator.
Not for compiling, just as script runner.
I see this practice often and I haven't found a good reason
I use it quite a lot for automating deployments - if you want to Terraform up a VM:
make colo1-fooserver01.vm
make colo1-fooserver01
I think the real reason is you have it all in one file rather than multiple scripts which makes it easier to edit and maintain.
... and if your scripts do all that, they've basically rebuilt make, but it's undocumented and worse.
(I say this as someone with LOTS of experience with make, and am not really a fan because I know too much and it's horrifying. But I dislike custom crippled versions even more.)
For script-running just is great, for full dep tracking build tool something like buck2 is an improvement.
The one place where make shines is when your workflow is truly file-based, so all steps can really be described as some variations of "transform file A to file B".
Make is a powerful tool. You just have to understand how it thinks about the world, and adjust your own thinking a bit if needed.
If you just want to have tasks that depend on other tasks, you don't need stamps, phony, or anything else.
But what happens when you want to say "only rebuild my venv if requirements.txt changed"? that's a file dependency that you can reasonably express between requirements.txt and venv/bin/activate. And then all of a sudden, you're squarely in Make's wheelhouse.
However, even after 10+ years of professional dev work during which I've regularly crossed paths with Makefiles, they still scare me, and I've still never written one from scratch myself. I cling to bash scripts, which I'm also a rookie at (or, for more complex cases, I write Python scripts, which I'm much more comfortable with).
I guess one day I'll read the manual, and digest some tutorials, and actually learn make. But I've made it this far...
Make has its warts (look at when and what it was designed for, after all!), but I've found it much easier to write Makefiles than YAML-for-github-runners.
If you're used to YAML, you get pleasantly surprised when using Make, which does execution of dependency trees in a more readable manner than most CI/CD YAML files do.
If the source code is kept tight, it doesn't yield much of a speed-up, though - precompiling hundreds of unnecessary headers can take a lot of time. Better to put some effort into moving unnecessary #includes out of header files. In fact, you can use your log of cache-hits to guide that work.
My opinion may be a bit skewed by the fact that I write code that gets built on a variety of different platforms, though, and make is essentially universal. It lets me have a consistent build process regardless of platform.
It's also very useful for automation that isn't related to building code.
.PHONY: foo
foo:
echo foo
bar: barin
cp barin bar
.PHONY: baz
baz: bar
echo bazhttps://marketplace.visualstudio.com/items?itemName=lfm.vsco...
It allows you to click above target to run target.
You can do that in Bash. And now you've reinvented Makefile, but poorly.
I made a bash script that takes your Makefile and gives you a nice dialog menu for the targets that have comments. Works nicely as a self documenting project command menu.
https://gist.github.com/oppianmatt/dcc6f19542b080973e6164c71...
https://private-user-images.githubusercontent.com/48596/3262...
- tab completion of targets
- automatic dependency execution
- automatic entry points between every task
- result caching
- parallel execution
Sure, you can do it in bash, or python, or whatever. But then you have a cumbersome, not particularly interesting piece of code full of boiler plate. Of course, you can design it a bit, organise things neatly, and then use a config file because fiddling with the code in each project is unsustainable in the long run. At this point, you’ve just made a poor copy of make and thrown away all the good bits that result from decades of experience and weird corner cases.
The syntax of Makefiles is terrible, but make itself is very useful and versatile.
And that pattern is not abuse, it’s the sort of things Make was designed for. It’s just that we’re used to think of make as this old thing that just runs a compiler and that’s such a pain to deal with that we need Makefile generators to do it properly. And certainly that’s true for complex software compilation, but make is more versatile than that.
Certainly abuse, but hey.
The match with substr is interesting. It's more complicated than setting the field separator to something like :|#+ but should mean : in the help text works. For something one only writes and debugs once, probably better to do the complicated thing that always works.
gawk will write the groups to an array, that's possibly more legible (and slower? should be slower than the leading non-capture //)
@gawk 'match($$0, /^([a-zA-Z_*.-]+):.*## (.*)$$/, arr) {printf " %-30s %s\n", arr[1], arr[2]}' $(MAKEFILE_LIST) | sortThe other options are here: https://github.com/ninja-build/ninja/wiki/List-of-generators...
It's also better from an architectural separation perspective. Your custom CLI will have custom commands and flags, and probably will need to be called in some standardized way. Make is very good at calling your toolchain / build-system (whatever that might be) with the exact arguments that you want. And things like "make all" or "make clean" are muscle-memory for hundreds of thousands of developers.
Why mix your custom tooling with the task of standardizing an entry point?
I've had really great success over the years with the pattern of "some build system or another (cmake, bazel, autotools, etc) orchestrated by a top-level Makefile." It's simple, portable, and flexible. What's not to like, other than ugly syntax?
But how is make dependency free? You need to install make. Which version? GNU make, or FreeBSD make? What platform are you installing it on? What version? In my team we had to get all our devs to manually upgrade from 3 to 4 as we were using modern make features to make it a nicer task runner.
These are all things you've already had to deal with in the custom CLI, which is also a perfectly good entry point. You also have a lot more control of command line arguments, rather than just make targets ("just" has also added this as a feature)
Whitespace padding output in sed is probably horrible, column looks simpler than printf via bash or trying to use make's $info.
https://www.grymoire.com/Unix/Sed.html
as a reference some might occasionally swear by.
Anyhow, sed is a marvelous utility. Unfortunately, most people never learn its real power. The language is very simple, but the documentation is terrible. The Solaris on-line manual pages for sed are five pages long, and two of those pages describe the 34 different errors you can get. A program that spends as much space documenting the errors as it does documenting the language has a serious learning curve.It is difficult for newcomers to guess that "extended regular expressions" refers to the barely-usable subset of "regular expressions" and "posix regular expressions" are terrible in comparison to either.
edit: alright, yes, one can program in that. Sed can recurse.
.PHONY: help3
help3:
sed -nE 's/^([a-zA-Z_*.-]+):.*## (.*)$$/\1 :\2/ p' \
$(MAKEFILE_LIST) | \
sed -E -e ':again s/^([^:]{1,16})[:]([^:]+)$$/\1 :\2/ ' -e 't again ' |\
sed -E 's/^([^ ]*)([ ]*):(.*)$$/\1:\2\3/' |\
sort
Composing the -n/p combination with act on everything is a stumbling block for merging the multiple invocations together but I expect it to be solvable.
HELP_PADDING := 30
.PHONY: awkhelp
awkhelp: ## Write this help using awk
@echo "awkhelp:"
@awk 'BEGIN {FS = ":.*#+"}; /^[a-zA-Z_*.-]+:.*## .*$$/ {printf " %-'$(HELP_PADDING)'s %s\n", $$1, $$2}' \
$(MAKEFILE_LIST) | \
sort
.PHONY: sedhelp
sedhelp: ## Write this help using sed
@echo "sedhelp:"
@sed -E \
-e '/^([a-zA-Z_*.-]+::?[ ]*)##[ ]*([^#]*)$$/ !d # grep' \
-e 's/([a-zA-Z_*.-]+:):?(.*)/ \1\2/ # drop :: and prefix pad' \
-e ':again s/^([^#]{1,'$(HELP_PADDING)'})##[ ]*([^#]*)$$/\1 ##\2/ # insert a space' \
-e 't again # do it again (termination is via {1, HELP_PADDING})' \
-e 's/^([^#]*)##([^#]*)$$/\1\2/ # remove the ##' \
$(MAKEFILE_LIST) | \
sortOnly if there's no dependencies. It's unusual that GP's type of usage has no dependencies.
I hear you, but you're running the other script unconditionally. If it downloads something, it will download it every time you run the first script.
In this simple case, make runs the other script conditionally, so it need not run every time.
The solution is not to ship complicated and bug infested build systems, it is to fix the dependency problem in the same way any other language has done so far and until we do this, ship the dependencies with the program.
And if you aren't making a GUI and don't need to target Windows, just wrap it in Docker, which enforces a functional dependency system and means you can ship your dependencies.
Even if that would be true, still every language that isn't C (or JS in the browser) needs to link or build against some C (or Fortran), which results in more or less working solutions on how to integrate with or build C sources. Of course you may not see that (as long as it works), because somebody else has wrapped that up in a package (or whatever) for your language of choice, but somebody has to do that.
> ship the dependencies with the program.
My post should have been the answer to this argument: this is not possible for example when "shipping" the source for a cross-platform library. Or a cross-platform end-user program/app. Or just about anything which isn't "just" some web-backend or a server of some kind.
Does that mean _you_ need a complicated (I'd call that "working for anything but the most basic stuff") build-system? No.
I am a bit puzzled by this phrase:
> The solution is not to ship complicated and bug infested build systems, it is to fix the dependency problem in the same way any other language has done so far
But any other language than C or C++ (sadly, _way_ less than "any other") solves that by using a complicated and bug infested build system(s) and package manager(s) or a combination of both.
Which would go away if C++ had a working package manager, that worked in the same way every other language did, was written once only and about as bug free as the compiler. This would also allow you to ship the source code for the library with a simple file that list the dependencies you need.
I guess my final issue boils down to this: CMake does too much and too little. It is too hard to get it reliably pull down the libraries I want to use and it can do too much as part of the build to the point that it becomes too complicated.
Do one thing, and do it well.
With maybe one (or two) exceptions, those other languages' build systems are incredibly susceptible to supply-chain attacks.
And, to be honest, unless you have a burning need for autoconf's main value proposition (cross-compiling for a different target system), plain gnu-make and storing your dependencies in your repo is probably a lot safer than many other build systems.
I've built software with dependencies on libpng, libcurl, libsodium and more and was confident in the security of the resulting binary. I've also done one or two node.js projects, and had much less confidence that it won't be supply-chain attacked on the next build.
In other words, if you also don't need to target MacOS or *BSD, which were the parent's stated requirements. MacOS can't run Docker, and BSDs seem to be unstable/unsupported targets, so the only stable way to get Docker on such a machine is to run Linux in a VM. Which isn't really a solution for cross-platform development, as it is a denial of it.
(Also, if you were to go down that route, why not just ship the VM; rather than dragging in all the crap that Docker entails?)
But you made the argument that adding another build dependency is somehow difficult or undesirable which I can't see; the tool is installed in seconds on all major OS-es and package managers.
So let's not conflate those two separate things, yeah?
just doesn't even do many of the things make does, as the README states: "just is a command runner, not a build system"
And yes, `just` isn't a build system per se, it still requires something to track modified source files and whether they map to the requisite build artifacts. Good for C/C++ and maybe stuff like Latex -> HTML but outside of that it's obsolete and thank the gods for that.
If it works well for you, cool. As mentioned up-thread I wouldn't attempt a switch with hundreds of Makefiles already working fine either.
But I'm very happy to tell you that your case is the vanishing minority. Happily the world has moved on.