Show HN: Making GNU Make a better Task Runner

Show HN: Making GNU Make a better Task Runner(github.com)

98 points by mitjafelicijan 2 years ago | 61 comments

I know this could be considered blasphemous, but I constantly find myself using Make as a task runner. I have written my own task runner in the past, but somehow I always end up using make.

I went, and I put together 3 quality of life snippets I use all the time and put it in a single makext.mk file that can be included in other Makefiles and wrote a basic readme for it.

This is not meant to be a replacement for other task runners, but I do think it can be useful to some of you.

https://github.com/mitjafelicijan/makext

Check it out and see if it makes sense to you.

Thanks for any feedback or comments.

Cheers

tpoacher 2 years ago |

Very nice.

Whenever I see comments about people trying to reinvent make, my conclusion is that make "as a tool" is inevitably far superior, but people are mostly familiar with a small subset of its functionality, which makes it appear clunky. That, and the syntax is very flexible, which means it can easily be abused to write ugly/convoluted makefiles.

If I had one thing I'd want to improve about make, it wouldn't be the tool itself, but its documentation, and a set of reasonable defaults/templates, like you have here.

The documentation is amazing in one sense, in that it's very comprehensive, but very clunky and hard to master in another, in that it's organised in a very high-level manner that doesn't allow it to be used as a quick reference / point of truth, if you don't know what you're looking for. I found myself making anki notes for it, as it's not organised in the kind of way that would simply allow me to refer to the docs for your usecase and get things done. You really either know some functionality exists and can look it up or you don't, and unless you're someone who uses make in expert mode on a daily basis, it's hard to know all the specialised components and be able to combine them all together seamlessly. Hence my anki notes, hahah.

But make really is an amazing tool. I wish people focused on improving make workflows instead of reinventing the wheel all the time with other tools (which are then forced on the user).

frankjr 2 years ago | |

> Whenever I see comments about people trying to reinvent make, my conclusion is that make "as a tool" is inevitably far superior, but people are mostly familiar with a small subset of its functionality, which makes it appear clunky.

If you're looking for a task runner, not a build tool, then make is not "far superior" in any sense and there are much better alternatives, the most prominent probably being "just". Even something as basic as accepting parameters to a task (make fetch <arg1> <arg2>) is awkward because make doesn't understand this syntax and interprets the arguments as targets. The only way to make it work is either through named arguments or empty target hacks and in both cases you need to write the validation logic yourself. In just it's simply:

  fetch PACKAGE VER:
    @echo fetching {{PACKAGE}} at version {{VER}}

  $ just fetch foo 
  error: Recipe `fetch` got 1 argument but takes 2
  usage:
    just fetch PACKAGE VER

  $ just fetch foo 1.2.3
  fetching foo at version 1.2.3

https://just.systems/man/en/chapter_1.html

nrclark 2 years ago | | |

Just is fine, don't get me wrong. But Make targets can take arguements too. They just have to be named, or (if you stretch the definition a bit) wildcard targets.

AFAIK, Just doesn't have any way to mark "this task is complete, so don't re-run it". It also doesn't have file-based dependendicies, which are pretty common for any kind of programming.

Why would I want to lose all of Make's flexibility and power in exchange for slightly prettier UI syntax?

nrclark 2 years ago | | |

    ---- 
    fetch PACKAGE VER:
        @echo fetching {{PACKAGE}} at version {{VER}}
    ---- 
    $ just fetch foo 1.2.3

Here is the Make equivalent, with equivalent error-checking

    ---- 
    fetch:
        $(if $(PACKAGE),,$(error PACKAGE was not set))
        $(if $(VERSION),,$(error VERSION was not set))
        @echo fetching $(PACKAGE) at $(VERSION)
    ----
    $ make fetch PACKAGE=foo VERSION=1.2.3

I agree that the `just` version is a little neater / tidier. But it's not like Make can't do the same thing. And Make is a much more powerful tool.

bityard 2 years ago | | |

I like the idea of Just (and the many tools like it), I only wish it had come along a few decades earlier. Make has the advantage of being a ubiquitous de facto standard. It is probably already installed anywhere you want to deploy your repo, or can be very easily installed with one command.

As a personal preference, I actually do tend to avoid Make as a task runner because a quick shell script is almost as easy to write and far more flexible. I am pretty familiar with Make but have still painted myself into a corner with it more times than I'd like to admit.

TylerE 2 years ago | |

Any tool that treats spaces and tabs differently is not sane.

nrclark 2 years ago | | |

Agreed, I don't like that about Make at all.

But also it's never something I have to worry about. Pretty much every syntax-aware text editor knows how to handle Makefiles, and knows that recipe steps start with a tab (unless you change Make's default). So for me, that falls in the "ugly, but not really a usability issue" bucket. And Make's power/expressiveness is worth accepting a few "I'd have done it differently" syntax things.

woodrowbarlow 2 years ago |

people often turn to make as a taskrunner, but i've never understood why. i've heard people say "i don't want to add another tool/language/dependency to my project, so i'll just use make" but, usually, `make` is a new dependency for the project. `make` (realistically) excludes Windows users, most Linux distros don't ship it by default, it is a unique syntax (no, it's not shell). even in a C project, it's not a given that Make is already a dependency. in languages like python or javascript, with rich package ecosystem, `make` makes even less sense; why limit your users to one OS, and why not write your tasks in your project's native language? as a taskrunner, `make` has so many limitations -- because that's not what it's designed for. `make` is good for tracking dependencies between files that are generated from other files, and that's about it.

a good taskrunner makes it easy to run a task while still exposing the tool's underlying flexibility. a good taskrunner lets me invoke the entire test suite with a short command, but also allows me to add custom options and arguments to, say, run a specific test case in an alternate environment.

`make` fails to expose the tools' underlying flexibility. sure, you can write a .PHONY target to run the full test suite, but `make` can't handle passing options or arguments (besides cumbersome Makefile variables).

a makefile tends to obscure the underlying tool, enshrining its launch arguments. anyone who has tried to cross-compile a makefile codebase authored by someone who didn't consider cross-compilation understands what i mean (you'll end up re-writing the Makefile 90% of the time).

smartmic 2 years ago |

Thanks, that is a good idea. I also prefer GNU Make for automatization if the number, diversity or dependencies of the tasks would exceed the benefits of a simple bash script.

I stumbled over this statement in your README:

> It is recommended to use .PHONY for targets that are not actual files. In the example below I am not doing that though, but it is wise to follow that rule.

Please also give the example as you think it would be correct. Or write the understandable reason why you don't do it here. I think many people don't read the text between the code snippets but only scan the code examples. It would be helpful to find the ‘best practices’ right there.

mitjafelicijan 2 years ago | |

You are correct. I will add this. I was not clear enought on this.

Galanwe 2 years ago |

> I know this could be considered blasphemous, but I constantly find myself using Make as a task runner

It's not, don't apologize, reusing battle tested tech is the way.

I have a quite close "template Makefile", though I prefer to store it as a cookiecutter (or more recently "copier") template for easier per-project bootstrap.

You could add a timestamped log function. I use that in mine so that targets have timestamped messages pre/post.

mitjafelicijan 2 years ago | |

This is a very good idea. I will add that. Thank you.

keypusher 2 years ago |

May want to look at Just. It is heavily inspired by Make and shares much of the same syntax, but removes a lot of the workarounds necessary to use Make as a task runner and adds a few other features.

https://github.com/casey/just

nrclark 2 years ago | |

If Just grew to have a large enough feature-set that it could replace GNU Make, it would collect an equal amount of warts and sharp-edges along the way.

It's a little more user-friendly, but not enough to justify how much power you lose.

I spent the last 20 minutes reading Just's documentation, and it seems like the only real wins are nicer argument parsing and && dependencies, and the --list argument. And in exchange for that, you lose file-based dependendicies and Make's powerful templating engine. Did I miss some other features that make the trade worth it?

zokier 2 years ago | | |

For lots of uses not having file-based dependency mechanism is a benefit, not a con. Sometimes less is more

rgoulter 2 years ago | | |

File dependencies are useful as part of a build system.

But often tasks are a step removed from that. Targets like "all" and "clean" are good examples of useful tasks to have, even if they're unrelated to files.

A "task runner" isn't intended to replace all of a Makefile's functionality. It takes a common use case of Makefiles, and improves the user experience for this.

I think it's fair to say that it's not a big step up; but it's a definite improvement for what it does aim to do.

bigstrat2003 2 years ago | | |

I don't want file-based dependencies. I don't understand why anyone would, frankly. Just is an upgrade in that respect, not a downgrade.

mitjafelicijan 2 years ago | |

Just is amazing. I used quite a lot. It is however difficult sometimes to convince others to use it, even though it is a no-brainer.

crabique 2 years ago |

Biggest showstopper for me personally is Make's inability to properly pass through arguments from `make run <arbitrary number and type of arguments>` to the underlying program.

Some of the issues could be avoided by requiring a -- and using the .SILENT modifier, but some are extremely difficult: e.g. it word-splits strings and you can't just pass "some string" as a single argument, everything will be separated by spaces.

In case the underlying program is a script-language wrapper that you also control, it is possible to hack around and let Make pass through its own $$PPID so that the underlying script could read the nul-terminated /proc/<Make's pid>/cmdline and pass it on to the actual program.

This, of course, only works where procfs is a thing, so e.g. on macOS (if it's among your target platforms) you'd have to reinvent the wheel and learn a whole lot of Darwin sysctl dark magic to read arbitrary process' arguments, at which point you'd ask yourself if this is even something a sane person would ever do...

0xbadcafebee 2 years ago |

I've never found the need for a task runner since learning shell scripting.

A lot of people recommend just; here's their sample file:

  alias b := build
  
  host := `uname -a`
  
  build:
      cc *.c -o main
  
  test-all: build
      ./test --all
  
  test TEST: build
      ./test --test {{TEST}}

Here's how I would do that with shell:

  #!/usr/bin/env sh
  set -eu
  
  b="build"
  
  host=`uname -a`

  _cmd_build () {  # build:       Build main program
      cc *.c -o main
  }
  
  _cmd_test_all () {  # test_all:    Run all tests
       _cmd_build
      ./test --all
  }
  
  _cmd_test () {  # test TEST:   Run a test TEST
      _cmd_build
      ./test --test "$1"
  }
  
  _cmd_help () {
      echo "Available targets:"
      grep -E "^_cmd_[^[:space:]]+ \(\) {.+" "$0" | sed -E 's/.*#/  /'
  }

  [ $# -gt 0 ] || _cmd_help
  set -x
  "_cmd_$1" "$@"


  $ chmod 0755 run.sh
  $ ./run.sh
  Available targets:
     build:       Build main program
     test_all:    Run all tests
     test TEST:   Run a test TEST
  ./foo.sh: line 23: $1: unbound variable
  $ ./run.sh build
  $ ./run.sh test_all
  $ ./run.sh test foo.t

But this example (from just's homepage) is clearly a build process, so I would use Make for it instead. When I need to be able to run individual sets of commands with arguments and options, I make a shell script. Many more features available, more flexibility, I can tailor it to my use case.

kstrauser 2 years ago | |

Two things Just gives you beyond that:

- It handles all that boilerplate for you. The only stuff in the justfile is code you want to execute, not the code to figure out which code to execute.

- Out-of-the-box tab completion. "What did I name that recipe? `just t<tab>` Oh, `test-all`, that's right!"

I like not reinventing those wheels. Let someone else manage that hassle for me.

fmajid 2 years ago |

remake is a more powerful tool, a fork of GNU Make that adds a debugger and profiler, among other extensions.

https://remake.readthedocs.io/en/latest/

simonmic 2 years ago |

I used make as a task runner for probably 30 years; using all the tricks to do most of the things you want in that use case (arguments, help, portability, reliability..). I worked around all the idiosyncracies continually. That's enough time sunk into wasteful friction for me. Now just is here, and the people pushing it are right. For this job (task/script manager for more than trivial commands) it's better enough and reduces cognitive load enough that it's very often worth the install requirement and new learning curve. In time it will be more installed-by-default and I'm certain it'll acquire some form of make-like dependent building as well.

[Apologies for contributing to the slightly off topic discussion. makext sounds great for people still using make for this and I totally would have used it in the past.]

mitjafelicijan 2 years ago | |

I totally get it. I know the feeling. I have written some complex Makefiles in the past for C projects that looked completely horrid.

For smaller projects I like make and people who work with me and know me always first check Makefile to see what is there and what the project is about. I use it as a launchpad for installing dependencies and things like that.

And it's all about what is appropriate for that project. Something a shell script is better. Sometimes soemthing else. So I completely understand what you are saying.

simonmic 2 years ago | | |

Appreciate you saying so mitjafelicijan. You're right, it's all about what is appropriate for the project at a given time.

blacklion 2 years ago |

I wonder, why GNU make is much more popular than BSD make/pmake.

Both are extensions to classic make (makefiles without advanced features works with both implementations), but, to my eye, pmake extensions is much more simpler and nicer to use, than GNU make extensions.

Simple ".if/.else/.endif" and true ".for" loop, like in sh is much more comprehensible than GNU make functional extensions to me.

Is it only me?

And BSD systems have extensive library for pmake already, which allows to build programs, libraries (both shared and static) and more in declarative way, without any custom rules. It is like these extensions, but battle-tested for 20+ years. With nice license, too.

barries11 2 years ago |

Nice, I will definitely use these techniques.

Here are a few settings that make GNU make a bit faster, and enable multi-line, strict-mode bash scripts as recipes (make recipes are normally sequences of single-line sh invocations)--and enable quietude & tracing: https://github.com/barries/polling_state_machine_cpp/blob/ma...

mitjafelicijan 2 years ago | |

This is really nice. Thank you for this!

wodenokoto 2 years ago |

A while back someone linked to this relevant comment on HN, about using make as orchestration: https://news.ycombinator.com/item?id=32441602

I also believe I've seen someone setup tools that can draw the dependency graph for you, so you can view (not edit) it airflow style.

pama 2 years ago |

Thanks! Typically I don’t like solutions that involve extra environment variables as things inevitably get messy over time, but your use cases are clean, simple, and common enough that it’s nice to have them documented in one place in this extra makefile.

jjgreen 2 years ago |

I saw MEX_LICENSE and thought: What, this requires a Matlab licence?!?

mitjafelicijan 2 years ago | |

I did not knew they use this naming convention. :) I should probably choose something else :)

jjgreen 2 years ago | | |

Matlab mex files are what they call their plugins ("Matlab executable" I guess), so they're just .so files on Linux etc; I don't think they actually use "MEX_LICENSE" as the envar for their keys, but it's feasible that they might which is why it jumped out at me ...

[tool.taskipy.tasks] fmt = "nox -s fmt" lock = "nox -s lock" install = "if [ $VIRTUAL_ENV ]; then pip install -r local-requirements.txt; else printf '%s\n' 'Please activate a venv first'; return 1; fi" test = "nox -s test test_without_toml typecheck -p 3.12" docs = "nox -s render_readme render_api_docs"