Proposal to add build graph output to GNU Make (2020)

Proposal to add build graph output to GNU Make (2020)(jonasdn.blogspot.com)

161 points by jonasdn 4 years ago | 37 comments

indygreg2 4 years ago |

I once helped maintain a nearly full featured implementation of make in Python (pymake) that grew out of the Firefox project.

As much as I would like to support efforts like this, I feel like it is ultimately doomed to suffer from usability limitations because make does not have a static DAG. Rather, the DAG evolves dynamically as make evaluates targets. There are pesky problems like $(call) and $(eval) where you can dynamically inject make expressions into a partially evaluated DAG. And targets can generate files which are later loaded via include directives.

Dumping the make database in a machine readable format (for visualization or otherwise) would be incredibly valuable for debugging and could improve understanding. But since many make files have N>>1 "snapshots" of the internal DAG during their execution, there is a really thorny problem around when and which of these "snapshots" to use and how to stitch them together. Many make files aren't "static" enough to support exporting a single, complete, and valuable usable snapshot of their internal DAG.

If debugging is the target goal, I think a potentially better approach would be to define a function that takes an output filename and optional list of targets and writes out the point-in-time build graph when that function is called. This way makefile authors could insert probes in the make file (including at the bottom of a make file so the function is called on file load) and capture the state(s) of the build graph exactly when they need to. There could also be a specially named variable holding the names of targets that when pre- or post-evaluated would trigger the dumping of the database.

Best of luck to the person doing this work. Debugging make is notoriously hard and any progress in this area will be much welcomed by its many users.

loo 4 years ago | |

This must be why after 45 years make still has no "list targets" option.

gjvc 4 years ago | |

Does pymake handle colons in filenames any better / easier / etc than GNU make? (I have a task where I have a bunch of files to transform and they have HH:MM:DD timestamps embedded in the basename of the file)

saagarjha 4 years ago |

Considering that all build systems must keep a dependency graph internally, it’s surprising how many of them make it incredibly difficult to inspect why things are being rebuilt. I don’t really get it either, I’ve talked to a lot of people and getting feedback on how the build system is doing its task is one of the biggest complaints I see (others are “what is the build system doing” and “why does this have the worst syntax and weird edge cases”) and I mean all the build tools have literally everything they need to show this externally, but basically none expose it in a reasonable manner (if at all). I should be able to go, “oh you rebuilt all of libFoo.a because I accidentally made it depend on autogenerated.cpp”. If you are making a new build system today (and I am sure some of you here are considering it :P) please make this one of your priorities when designing the tool.

eddieh 4 years ago | |

GNU Make is pretty good about telling you why things are happening with -d

If you can get away with it use -rR to skip builtins, it makes the output of -d much nicer.

I usually end up running make with -rRd or -rRp when I want to inspect things. -d tells you why things happen and -p shows you what rules and variables are defined.

sideshowb 4 years ago | | |

My bugbear with make (and even the better remake) is finding out why things are not happening. "No rule to make target" is such an uninformative error message. Could it not identify rules which would make the target were they not failing?

pjc50 4 years ago | |

Msbuild (as used in Visual Studio and dotnet core) has an amazing implementation of this: https://msbuildlog.com/

Not only can you see what was rebuilt and why, you can see all the input into the rules at all the stages of the build, conveniently hyperlinked into the master rule files provided in the SDK.

a-dub 4 years ago | |

i think one of the big problems is that when projects get unwieldly with code generation and external systems, build steps often have unintended side effects which then trigger other build rules unexpectedly.

i remember reading about some wild system on here that exposes the entire source tree via fuse so that it can enforce rules about what changes and what doesn't for each step. wish i could remember the name...

edit: ah, yes. tup. https://gittup.org/tup/

edit 2: see tup graph: https://gittup.org/tup/ex_dependencies.html

chriswarbo 4 years ago | |

FYI you can get this from Nix by giving a .drv file to 'nix-store --query --tree' (a .drv file defines how to build something; it's like an individual Make rule (after all variables, etc. have been evaluated)).

You can get a .drv file using 'nix-instantiate' (or 'nix repl'), e.g.

    $ nix-store --query --tree "$(nix-instantiate -E '(import <nixpkgs> {}).bash')"
    /nix/store/378a0jpkxiwzyk6jj6w2y2qdpn8790nc-bash-4.4-p23.drv
    +---/nix/store/j0bqs7l21myz0llnjp9wq73m4nfarzg1-bootstrap-tools.drv
    |   +---/nix/store/71d9qf26s1nzrp34cqw1hh0yrx0h1kbx-cpio.drv
    |   +---/nix/store/7jl6nhfsc0mchccg5w40rfm4zfshpsag-bzip2.drv
    |   +---/nix/store/bci5x35014bbpc87x7ds2n2ffh5syns1-sh.drv
    |   +---/nix/store/fnn1nmssk7fsfn83ik66cxddqxzvnwx6-bootstrap-tools.cpio.bz2.drv
    |   +---/nix/store/p1bhqmlpf6vr9bc5mb99w29mvllcqmsm-mkdir.drv
    |   +---/nix/store/spcnl3whhm392kir1snaar4wlcawr3yq-unpack-bootstrap-tools.sh
    +---/nix/store/04npl9bjcjkpqgwx1s94azbimfgr531v-bash44-012.drv
    |   +---/nix/store/j0bqs7l21myz0llnjp9wq73m4nfarzg1-bootstrap-tools.drv [...]
    |   +---/nix/store/z6aidzdk3yl2k92g8qj8xgzr8glp0vjr-bootstrap-stage0-stdenv-darwin.drv
    |   |   +---/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh
    |   |   +---/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh
    |   |   +---/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh
    ...

It can also output other formats, like GraphML and GraphViz. Note that querying a build output will get its runtime dependencies instead of build-time dependencies.

https://man.archlinux.org/man/community/nix/nix-store.1.en#S...

jhasse 4 years ago | |

    ninja -d explain

?

drothlis 4 years ago | | |

Ninja's "-d explain" is next to useless in a large build: It says what's dirty but it's hard to relate which dirty input caused which output to be re-built. At the beginning of your build you get pages & pages of "these things are dirty" and then it starts the actual build. Also, this "explain" output is printed before ninja has realised that a lot of those things aren't actually dirty (thanks to "restat").

I've been using this in my build system to make the explain output slightly more useful: https://github.com/ninja-build/ninja/pull/2067

It's just a quick hack that took me half a day (most of which was remembering how to C++ after 8 years...) but I think it makes "-d explain" much more useful.

Really, the "explain" implementation could do with an overhaul. I'd love ninja to serialise the build graph to disk (or maybe just the subset of the graph that actually needed rebuilding). This could be augmented with the reason why each rule was run, and possibly even with the command output for each rule. Then the existing ninja tools like "graph", "browse", and "query" could optionally use this graph instead of the "build.ninja" file, so that you can inspect what happened the last time you ran "ninja".

Realistically I'll never get around to implementing this, and my pull request above gets me close enough.

jahnu 4 years ago | |

I don't understand why they can't all have interactive debugging either.

vasvir 4 years ago |

Well there is makefile2graph https://github.com/lindenb/makefile2graph. It is even packaged be Debian and it does exactly that.

I use it like this from inside the Makefile:

  make-db.png:       
      make -f $(MAKEFILE) -Bnd | sed -e 's,$(MAKEDIR)/,,g' | make2graph | dot -Tpng -o $@

but with some gymnastics in order to remove very long path names (MAKEDIR) can be used from the shell directly.

etangent 4 years ago | |

EdSchouten 4 years ago |

Funnily enough, Bazel/Blaze has already supported this for at least a decade. You can just run:

    bazel query ${some_expression} -output graph | dot -Tpdf > out.pdf

https://docs.bazel.build/versions/main/query-how-to.html

ReleaseCandidat 4 years ago |

Irix (SGI's) had a make GUI frontend called 'Build Manager' that had a graph view called 'Build Analyzer' of the dependencies. You could even run 'what would happen, if I change this file?' queries on the graph.

https://techpubs.jurassic.nl/manuals/0650/developer/Debugger...

jgrahamc 4 years ago |

You can hack some of this functionality into GNU Make makefiles by modifying $(SHELL).

https://www.cmcrossroads.com/article/tracing-rule-execution-...

dolmen 4 years ago | |

Clever hack!

eqvinox 4 years ago |

Wouldn't it be possible/easier to create a new/separate script that converts the `make -p` output to the desired `.dot` graph? (Just to have this not be a thing that needs to pass through the GNU make project.)

Also I feel like on most nontrivial projects the graph would be gigantic & to get any usability you'd need something interactive where you can collapse/expand the nodes you're (not) interested in. Don't we have a bunch of JS libraries these days that provide this?

mhh__ 4 years ago | |

It might be easier but having tooling build info your... tooling is genuinely useful

glandium 4 years ago |

ISTR there was a thread about it here a few months ago.

pjmlp 4 years ago |

Some clarification needed,

> The Android build system works by including all recipes to be built (programs / libraries / etc) using the GNU Make include directive, so that you end up with one giant Makefile that holds all rules for building the platform. Possibly to avoid the problems laid out in the paper Recursive make considered harmful.

The old build system (ndk-build) works like this.

Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong.

darkamaul 4 years ago | |

> Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong.

Soong build files are easy to parse and you can create an Unified Dependency Graph from them. That's what I did in BGraph [0] for security applications.

[0] https://github.com/quarkslab/bgraph

malkia 4 years ago | |

AFAIK, Soong -> Bazel - https://blog.bazel.build/2020/11/12/aosp_migrating_to_bazel.... (2020)

sydthrowaway 4 years ago |

Ive tried this before but with a complex build the graph is unwieldy and useless

kergonath 4 years ago | |

It’s probably more suited for the OP’s use case (a couple of files causing some re-building) because otherwise yeah, the build tree can be humongous.

That said, isn’t there some software to visualise dot files in a more sophisticated way than just generating a picture? It would be handy to be able to collapse/expand sub graphs in a GUI, for example.

datashaman 4 years ago | | |

Something like this could be handy. (Disclaimer: never used it)

https://pypi.org/project/pydot/

JNRowe 4 years ago | | |

[Noting that you said GUI, and what I'm about to say isn't]

There's a lot more to Graphviz than just dot, as it comes with some really good filtering tools that people often seem to miss. You can achieve an awful lot with just bcomps/ccomps. And, when you're running queries gc is useful to see that you're heading down the right path.

jcranmer 4 years ago | | |

There's a couple of interactive graph visualization software out there that could be used instead of static dot file generation.

(One of the advantages of DOT is that it's a pretty common de facto standard for describing graphs, so you can import it into most graph manipulation libraries and mangle it further using that.)

kergonath 4 years ago |

This would be quite useful. We have a couple of large-ish projects with a couple of hundreds files compiled in parallel and there are snags every now and then (which disappear when re-running make either in parallel or sequential modes). Getting a graph in these cases would be a nice time saver.

I’ll try the fork.

Ericson2314 4 years ago |

I would very much like this for https://github.com/NixOS/rfcs/pull/92

LAC-Tech 4 years ago |

I get the impression that GNU Make is a fairly hard program to update - to the best of my knowledge it still can't handle filenames with spaces in them, for example. So for me it's a lot like a lot of GNU projects - great idea, but a little bit long in the tooth and unlikely to substantially improve at this point.

danny0z 4 years ago |

xmake can also easily write some lua scripts to dump the build dependency graph https://github.com/xmake-io/xmake

mplanchard 4 years ago |

I have found the `--debug=v` flag to be super helpful for debugging make targets. The `v` flag generates a lot more output than `--debug` with no arguments. Would also love a graph though

tpoacher 4 years ago |

This may be a naive comment, but, what's stopping one from simply appending a graphviz command at the end of each recipe? (manually or otherwise)

asdff 4 years ago |

snakemake has dag output built in

$ nix-store --query --tree "$(nix-instantiate -E '(import <nixpkgs> {}).bash')" /nix/store/378a0jpkxiwzyk6jj6w2y2qdpn8790nc-bash-4.4-p23.drv +---/nix/store/j0bqs7l21myz0llnjp9wq73m4nfarzg1-bootstrap-tools.drv | +---/nix/store/71d9qf26s1nzrp34cqw1hh0yrx0h1kbx-cpio.drv | +---/nix/store/7jl6nhfsc0mchccg5w40rfm4zfshpsag-bzip2.drv | +---/nix/store/bci5x35014bbpc87x7ds2n2ffh5syns1-sh.drv | +---/nix/store/fnn1nmssk7fsfn83ik66cxddqxzvnwx6-bootstrap-tools.cpio.bz2.drv | +---/nix/store/p1bhqmlpf6vr9bc5mb99w29mvllcqmsm-mkdir.drv | +---/nix/store/spcnl3whhm392kir1snaar4wlcawr3yq-unpack-bootstrap-tools.sh +---/nix/store/04npl9bjcjkpqgwx1s94azbimfgr531v-bash44-012.drv | +---/nix/store/j0bqs7l21myz0llnjp9wq73m4nfarzg1-bootstrap-tools.drv [...] | +---/nix/store/z6aidzdk3yl2k92g8qj8xgzr8glp0vjr-bootstrap-stage0-stdenv-darwin.drv | | +---/nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh | | +---/nix/store/bnj8d7mvbkg3vdb07yz74yhl3g107qq5-patch-shebangs.sh | | +---/nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh ...