Clodl: Turn dynamically linked ELF binaries into self-contained closures

Clodl: Turn dynamically linked ELF binaries into self-contained closures(github.com)

56 points by vsurabhi 5 years ago | 27 comments

ris 5 years ago |

Nix can do this for basically anything - even python scripts with all their dependency modules, binary extensions and all dynamic lib dependencies, packed up into a single .nar file.

nix-shell can even be used as a shebang-line for scripts: https://nixos.org/manual/nix/stable/#use-as-a-interpreter

PaulDavisThe1st 5 years ago | |

On the other hand, Nix is also the only Linux distribution to have broken ("changed") the LD_LIBRARY_PATH behavior of the runtime linker, which means 3rd party "self-contained" packages can't work there, even though they do on every other Linux distro.

So there's that.

tadfisher 5 years ago | | |

The NixOS solution to that is `buildFHSUserEnv` (soon to use bubblewrap under the hood) but obviously that's not ideal.

`nix bundle` (coming with the upcoming flakes feature) promises to make Nix-built binaries runnable without nix, for the opposite of what you want.

jart 5 years ago | |

How useful is that though, for liberating and expropriating binaries? That things got to be at least several megabytes. If I used that as an interpreter or shell script prefix hack then I'd feel like an ant riding on the JVM's coattails.

That's one of the reasons I wrote APE. The bootloader shell script prefix is only 12kb so you can write a small program and it'll feel like it's actually your program. When you distribute it to your friends (respecting their time not asking them to become build system experts, just to try the cool thing you've written) you'll find that APE lets it run on seven operating systems too and most importantly, it'll just feel like something you've written, and that is yours, rather than, "hey grandma install oracle java first, here's how to remove the toolbar" kind of things where you're just evangelizing someone else's platform.

jart 5 years ago |

Author should consider using Actually Portable Executable for the bootstrapper.

fao_ 5 years ago |

    That is, given a shared library or a position independent
    executable (PIE), it returns a single, self-contained file
    packing all dependencies. 
    [...]
    but closures do allow for deploying to other machines without
    concerns about missing dependencies.

So. It's a statically linked binary, but worse? Or like a PAR file, but for C? I'm really not sure what the benefits are here over statically linked binaries.

In fact, since it extracts to a directory in /tmp, it seems to be worse in comparison -- since you have to take the increase in size into account.

I really do not see what the point of this is, or what niche it's supposed to fill over flatpaks or appimages (which both have sandboxing), or why you would pick this over static libraries -- can someone explain this?

toast0 5 years ago | |

It can be tricky to get static executables out of a complex build process that wasn't put together with that goal in mind.

For reasons, I was trying to do that for a while, but it was easier for me to bundle a dynamic exe with the dependencies (including an appropriate dynamic linker) rather than continue to fight with complex makefiles. Without looking at the code, it seems like this ended up similarly.

cmm 5 years ago | |

when have you last tried to create a statically-linked binary that used glibc, and how did that go for you? or how about statically relinking a dynamically-linked binary you don't have sources for?

(but hey, I also felt cheated when I read how clodl works. the hope for magic is always there, isn't it)

choeger 5 years ago | | |

> or how about statically relinking a dynamically-linked binary you don't have sources for?

Even though I don't know why you would do that, I think you could do it by running a modified version of the dynamic linker. That thing basically runs like an interpreter for your binary. It links the code and then executes it. It should not be impossible to modify it to instead dump your ELF again, no?

fao_ 5 years ago | | |

> when have you last tried to create a statically-linked binary that used glibc, and how did that go for you?

I don't write code that requires Glib, but last I remember, there are explicit (lesser-known) flags that give complete static independence :P

My implicit point is that essentially, there is a slew of approaches that seem to be reinventing the wheel to solve the obvious problems in dynamic linking, but they seem to combine the problems of static linking with the problems of dynamic linking. Because people have the memoized knowledge that "static linking old and bad", they twist software into worse solutions because what they want are static libraries and binaries, but for whatever reason they don't feel they can use them.

> or how about statically relinking a dynamically-linked binary you don't have sources for?

Most open source developers don't tend to stray that far off the beaten path, at least in the communities where snap/flatpak/appimage are commonly used.

gravypod 5 years ago | |

My guess is that the benefit is they can write their applications in Haskell, package up the binary into a JVM application/shim (which can be templated in a bazel macro) and deploy a service into something like AWS Lambda or existing infrastructure for their clients. They're a consultancy that favors Haskell https://www.tweag.io/services

nh2 5 years ago | |

I often work on static linking with Nix.

Creating statically linked end user programs that aren't minimal CLI utilities is not trivial.

You need static linking working for all your dependencies (`.a` files) of all your recursive dependencies (hundreds of C libraries, and often up to hundreds of libraries in the higher-level language you may be using).

You need to either use a very configurable package distribution like nixpkgs, or a distribution that specialises in static Linking like Alpine Linux.

These toolchains now make static linking possible and much easier, and nixpkgs can build hundreds of C and Haskell programs statically. Go is also easy. When you can statically link, it is often better, but there are still many programs for which it doesn't work yet, and just "packing up all the .so files" as proposed in the announcement is certainly easier.

For people who want to contribute to static linking, consider contributing to nixpkgs/NixOS and Alpine Linux.

Wowfunhappy 5 years ago | |

Pretty please, tell me how to compile static binaries and I’ll do that forever. I always end up with certain external libraries still being required.

hctaw 5 years ago | | |

You need to compile a static executable (not with static libraries, there's a subtle difference). The flags depend on your compiler.

If you're on Linux, musl libc and musl-tools are useful for dealing with this reliably.

PaulDavisThe1st 5 years ago | | |

Step 3b is to make sure you don't use libraries that themselves cannot be fully statically linked (hello, GTK!).

It's cool that they use run-time dynamic loading to "modularize" certain features (such as image loading), but it seriously damages the benefits of static linkage.

jart 5 years ago | | |

Use Cosmopolitan Libc. Everything is statically linked. See for example https://justine.lol/redbean/index.html don't you want binaries like that?

CalChris 5 years ago |

How is this different from prelink?

https://linux.die.net/man/8/prelink

saagarjha 5 years ago | |

Prelinking just attempts to fix up load addresses so there are no conflicts at runtime that require relocations; it doesn’t bundle the libraries and as such solves a fundamentally different problem.

q3k 5 years ago |

Great! I was considering writing some Bazel rules for _exactly_ this yesterday.

That is, given a shared library or a position independent executable (PIE), it returns a single, self-contained file packing all dependencies. [...] but closures do allow for deploying to other machines without concerns about missing dependencies.