Hello HN. Earthly is a tool for build automation and we just had our biggest release yet.

Earthly is a modern build solution that is similar to a sandboxed makefile. It works with your existing CI and the #1 reason people use it today is to able to reproduce a CI build locally.

Today we promoted a number of important features to GA status in our 0.6 release including:

    - WITH DOCKER : Earthly can execute any number of containers in parallel, via an isolated Docker daemon that it starts up within.
    - User-Defined Commands: Extract common repetitive build steps into a command you can use across your projects.
    - Shared Cache:  Earthly v0.6 now provides shared caching. This  extends our existing build caching to work in ephemeral CI runners by storing the cache in regular container registries.

We started working on this project in 2020[1] on GitHub [2] and while time has gone quickly the number of people and projects using Earthly now is truly exciting.

Let me know what you think! Feature requests always welcome :)

[1] https://news.ycombinator.com/item?id=22890612

[2] https://github.com/earthly/earthly

I love earthly! It has been a joy to use.

One thing I haven’t gotten right is caching. Some of my builds have targets they depend on that are very expensive but rarely change, and I haven’t been able to make sense of when those targets are (re)computed and when they aren’t. This results in many builds taking a long time, when the targets could be reused from a cache.

Earthly is also missing something that I find to be a glaring omission from make—give me a command to simply list the targets in an Earthfile. ‘earthly ls’? Maybe have a column or two that indicates whether the target saves an image or artifact. When building a project I didn’t author, it’d be so helpful for the targets to be discoverable.

> The #1 reason people use Earthly today is to be able to reproduce CI builds locally. Earthly helps create a consistency layer, providing a level of guarantee that the build will run the same on your machine as on your colleagues machine, and as in CI. We call this consistency repeatability (note that this is distinct from reproducibility as full determinism is not guaranteed).

> This helps avoid cases where tweaks to the CI script take long interaction cycles to test (git commit -m "try again..." anyone?)

Now you have my attention. I haven't done much CI work and the long iteration times are the main reason why. I have to edit a file, commit, push, wait for CI to start, wait for CI to complete, search through 10k lines of logs to find the error, repeat... seriously? This is the best we can do? I assumed that I was missing something.

I should be able to just `gh run-ci` in a repository on my machine. Or take one of those fancy online dev environments that everyone's making now and specialize it for editing CI files. Open an editor, let me click Run to run CI on-demand, have live error messages for syntax and missing variables, show a pane with a description of the CI jobs, show what config options are available for each job as it's being edited. Then once the changes are working I can commit and open a PR.

Nice. Been looking for something like this - a Dockerfile equivalent that isn't governed by the Docker folks, with whom I generally do not see eye to eye with.

Bookmarked for later. Glad someone is thinking about this problem space!

I tried using Earthly, great product.

Unfortunately, we rejected it at our organization due to how verbose the commands are. I want my users to be able to say `earthly plan-stage` for a terraform repo, and for it to work. But you cannot.

You cannot have earthly automatically use environment variables from your env, you have to explicitly list the commands yourself using flags:

$ earthly \

  --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \

  --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \

  --build-arg AWS_DEFAULT_REGION=us-west-2 \

  --build-arg AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \

  --build-arg AWS_SECURITY_TOKEN=$AWS_SECURITY_TOKEN \

  +plan-stage

This means if I want to `terraform plan` something, I need to explicitly set all of my AWS environment variables.

Compare this to `task plan-stage` or `make plan-stage`.

It is much too verbose and will only get worse as we add features to our pipelines.

Oh geez.

Dockerfiles are reason #1 for docker's popularity.

Shipping 'the lot' so the artifact works everywhere is #2. This is incredibly inefficient for most use cases.

`docker pull random-shit` is #3.

Everything else it gives is largely immaterial, or maybe even a negative.

You guys are solving #1 and #3, and providing a better alternative to #2. I'm very excited for you and your project!

I haven't tried the product but looks interesting.

I've actually been looking into nixos to generate docker containers after that repl post the other day.

Aside from the software itself I wanted to compliment you on this image.

https://earthly.dev/blog/assets/images/earthly-v0-6/earthly-...

I really appreciate how much context is added there for what earthly actually does or is trying to do. I'll probably save that just to have it as an example to ask people to recreate when I'm baffled by their landing page.

I think in your Articles / Tutorials you should show how to migrate(?) e.g. a "Hello World" Maven project running in GitLab CI to Earthly. This will help people understand it better (from a concept they're already familiar with).

Why is it called "earthly"?

How does this differ from Mesos?

xkcd #927 irl.

You may want to use secrets like this:

     earthly \
      --secret-file config=/home/user/.aws/config \
      +build

But yes, Earthly optimizes for reproducibility, so implicitly pulling in ENVs is not something it does.

Interesting. Dev tools post always catch my attention. For me, your comment regarding `build-arg` is in the same bucket as the comment regarding the need for a `ls` command. Ultimately I predict that the requested feature set will eventually be equivalent to what is already in Bazel -- like `.bazelrc` for setting up build variations and environment variables, and `query` for figuring out what is going on with the project. If Bazel has been working on these features and issues for years, I'd expect the same here.

Earthly seems very useful to work around issues like cross-compiling. A simple example that I'd expect a lot of people to run into is running into libc/stdlibc++ differences with Alpine containers. That is easy to come across when doing something as simple as pulling in a Python pip dependency that wants to compile native C/C++ code. I find this extremely difficult to learn to do in Bazel.

However if you're in the league of complex cross-compiling, then you may already be on CMake and/or Bazel and have the dev resources to define the toolchains as needed.

It’s a build tool, I’d be surprised if something like it worked well for non build related tasks. I suspect bazel would be equally verbose (if not even worse or impossible).

I think you’re better of using something like taskfile w/ earthly for a really good workflow, it gives you the best of both worlds.

(Actually I think I saw this on their slack)

That's my biggest complain about earthly as well. There is `.env` support that ease the problem a bit: https://docs.earthly.dev/docs/earthly-command#environment-va...

Worth watching the reported issue at https://github.com/earthly/earthly/issues/707

could combine it with something like: https://github.com/bbugyi200/funky