I'll think twice before using GitHub Actions again(ninkovic.dev) |
I'll think twice before using GitHub Actions again(ninkovic.dev) |
https://medium.com/@bitliner/why-gitlab-can-be-a-pain-ae1aa6...
I think it’s not only GitHub.
Ideally we should handle it as any other code, that is: do tests, handle multiple environments including the local environment, lint/build time error detection etc
If it's not open source, I have no idea why you'd use GitHub at all. (And even then.)
Keep your eggs in your own nest.
if they're really independent out them in separate repos.
Yeah-yeah, but it's not like they allow you to run your build definitions locally nor they address some other concerns. With GHA you may use nix-quick-install in a declarative manner, nixify your builds and then easily run them locally and under GHA. In case of jenkins/tc you would have to jump through much more hoops.
Similarly, we are adopting Dagger, more as part of a larger "containerize all of our CI steps" which works great for bringing parity to CI & local dev work. There are a number of secondary benefits and the TUI / WUI logs are awesome.
Between the two, I have removed much of the yaml engineering in my work
Pros:
- pipeline as code, write it as golang, python, typescript or a mix of thr above.
- Really fast once cached
- Use your languages library for code sharing, versioning and testing
- Runs everywhere local, ci etc. Easy to change from github actions to something else.
Cons:
- Slow on the first run. Lots of pulling of docker images
- The DSL and modules can feel foreign initially.
- Modules are definitely a framework, I prefer just building having a binary I can ship (which is why the rust SDK doesnt support modules yet).
- Doesn't handle large mono repos well, it relies heavily on caching and currently runs on a single node. It can work if you don't have 100 of services especially if the builder is a large machine.
Just the fact that you can actually write ci pipelines that can be tested, packaged, versioned etc. Allows us to ship our pipelines as products which is quite nice and something we've come to rely on heavily
I'm a fanboy of Rust, Containerization, and everything-as-code, so on paper Dagger and your Rust SDK seems like it's made for me. But when I read the examples... I dunno, I just don't get it.
In the end you do end up slinging apt install commands for example, but you can test those parts in isolation. Does my ci actually scan this kind of vulnerability, install postgres driver, when I build a rust binary is it musl and working on scratch images.
In some sense dagger feels a little but like a programmatic wrapper on top of docker, because that is actually quite close to what it is.
You can also use it for other things because in my mind it is the easiest way of orchestrating containers. For example running renovate over a list of repositories, spawning adhoc llm containers (pre ollama), etc. Lots of nice uses outside of ci as well even if it is the major selling point
That conceptually allows one to have two different "libraries" in your CI: one in literal golang, as a function which takes in a source and sets up common step configurations, and the other as Dagger Functions via Modules (<https://docs.dagger.io/features/modules> or <https://docs.dagger.io/api/custom-functions#initialize-a-dag...>) which work much closer to GHA uses: blocks from a organization/repo@tag style setup with the grave difference that they can run locally or in CI, which for damn sure is not true of GHA uses: blocks
The closest analogy I have is from GitLab CI (since AFAIK GHA does not allow yaml anchors nor "logical extension"):
.common: &common # <-- use whichever style your team prefers
image: node:21-slim
cache: {} # ...
environment: [] # ...
my-job1:
stage: test
<<: *common
script: [ npm, run, test:unit, run ]
my-job2:
stage: something-else
extends: .common
script: echo "do something else"
I do have a GitHub Actions file for a purpose which is not related to the program itself; specifically, for auto-assignment of issues. In this case, it is clearly not intended to run locally (although in this case you could do so anyways if you could install the "gh" program on your computer and run the command mentioned there locally, but it is not necessary since GitHub will do it automatically on their computer).
on:
issues:
types:
- opened
pull_request:
types:
- opened
permissions:
contents: read
issues: write
pull-requests: write
jobs:
default:
runs-on: ubuntu-latest
steps:
- run: gh issue edit ${{ github.event.issue.number }} --add-assignee ${{ github.repository_owner }}
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}Basically: you set GitGuard as your required check and then write a simple GitGuard workflow like this:
if anymatch(pull_files,"src/backend/.*") {
assert(checkpassed("backend-tests"))
}
However, I’ve come full circle on this topic. My current position is that you must go all-in with a given CI platform, or forego the benefits it offers. So all my pipelines use all features, to offer a great experience for devs relying on them: Fast, reproducible, steps that are easy to reason about, useful parameters for runs, ...
I used to be a multirepo proponent, but have fallen in love with Bazel and “everything bagel” repo.
Of course; I was only quoting the article. I am a firm believer in making things as simple as possible until you need something complicated, and monorepos/Bazel definitely get a “complicated” label from me.
It's very convenient and avoid having to mess with the CI limitations on the matter
For example, if you've ever had to wade into the codesigning/notarization quagmire, observing the methods projects use with Github Actions to do it, can teach you a lot about how to do things, locally.
I really hope there will be a nice, official tool to run gh actions locally in the future. That would be incredible.
Apart from the nix gh action that just runs "nix flake check", the only other actions are for making a github release on certain tags, and uploading release artifacts - which is something that should be built-in IMO.
Disclaimer I am the tool author - https://github.com/melezhik/sparky
We are very much in the part of the platform cycle where best-in-breed is losing out to all-in-one. Hopefully we see things swing in the other direction in the next few years where composable best-in-breed solutions recapture the hearts and minds of the community.
I've used Tekton, Jenkins, Travis, Hudson, StarTeam, Rational Jazz, Continuum and a host of other CI systems over the years but GitHub Actions ain't bad.
This month I start to create to my team our own tool to build CI, I'm using go lang and create a webhook who call my API and apply what is need.
I'm saying this because you can create the CI with your features.
This basically brings docker layer caching to CI. Only Things that changed are rebuilt and tested.
Seen a couple posts on here say otherwise.
I also enjoy the "randomly and undebuggably hang until timeout" (25mins?) which is annoying and incomprehensible and costs money.
Rerunning the same jobs always passes.
GitHub cares. GitHub cares about active users on their platform. Whether it's managing PRs, doing code reviews, or checking the logs of another failed action.
(Whitespace is just noise in a typescript repo with automatic formatting)
My policy is to never let pipeline DSLs contain any actual logic outside orchestration for the task, relying solely on one-liner build or test commands. If the task is more complicated than a one-liner, make a script for it in the repo to make it a one-liner. Doesn't matter if it's GitHub Actions, Jenkins, Azure DevOps (which has super cursed yaml), etc.
This in turn means that you can do what the pipeline does with a one-liner too, whether manually, from a vscode launch command, a git hook, etc.
This same approach can fix the mess of path-specific validation too - write a regular script (shell, python, JS, whatever you fancy) that checks what has changed and calls the appropriate validation script. The GitHub action is only used to run the script on PR and to prepare the CI container for whatever the script needs, and the same pipeline will always run.
The reason it gets unbearably messy is because most people google "how to do x in github actions" (e.g. send a slack message) and there is a way and it's almost always worse than scripting it yourself.
Our NX is pointed to store artifacts in GHA, but our GHA scripts don't do any caching directly, it is all handled by NX. It works so well I would even consider pulling a nodejs environment to run it in non-nodejs projects (although I haven't tried, probably would run into some problems).
It is somewhat heavy on configuration, but it just moves the complexity from CI configuration to NX configuration (which is nicer IMO). Our CI pipelines are super fast if you don't hit one of one of our slow compilling parts of the codebase.
With NX your local dev environment can pull cached items that were built from previous CI ran-jobs or other devs. We have some native C++ dependencies that are kind of a pain to build locally, our dev machines can pull the built binaries built by other devs (since all devs and CI also share the same cache-artifacts storage). So it makes developing locally a lot easier as well, I don't even remember last time I had to build the native C++ stuff myself since I don't work on it.
The only functionality a CI tool should be providing is:
- starting and running an environment to build shit in
- accurately tracking success or failure
- accurate association of builds with artifacts
- telemetry (either their own or integration) and audit trails
- correlation with project planning software
- scheduled builds
- build chaining
That’s a lot, but it’s a lot less than any CI tool made in the last 15 years does, and that’s enough.
There’s a big difference for instance between having a tool that understands Maven information enough to present a build summary, and one with a Maven fetch/push task. The latter is a black box you can’t test locally, and your lead devs can’t either, so when it breaks, it triggers helplessness.
If the only answer to a build failure is to stare at config and wait for enlightenment, you fucked up.
I write a script called `deploy.sh` which is my wrapper for my ci/cd jobs. It takes options and uses those options to find the piece of code to run.
The ci/cd job can be parameterized or matrixed. The eventually-run individual jobs have arguments, and those are passed to deploy.sh. Secrets/environment variables are set from the ci/cd system, also parameterized/matrixed (or alternately, a self-hosted runner can provide deploy.sh access to a vault).
End result: from my laptop I can run `deploy.sh deploy --env test --modules webserver` to deploy the webserver to test, and the CI/CD job also runs the same job the same way. The only thing I maintain that's CI/CD-specific is the GitHub Action-specific logic of how to get ready to run `deploy.sh`, which I write once and never change. Thus I could use 20 different CI/CD systems, but never have to refactor my actual deployment code, which also always works on my laptop. Vendor lock-in is impossible, thanks to a little abstraction.
(If you have ever worked with a team with 1,000 Jenkins jobs and the team has basically decided they can never move off of Jenkins because it would take too much work to rewrite all the jobs, you'll understand why I do it this way)
I've struggled in some teams to explained why it's better to extract your command in scripts (ShellCheck on it, scripts are simple to run locally etc...) instead of writing a Frankenstein of YAML and shell commands. I hope someday to find an authoritative guidelines on writing pipeline that promote this approach so at least I can point to this link instead of defending myself being a dinosaur!
Getting anything done in such environments are just a pain. You spend more time fighting the systems than you do actually solving problems. It is my opinion that a CI/CD system needs just the following features: Triggers (source code repo, http endpoints or manually triggered), secret management and shell script execution. That's it, you can build anything using that.
"But then we can reuse the same pipeline for all our projects!"
CI/CD is a method to automate tasks in the background that you would otherwise run on your laptop. The output
of the tasks are used as quality gates for merging commits, and for deployments.
- Step 1. Your "laptop in the cloud" requires some configuration (credentials, installed software, cached artifacts)
before a job can be run.
- Requires logic specific to the CI/CD system
- Step 2. Running many jobs in parallel, passing data from step to step, etc requires some instructions.
- Requires logic specific to the CI/CD system
- Step 3. The job itself is the execution of a program (or programs), with some inputs and outputs.
- Works the same on any computer (assuming the same software, environment, inputs, etc)
- Using a container in Step 1. makes this practical and easy
- Step 4. After the job finishes, artifacts need to be saved, results collected, and notifications sent.
- Some steps are specific to the CI/CD system, others can be a reusable job
Step 3 does not require being hard-coded into the config format of the CI/CD system. If it is instead
just executable code in the repo, it allows developers to use (and work on) the code locally without
the CI/CD system being involved. It also allows moving to a different CI/CD system without ever rewriting
all the jobs; the only thing that needs to be rewritten are the CI/CD-specific parts, which should be
generic and apply to all jobs pretty much the same.
Moving the CI/CD-specific parts to a central library of configuration allows you to write some code
once and reuse it many times (making it DRY). CircleCI Orbs, GitHub Actions, Jenkins Shared Libraries/
Groovy Libraries, etc are examples of these. Write your code once, fix a bug once, reuse it everywhere.Most programmers have never written a shell script and writing CI files is already frustrating because sometimes you have to deploy, run, fix, deploy, run, fix, which means nobody is going to stop in the middle of that and try to learn shell scripting.
Instead, they copy commands from their terminal into the file and the CI runner takes care of all the rough edges.
I ALWAYS advise writing a shell script but I know it's because I actually know how to write them. But I guess that's why some people are paid more big bux.
The 'Frankenstein' approach isn't what makes it necessarily worse. Eg Makefiles work like that, too, and while I have my reservations about Make, it's not really because they embed shell scripts.
and it feels like fighting against the flow when you're trying to make it reusable across many repos
This syndrome is very common these days. Things are explained differentially: it's like Circle CI but in the GH repo. Well that's no use if the audience wasn't around when Circle CI was first new and readily explained (It's like Jenkins but in the cloud...).
I call this “isomorphic CI” — ie: as long as you set the correct env vars, it should run identically on GitHub actions, Jenkins, your local machine, a VM etc
I just did my own thing and wrapped everything deploy.sh and test.sh and when the shift to another system came... well it was still kind of annoying, but at least I wasn't recreating the whole thing.
Even if the ”meat” of the script is a single build.ps oneliner, I quickly end up with 200 line yaml scripts which have no chance of working locally.
The trick to working with this is that you don't need any of their custom Azure DevOps task types, and can use the shell type (which has a convenient shorthand) just as well as in any other CI environment. Even the installer tasks are redundant - in other CI systems, you either use a container image with what you need, or install stuff at the start, and Azure DevOps works with both of these strategies.
So no, it's neither hard nor impossible, but Microsoft's half-assed approach to maintaining Azure DevOps and overall overcomplicated legacy design makes it a bit hard to realize that doing what their documentation suggests is a bad idea, and that you can use it in a modern way just fine. At least their docs do not recommend that you use the dedicated NPM-type task for `npm install` anymore...
(I could rant for ages about Azure DevOps and how broken and unloved it is from Microsoft's side. From what I can tell, they're just putting in the minimum effort to keep old Enterprise customers that have been there through every rename since Team Foundation Server from jumping ship - maybe just until Github's enterprise side has matured enough? Azure DevOps doesn't even integrate well with Azure, despite its name!)
You can, of course, chew it down to a bare minimum. But I really wish more CI systems would just show up with "you configure us with scripts" instead of the "declarative" nonsense.
But that caters only for each individual command... as you mention the orchestration is still coded in, and duplicated from what rake knows and would do.
So I'm currently trying stuff that has a pluggable output: one output (the default) is that it runs stuff, but with just a rake var, instead of generating then running commands it generates workflow content that ultimately gets merged in an ERB workflow template.
The model I like the most though is Nix-style distributed builds: it doesn't matter if you do `nix build foo#bar` (local) or `nix build -j0 foo#bar` (zero local jobs => use a remote builder†), the `foo#bar` "task" and its dependents gets "built" (a.k.a run).
† builders get picked matching target platform and label-like "features" constraints.
Ever since there has been gitlab-runner, I've wondered why the hell can't I just submit some job to a (list of) runner(s) - some of which could be local - without the whole push-to-repo+CI orchestrator? I mean I don't think it would be out of this world to write a CLI command that locally parses whatever-ci.yml, creates jobs out of it, and submit them to a local runner.
Still, some actions or CI steps are also not meant to be run locally. Like when it publishes to a repo or needs any credentials that are used by more than one person.
Btw, Github actions and corresponding YAML are derived from Azure DevOps and are just as cursed.
The whole concept of Github CI is just pure misuse of containers when you need huge VM images - container is technically correct, but a far fetched word for this - that have all kinds of preinstalled garbage to run typescript-wrapped code to call shell scripts.
Not sure how much of that does Github do, but it could parse test output (I know we used that feature in Gitlab back when I was using that on a project), track flaky tests for you or give you a link to code for a failing test directly.
Or it could highlight releases on their "releases" page, with release notes prominently featured.
And they allow you to group pipelines by type and kind, filter by target environment and such.
On top of that, they provide a library of re-usable workflows like AWS login, or code checkout or similar.
With all that, the equation is not as clear cut: you need to figure out how to best leverage some of those (or switch to external tools providing them), and suddenly, with time pressure, just going with the flow is a more obvious choice.
This is one thing I really love about Buildkite[0] -- being able to run the agent locally. (Full disclosure: I also work for Buildkite.) The Buildkite agent runs as a normal process too (rather than as a Docker container), which makes the process of workflow development way simpler, IMO. I also keep a handful of agents running locally on my laptop for personal projects, which is nice. (Why run those processes on someone else's infra if I don't have to?)
>Reusability and YAML
This is another one that I believe is unique to Buildkite and that I also find super useful. You can write your workflows in YAML of course -- but you can also write them in your language of choice, and then serialize to YAML or JSON when you're ready to start the run (or even add onto the run as it's running if you need to). This lets you encapsulate and reuse (and test, etc.) workflow logic as you need. We have many large customers that do some amazing things with this capability.
Asking because Github Action also supports Self-Hosted runners [1].
[1] https://docs.github.com/en/actions/hosting-your-own-runners/...
CI must only be an application of tools available to developers locally.
This, other than for practical reasons, is a tax on complexity.
The pipeline DSLs, because they are not full programming languages have to include lots of specific features and options and if you want something slightly outside of what they are designed for, you are out of luck. In a way it feels like how graphics were in the age of fixed-pipeline, when there had to be a complex API to cover all use cases yet it was not flexible enough.
This is not possible with GHA.
However, replicating environmental details is only relevant where the details are known to matter. A lot of effort has been wasted and workflows crippled by the idea that everything must be 100% identical irrespective of actual dependencies and real effects.
How would you suggest I do this following your advice?
I suspect the author of the article could greatly simplify matters if they used a task running tool to orchestrate running tasks, for example. Pick whatever manner of decoupling you want really, most of the time this is the path to simplified CI actions. CI is best when its thought of as a way to stand up fresh copies of an environment to run things inside of.
I have never had the struggles that so many have had with CI as a result. Frankly, I'm consistently surprised at how overly complex people make their CI configurations. There's better tools for orchestration and dependency dependent builds, which is not its purpose to begin with.
It seems to me that a big part of the problem here (which I have also seen/experienced) is that there's no one specific thing that something like GitHub Actions is uniquely suited for. Instead, people want "a bunch of stuff to happen" when somebody pushes a commit, and they imagine that the best way to trigger all of that is to have an incredibly complex - and also bespoke - system on the other end that does all of it.
It's like we learned the importance of modularity in the the realm of software design, but never applied what we learned to the tools that we work with.
https://github.com/Pernosco/gha-runner
but I'm not sure how complete it is, and probably doesn't satisfy the author's use cases
(Disclaimer: I work at CircleCI)
At CircleCI for example, we have added valuable features like a VSCode extension[0] to validate and "dry-run" config from within your IDE, we have local runners[1] that you can use to test and run pipelines on your local machine and your own infra, we have dynamic config[2], a Javascript/Typescript SDK[3], a CLI that can validate and run workflows locally[4], and QoL additions like a no-op job type[5] and flexible requires, along with flexible when statements and expression based job filters[6].
And finally, it's of course also possible to combine different approaches into a "best of both worlds" approach, f.e. combining Dagger with CircleCI[7].
[0]https://circleci.com/docs/vs-code-extension-overview/
[1]https://circleci.com/blog/using-runner-for-local-testing/
[2]https://circleci.com/docs/dynamic-config/
[3]https://circleci.com/docs/circleci-config-sdk/
[4]https://circleci.com/docs/how-to-use-the-circleci-local-cli/
[5]https://circleci.com/changelog/new-job-type-no-op-job-can-ma...
[6]https://circleci.com/changelog/more-flexible-job-required-ca...
If you write behaviour in python/ruby/bash/hell-rust-if-you-really-want and leave your github action at `run: python some/script.py` then you'll have something that's much easy to test locally, and save yourself a lot of pain, even if you wind up with slightly more boilerplate.
Much more intuitive, taking shell scripts and other script commands natively and not devolving into a mess of obfuscated typescript wrapped actions that need a shit ton of dependencies.
I’m not even sure when I started feeling like that was a bad thing. Probably when they started glueing a bunch of badly executed security crud onto the main product.
GitHub Actions allows you to only run checks if certain conditions are met, like "only lint markdown if the PR contains *.md files". The moment you decide to use such rules, you have the "Pull request and required checks" problem. No "monorepo" required.
GitHub required checks at this time allow you to use with external services where GitHub has no idea what might run. For this reason, required checks HAVE to pass. There's no "if it runs" step. A required check on an external service might never run, or it might be delayed. Therefore, if GH doesn't have an affirmation that it passed, you can't merge.
It would be wonderful if for jobs that run on GH where GH can know if the action is supposed to run, if required checks could be "require all these checks if they will be triggered".
I have encountered this problem on every non-trivial project I use with GitHub actions; monorepo or not.
Also: I'm not asserting that the below is good, just that it works.
First, don't make every check a required check. You probably don't need to require that linting of your markdown files passes (maybe you do! it's an example).
Second, consider not using the `on:<event>:paths`, but instead something like `dorny/paths-filter`. Your workflow now runs every time; a no-op takes substantially less than 1 minute unless you have a gargantuan repo.
Third, make all of your workflows have a 'success' job that just runs and succeeds. Again, this will take less than 1 minute.
At this point, a no-op is still likely taking less than 1 minute, so it will bill at 1 minute, which is going to be $.008 if you're paying.
Fourth, you can use `needs` and `if` now to control when your 'success' job runs. Yes, managing the `if` can be tricky, but it does work.
We are in the middle of a very large migration into GitHub Actions from a self-hosted GitLab. It was something we chose, but also due to some corporate choices our options were essentially GitHub Actions or a massive rethink of CI for several dozen projects. We have already moved into code generation for some aspects of GitHub Actions code, and that's the fifth and perhaps final frontier for addressing this situation. Figure out how to describe a graph and associated completion requirements for your workflow(s), and write something to translate that into the `if` statements for your 'success' jobs.
The required check configuration on github is just based off of job name, so either the trigger condition is true, and the real one has to succeed or the trigger condition is false and the no op one satisfies the PR completion rules instead.
It seems crazy to me that such basic functionality needs such a hacky workaround, but there it is.
Managing a monorepo with acyclic dependencies is super easy: dornys path filter in one job and the other jobs check
1. whether their respective or any dependency's path got changed 2. and all dependency jobs were either successful or skipped.
Done. No need to write an article.
> GitHub doesn't care
My take: GitHub only built Actions to compete against GitLab CI, as built-in CI was taking large chunks of market share from them in the enterprise.
As far as I can tell, they are identical as far as testing locally. If you want to test locally, then put as much logic in shell scripts as possible, decoupled from the CI.
Common examples are secrets. Organization or repository secrets are very convenient, but they are also massive security holes just waiting for unsuspecting victims to fall into.
Repository environments have the ability to have distinct secrets, but you have to ensure that the right workflows can only access the right environments. It's a real pain to manage at scale.
Being able to `inherit` secrets also is a massive footgun, just waiting to leak credentials to a shared action. Search for and leak `AWS_ACCESS_KEY_ID` anyone?
Cross-repository workflow triggering is also a disaster, and in some circumstances you can abuse the differences in configuration to do things the source repository didn't intend.
Other misc. things about GHA also are cool in theory, but fall down in practice. One example is the wait-timer concept of environments. If you have a multi-job workflow using the same environment, wait-timer applies to EACH JOB in the environment. So if you have a build-and-test workflow with 2 jobs, one for build, and one for test, each step will wait `wait-timer` before it executes. This makes things like multi-environment deployment pipelines impossible to use this feature, unless you refactor your workflows.
Overall, I'd recommend against using GHA and looking elsewhere.
Would GitLab have been better? Maybe. But chances are that there is another edge case that is not handled well there. You're in a PaaS world, don't expect the platform to adjust to your workflow; adjust your workflow to the platform.
You could of course choose to "step down" (PaaS to IaaS) by just having a "ci" script in your repo that is called by GA/other CI tooling. That gives you immense flexibility but also you lose specific features (e.g. pipeline display).
You'd think, especially with the deep VS Code integration, they'd have at least a basic sanity-check locally, even if not running the full pipeline.
git commit --allow-empty -m "bump ci"
> yolo = "!git commit --all --amend --no-edit && git push --force #"
> In GitHub you can specify a "required check", the name of the step in your pipeline that always has to be green before a pull request is merged. As an example, I can say that web-app1 - Unit tests are required to pass. The problem is that this step will only run when I change something in the web-app1 folder. So if my pull request only made changes in api1 I will never be able to merge my pull request!
Continuous Integration is not continuous integration if we don’t test that a change has no deleterious side effects on the rest of the system. That’s what integration is. So if you aren’t running all of the tests because they’re slow, then you’re engaging in false economy. Make your tests run faster. Modern hardware with reasonable test runners should be able to whack out 10k unit tests in under a minute. The time to run the tests goes up by a factor of ~7-10 depending on framework as you climb each step in the testing pyramid. And while it takes more tests to cover the same ground, with a little care you can still almost halve the run time replacing one test with a handful of tests that check the same requirement one layer down, or about 70% moving down two layers.
One thing that’s been missing from most of the recent CI pipelines I’ve used is being able to see that a build is going to fail before the tests finish. The earlier the reporting of the failure the better the ergonomics for the person who triggered the build. That’s why the testing pyramid even exists.
Obviously this would be a real pain to implement just to fix the underlying problem, but it's an interesting (awful) solution
1. We apparently don’t even have a name for it. We just call it “CI” because that’s the adjacent practice. “Oh no the CI failed”
2. It’s conceptually a program that reports failure if whatever it is running fails and... that’s it
3. The long-standing principle of running “the CI” after merging is so backwards that that-other Hoare disparagingly called the correct way (guard “main” with a bot) for The Not Rocket Science Principle or something. And that smug blog title is still used to this day (or “what bors does”)
4. It’s supposed to be configured declaratively but in the most gross way that “declarative” has ever seen
5. In the true spirit of centralization “value add”: the local option of (2) (report failure if failed) has to be hard or at the very least inconvenient to set up
I’m not outraged when someone doesn’t “run CI”.
Martin Fowler apparently calls this “continuous build” (the build itself without CI necessarily). And that’s better.
Article body: "So we use a monorepo and-"
Tale as old as time
Information should only from from the user scripts to the CI, and communication should be done by creating files in a specific format and location. This way the system can run and produce the same results anywhere provided it has the right environment/container.
Still a lot of computation & some wait time, but you can just click & forget. You can also parallelize it; since branches are rebased on each other, you can run CI in advance and, assuming your predecessor is also successful, reuse the result from yours.
Only available for enterprise orgs though.
It is nice to know that if humans thought your change is OK, you're done. I've only committed small changes (compiler diagnostics, documentation) nothing huge, so perhaps if you really get up in it that's more work, but it was definitely a pleasant experience.
... and sure enough it turns out that work on one of the bors successors was in fact discontinued because you should just use this GitHub feature. TIL.
GHA/Gitlab CI/Buildkite/whatever else floats your boat then just builds a bunch of Bazel targets, naively, in-order etc. Just lean on Bazel fine-grained caching until that isn't enough anymore and stir in remote build execution for more parallelism when you need it.
This works up until ~10M+ lines of code or ~10ish reasonably large services. After that you need to do a bit more work to only build graph of targets that have been changed by the diff. That will get you far enough that you will have a whole team that works on these problems.
Allowing the CI tools to do any orchestration or dictate how your projects are built is insanity. Expressing dependencies etc in YAML is is the path to darkness and is only really justifiable for very small projects.
CI works _fantastic_. Absolute top-notch.
Then I have to sort out deployment...
This just seems like a bad implementation to me?
There are definitely ways to set up your actions so that they run all of the unit tests without changes if you'd like, or so that api1's unit tests are not required for a web-app1 related PR to be merged.
Additionally, you can use conditionals based on inputs in the `workflow_dispatch` meaning that you could easily setup a "skip api tests" or "include web tests" option.
If this is true, and you still have problems running specific Actions, why not break this into separate repositories?
all-done:
name: All done
# this is the job that should be marked as required on GitHub. It's the only one that'll reliably trigger
# when any upstream fails: success
# when all upstream skips: pass
# when all upstream success: success
# combination of upstream skip and success: success
runs-on: ubuntu-latest
needs:
- calculate-version
- cargo-build
- cargo-fmt
- cargo-clippy-and-report
- cargo-test-and-report
- docker-build
- docker-publish
if: |
always()
steps:
- name: Fail!
shell: bash
if: |
contains(needs.*.result, 'failure') ||
contains(needs.*.result, 'cancelled')
run: |
echo "One / more upstream failed or was cancelled. Failing job..."
exit 1
- name: Success!
shell: bash
run: |
echo "Great success!"
Now, I did end up running the tests of the front-end and back-end because they upload coverage, and if my coverage tool doesn't get both, it'll consider it as a drop in coverage and fail its check.
But in general, I agree with the writer of the post that it all feels like it's not getting enough love.
For example, there is no support for yaml anchors, which really hampers reusability on things that cannot be extracted to separate flows (not to mention separate flows can only be nested 4 deep).
There is also the issue that any commit made by GitHub actions doesn't trigger another build. This is understandable, as you want to avoid endless builds, but sometimes it's needed, and then you need to do the ugly workaround with a PAT (and I believe it can't even be a fine-grained one). Combine that with policies that set a maximum time limit on tokens, your build becomes brittle, as now you need to chase down the person with admin access.
Then there is the issue of Docker actions. They tell you to pin the action to an sha to prevent replacements. Except the action itself points to a replaceable tag.
Lastly, there is a bug where when you create a report for your action, you cannot specify the parent it belongs to. So your ESLint report could be made a child of your coverage report.
The Azure machines GitHub uses for the runners by default have terrible performance in almost every regard (network, disk, CPU). Presumably it would be more reliable when using your own runners, but even the Actions control plane is flaky and doesn't always schedule jobs correctly.
We switched to Buildkite at $DAYJOB and haven't looked back.
A healthier workflow is to keep all the logic (build, test, deploy) in portable scripts and let the CI only orchestrate each script as a single step. It’s easier to troubleshoot, possible to run everything on a dev machine, and simpler if you ever migrate away from GitHub.
For monorepos, required checks are maddening. This should be a first-class feature where CI can dynamically mark which checks apply on a PR, then require only those. Otherwise, you do hacky “no-op” jobs or you force your entire pipeline to run every time.
In short, GitHub Actions can be powerful for smaller codebases or straightforward pipelines, but if your repo is big and you want advanced control, it starts to feel like you’re fighting the tool. If there’s no sign that GitHub wants to address these issues, it’s totally reasonable to look elsewhere or build your own thin orchestration on top of more flexible CI runners.
I wrote about it in detail https://ashishb.net/tech/common-pitfalls-of-github-actions/ And even created a tool to generate good configs http://github.com/ashishb/gabo
No, applying the logic to something like Lambda would mean implementing handlers like:
function handle(lambdaRequest, lambdaContext) {
return myFunction(
stuffExtractedFromLambdaRequest(lambdaRequest),
stuffExtractedFromLambdaContext(lambdaContext)
);
}
Dynamo isn't the same, since it's just a service/API that we call; we don't implement its logic, like we do for CI tasks, Lambda functions, etc.
Whilst you're right that it's a hassle to run DynamoDB locally (although not too bad, in my experience); that's also not necessary. It's fine to run code locally which talks to a remote DynamoDB; just set up the credentials appropriately.
dynamodb: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
doesn't seem any harder than running any other db
With Woodpecker/Jenkins you know exactly what your pipeline is doing. With GitHub actions, not even the developers of the actions themselves know what the runner does.
What does this even mean? Are you talking about Forgejo Actions, or are you somehow hosting your code on a Forgejo instance but running CI through GitHub?
> With Woodpecker/Jenkins you know exactly what your pipeline is doing.
If you wrote it from the ground up, sure. On the other hand, I've inherited Jenkins pipelines that were written years before I got there and involved three to four different plugins, and they're way worse to work with than the GitHub Actions that I inherited.
yes, Forgejo Actions which is supposed to be a drop in replacement for GH Actions. You can say they're different things but the general idea and level of complexity is the same.
Well that's just someone being a dumbass, since AssumeRoleWithWebIdentity (and its Azure and GCP equivalent) have existed for quite a while. It works flawlessly and if someone does do something stupid like `export HURP_DURP=$AWS_ACCESS_KEY_ID; printenv` in a log, that key is only live for about 15 minutes so the attacker better hurry
Further, at least in AWS and GCP (I haven't tried such a thing in Azure) on can also guard the cred with "if the organization and repo are not ..." then the AssumeRole 403s to ensure that my-awesome-org/junior-dev-test-repo doesn't up and start doing fun prod stuff in GHA
I hate GHA probably more than most, but one can footgun themselves in any setup
I wrote a little about it in this blog post: https://joshstrange.com/2024/04/26/nightly-postgres-backups-...
I guess what I’m saying is, it’s GHA all the way down.
IME, much of the complexity in using Github Actions (or Gitlab CI, or Travis) is around communicating that information to scripts or build tools.
That and running different tasks in parallel, and making sure everything you want passes.
My impression of gitlab CI is that it's also not built for monorepos.
(I'm a casual gitlab CI user).
this makes me feel like you’re really asking “can i split up my gitlab CICD yaml file or does everything need to be in one file”.
if that’s the case:
yes it does eventually all end up in a single pipeline (ignoring child pipelines).
but you can split everything up and then use the `include` statement to pull it all together in one main pipeline file which makes dealing with massive amounts of yaml much easier.
https://docs.gitlab.com/ee/ci/yaml/includes.html
you can also use `include` to pull in a yaml config from another project to add things like SAST on the fly.
previous workplace i had like 4 CICD template repos and constructed all 30 odd actual build repos from those four templates.
used `include` to pull in some yaml template jobs, which i made run when by doing something like (it’s been a while, might get this wrong)
include:
project: 'cicd/templates'
file: 'builds.yml'
stages:
- build
job_a:
stage: build
extends: .job_a_from_template
variables:
IMAGE_NAME: "myimage"
IMAGE_REPO: "somerepo.org"
you can also override stuff like rules on when it should run if you want to. which is handy.
gitlab CICD can be really modular when you get into it.
if that wasn’t the case: on me.
edit: switched to some yaml instead of text which may or may not be wrong. dunno. i have yet to drink coffee.
If the unit tests are slow enough to want to skip them, they likely are not unit tests but some kind of service-level tests or tests that are hitting external APIs or some other source of a bad smell. If the slow thing is the build, then cache the artifact keyed off the directory contents so the step is fast if code is unchanged. If the unit tests only run for a package when the code changes, there is a lack of e2e/integration testing. So, what is OP's testing strategy? Caching? It seems like following good testing practices would make this problem disappear.
For example, tests involving read hardware can only run at 1x speed, so you will want to avoid running those if you can. If you are building a custom compiler toolchain, that is slow and you will want to skip it if the changes cannot possibly affect the toolchain.
definitely interesting!
I do wonder if this really solves the author problem because by the looks of it , you just have to run meta command and it would run over each of the sub directory. While at the same time , I think I like it because this is what I think people refer to as "modular monolith"
Combining this with nats https://nats.io/ (hey if you don't want it to be over the network , you could use nats with the memory model of your application itself to reduce any overhead) and essentially just get yourself a really modular monolith in which you can then seperate things selectively (ahem , microservices) afterwards rather easily.
I'm not sure what NATS has to do with anything in this post or discussion. Also, a modular monolith is almost the antithesis of microservices.
[1]: https://www.thoughtworks.com/en-us/insights/blog/microservic...
Running dotnet test on the other hand with 10k empty tests took 95 seconds.
Honestly, 10k no-op tests should be limited by disk IO, and in an ideal world would be 10 seconds.
If no one else is doing anything about the mess, then it falls to the senior person to sort it out.
As a rule of thumb:
- Ideally your people do the Right Thing by themselves by the magic of 'leadership'. - Second best: you chase the people to do the Right Thing. - Third: you as the senior person do the Right Thing. - Least ideal: no one fixes the mess nor implements the proper way.
I guess some people can achieve the ideal outcome with pure charisma (or fear?) alone, but I find that occasionally getting your hands dirty (option 3) helps earn the respect to make the 'leadership' work. It can also help ground you in the reality of the day to day work.
However, you are right that a senior person shouldn't get bogged down with such work. You need to cut your losses at some point.
* TC offers sh, full stop. If you want to script something that depends on bash, it's a PITA and you end up with a kludge to run bash in sh in docker in docker.
https://www.jetbrains.com/help/teamcity/command-line.html#:~...
Your "docker in docker" comment makes me wonder if you're conflating the image that you are using, that just happens to be run by TeamCity, versus some inherent limitation of TC. I believe a boatload of the Hashicorp images ship with dash, or busybox or worse, and practically anything named "slim" or "alpine" similarly
For me it was an epiphany as software dev - not to write reusable extensible scripts - I am so much more productive after that.
oh god just reading that gave me PTSD flash backs.
At $priorGig there was the "omni-chart". It was a helm chart that was so complex it needed to be wrapped in terraform and used composable terraform modules w/ user var overrides as needed.
Debugging anything about it meant clearing your calendar for the day and probably the following day, too.
The few bits that end up getting reused are the externalized bash scripts.
The ideal for me is (this is very silly and glib and a total category error) LSP but for CI. Tooling that is relatively normalized, letting me (for example) have a pytest plugin that "does sharding" cleanly across multiple CI operators.
There's some stuff and conventions already of course, but in particular caching and spinning up jobs dynamically are still not there.
For one, with GitHub Actions, hosted and self-hosted runners are fundamentally different applications; hosted runners are fully configured container images, (with base OS, tools, etc., on board), whereas self-hosted runners are essentially thin, unconfigured shell scripts. This means that unless you're planning on using self-hosted runners in production (which some do of course, but most don't), it wouldn't make sense to dev/test with them locally, given how different they are. With Buildkite, there's only one "way" -- `buildkite-agent`, the single binary I linked to above.
The connection models are also different. While both GHA self-hosted runners and the Buildkite agent connect to a remote service to claim and run jobs, GHA runners must first be registered with a GitHub org or repository before you can use them, and then workflows must also be configured to use them (e.g., with `runs-on` params). With Buildkite, any `buildkite-agent` with a proper token can connect to a queue to run a job.
There are others, but hopefully that gives you an idea.
CI shines for running tests against a clean environment for example.
Really any task that benefits from a clean image being stood before running a task.
The key though is to decouple the tasks from the CI. Complexity like pre-packaging artifacts is not a great fit for CI configuration, that is best pushed to a tool that doesn’t require waterfall logic to make it work.
There is a reason task runners are very popular still
Looks interesting. We’ve picked tools based on time saved without too much extra knowledge or overhead required, so this may prove promising.
NX is good because it does the caching part of CI in a way that works both locally and on CI. But of course it doesn't really help at all with the other points raised by the OP.
One interesting thing about NX as well is that it helps with you managing your own local build chain, like in the example I mentioned above, when I run a project that requires the C++ native dependency, that project gets built automatically (or rather my computer pulls the built binaries from the remote cache).
But for all of this to work you need to set up these dependency chains explicitly in your NX configuration, but that is formalizing an actual requirement instead of leaving it implicit (or in Makefiles or in scripts that only run in CI).
I do have to say that our NX configuration is quite long though, but I feel that once you start using NX it is just too tempting to split your project up in individual cacheable steps even if said steps are very fast to run and produce no artifacts. Although you don't have to.
For example we have separate steps for linting, typescript type-checking, code formatting, unit testing for each unique project in our mono-repo. In practice they could be all the same step because they all get invalidated at the same time (basically on any file change).
role-session-name: GitHubActionSession
role-session-name: gha-${{ github.run_id }} # or your favorite
include:
project: 'cicd/templates'
file: 'builds.yml'
variables:
IMAGE_NAME: something
IMAGE_REPO: some.org
job_b:
rules:
- when: never
less useful when just splitting things into multiple files to make life simpler.
super useful when using the same templates across multiple independent repositories to make everything build in as close to the same way as possible.
I have had many tests which manipulate the global environment (integration tests should do this, though I'm not convinced the distinction between integration and unit tests is valuable) so the ability run the same tests as CI is very helpful in finding and verifying you fixed these.
The trick is to NOT have your script depend on the specifics of the environment, but reverse the dependency. So replace all `If CI then Run X else if Local Run Y` with the ability to configure the script to run X or Y, then let the CI configure X and local configure Y. For example.
I'm not saying it is always easy and obvious. For bigger builds, you often really want caching and have shitloads of secrets and configurations going on. You want to only build what is needed, so you need something like a DAG. It can get complex fast. The trick is making it only as complex as it needs be, and only as reusable as and when it is actually re-used.
> I'm not saying it is always easy and obvious. For bigger builds, you often really want caching and have shitloads of secrets and configurations going on.
Here's the thing. When you don't want secrets, or caching, or messaging, or conditional deploys, none of this matters. Your build can be make/mvn/go/cargo and it just works, and is easy. It only gets messy when you want to do "detect changes since last run and run tests on those components", or "don't build the moon and the stars, pull that dependency in CI, as local users have it built already and it won't change." And the way to handle those situations involves running different targets/scripts/whatever and not what is actually in your CI environment.
I've lost count of how many deploys have been marked as failed in my career because the shell script for posting updates to slack has an error, and that's not used in the local runs of CI.
What you _actually_ need is a staging branch of your CI and a dry-run flag on all of your state modifying commands. Then, none of this matters.
Automation renders knowledge into a set of executable steps, which is much better than rendering knowledge into documentation, or leaving it to rot in people's minds. Compiling all rendered knowledge into a single step is the easiest way to ensure all elements around the build and deployment lifecycle work in unison and are guarded around failures.
The only difference is that you are now tied to a vendor for executing that logic, and the issue is really that this tooling is proprietary software (otherwise, you could just take their theoretical open source runner system and run it locally).
To me, this is mostly a question of using non-open-source development tools or not.
To be fair, testing actions with side effects on the wider world is terrifying even if you’re running it locally, maybe more so because your nonstandard local environment may have surprises (e.g. an env var you set then forgot) while the remote environment mostly only has stuff you set/installed explicitly, and you can be sloppier (e.g. accidentally running ./deploy when you wanted to run ./test). That part isn’t a GH Actions problem.
Without tooling like this any sufficiently complex system is guaranteed to evolve into a spaghetti mess, because no sane way exists to maintain such a system at scale without proper tooling, which one would need to hand roll themselves against a giant, ever changing mostly undocumented black box proprietary system (GitHub Actions). Someone tried to do this, the project is called “act”. The results are described by the author in the article as “subpar”.
The only sane way to use GitHub Actions at scale is to take the subset of its features that you leverage to perform the execution (event triggers, runs-on, etc) and only use those features, and farm out all the rest of the work in something that is actually maintainable eg Buildkit, Bazel, Gradle etc
Ironically, Jenkins does this one correctly, they just give you a regular programming language. It's a shame there are so many other pitfalls, though.
It's like the circle of life for CI systems.
That's generally a bit less convenient, ie it takes a few more lines, but it has significantly fewer sharp edges than your typical shell script. And more people have written Python scripts, I guess?
Python’s subprocess has communicate(), check_output() and other helpers which takes care of a lot but (a) you need to know what method you should actually call and (b) if you need to do something outside of that, you have to use Popen directly and it’s much more work than just writing a shell script. All doable if you understand pipes and all that but if you don’t, you’ll be just throwing stuff at the wall until something sticks.
Often the shell script just calls a python/Ruby/rust exec anyways...
Shell scripts are for scripting...the shell. Nothing else.
That day, they very effectively communicated that they had decided they were only interested in serving Enterprises, and everything about their product has predictably degraded ever since, to the point where now they're now branding themselves "the most comprehensive AI-powered DevSecOps Platform" with a straight face.
They have since had other also severe CVEs. That has made me feel pretty confident in my decision.
The only difficult part for the attacker was finding an email address that was used by the target; though thats hsually the same as you use for git commits; and gitlab “handily” has an email address assigned to each user-id incrementing from 1;
Usually low numbers are admins, so, a pretty big attack vector when combined.
I wish Microsoft would just announce AzDO's time of death and save companies increasingly crazy AzDO blinders and/or weird mixes of GitHub and AzDO as GitHub-only is clearly the present/future.
Keeping some separation between AzDo itself and GH also requires some balancing. But so far I’m pretty sure I could never sell our enterprise on a shift to GH. Simply not enough jira-esque features in GH with complex workflows, time reporting etc so I can’t see them doing the bigger GH/AzDo merger.
Your average person is going to be blindsighted.
Especially for Github Actions, which is stateless. If you want to reuse computation within their VMs (i.e. not do a fresh build / test / whatever), you can't rely on Just or Make
A problem with Make is that it literally shells out, and the syntax collides. For example, the PID in Make is $$$$, because it's $$ in shell, and then you have to escape $ as $$ with Make.
I believe Just has similar syntax collisions. It's fine for simple things, but when it gets complex, now you have {{ just vars }} as well as $shell_vars.
It's simpler to "just" use shell vars, and to "just" use shell.
Shell already has a lot of footguns, and both Just and Make only add to that, because they add their own syntax on top, while also depending on shell.
It is so much better to run scripts with Just than it is doing it with Make. And although I frankly tend to prefer using a bash script directly (much as described by the parent commenter), Just is much less terrible than Make.
Now the only problem is convincing teams to stop following the Make dogma, because it is so massively ingrained and it has so many probems and weirdnesses that just don't add anything if you just want a command executor.
The PHONY stuff, the variable scaping, the every-line-is-a-separate-shell, and just a lot of stuff that don't help at all.
Make targets look suspiciously like functions (or procedures), but they actually aren't.
Here is an example of a "complex" Makefile I use to help manage Helm deployments (https://github.com/peterwwillis/devops-infrastructure/blob/m...). It uses canned recipes, functions (for loops), default targets, it includes targets and variables from other Makefiles, conditionally crafts argument lists, and more. (It's intended to be a "default" Makefile that is overridden by an additional Makefile.inc file)
I could absolutely rewrite that in a shell script, but I would need to add a ton of additional code to match the existing functionality. Lines of code (and complexity) correlates to bugs, so fewer lines of code = less bugs, so it's easier to maintain, even considering the Make-specific knowledge required.
They say "use the best tool for the job". As far as I've found, for a job like that, Make fits the best. If some day somebody completely re-writes all the functionality of Make in a less-obnoxious way, I'll use that.
They are still phony at heart in this case, even if you don't declare them .PHONY.
Make really wants to produce files; if your targets don't produce the files they are named for you are going to run into trouble (or have to be rather careful to avoid the sharp edges).
> They say "use the best tool for the job". As far as I've found, for a job like that, Make fits the best. If some day somebody completely re-writes all the functionality of Make in a less-obnoxious way, I'll use that.
You could try eg Shake (https://shakebuild.com/), but it requires some Haskell.
Good facilities for arguments, builtins like absolute locations -- there are a ton of benefits (see the README).
the history problem goes away if you always enforce squash merge...
$ hg init
$ hg commit -m"initial"
nothing changed hg ci -m "empty commit" --config ui.allowemptycommit=1
I definitely get the backwards compatibility thing and "don't break someone's automation", but at the same time, Microsoft could at least mark AzDO's official Roadmap as "Maintenance Only" and send the message that feels obvious as a user that GitHub is getting far more attention than AzDO can, but is hard to convince management and infosec that a move to GitHub is not just "the future" but "the present" (and also maybe "the past", now, given AzDO seems to have been frozen ~2020).
I've rarely seen a feedback loop with containers that's not longer than 10s only due to containerization itself, and that breaks the "golden" 10s rule (see https://www.nngroup.com/articles/response-times-3-important-...).
If you aim for quicker turn-around (eg. just running a single test in <1s), you'll have to either aggressively optimize containers (which is pretty non-idiomatic with Docker containers in particular), or do away with them.
Sounds like a skill issue tbh.
`time podman run —-rm -it fedora:latest echo hello` will return in a few milliseconds, whatever delay you are complaining about would be from the application running in the container.
Lastly containers != docker.
Why would it be slow?
It needs to be rebuilt? (on a fast moving project with mid-sized or large team, you'll get dependency or Dockerfile changes frequently)
It needs to restart a bunch of dependent services?
Container itself is slow to initialize?
Caching of Docker layers is tricky, silly (you re-arrange a single command line and poof, it's invalidated, including all the layers after) and hard to make the most of.
If you can't get a single test running in <1s, you are never going to get a full test suite running in a couple of seconds, and never be able to do an urgent deploy in <30s.
1. Image tags ("latest", etc.) can change over time. Does any layer in your Dockerfile -- including inside transitive deps -- build on an existing layer identified by tag? If so, you never had reproducibility.
2. Plenty of Dockerfiles include things like "apt-get some-tool" or its moral equivalent, which will pull down whatever is the latest version of that tool.
It's currently common and considered normal to use these "features". Until that changes, Docker mostly adds only the impression of reproducibility, but genuine weight and pain.
Or write your stuff in eg Python in the first place. GitHub's CI yaml supports scripts in arbitrary languages, not just shell.
Less snarky: https://learn.microsoft.com/en-us/virtualization/windowscont... seems to be a thing?
And in any case, you can use VMs instead of containers.
I've had the unfortunate pleasure of working with Windows Containers in the past. They are Containers in the sense that you can write a dockerfile for them, and they're somewhat isolated, but they're no better than a VM in my experience.
> And in any case, you can use VMs instead of containers.
They're not the same thing. If that's the case running linux AMI's on EC2 should be the same as containers.
It doesn’t really matter if you have to spin up a Linux instances and then run your build environment as a container there vs spinning up a windows VM.
What in particular did you find "bad" with Python tech stack?
(I've got my gripes with Python and the tooling, but it's not this — I've got bigger gripes with containers ;-))
Stateful virtualenvs with no way to check if they're clean (or undo mistakes), no locking of version resolution (much less deterministic resolution), only one-way pip freeze that only works for leaf projects (and poorly even then), no consistency/standards about how the project management works or even basic things like the directory layout, no structured unit tests, no way to manage any of this stuff because all the python tooling is written in python so it needs a python environment to run so even if you try to isolate pieces you always have bootstrap problems... and most frustrating of all, a community that's ok with all this and tries to gaslight you that the problems aren't actually problems.
With docker containers, you can shell into it, do a couple of changes and "docker commit" it afterwards: similarly stateful, right? You resolve both by recreating them from scratch (and you could easily chmod -w the entire virtualenv directory if you don't want it to change accidentally).
The pattern of using requirements.txt.in and pip-freeze generated requirements.txt has been around for a looong time, so it sounds like non-idiomatic way to use pip if you've got problems with locking of versions or non-leaf projects.
As for directory layout, it's pretty clear it's guided by Python import rules: those are tricky, but once you figure them out, you know what you can and should do.
Can you clarify what do you mean with "structured unit tests"? Python does not really limit you in how you organize them, so I am really curious.
Sure, a bootstrapping problem does exist, but rarely do you need exactly a particular version of Python and any of the dev tools to be able to get a virtualenv off the ground, after which you can easily control all the deps in them (again a requirements-dev.txt.in + requirements-dev.txt pattern will help you).
And there's a bunch of new dev tools springing up recently that are written in Rust for Python, so even that points at a community that constantly works to improve the situation.
I am sorry that you see this as "gaslighting" instead of an opportunity to learn why someone did not have the same negative experience.
Also in many projects there’s things other than code that need to be “built” (assets, textures, translations, etc). Adding custom build targets to maven’s build.xml is truly not ideal then there’s people who actually try to write logic in there. That’s objectively worse than the YAML hell we were complaining about at the top of the thread.
Like I said, any version from the last, like, 10+ years (Java and Maven are both serious about backward compatibility), and if you install an ancient version you at least get fail-fast with a reasonable error.
> we really should be using gradle by now
We really shouldn't.
> Adding custom build targets to maven’s build.xml is truly not ideal then there’s people who actually try to write logic in there.
Maven doesn't have a build.xml, are you thinking of ant? With maven you write your custom build steps as build plugins, and they're written in plain old Java (or Kotlin, or Scala, or...) code, as plain old Maven modules, with the same kind of ordinary unit testing as your regular code; all your usual code standards apply (e.g. if you want to check your test coverage, you do it the same way as for your regular code - indeed, probably the same configuration you set up for your regular code is already getting applied). That's a lot better than YAML.
I guess theoretically you could, but I don't think that's part of anyone's normal workflow. Whereas it's extremely easy to run "pip install" from project A's directory with project B's virtualenv active (or vice versa). You might not even notice you've done it.
> You resolve both by recreating them from scratch
But with Docker you can wipe the container and start again from the image, which is fixed. You don't have to re-run the Dockerfile and potentially end up with different versions of everything, which is what you have to do with virtualenv (you run pip install and get something completely different from the virtualenv you deleted).
> you could easily chmod -w the entire virtualenv directory if you don't want it to change accidentally
But you have to undo it every time you want to add or update a dependency. In other ecosystems it's easy to keep my dependencies in line with what's in the equivalent of requirements.txt, but hard to install some random other unmanaged dependency. In the best ecosystems there's no need to "install" your dependencies at all, you just always have exactly the packages listed in the requirements.txt equivalent available at runtime when you run things.
> The pattern of using requirements.txt.in and pip-freeze generated requirements.txt has been around for a looong time, so it sounds like non-idiomatic way to use pip if you've got problems with locking of versions or non-leaf projects.
I've literally never seen a project that does that. And even if you do that, it's still harder to work with because you can't upgrade one dependency without unlocking all of your dependencies, right?
> As for directory layout, it's pretty clear it's guided by Python import rules
I don't mean within my actual code, I mean like: where does source code go, where does test code go, where do non-code assets go.
> Can you clarify what do you mean with "structured unit tests"?
I mean, like, if I'm at looking at a particular module in the source code, where do I go to find the tests for that module? Where's the test-support code as distinct from the specific tests?
> rarely do you need exactly a particular version of Python and any of the dev tools to be able to get a virtualenv off the ground
Whether virtualenv is available is a relatively recent change, so you already have a fractal problem. Having an uncontrolled way of installing your build environment is another of those things that's fine until it isn't.
> And there's a bunch of new dev tools springing up recently that are written in Rust for Python
Yeah, that's the one thing that gives me some hope that there might be light at the end of the tunnel, since I hear they mostly ignore all this idiocy (and avoid e.g. having user-facing virtualenvs at all) and just do the right thing. Hopefully once they catch on we'll see Python start to be ok without containers too and maybe the container hype will die down. But it's certainly not the case that everything has been fine since 2007; quite the opposite.
Sure, if I try to rebuild a docker image from 5 years ago, it may fail, or produce something different, because it was pulling in some packages that have changed significantly in apt, or perhaps pip has changed encryption and no longer accepts TLS1.1 or whatever. And if you use ':latest' or if your team has a habit of reusing build numbers or custom tags, you may break the assumptions even sooner.
But even so, especially if using always incremented build numbers as image tags, a docker image will work the same way om the Jenkins system, the GitHub Actions pipeline, and every coworker's local build, for a while.
You can use specific image hashes to work around image tag changes.
For problem 2 ideally use something like NixOS as a base or at least nix package manager.
But IMO Nix is quite complicated, even though the working model is really good. So this is mostly if you really need deterministic builds.
Docker already gets you 80% of the way with 20% of the effort.
But really containerizing the build environment is not related to deterministic builds as there’s a lot more work needed to guarantee that. Including possible changes in the application itself.
Lastly you don’t need to use docker or dockerfiles to build containers. You can use whatever tooling you want to create a rootfs and then create an image out of that. A nifty way of actually guaranteeing reproducibility is to use nix to build a container image.
What would you say the goal of containerising builds is, if not reproducibility?
If you use a container with the same apt-get commands and the same OS, you already separate out almost all the reproducibility issues. What you get will be a very similar environment.
But it's not deterministic as you point out, that takes a lot more effort.