Parallel development without the headaches using Git worktree

Parallel development without the headaches using Git worktree(barrd.dev)

59 points by oogali 42 days ago | 49 comments

zmmmmm 42 days ago |

The real problem for parallel dev is ensuring parallel dev environments can seamlessly co-exist without treading on each other. As soon as one of them wants to open a port, talk to an external database or write to a shared location as part of testing you have them conflicting with each other.

Much of this is a legacy dev issue where there was never so much of an assumption that parallel ephemeral dev environments would be in play in the first place. But legacy dev is still most of dev.

SimionBaws 40 days ago | |

Easy: each worktree starts services on new ports. https://termic.dev goves you ENV variables for this, oir you definee your own custom named variables, and termic assigns ports for each wotktree.

Or if the stack is too complex to start it in parallel, use spotlight to sync changes to main checkout from a single worktree at a time. You can only test one at a time, let's be serios. https://termic.dev/docs/spotlight/

lefra 42 days ago | |

To avoid that, the code should have ways to override default ports or paths for testing. Then, when testing, ask the OS for a free port or temporary file, and tell the tested code to use that instead of its default. This also allows running tests concurrently.

In what case would talking to an external DB be a problem? Databases support parallel queries.

pprotas 41 days ago | | |

On your last point, imagine worktree A talks to a database assuming some specific schema. If worktree B changes the schema (let’s day, we remove a column from a table), then worktree A can run into some errors.

mock-possum 41 days ago | |

I just have a handful of ‘hot desks’ I assign different agents to - each desk is numbered, and the numbers inform the ports. So my first agent (001, because it’s an agent after all) uses 8081 for the web server, 3001 for the database, 8001 for the api service, etc - and 002 would use —2 for all the same.

Nice and neat, nobody’s toes get stepped on.

mkirsten 41 days ago |

My mind inserted an 'of' in the title. I thought the article was about doing parallel development without using Git worktrees. Prior to AI, I never used worktrees. Now I use them all the time. I think it's both due to that AI agents are well suited to do parallel development, but also due to that Claude helps me avoiding the headaches of using Git worktrees

therealmarv 42 days ago |

Maybe I'm stubborn, but even in the age of AI, I still use multiple git clones/directories of the same project, e.g.:

  ~/dev/projectx
  ~/dev/projectx2
  ~/dev/projectx3
  ~/dev/projectx4

very rarely use more than 4–5 per project. Maybe I'm just avoiding wrapping my head around worktrees and actually trying them out.

Benefits: These clones act as semi-permanent directories:

- Helps with caching for heavy Docker usage (think of repeated parallel unit, e2e tests)

- I've color-coded my terminal tabs for each clone so I can instantly tell where I am at a glance (kinda like tab groups just with colors)

Maybe if for some reason I need double digits clones of a project I will be more forced to use git worktrees because then it will be annoying to remember in which directory a branch clone lives.

agrm 42 days ago | |

this is basically what worktrees do, though? except git 'knows' about all the associated branches, so you don't have to deal with pushing/pulling in order to merge xyz branch, etc.

fireant 41 days ago | | |

The main issue with worktrees is that you can't have the same branch opened in multiple worktrees which makes many things harder than the ought to be. I don't really understand why we need that limitation, otherwise the tooling is good enough.

fergie 41 days ago | | |

The article said that a "limitation" of worktrees is that a branch can only be checked out in one worktree at a time- however, this seems like a feature, because if you could have several worktrees each with their own overlapping sets of branches then surely that would be equivalent to many git clones?

But yes, personally, in my ignorance, I shamelessly git clone the same repo many times.

ianmarcinkowski 41 days ago | |

I'm also stubborn. I clone multiple copies of the repo. I'm currently at two at my job right now, because I was trying to vibe-code an E2E test suite.

I'm probably missing something that people see as a huge advantage of having multiple streams of work large enough to require a long-running branch. I just finish working on the current branch and then switching to another branch. If work can't be accomplished and merged into `main` within a small amount of time, I'm almost certainly doing something wrong.

irskep 42 days ago |

I use git worktrees daily. I'm surprised by how hard it can be to explain them to people who have never used them. Lately I've settled on "like clones, but sharing a .git directory." The article tries to get this across by comparing them to branches, but I think clones are a more intuitive concept to compare against.

To solve some of the ergonomics issues (command verbosity, manual commands to copy over .env files and install dependencies), I wrote autowt, which is a lightweight but powerful worktree manager: https://steveasleep.com/autowt/

Once I dialed in the cli experience, I basically stopped typing 'git checkout <branch>' to change tasks, because it's easier to ignore working directory state when flipping between tasks.

rhyperior 42 days ago | |

I just ran into a situation where I had two Claude instances working in the same repo on two different, independent issues. I noticed the sessions were thrashing a bit on which branch they were on because each was taking control from the other session. That’s when it hit me why I need worktrees.

anon7000 41 days ago | | |

And depending how much you let Claude do, you can trivially say “in a new worktree, $prompt” and it’ll happily set it up and use it

esposito 42 days ago |

One struggle I've had when working with worktrees is not a problem with the worktrees themselves but instead how my Intellij manages project configurations. I can't for the life of me get it set up so my configuration can sit relative to multiple worktrees. There's probably a plugin which solves the issue, but at least in my experience the .idea config folder is rooted in its current directory and very difficult to migrate to another folder or worktree. I end up having to re-establish project linting rules, run configurations, what have you, for each new worktree and it becomes too much of a hassle for anything short-lived.

sa46 42 days ago | |

IntelliJ added worktree “support” but it’s really just a way to create a new project.

I suspect it’ll take a while before we see a shared configuration across worktrees because the .idea folder contains mutable state.

https://www.jetbrains.com/help/idea/use-git-worktrees.html

Klaster_1 41 days ago | |

I had a related issue in VSCode - my project definition is stored in the main checkout and is git ignored. When I WT the repo and open an instance of VSCode for it, no project is attached to it, breaking the dev env (scripts, settings, etc.). What worked for me was a VSCode WT extension that allows you to run arbitrary hooks before and after WT checkout. In checkout one, I symlink the VSCode project. This solves the issue and as a bonus, and project changes propagate between all open instances. The only drawback is that VSCode chats/extensions are path scoped, so you lose conversation history and per project extension on/off state that's stored in the VSCode global DB.

thunderbong 42 days ago |

Fossil SCM, conventionally, has been thus way by default.

https://fossil-scm.org/home/doc/trunk/www/gitusers.md

pkghost 42 days ago |

Having gone down the worktree rabbit hole for a month or so, I am giving up in favor of multiple checkouts to enable many agents to work across many repos.

Worktrees worked great for me when my agents' work was mostly contained to a single repo (they got me to finally hitting rate limits, not that that was a goal). As my homelab scales, agents increasingly work across multiple repos, and that's where (my approach to) worktrees broke down; agents were spawned in a repo worktree and so avoided stepping on the toes of other agents in the same repo without any special instruction, but as soon as they needed to touch another repo, they would default to working in that repo directly, without a worktree, and thus collide with other agents, and muddy a merge process that expected the main repo clone to be clean (which turns out to have been an unnecessary design quirk, but resolving it would still not stop agents from stepping on each other's toes in secondary repos).

After a detour through ZFS dataset clones and some mounting magic that made /srv/src/ appear to be a distinct hierarchy for each agent process, I am simplifying even further and giving each agent a bare directory (sth like /srv/dev/<slug>/) where they can check out any repos they want. The git remote (/srv/git/) becomes the only integration point.

Maybe I should just bite the bullet and move into containers, but performant as they are the ergonomics still bum me out.

Edit: I may actually hold on to ZFS datasets with mounting magic. It does add a bit of complexity to dev tools, but it also reduces path-based trust bloat in harness configs, provides some interesting zfs features (zfs diff, etc)... And it's done.

2pigeons 38 days ago | |

Maybe it's simple, but I just put the agents in a chat room and have them talk to each other.

tlarkworthy 41 days ago |

I use a top level meta-repository to create a virtual monorepo of all my repositories and vendor their upstreams, then use worktrees off off the submodules to prepare patches. I never need to change directory from the root meta-repository even with multiple agents. Gives you a full monorepo experience without actually having to own any of the parts, every agent has its own isolated worktree so does not conflict.

Klaster_1 41 days ago | |

Doesn't that mean that if you need to work on several repos for a feature at the same time, you'd have to WT each submodule compared to a single WT for a monorepo? Do you use a custom orchestration layer to fire all commands or lets agents manage the WTs?

tlarkworthy 40 days ago | | |

Yeah, but actually the agent recreates the folder structure, so work trees/task1 subfolder contains a sparse subset of work trees for task1 if it crosses repos. Yeah I have a skill for creating the isolated worktrees. The agents manage all this complexity and the complexity is isolated to the meta repository, it does not spill into the real code repos

Vanit 42 days ago |

The one real caveat to this is that if your application has particularly heavy cold starts it can be hard to actually run your worktrees. I usually end up swapping to a regular checkout as each worktree is due for hands-on attention.

lacunary 42 days ago | |

I'm curious what the constraint preventing you from optimizing the cold start time - can you share more?

thejokeisonme 41 days ago | |

Same!

hankbond 42 days ago |

Dang I read this as

> Parallel development without the headaches from using Git worktree

Read the whole thing just waiting for the complaints to start!

w4lker 41 days ago | |

Same

on_the_train 42 days ago |

There's a huge limitation with work trees that makes them borderline unusable: they can never be active on the same branch. Which includes that you can only ever have one on the master branch.

modeless 42 days ago | |

Yeah they aren't worth the trouble just because of this. Multiple checkouts are the way to go unless your repo is so huge that you can't afford the disk space. You can clone them from each other so you don't need to download the repo over and over and you can send branches back and forth locally if you really need to, though I never do.

skew-aberration 42 days ago |

Working on an embedded device that requires building firmware images, etc, I've taken to using docker containers with a COW overlay. I have a vibe-coded tool which checks out the latest and does a clean compile, creating a base directory. Then I spawn a build container on top of that. Every build is warm, as they share ccache and page cache. Huge speed up for parallel builds and I never OOM. Every build also sees the same directory structure, simplifying instructions, etc for the agents.

pydry 42 days ago |

I do the same thing just checking out the repo multiple times in different directories.

I'm not really sure what worktrees adds on top of that.

MaulingMonkey 42 days ago | |

Worktrees share more git state (remotes, blobs, etc.) and don't require hunting down the repository url to feed to git clone, or a network connection to use said url. While you could get the same benefit from cloning your local repository, you're then in the weird state where `origin` is a local non-bare repository.

If you're setting up long-lived checkouts that you reuse, it doesn't help much (except perhaps saving space or bandwidth) vs multiple clones and some once up-front reconfiguration. On the other hand, if you want something more temporary - and perhaps based on your current HEAD without having to hunt down a commit id to feed a subsequent git clone / git checkout command - worktrees save you some boilerplate (re)configuration.

pydry 41 days ago | | |

if it's a my main repo where im regularly doing hotfixes in one folder while i work simultaneously on a feature branch on another I will have multiple long lived checkouts (usually 4).

it makes sense to keep them warm because while cloning a new repo is almost always quick, regenerating the build artefacts from scratch each time is not.

recursivecaveat 42 days ago | |

It makes coordination easier. For eg if you want to fetch and pull master, you only have to do that once. If you want to rebase checkout A's branch on top of B's, you can just do that, you don't have to push between the two.

esafak 42 days ago | |

Less space.

ianmarcinkowski 41 days ago |

My ai-maximalist coworker spent a day last week getting Claude setup with work trees. He asked me how I did it and I said "I just clone the repo a second time and run claude's CLI in a new terminal window."

His mind was blown that there was .... a simple way ........ to ..... do this...?

Is there a personality type that's like "i will invent a new way of not needing to use my hands to drink soup directly from the hot pot on the stove!!!" ... "you mean a spoon?" .... "yes.... a spoon... how quaint!"

kajika91 42 days ago |

This is very interesting and could be very helpful for A/B testing or comparing outputs/speed between 2 branches.

I already know a place I want to use this.

jemmyw 42 days ago |

wt is a useful little tool for helping with git worktrees: https://github.com/timvw/wt

I use it's config to copy in some non-repo config and then hooks to start an agent doing analysis.

matltc 42 days ago |

git worktree add -b some-branch ../wts/repo-some-branch origin/main

git worktree list

git worktree remove

All you need

jmpeax 42 days ago |

What a peculiar title. The title suggest parallel dev without using worktrees, but the article is about using worktrees.

michaelsmanley 42 days ago | |

I thought that as well and then realized I had inserted the word "of" before "using" where there's an implied "by."

psanchez 42 days ago |

In the same line as therealmarv's comment, I think I prefer working with agents and multiple clones rather than with worktrees so far. Probably the main reason is that it has better isolation and I don't need to copy non-committed files (e.g config) over and over. I can just keep working with agents on the very same folders, even when I need a new feature.

I think there are some pros/cons to each approach and would depend on each person's individual workflow and project(s) more than anything.

In my case, I've been working on a monorepo the past weeks, and this is the setup that I've found useful:

  - project/
  - project2/
  - project3/
  - project4/

You can easily spawn (or even recreate) a new folder by doing:

  $ cp -r project project5

All my non-committed config files were copied from project/ to the other folders.

Before I had this setup, I was working with just 1 docker-compose.yml file (e.g. docker compose watch), however, when working with multiple folders and agents in the same computer at the same time, this approach stopped being useful.

Since my local setup for this project had a db+backend+frontend, I instructed AI to create a command to just launch variants of my local setup but using different ports to avoid conflicts with the other copies of my project.

  $ ./build.sh --watch 3

This number, I use it as a sort of prefix on the existing exposed ports in my local machine, to help me mentally map ports easily. For example, for 3:

  - 3080    (http)
  - 3443    (https)
  - 33306   (mysql)
  - ...
  - 35432   (postgres)

The reason I'm using it as a prefix is convenience, it's very easy to remember which port should I open for each of the clones.

I found using 3 to 4 parallel clones/AI workers to be my sweet spot in terms of being able to manage the development of different features in parallel and not lose track of what's where, while getting the most of my AI subscriptions. I usually keep some notes in a piece of paper like this:

  1) feature blah
  2) bugfixing of blah
  3) research blah
  4) bugfixing of blah blah

Then if I'm reviewing the work of AI on 4, I know I can just go to the browser to 4443 and do some manual testing if I need to (of course, there are unit / end2end tests, but that does not prevent me from doublechecking and ensuring what I had in my mind got translated in code to the actual thing I wanted)

When I finish with one of the features/bugfixes I'm working on, I push changes, pull master repo again, and start another branch & feature. I keep using that same folder and ports.

Another thing I found to be very useful is to set up these local servers with the fixtures I use for the end2end tests, and that way I have a common setup with the same data to do my manual tests. I only use one command to set everything up (the one above) since I don't want to remember nor type a bunch of commands every time.

Sometimes I don't even use clones/worktrees at all, but just work on multiple projects in parallel (again, 3 being the sweet spot, 4 is already challenging to keep track of what I'm doing, and I would probably only go 4 or 5 if I'm doing really long tasks from 1 to 3). BTW having fully autonomous agents working in parallel and doing PRs automatically is totally separate from this local workflow I'm describing, both things can coexist.

Just as a side-note, if agents were 10x-100x faster (at current Sol/Opus5 level) I would probably work only on a single repo/task at a time in my local computer. The reason I multitask today is just because waiting for the models takes minutes and not seconds.

Edit: minimal changes/formatting for clarity

whateveracct 41 days ago |

use git prole