Steve's Jujutsu Tutorial(steveklabnik.github.io) |
Steve's Jujutsu Tutorial(steveklabnik.github.io) |
The ability to easily work on top of an octopus merge and then push changes "down" into the contributing branches has been a live saver when my team had to do a big refactoring in a mono repo and split the changes into small PRs for individual teams (code owners).
The auto committing behavior is a bit weird at first, but now I don't want to go back to git. It feels a bit like the step from SVN to git back in the the day. ("this feels weird" -> "how did people ever tolerate the old way?")
I do wish there was a better front end for it though.
One of those for me was branch names that don’t automatically “follow” new commits. At first it felt weird but it unlocks the ability to do consecutive work as one linear set of changes, even when those changes need to be merged in discrete chunks. The git approach for this (stacking branches) is so painful, particularly when you need to edit an earlier change or add a new commit between earlier ones. This went from being so frustratingly difficult I wouldn’t even consider it to being utterly trivial.
Also rebase conflicts. Not being unceremoniously dropped into a half-broken “fix this NOW state” with no ability to inspect and poke at other commits in the chain and not being able to fix things incrementally in any order is something I couldn’t have imagined. And like you said now it’s insane to me that people continue to put up with it.
1. I am going to work on $X
2. autocommit
3. My work on $X is done
rather than
1. I make changes
2. I am done making changes
3. Now I have to describe what I changed and how
Maybe this is just me, but with git it is at times hard at times to hit the right balance in terms of commit granularity — and for my flow planning forward ("I am gonna do $X") rather than describing what I did ("I did $X") seems more.. focused?
I am a bit skeptical about this, because this requires a jj daemon?
But you can integrate with https://github.com/facebook/watchman/ in order to have a truly daemon-ified option where any filesystem write will cause a snapshot to be taken.
I haven't checked the source.
I wonder what it is about descriptions of stacked diffs that doesn't land - it's literally just a rebase-centric workflow instead of the merge-centric workflow popularised by GitHub et al.
This is why jj is on my todo list. I’m not calling it jujutsu no matter how much someone pays me though.
IMO, Gerrit is the best currently available option by a large margin, notwithstanding its quirks.
And yeah, the lack of good review tooling is certainly a big issue.
Zellij looks powerful but also a bit too complex, following the "kitchen sink" school of design :-). No biggie but its name is too close to IntelliJ imho. What kind of workflow do you use with it?
I used tmux a bit back in the day, but these days I feel like good old tabs and app windows cover my needs. When I want to multiplex processes in a single window, I reach for Overmind. [1]
--
Is there more advanced stuff that's more complex that I just haven't seen?
> Helix, the editor? How would you pitch it?
I’m especially interested after learning about the git compatible backend:
> There's one other reason you should be interested in giving jj a try: it has a git compatible backend, and so you can use jj on your own, without anyone else you're working with to convert too.
I even started writing one but that was a pretty big project and I lost the motivation for it.
From a tiny bit of previous jj experience, my mental model is "a commit is the snapshot, and a change is what happened between snapshots", but that might be wrong. It would be great if this could be clarified a bit more in the tutorial.
> The command-line tool is called jj for now because it's easy to type and easy to replace (rare in English). The project is called "Jujutsu" because it matches "jj".
What caught my interest:
* Separate operations and data
* Partial repos = no big repo issues, no need for shallow clones and such
* Proper and easy merging with no shuffled lines
* Patch-based model is much more intuitive (e.g. rebase and merge are the same operation)
* Conflict resolutions are stored and can be reapplied!
Just want to point out that this hasn’t been updated in a minute, and in particular, you’ll get some messages about branches being bookmarks now: https://github.com/steveklabnik/jujutsu-tutorial/pull/34
I have started on a second iteration of the tutorial in private, and am gonna see if I can get it in shape this weekend.
Happy to answer questions about jj!
English is so great and so confusing!
I swear the modern programmer doesn’t realize how extremely bad Git is. It does do a lot of things better than SVN. But it’s a long, long ways from “good” imho.
I blame GitHub. Git didn’t win because it’s good. Git won because GitHub won. If only HgHub had won instead, alas.
My dream VCS system would have a virtual file system, copy-on-write storage, and a system wide blob cache. The goal being to allow open source repos to commit *ALL* their dependencies, up to and including toolchains in many cases.
I don't particularly like git and for personal projects use fossil instead.
Without going through the whole tutorial, and doing a lot more reading, why should I consider using this over fossil?
What I care about is the tooling around it: GitHub and its ecosystem mainly. I also want my open source projects to be on GitHub specifically, and I don’t want to ask contributors to use something other than Git.
I've never really messed around with VCSes other than git and Subversion, there's people who really like Mercurial though, so I wonder how jj compares to that.
I thought about it and I don't know what a better name would be. Off the top of my Head, I know Perforce, BitWarden, Subversion, fossil and git. And then the abbreviations CVS, RCS and SVN.
Do any of these qualify as a descriptive name?
I recommend looking up Bartitsu (that Conan Doyle spelled Baritsu), a short-lived but very interesting martial art.
What you do is, you treat @ like the index, and you work on @-. This is the "squash workflow" https://steveklabnik.github.io/jujutsu-tutorial/real-world-w...
With some respect, I think “rebase centric workflow” doesn’t really cover it: I use rebasing heavily with GitHub. A “trunk based development where all branches are rebased before merge so there’s never merge commits, please add to commits and rebase the branches in response to review comments” development style is still very rebase centric, but not stacked.
You also have to remember (though there’s no reason you should know this) that GitHub came out of the community I was heavily involved in at the time of its birth. I’ve been using GitHub for longer than most people, and so the PR-style workflow has been the air I’ve breathed for so long, it can be hard to understand other things at first. I had used subversion before git, but didn’t really understand much about it.
Anyway, that’s just a roundabout way of saying that I think this space is super interesting because these tools are so flexible that they can be used in so many different ways, and it’s easy to assume that “how you use git” is some sort of shared experience when that’s not really true.
refactor/a # should be merged as independent PR
feature/a-b # depends on a; should be merged independently, after
fix/c # independent
How else than "stacked diffs" would you submit all of this for review concurrently while still working on all of them? It sounds like a new word for what we were always expected to do? What's an alternative which doesn't rely on compromising merge-strategy?
At this point I'm suspecting "stacked diffs" is just a crotch for people still barely coping with DVCS but I could be missing something.
Meta runs a custom mercurial-compatible monorepo that was scaled up to handle thousands of commits per hour, and they have a culture of constant codebase-wide refactoring commits (i.e. the ongoing live migration of PHP -> Hack) - that rate of change causes problems that most GitHub projects have no reason to solve.
Git and Github do not support this workflow directly, and this creates space for all the other tools.
Jujutsu comes close to supporting this workflow directly, without having to provide any specialized utilities that mention "stacked diffs" at all.
https://github.com/spacedentist/spr (not to be confused with ejoffe/spr) is a utility that maps a local change to a remote branch, updating the branch (without force pushes) when the local change is updated. (Changes work the same as they do in JJ, but the implementation is different.) And this works even when you have a stack of changes and update one in the middle of the stack.
As you iterate on one change, new underlying git commits are made to snapshot the latest state of the current change being edited. The change ID remains stable though the backing commit ID varies as the change’s contents are modified.
You can always revert a change back to a previous backing commit.
This has a few cool implications.
First, you never have uncommitted changes. You are always editing a “current” change with an ID that is regularly persisting your work (either every time you run a jj command or with a filesystem event monitor).
Second, you’re never “on” a named branch: you’re only ever “on” a change and named branches are just mutable pointers to change IDs. This is awesome for a long-lived linear bit of work that needs to be merged in piecemeal. In git you’d need multiple branches based on one another and nothing tracks the relationship between them. Updating earlier work is painful. With jj, it’s all just one linear branch with some changes having names. If you need to edit an earlier change or insert new ones, you just… do that.
I should probably use jj at this stage and figure all this out for myself, rather than just read about it. Thank you!
Your mental model sounds decent. I think there’s a few ways to describe things. I like to think of changes as a stable ID for something I want to accomplish, and a commit as the intermediate steps that make up a change. You can kind of think of a change as a branch… but I think that’s stretching it.
After your comments, the entire tutorial is more solid, but still I feel like I have gaps. My feedback would be to really focus on commits vs changes and define their relationships, what affects them, etc before you go into anything else.
Clarifying early on what exactly a change ID is, when it changes, and what the relationship between changes and commits is would really help, I think.
If you are already using Sapling then at least trying it should be fairly easy and familiar. You could also write a JJ backend for Mononoke and EdenFS if you wanted and then use it at work ;) You wouldn't be the first JJ user from Meta, actually...
We do have plans to explore server-side designs with virtual filesystems, chunked storage, etc. There's nothing concrete yet.
(Another benefit is that JJ is much easier to write patches and contribute to due to being a relatively new, small Rust project, whereas Sapling is much more developed but much bigger and harder to get into, I think.)
I'm not sure how I feel about JJ's "working copy commit". One of the great things about Meta VCS is that all commits are automatically backed up into the cloud. Which seems incompatible with the JJ model? Not sure. I think the D in DVCS is *wildly* overrated. 99.999% of projects are defacto centralized.
I'm #TeamMonoRepo 100% of the way. My background is gamedev and perforce. An industry which still uses Perforce because Git is poopy poop poop. The Git integration I want to see is the ability to easily sync a monorepo subfolder with an external GitHub repo. Syncing commits for internal projects that are open sourced requires a big ugly custom set of tooling. And I'd kind of like a way to do an "inner fork" within a monorepo if that makes sense.
If you're interested here's a pair of blog posts I wrote that have at least some of my thoughts on source control.
https://www.forrestthewoods.com/blog/dependencies-belong-in-... https://www.forrestthewoods.com/blog/using-zig-to-commit-too...
I can’t tell you how many times I got midway through a git rebase, realize I’d fucked up earlier, and had to abort and redo everything from scratch. With jj not only is this not a problem (thanks to conflict markers being first class citizens) but also any intermediate changes are being snapshotted without me having to do anything. So I can always go back to an earlier change if need be.
The first time I had to do this I’d royally fucked up a complicated commit reordering sequence while screen sharing during a meeting. I’d never needed to use the op log before and had no idea how to use it. It took maybe a minute or two to fix things completely.
Cons: no session save, good Helix keybindings not available in other tools so confusing to switch between vim/hx mental models when in colab/vscode, no AI-assistants since no plugin system yet.
Standard Japanese pronunciation for "jutsu" does not contain an i, and it's also judo not "jiu-do" that the old system would have called for.
additionally you can enable automatic snapshots when files in the working copy are updated: https://martinvonz.github.io/jj/latest/config/#watchman
You simply don't attach a bookmark to @, and `jj git push` will NEVER push it (to what branch would it push?)
The branch you're working on is instead on @-. So when you squash, it updates the branch.
The older ones are there in your local repo but they’re invisible unless you need to look through the “obsolete log” to find earlier iterations of changes. This is similar to (and built on, essentially) git’s reflog that holds on to commits that are no longer in active use.
There’s also an “operations log” that, instead of tracking history of individual changes, tracks history of the entire repo. So if you reorder or drop changes, you can always undo those operations.
Yes? I mean, it even has a system to install plugins made with WASM, from what I saw in the docs. I guess you could just use the basics and be okay with it.
For me, after years of tinkering with apps like Vim, Emacs, and AwesomeWM [1], I've developed a bit of PTSD over the amount of time these kinds of tools can take to configure and master [2]. Zellij feels like it belongs in this category of tools, and perhaps I'm overreacting or flinching. :-)
--
2: I've benefitted a lot from being able to use vim key bindings with extensions in apps like VS Code and emacs key bindings on places and I'm somewhat glad I spent the time, though.
The advantages people first think of when they hear about jj's conflict handling are usually that you can collaborate on conflicts and that you can leave conflicts for later. What's less obvious [1] is that being able to store conflicts in commits means that we can always rebase descendants, so there are states like what Mercurial (and Sapling, I think) call "obsolete" and "orphan". There is also no "interrupted rebase" state when you're resolving conflicts.
These things simplify for the user. They also simplify a lot for developers. An example is how I spent about 2 weeks on a `hg amend --into` command for amending the changes in the working copy into an ancestor. I then implemented that in under an hour in jj. Much of the complexity in hg stemmed from dealing with the interrupted states while dealing with conflicts. (Other complexity in hg that jj doesn't have is dealing with a dirty working copy, dealing with concurrent operations, and simply complexity in the APIs for creating new commits in memory.)
[1] IIRC, it took me about a year after I added support for "first-class conflicts" until I figured out that it meant that we should simply always rebase descendants. Jujutsu had a orphans and a `jj evolve` command before then.
JJ has an abstract notion of a working copy and even an abstract notion of a backend that stores commits. It is a set of Rust crates, and these are interfaces, so you can just implement the interfaces with whatever backend you want. For example, the working copy can be described in terms of the filesystem (POSIX copy/rename/move files around) or in terms of a virtual filesystem (RPC to a server that tells you to make path x/y/z look like it did at version 123.)
You can absolutely have "cloud commits" like Mononoke/Sapling does, and that is a central feature desired at Google too, where Martin works. Skip to the end of this talk from Git Merge 2024 a few weeks ago, and you can see Martin talk about their version of `jj` used at Google, that interacts with Piper/CitC: https://www.youtube.com/watch?v=LV0JzI8IcCY
My understanding is that the design works something like this (no promises but we may approximate something like this in JJ itself): A central server (Piper) stores all the data. A local pair of daemons runs on your machine: a proxy daemon that talks with the server on your behalf (is used to reduce RPC latency for e.g. commits and writes, and does asynchronous upload) and a virtual filesystem daemon (CitC). The virtual filesystem daemon manages your working copy. It "mounts" the repository at version 123 (the "baseline version") onto a filesystem directory, by talking to the server. Then it tracks changes to the directory as you update files; sort of like OverlayFS does for Docker.
When you make a commit, tell the VFS layer to snapshot the changes between your working copy and the baseline, uploading them to the server. Then tell the backend that you've created version 14 out of those files/blobs. The new version 124 is now your new baseline. Rinse and repeat to create version 125, 126, etc...
The TL;DR on that is the VFS layer manages the working copy. The server/proxy manage interaction with a backend.
OK, monorepo export of subprojects. For the purposes of exporting monorepo subfolders, I'm actively exploring and thinking about "filtering" tools that can be applied to a commit graph to produce a new commit graph. Mononoke at Meta can do something like this. Check out this tool Josh that does it for native Git repositories: https://josh-project.github.io/josh/reference/filters.html
The idea is let's say you have commits A -> B -> C and you want to export a repository that only contains a project you want to export, located under paths path/to/oss/thing. You basically sweep the commit graph and only retain commits that touch path/to/oss/thing. If a commit does not touch this path, remove it from the graph and "stitch" the graph back together. If a commit does touch this path, then keep it.
If a commit touches that path and other paths that don't match, then create a new commit that only touches files under that path. In other words, you might derive a new commit graph A' -> C' where B was thrown away because it touched private code, and A' and C' are "filtered" versions of the originals that retain a subset of their changes. The secret is that this algorithm can be fully deterministic, assuming the input commit graph is "immutable" and cannot be changed.
This idea actually is kind of like applying a MATERIALIZED VIEW to a database. In fact it could actually be a kind of continuous materialized view where a new derived graph is computed incrementally and automatically. But how do you express the filters? The underlying filter is really a way of expressing a functional map between graphs. So should filters only be bijective? What if they're injective? Etc. I haven't fully figured this out.
So anyway, I've never really worked in gamedev or huge monorepo orgs before, but these are all problems I've acutely felt, and Git never really solved. So, I can say that yes, we are at least thinking about these things and more.
I mean I work at Meta. We have lots of people working on stuff at the same time. :) AraxisMerge auto-merges quite nicely. Text conflicts have just never been a meaningful issue for me in my 15+ year career.
Now binary conflicts are a source of pain! Unfortunately require Perforce style file-locking once you hit a team size.
> I've never really worked in gamedev or huge monorepo orgs before, but these are all problems I've acutely felt, and Git never really solved. So, I can say that yes, we are at least thinking about these things and more.
Good to hear! Yeah the workflow for subproject syncing it's super obvious. It definitely can be done but will require a lot of thinking and a few iterations.
Well, guess I’m on my way to read the tutorial.