Could something like this enable SSI distributed computing entirely in user space? Is it practical to run some sort of distributed shared memory, relying on user-managed page faults to ensure coherence?

Thanks for sharing! I am still proud of this hack :)

Sort of, but not exactly. You can't redirect the program to a different address, but you can respond to the fault by mapping memory at address 0x4000 and placing whatever you want in it. Perhaps, you can map the memory as an alias of some other page, so that they refer to the same underlying physical memory. But the program will still see the memory as living at 0x4000, it won't jump to a different address. (Though I suppose if you're specifically thinking of the instruction pointer faulting at 0x4000, you could respond by filling in that address with a jump instruction jumping to the address of your choice...)

To map memory at a specific address, you use mmap() with the MAP_FIXED flag. This lets you tell the kernel: "Please put these pages at this virtual address, replacing what's already there."

Note that userfaultfd is not the only way to detect and handle faults. You can also register a signal handler for SIGSEGV. The signal handler function receives information from the kernel specifying what address caused the fault. If, in the signal handler, you use mmap() to map memory at that address, and then return, the program will continue as if nothing happened.

userfaultfd allows you to do the same but from a different thread or process. The SIGSEGV signal handler would actually run inside the thread that faulted, but with userfaultfd the faulting thread pauses and some other thread or process receives the notification.

You don't need userfaultfd for that. Just mmap the same file twice, at different addresses:

    int mem_fd = memfd_create("", 0);
    ftruncate(fd, 4096);
    void* mem = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, 0);
    void* alias = mmap((void*) 0x4000, 4096, PROT_READ | PROT_WRITE, MAP_FIXED | MAP_SHARED, mem_fd, 0);
    close(mem_fd);
    // alias now points to the same physical memory as mem

This is commonly used to create ring buffers that wrap around transparently.