Show HN: Kedge – Full-stack cloud with forkable VM snapshots and global SQLite

Show HN: Kedge – Full-stack cloud with forkable VM snapshots and global SQLite(kedge.dev)

35 points by wgjordan 3 hours ago | 9 comments

I'm building Kedge, a globally distributed platform for stateful serverless apps. Here's how you make a simple static site: `echo '# Hello world!' | ssh kedge.dev`

I helped build Fly.io for 4 years and shared enthusiasm for the founders' vision of a 'global Heroku'. While there, I wrote "The Serverless Server" (https://fly.io/blog/the-serverless-server/) as a study of Lambda and a sketch of a modern serverless product built around lightweight VMs. That essay was the initial inspiration for Kedge.

Kedge has a fast VM orchestrator that can create code sandboxes or scale service instances in 3ms, using a combination of forkable VM snapshots and a tree of warm pools (Linux kernel -> base runtime -> app). VMs are memory-dense thanks to shared copy-on-write memory pages. You can run lightweight CGI-style functions, public OCI images, or source code for BuildKit to compile and deploy.

Kedge's global control plane sits on an eventually-consistent SQLite database. Taking inspiration from Corrosion and Litestream, I built a local-first, multi-writer CRDT-based replication system backed by object storage, and just recently made it open source (https://github.com/wjordan/syzy).

You can also use a SQLite client to query `/shared.db` from any instance for a build-in replicated database in your app. This lets Kedge autoscale services close to your users while each instance queries its local replica for eventually-consistent data, with no need to micro-manage instance or volume placement. (There's also a /shared/ filesystem adapter for convenience.)

Kedge can even use this same database for stateful, server-rendered HTML apps. Data attributes bind forms, buttons, and values to records in the app database, Kedge compiles the schema and operations at deploy-time, and then queries the local data to serve requests. As a demo, I made a Hacker News clone with story submission, votes, comments and auth in about 60 lines of Markdown, plus CSS (https://kedge.dev/docs/html-apps#kedger-news).

I've just started collecting public feedback, so please let me know what you think! I'm particularly interested in feedback on the stateful HTML app model, which is the newest (and most ambitious) piece. The preview is currently running in 11 regions for you to kick the tires. There's no billing yet, so the pricing page is an estimate. Thanks for taking a look!

nicksuperb 2 hours ago |

https://kedge.dev/docs/sandboxes

Caught my eye with this one. I'll be taking a closer look as this is one of those "I'd rather let someone else do it." type infrastructure bits :)

wgjordan 1 hour ago | |

Thanks! I tried to make the sandbox-eval stdin interface as simple as possible so you could easily plug it anywhere. The executable docs run an actual full shell in a VM, since they're so fast and cheap to spin up- I hope the example made it clear how to use this feature!

devmor 1 hour ago | |

This is also extremely interesting to me.

An instant, ephemeral cloud sandbox is pretty handy.

tekacs 1 hour ago |

This is phenomenal, great work!

On the HN frontpage is Superlogical (I have no affiliation) right now, and I used the shell provided by the Kedge News demo to `ssh superlogical.jobs` from my browser on a phone on an instance started seconds earlier. :)

I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?

wgjordan 1 hour ago | |

Thanks for the kind words!

> I didn't immediately spot the durability contract for volumes -- local NVMe like Fly, or sitting on some underlying system like EBS?

Persistent volumes use local NVMe that continuously syncs to object storage, so a similar architecture to Fly Sprites. It automatically recovers from host failures (with on-demand lazy restore), and you can manually migrate stable members to another region if needed.

I've positioned persistent volumes as an escape hatch for when you need an existing database app running in a VM, or when performance requirements outgrow the built-in replicated SQLite.

vyedin 2 hours ago |

this is super cool, bookmarking to give it a whirl

> As a demo, I made a Hacker News clone with story submission, votes, comments and auth in about 60 lines of Markdown, plus CSS

although why would you use it for evil

wgjordan 2 hours ago | |

> although why would you use it for evil

I couldn't help it! Ben Johnson recently wrote a Litestream post touching on this topic [1]:

> multiple-writer distributed SQLite databases are the Lament Configuration and we are not explorers over great vistas of pain

I offer my sincerest apologies if this database release may have inadvertently opened the gates of Hell.

[1] https://fly.io/blog/litestream-writable-vfs/

PersonalJarvis 1 hour ago |

Two questions about the fork boundary. The snapshot is taken once when the process calls listen, so every later instance is a clone of the same frozen memory image, and anything seeded during init is identical across every clone in every region. That is exactly what the Linux vmgenid driver exists for, and it is why Lambda SnapStart had to ship explicit reseeding guidance. I could not find entropy, RNG or reseed anywhere in the runtime, performance or security docs, so is there a post-restore reseed hook, and does it fire before the first accept?

The other one is syzy. CRDT.md says eviction is operator-driven, that PRUNING.md is a design-stage contract and not yet code, and that a replica lagging beyond the hold window may transiently observe the departing duplicate, while the product headline offers scale to zero when idle, which mints offline replicas continuously. Who calls syzy_evict in the managed product?

wgjordan 41 minutes ago | |

These are both later-stage detail questions for when this project is further hardened for production workloads, but here's where each of these currently stands:

- Yes, there is a post-restore kernel reseed hook that happens after fork, and just before the trapped listen is continued. It doesn't use the vmgenid driver, it just calls the RNDRESEEDCRNG ioctl, so it would not be compatible with applications that depend on vmgenid notifications to reseed userspace PRNGs.

- In Syzy, the full pruning implementation is deferred for now; at the managed product's current small scale, it can simply eat the tiny overhead costs of un-collected garbage, while the user is only billed for the logical storage space consumed.