I started building my own computer use agent - and ran into a bunch of issues. While there are platforms that offer virtual desktops, I wanted full control over the configuration of the OS and programs that were preinstalled. Looking at the open source computer use projects available, most were either proof of concepts or tightly coupled to a very opinionated designs. It ended up being a pain just to get a basic setup going!

Here’s my entry into the mix: A docker container that encapsulates any desktop, with bindings for computer use actions, and a VNC client for viewing. By default it comes with a preconfigured lightweight Linux image that has Chrome and some other tools preinstalled.

I built it over a weekend and I’ve been tweaking it over the past week. It’s meant to be fully extensible - build your agent in whatever language you like, using whatever model (via api or locally), swap in whatever OS image you’d want, and deploy onto whatever hardware.

In the next few weeks I’ll be tackling some low-hanging fruit including adding MCP support, native Terminal and Text Editor implementations, sensible defaults for deployment machines, as well as my own default “batteries included” agent.

Check it out, give it a star, and leave a comment below if you’re building a computer use agent!