Best Open Source Offline AI Agent

3 points by Ms-J 61 days ago | 9 comments

With the rise of AI agents, I've been looking for an agent that can use a combination of local models or models that are hosted on servers I control.

For the local models, the agent must be able to run completely offline. When using remote models, it must only send network traffic to my servers. No checking for "updates" or pulling any type of dependencies from elsewhere.

When trying Opencode, it was a failure.

See: https://github.com/Chetic/opencode-offline

"This is the offline fork of OpenCode. Upstream OpenCode requires internet access for several core functions: the web UI proxies all requests to app.opencode.ai, ripgrep and LSP servers (TypeScript, Python, C++, Rust) are downloaded on first use, and model metadata is fetched from models.dev. In air-gapped or restricted network environments, this means no web interface (HTTP 500 on every request), no file search, no code intelligence, and no model definitions."

The problem is this project isn't maintained to standards and still uses Opencodes base.

Thanks for any suggestions.

Archerlm 61 days ago |

From what I know of, Ollama works offline, but if your ram is only 8 GB and without a GPU, latency would be severely limiting. The closest local model I can suggest is LM Studio (no cloud dependencies, own model registry, etc.). If that doesn't work out for you then GPT4All, but its models are quite small.

Ms-J 61 days ago | |

I'm looking for an agent, but thanks.

I forget the issue with Gpt4all as some have blended together when they weren't suitable for me.

Archerlm 61 days ago | | |

My bad I misread your post. If GPT4all didn't work out, go with Aider. It’s a CLI tool, doesn't have a UI trying to proxy requests to a dev's server. You just point it at your local model (via Ollama or vLLM) and it stays in its lane. Since it’s Python-based, you can grep the source code to confirm there are no hidden update pings. If that's not for you, and you need the IDE experience, pick Continue. It’s the only one that handles air-gapped setups properly. You can manually install the .vsix file and kill all telemetry in the config.json. Unlike OpenCode, it doesn't try to be it’s just a bridge between your code and your model server. OpenCode failed because it’s basically "cloud-first" pretending to be local. Aider and Continue are actually built for what you want.

josefcub 61 days ago |

Try charmbracelet's crush, found here:

https://github.com/charmbracelet/crush

Crush is pretty new, but getting better all the time. It's written in Go, so no node hijinks to get it working. It works fine with my ollama or llama-server localhost endpoints, and I've used it to make up a couple of internal projects without any issues.

It does have internal telemetry and such (including updating its list of external models it can use) that can be turned off in the crush.json configuration file.

If you're on a Mac, you can install via homebrew or use the more traditional route via Github.

Ms-J 61 days ago | |

Thanks for the recommendation, I took a look and maybe you can answer a few questions that I couldn't find a clear answer when doing some quick searching.

Regarding local models, can it use them? I found this discussion:

https://github.com/charmbracelet/crush/discussions/775

I didn't appreciate the meow maintainer's attitude converting it into a discussion and ignoring the issue even to this day.

"It does have internal telemetry and such (including updating its list of external models it can use) that can be turned off in the crush.json configuration file."

Is there a page or guide which explains the telemetry and any internet connected settings?

Forgot to add, I use Linux.

josefcub 61 days ago | | |

My google-fu is failing me at the moment to cite sources, but here's an example ~/.config/crush/crush.json file (based on my own) showing the options to remove telemetry and provider auto updates, and the connection info to connect to a localhost model on an OpenAI-compatible endpoint:

{ "$schema": "https://charm.land/crush.json", "options": { "disable_provider_auto_update": true, "disable_metrics": true }, "providers": { "ollama": { "name": "Local Models", "base_url": "http://localhost:11434/v1", "api_key": "nunya", "type": "openai-compat", "models": [ { "name": "Qwen 3.5 Local", "id": "qwen-3.5-35b-planning", "cost_per_1m_in": 0.01, "cost_per_1m_out": 0.01, "context_window": 131072, "think": true, "default_max_tokens": 5120, "supports_attachments": true } ] } } }

...or not, thanks to formatting. I can't even search for help formatting this text box, because of HN's nature haha

verdverm 61 days ago |

I have a custom agent setup and just added ollama support. The main issue with local is model quality, it's just not there for the most part. I have had some good results using gemma4 31b on cpu, but the latency is really long. I do plan to use it for higher level automation that could run for a day or two without me needing to look at it (what would be ~minutes to hours with Gemini flash, i.e.)