Show HN: GoModel – an open-source AI gateway in Go

Show HN: GoModel – an open-source AI gateway in Go(github.com)

217 points by santiago-pl 26 days ago | 75 comments

Hi, I’m Jakub, a solo founder based in Warsaw.

I’ve been building GoModel since December with a couple of contributors. It's an open-source AI gateway that sits between your app and model providers like OpenAI, Anthropic or others.

I built it for my startup to solve a few problems:

  - track AI usage and cost per client or team
  - switch models without changing app code
  - debug request flows more easily
  - reduce AI spendings with exact and semantic caching

How is it different?

  - ~17MB docker image
    - LiteLLM's image is more than 44x bigger ("docker.litellm.ai/berriai/litellm:latest" ~ 746 MB on amd64)
  - request workflow is visible and easy to inspect    
  - config is environment-variable-first by default

I'm posting now partly because of the recent LiteLLM supply-chain attack. Their team handled it impressively well, but some people are looking at alternatives anyway, and GoModel is one.

Website: https://gomodel.enterpilot.io

Any feedback is appreciated.

nzoschke 26 days ago |

Looks nice, thanks for open sourcing and sharing.

I'm all in on Go and integrating AI up and down our systems for https://housecat.com/ and am currently familiar and happy with:

https://github.com/boldsoftware/shelley -- full Go-based coding agent with LLM gateway.

https://github.com/maragudk/gai -- provides Go interfaces around Anthropic / OpenAI / Google.

Adding this to the list as well as bifrost to look into.

Any other Go-based AI / LLM tools folks are happy with?

I'll second the request to add support for harnesses with subscriptions, specifically Claude Code, into the mix.

ewhauser421 26 days ago | |

If you're all in on Go and AI, you might want to take a look at: https://github.com/ewhauser/gbash

It's a just-bash like variant implemented in Go. Useful for giving a managed bash tool to your agents without a full sandboxing solution.

markusw 25 days ago | |

Hi, I'm the author of GAI. I'm glad that you are happy with it! I'm using it a lot myself, both for own and client projects, but I wasn't sure if anyone else was. :D

I'm actually trying to build it out in a way so that gateways aren't necessarily necessary. Cost and token tracking happen through OpenTelemetry. Fallbacks and retries are handled through the new “robust” package, and I have other plans as well. You're always welcome to file issues in the repo for things you'd like to see but aren't there yet. :-)

santiago-pl 24 days ago | | |

I'll definitely take a look at GAI myself! I like this beaver(?) at README.

santiago-pl 26 days ago | |

I'll take a closer look at it over the next few days.

However, it might be challenging, considering that Claude Code with a subscription no longer officially works with OpenClaw.

nzoschke 26 days ago | | |

Yeah I share the same uncertainty here. My understanding is personal and interactive use should be fine. I use Conductor all day every day and it wraps a subscription.

Perhaps fully automated use is where the line is drawn.

But I also suspect individuals using it for light automated dispatching would be ok too.

arcanemachiner 26 days ago | | |

That... might have changed?

https://news.ycombinator.com/item?id=47844269

lackoftactics 26 days ago | | |

It suppose to work again based on todays news

jy-tan 26 days ago | |

I built Fence (https://github.com/Use-Tusk/fence) in Go, a lightweight process sandbox for CLI agents (or any command really) with filesystem and network restrictions. It's also available as a Go library if you wish to add sandboxing to Shelley.

rolls-reus 26 days ago | |

i use this for my personal projects. some features are gated behind a license but the basics like provider proxy, logs, metrics are covered in the free version. https://github.com/maximhq/bifrost

verdverm 26 days ago | |

> Any other Go-based AI / LLM tools folks are happy with?

I can throw my hat into the ring, built on ADK, CUE, and Dagger (all also in Go); CLI, TUI, and VSCode interfaces. It's my personal / custom stack, still need to write up docs. My favorite features are powered by Dagger, sandbox with time travel, forking, jump into shell at any turn, diff between any points.

Good entrypoint folder: https://github.com/hofstadter-io/hof/tree/_next/lib/agent

pizzafeelsright 26 days ago |

I have written and maintained AI proxies. They are not terribly complex except the inconsistent structure of input and output that changes on each model and provider release. I figure that if there is a not a < 24 hour turn around for new model integration the project is not properly maintained.

Governance is the biggest concern at this point - with proper logging, and integration to 3rd party services that provide inspection and DLP type threat mitigation.

ijk 26 days ago | |

Given the LiteLLM Supply Chain Incident, threat mitigation and governance are definitely major concerns.

crawdog 26 days ago |

I wrote a similar golang gateway, with the understanding that having solid API gateway features is important.

https://sbproxy.dev - engine is fully open source.

Another reason golang is interesting for the gateway is having clear control of the supply chain at compile time. Tools like LiteLLM the supply chain attacks can have more impact at runtime, where the compiled binary helps.

lackoftactics 26 days ago | |

Maybe worth showing on SHOW HN

crawdog 26 days ago | | |

Thanks I am finishing up some performance comparison work looking at rust vs golang and plan a deeper write up for that group. I hope to publish soon.

hgo 26 days ago |

Hey, this looks super nice. I do like the 'compact' feel of this. Reminds me of Traefik. It seems very promising indeed!

One problem I have is that yes, LiteLLM key creation is easier than creating it directly at the providers and managing it there for team members and test environments, but if I had a way of generating keys via vault, it would be perfect and such a relief in many ways.

I see what I need on your roadmap, but miss integration with service where I can inspect and debug completion traffic, and I don't see if I would be able to track usage from individual end-users through a header.

Thank you and godspeed!

santiago-pl 26 days ago | |

"... and I don't see if I would be able to track usage from individual end-users through a header".

Currently we have a unified concept of User-Paths. Once you add a specific header OR assign User-Path to an API key, you can track the usage based on this. The User-Path might be youe end-user, internal user or some service. Examples:

  /client1/app1
  /agents/agent1
  /team2/john
  /team2/adam

Would this work for you?

https://gomodel.enterpilot.io/docs/features/user-path

PS Thanks for the feedback on the Vault integration. Noted.

hgo 25 days ago | | |

Ah, seems like the right thing. To be more clear on what I'm looking for is this: the system using the LLM gateway would present an arbitrary user id. Let's say the system has thousands of end-users (completely managed by that system and not configured in the LLM proxy). The admin is interested in blocking end-users from using more than a certain allowed quota.

neilly 26 days ago |

Given this app seems to expose itself via REST calls, why would anyone care that it’s written in Go? I guess it matters to potential contributors but the majority of interest would be from users.

santiago-pl 26 days ago | |

It's like fuel costs in a supply chain. When you buy apples at the store, you don't think about oil prices. But if trucks ran on something cheaper, more efficient, or less taxed, the apples on the shelf would be cheaper too.

mosselman 26 days ago |

Does this have a unified API? In playing around with some of these, including unified libraries to work with various providers, I've found you are, at some point, still forced to do provider-specific works for things such as setting temperatures, setting reasoning effort, setting tool choice modes, etc.

What I'd like is for a proxy or library to provide a truly unified API where it will really let me integrate once and then never have to bother with provider quirks myself.

Also, are you also planning on doing an open-source rug pull like so many projects out there, including litellm?

santiago-pl 26 days ago | |

1. Yes, we have OpenAI-compatible API and we develop GoModel with Postel’s law in mind: https://gomodel.enterpilot.io/docs/about/technical-philosoph... .

2. Regarding being open-source and the license, I've described our approach here transparently: https://gomodel.enterpilot.io/docs/about/license

sowbug 26 days ago |

Are these kinds of libraries a temporary phenomenon? It strikes me as weird that providers haven't settled on a single API by now. Of course they aren't interested in making it easier for customers to switch away from them, but if a proprietary API was a critical part of your business plan, you probably weren't going to make it anyway.

(I'm asking only about the compatibility layer; the other tracking features would be useful even if there were only one cloud LLM API.)

pjmlp 26 days ago |

Expectable, given that LiteLLM seems to be implemented in Python.

However kudos for the project, we need more alternatives in compiled languages.

smcleod 26 days ago | |

LiteLLM proxy has to be about the worst codebase and performing Python code I've used in a long time. Incredibly poor performance, riddled with serious bugs and to be honest devs that don't seem to understand them.

santiago-pl 26 days ago | | |

TBH I decided to write GoModel because I needed something like this for my startup, enterpilot, and LiteLLM didn’t meet my needs.

santiago-pl 26 days ago | |

Agree and thank you! Please let us know if you'd like to give it a try and if you miss any feature in GoModel.

goodkiwi 26 days ago | |

It’s also badly implemented - everything is a global import. Had to stop using it

glerk 26 days ago |

This is awesome work, thanks for sharing!

How do you plan on keeping up with upstream changes from the API providers? I have implemented something similar, and the biggest issue I have faced with go is that providers don’t usually have sdk’s (compared to javascript and python), and there is work involved in staying up to date at each release.

santiago-pl 26 days ago | |

First, GoModel is designed to be flexible. If you add an extra field, it tries to pass it through in the appropriate place (Postel's law)

Therefore there's a good chance that if they make a minor API-level change, GoModel will handle it without any code changes.

Also, changes to providers' API formats might be less and less frequent. Keeping up typically means adding a few lines of code per month. I'm usually aware of those changes because I use LLMs daily and follow the news in a few places.

As a fallback, GoModel includes a passthrough API that forwards your request to the provider in its original format. That might be useful when an AI provider changes their contract significantly and we haven't caught up yet.

Also, official SDKs aren't bug-free either. Skipping that extra layer and hitting the API directly might actually be beneficial for GoModel.

vorticalbox 26 days ago | |

Most APIs provide some sort of documentation. If it’s swagger you can just update the application from that.

lackoftactics 26 days ago | |

Almost impossible without backing from some VC like litellm

swyx 26 days ago | | |

ridiculous statement. most people dont need long tail.

nopointttt 25 days ago |

Nice work. How are you doing the cache key when the prompt has a timestamp or session id in it? Regex pass to strip volatile fields before hashing, or do you make the user declare what's stable upfront? I've tried both in my own stuff and neither ends up clean.

giorgi_pro 25 days ago | |

Thank you. No config atm to declare which fields are "stable" for caching. I assume you mean cases where the user would provide a different session ID but still would prefer the key to match. Am I right?

StingSS 25 days ago |

Really slick and lightweight! My biggest question is about the upstream API evolution. Keeping up with every provider's updates feels like a full-time job—how do you automate or manage the tracking of these constant protocol shifts?

santiago-pl 25 days ago | |

I'm wroking on it full-time right now. It might be challenging, especially when it comes to interactions with video, audio, and image models. I'm just trying to stay on top of what's happening and add new things day by day.

An even bigger challenge might be integrating it tighter with the cloud platforms, Prometheus, OAuth, Datadog, vaults, and DLP software.

Talderigi 26 days ago |

Curious how the semantic caching layer works.. are you embedding requests on the gateway side and doing a vector similarity lookup before proxying? And if so, how do you handle cache invalidation when the underlying model changes or gets updated?

giorgi_pro 26 days ago | |

Hey, contributor here. That's right, GoModel embeds requests and does vector similarity lookup before proxying. Regarding the cache invalidation, there is no "purging" involved – the model is part of the namespace (params_hash includes the LLM model, path, guardrails hash, etc). TTL takes care of the cleanup later.

rvz 26 days ago |

I don't see any significant advantage over mature routers like Bifrost.

Are there even any benchmarks?

santiago-pl 25 days ago | |

My thoughts about this:

Benchmarking AI gateways properly is harder than it looks. Feature sets differ meaningfully - exact vs semantic caching, cluster mode, guardrails, audit logging - and each carries its own latency cost. What actually matters for most users is end-to-end latency including provider overhead (200–2000ms), and in that frame Bifrost, LiteLLM, and GoModel are all perfectly fine.

I ran some comparisons but I'm not happy with the methodology, and I'd rather not spread misleading information. Once I have time to do it properly I'll write it up and share a link here. Honestly, I'd also love to see benchmarks done by someone other than the AI gateway builders. :)

Where GoModel actually differs today:

  - image size: 16.96 MB vs Bifrost's 69.84 MB. It matters for sidecar, edge, and cold-start scenarios.
  - per-tenant keys, guardrails, and audit logs are all in the OSS repo - not gated.
  - AI interaction visualization that makes debugging individual request/response flows much easier.

lackoftactics 26 days ago | |

It’s a heavily vibe coded project with only proxy with terrible benchmarks design. Basically vibe coded benchmarks that lie through ignorance of mocked super fast endpoint without using full power of litellm in multiple processes.

Other than that almost useless it’s faster when this will be io bound and not cpu bound.

eikenberry 26 days ago | | |

Which project are you talking about, GoModel or Bifrost?

driese 26 days ago |

Nice one! Let's say I'm serving local models via vllm (because ollama comes with huge performance hits), how would I implement that in gomodel?

santiago-pl 25 days ago | |

I've released a new version of GoModel (0.1.20) with explicit support for vllm. You can now use it even with a few vLLM instances. Like this:

  docker run --rm -p 8080:8080 \
    -e VLLM_BASE_URL=http://host.docker.internal:18000/v1 \
    -e VLLM_BASEMENT_BASE_URL=http://host.docker.internal:18000/v1 \
    enterpilot/gomodel:latest

devmor 26 days ago | |

This is way more interesting to me as well. I have projects that use small limited-purpose language models that run on local network servers and something like this project would be a lot simpler than manually configuring API clients for each model in each project.

santiago-pl 26 days ago | | |

Thanks for raising it! Since vLLM has an OpenAI-compatible API, this should work for now:

  docker run --rm -p 8080:8080 \
    -e OPENAI_API_KEY="some-vllm-key-if-needed" \
    -e OPENAI_BASE_URL="http://host.docker.internal:11434/v1" \
    ...
    enterpilot/gomodel

I'll add a more convenient way to configure it in the coming days.

tahosin 26 days ago |

This is really useful. I've been building an AI platform (HOCKS AI) where I route different tasks to different providers — free OpenRouter models for chat/code gen, Gemini for vision tasks. The biggest pain point has been exactly what you describe: switching models without changing app code.

One thing I'd love to see is built-in cost tracking per model/route. When you're mixing free and paid models, knowing exactly where your spend goes is critical. Do you have plans for that in the dashboard?

santiago-pl 26 days ago | |

This comment looks like AI-generated.

However IIUC what you're asking for - it's already in the dashboard! Check the Usage page.

indigodaddy 26 days ago |

Any plans for AI provider subscription compatibility? Eg ChatGPT, GH Copilot etc ? (Ala opencode)

santiago-pl 26 days ago | |

You are not the first person who has asked about it.

It looks like a useful feature to have. Therefore, I'll dig into this topic more broadly over the next few days and let you know here whether, and possibly when, we plan to add it.

Trenchcole 25 days ago |

Love the progress!! keey going

phoenixranger 26 days ago |

looks interesting, will defo give it a try. thanks for open-sourcing it!

immanuwell 26 days ago |

it's nice that it supports different providers

anilgulecha 26 days ago |

how does this compare to bifrost - another golang router?

santiago-pl 26 days ago | |

First of all, GoModel doesn't have a separate private repository behind a paywall/license.

It's more lightweight and simpler. The Bifrost docker image looks 4x larger, at least for now.

IMO GoModel is more convenient for debugging and for seeing how your request flows through different layers of AI Gateways in the Audit Logs.

anilgulecha 26 days ago | | |

That would be valuable if there's a commitment to never have a non-opensource offering under GoModel? If so, you can document it in the repo.