Running our Docker registry on-prem with Harbor(dev.37signals.com) |
Running our Docker registry on-prem with Harbor(dev.37signals.com) |
The issues we’ve had so far:
- No programmatic way to retrieve your token that’s required for ‘docker login’. So we had to create a robot account per user and pop their creds into our secrets store.
- Migrating between sites by cloning the underlying S3 bucket and spinning up the new Harbor instance on top of it, does not work. Weird issues with dropping pulls.
- RBAC goes down to project, not repository level, complicating some of our SDLC controls.
- CSRF errors every time you try to do anything in the UI
- Lenient API and lack of docs means things like setting up tag immutability rules via Terraform was a bit of a PITA to figure out the right syntax
So some small issues, but definitely a great piece of software.
One glaring omission is lack of support for proxy docker.io without the project name i.e pulling nginx:latest instead of /myproject/nginx/nginx:latest
The workaround involves URL rewrite magic in your proxy of choice
Contrary to the author of this post, we just run one (the "source of truth") and use caching proxies in other regions. Works fine for us.
1. Doesn't work with ExternalSecretsOperator and ArgoCD, which I happen to use. This is because the author of the Harbor chart decided not to use k8s concepts like secretRef in a podTemplate. Instead, at Helm template time, it looks up the secret data and writes it into another secret, which is then included as a envFrom. This interacts poorly with ExternalSecretsOperator in general, because it breaks the lifecycle control that ESO has. It's completely broken with ArgoCD because ArgoCD disables secret lookups by charts for pretty valid security concerns. No other chart I've come across does secret lookups during helm template time. Even the helm docs tell you it's not correct.
2. Harbor requires redis, but the Helm chart doesn't correctly pipe in the connection configuration. Redis can't be behind TLS, or the chart won't work.
I wonder what they were doing that resulted in blowing out their Docker layer cache on every pull and push.
Normally only a layer diff would be sent over the wire, such as a code change that didn't change your dependencies.
The last time I used it earlier this year for a company already on AWS, it was ~$3 / month per region to store 8 private repos and it was really painless to have a flexible and automated life cycle policy that deleted old image tags. It supports cross region replication too. All of that comes without maintenance or compute costs and if you're already familiar with Terraform, etc. you can automate all of that in a few hours of dev time.
In my experience Nexus is a bit weird to administer and sometimes the Docker cleanup policies straight up don't work (junk left over in the blob stores even if you try to clean everything), but it also supports all sorts of other formats, such as repositories for Maven and NuGet. Kind of hungry in regards to resources, though.
It is pretty easy to just run the basic registry for this purpose.
We have a similar setup for NPM and Pypi on the same machine. It doesnt really need a lot of attention. Some upgrades every once and a while.
With the self hosted official registry, the stop-GC-restart process is a PITA.
One issue to solve is auto-deletion of old images, so that the storage does not swell. Any tips?
This is incorrect, the word you are looking for here is "on-premises" - a "premise" is something entirely different.
The same Nexus is also hosting our Maven and NodeJS repositories. That has 1,800,000 hits per day, although all but 120,000 of them return HTTP 404. (I think one of our clients has misconfigured their build server to use our Maven repository as a mirror of Maven Central, but as it's just an AWS IP I don't know who it is.)
I'm sure it's overprovisioned, but the marginal cost of 2, 4 or 8 cores and 4, 8 or 16GB RAM isn't much when we buy our own hardware.
this is the crux of it all. RENTING bare metal (eg from hetzner) is 10x cheaper than aws ec2. So, I can only imagine how much cheaper it is when you buy the hardware directly.
Hell Git doesn't even need the Git protocol if you do `update-server-info`
https://github.com/jpetazzo/registrish
I haven’t tried running this yet, but it seems worth keeping in mind. It’s relatively simple software so the idea could probably be pretty easily adapted to other situations.
0. https://github.com/replicatedhq/ttl.sh/blob/main/registry/en...
> During this time, Harbor has served more than 32,000 pulls under company-wide use in day-to-day business.
They never say that it needs these resources, they say that the current VM has this config. Probably overkill by 1000%.
I run distribution [0] / registry [1] as a docker (via Nomad) job, and it just uses a shared nfs mount in my cluster - ie. EXT4 FS underneath.
Currently has 12GB and 52 repositories, and is very performant. It's just a file server.
Unless your size requirements are huge, the complexity of having your registry rely on a self-hosted S3-alike sounds like a lot of pain for future-muhehe to contend with if / when the S3-alike hiccups.
I'm using it in my lab, along with an older instance of minio, and both are excellent choices I think. (I'm running both as HA jobs within a Hashicorp Nomad cluster, which complicates / eases various things.)
I had a vague memory of minio losing some favour a while back because they switched their underlying storage from basic object : normal file mapping to something more complex. A lot of home users, and I guess developers, liked the ability to navigate their S3-esque buckets via normal linux file system tooling.
If you can find an example of OIDC with Github Actions and Harbor I'd love to see it.
However, when getting the config prod-ready, we started to trip over one thing after the other. First, my colleague was struggling to get the scale-out clustering to work in our container management. Right, use the other deployment way for HA. Then we found that apparently, if you enable OIDC, all other authentication methods get deactivated, so suddenly container hosts would have to login with tokens... somehow? And better hope your OIDC provider never goes down. And then we found a bug on top that Zot possibly doesn't remove blobs from minio during GC.
At that point we reconsidered and went with Harbor.
I use mine primarily for long-term stores for Grafana's Mimir and Loki products.
Anyway, minio & garage are both lightweight, and happily run in docker, so if you've got a small cluster at home it would take maybe an hour or two to install them both from scratch.
Sometimes you'll need admin skills, but only if you spread your cluster out over availability zones or poor connections
Add to that rackspace costs, staff time doing installation (or cost of remote hands etc). I don't have our figures for rackspace costs, but staff time is fairly low — an hour or two when the server is purchased and later thrown out, and maybe the same to replace a HDD at some point in the 5-8 years we keep it.
Also add the time to get competing quotes from Dell, HP etc and decide on the configuration, but then a similar process is needed to choose from Hetzner vs their competitors.
—-enable-helm isn't supported everywhere but Argo definitely allows it