Containerize Go and SQLite with Docker(awstip.com) |
Containerize Go and SQLite with Docker(awstip.com) |
Dependencies is not really a problem with Docker, nor the thing it is designed to solve. If dependencies was the problem people cared about, everyone would just go with the single statically linked executable/fat JAR and no one would ever be bothered with Docker.
Docker is primarily about containerization, but it's also about ease of packaging and deployment. It's also a deployment format that provides horizontal scaling for free.
Also, the one-database-per-service architecture pattern is quite common, as well as ephemeral databases and local caching, and keep in mind that SQLite also supports in-memory databases.
> It's also a deployment format that provides horizontal scaling for free.
Um. Docker does not provide horizontal scaling at all, for that you need orchestration. And those tools are anything but free if your time has any value.
Docker swarm mode is pretty good and terribly easy to get up and running in no time.
I have a few small personal projects hosted on Herzner on a couple of Docker swarm mode deployments with 100% uptime in the past two years, and all it took to get that infra up and running is installing Docker on a bare Linux node.
The only downside I'm aware is that inter-node traffic speeds can be relatively low.
Or: modernc.org/sqlite (plus https://github.com/zombiezen/go-sqlite), "an automatically generated translation of the original C source code of SQLite into Go"
as mentioned yesterday: https://news.ycombinator.com/item?id=29959193#29960726
The discussion of building fully static go binaries for docker has been rehashed a million times on the internet, the modernc.org/sqlite is at version 3+, so nothing in this should be new information for anyone that has an interest in the topic.
I strongly dislike this type of content that just repacks basic existing information and uses it as an excuse to click bait people on to a spammy blog.
At $work we use the following template. Sharing in case you find it interesting:
FROM golang:1.17.1-alpine3.14 as builder
RUN apk update && apk add ca-certificates curl git make tzdata
RUN adduser -u 5003 --gecos '' --disabled-password --no-create-home mycompany
COPY . /app
WORKDIR /app
RUN make build
FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app/bin/app-3 /bin/app-3
COPY --from=builder /etc/passwd /etc/passwd
USER mycompany
CMD ["app-3"]
help:
@echo "Please use 'make <target>' where <target> is one of the following:"
@echo " test to run unit tests."
@echo " build to build the app as a binary."
@echo " build-image to build the app container."
@echo " run to run the app with go."
run:
go run -ldflags="$(govvv -flags -pkg) -w -s" ./cmd/app-3/main.go
test:
go test -v -coverpkg=./... -coverprofile=profile.cov ./...
go tool cover -func profile.cov
rm profile.cov
build:
CGO_ENABLED=0 go build -mod=readonly -a -ldflags="$(govvv -flags -pkg $(go list ./info)) -w -s -linkmode external -extldflags '-static'" -o ./bin/app-3 ./cmd/app-3/*
build-image:
docker build -t mycompanytown/app-3:latest .See also: -trimpath
https://blog.filippo.io/shrink-your-go-binaries-with-this-on...
Trying to decide if it's worth working into the containers that I maintain.
In reality you'll tell whatever deployment system to deploy your binary/container, and you'll just wait until it's done. In such cases, 150ms won't make a difference in the grand scheme of things due to other systems such as your CI and other things in your pipeline.
goupx is not necessary after go 1.6. https://github.com/pwaller/goupx#update-20160310
I mean sure, in theory you shouldn't get any of those in production, but no software is perfect.
Our backend teams rarely do anything with containers, but all apps deploy to K8s (platform team manages).
We deploy SQLite… don’t remember any Cgo requirements, however we’re also deploying each SQLite “instance”, or db-files, to a node directly, and for low data gravity on SSDs. Go interacts with database/sql.
Has anyone actually used it?
If you haven't jumped on the bandwagon yet, don't do it. It's a sucker's game.
I got sucked into the Docker hype some years ago and the demos worked well enough in isolation.
I pushed it on a small team that didn't need it. One of the biggest regrets of my career.
Docker may be useful if an application has enormous reach and needs Google scale, but very few apps do.
1) It is incredibly slow to build
2) It is slow to run
3) It is an unbelievable resource hog
4) It has an entirely new set of problems (file system, networking, etc.) that must be understood to get it working
5) Hard deploy time dependency on Docker's web app?!
6) If you do not believe me on the complexity, just observe how docker arguments always seem to turn into k8s (or other orchestration) arguments. It is hard to reason about, hard to deploy, hard to coordinate.
K8s is a total disaster for a small team working on a small project.
We lost many months by choosing these tools for a startup that didn't (yet) need them, and the ongoing, neverending pile of problems and distractions they generated was shocking.
I still feel bad about it. The company could have run on a couple $400 machines under a desk for years, but instead spent tens of thousands on AWS services it didn't need.
Of course, YMMV
A bunch of other options in the docs as well
[0]: https://kubernetes.io/docs/tasks/debug-application-cluster/d...
[1]: https://kubernetes.io/docs/tasks/configure-pod-container/sha...
I was originally turned onto this by a post on the repl.it blog about using Nix this way.
You can also do this with other build systems or using weird docker file hacks by hand.
Here's some idea of what the syntax looks like:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-supp...
It sounds like (no offence) issues that containerization solved were missed and you had no dedicated operations/build-engineer guy on your team that could've explained/fixed these things for you and steer your infra in a right direction.
> I still feel bad about it. The company could have run on a couple $400 machines under a desk for years, but instead spent tens of thousands on AWS services it didn't need.
Does not sound like a docker issue. If you actually plan to run a money making business that your livelihood depends on, then running it under your desk is not something that you can reliably do even 10-15 years ago.
AWS is a money hog with a lot of things that are a distraction for a small project and it is easy to assume that their products are necessary to run your app effectively, but most often it is not. There is plenty of cloud providers that are not as expensive, and are much easier to manage compared to AWS.
> K8s is a total disaster for a small team working on a small project.
This is truly a YMMV thing. in my experience managed k8s for a small team can be a godsend for infra management after some level of complexity is reached, while self-hosted bare-metal k8s can utterly decimate the team. But again YMMV.
I am currently running several clusters of Nomad for different types of workloads each. CI, HPC, VM's, public facing services you name it. Containerization makes it all possible to run (mostly) single-handedly.
Swarm pros:
- Easy to setup
- Easy to run
- Relatively easy to debug
Swarm cons:
- Many problems persist for years. Some because of lack of resources, others because the problem is simply too hard.
- The community and automation around swarm is small
- Problems solved by third party tools, apps, etc. in kubernetes require in-house workarounds or solutions (e.g. there was an API to perform autoscaling, but we had to write the python app that will read data from prometheus and scale-in/out the deployment)
What I found it with Swarm is that it was extremely resilient. At some point the swarm cluster was running on AWS for ~2 years with minimal maintenance. That would have been impossible even for a managed EKS cluster for example. There are simply too many things that can go wrong.
You only need gcc during the build fase, that doesn’t change just because you run without Docker.
There’s nothing preventing you from statically link C programs either and have a single binary to deploy either. You just have to rebuild everything every single time security updates are a available. But that no different from Docker images, they need to be rebuild constantly as well.
However, many people already know how to use Docker and have flows around it so it makes sense for them to do this.
If that works for you, then there's no need to change. But if you ever end up in an environment where multiple different applications are deployed, or if your app needs to be deployed a thousand times, that's where e.g. Docker may come in handy.
The complexity of Docker is not that big for a Go deployment though, especially if you have all the other bits for orchestrating your Docker containers (for the rest of your stack) already in place. You mostly just copy the binary into a slim image and you are done.
You don't need docker for that, just 'tar czf my-layer.tar.gz my-dir'. If you want a manifest file, you can get the digest using `sha256sum my-layer.tar.gz`.
Docker does this quite well, and solves a bunch of other problems you're likely to have regardless. One really simple example is "how do you copy it to the server?" Do you have ssh keys for your server on development machines? How do you handle the "Oh I'll just remote in and fix this one X"?
It's also _crazily_ easy to get started with, and handle middling amounts of scale. If you're on AWS, Elastic Beanstalk is plug and play. If you're not, DigitalOcean App platform will host it for you, with automated deployments from git for $5/month with basically no configuration needed from you.
Publishing your docker image to some open or private store still needs to happen. Then the host needs to be updated. This is not really that much simpler than "scp your binaries to the server". And has many more moving parts that can fail.
Now, there are better ways to distribute go projects than scp, for example heroku style or by just abusing its builtin git support.
Of course, if you're doing somethign in Python/Ruby/Node.js then it's useful because these language do not provide a reliable method to compile source code into an independent program/application.
But that does not make it "awesome" or anything. It's just a band-aid. It's an extra layer to hide fundamental design problems.
With Go, the root problem is solved: the language has a compiler that reliably produces statically linked binary executable programs.
This is the very first time I ever heard such nonsense. In all companies I've been, Docker is a renowned problem solver, not only for production deplyments but also for local testing environments and deployments.
It even shines as a stand-alone barebones clustering solution with Docker swarm mode.
I find it quite amusing that projects that try to position themselves as Docker alternatives end up basing their presentation on how their project can be used just like Docker, down to Docker's choice of command line interface.
Of course, if you're in a tightly managed environment and can't get on the nodes themselves things get harder, but probably not completely impossible :)
Maybe it will be some day, but these orchestration and deployment tools built on Docker have enormous hidden costs.
IME, Docker takes an enormous amount of focus away from the customer problem and moves it to the how to get this mess working problem.
Now may be the right time in a given business to make that shift in focus, but to claim that it will be "free" is just misleading (IME).
This is the opposite in my experience: Docker lets me focus on the business problem by making deployment easier.
It does. Please take the time to learn about Docker and it's Docker swarm feature.
At least, that's been my experience when maintaining a mess of other people's Docker crap.