Do you think you're better than the OP because you're using a simpler stack?

My one-man-SaaS setup:

- Static frontend hosted on Netlify (free unlimited scale)

- Backend server on Google App Engine (connecting to Gcloud storage and managed DB via magic)

I realize I'm opening myself up to vendor lock-in and increased costs down the road (if I even get that far), but I've wrangled enough Docker/k8s/Ingress setups in the past to know it's just not worth the time and effort for a non-master.

  1. Firebase Hosting for the React frontend
  2. GraphJin (Automatic GraphQL to SQL Engine) on App Engine for the backend
  3. Cloud SQL Postgres for DB

Anthony - if you're reading this, thank you!! To arrive at this architecture takes 100s if not 100s of hours, and to share it with the community is dang inspiring.

I was feeling a bit down on my projects, but this has me amped up seeing how the ultimate goal of working on features rather than deployment is possible, and very real!

Best of luck with Panelbear!

The author almost seems to apologize for having a django monolith.

But it's worth realising that one purpose of code organisation in larger companies is to mirror the team organisation. That's a constraint on code that can interfere with the best technical architecture.

You can do better with a monolith in a one-man team!

How do you start learning this breadth of software engineering? I consider myself good in the python / django space, but where do I start with learning these infrastructure technologies? I find that I use them once or twice periodically, and then don't touch them for so long, so I forget much of what I have learned.

My one-person SaaS architecture with over 250k users:

* Flask + Flask-Login + Flask-SQLAlchemy [1]

* uWSGI app servers [2]

* Nginx web servers [3]

* Dramatiq/Celery with RabbitMQ for background tasks

* Combination of Postgres, S3, and DigitalOcean Spaces for storing customer data [4]

* SSDB (disk-based Redis) for caching, global locks, rate limiting, queues and counters used in application logic, etc [5]

I like how OP shows the service providers he uses, and why he decides not to self-host those parts of his infra. Also, there's a large up front cost involved for any stack (Rails, Django, k8s). I'd be interested in a more detailed writeup with configs, to try out OP's auto-scaling setup. My configs are linked in the gist below [2] for my non-auto-scaling Flask setup.

I spend about $4,000/mo on infra costs. S3 is $400/mo, Mailgun $600/mo, and DigitalOcean is $3,000/mo. Our scale/server load might be different, but I'm still interested in what the costs would be with your setup.

[1] https://wakatime.com/blog/33-flask-part-2-building-a-restful...

[2] https://gist.github.com/alanhamlett/ac34e683efec731990a75ab6...

[3] https://wakatime.com/blog/23-how-to-scale-ssl-with-haproxy-a...

[4] https://wakatime.com/blog/46-latency-of-digitalocean-spaces-...

[5] https://wakatime.com/blog/45-using-a-diskbased-redis-clone-t...

I'd argue that just about every infrastructure that looks like this benefits from Kubernetes (that you're not setting up and managing), and that's a lot of them. The biggest problem is that not enough people have boiled down Kubernetes enough to look like heroku yet. Google Cloud Run is possibly the best example of what Kubernetes can look like/run like -- it runs on (probably a relatively heavily modified) KNative, a project that runs on top of kubernetes.

The "point" of Kubernetes is to drop the difficulty of building a service like Cloud Run to zero. It drops the cost of building a Heroku down to zero. I'd bet my bottom dollar that fly.io and render are running on Kubernetes (maybe they mentioned it somewhere already and I just missed it). With the right cluster set up, building one of those platforms (or others that I won't mention) is almost as simple as setting up stripe checkout and writing a web interface to turn form fields into JSON fields and send them to a kubernetes cluster (possibly with hard multi-tenancy! not to get too into it, but you can literally provision kubernetes clusters from kubernetes clusters, ephemeral or otherwise).

No other tool in the devops world except for maybe the initial orchestrator wave (ansible/puppet/salt/chef) has been this much of a force multiplier. Ok, maybe that's hyperbole, but if adhoc-bash-scripts->ansible is 1->2, Ansible->Kubernetes is similarly 1->2, especially if you consider baked in cloud provider support/innovation.

But here's the secret -- perversely, I'm happy deep down that everyone thinks k8s is too complicated/is a buzzword/isn't worth the effort. All it means to me is that I'm still ahead of the curve.

I am also a one-man SaaS (though not a successful one). The following tends to be my stack (on Google Cloud, if you will):

- Cloud Run (serverless containers)

- Cloud SQL (via proxy)

- Cloud Monitoring & Logging (formerly Stackdriver)

- Compute Engine (if necessary, e.g. websockets)

- Cloud Build for GitOps (deploy on push)

It's clean and simple (to me). Billing is in one place, nicely separated by projects. Monitoring & Logging is already built in. No need to span multiple dev SaaS tools. So far managed to avoid Redis caching because Golang + Postgres is fast enough, so far. But if you need Redis you can DIY on Compute Engine or try Cloud Memorystore (configure the memory to a low amount for cost savings).

Google Cloud drawbacks: Additional charges necessary to connect Cloud Run to VPC (via proxy instances). Load balancing on GCP ain't cheap ($18/month, though to a larger enterprise that is a rounding error). But in my setup I didn't need these things.

As shown above, I have heavily optimized for cost and simplicity in my setup.

Good article. My comment might be off topic in which case, please ignore.

If you have a one-person SaaS company, how do you get past customers’ resistance to a single point of failure, namely you?

Do you pretend you’re not just one person? Do you only have customers who could handle losing the service when you, say, run away to meditate on the mountaintop? (Or get run over by a beer truck, or whatever.) Is there some non-obvious solution?

And — back on topic — is the architecture part of that sales pitch? “I’m just one dude, but look how simple this is, it can run itself if I am devoured by mice!”

This is really well done.

A suggestion, hopefully helpful: a better approach to securing your admin console than simply layering 2FA onto it would be to expose it to a private WireGuard network. One very easy way to do that is with Tailscale, which will hook up to your GSuite authentication --- Google's 2FA stack will be far better than anything you'd likely build on your own.

Tailscale is disgustingly simple to set up. If you're a product person, it's actually upsetting how easy they've made it to get set up.

An important distinction here is that PanelBear (OP's One Man SAAS) is something I would define as an "analytics" SAAS and as such has requirements that are way above what a typical CRUD SaaS might have.

That's not to take anything away from the excellent writeup, but more so that someone who is thinking about starting a SaaS maybe doesn't jump to the conclusion of "I should go learn Kubernetes".

Then reality hits, and most SaaS's typically only need to handle about ~10 reqs/day, those of the "founder".

This is probably not the best place to ask this question, but as a solo founder or just to reduce costs/time are there some standard free software packages that are used when creating sites? For example most sites need a user sign up mechaism, a authN and authX mechanism to gate access to different pages. Are there open source projects that provide this? Or do site owners develop these from scratch every time?

My infrastructure is $2,800/year.

There are two Servers load balanced with DNS.

Each Server has 3 jails (Nginx, App, DB) and 2 NICs

The internal NIC is for replicating the DB, and for the App Servers to target the Primary one.

Diagram and Configs: https://blog.uidrafter.com/engineering/freebsd-jails-network...

Good on them. I wish I could use K8 as effectively as the author, it is an incredibly overwhelming list and an impressive range of knowledge.

In my situation I am finding the lack of consistent environment a reoccuring issue, the developer environment does not match production. However I kept it simple with Google App Engine Standard and Flex environments, I found the deployment process simple and was enough for me (at the time) - however I am finding we are going to step into dockerland; however I feel like it is very over my head!

I always feel like these write-ups about SaaS's are written by people who make SaaS's for other SaaS's. Application monitoring, email marketing, etc.

In the beginning my startup only had 2 people. A designer (my friend) and me (a developer).

For our frontend we used Webflow. My friend was able to create the entire marketting site, and all the app UI's without needing help from me. Webflow is an awesome tool for that sort of thing.

For the backend, I built a simple Node/Express API and hosted via Heroku.

To this day, everything is still running fine and the API is processing roughly 200 million requests a month. The total cost to host that on heroku is $50/mo.

You can definitely have a simple stack but have it be highly scaleable!

I really enjoy reading these to learn the infra behind one man bands.

Another good read is Wenbin from Listen notes. https://www.listennotes.com/blog/how-i-accidentally-built-a-...

Super interesting! Definitely feels like a lot of fairly low-level tech to have to deal with for a one-person company, but I guess that doesn't surprise me any more :)

For my typing SaaS, I found firebase hosting/realtime DB a ginormous time saver.

And since it hasn't taken off (and probably won't ever), it just costs me a pennies a month since I'm under their free limits, plus the domain.

I was under the impression that Kubernetes was a complicated beast not meant for small teams / startups. What is the value of it in this monolith environment? Is the key to using it in a startup context to use it as a basic monolith auto-scaling orchestrator but no more than that? If you or anyone else here can comment about how to use Kubernetes strategically without falling into an unnecessary over-engineering rabbit hole, I'm willing to learn from you.

Regarding the rate limiting, you're load balancing into nginx services that you've configured to limit requests. Are they synchronizing rate limiting state? I can't seem to find nginx documentation supporting this. What value is there in this style of rate limiting, considering User X can send a sequence of requests into a load balancer that routes them to nginx boxes A, B, and C? The big picture that 3 requests were processed for user X gets lost. Your endpoint-level rate limiting, however, may potentially be achieving the synchronized rates if the redis servers in a cluster are synchronizing. I guess I'm asking about the strategy of using multiple lines of rate limiting defense. Is nginx-level rate limiting primarily for denial of service?

The horizontal autoscaler should be based on throughput rather than hardware consumption, shouldn't it? If the req/sec goes below a threshold, spawn a new service. Can anyone comment?

> From a technical point of view, this SaaS processes a large amount of requests per second from anywhere in the world, and stores the data in an efficient format for real time querying.

That is the closes thing to a number of requests I could find. So this architecture, no matter how solid, is somewhere between „way to large“ and „matches perfect“.

It seems like a solid breakdown on how to deploy your services to k8s and how to properly do CD deployments. But it does never mention whether it actually makes sense at the scale he actually has.

My one-man-SaaS setup:

- A single VPS server to host the app. I love DigitalOcean.

- A single docker-compose file to bring up the entire stack containing the front-end, the back-end and the database.

- Caddy for automatic SSL certificates and proxying.

- JavaScript/TypeScript for building stuff.

- Cloudflare For DNS

I am running a few one man saas's on a 2.99$ vps (1 per project) with php, mysql and nginx.

A lot of people are going to jump on the "he used k8s and he doesn't even work at Google scale!" part of this writeup, but I think it's a perfect demonstration of the concept of innovation tokens [1]. He admits in TFA that clickhouse was the only new piece of tech in his stack, and he was already familiar with k8s et al - so he's able to focus on actually building the products he wants. I could see somebody unfamiliar with k8s (but very familiar with all other pieces of tech in the system they want to build) being able to learn it as part of a side project, if it's the only new thing. Where the wheels come off is when you've never touched k8s, postgres, aws, rust, graphQL or vue - and you try to mash them all together in one ambitious project.

[1] https://mcfunley.com/choose-boring-technology

As a one person company I find it not just helpful but a core principle to minimize the number of stacks/tools/services being used. Overhead of task switching and learning curves.

It only takes two things – curiosity and time.

When you are working on a project, if you hit the edge of your current knowledge / skills, push just a little bit further when it’s something that interests you instead of just aiming to hit the basic requirements / lean on other people.

This minor effort compounds over time; do it for twenty years and you’ll be an expert in multiple disciplines and also an expert in how to tie them all together into one cohesive whole. Aim to be a “T-shaped” person, and just expand over time.

Document everything in excruciating detail - I go so far as to record all the commands I run; and when complete o destroy the machine and start again (or use a separate system) and verify that I accurately recorded every step.

You can add additional text about why you did certain things - and then store the data in a wiki or checked into git or similar so you can find it when you need it.

What has helped me most over the years is working in smaller companies, where you necessarily need to take on more responsabilities.

My first job ~2005 was at a small shop with like 4-5 people and around 20 physical servers under our control and the same amount on-premise with clients (mix of Windows Servers, Linux distros and BSDs). We did have a sys admin person, but he was only responsible for the servers themselves and the base configuration. Everything application related running on it was our responsilibity as developers.

And after that, in the following jobs and as a freelancer, there were I wide variety of things I had to ramp up on quickly. Different build processes, application monitoring, backups, different cloud providers, hidden costs, etc.

Also I have been keeping a "Today I Learned" journal, where I just put small comments and snippets. It is hardly ever any deep insight, but for the most part "to do x in framework y solution z worked". It is also mostly a write-only journal. Just writing things down helps a lot with memory.

The most natural way is to join a startup that is scaling. You can of course learn by doing it yourself on the side, but in practice "learning by doing" on the job is by far the most effective in my experience.

I also hope we don't need to know all this stuff in the future. It's pretty really low-level and it's much better if we can focus more on creating differentiation and building your actual product. (Full disclosure: I've founded a startup that's trying to do exactly that, so I guess I'm biased!)

Addressing the implicit "am I doing career wrong?": there's folks that find working broadly across the whole stack really compelling (I am one of them) that will visibly have exposure to a _lot_ of stuff. Those folks (I am again one of them) are likely looking at your work and worriedly feeling as though they are Doing Career Wrong because they don't have your depth. Apologies if I'm merely projecting; my hope is that this is supportive.

For me the magic trick has always been side-projects. I have a lot of them, and each one is an opportunity to learn new tricks.

(Over time I've learned that it's best to avoid side-projects which have user accounts and store data on behalf of other people, because that's not a side-project: it's an unpaid job.)

Work somewhere where you're the big fish in a small pond. You're forced to wear a bunch of different hats and learn multiple tools across multiple business functions.

For me, it's mostly learning by doing. At my day job as well as with my hobby projects. I initially started learning about the containerization and Kubernetes stuff mostly out of interest before I realized that there is a massive benefit to them, even during the development phase. I guess that's the reason why they are parts of most DevOps toolkits.

While hobby projects can be a great start, the best way to learn is in a team of experience coworkers. The basic concepts of something like Kubernetes are very easy to grasp, leading people to believe Kubernetes is easy and completely missing the giant complexity the system introduces (that's way many people on here say its overpowered for 99% of projects, which I tend to agree on). Even with seemingly simple things like Docker, there is a massive amount of depth that's in my experience very hard to find in blog articles or YouTube tutorials.

That being said, if you have to chance to learn about such things from your coworkers by applying them on your day job, I think the best choice is stell do have hobby/testing projects and combine the learning by doing aspect with some good books. I also recently learned about two YouTube channels that do a pretty good job with explaining such tools and applying them to the real world in a beginner friendly way. [1][2]

[1] https://www.youtube.com/channel/UCdngmbVKX1Tgre699-XLlUA

[2] https://www.youtube.com/user/Kamakazihoer

> I find that I use them once or twice periodically, and then don't touch them for so long, so I forget much of what I have learned.

This is why I "write". I started a decade ago capturing short notes for myself about the technologies I use. Writing it down helps me remember it in two ways. First, the act of writing (primarily by pen) is proven to increase your memory of a thing. Second, I can open my notes for step-by-step reminders.

You don't have to blog publicly. Checkout the Zettelkasten method if you want to use Index cards. Keep a set of Markdown files in a private repo. Whatever floats your boat.

If you keep notes in a notebook I found that labeling mine as "Stray Thoughts" was one of the best things for me. That prevents me from moving away for that notebook trying to categorize my thoughts. If they are just stray thoughts, I can put any random thought in that same notebook. The same thing works in a set of text files or a Zettel.

Nothing like bringing down the entire production cluster and all services with it on a Friday afternoon due to a seemingly innocent "hotfix". Big learnings at the time, but now these make for good stories.

I learned most of these tools at my day job through some catastrophic failures. From my experience, failure has always been the best teacher.

Kubernetes just happens to be a great sandbox for failing hard :) Lots of stories here: https://k8s.af/

However, I wouldn't reach for tools that didn't solve a problem I truly have, be it cost-effective scaling (my day job), or reusing what I already know best even if unconventional (my SaaS).

I guess what I'm trying to say is: focus on solving your immediate problems first with the tools you already know. Your toolbelt will expand without you realizing it.

I've wondered about this as well.

I don't have a complete answer, but so far documenting things as I go about doing them helps, especially if I write down what I tried, what went wrong, what worked and why. Its a lot while starting off, but over time as the concepts sink and become habits, my docs move to higher abstractions automatically and then it is mostly clear. The key words for me are 'train of thought'. The solution (the how) is important obviously, and always useful when quickly referring, but when making bigger changes it is more important to remember th why.

It is hella time consuming, needs dedication and practice and good tools (I couldn't start without org-mode myself)

Work at a company that does DevOps well; this is a pretty common deployment pattern because it’s easy to test and gives a lot of flexibility. Many of the things OP is describing are only things you learn in a role with a production support component, which is where you find the tiny details like your health checks were inadequate and driving error rate spikes in certain scenarios without a specific config option, etc. Many of them are things that only pop up when you have some scale to deal with over a period of time, which makes it hard for hobbyists to pick up if they’re not on a team with prod support responsibilities.

Give yourself an ambitious pet project that requires you to learn and practice new things. Add to it over time so you continue to revisit the project with new requirements. And read all the docs, not just the parts you need to know.

My strategy over the years has been to build a whole lot of otherwise useless side projects with incrementally different stacks, optimizing for doing it well with different technology instead of cobbling them together quickly.

work on an infrastructure team. This is standard cloud infrastructure -- the networking (cloudflare/ingress LBs), k8s, terraform, CI/CD etc etc

I think time will tell that ANY cluster setup is a high risk for smaller businesses. The amount of knowledge needed to run it in production is much higher than to set it up.

I have setup Kubernetes but never run it myself in production. But I work with a Hashicorp equivalent setup with Docker, Nomad and Consul. I also have several Service Fabric clusters. I think it all is just a complete waste of money. Buying services/metal in the cloud or going serverless or whatever is cheaper and with much lower risks for most minor businesses.

Can you point out any tutorials or guides on how to set up kubernetes simply? I'm wondering what the best way to deploy my app with minimal effort is - you make it sound like kubernetes is the answer.

Fly.io is running on Nomad last I heard.

This is a great question. It never ever comes up during the sales process, I don't go out of the way to show them my (lack of) org chart.

If you do customization for larger customers (and you should), like boiling a frog, one day you become mission critical to their business. Once they recognize that, then they will start asking questions. Now they're kinda stuck with you. You did charge enough money, right?

At that point you must appease them with a plan. Have their code and database on a dedicated server instance. Have them pay for it and own the account (you just saved money). Make sure you're using their domain they control. Give them access to the source code. It's on the server, so that's easy. Write up a doc with how to access everything and all the frameworks and tools you use. After this, they will never bring it up again.

Worst case scenario, sell them the software outright. Price it much higher than you think they will pay. Then double that. Trust me, I've done this a few times.

I've made a living running one-person SaaS sites for over 20 years, many of them in the same space as OP (analytics stuff). I can't recall a customer ever asking how many employees there are. It just doesn't come up. I don't think small business customers care. Maybe it matters for more enterprise salesy type businesses.

I run a b2b saas that is used daily by some fortune500s. I have been going through rigorous vetting processes but have only once been asked about this. I lied and told them it is no problem. I usually pretend my company is larger than it is (me).

It gives me a headache now and then, and am now in the process to get someone else onboard.

I've had some cases of customers asking "what happens if your company dies" for cloud services even knowing the company I work for is +30 employees, so not just for "one person company", some people seems to care about what happens to their data / service. I'm just wondering if that comes from conferences or talks that points out that a lot more/less.

> Where the wheels come off is when you've never touched k8s, postgres, aws, rust, graphQL or vue - and you try to mash them all together in one ambitious project.

In my experience (both myself and observing others) this is the cause of lots of side project (sometimes even startup) failure. Lots of people choose a tech stack that's far away from what they've worked with, so they never get past the "read the docs and try to get anything working" stage. For a real chance at completion it seems like the recipe for success is choosing a stack that's 1ish derivative away from a dev's competencies so they have a new and exciting thing to learn, but are able to continue progressing and adding value.

I am also a person that, prior to using Azure, was an absolute "Kubernetes is a big waste of my time and I'll just skip it" person. I wrote it off as predominantly "resume-driven". Now, having used Azure for about a year, I'm rewriting all my Azure infra to use AKS to better insulate me from the inevitable issues that come up when I GTFO of the Azure sphere as soon as our credits run dry. And, what I'm learning, is Kubernetes is a just-fine deployment/orchestration/management tool for containerized infrastructure that is _not_ a massively complex microservices infra. It's just a more streamlined approach to scaling and managing cloud-agnostic tooling/containers.

Innovation tokens is such a fantastic concept. When I brainstorm products I like to make a simple plan, and then map out "Whats new to me" to help me decide what I am really trying to accomplish. Prototyping a new idea is very hard and requires a lot of iteration, using a new tech complicates that even if its better. Alternatively, if the prototype is purely for fun, then new tech can be a great value add, as even if the prototype takes a turn for the worse (i.e. too hard to finish), you get satisfaction from learning some new tech. I've noticed since being very intentional about it, side project work has become considerably more enjoyable.

Picked up k8s as part of my a side project, that become my startup. I would say if you know docker, it's not hard. Especially when using managed google style. Setting up your own k8s cluster is a whole other thing..

That said I agree with the innovation token concept. None of this junk makes you money, solve a problem first.