I Don't Maintain My Homelab(cleberg.net) |
I Don't Maintain My Homelab(cleberg.net) |
Edit: zero minutes old already downvoted.
Practically Luddite
So using AI is not the point of the article but neither was it mine.
My point was I also attempt to implement homelab automation rather than manual maintenance, and I listed a few things that are onerous to do regularly by hand just like the article.
But I totally expected people to just skim my message, see “AI” and dismiss it, so I’m not terribly upset.
The rest of the lab is a few ephemeral instances on Google, with dual A100s that spin up when I need to train things.
I put Ubuntu on the old beast, and never touch it. If the power goes out, it automatically comes on and Docker launches all the services when it comes up.
About the only thing that needs watching is the tiny SDR radio plugged into it, which I use for pure random numbers and talking to it with a hand held radio from the other house. Sometimes I have to unplug it and then plug it back in to get it back into service. No amount of finagling seems to fix it from software.
You are an interesting person! We would be friends IRL :-)
May I ask what you use the pure random numbers for? And what you use the radio link for?
I've built an SDR radio stack to integrate with a single pane navigation and chart plotting app I'm building for my new company. You use the radio to talk to the local agent (I just finished a submission to the Gemma Challenge on HF to learn how to train models). Wake words show up on the glass, for security. Been working on training a small model to do agentic controls, including changing autopilot and switching displays when you want to see other content on the screen. I've been working on isochrone routing and have it working well now. Waiting for Fabel to come back to continue the work...
Everything is here: https://deepbluedynamics.com. It's just me as an LLC. No VC. No backers. No users, yet. Few stars, but 100% written by me, and most of it is open source. FWIW, I've been around the block and I'm getting old now. So the radio helps with arthritic fingers! :) The radio stack stuff lives here: https://nuts.services/sdrrand
I'm also integrating the voice stuff into Hyperia, my terminal emulator forked from Hyper Terminal. It's on Github. Hyperia is also agentic controlled, so I can talk to it by pane name to inject text into the prompts. This lets me get up and roam while I'm roaming around. I'm remodeling my front house, and for a while I had one of those smart lights turning red or blue when things happened in the sessions, but I want something I can talk to without having to click and type. I use the Windows transcription feature a lot, but talking on the radio to it is much easier.
Oh and to answer your question about random numbers, I use them for various Monte Carlo based approaches. For example, I inject those numbers into training runs, rolls for the iChing (helpful for feeding agents for decision busting), and seeding other things like simulated wind speeds and current (in the sim beause I'm not sitting on a boat). I even use it for sampling documents I've been indexing (cut up a bunch of business books and have a local model use them for reference).
I have the SDRrand sources wired into almost everything now that needs randomness. Is it necessary? Maybe. We only get pseudo random numbers from computers, so I just attached to the purity of it, if anything.
After I set it up and stopped fiddling with it it's just run flawlessly for the last 6 months.
Building/tinkering/playing around is fun, but once you are actually self-hosting services you rely on, it needs to "just work" or you will eventually burn out or lose interest. Especialy when you take on more users than just yourself. The day my wife cancelled her audible subscription because audiobookshelf was just as good (IMHO better) was a good day, but that only happens because it is stable/reliable.
Indeed. And if you never test your recovery then you don't actually have a workable backup.
What this skips though is the complexity of services like NextCloud (stuck in maintenance mode again?), Immich (needs a compose file edit?), MineCraft worlds (Dad! my client is on another version again!), (dmn) AlbyHub (needs re-login and closed its channel).
But to be fair this is really getting quite minimal these days indeed. I didn't really realize it but I too have a mostly hand-off home-lab... Ok, then it's not really a lab anymore, its more "stable home-infra" ;)
1. How often do I have to touch it during the next ten years?
2. How many of the times that I have to touch it are because I decided to do so?
3. How much pain is it to fix and understand if I had my mind erased?
* Docker Compose files and various folders for containers live on an NFS share
* SQLite and other databases run off a local SATA SSD for speed and reliability
* Cronjob tarballs the critical stuff nightly and throws it on another NFS share to get ingested into Backblaze B2.
Now I just get to kick back and actually experiment with new things instead of babysitting a convoluted Proxmox upgrade or shunt onto a new container standard.
Does it run rootless? Not atm (blame FreshRSS, my sole holdout). Is it super secure? Probably not, but I’m not doing anything goofy like mounting the Unix socket into a container at the very least, and the server credentials don’t work anywhere else should it get popped. The blast radius is contained, and that’s more important to me than Enterprise-grade security for my homelab (a la Wazuh, another backlog project TBD).
It doesn’t change.
Many people keep swapping gear in so they can learn BGP on Cisco edge gear or run clusters on salvaged IB.
OP is not that person.
It's been normal for me for the past 3 years thanks to using NixOS for all server infrastructure.
Helps that things are really easy to test too, spin up a new test VM with your new config and copy of real data, check if it works, then apply the change to the real hardware and you're good to go. Alternatively, do it live with a copy of real data, then rollback in case it doesn't work.
Switched to NixOS a few years ago, and I can't overstate the amount of peace it has brought to my life. It just takes so much stress away, compared to everything else I've used before.
My only criticism is that the Nix language is not super ergonomic or easy to learn. But with LLMs nowadays, even that is barely ever a problem.
I'm not sure what's here to talk about. Things break. We don't have to overthink this. But if you want more predictability, stable distros exist.
Don’t super care about updates. If it isn’t too ancient and not internet facing then it’s probably ok
Yeah, right until the moment it bricks after an update.
Longer term goal is a sleek plug-and-play box anyone can connect to their ISP modem with minimal technical knowledge.
I'm currently running it on a Aoostar WTR Max NAS with my AT&T connection. Got another NUC connected to a Spectrum modem. My goal is to be able to flip back and forth between the two with a backup bundle within minutes.
Considering breaking up the router and app server functionality so they can be run separately. Another idea is to use custom a 3D printed case with Framework laptop motherboard and battery, switch, and wifi AP to make a true all-in-one box. I currently need an external switch, backup battery, and wifi access point.
Once the system feels mature, next steps would be things like federated tailnets with friends and family for things like distributed backups, compute/GPU, CDN, social networking, etc. Hoping that decentralized model training is cracked by someone at some point.
From a coding perspective I'm hoping to modularize everything (since it's NixOS) and add thorough testing and hardening. It's already relatively modularized considering it's built on Nix flakes.
Technology has come along way. But I think that in tech we should be careful to not fall prey to monkey see monkey do.
We should not be deploying technology in our homes to "mimick our employers"
Remember they are miserable for a reason.
Frankenstein couldn’t build a monster without influence. Same thing here.
“CCNA? I’ll show you CCNA…”