Raspberry Pis are notorious for SD card corruption with unknown causes. In this era you might replace it with a mini-PC (NUC style).

Classic power failure and raspberry pi with SD cards. When I last set up things for my parents, I simply imaged a bunch of SD cards and had the data stored on an external hard drive. I live in the US and they live in India and this is for their RaspBMC. With the SD cards they just swap a broken one with a new one and call it a day.

Been 12 years or so now. Went back to their home a few years ago and everything but the opensubtitles integration works. I was surprised. I could never recreate it now. Even has one of these flirc setups so the remote works.

India has unreliable power too so the SD card death was inevitable. I made a few more images and called it a day.

My Raspberry Pi 4 has been used to run both a Jitsi and Trackmania server over the years. I never considered using the SD card since it can boot and run from a USB 3 flash drive. Later on, I picked up an Argon One case to add a SATA SSD.

Thankfully, modern Rockchip SBCs tend to come with NVMe slots.

The failure everyone plans for is the disk; the failure that actually gets you is the boot media. The fix is making a rebuild boring, not making the hardware immortal.

I use a Waveshare hat for my Pi 5s. It does PoE and has a spot for an SSD. What I particularly like about this setup is the single wire running to each board, looks very clean.

I'm not a sysadmin but I often hear about server failures due to hardware just wearing out. Is there an expected shelf life of the hardware? I would expect for sure on the storage devices. So if that were the case then why isn't there some sore of daemon or system software service that tracks the expected life of the hardware? I would expect that the software would start showing warnings of imminent useful life of devices to expire.

If you're using a USB adapter on a Pi 4 you can use raspi-config to enable booting from an external drive and not worry about SD cards at all

> I chose to enable zram for swap

swap is used when you run out of ram. using a ramdisk for swap ... You see how that doesnt make sense?

> Not eager to ruin the evening by starting a debugging session

Use a clanker. Nothing has brought back so much joy into homelabbing for me as having LLMs handle the boring stuff.

I launch it in opencode on the system or elsewhere to ssh in (I think giving it KVM is an overkill but an option nonetheless) and tell it to fix things. Not just ask questions and generate configs. I have backups, let it rip. They have gotten shockingly good at it even when they write bash wrappers just to catch some logs.

this would have been the perfect era for home servers - but damn Ram is expensive.

every other component is cheap enough but not Ram.

The RPi Compute Module is mush better with on board flash. Completely removes the SD Flash Achilles heel of the RPi.

I've been running a CUPS server on a Pi Zero W since 2017. I suppose I should back up the SD card.

Back in the day I ran s××× coin miners off flash drives without any apparent problems.

Teslas though... ;)

What a great example of nice writing. I just liked reading it.

If i'm reading this correctly, can run Immich from a Raspberry Pi 4? That's pretty cool.

I bought all the stuff (new case, networking card and a Jonsbro N4 to repurpose some intel i7-9700 era processor and motherboard, but didn't get it setup yet, as I was starting to rethink the power consumption....

Also looking to use this rebuild as an opportunity to learn Nix.

Let me know if you all have some good article references, to point me at. I've been doing some research, but could always do more.

I may end up putting a low power NUC in the case, or something. I don't know...

A few years ago I decided to give up on sdcards for my main 24/7 homelab server.

I now use a small SSD connected via USB3 and a slightly modified version of this : https://framps.github.io/raspiBackupDoc/introduction.html

...to take snapshots to my NAS periodically.

I keep a similar small SSD in the drawer ready for a restore and swap if the main system drive fails.

Lack of proper IO connector is the reason I abandoned Raspi. SD card crashes are nearly a guarantee. Total pita too since there’s no display during the BIOS.

Another useful advise would be to keep your boot drive 60% used at max and run TRIM after freeing a huge chunk of files to let the wear leveling algo do its job.

And every few years reinstall the OS to free up unwritten cells.

Why not get a Beelink N95 and call it a day?

There is for hard drives - SMART - the problem is, unless you're an enterprise (and let's all stop and laugh because we KNOW that most "enterprises" run at least some hardware until the only person who even knows it exists is long in the grave) you're not going to do preventative replacement.

And especially for home use, where nien 9s is more realistic than nine 9s, you're better off having a solid backup strategy and a source of spare parts.

My main "tank" server is 16 years old this year, and has blown out one motherboard - it was a bit annoying to be "offline" for a few days as a replacement wandered in from eBay, but it worked. If it happened today I might import the zpool into another device temporarily - or permanently, who knows? This R510 probably takes more power than a datacenter.

I'm running my own home server on an old 2014 Mac mini, 24/7 basically. It both has a SSD and a HDD (old "Fusion" disk type of machine, but I use both parts separately, the SSD part for the OS, the HDD part for the data.)

A few days ago the HDD had its 100,000 Power_On_Hours anniversary. :) According to SMART this is way past the expected/ designed life time of the disk (the normalised value reports "1" since a few years when the disk hit about 42,000 Power_On_Hours)

So you never know. It either works or it doesn't. You have backups for the worst case.

My primary home zfs server is from 2010, running 24x7 since then. It has a 4-way mirror zfs pool for data. Over the years I have replaced two of the four drives, so two are still original from 2010. Everything else on the machine is from 2010.

You can get unlucky and have components fail quickly, but usually if they last, they'll last for a long time.

But trying to run a server from an SD card like in the article, seems silly. False economy, just get a good SSD.

For drive health monitoring there is SMART, but IME while it does monitor drive usage and lifetime, it rarely predicts catastrophic failure since that happens suddenly. Both drives that failed in this system went from healthy (according to SMART) to unresponsive overnight.

I run a fairly large homelab (probably ~10 nodes when including OPNSense routers, backup NAS at our cottage, etc.), and hardware failures have been incredibly rare, especially compared to what it used to be a couple of decades ago (I had quite a few motherboards fail due to blown capacitors due to the stolen electrolyte formula scandal).

Hardware failures tend to follow a bathtub curve: there are more frequent failures early on due to manufacturing defects, then they become rare until you reach the end of the natural lifespan of the equipment. That is shorter for some components (spinning rust HDDs, fans, CMOS batteries, power supplies in noisy environments), but extremely long for others.

> So if that were the case then why isn't there some sore of daemon or system software service that tracks the expected life of the hardware?

Some linux distribution will regularly check SMART data from disk and warn you if some threshold has passed (like number of corrected errors, spin up/down, total hours etc).

That said, if we talk about consumer hardware in a home lab or the like, of the disks that I bought in the past 15 years, only one broke down, and it did fairly quickly so my suggestion is to use redundant storage, make backup and change the disk when it breaks down. (Enterprise disk and usage is another can of worms of course).

Life expectancy really depends on how the hardware gets used. I run a small fleet of servers that are heavily CPU / GPU loaded, and the most frequent hardware failure I see is power supplies, with RAM as #2 and spinning HDDs as #3. Each of these (on enterprise-grade gear) has some amount of reporting / monitoring, but it's rarely the case that I get any advance warning before a failure. The RAM is probably the nicest, as it sometimes starts with correctable ECC warnings. Nearly all of the PSU failures have been sudden deaths, with the only saving grace being N+1 redundant supplies usually keeping the victim server alive.

Hardware wearing out is not common at the individual component level. It only becomes common when you are running a large number of servers, those individual small probabilities add up.

About the only thing that regularly wears out is spinning drives, and those have SMART data to give you an idea of when they'll die, but generally yif you're actually committing to high uptime you'll have redundant hardware so any one server failing doesn't take the service down, giving a bit of a grace period on hardware failures

SSDs have a fairly linear wear-out mechanism although the exact end point is unpredictable as always. HDDs fail randomly or from mechanical shock.