We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.

Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.

On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.

IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.

With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.

Kopia does not let you pipe into it without intermediate files, or at least not in a friendly way I could figure out.

Kopia seems quite designed around the backup server being the place where the config is set. IMO it's harder to configure if you have your own declarative config tooling (like Ansible, NixOS, etc)

The nicest way I found to ignore files declaratively is to configure a ignore-file pattern in the repository policy and then declaratively create `.kopiaignore` files.

The Kopia server is convenient if you have a server at home and want to back up laptops to it, with them being in a shared deduplicated repository without having to trust the clients not to get pwned and wipe out the backups. However I'm not aware of a nice way to then transfer those backups to aggregate them into a bigger repository. (So I just rsync them off-site and don't worry about getting deduplication with other backups) Also learnt the hard way that you can't set a bandwidth limit when backing up to the kopia server, which can tank wifi performance on laptops during the backup.

I am thinking of switching to Restic as it seems like it is more suitable for people who want a tool at the complexity level of tar/git CLI with declarative config, rather than something imperatively configured.

The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything.

The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files.

The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously.

And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.

When I was a teenager, I was the reason for data loss for my dad, twice. Both times it was because I was re-partitioning a hard drive to install linux.

You would think that taught me a lesson about backups, instead it reminds me every now and again to be grateful for an awesome dad and aspire to handle situations with my kid similarly :)

  We are not in the backup business.  We are in the restoration
  business.

    [1]: https://github.com/jimsalterjrs/sanoid
    [2]: https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs_proxmox/

All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.

I plan to implement it like this:

  vps1:
  - restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
  homelab:
  - backrest container, to back up volumes/, do prune/check, replicate repo to offsite
  - rest-server container, will store backups from vps1, homelab, offsite
  offsite:
  - restic container, backs up volumes/ to homelab every 24 hours
  - rest-server container, store copy of backups from homelab

The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost

This is an empirical claim, but is it grounded in reality?

99% of people don't backup, and that's probably the right choice because the risk is low, they can't meaningfully improve their restoration rate themselves and they don't care enough about their data to classify its loss as catastrophic.

When a person hears someone talk about the importance of backups, but also haven't heard friends/family suffer this date, they will rightfully ignore this warning.

Or do we all follow the best practices as it relates to backups, exercise, sleep, nutrition, accounting, house maintenance, ... Etc.

I do not. Backups aren't near the top of that list.

This stuff just isn't very important to most people and that's okay.

Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.

https://www.jwz.org/doc/backups.html

(maybe copy/paste this into browser instead of following link, since referrer from hn apparently does something)

Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK.

Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself).

They ran via VPN out of our data centre for most of the pandemic.

That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)

https://relax-and-recover.org/

This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file.

I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions.

I finally have rsync configured for some scratch temporary files.

I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most.

My replacements do not like the complexity (and I am retiring).

GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive.

The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention.

My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).

The best part: it is just files, not some proprietary archive. I have high confidence in this. I can restore with the most rudimentary tools.

This setup has worked for me well on linux, windows and now mac. I migrated to new computer&os by restoring the backup mostly.

[1]: https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot...

We finally got a nice setup with CloudNativePG + Barman. This allows for point-in-time restores, but there were a lot of lessons to learn along the way.

- The various types of (database) backups (logical, binary, onsite, offsite, snapshots, write-ahead log...) in combination with the various types of data (database, files, cluster configuration...)

- In our earlier approaches, we tried to preserve the old database volume if it was not corrupt, and use that in our restore. This caused so many complications, because you are fighting the recommended approach. So now, when we need to restore, we always restore from backups and the 'live volume' is dropped.

- For a restore, we just spin up a completely new Kubernetes cluster, instead of trying to restore in-cluster. This is a lot easier.

- Many object stores allow for retention periods, which you can put to good use to prevent malicious or accidental removal of backups. HOWEVER, not all of them are really 'locked'. In some services, you can still override the lock with a forced delete; in others, you can still remove the project holding the storage buckets, which will delete the buckets, and so on... so test those things, instead of just blindly depending on a 'retention period' claim.

- We now automatically run a scheduled restore with verifications on a weekly basis. This requirement does shape your environment, so keep that in mind! There is also the question of how you can reliably and automatically verify that the restore restored the latest data (of a live prod environment). Various solutions exist here, but most are not very elegant!

Honestly, this is only worth it if you are already handling sufficient volume. If you are just starting out, then the easier approach is to just go with a hosted database, which will handle backups and point-in-time restores for you.

- do several backups (not using the same hardware brand or host if you do it online)

- check your backup (if something was wrong, you have another backup from the first rule)

- do not store your backups at the same place (if your place get flooded or burn, it will be pretty handy)

I wrote about this here [1].

[1] https://staticnotes.org/posts/cloud-backups-with-restic/

and a privesc also.

Also "There are just two types of equipment; those which are already broken and those which will be broken".

I've found a good way to point out anyone thinking right direction not to trust their valuable data in any single entity, product, location, etc.

When I heard it decades ago IIRC 80's Nokia Data Unix courses I went. The distinction made Backups is just 1/3 of the triad Redundancy, Backups, Archives.

1) Redundancy is what you get using RAID, multiple network connectivity etc. ie. good for avoiding single point of failures and avoiding loss service or product availability.

2) Backups are for recovering lost data needed returning lost running known state right before fault or some time state before what backup schedule and used rotation cycle can provide.

3) Archives are meant to saving valuable data not expected to change or active use and but there is need saving much longer periods.

Not seeing and understanding usefulness of difference between these seem hard some people I met over decades. Which then led to confusion, not knowing what they are up to with their data and also often later disappointment when they found out that for example backup programs are not great if you expected archiving instead.

One day I found an interesting new command called DriveSpace, and it looked exciting. I ran it on the C: drive and then got cold feet and cancelled the operation half way through. Needless to say this was a disk compression application, and I had just unknowingly compressed half my hard disk. Took a long time to figure that one out and I was in trouble with my Dad for bit, but we called a few experts we knew for advice and he included me along the journey to fix it. Something I always try to remember when my kids do something similar.

Moments later their dad (presumably) comes back and asks for his phone back and sits down with their food. The kids are eating and the dad is fumbling with his phone with odd looks and I can see his frustration growing.

Then out I hear something like "What did you do to my phone?" repeated loudly over and over. "You erased my phone, it's gone."

The small children (guessing) had figured out how to factory reset his phone.

My thought at the time was something like : How many times did I do something as a toddler I don't remember to piss of my dad like that and that he never got mad or brought it up later (Lots of times knowing myself).

Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?

https://www.postgresql.org/docs/current/backup-file.html

Over 40 years (even with changing disks, upgrading PCs, etc.), the chances of you ending up with your data intact is 0.99^40=0.67. May not seem too bad, but this is just a hardware failure of your main drive we're talking about. When you factor in all the other types of screwups that happen, the annualized failure rate will skyrocket.

So a more conservative version of this quote is that a 20-year old today is very likely to experience data loss over his lifetime.

Thankfully with cloud being ubiquitous at least photos are given one copy in the cloud, but IMO even that is risky. People have different appetites for risk, and that's OK. But over the last 20 years the all of our data has become pretty important. I don't know how many people could stomach the loss of important documents, photos, etc.

Maybe they just haven't talked to you about it? Most people I know have suffered some form of (partial) data loss or another, and lived with it.

It’s still vitally important if you are self hosting your data.

This depends on your viewpoint. Using gmail for example is an implicit backup of your e-mails. Uploading your pictures to Facebook the same. The move from local, private compute and storage to the cloud solved this issue for many to a degree that it is now less important.