Beware that this script only uses rsync with the "--archive" flag.

This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises.

rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies.

Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most file metadata and they usually need many non-default flags to make complete file copies.

Nevertheless, rsync is the best of all alternatives, because when invoked correctly it accurately copies all file metadata even between different file systems and different operating systems, in cases when other copying utilities may lose some file metadata without giving any errors or warnings for the users.

  echo "Syncing ~/.ssh directory"
  rsync --archive --delete --xattrs ~/.ssh/ laptop:.ssh/

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below.

- Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong.

- As an example of the above: This script notes that it fails if a backup takes more than 24 hours.

- The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok.

I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/...

Features of it are:

- As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination.

- It goes out of its way to detect when rsync has failed and log that.

- It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases).

- This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days.

- In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails.

The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.

The first two lines of the script are already wrong;

    #!/bin/bash

Should be:

    #!/usr/bin/env bash
    set -euo pipefail

That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

rsnapshot[1] is what I used on FreeBSD with a snapshot. It is like a well-tested version of the author's rsync and ssh blog post. I have a blog post here[2] describing my setup. It saved my bacon multiple times. Also, rsnapshot works in pull mode, so no client is needed on my Linux/macOS desktop or server except ssh and rsync.

[1] https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-rsnapshot-filesy...

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encryption passphrase) mount the backup archive as a read-only file system through FUSE and access it through normal file navigation. It is impressive.

I use Restic (https://restic.net/) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends.

Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455

Perfect? Ha.

Wrong ps grep usage. MacOS specific. No shell quoting.

At least he doesn't advocate rm -rf / by an incorrect usage of --delete.

It's nice that the script works both ways, but keep in mind that it's always best to pull backups from a remote system, so that an attacker in the production system can't log into the backup system and delete the backups.

If your prod system(s) is/are logging into the backup system, then you have a big problem because if any of them are compromised, they can wipe out / corrupt / etc all of their own backups, and possibly the other server's backups as well (if the same user account is used for all of them.) This problem goes away if your (isolated, hardened) backup server logs into the prod systems. Of course, the inverse problem is that if an attacker manages to get into your backup system, then they also have (at least read) access to all of your prod systems!

What seems like a long time ago, I was working at a place that used Linux, but not really any Linux experts. They had backup scripts that assumed (and did not check) that you would connect an external drive. If you didn't, the script would gladly copy files to /mnt/usbdrive, and if the server was full enough, backing up itself to itself would fill the drive and bad things would then happen.

There was no email notification if the backup completed or failed, or even started to run. There was no notification of how long it took, how much data was backed up, etc.

There were plenty of incidents of data loss.

For rsync tasks that require scheduling and need to ensure only a single instance executes, I personally prefer creating a systemd service (type=oneshot) and have that run under a systemd timer (set OnBootSec, OnUnitActiveSec)

https://wiki.archlinux.org/title/Systemd/Timers

EDIT: I'd love to hear if anyone knows of any downsides, edge cases for running rsync under systemd timers.

> # avoidng collisions with other rsync processes

Use https://github.com/instacart/ohmycron

> MONTHROTATE=monthrotate # use DD instead of YYMMDD

Use https://rotate-backups.readthedocs.io/en/latest/readme.html

> $RSYNC -avR "$SOURCE" "${RSYNCCONF[@]}"

Create a one-command script with the hardcoded rsync command you want to use and replace the directory to sync as a command-line argument, e.g.

  #!/bin/sh
  rsync -avR \
    --delete \ 
    --exclude=/Volumes/Raid/.DocumentRevisions-V100 \
    --exclude=/Volumes/Raid/.TemporaryItems --exclude=/Volumes/Raid/.Trashes \
    --exclude=/Volumes/Raid/.apdisc \
    "$@"

> $DATE >> $LOG

Use logrotate

> mail a report

Use Cron's built-in mail feature

I didn't read the whole article but the reason I like rsync is that I get files on a normal filesystem. I don't need any special backup software to read or access the files.

Restic is currently the best solution in my view.

The backup script is much simpler, the repository is properly encrypted, takes snapshots, dedup, mounts the remote, integrates with backends, has clean output, and various useful features for working with repositories.

Rsync over SSH is not even encrypted at rest.

My go-to backup solution, that can also manage rsync over ssh (among plenty other things), is synbak[0]. Short wrapper script to automatically mount the backup medium and it's really simple to (automatically) run a backup job. I use encrypted (LUKS) USB drives for that at home. Highly recommend it.

For encrypted backups to e.g. a NAS, Duplicity[1] is my go-to choice (full backup every month or so, with incremental backup every day inbetween).

[0]: https://github.com/ugoviti/synbak

[1]: https://duplicity.gitlab.io/duplicity-web/

Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part:

    pro_on=$(ps aux | grep -c rsync)

    # if someone is using rsync
    # grep is also producing one entry so -gt 1

`grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so:

    pro_on=$(ps aux | grep -c rsyn[c])

Also, this:

    minutes=$(($minutes - 1))

Can be more easily written as:

    ((minutes--))

Guess they never heard of rsnapshot?? [0]

[0]: https://github.com/rsnapshot/rsnapshot

I love the irony of a document about a "perfect" script with a grammatical error in the very first word of the title.

Does anyone have experience with Duplicati [1] or recommend it? I am tentatively looking to make the switch to something a little more sophisticated than manually tar-balling + rsyncing backups.

[1] https://www.duplicati.com/

About 10 years ago, I was earning my CS degree, and my Intro to Unix teacher was showing us how to use rsync. He somehow made an error that synced an empty directory to his home directory, deleting everything from his home directory.

The lesson I learned in that class was to not use rsync, sadly.

I think there's a bug in this; the proc_on calculation doesn't account for multiple lines, it just calculates if the output of grep is greater than 1? grep -c may not be available on all systems, so a more portable way to do this is piping to 'wc -l'

Honestly, I just use duplicity. It is easy to test backups and has a nice GUI if you're that way inclined. I'm too old for bash scripts when the stakes are a lifetime of memories or critical data at work.

I wish more of these backup solutions (even simple scripts like this) would include zero trust features.

I'd like to have a backup system that I didn't just kludge together myself over a weekend that accomplishes both delta change detection and rotation as well as allows me to keep my data encrypted with a key only I control.

Haven't read the page yet but the header image struck me as interesting.

"Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there.

If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)

The script could use --rsync-path='sudo rsync', so that your rsync backup script can access root-owned files on the remote system. It looks like you can add that to the RSYNCCONF part of "subpart of part 2".

This SO post goes over the semantics of `--inplace` and `--no-whole-file`

  --inplace                update destination files in-place
  --whole-file, -W         copy files whole (w/o delta-xfer algorithm)

It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container.

https://superuser.com/questions/576035/does-rsync-inplace-wr...

Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same.

MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync`

   rsync  version 2.6.9  protocol version 29

This is kinda a large footgun.

> - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails.

I've always thought this isn't the right approach.

How do you know if the email server is borked or you commented out the script in cron to debug it and forgot to put it back in?

Either there's a weekly status report to tell things have been green or you could place cron checks like healthchecks.io (You can self host it.)

Also it's much better to use 'zfs send' with large volumes to backup if both ends have zfs as zfs knows which files have changed and it doesn't have to scan for what has changed on each go as any other tools do like rsync.

https://arstechnica.com/information-technology/2015/12/rsync...

I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105

It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.

If I may add a few critiques...

  # Make sure no one else is using rsync
  pro_on=$(ps aux | grep -c rsync)

A better way to do that is with the flock utility.

  (
    flock -n 9 || exit 1 # Critical section-allow only one process.

    ...single thread shell script

  ) 9> ~/.empty_lock_file

Note that the flock utility is specific to Linux, but POSIX mkdir() is atomic and could be more portable.

  "${SOURCES[@]}"

POSIX shells do not support arrays. Iterating with read over a here document is more portable.

  minutes=$(($minutes - 1))

POSIX is specific that the $ prefix on a variable name can be omitted in an arithmetic expression.

  ECHO="/bin/echo"

Many shell scripts never use echo, and this is a good idea. 'NEVER use echo like this. According to POSIX, echo has unspecified behavior if any of its arguments contain "\" or if its first argument is "-n".' http://www.etalabs.net/sh_tricks.html

Perhaps use this instead, in a subshell to avoid stomping on variables:

  myecho () ( z=''; for x; do printf "$z%s" "$x"; z=' '; done; )

#!/usr/bin/env bash and set -u are always good ideas.

There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures.

Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted.

When your script does fail unexpectedly, you'll want to re-run it with at least tracing enabled, so the third line should be something like

  [ "${DEBUG:-0}" = "1" ] && set -x

  dont_mind_failure || true
  important_process

I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues.

Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.

Agreed on fail on error, but the first one is needlessly pedantic.

Find me a single Linux distro where bash, if installed, is not available in /bin

Not using “env” isn’t “wrong”. It depends how many platforms they want to supply

Bet yes pipefail and nounset should definitely be set.

Make sure your borg repos are copying properly and in full!

I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair"

Course following the 3-2-1 rule you're probably good, but aye I'm treating borg repos as delicately as I would a striped raid now. I am also monitoring that `borg check` actually comes back successfully before considering the backup complete too :)

Also if you're in a small company, start up, etc.. Set yourself a 3-2-1+ backup system. + is the miracle backup that makes you look good.

Fully agree, Borg is awesome. I wrote up how I'm using it here[1]. In short, borg backup to a local machine, and that machine uses rclone to copy the backups to an S3 bucket off-site. I've had multiple occasions to restore stuff successfully, and I never have to think about whether my data is retrievable.

[1]https://opensource.com/article/17/10/backing-your-machines-b...

I used to do something similarly complex to the OP with ssh and --link-dest to make it share inodes so I could easily keep N days of backups with file level deduplication [1].

Then I moved to Borg and haven't looked back. It does the same end goal in a better way, is way faster, and easier to work with overall [2].

[1]: https://nuxx.net/blog/2009/12/06/time-machine-for-freebsd/ [2]: https://nuxx.net/blog/2019/11/10/using-borg-for-backing-up-n...

The cheapest I can find on https://www.hetzner.com/storage/storage-box is 3.49 Euro. I assume your talking about the bigger plans with price / TB?

I used rsync 20 years ago, switched to Borg, then switched to Restic. It’s nice to send backups to destinations like B2 or S3 without any special requirements. But the Hetzner Storage Box is good deal.

Now all I need is better client side software for these things, something I could use on my moms computer. Vorta is not that good, although I commend the effort.

Until the UI side is sorted, my moms laptop stays on Backblaze.

As a small film production we have approx. 120 TB to backup. Thats a lot of money per month, if you check AWS it is a fortune (half the value of a car) per month.

>Storage Box backup product

The script here really only makes sense for servers you physically own. You wouldn't accept SSH access from a key located on a plain VPS, right? Also this script doesn't seem to encrypt the data at all!! Very dangerous on a VPS.

"Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it."

I'm still not clear - does the Hetzner storage box have the borg binary installed on their end ? As in, one could run:

  ssh user@hetzner borg --version

... or are you accessing borg over an sshfs mount, etc. ?

Asking for a friend ...

Is it possible to use it as a read/write mounted drive on macOS?