"So that a truncated partial download doesn't end up executing half a script"

"So that a truncated partial download doesn't end up executing half a script"(tailscale.com)

78 points by obi1kenobi 2 years ago | 81 comments

obi1kenobi 2 years ago |

The shell script starts with the following comment:

    # All the code is wrapped in a main function that gets called at the
    # bottom of the file, so that a truncated partial download doesn't end
    # up executing half a script.

arp242 2 years ago | |

Most of these scripts have been doing that for years.

obi1kenobi 2 years ago | | |

I'm sure of it. And yet based on this rapidly getting to the front page, it seems like many of us are part of today's lucky ten thousand: https://xkcd.com/1053/

Might be more than ten thousand, even, based on the reactions :)

TrianguloY 2 years ago |

Another trick I've also seen is to enclose the whole script in a code block

    { 
    code 
    }

That way, if the file is not fully loaded, the block will not end and the script will not parse

noisy_boy 2 years ago |

A simple (not perfect) approach could be to have a comment containing a "unique" string on the last line and grep for it as the first check to ensure that the entire script has downloaded.

    #!/usr/bin/env bash
    
    set -u
    
    grep -wq '^# asfewdq42d3@asd$' $0
    [ $? -ne 0 ] \
        && echo "script is not complete - re-download" \
        && exit 1
    
    echo "script is complete"
    
    # asfewdq42d3@asd

kevincox 2 years ago | |

This doesn't work for something like `curl ... | sh`

bombcar 2 years ago |

I've seen scripts (self-extracting archives for Linux, for example) that checksum themselves either by some trickery, or just ignoring the first line after the shebang (which itself is the computed checksum of the rest of the file).

Scaevolus 2 years ago | |

Incorporating an MD5 quine into a shellscript would be funny.

seeknotfind 2 years ago | | |

And a sha256 quine would be terrifying. :)

kazinator 2 years ago | |

The problem is, the wrong party is doing the check (from a security point of view, not integrity).

When we download a script from a remote domain we don't trust, we have to validate its checksum against the known one; we can't leave that to the script, which we don't trust.

joshspankit 2 years ago | | |

In this case we’re specifically talking about the possibility of a truncated script from a trusted source

kevincox 2 years ago | | |

99% of the time you are downloading from a domain that you do trust. This check is to detect corruption, not malice.

But yes, if you were downloading from an untrusted mirror you would want to check the signature or trusted hash before running the script at all.

cassianoleal 2 years ago | |

That can be useful against download corruption but wouldn't do much against an actual attack (in this case, the attacker can just update the checksum).

JonChesterfield 2 years ago |

This probably means you can edit the script while it's running without it falling over confusingly. Might cargo cult this pattern - I'm very prone to editing a build.sh while it runs.

exe34 2 years ago | |

Shot myself in the foot so many times with this!

kseifried 2 years ago |

    #!/bin/bash
    SHA512="485fe3502978ad95e99f865756fd64c729820d15884fc735039b76de1b5459d32f8fadd050b66daf80d929d1082ad8729620925fb434bb09455304a639c9bc87"
    # This line and everything later gets SHA512'ed and put in the above line.
    # To generate the sha512 simply: tail -n +3 [SCRIPTNAME].sh | sha512sum
    check_sha512() {
        # Compute the SHA512 hash of the script excluding the first two lines
        local current_sha=$(tail -n +3 "$0" | sha512sum | awk '{print $1}')
    
        # Compare the computed SHA512 hash with the predefined one
        if [[ "$current_sha" != "$SHA512" ]]; then
            echo "Error: SHA512 hash does not match!"
            exit 1
        fi
    }
    
    # Call the function to perform the hash check
    check_sha512
    
    # Rest of your script starts here
    echo "Script execution continues..."

The idea is simple: if the first line get's mangled (#!/bin/bash) the script probably won't execute at all. If the second line gets mangled than obviously the SHA512 comparison won't work (variable name or value).

Finally if the rest of the script gets mangled or truncated it won't SDHA512 the same and it'll cause the function to exit.

For bonus points you can add a check if first line of script is exactly "#!/bin/bash" as well.

o11c 2 years ago | |

If the file is truncated after the function's closing brace, it will succeed but do nothing.

If the file is truncated in the middle of the word `check_sha512` it will try to execute a hopefully-not-existing command.

Wrapping in simple { braces } should fix this - if the brace is missing, you get a syntax error, and if present, you can execute the full thing, regardless of whether a trailing newline is available. This is admittedly bash-specific, so won't work for the linked script, but (subshell) doesn't cause too many problems

Using a function and checking the SHA don't really add anything after these fixes.

Checking the shebang is hostile to environments that install bash elsewhere.

An almost-working possibility would be:

  exec some-interpreter -c 'commands' "$0" "$@" ""

which will fail if the second ' is missing. The child interpreter can then check for later truncation by checking that at least 2 arguments were passed and the last one is an empty string. However, this is still prone to truncation before the -c.

deanCommie 2 years ago |

A serious question for any Linux-heads here, no insult intended.

How is it possible that there are ELEVEN different possible package managers that need to be supported by an installation script like this?

I can understand that some divergences in philosophical or concrete requirements could lead to two, three, or four opinionated varieties, but ELEVEN?

Does that mean that if I want to write an app that runs on Linux I should also be seeking to support 11 package managers? Or is there something unique about tailscale that would necessitate it?

edit: Thank you for the responses so far, but noone has yet answered the core question: WHY are there eleven of them?

oneepic 2 years ago |

I read TFA. Why would a truncated partial download happen and still run the script?

loeg 2 years ago | |

I believe 'curl XXX | sh' will start executing any complete shell statements in the input, before the full script is downloaded.

LukeShu 2 years ago | |

The most obvious answer is `curl | sh`. But also perhaps a network blip interrupting curl/wget, but the user failing to notice and going ahead and executing the file anyway.

wongarsu 2 years ago | |

A truncated download might happen for all sorts of reasons, like your internet connection dropping while you download the script. If you don't notice you might accidentally run an incomplete script and leave your system in some broken or at least confusing state. They wrapped everything in a main function to prevent that from happening

loeg 2 years ago | | |

Browsers typically emit downloads to temporary files until they are complete, then rename them into the final location, to prevent this kind of issue.

kazinator 2 years ago |

I mean, could you put the main function at the top of the script, so that it calls later definitions?

The problem is that the script could be truncated in such a way that it executes successfully. It defines a bunch of functions and then quits.

If you're not checking for the success or failure of the download, you're probably not checking for the success or failure of the script; something is just going to assume the script worked.

latchkey 2 years ago |

If only there was a way to transactionally run shell scripts such that if they don't complete fully, the changes are automatically reverted.

Edit: cue the HN responses to use nix, and other solutions

slimsag 2 years ago | |

Make curl | sh automatically upgrade the user's system to nix?

hwbunny 2 years ago | | |

Nix, as the name implies, it's a moissanite, a fuck-o, a bullshit.

treve 2 years ago | |

Potentially sounds like a job for a BTRFS snapshot?

rkeene2 2 years ago | | |

Not really... although there have been attempts at doing this (e.g., Solaris) I've never seen it work out well for systems which could have multiple concurrent users. The main problem is that there are unrelated changes that could occur while performing an installation, for example adding a user while (but not part of) installing a package is a valid configuration right now but wouldn't be in the described scenario.

Solaris handled this by having a config file that you could specify which files to copy between boot environments. Don't have /var/mail,/etc/passwd,/etc/shadow,/etc/hosts,.... in a different FS ? Better remember to copy it, or you lose your emails,users,hosts,etc

The problem with this kind of failure mode is that it's silent and generally irreversible. There generally aren't tools to MERGE files that conflicted later on if you discover the issue while the old boot environment snapshot is still around.

On the other hand, this is sort of how Docker (and Solaris zones) work -- and why you can't upgrade a container in a container-like way, you must replace the entire container (i.e., any upper-COW is lost, unless you export it and build something new atop it outside of any sort of reconciled process).

On the other other hand, I've actually used BtrFS snapshots for exactly this successfully in exactly 1 case -- a read-only volume containing my OS files (12 files total; Kernel, Kernel Debug, Several Initramfs) for PXELINUX/EXTLINUX booting, for which can be atomically upgraded. Since it was read-only and the only operation supported on that volume was replacing the files, and it only had those 12 files, it was safe to do so.

riffic 2 years ago | |

Idempotence isn't guaranteed either

Arcuru 2 years ago |

...isn't that a common pattern? I'm pretty sure most install scripts I download and run already do that, though I don't run those very often.

yau8edq12i 2 years ago |

Don't pipe curl/wget a script to a shell without reading what you've downloaded. This should be common sense. Do `wget $url; most install.sh` and only if you're satisfied with what you read, execute `sh install.sh`.