Advanced Bash Scripting

120 points by mbowcock 14 years ago | 34 comments

Adaptive 14 years ago |

This is hands down the best Bash resource online or in print. Don't let the "advanced" throw you off. It's the source of pretty much everything (of significance) that I know about bash.

For other applications it makes sense to use your scripting language of choice (libraries, etc.) but in any situation where you need absolute portability or availability on an otherwise bare system, bash is the way to go.

There are whole projects like the Arch Linux Installation Framework (AIF) written in bash for this reason.

Named arrays, quick string handling and the [[ ]] test syntax alone make this worth reading, and that's just scratching the surface.

mbubb 14 years ago | |

Also on the tldp site is a 'beginner's BASH guide' - in fact there are a number of guides:

http://tldp.org/guides.html

Unfortunately a lot of the stuff has not been updated in 5 yrs or so - but it is still fun to read through from time to time.

It is not exactly a BASH guide but another good guide is "UNIX for Poets" - I think there have been a number of news.ycombinator posts regarding it. This is a good intro for the text manipulation tools you would tend to use in a BASH script.

(see http://news.ycombinator.com/item?id=2456929 )

avar 14 years ago | |

If you need portability you should write your script in POSIX shellscript, not Bash shellscript.

derleth 14 years ago | |

> in any situation where you need absolute portability or availability on an otherwise bare system, bash is the way to go.

Interestingly, this is why Debian uses dash: A smaller shell is more portable.

tingletech 14 years ago | | |

How is that portable, across different architectures?

I have to work in Solaris, SUSE SLES, some other linux where I don't know what they run, and OS X and I don't have root on most of those. bash seems to work the same everywhere and always seems to be there when I get access to a newly provisioned environment.

eliben 14 years ago |

While bash is certainly more pleasant to write and use than the other shell languages, I made myself a rule I abide by, a long time ago. Never use shell to write anything non-trivial. Unless it's a one liner that just stitches some Unix commands together, just use a real language (Python, for example). I can't count how many times I found myself being grateful for this decision. Shell scripts just have a way of growing in size and complexity, and swallowing the shell pill is the quick road into abyss.

vlisivka 14 years ago | |

I wrote large bunch of software in shell, so I have habit to use strict mode ("set -ue"), so shell scripts are much safer.

I write scripts like large programs from scratch: decomposition, error handling, proper logging, unit testing, packaging, etc.

I wrote my own library with few useful modules, which is designed for strict mode (bash-modules on assembla), which I recommend to use even for trivial programs.

I have no problems with bash.

sanderjd 14 years ago | | |

Why did you do all those things? Did you have to use bash for all that software? Were you concerned about portability? Do you just dislike python?

I have no trouble understanding how complex programs can be written in bash, but I have never understood why.

eliben 14 years ago | | |

This is all good, until you want your program to be really large and/or use some serious external libraries which bash lacks. Or, if you need to attach some kind of non-CLI UI to it (GUI or web), or (heavens forbid) if you need to port it to windows. Then all your plans crash and burn, and you either suck it up and port the app to Python/Perl/Ruby/C++/whatever, or just keep building an un-maintainable mess. And don't forget someone else will probably have to maintain it at some point.

jhaglund 14 years ago |

I recently found this site useful. I wrote a script that glued together some commands to be run by cron. Then the requirements grew and the script's complexity grew, and grew, and before long I learned that bash scripts support functions.

At this point, I wished I had started in python.

To answer maven911, I usually do LAMP stack development but pick up all kinds of technologies along the way.

Maven911 14 years ago | |

Thanks for answering, I should have given some background on why I am asking this, bash is one of many technologies that I need to learn for the job - our product unfortunately uses thousands of lines of bash scripting, and I have never gotten around finishing this guide - mostly due to laziness (after work) since I don't have time during the job to look into it as we are always in fire-fighting mode.

1010011010 14 years ago |

Here's the short version of the Advanced Bash Scripting Guide: "Use Python".

Long bash scripts are less readable and maintainable than python scripts.

draebek 14 years ago | |

Speaking as a person that converted a big shell script to Python a while back, I think both languages have their strengths. Writing something like:

    prog1 2> prog1.log | prog2 2> prog1.log | prog3

in Python is not nearly so succinct as in the shell (using only Python's standard library). And don't screw up waiting for processes or flushing buffers, things that the shell takes care of for you AFAIK.

On the other hand, things like string or date manipulation are easier in Python than in the shell.

tikhonj 14 years ago | |

It doesn't have to be Python--you can use pretty much any language you want, so just use your favorite one be it Ruby or Python or Haskell or whatever.

I personally only use Bash when the script is just a trivial translation of commands I enter.

Maven911 14 years ago | | |

sometimes you have no choice, such as working on telecom systems which have a stripped down Linux version running on it and there is no python/perl interpreter available.

rhizome31 14 years ago | | |

Sometimes you could write in Ruby/Python/Perl but it's a good idea to make a shell script instead, so that sysadmins can understand it.

tingletech 14 years ago | |

I needed this guide to help me write my bash script that I use to build python and all its dependencies on all the machines I need to work on. That way I'm not at the mercy of whatever out of date python happens to be installed with the package manager by the admin.

wazoox 14 years ago | |

Actually perl makes it way easier than python, because it supports more of bash features directly, from backticks to here-docs.

burgerbrain 14 years ago | |

This guide is a great read even if all you do is short bash scripts.

vlisivka 14 years ago | |

Shell is Lingua Franca for system administrators.

kaichanvong 14 years ago |

I'd really like this as an Epub, Pdf or something I can just stick on my iPad. Anyone know of a link for this... or am I going to have to do this for everyone? :]

anoother 14 years ago | |

Please share if you do! epub would be perfect

icebraining 14 years ago | |

If you go "one directory up", they already have PDF versions: http://www.tldp.org/LDP/abs/

Maven911 14 years ago |

has anyone gone through the entire book/guide, and i wonder is it because of your job and what other technologies you have to know in-depth

tingletech 14 years ago | |

I use it more as a reference / scan it for inspiration when I'm stuck. Way better than the bash man page...

I use it like I use 'perldoc perl' or http://docs.python.org/ or https://docs.djangoproject.com/en/1.3/ or https://developer.mozilla.org/en/JavaScript/Guide or http://docs.jquery.com/Main_Page or http://www.dpawson.co.uk/xsl/sect2/sect21.html etc.

nosequel 14 years ago | |

I've read the whole thing at one point in time. I was an embedded programmer for years, and the availability of bash, and the lack of python/ruby/whatever made it pretty much a necessity.

Once you grok it, it is a pretty excellent tool for testing, monitoring, coordinated pieces, etc. Besides Solaris, I can typically sit down at any type of machine or embedded device and write a fairly non-trivial script without having to ask, "what version of python is installed?" or "is it hooked up to the outside world, so I can install these libraries?".

There's power in availability and commonality even if you give up some readability.

przemoc 14 years ago |

ABS can be handy for some quick look up, but if you want to learn Bash then go with Greg's Bash Guide:

http://mywiki.wooledge.org/BashGuide

Check also the list of Bash online tutorials:

http://wiki.bash-hackers.org/scripting/tutoriallist

mbubb 14 years ago | |

Thanks for the link to Gregg's BASH guide. Useful information.

taf2 14 years ago |

hasn't this resource been around the block for 10+ years? I remember googling' "advanced bash scripting" function in college and finding a resource just like this one...