1. Bash shouldn't be used, not because of portability, but because its features aren't worth their weight and can be picked up by another command, I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for.
2. Never use #!/usr/bin/env bash. Even if you are using full bash, bash should be installed in /usr/bin/bash. If you don't even know something this basic about the environment, then you shouldn't be programming it, the script is likely to create a mess somewhere in the already strange environment of the system.
3. Don't use extensions unless you're writing for Windows machines. Do you add extensions to any other executable? head, sed can help you retrieve the first line of a file and neither of them have extensions.
4, 5, 6. You may do this is obscure scenarios where you absolutely cannot have a script run if there is any unforeseen error, but it's definitely not something that should be put on without careful consideration, http://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail explains this better. And it goes without saying that this is not a substitute for proper error handling.
7. I agree that people should trace their shell scripts, but this has nothing to with shell.
8. [[]] is powerful, so I very often see it used when the [] builtin would suffice. Also, [[ is a command like done, not a bash builtin.
9. Quote only what needs quoting. If you don't know what needs quoting, then you don't understand your script. I know it seems like a waste of time, but it will make you a much better shell programmer then these always do/don't do X unless Y then do Z, rules that we are spouting.
10. Use either local or global variables in functions, depending on which you want. I see no reason to jump through this weird hoop because it might become an easily fixable problem later.
11. This is a feature, make usage appear when you blink, I don't care, if anything variations of -h too limited,
12. Finally, one "opinion" we agree on, not sure how else to redirect to stderr, but I'm sure that other way isn't as good as this one.
13. No, read the usage. If you want inferior long options, then you can add them to your scripts, but they are not self documenting, they only serve to make commands less readable and clutter completion.
14. No, it's not usually appropriate, do you want all installed scripts writing to /bin? The directory the script is running in should be clearly communicated to the user, with cd "$(dirname "$0")", "It runs in the directory the script is in." Needs to be communicated somewhere, or you have failed.
15. Yes, use ShellCheck.
16. Please call your list Bash Script Practices if it's unrelated to shell.
8. '[' and '[[' are both bash builtins.
9/15. If you're using shellcheck, you'll need to quote (almost) all vars anyway.
Yup, nix and guix consistently put their binaries in "very weird places" - unless you want to make users of those tools unhappy (among others!) please use env. The user knows where their shell is more than you do.
'[' is a builtin, '[[' is a keyword. Can use bash's builtin 'type' to check.
I agree but
> I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for.
That's just terrible recommendation. It's saying "well, bash is terrible, use a terrible one that also have less functions"
Credibility gone.
Adding an extension to make it easier to tell what's inside without opening it is being lazy rather than following best practices. Best practice is half century of leaving them off.
Unlike Windows, which ignores extensions and lets you run a command omitting them, Unix has a better (I'm not saying perfect) approach which allow the metadata to pulled from the first line of the file, tuned exactly to what the script needs. No sane extension is going to capture this info well.
Extensions expose (usually incompletely) the implementation details of what's inside, to the detriment of the humans using them (the OS doesn't care), who will then guess at what the extension means.
However, many extensions are WRONG, or too vague to actually tell what interpreter to call on them - which this subgroup of devs does all the time, mostly commonly using the wrong version of python (wrong major, wrong minor, not from a specific python env) and breaking things. .sh is manifestly wrong as an extension for Bash scripts, which have different syntax.
The exception is scripts that should be "."-ed in (sourced), where having a meaningful .sh or .bash (which are NOT interchangeable) is ACTUALLY good, because it highlights that they are NOT COMMANDS. (and execute isn't enabled)
If you want a script to make it easier to list commands that are shell scripts or whatever, there's a simple one at the end of:
https://www.talisman.org/~erlkonig/documents/commandname-ext...
I've seen several cases of .sh scripts which contained perl code, python, or were actually binary, because the final lynchpin in this (abridged) case against extensions is that in complex systems the extensions often have to be kept even after the implementation is upgraded to avoid breaking callers. It's very normal for a program to start as shell, get upgraded to python, and sometimes again to something compiled. Setting up a situation which would force the extension to be changed in all clients in a large network to keep it accurate is beyond stupid.
Don't use extensions on commands, and stop trying to rationalize it because you (for those to whom this applies) just like to do "ls *.sh" (on your bash scripts). These are a violation of Unix best practices, and cause harm when humans try to interpret them.
shellcheck
It's like pylint for your shellscripts.
Also, write a "find" command without checking the manpage or internet
- `set -o errtrace`: trap errors inside functions
- `shopt -s inherit_errexit`: subprocesses inherit exit error
Unfortunately the list of Bash pitfalls is neverending, but that's a good start.
And the right-hand-side of a variable assignment.
And the WORD in a case statement. (Not in the patterns, though).
Plus a bunch of other single-token(?) contexts.
I don't recommend relying on the context though, it's clever and makes it hard to verify that the script does not have expansion bugs.
1. end all your lines C-style; this may save your life many times;
2. declare -is variables and -r CONSTANTS at the beginning, again, C-style;
3. print TIMESTAMP="$(date +%Y-%m-%d\ %H:%M:%S)"; where appropriate if your script logs its job;
4. Contrary to OPs reommendation I strongly try to stick to pure SH compatibility in smaller acripts so they can run on routers, TVs, androids and other busybox-like devices; BASH isn't everywhere.
date -u +%Y-%m-%dT%TZ
cd "$(dirname "$0")"
EDIT: I should've just tested this to see :) I did and it does exactly that. Very helpful. I didn't realize $0 is always the first argument. Kind of like how `self` is the first implicit argument in OOP methods?
It has some great tools for user interaction, too, including secure string handling for credentials, a TUI framework, easy parallelism, unit tests and lots more.
To be honest these days I use shell scripts, and if they get too large I'll replace with either golang or python. I don't love python, especially when dependencies are required, but it is portable and has a lot of things built-in that mean executing "standard binaries" isnt required so often.
Kudos for nicely put tips that are easy to follow and understand.
I'm newcomer in bash
Usually there is no need to return to original directory. Change of directory is process-local (script-local) so the calling process is not affected by this 'cd' in the script.
For example, instead of
USER=mail
UID=8
USER=mail
UID=$(id -u $USER)
Also note that this is something that should be done in any programming language, not just shell scripts.
The explanation for that wasnt really an explanation either ...
I won't check with for version those tips applies, and continue writing POSIX shell as much as can. I might check which or those suggestions are POSIX, though.
Any reason to avoid writing bash scripts, other than purism?
1. use shellcheck
2. use shfmt (to format your shell script)
3. set -euo pipefail (much shorter)
My order preference would be:
1. use shellcheck.
… rest …Ah, got it. Another failure mode. There is a /bin/bash, but it's an ancient, crummy thing, that is difficult to upgrade. MacOSX does this, so users paper this over by installing a private copy as ~/bin/bash. Thank you.
As soon as output needs to be parsed — especially when it’s being fed back into other parts of the script — it gets harder. Handling errors and exceptions is even more difficult.
Things really fall down on modularity. There are tricks and conventions: for example you can put all functions to do with x in a file called lib/x.sh, prefix them all with x_, and require that all positional parameters must be declared at the top of each function with local names.
At that point though, I would rather move to a language with named parameters, namespaced modules, and exception handling. In Python, it’s really easy to do the shell bits with:
def sh(script):
subprocess.run(
[‘sh’, ‘-c’, script, ‘--‘, *args],
check=True,
)
(Disclaimer: I'm one of the authors)
After falling in love with ShellCheck several years ago, with the help of another person, I made the ShellCheck REPL tool for Bash:
https://github.com/HenrikBengtsson/shellcheck-repl
I found it to be an excellent way of learning about pit falls and best practices in Bash as you type, because it gives you instant feedback on possible mistakes. It won't execute the command until the ShellCheck issues are fixed, e.g. missing quotes, use of undefined variables, or incorrect array syntax.
It's designed to be flexible, e.g. you can configure ShellCheck rules to be ignored, and you can force executtion by adding two spaces at the end.
License: ISC (similar to MIT). Please help improve it by giving feedback, bug reports, feature requests, PRs, etc.
Traditional shell might be:
grep -q thing < file
if [ $? -eq 0 ] ; then echo "thing is there ; fi
if
grep -q thing < file
then
echo "thing is there"
fi
I'm not saying *never* write shell scripts, but always consider doing something else, or at least add a TODO, or issue, to write in a more robust language.
always quote filenames, because you never know if there's a space in them.
filenames with dashes or periods will kill you
prepend current directory file manipulation filenames with "./", because the file might start with a period or dash
Dashes in filenames still might kill you, especially if you pass those to another command
I'd also say that in most cases Python is also a better choice, especially when you use the ! syntax.
But that's just a personal thing and not something that I can realy blame the language.
gci *.txt | %{ $tot += $_.length }; echo $tot
ls -l *.txt | awk '{ tot += $5 } END { print tot }'
Get-ChildItem *.txt | ForEach-Object { $tot += $_.length }; Write-Output $tot
Note that 'ls -l' and guessing that you want to total up field 5 is brittle in way that the Powershell snippet isn't, but I'm leaving that issue aside.
exec &> >( ts '[%Y-%m-%d.%H:%M:%S] ' | tee ${LOGFILENAME} )
Case in point: Declaring an array. IMHO, it’s just not ergonomic at all. Especially not in sh/dash.
* use bats for testing * use shfmt for code formatting * use shellcheck for linting
export PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}: '
[0]: https://marketplace.visualstudio.com/items?itemName=timonwon...
I think this site is amazing, and it must be older than at least two decades.
A better resource is https://mywiki.wooledge.org/BashGuide Also, a preliminary read of https://mywiki.wooledge.org/BashPitfalls is advised.
Using shellcheck as a bash/shell linter is the ultimate. When you get a new warning, you can look up the code and learn why it's complaining.
echo_command()
{
echo
echo '$' "$@"
}
echo_and_run_command()
{
echo_cmd "$@"
"$@"
}
main()
{
# For simple commands that do not use | < > etc.
echo_and_run_command cp --verbose ...
# More complex commands
echo_command grep ... '|' find ...
grep ... | find ...
}
main "$@" ### Usage and help - change this for each script
##############################################################################
# shellcheck disable=SC2015
[[ "${__usage+x}" ]] || read -r -d '' __usage <<-'EOF' || true # exits non-zero when EOF encountered
-t --timestamps Enable timestamps in output
-v --verbose Enable verbose mode, print script as it is executed
-d --debug Enables debug mode
-h --help This page
-n --no-color Disable color output
EOF
set -x
sh -x script
#!/usr/bin/env bash
(
foo
bar
) 2>&1 | print_and_log "$logfile"Don’t.
Use a proper programming language instead. bash (and similar scripting languages) are non-portable (Linux/Windows) and the opposite of what I want in a good programming language.
I agree we need a shell scripting language, I disagree that bash zsh or anything that frequently uses double square brackets and awful program names is the epitome of shell scripting language design.
giving up on the notion "others will use or collaborate with my scripts" was the single most productive thing i've done for my scripting.
A couple of days ago this link was posted to hn http://mywiki.wooledge.org/BashFAQ/105
It showed me once again how little bash I know even after all those years. I checked the examples to see if only set -e is dangerous or also set -o like the author suggested and sure enough it's just as bad es set -e. You just got to thoroughly check your bash scripts and do proper error handling.
Perhaps not surprisingly, it's bourne shell, not bash. But still, it's an actual published standard all can refer to when the language in question is "shell scripts", i.e. .sh files, or "shell commands" in some context where a shell command is called for (e.g. portable makefiles).
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html* Write help text to stdout, not stderr, so I can grep it
* Set exit status to 0 on success and 1 or some other small positive integer on failure so I can use || and &&
> 9. Always quote variable accesses with double-quotes.
Does the author refer to "$MYVAR"? Why would you want to use that over ${MYVAR}?
That said, I like doing the usage like so for short scripts:
#!/bin/sh
#
# Sleep until a specific time. This takes a time in 24-hour clock format and
# sleeps until the next instance of this time is reached.
#
# % sleep-until 15:30:45
# % sleep-until 15:30 # Until 15:30:00
# % sleep-until 15 # Until 15:00:00
#
# Or to sleep until a specific date:
#
# % sleep-until 2023-01-01T15:00:00
#
# Or space instead of T; can abbreviate time like above.
echo " $@" | grep -q -- ' -h' && { sed '1,2d; /^[^#]/q; s/^# \?//;' "$0" | sed '$d'; exit 0; } # Show docs
% sleep-until -h
Sleep until a specific time. This takes a time in 24-hour clock format and
sleeps until the next instance of this time is reached.
…
I'm not 100% sure if I thought of this myself or if it's something I once saw somewhere.
This works around malicious filenames that may start with a '-'. Especially important if you're running an 'rm' command
Edit: another workaround is to ensure that files are always absolute pathnames or even starting with './' for relative ones.
But I use, set -euo pipefail. I think -u is -o unset ,etc? Just easier to type.
2. Shell scripts are wonderful, but once they exceed a few lines (give or take 50), they've entered the fast track on becoming a maintenance headache and a liability.
.sh is appropriate for a shell library module which you source from another shell script. It is not really appropriate for something which is more abstract (such as a "program" inside your PATH).
set -e / set -o errexit will only be helpful if you fundamentally understand exactly what it does, if you don't, you are bound to end up with broken code. Once you fundamentally understand set -e you will be better placed to decide whether it is appropriate to use it or more appropriate to simply do proper error handling. The oft repeated mantra of using set -e is really misleading a lot of people into thinking that bash has some sane mode of operation which will reduce their chance of making mistakes, people should never be mislead to think that bash will ever do anything sane.
set -u / set -o nounset breaks a lot of perfectly sensible bash idioms and is generally bad at what proponents of it claim it will help solve (using unset variables by accident or by misspelling). There are better linters which solve this problem much better without having to sacrifice some of what makes bash scripts easier to write/read.
set -o pipefail is not an improvement/detriment, it is simply changing the way that one of bash's features functions. pipefail should only be set around specific uses of pipelines when it is known that it will produce the intended result. For example, take this common idiom:
if foo | grep -q bar; then ...
Guarding set -x / set -o xtrace seems unnecessary, -x is already automatically inherited. Just set it before running the program.
Good advice on using [[ but it is important to fundamentally understand the nuances of this, quoting rules change within the context of [[.
Accepting h and help seems incredibly unnecessary. If someone who has never used a unix-like operating system happens upon your script then they may find it useful. But I don't think catering to such a low common denominator makes sense. Your script should just handle invalid arguments by printing a usage statement with maybe a hint of how to get a full help message.
I'd say changing to your script's directory is almost never appropriate.
Shellcheck, while useful, is useful only if you understand bash well.
The lesson here is that if you think that you have spent enough time writing bash to suggest best practices, you've not spent enough time writing bash. Only when you realise that the best practice is to not use bash have you used bash long enough (or short enough).
If you want to write a script which you're going to rely on or distribute, learn bash inside out and then carefully consider if it's still the right option.
If you are unwilling or unable to learn bash inside out then please use something else.
Do not be fooled into thinking that some "best practices" you read online will save you from bash.
It occupies a sweet spot of being ubiquitous, quick to write/deploy and naturally interfaces with OS commands. It's the glue that holds unixes/linuxes together.
Your articles on awk and sed were a huge inspiration to me around 2008-09, and I super-looked up to you. Never have I imagined you would check out my blog one day!
Thank you for all your work dude! Stay awesome.
Are you listening Apple?
I might add only a minor note that his construct for h/help breaks globbing in a directory that happens to contain a file named "h" or "help" (even without a leading dash '-'), but only if that happens to be first alphabetically. No footgun there..Lol. He also does no "--" support, about the only convention to make globbing work reliably.
#!/bin/bash
if [[ `wc -l $0|cut -f 1 -d ' '` -gt 100 ]]
then
echo "No, this is too long!"
exit
fiPretty sure there are other dyslectics who will find that it helps them. So I guess this all depends on person to person
My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different programming paradigm than languages like Perl, Python, whatever.
Obviously, if your mental model is BASIC and you try to write Python, then you encounter lots of friction and it's easy for the latter to feel hacky, bad and ugly. To enjoy and program Python well, it's probably best to shift your mental model. The same goes for Shell.
What is the Shell paradigm? I would argue that it's line-oriented pipelines. There is a ton to unpack in that, but a huge example where I see friction is overuse of variables in scripts. Trying to stuff data inside variables, with shell's paucity of data types is a recipe for irritation. However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against.
/2cents
What Python is to Java, Shell is to Python. It speeds you up several times. I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away.
While neither Shell or SQL are perfect, there have been many ideas to improve them and for sure people can't wait for something new like oil shell to get production ready, getting the shell quoting hell right, or somebody fixing up SQL, bringing old ideas from Datalog and QUEL into it, fixing the goddamn NULL joins, etc.
But honestly, nothing else even comes close to this 10x productivity increase over the next best alternative. No, Thank you, I will not rewrite my 10 lines of sh into python to explode it into 50 lines of shuffling clunky objects around. I'll instead go and reread that man page how to write an if expression in bash again.
Shameless plug coming, it this has been a pain point for me too. I found the issue with quotes (in most languages, but particularly in Bash et al) is that the same character is used to close the quote as is used to open it.m. So in my own shell I added support to use parentheses as quotes in addition to the single and double quotation ASCII symbols. This then allows you to nest quotation marks.
https://murex.rocks/docs/parser/brace-quote.html
You also don’t need to worry about quoting variables as variables are expanded to an argv[] item rather than expanded out to a command line and then any spaces converted into new argv[]s (or in layman’s terms, variables behave like you’d expect variables to behave).
Language aside, the ecosystem and culture do not afford enough in way of testing, dependency management, feature flags, static analysis, legibility, and so on. The reason people say to keep shell programs short is because of these problems, it needs to be possible to rewrite shell programs on a whim. At least then, you can A/B test and deploy at that scope.
* Shell scripts force you to think in a more scalable way (data streams)
* Shell scripts compose rich programs rather than simplistic functions
* Shells encourage you to program with a rich, extensible feature set (ad-hoc I/O redirection, files)
The only times I don’t like shell scripts are when dealing with regex and dealing with parallelism
Do you not have a ~/.python_history? The exact same search functions are available on the REPL. Ctrl-R, type your bit, bam.
It's a shell that is actually built for structured data, taking lessons learned from PowerShell and others.
Running `parallel --shellquote --shellquote --shellquote` and pasting in the line you want to quote thrice may alleviate some of the pain.
By no means ideal, though.
The comparison should be to perl or Ruby, both of which will fare better than Python for typical shell-type tasks.
But different strokes and all that.
This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.
Do you have an example of this? I didn’t even know you could make sql calls in scripts.
It is "opinion" based on debugging scripts made by people (which might be "you but few years ago") that don't know the full extent of death-traps that are put in the language. Or really writing anything more complex.
About only strong side of shell as a language is a pipe character. Everything else is less convenient at best, actively dangerous at worst.
Sure, "how to write something in a limited language" might be fun mental excercise but as someone sitting in ops space for good part of 15 years, it's just a burden.
Hell, I'd rather debug Perl script than Bash one...
Yeah, if it is few pipes and some minor post processing I'd use it too (pipe is the easiest way to do it out of all languages I've seen) but that's about it.
It is nice to write one-liners in cmdline but characteristic that make it nice there make it worse programming language. A bit like Perl in that matter
It's possible to use the wrong tool for the job in any language - including language choice itself.
Dismissing a programming language because it's not shell and dismissing shell because it's not a proramming language are the same thing - a bad idea if that's your only decision criteria.
Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data.
dmenu + fzf + jq + curl is my bread and butter in shell scripts.
However, I still haven't managed to find a way to do a bunch of tasks concurrently. No, xargs and parallel don't cut it. Just give me an opinionated way to do this that is easily inspectable, loggable and debuggable. Currenly I hack together functions in a `((job_i++ < max_jobs)) || wait -n` spaghetti.
I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell.
The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need.
And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.
Do you have an example of a script you wrote?
I swear, there should be a HN rule against those. It pollutes every single Shell discussions, bringing nothing to them and making it hard for others do discuss the real topic.
Unless, the language is APL, in which case, 10 lines is an operating system.
Which python can do realitively well, by using the `subprocess` module.
Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr`
import subprocess as sp
cat = sp.Popen(
["cat", "README.md"],
stdout=sp.PIPE,
)
grep = sp.Popen(
["grep", "#"],
stdin=cat.stdout,
stdout=sp.PIPE,
)
tr = sp.Popen(
["tr", "[:lower:]", "[:upper:]"],
stdin=grep.stdout,
stderr=sp.PIPE,
stdout=sp.PIPE,
)
out, err = tr.communicate()
print(out.decode("utf-8"), err.decode("utf-8"))
Yes, that is a convoluted example, but it illustrates the point I am trying to make. Everything outlined could probably done in a bash script, but I am pretty certain it would be much harder, and much more difficult to maintain, than doing this in python.
Bash is absolutely fine up to a point. And with enough effort, bash can do extremely complex things. But as soon as things get more complex than standard unix tools, I rather give up on the comfort of having specialiced syntax for pipes and filehandles, and write a few more lines handling those, if that means that I can do the more complex stuff easily using the rich module ecosystem of Python.
I am not sure I would agree. Sed fills this role quite nicely.
cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'
Doesn't sound so bad. A quick argument parser, a call out to grep or sed, pipe to curl, then to graphviz I guess (I don't really know much about image generation tools though), then compose the mail with a heredoc and run sendmail. Sounds like 10 to 15 lines for a quick and dirty solution.
Bourne Shell Scripting is literally a bunch of weird backwards compatible hacks around the first command line prompt from 1970. The intent was to preserve the experience of a human at a command prompt, and add extra functionality for automation.
It's basically a high-powered user interface. It emphasizes what the operator wants for productivity, instead of the designer in her CS ivory tower of perfection. You can be insanely productive on a single line, or paste that line into a file for repeatability. So many programmers fail to grasp that programming adds considerations that the power user doesn't care about. The Shell abstracts away all that unnecessary stuff and just lets you get simple things done quickly.
- no standard unit testing
- how do you debug except with printlns? Fail.
- each line usually takes a minimum of 10 minutes to debug unless you've done bash scripting for... ten years
- basic constructs like the arg array are broken once you have special chars and spaces and want to pass those args to other commands. and UNICODE? Ha.
- standard library is nil, you're dependent on a hodgepodge of possibly installed programs
- there is no dependency resolution or auto-install of those programs or libraries or shell scripts. since it is so dependent on binary programs, that's a good thing, but also sucks for bash programmers
- horrid rules on type conversions, horrid syntax, space-significant rules
- as TFA shows, basic error checking and other conventions is horrid, yeah I want a crap 20 line header for everything
- effective bash is a bag of tricks. Bag of tricks programming is shit. You need to do ANYTHING in it for parsing, etc? Copy paste in functions is basically the solution.
- I'm not going to say interpreter errors are worse than C++ errors, but it's certainly not anything good.
Honestly since even effing JAVA added a hashbang ability, I no longer need bash.
Go ahead, write some bash autocompletion scripts in bash. Lord is that awful. Try writing something with a complex options / argument interface and detect/parse errors in the command line. Awful.
Bash is basically software engineering from the 1970s, oh yeah, except take away the word "engineering". Because the language is actively opposed to anything that "engineering" would entail.
Any example with this? The following works reasonably well for me.
args=(-a --b 'arg with space' "一 二 三")
someprog "${args[@]}"With Trace. Which is talked about in TFA.
By the way nobody use exclusively bash. When i worked for a cloud provider, it was basically 30% python(ansible), 30% perl, 5 to 10% bash, and a bit of other languages depending on the client needs (mostly java, but also Julia and R).
And sure sure you can call any process from a language but the assumptions are different. No one wants to call a Java jar that has a dependency on the jq CLI app being available.
Shell is a tool that turns out to be pretty good for some things, particularly composing functionality out of other programs and also doing system configuration/tuning stuff to tailor an environment for other programs. It's also really handy for automating tasks you find yourself repeating.
Programming languages are a tool that are pretty good for other things - making new programs, tricky logic, making the most (or at least more than a shell script launching 1000s of new processes) efficient use of a computer.
Trying to replace one with the other is not really useful - they have different jobs. Learning to use them in conjunction on the other hand... there's a lot of power in that.
By comparison - javascript and html. They don't replace each other - yet they are both computer languages used in the same domain, and both have strengths and weaknesses. They have different jobs. And when you use them in conjunction you get something pretty darn powerful.
Handling binary data can also work in Bash, provided that you just use it as a glue for pipelines between other programs (e.g. feeding video data into ffmpeg).
One time, while working on some computer vision project, I had a need to hack up a video-capture-and-upload program for gathering training data during a certain time of day. It took me about 20 minutes and 50 lines of Bash to setup the whole thing, test it, and be sure it works.
It's also important to learn your system's environment too. This is your "standard library", and it's why POSIX compatibility is important. You will feel shell is limited if you don't learn how to use the system utilities with shell (or if your target system has common utilities missing).
As an example of flexibility, you can use shell and system utilities in combination with CGI and a basic web server to send and receive text messages on an Android phone with termux. Similar to a KDE Connect or Apple's iMessage.
You think the complaints about rickety, unintuitive syntax are "socially founded"? I can't think of another language that has so many pointless syntax issues every time I revisit it. I haven't seen a line of Scheme in over a decade, and I'm still fairly sure I could write a simple if condition with less likelihood of getting it wrong than Bash.
I came at it from the other end, writing complex shell scripts for years because of the intuition that python would be overkill. But there was a moment when I realized how irrational this was: shell languages are enough of a garbage fire that Python was trivially the better choice for my scripts the minute flow control enters the picture.
Bash has dynamic scope with its local variables.
The standard POSIX language has only global variables: one pervasive scope.
* I like to the HGPPL data structures and convenient library for manipulating them (in my case this is Clojure which has a great core library). Bash has indexed and associative arrays.
* Libraries for common data formats are also used in a consistent way in the HGPPL. I don't have to remember a DSL for every data format - i.e. how to use jq when dealing with JSON. Similarly for YAML, XML, CSVs, I can also do templating for configuration files for nginx and so on. I've seen way too many naive attempts to piece together valid YAML from strings in bash to know its just not worth doing.
* I don't want to switch programming language from the main application and I find helps "break down silos" when everyone can read and contribute to some code. If a team is just sysadmins - sure, make bash the official language and stick to it.
* I can write scripts without repeating myself using namespaces and higher-order functions, which my choice of paradigm for abstractions, others write cleanly with classes. You can follow best practices, avoid the use of ENV vars, but that requires extra discipline and it is hard to enforce on other for the type of places where bash is used.
This argument is essentially the same as "dynamic typing is just a different programming paradigm than static typing, and not intrinsically better or worse" - but to an even greater extent, because bash isn't really typed at all.
To those who think that static (and optional/gradual) typing brings strong benefits with little downsides over dynamic typing and becomes increasingly important as the size of a program increases, bash is simply unacceptable for any non-trivial program.
Other people (like yourself) that think that static typing isn't that important and "it's just a matter of preference" will be fine with an untyped language like bash.
Unfortunately, it's really hard to find concrete, clear evidence that one typing paradigm is better than the other, so we can't really make a good argument for one or the other using science.
However, I can say that you're conflating different traits of shell languages here. You say "dynamic scope, focus on line-oriented text, and pipelines" - but each of those are very different, and you're missing the most contested one (typing). Shell's untypedness is probably the biggest complaint about it, and the line-oriented text paradigm is really contentious, but most people don't care very much about the scoping, and lots of people like the pipelines feature.
A shell language that was statically-typed, with clear scoping rules, non-cryptic syntax, structured data, and pipelines would likely be popular and relatively non-controversial.
declare -A mydict( [lookma]=initalization )
mydict[foo]=bar
echo "${mydict[foo]}"
list=()
list+=(foo bar baz)
echo "${list[0]}"So if you don't care about error cases everything is fine, but if you do, it gets ugly really fast. And that is the reason why other languages are probably be better suited if you want to write something bigger that 10 lines.
However, I have to admit, I don't follow that advice myself...
I guess you're referring to http://mywiki.wooledge.org/BashFAQ/105. Got recently hit by these as well.
There is something to be said in favor of the shell being always available, but Perl is almost always available. FreeBSD does not have it base of the base system, but OpenBSD does, and most Linux distros do, too.
But it is fun to connect a couple of simple commands via pipes and create something surprisingly complex. I don't do it all the time, but it happens.
However the biggest issues I've had is that the code is really hard to test, error handling in shell isn't robust, and reusability with library type methods is not easy to organize or debug.
Those are deal breakers for me when it comes to building any kind of non trivial system.
I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced features, which you don't need to use (and many people are unaware of them anyway), but will very quickly become useful; in bash all sorts of things require obscure incantations and/or shell pipelines that almost make APL seem obvious in comparison.
In my experience few people understand bash (or POSIX sh) in the first place, partly because everything is so difficult and full of caveats. Half my professional shell scripting experience on the job is fixing other people's scripts. So might as well use something that doesn't accidentally introduce bugs every other line.
Most – though obviously far from all – scripts tend to be run in environments you control; portability is often overrated and not all that important (except when it is of course). Once upon a time I insisted on POSIX sh, and then I realised that actually, >90% of the scripts I wrote were run just by me or run only in an environment otherwise under my control, and that it made no sense. I still use POSIX sh for some public things I write, when it makes sense, but that's fairly rare.
I think bash is really standing in the way of progress, whether that progress is in the form of fish, zsh, oil shell, or something else, because so many people conflate "shell" with "bash", similar to how people conflate "Google" with "search" or "git" with "GitHub" (to some degree).
I don't agree with this one. When I name my script without extension (btw, .sh is fine, .bash is ugly) I want my script to look just like any other command: as a user I do not care what the language program is written in, I care about its output and what it does.
When I develop a script, I get the correct syntax highlight becuase of the shebang so the extension doesn't matter.
The rest of the post is great.
Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior.
"Prefer to use set -o nounset."
ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". There is no real downside.
"Use set -o pipefail."
Waste of time. You will spend so much time debugging your app from random pipe failures that actually didn't matter. Dont use this option; just check the output of the pipe for sane values.
"Use [[ ]] for conditions"
No!!! Only use that for bashisms where there's no POSIX alternative and try to avoid them wherever possible. YAGNI!
"Use cd "$(dirname "$0")""
Use either "$(dirname "${BASH_SOURCE[0]}")" or grab a POSIX readfile-f implementation.
"Use shellcheck."
This should have been Best Practice #1. You will learn more about scripting from shellcheck than 10 years worth of blog posts. Always use shellcheck. Always.
Also, don't use set -o nounset when set -u will do. Always avoid doing something "fancy" with a Bashism if there's a simpler POSIX way. The whole point of scripts is for them to be dead simple.
Pass all scripts through https://www.shellcheck.net/ or use `shellcheck` on the commandline.
Learn the things it tells you and implement them in future scripts.
I'm almost tempted to put in a self-linting line in scripts so that they won't run unless shellcheck passes completely. (It would be unnecessary to lint the same script every time it's called though, so it's not a serious suggestion).
There should be an option in bash to auto-lint scripts the first time that they're called, but I don't know how the OS should keep track of when the script was last changed and last linted.
He suggests to `set -eu`, which is a good idea, but then immediately does this:
if [[ "$1" =~ ^-*h(elp)?$ ]]; ...
if [[ "${1-}" =~ ^-*h(elp)?$ ]]; thenUnfortunately, `errexit` is fairly subtle. For example
[ "${some_var-}" ] && do_something
However! If that's the last line of a function, then the function's return code will inherit the exit code of that line, meaning that
f(){ [ "${some_var-}" ] && do_something;}; f
Anyway, there are a few subtleties like this that are worth being aware of. This is a good, but dated, reference: https://mywiki.wooledge.org/BashFAQ/105
I'm a fan of using BASH3 boilerplate: https://bash3boilerplate.sh/
It's standalone, so you just start a script using it as a template and delete bits that you don't want. To my mind, the best feature is having consistent logging functions, so you're encouraged to put in lots of debug commands to output variable contents and when you change LOG_LEVEL, all the extraneous info doesn't get shown so there's no need to remove debug statements at all.
The other advantage is the option parsing, although I don't like the way that options have to have a short option (e.g. -a) - I'd prefer to just use long options.
> And it’s usually always appropriate.
I wouldn't think so. You don't know where your script will be called from, and many times the parameters to the script are file paths, which are relative to the caller's path. So you usually don't want to do it.
I collected many tips&tricks from my experience with shell scripts that you may also find useful: https://raimonster.com/scripting-field-guide/
Edit: just had a quick look at your recommended link and spotted a "mistake" in 4.7 - using "read" without "-r" would get caught out by shellcheck.
Fixed, thanks!
If a shell script needs any kind of functionality beyond POSIX, then that's a good time to upgrade to a higher-structure programming language.
Here's my related list of shell script tactics:
Agreed on the powerful bit, however [[ ]] is not a "builtin" (whereas [ and test are builtins in bash), it's a reserved word which is more similar to if and while.
That why [[ ]] can break some rules that builtins cannot, such as `[[ 1 = 1 && 2 = 2 ]]` (vs `[ 1 = 1 ] && [ 2 = 2]` or `[ 1 = 1 -a 2 = 2 ]`, -a being deprecated).
Builtins should be considered as common commands (like ls or xargs) since they cannot bypass some fundamental shell parsing rules (assignment builtins being an exception), the main advantages of being a builtin being speed (no fork needed) and access to the current shell process env (e.g. read being able to assign a variable in the current process).
For example, using cd "$(dirname "$0")" to get the scripts location is not reliable, you could use a more sophisticated option such as: $(dirname $BASH_SOURCE)
When it comes to bash search for even simplest command/syntax always ALWAYS leads to stackoverflow thread with 50 answers where bash wizards pull oneliners from sleeves and nitpick and argue about various intricancies
I'm more likely to use the BashFAQ though for actual snippets: https://mywiki.wooledge.org/BashFAQ
I start scripts from the very useful template at https://bash3boilerplate.sh/
Also, did you mean to write...?
$(dirname "$BASH_SOURCE")"Luckily, most commonly encountered scripting issues are with whitespace in filenames/variables and running a script through shellcheck will catch most (all?) of those problems.
It's amazing how edge cases can make a simple command such as 'echo' break. (Top tip - use printf instead of echo)
This is why you'll see code like this: echo 'prompt: ' | tr -d '\012'
No other simple mechanism was portable at the time. Seriously portability-minded coders still use that line, because although the issue is finally dead in linux+bash (i.e. /bin/echo is enough like bash's builtin) - it's likely still broken in other Unixen out there.
> People can now enable debug mode, by running your script as TRACE=1 ./script.sh instead of ./script.sh.
The above "if" condition will set xtrace even when user explicitly disables trace by setting TRACE=0.
A correct way of doing this would be:
if [[ "${TRACE-0}" == "1" ]]; then set -o xtrace; fiMy girlfriend complained about Firefox aalllways needing updates every time she starts it. Yeah, because she used Chrome most of the time, if you start Firefox once every other month, of course that's going to happen every time. This sounds like a similar issue: the software may not be the friendliest, but you can't really expect another outcome if you never use it because you don't like it because you never use it.
while getopts :hvr:e: opt
do
case $opt in
v)
verbose=true
;;
e)
option_e="$OPTARG"
;;
r)
option_r="$option_r $OPTARG"
;;
h)
usage
exit 1
;;
\*)
echo "Invalid option: -$OPTARG" >&2
usage # call some echos to display docs or something...
exit 2
;;
esac
doneWhen I get to a new command I find it a bit anti-social when it takes effort to find the help.
Running without any arguments? Yes, that should output info in most cases, identical to -(-)h(elp) or even /? and /h(elp) if you're feeling Windowsey that day. Outputting your full usage info, especially when spanning more than half a terminal in full screen on a modern resolution, when "nothing matches"? Please no.
For someone who knows errexit can't be trusted, and codes defensively anyway, it's fine.
* safe ways to do things in bash: https://github.com/anordal/shellharden/blob/master/how_to_do...
* better scripting: https://robertmuth.blogspot.in/2012/08/better-bash-scripting...
* robust scripting: https://www.davidpashley.com/articles/writing-robust-shell-s...
Now when I'm processing files with BASH, I nearly always end up copying stuff from there as it just bypasses common errors such as not handling whitespace or filenames that contain line breaks.
The order of commandline args shouldn't matter.
Env vars are better at passing key/value inputs than commandline arguments are.
Process-substitution can often be used to avoid intermediate files, e.g. `diff some-file <(some command)` rather than `some command > temp; diff some-file temp`
If you're making intermediate files, make a temp dir and `cd` into that
- Delete temp dirs using an exit trap (more reliable than e.g. putting it at the end of the script)
- It may be useful to copy `$PWD` into a variable before changing directory
Be aware of subshells and scope. For example, if we pipe into a loop, the loop is running in a sub-shell, and hence its state will be discarded afterwards:
LINE_COUNT=0
some command | while read -r X
do
# This runs in a sub-shell; it inherits the initial LINE_COUNT from the parent,
# but any mutations are limited to the sub-shell, will be discarded
(( LINE_COUNT++ ))
done
echo "$LINE_COUNT" # This will echo '0', since the incremented version was discarded
LINE_COUNT=0
while read -r X
do
# This runs in the main shell; its increments will remain afterwards
(( LINE_COUNT++ ))
done < <(some command)
echo "$LINE_COUNT" # This will echo the number of lines outputted by 'some command'The programmer, who was very proud of his mastery of C, said: “How can this be? C is the language in which the very kernel of Unix is implemented!”
Master Foo replied: “That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C.”
The programmer grew distressed. “But through the C language we experience the enlightenment of the Patriarch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!”
Master Foo replied: “All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C.”
The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”
The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so.
“And how many hours would you require to implement and debug that C program?” asked Nubi.
“Many,” admitted the visiting programmer. “But only a fool would spend the time to do that when so many more worthy tasks await him.”
“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”
Upon hearing this, the programmer was enlightened.
`-h` and `--help` are fine. But `help` and `h` should only display the help if the script has subcommands (like `git`, which has `git commit` as a subcommand). Scripts that don't have subcommands should treat `h` and `help` as regular arguments -- imagine if `cp h h.bak` displayed a help message instead of copying the file named "h"!
I wouldn't encourage `-help` for displaying the help because it conflicts with the syntax for a group of single-letter options (though if `-h` displays the help, there is no legitimate reason for grouping `-h` with other options).
And ideally scripts that support both option and non-option arguments should allow `--` to separate them (e.g. `rm -- --help` removes the file called "--help"). But parsing options is complicated and probably out of scope for this article.
> If appropriate, change to the script’s directory close to the start of the script. And it’s usually always appropriate.
This is very problematic if the script accepts paths as arguments, because the user would (rightly) expect paths to be interpreted relative to the original working directory rather than the script's location. A more robust approach is to compute the script's location and store it in a variable, then explicitly prepend this variable when you want paths to be relative to the script's location.
Clearly, it's for a different purpose, and there are some things that wouldn't work in a general-purpose language that isn't as focused on line-based string processing, but we are really happy with the things we took from bash.
Shell quoting though, Aieeee...
I find I have to shift gears quite substantially moving from shell or powershell to anything else...
"I'll just pipe the output of this function into.. oh, right"
Of course Bash is ubiquitous so I use them whenever I can in the company. A golden rule for me is: if it has more than 50 lines then I should probably write in a decent programming language (e.g. Ruby). It makes maintenance so much easier.
However, neither Bash nor Fish were created with Composability in mind, which is a show-stopper for me.
IMO don't use Bash if the script is longer than 20 lines and don't use Fish if it's longer than 50. Use Python. If you want to use a proper(!) language use any LISP-Dialect like Babashka, Guile Scheme, Racket, etc. If you need Types have a look at Haskell-Scripting.
EDIT: To clarify, use Fish for its bling-bling capabilities, don't use it for scripting and configuring your machine(s).
> If I want to run some POSIX-compatible script then I just use `bash scripts.sh`
Shouldn't you be using a shebang?
Ten years or so ago the cool kids were using zsh: which is in general a pretty reasonable move, it’s got way more command-line amenities than bash (at least built in).
Today fish is the fucking business: fish is so much more fun as a CLI freak.
But I guess I’ve got enough PTSD around when k8s or it’s proprietary equivalents get stuck that I always wanted to be not only functional but fast in outage-type scenarios that I kept bash as a daily driver.
Writing shell scripts of any kind is godawful, the equivalent python is the code you want to own, but it’s universality is a real selling point, like why I keep half an eye on Perl5 even though I loathe it: it may suck but it’s always there when the klaxon is going off.
The best possible software is useless if it’s not installed.
I was only talking about scripting; I know fish scripting is different, but I have no idea if it's any good. For interactive shells I don't care what people use: it's 100% a personal choice.
There is a general problem in the fact that a radical evolution of glue languages wouldn't be popular because devs rather use Python, and small evolutions wouldn't be popular (ie. zsh), because they end up being confusing (since they're still close to Bash) and not bringing significant advantages.
I'm curious why there haven't been attempts to write a modern glue language (mind that languages like Python don't fit this class). I guess that Powershell (which I don't know, though) has been the only attempt.
This is kind of a good example of what I meant when people conflate "bash" with "shell".
As for your larger point: I kind of agree, but I think what zsh offers is the advantages of shell scripts with compatibility with existing scripts while still improving on it. That said, I believe oil also offers compatibility, but I haven't had the chance to look deeply in to it; just haven't had the time, and wanted to wait until it's stable (maybe it is now?)
Perl was initially invented as the "modern glue language" to replace shell. It's fallen a bit out of fashion these days though, and to be honest I never cared all that much for Perl myself either. Raku looks nice though. TCL also works well as a kind of "glue language", although it has some really odd behaviour at times due to everything being a string and I know some people hate it with a passion, but it always worked fairly well for me. But that has also fallen out of fashion.
I've also been told PowerShell is actually quite nice and has interesting concepts (and now also open source, and you can run it on e.g. Linux), but I could never get over the verbosity of it all. I'm an old unix greybeard and I want my obscure abbreviations dammit!
If you're at that spot, don't use shell in the first place but whatever other scripting language your team uses. Well, unless it's "pipe this to that to that", sh has no parallel here
See also <https://apple.stackexchange.com/questions/371997/suppressing...>. I went with the "use an updated brewed Bash" approach, which has been working well. Using `sudo chfn` means you don't need to futz around with editing /etc/shells.
Same PS1, aliases, functions, etc. but with a couple of slight variations due to syntax differences
One reason is that there are thousands of command line tools in the UNIX ecosystem that process text streams and are designed to work with shells like bash. You have much less options when you are processing PowerShell objects.
Note: I think the first thing I tried to do in PowerShell was a script that scanned a directory recursively for files containing a CRC in their name, and then check it, or something along these lines. After several hours of trying, I simply couldn't do it while it was relatively straightforward in bash, even with spaces in file names.
And that's not that I like UNIX shell scripting, in fact I hate it, so many footguns, that's why I wanted to try PowerShell, but it didn't fit my needs.
I had to work with Sencha/ExtJS early 2010. It was the same feeling. Yes, it is powerfull, but too much magic happen for something without a clear orientation (at the time, now it is used for data loaded frontend i think). PS i don't understand what it wants me to do.
The language is fine. The interface is now fine, but in 2015 it was the shittiest tty available on modern computers. It's okay since at least early 2021 (when i restarted using windows). I know it should be reasonably better, but i wouldn't trust any PS script written before 2021 to run on my workstation. I run bash scripts i wrote when i started coding.
Still, if you're new to the gig and don't care about free software and commons, you should learn PS (unless you want to work on baremetal or on MC, in this case, bash will be enough).
Bash, awk, grep et alii don't do OOP. But they are close to the data and are powerful.
Compaints about the notation of these tools (e.g. "line noise") are becoming silly now that we've seen the appearance of some Powershell and Python statements. Any non-trivial notation will have to make choices. ^_^
=name will expand to the entry in your PATH. e.g. =ls expands to /usr/bin/ls. So == expands to an executable named =, or rather, it tries to as you probably don't have = in your PATH.
[[ ]] disables expansions (e.g. [[ * = * ]] will work too) so it's not an issue there.
Not sure if this convention is actually documented anywhere.
Random examples:
- https://github.com/PyCQA/isort/blob/main/pyproject.toml#L100
- https://github.com/pypa/pip/blob/main/setup.py#L78
- https://github.com/11ty/eleventy/blob/master/package.json#L1...
One advantage of no-extension is that you can swap the implementation language later without "breaking" shell history for people in your team.
"Ugly" is subjective. If I encountered a file with that extension, I'd assume it uses Bash-specific features and that I shouldn't run this script with another shell.
For most people, YAGNI means using convenient Bash-isms, because their scripts won't ever be run on environments that don't have Bash.
Edit: Admittedly, someone in this thread pointed out the flaw in my argument, there are plenty of cases where you can't assume you have Bash. I still hold that proofing something for all possible environments is itself a YAGNI.
This seems like really bad advice because the number of people writing bash massively massively outnumbers the people writing sh. Regex matching, glob matching, proper parsing, &&/||, no need to quote.
I would say the opposite, enjoy all the bashisms like (( )) [[ ]], numeric for loops, extended globs, brace expansion, ranges, OH GOD YES VARIABLE REFERENCES, and only rewrite when you absolutely have to make it work on sh.
Silently ignoring sub-commands that exit with a non-zero code is not the same thing as "never failing". Your script might return 0, but that doesn't mean it did what you expect it to.
> Also, don't use set -o nounset when set -u will do.
`set -o nounset` is a lot easier to understand for the next person to read the script. Yes, you can always open the manpage if you don't remember, but that is certainly less convenient than having the option explained for you.
What shell are you using that doesn't support `set -o nounset`? Even my router (using OpenWRT+ash) understands the long-form version.
> Only use that for bashisms where there's no POSIX alternative
I totally disagree. You expect people to know the difference between `[[ ... ]]` and `[ ... ]` well enough to know what the bash version is required? You expect the next person to edit the script will know that if they change the condition, then they might need to switch from `[` to `[[`?
How do you even expect people to test which of the two that they need? On most systems, `/bin/sh` is a link to `/bin/bash`, and the sh-compatibility mode of bash is hardly perfect. It's not necessarily going to catch a test that will fail in `ash` or `dash`.
I think the "YAGNI" applies to trying to support some hypothetical non-bash shell that more than 99% of scripts will never be run with. Just set your shebang to `#!/bin/bash` and be done with it.
I totally agree about `pipefail`, though. I got burned by this with condition like below: ``` if (foo | grep -Eq '...'); then ```
Since `-q` causes grep to exit after the first match, the first command exited with an error code since the `stdout` pipe was broken.
Well I meant the former. Very useful for things like init scripts where you would prefer the script do as much as it can to get something online.
> What shell are you using that doesn't support `set -o nounset`?
You're right, this does appear to be in POSIX, so I guess it's fine. But it is unusual to see in my experience.
> You expect people to know the difference between `[[ ... ]]` and `[ ... ]` well enough to know what the bash version is required?
No, I want them to use POSIX semantics until they have to do something Bash-y. Simplicity when it doesn't cost anything extra is best practice.
> Just set your shebang to `#!/bin/bash` and be done with it.
Homebrew, Jenkins, Asdf, etc may provide their own version of Bash that is required rather than the system default, and some systems have no /bin/bash at all. So you should use #!/usr/bin/env bash for Bash scripts and #!/usr/bin/env sh for POSIX Shell scripts. This lets the user override the PATH with their required version of Bash for this script. (and the script itself can check for versions of Bash, and even re-exec itself)
To that end, would it not make more sense to always use `[[ ... ]]` for conditions, when I know my .bash scripts will always be invoked by bash?
Consistency is simple.
> Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail.
Then handle those errors explicitly. The above will catch those error that you did not think about.
Oh how I hate the double square bracket. It is the source of many head scratching bugs and time wasted. "The script works in my machine!" It doesn't work in production where we only have sh. It won't exit due to an error, the if statement will gobble the error. You only find the bug after enough bug reports hit that particular condition.
After a couple shots to the foot I avoid double square brackets at all cost.
EDIT: I should have said, "could be fixed once and for all", "should" is just my opinion.
https://stackoverflow.blog/2022/07/06/why-perl-is-still-rele...
https://stackoverflow.blog/2022/09/08/this-is-not-your-grand...
Python is much cleaner code than Perl for the most part as well.
However, for anything that should run forever, make sure you have a copy of all of its source code AND its libraries AND the source code for it's compiler. Repo rot is a serious problem over time.
I only ever use it on my own systems so awk works fine for me and I use it regularly, but iirc it's not true that you don't have to worry about versions.
Python is where I don't worry about versions. Everyone's got python3 by now (word got round) and most basics, like print(), works just fine in 2.7 (the main py2 backwards compatibility thing I run into is bytes vs unicode strings; if the script needs to work with raw bytes, you'll just need any python3 version). The issue I run into is with Windows people not having python installed, and (worse) not being able to install it in 30 seconds with one command, but that would be the same with awk.
FOO_ARGS=(
# Some explanatory comment
--my-arg 'some value'
# More comments
some other args
#...
)
myCondition && FOO_ARGS+=(some conditional args)
foo "${FOO_ARGS[@]}"I only use it to select the n'th word in a csv-like line. Anything more than that, I need to search stackoverflow for the invocation.
Don't you find its syntax cumbersome?
For me, the additional features that bash provides are much more important than a portability that I'll never need to use.
Bash in --posix mode does that perfectly.
But, more importantly, people will google for how to set cwd to the script directory more often then will google how to go to an absolute path. Having 'cd "$(dirname "$0")"' as reference in an article discussing best practices and the topic of changing the directory early, is a good idea.
people will google for how to set cwd to the script directory more often
The answer should suggest setting $script_dir instead of chdir and refer to it when needed, explaining why chdir is a wrong shell mindset except for a very few special cases. It’s okay for personal use, but inheriting such scripts would be an awful experience, imo.
I think it would be more correct to just use vcs to get the root of a project (or fail if you can't), instead of essentially hardcoding path to the script.
For example if you put your script in helpers/ then someone else did a refactor and moved all of the stuff into cmd/helpers/, any relative reference you put into script is now invalid and your script is doing the wrong thing
project=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
I think an easier way is to find script's location and construct paths for accessing script dependencies, for example (works on Linux & macOS):
script_root="$(cd "$(dirname "$(readlink "$([[ "${OSTYPE}" == linux* ]] && echo "-f")" "$0")")"; pwd)"
source "${script_root}/common.sh"
source "${script_root}/packages.sh"
source "${script_root}/colors.sh" [1]: https://github.com/Mitigram/mg.sh/blob/cbeb206d67fe08be2107deee50acf877f990dbdf/bootstrap.sh#L6$OLDPWD is set when you 'cd'. Also 'cd -' will take you back to the last directory.
Ugh I've seen so many (bash and non bash) cmdline tools that made it utterly annoying like that
The special place in hell goes to people who force users to write
cmd help subcmd
cmd subcmd --help
cmd subcmd --verbose
or ones where you need to write
cmd --option1 subcmd --option2 subsubcomd --option3
and if you go "well but the option for command and subcommand might have same name" DONT NAME THEM THE SAME, that's just confusing people and search results.
Why not use pushd/popd instead?
b) you need to mentally keep the stack in your head when you write the script. And anyone else who would be reading your script. (Edit: including yourself a couple of months/years later)
c) pushd $script_invocation_path is easier to understand and remember.
Eg:
$global:MainScriptRoot = $PSScriptRoot
$global:configPath = Join-Path $PSScriptRoot config
$global:dataPath = Join-Path $PSScriptRoot data
$dirsToProcess = gci -Path $PSScriptRoot -Directory | ? Name -Match '\d+-\w+' | Sort-Object Name
foreach ($thisDir in $dirsToProcess) {
foreach ($thisFile in $moduleFiles) {
. $thisFile.FullName
}
}
Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement.
This presentation sums up the woes of an implementor:
https://archive.fosdem.org/2018/schedule/event/code_parsing_...
Bash scripts have a nasty tendency to grow, sometimes in ways that are disproportional to the bit of extra functionality that is suddenly required. Very quickly, a small quick'n dirty solution can blow up to a compost-heap ... no less dirty, but now instead of a clean-wipe, I'd need a shovel to get through it.
I think my handle speaks for itself as to how much I like bash. But I have had the pleasure of getting handed over bash scripts, hundreds of lines long, with the error description being "it no longer works, could you have a look at it? and the original author both unreachable and apparently having string feelings against comments.
And in many of these cases, it took me less time to code a clean solution in Python or Go, than it took me to grok what the hell that script was actually doing.
# shellcheck disable=SC2154
(That's for not warning about using a variable that hasn't been defined)
Why would they use a shebang? `bash script.sh` works perfectly fine. Ever used a terminal?
Parsing text isn't anything fancy.
It's just knowing what the marker is for a word/item boundary.
For bash, that marker is defined in IFS
The scripts are similar to this one:
https://github.com/debxp/dmenu-scripts/blob/master/dmenu-kil...
It converts to/from the listed formats.
There is also `jc` (written in Python) with the added benefit that it converts output of many common unix utilities to json. So you would not need to parse `ip` for example.
This is a wrapper to jq that also supports yaml and other file formats.
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
PSQL="psql postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$DATABASE_HOST:$DATABASE_PORT/$POSTGRES_DB -t -P pager=off -c "
OLD="CURRENT_DATE - INTERVAL '5 years'"
$PSQL "SELECT id from apt WHERE apt.created_on > $OLD order by apt.created_on asc;" | while
read -r id; do
if [[ $id != "" ]]; then
printf "\n\*\* Do something in the loop with id where newer than \"$OLD\" \*\*\*\n"
# ...
fi
doneI don't do much sql in bash scripts but I do keep some wrapper scripts that let me run queries from stdin to databases in my environment
If i need to run 11 commands in a row, suddenly i need to make sure new tooling is installed in my instance and/or ship a binary?
What if that 11 lines is setting up some networking? Now i need to go write a 400 line go program to use netlink to accomplish the same task? Or should I condense that to 80 lines of go to shell out the commands that replicate the 11 lines of simple bash?
There are plenty of reasons to do this, I have done it more than once. None of those reasons are "crossed an arbitrary magic number of 'lines of shell'".
/s
I’m aware that ZSH has been the default shell since Catalina.
Started using fish a month ago and really liking it.
I think being the de-facto default on Linux as part of "GNU plus Linux" has more to do with it.
Like aliases but they expand in-place, so auto completion friendly, easily modifiable, etc. Love them
$ abbr s sudo
$ s<space> -> sudo
What is used both in case/esac and globbing are "shell patterns." They are also found in variable pattern removal with ${X% and ${X#.
In "The Unix Programming Environment," Kernighan and Pike apologized for these close concepts that are easily mistaken for one another.
"Regular expressions are specified by giving special meaning to certain characters, just like the asterix, etc., used by the shell. There are a few more metacharacters, and, regrettably, differences in meanings." (page 102)
Bash does implement both patterns and regex, which means discerning their difference becomes even more critical. The POSIX shell is easier in memory for this reason, and others.
Wow, for me parallelism is one of the best features of a unix shell and I find it vastly superior to most other programming languages.
Not really; just seems the same as most other dynamic languages. Awk does a lot of stuff for you (the "implied loop" your program runs in, field splitting) that's certainly possible (even easy) to replicate in Python or Ruby, but Awk it's just so much more convenient.
I use it for things like processing the Unicode data files, making some program output a bit nicer (e.g. go test -bench), ad-hoc spreadsheets, few other things. I got started with it as I needed to process some C header files and the existing script for that was in Awk; it worked pretty well for that too.
The Awk Programming Language book is pretty good. GNU Awk has a bunch of very useful extensions, but pretty much everything in the book still works and is useful today. You can get it at e.g. https://archive.org/details/awkprogrammingla00ahoa or https://github.com/teamwipro/learn_programing/blob/master/sh...
The GNU Awk docs are also pretty decent.
If it's not a CSV with quotes to allow for commas inside values (which I think AWK will also fail in), you can use `cut`.
cut -d, -f n
That's not to say I'd do everything in shell. Most stuff fits well into SQl, but when it comes to optimizing processing over TB or PB scale, you won't beat shell+massive hw orchestration.
In python I can use `opt/argparse`, and use the error output from `re.compile` to do this.
Of course this is also possible in bash, but how easy is it to code in comparison, and how maintainable is the result?
If I wanted to parse cli args I would use case on the input to mux out the args. I personally prefer writing cli interfaces this way (when using a scripting language).
while test $# -gt 0; do
case "$1" in
-f|--flag) shift; FLAG="$1";;
esac
shift
donesed is the thing that handles shell regular expressions for shellscripts.
This rule of ten lines or less is clearly meant as an illustrative guideline. Obviously if you have a shell script that has 11 lines, but does what it has to do reliably, nobody will be bothered.
The idea that the rule is trying to convey is "don't write long, complex programs in shell". Arguing about exact numbers or wording here is detracting from the topic at hand.
It doesn't. You now have 4 numbers.
prinf -- "fmt str here..." "$carefully" "$quoted" "$args"It's a little more wordy if you're use to C or Bash. But hands down it's one of my favorite languages for slicing dicing data. No need for 3rd party libraries or binaries. No need to learn a bunch of weird awk/jq syntax which is only useful for those two tools (yay, lets learn 3 language instead one?). Plus, most of the structure translates over to C#, and you can integrate C# code directly into your POSH code if desired, as well as access pretty much any low level C# methods directly.
Working with strings? Pretty much any/every tool you could want to slice and dice strings.
The POSH REPL is amazing. You have far more flexibility around interacting with the command line than you do with Python. It's both a shell and a true language. As with any language, there are ISMs, but far fewer footguns than any other language I've spun up.
Cross platform as well with 6.0+
Intellisense ON the commandline (did I mention the awesome REPL?). Hands down one of the best built-in parameter/args/help parsing I've encountered across any language. Debugging? Amazing in vscode. And can be done strictly from the commandline as well (dynamic breakpoints? You've got it, drop you right into your catch block with an interactive shell so you can see the current status of any/all variables and manipulate them live and resume if desired)
Okay, I'm done shilling for POSH. It's hands down one of my favorite shells/languages for doing POC work, or writing utility functions. Treat it more like Python than bash. But realize that you can easily use that Pythonic-esque code right inside your shell.
The nice thing with Powershell is that it's not verbose, but the arcane abbreviations are actually quite a bit easier to remember and discover than bash. What mixes people up is that in documented examples and reusable scripts, it makes sense to use the full, canonical name, which looks aesthetically different coming from a Unix background.
Here's what it might actually look like to check a JSON file that has an array of file metadata objects, and delete the ones that have been processed (this includes one user-defined alias, cfj, for "ConvertFrom-Json"):
gc queue.json | cfj | where status -eq processed | ri
jq '.[] | select(.status == "processed")' queue.yaml | xargs rm
It's even better if you're dealing with network resources:
$api = https://api/
irm $api/queue.json | ?{ $_.status -eq processed } | `
%{ irm -M Delete $api/files/$_.file }
api=https://api/
curl $api/queue.json | \
jq '.[] | select(.status == "processed") | .file' | \
while read file; do curl -X DELETE "$api/files/$file"; done
Get-Content -Path queue.yaml | ConvertFrom-Json | `
Where-Object -Property status -Eq 'processed' | Remove-Item
I didn't want to clutter this with too many examples but I'll reiterate that Powershell is a shell. It invokes your Unix commands just fine. For example, if you forgot what the option to Get-Date is, to get a Unix-style timestamp (it's 'get-date -uf %s' so not exactly hard to use): you can just type 'date +%x'. In the example above, I could have used 'cat' and 'xargs rm' instead of 'gc' and 'ri'. So it's not like you have to buy the whole kit and kaboodle right away, either.
And OSH implements almost all of bash
That is, The posix shell spec is missing a lot of reality. It’s not very actively maintained, unfortunately
The canonical example is not having local vars, which basically every shell supports
POSIX was primarily intended a descriptive specification, rather than a prescriptive.
That is, it attempted to document and standardize the common behaviour found on many platforms during the Great Unix Wars, rather than say "hey we thought of this great new thing and released a spec, go implement it!", which is more how, say, web standards work. I does/did have some of that, but it was never the main goal.
These days "whatever Linux does" is the de-facto standard, for better or worse, and the need for POSIX is much less.
> It’s not very actively maintained, unfortunately
Because that's how standards works. If there is enough interest a new standard will be made, but right now, bash isn't it.
* local keyword * 2 or 3 variable expansion tricks (string replacement etc) * pipefail
But with those, I think the spec isn't too bad. I've been writing (imo) high-quality (and shellcheck compliant) shell scripts for a decade, and _always_ try to be as pendantic about being posix compliant where humanly possible. Sometimes things are a _little_ harder (no sarcasm here), but it's really quite doable once you get the hang of it.
You scripts then end up being far more portable, and as stated below, MUCH easier to read. The trick is, to write readable code (which is hard enough for most people, regardless the language anyway :p)
e.g. (a very simple example), and an addition to Shrikant's post, avoid the terse double ampersant 'and' (&&) and double pipe 'or' (||) operators, in normal calls. do e.g. ```sh if ! external_command; then do something with failure fi ``` Rather then `external_command || do something with failure`
These can get up becoming really hard to read once they become longer, where reading a stupid "if" statement is easy to comprehend. Readability over saving number of lines.
One important hint left forgotten, always make sure the last line of your file is `exit 0` or similar. This is more a 'weak' security feature. Prevent people from appending files and executing stuff instead. Gives a known exit point.
Another addition would be to actually favor single quotes for pure strings, and use double quotes where you expect the shell to do something with your string (true for most cases, but there's plenty of strings that benefit from single quotes, and hint the reader what to expect). Also integers should never be quoted (as it would turn then into strings), which shows a 'bad' example in the trace statement, you are now comparing the strings '0' and '1' rather then the number. Best use something easier to read in that case.
One thing I will pick up from this post for sure, is the trace hint, I'm adding that to my scripts, but probably a little more tunable.
```sh set -eu if [ -n "${DEBUG_TRACE_SH:-}" ] && \ [ "${DEBUG_TRACE_SH:-}" != "${DEBUG_TRACE_SH#"$(basename "${0}")"}" ] || \ [ "${DEBUG_TRACE_SH:-}" = 'all' ]; then set -x fi
echo 'Hello World'
exit 0 ``` Though I'll probably just settle for one of those keywords, I like 'all' the best atm. This would run the trace only if this special keyword is given, or if the variable contains the name of the script. I initially had `basename "${0%%.sh}"` but that makes things like `test,test1,test2` impossible, though that only helps if the extension is used ;)
While not needing be part of this list, I personally also always use functions, unless the script really is less then a handful lines, like google's bash-guide, always have a main etc.
In the end though I do disagree writing bash and bashisms. There's plenty of posts about that though; kind of like the whole C++ 'which version to use' discussion, where bash tends to be inconsistent in itself.
Shameless plug: See some of my older stuff here https://gitlab.com/esbs/bootstrap/-/blob/master/docker_boots...
You have a level of control over things popping up too
Though Ruby makes it confusing AF because there are two quoting types for both strings and symbols, and they're different. (%Q %q %W %w %i) I can never remember which does which.... the letter choice feels really arbitrary.
https://elixir-lang.org/getting-started/sigils.html#strings-...
Elixir has sigils, which are useful for defining all kinds of literals easier, not just strings:
https://elixir-lang.org/getting-started/sigils.html#strings-...
You can also define your own. It's pretty great.
$X=q( foo() )
I agree that this qualified quotation is really underutilized.
I also write shell scripts, but I'm just curious what you would think about a comparison.
So for somebody just coming from Windows, this isn't just the wrong mindset for shell scripts, more dangerously it's a mindset they find appealing, because it matches their prior experience on Windows better.
That was the intention of my comment. Because the rest of the post (or most of it for sure) is not subjective.
Bash and gawk don't have direct in-language support for OOP, but can be used/abused to do things in an oop manner. aka without in-language support (abstractions/api), takes much more effort to do/understand OOP approach.
Painful, ugly, unpleasant?
Code that we might deem "beautiful" may not even compile. ^_^
Does the tool do what it needs to do without getting fussy, gobbling RAM, and requiring a small army of maintainers to feed the monster? There is most definitely a place in technology for that kind of sanity.
There is something to be said for a design that doesn't bring with it a brigade of Opinions-As-A-Service proponents.
Any non-trivial notation is going to require learning. There is absolute value in learning. I think of many flexible and effective notations in tooling that has literally been the foundation of modern computing.
Maybe we are in the age of the Mono-Notation Luxury Coder. ^_^
The performance difference was night and day.
The biggest issue with PowerShell is that PowerShell Core is not yet default on Windows 10/11 and Windows Server. That should be Microsofts highest priority for PowerShell.
Bit off-topic, but I worked with ExtJS around the same time, and I found it one of the most confusing development experiences I ever had. "It's so easy, just add this one property to this deeply nested object you're passing to this function!" Thinking back on it, it's a really good example of "simple vs. easy". It didn't help it gave not a peep if you got one of those data structures wrong (capitalisation typo, wrong location, etc.)
Plus in hindsight the whole idea of "OS semantics in your browser!" was never a good one to start with, although that wasn't as obvious to me at the time.
ExtJS (on netbeans with windows server 2004) was always my worst development experience, it was my first internship too. That's probably the reason why it took so long (and a bag of money) for me to try JS, an IDE and developing on windows again (I only used Windows for CTFs and AndroidStudio).
For things in my PATH I drop any suffixes like that.
This died a long time ago with the pervasive use of NPM and PIP and the likes.
Most developers probably run a lot of random unchecked shit all the time with local user privileges today without a blink.
Somehow people are ready for all this, but are still afraid to run a random shell script from the internet. I guess this fear is one of our chances to explain how NPM and PIP can be dangerous.
I don't check everything I download from the internet; I don't think anyone does. It depends on what it is, where I'm getting it from, where I'm running it, etc. There are certainly some things I will review carefully, but other things I give just a quick check to see it's not in complete shambles, and others I barely check at all. I typically run the latest Vim from master, do I check every patch to see if after 30 years Bram finally sneaked in a crypto miner or password stealer? Do the people who package Vim for the Linux distros?
declare -A outer=(
[inner]="_inner"
)
declare -A _inner=(
[key]="value"
)
declare -n inner="${outer[inner]}"
echo "${inner[key]}"
# value
Try in Python to make a nested defaultdict you can access like the following.
d = <something>
d["a"]["b"]["c"] # --> 42
Edit: Dang it, I mean arbitrary depth.
c = defaultdict(lambda: 42)
b = defaultdict(lambda: c)
a = defaultdict(lambda: b)
a["a"]["b"]["c"] # --> 42Otherwise I'm not sure of a mainstream language that would let you do a.get(x).get(y) == 42 but a.get(x).get(y).get(z) == 42, unless you resorted to monkey patching the number type, as it implies 42.get(z) == 42, which seems.. silly
Not trying to be combative, but learning any new language is going to require some dogfooding and digging in order to be as efficient as something you're already familiar with. I use bash and PowerShell daily. I can't think of any one thing I couldn't do in one or the other. Bash usually requires tying in another tool though. It's not strictly bash at that point. People saying stuff like jq. I rarely find that pre-installed. And if it's your machine, you can just as easily install something that doesn't require you to learn multiple languages (jq is a language).
I here you. I've been there. I'd lazily prod you to give POSH another shake. I've no horse in this race but I think you're missing out if you weren't able to accomplish in POSH what you were able to do in bash.
There is simply no need for the kind of extensive text processing common in Linux because every command returns an object whose fields can be directly referenced. Combined with the ConvertTo-Json command this is incredibly powerful. Honestly it seems like you are attempting to do things in PowerShell the bash way instead of the PowerShell way.
" I think the first thing I tried to do in PowerShell was a script that scanned a directory recursively for files containing a CRC in their name, and then check it"
I wrote a PowerShell script that recursively scanned every filed and folder on our fileshares and wrote the permissions to a file for later indexing.
Or there's some registry hacks to remove the limit from regular paths.
https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...
ls -File | % { $fh = Get-FileHash $_ SHA1; if ($_.Name -match $fh.hash) {$_} }Anyways, it was a long time ago, maybe I will give PowerShell a retry at some point.
Five There's five numbers in this industry 0, 1, 3, 4, 5 and infinity
Wait, I'll come in again
The 7 being for design. If there are more than 7 boxes on the whiteboard, try again.
Unicode character can have more than 7 font boxes associated with one character box and still be a valid determinate character form.
log base 2 of 4 is only 16bits
For those that haven't taken the time yet, I think this is a good place to start:
https://learnxinyminutes.com/docs/awk/
Of course, some people do very advanced things in awk and I absolutely agree that 1 hour of study isn't going to make you a ninja, but it's absolutely enough to learn the awk programming paradigm so that when the need arises you can quickly mobilize the solution you need.
For example: If you're quick to the draw, it can take less time to write an awk one liner to calculate the average of a column in a csv than it does to copy the csv into excel and highlight the column. It's a massive productivity booster.
There are people who have asked me scripting questions for over a decade, who will not read this for some reason.
It could be read in an hour, but not fully retained.
[1] https://www.gnu.org/software/gawk/manual/gawk.html
For the sarcasm impared among us, everything above this, but possibly including this sentence is sarcasm.
The quick one ~ true ~ fix was ! or #! without the 1024k copyright.
s-expression notation avoids the issue with (."contents")
MS Windows interpretation is much more terse & colorful.
Submitted yesterday:
Learn to use Awk with hundreds of examples
https://github.com/learnbyexample/Command-line-text-processi...
Printing based on another field, example gets UIDs >= 1000:
awk -F: '$3 >= 1000 {print $0}' /etc/passwd
Here's a nice example of something similar: https://drewdevault.com/dynlib
Scripted cholmskey grammar ( https://en.wikipedia.org/wiki/Universal_grammar ) to unleash the power of regular expressions.
I would tend to agree that scripts for downloading from the internet should have a '.sh' extension to make it clear that it's a script as opposed to a binary.
When a malicious package is found on NPM or PIP, it will get removed. However, it is quite unlikely that a website will be taken town for a malicious script (or only after a long time).
I really doubt that most readers of HN would run a random script unless it comes from a source they trust (trusted enough to least to remove a malicious script in a timely fashion).
If I contribute to Nextcloud or write an app for it, I need to run npm. If I want to run PeerTube, I need to run npm. They both pull a shitload of dependencies I can't possibly review.
I personally avoid building anything using NPM and advocate for fewer / no dependencies, or for using dependencies packaged by reputable entities like Debian, but what can I do? I can't build everything myself.
Am I committed to being an idiot?
Also d["a"] and d["a"]["b"] aren't 42.
d["a"]["b"]["c"] == (d["a"]["b"])["c"]
d = magicdict()
is42 = d["foo"]["bar"]["baz"]
# -> You can read any path and get a default if it doesn't exist.
d["hello"]["world"] = 420
# -> You can set any path and d will then contain { "hello": { "world": 420 }
I feel like we're maybe too in the weeds, I should have just said "now have two expressions in your lambda."
I have a hard time understanding any use case outside of such mocking.
# One
a = d["a"]["b"]["c"]
# Two
a = d["a"]["b"]
b = a["c"]