Interesting commands for the Linux shell

Interesting commands for the Linux shell(lopezferrando.com)

372 points by victorlf 8 years ago | 129 comments

sethrin 8 years ago |

The ones I get a lot of use out of are:

curly brace substitution:

  $ mkdir -p new_project/{img,js,css}
  mkdir -p new_project/img new_project/js new_project/css
  $ mv some_file.txt{,.old}
  mv some_file.txt some_file.txt.old

Caret substitution:

  # systemctl status mysql.service
  -- snip output --
  # ^status^restart
  systemctl restart mysql.service

Global substitution (and history shortcut):

  $ echo "We're all mad here. I'm mad. You're mad."
  we're all mad here. I'm mad. You're mad.
  $ !!:gs/mad/HN/
  we're all HN here. I'm HN. You're HN.

I have a (WIP) ebook with more such tricks on GitHub if anyone is interested: https://tenebrousedge.github.io/shell_guide/

godelski 8 years ago | |

Curly brace expansion is one of the most useful and overlooked topic in these kind of posts. And I'm always surprised it isn't mentioned with compiling.

g++ -o foo{,.cpp}

tych0 8 years ago | | |

Worth noting that for the particular case you've cited, you can just use make, even without a makefile:

  ~ cat test.cpp 
  #include <iostream>
  
  using namespace std;
  
  int main()
  {
      cout << "hi\n";
      return 0;
  }
  ~ stat Makefile
  stat: cannot stat 'Makefile': No such file or directory
  ~ 1 make test
  g++     test.cpp   -o test
  ~ ./test
  hi

skykooler 8 years ago | | |

also, useful to know - zsh supports tab-completion for curly brace expansion.

agumonkey 8 years ago | |

I should use history substitution a lot more. Thanks

sethrin 8 years ago | | |

Everyone should probably have

  $ sudo !!

ingrained in muscle memory. The other even more useful one is !$ to get the last word of the previous line. It's probably the terminal feature I use most.

disconnected 8 years ago |

Disappointingly, this list treats "yes" like a toy that just prints things over and over, and doesn't mention actual useful uses for "yes", like accepting all defaults without having to press enter.

Practical example: when you are doing "make oldconfig" on the kernel, and you don't care about all those questions:

yes "" | make oldconfig

Or, if you prefer answering no instead:

yes "n" | yourcommand

Also, the author refers to watch as a "supervisor" ("supervise command" - his words). That is bad terminology. Process supervision has well defined meaning in this context, and watch isn't even close to doing it.

Examples of actual supervisors are supervisord, runit, monit, s6, and of course systemd (which also does service management and, er, too much other stuff, honestly).

tych0 8 years ago | |

The kernel also recently (4.8-ish?) introduce an `olddefconfig` command, so you don't have to mess around with yes.

feelin_googley 8 years ago | |

"...("supervise command" - his words). That is bad terminology."

"Examples of actual supervisors are..."

1999-2001

http://cr.yp.to/daemontools/supervise.html

runit and s6 are copies of daemontools.

Otherwise I liked your comment about use of yes.

skocznymroczny 8 years ago | |

Unfortunately it doesn't work when trying to connect via ssh and warning about unknown key pops up.

hnlmorg 8 years ago | | |

That is intentional behavior on the part of `ssh`. You can still disable that prompt within `ssh` (though it's not recommended):

    ssh -oStrictHostKeyChecking=no 192.168.0.100

...or by adding rules in /etc/ssh/ssh_config or ~/.ssh/config

    Host 192.168.0.*
        StrictHostKeyChecking no

hnlmorg 8 years ago |

Some handy tips there but I would recommend changing some of the `find` examples:

    find . -exec echo {} \;      # One file by line

You don't need to execute echo to do that as find will output by default anyway. There is also a `-print` flag if you wish to force `find` to output.

    find . -exec echo {} \+      # All in the same line

This is think is a dangerous example because any files with spaces will look like two files instead of one delimited by space.

Lastly, in both the above examples you're returning files and directories rather than just files. If you wanted to exclude directories then use the `-type f` flag to specify files:

    find . -type f ...

(equally you could specify only directories with `-type d`)

Other `find` tips I've found useful that might be worth adding:

* You don't need to specify the source directory in GNU find (you do on FreeBSD et al) so if you're lazy then `find` will default to your working directory:

    find -type f -name "*.txt"

* You can do case insensitive named matches with `-iname` (this is also GNU specific):

    find -type f -iname "invoice*"

joosters 8 years ago |

   grep -P "\t"

Not criticising the author here, as grep -P is good, but you might not also know that you can enter tabs and other codes in bash by pressing ctrl-v. So you could also type:

   grep "[ctrl-v]TAB"

reacweb 8 years ago | |

I know Linux since 1992 (main OS since 2009) and I did not know that. thank you very much

l0b0 8 years ago | |

The first alternative is still preferable, since it's actually readable.

You can also do $'\t' in at least Bash (and probably Zsh).

joosters 8 years ago | | |

Oh, agreed. But the ctrl-v trick is useful in general anywhere where you'd like to put a special character in an input or command.

michaelcampbell 8 years ago | |

Under the presumption that he means `bash` when he says "the Linux shell", does the Ctrl-v trick work in both emacs and vi modes?

Pete_D 8 years ago | | |

Yes; it's a readline command called quoted-insert and is by default bound to ctrl-V in both modes.

    $ set -o vi; bind -q quoted-insert
    quoted-insert can be invoked via "\C-v".

flachsechs 8 years ago | |

this works in vim also.

vram22 8 years ago | | |

And Ctrl-V does not work in GVim on Windows, but Ctrl-Q does instead. E.g. if you have a text file with mixed Unix and DOS/Windows line endings, like Ctrl-J (line feed == LF == ASCII 10) and Ctrl-M (carriage return == CR == ASCII 13) + Ctrl-J, you can search for the Ctrl-Ms with /Ctrl-Q[Enter] and for tabs with /Ctrl-Q[Tab]. When you type the Ctrl-Q, it does not show as Ctrl-Q, but as a caret (^). And you can also use the same quoted Ctrl-M (Enter) pattern in a search and replace operation to remove them globally, by replacing them with a null pattern:

:%s/Ctrl-Q[Enter]//g[RealEnter]

or replace each tab with 4 spaces:

:%s/Ctrl-Q[Tab]/[4 spaces]/g[RealEnter]

where [RealEnter] means an unquoted Enter.

This is mainly useful for making such changes in a single file (being edited) at a time. For batch operations of this kind, there are many other ways to do it, such as dostounix (and unixtodos), which come built-in in many Unixen, or tr, maybe sed and awk (need to check), a simple custom command-line utility like dostounix, which can easily be written in C, Perl, Python, Ruby or any other language that supports the command-line interface (command-line arguments, reading from standard input or filenames given as command line arguments, pipes, etc.).

stiangrindvoll 8 years ago |

Doesn't matter how long you've been using a GNU/Linux shell, you'll always learn something new every day. Thanks for this.

hnlmorg 8 years ago | |

Funny enough I was saying this to a work colleague yesterday when I was looking at a shell script and scratching me head wondering how it was working when a variable named $RANDOM wasn't assigned a value. After a little investigation it turned out Bash (and possibly other $SHELL's?) returns a random number with each call of the $RANDOM variable. Handy little trick.

Tepix 8 years ago | | |

Indeed! Also if $RANDOM is too small for your needs, you can use $RANDOM$RANDOM (and it will have different values)

vacri 8 years ago | | |

There is also the inbuilt variable $SECONDS, which seconds since the instance of bash was started. Makes it really easy to do 'time elapsed' at the end of a bash script.

    $ echo $SECONDS
    83783
    $ echo $SECONDS
    83784
    $ echo $SECONDS
    83787

Aissen 8 years ago | |

Indeed, I was about to say this. I clicked thinking "No way I'll learn something here". Turns out there's 2 or 3 commands that I didn't know and might prove useful!

kps 8 years ago | |

  > Doesn't matter how long you've been using a GNU/Linux shell

Or a Unix shell, even.

Tepix 8 years ago |

> 6. List contents of tar.gz and extract only one file

> tar -ztvf file.tgz

> tar -zxvf file.tgz filename

Tar no longer requires the modifier for the compression when extracting an archive. So no matter if it's a .tar.gz, .tar.Z, .tar.bz2, etc you can just use "tar xvf"

kaushalmodi 8 years ago | |

And the same for compression..

    tar caf foo.tar.xz foo/

The extension of the file after the `f` switch tells tar what compression to use.

c - compress

a - auto

otherflavors 8 years ago | |

by tar you mean gnu tar

abhirag 8 years ago |

As long as we are on the topic of Linux shell commands I would like to share a tip which has helped me, whenever you are using wildcards with rm you can first test them with ls and see the files that will be deleted and then replace the ls with rm. This along with -i and -I flags makes rm just a tad bit less scary for me. Kinda basic but hopefully somebody finds it helpful :)

vram22 8 years ago | |

As mbrock says, you can also use echo instead of ls for that. And "echo *" serves, in a pinch, as a rudimentary ls, when the ls command has been deleted and you are trying to do some recovery of a Unix system. Similarly dd can be used to cat a file if cat is deleted: dd < file

And in fact whenever you want to do some command like:

cmd some_flags_and_args_and_metacharacters ...

you can just replace cmd with echo first to see what that full command will expand to, before you actually run it, so you know you will get the result you want (or not). This works because the $foo and its many variants and other metacharacters are all expanded by the shell, not by the individual commands. However watch out for > file and >> file which will overwrite or append to the given file with the output of the echo command.

dotancohen 8 years ago | | |

Knowing how to cope with a deleted `ls` and `cat` sounds like a horrible battle scar to carry. How on Earth did you ever get into that situation?

crazybigdan 8 years ago | |

You can also use M-? to list all the files.

So typing 'rm foo* ' and pressing Alt-? (on non-OSX computers) will give you a list of what your wildcard would expand to.

M-* does the same thing but actually expands the wildcards, instead of just showing what that expansion would be.

jcranberry 8 years ago | | |

Still works with ESC on OSX

mbrock 8 years ago | |

Or echo instead of ls.

sethrin 8 years ago | |

Using find to do that is a much better idea. The author shows this as the first examples in #9.

aembleton 8 years ago |

MMV is the latest interesting one I've found recently: https://ss64.com/bash/mmv.html

For example, I can do the following:

   mmv "flight.*" "flight-new.#1"

and this will rename all of my files that start with flight. to flight-new. preserving the file extension. So useful, when you've got a bunch of different files with the same name but with different extensions such as html, txt and sms.

pmoriarty 8 years ago | |

Check out qmv from renameutils.[1] Using it you can use your $EDITOR to batch-rename files. It does sanity checking like making sure you don't rename two files to the same name and lets you fix such mistakes before trying again.

[1] - http://www.nongnu.org/renameutils/

akx 8 years ago | |

I use this `rename` (available from Homebrew as `rename`): http://plasmasturm.org/code/rename/

leephillips 8 years ago |

I probably should be embarrassed to admit I didn't know about much of the stuff on this page about parameter expansion:

http://wiki.bash-hackers.org/syntax/pe

Bash is insane.

chubot 8 years ago | |

It is a horrible language design, which was impressed upon me after implementing it for my shell Oil. Here are some pathological cases I pointed out:

${####}: http://www.oilshell.org/blog/2016/10/28.html

Each # means a different thing.

${foo//z//}: http://www.oilshell.org/blog/2016/10/29.html

There are three different meanings of / there.

"${a[@]}" -- http://www.oilshell.org/blog/2016/11/06.html

You need 8 punctuation chars in addition to the array name to correctly interpolate it. Every other way gives you word splitting issues.

michaelcampbell 8 years ago |

I guess I'm just overly sensitive (and maybe English is not the poster's first language), but I cringe at "the Linux shell"... I have used ksh, bash, (a little) csh, and zsh over the years, and love the architecture that makes "the shell" just another non-special binary that's not unique to the OS.

And yes, a lot of the things mentioned work in a lot of shells, but some don't, or act differently.

testestx 8 years ago | |

I would find it a little if it were called the GNU or the GNU/Linux default shell

Exuma 8 years ago |

Here are some that I use in Pet:

Description: Remove executable recursively for all files

    Command: chmod -x $(find . -type f)

Description: List files with permissions number

    Command: ls -l | awk '{k=0;for(i=0;i<=8;i++)k+=((substr($1,i+2,1)~/[rwx]/) *2^(8-i));if(k)printf("%0o ",k);print}'

Description: Output primary terminal colors

    Command: for i in {0..16}; do echo -e "\e[38;05;${i}m\\\e[38;05;${i}m"; done | column -c 80 -s '  '; echo -e "\e[m"

Description: NPM list with top-level only

    Command: npm list --depth=0 2>/dev/null

Description: Show active connections

    Command: netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | sed -e 's/::ffff://' |cut -d: -f1 | sort | uniq -c | sort -rn | head

xelxebar 8 years ago | |

While we're here sharing random cli-fu, you might like `find`'s native ability to spawn processes. Check out the `-exec` flag:

    find . -type f -exec chmod -x {} \+

schoen 8 years ago | | |

This is more correct than the previous poster's approach, because it will work with files that contain space characters, while the previous poster's version breaks in this situation.

Another approach that works is

  find . type f -print0 | xargs -0 chmod -x

although find's built-in -exec can be easier to use than xargs for constructing some kinds of command lines.

Exuma 8 years ago | | |

Nice, thanks for that revision

a3_nm 8 years ago |

Nice! Point 23 is a bit misleading though: comm only works on sorted input files. Also, "disown -h" in Point 21 works on bash but not on zsh. Also, in Point 22, "timeout" only works if the command does not trap SIGTERM (or you have to use -k).

ptero 8 years ago |

> If a program eats too much memory, the swap can get filled with the rest of the memory and when you go back to normal, everything is slow. Just restart the swap partition to fix it: sudo swapoff -a; sudo swapon -a

Is this true? Not impossible, but I am surprised. If true, what is this fixing? In my naive view (never studied swap management), if at the current time a page is swapped out (and by now we have more memory -- we can kill swap completely and do fine), it should get swapped in when needed next time. As there is more memory now it should not, in general, be swapped out again.

If true we are exchanging a number of short delays later for a longer delay now, which to me hardly looks like a win.

xearl 8 years ago |

nice collection! upvoted for "15. `namei -l`" alone, which is far too little known.

a better "20. Randomize lines in file":

  shuf file.txt

instead of

  cat file.txt | sort -R

(sort -R sorts by hash, which is not really randomisation.)

schoen 8 years ago | |

> (sort -R sorts by hash, which is not really randomisation.)

I looked at the source code for GNU sort and what they're doing is reading 16 bytes from the system CSPRNG and then initializing an MD5 digest object with those 16 bytes of input. Then the input lines are sorted according to the hash of each line with the 16 bytes prepended.

Although they should no longer use MD5 for this, I don't think we know anything about the structure of MD5 that would even allow an adversary to have any advantage above chance in distinguishing between output created this way and an output created via a different randomization method. (Edit: or distinguishing between the distribution of output created this way and the distribution of output created via another method!)

The output of sort -R is different on each output and ordinarily covers the whole range of possible permutations.

  $ for i in $(seq 10000); do seq 6 | sort -R | sha256sum; done | sort -u | wc -l
  720

xearl 8 years ago | | |

Sorting by hash will always cluster lines with equal content together, whereas true randomisation won't.

Eg `(seq 3; seq 3; seq 3) | sort -R`.

lathiat 8 years ago | |

> namei -l /path/to/file

This is fantastic, a game changer for me.. I often give people a command like ls -lad /path /path/to /path/to/file

Thanks!

trapperkeeper74 8 years ago | |

And prefer redirection to cat.

    whatever < file.txt

not

    cat file.txt | whatever

Also, there is no need for - or z on GNU tar in t or x modes:

    tar tf whatever.tgz
    tar tf whatever.tar.bz2
    tar tf whatever.txz
    ...

    tar xf whatever.tar.gz
    tar xf whatever.tar.Z
    ...

all work just fine

TorKlingberg 8 years ago | | |

I know whatever < file.txt is slightly more efficient, but there is value is keeping things going left to right with pipes in between. It makes it easy to insert a grep or sort, or swap out a part.

kleff 8 years ago |

Never knew about readlink for files, but I use

  pwd -P

all the time to get the real full path (no symlinks) of the current directory. Really easy to remember as well, Print Working Directory.

kazgul 8 years ago | |

That is not the same use case.

You could use this to get the current location of the script being run (not the location where it runs).

   readlink -f $0

yodsanklai 8 years ago |

I really like the "ag" command. Very convenient to grep in a bunch of files filtered by type. Example:

    ag --ocaml to_string

Very fast and simple syntax.

kaushalmodi 8 years ago | |

Now try rg (ripgrep). It has a much more robust file ignore handling than ag in my experience.

When I last tried, ag couldn't handle this in .ignore file:

    !foo.1
    foo.*

That would ignore all foo.* files except foo.1 in rg.

Also rg is a bit faster than ag for my use cases.

VeejayRampay 8 years ago | | |

Is it possible to register new file formats in ripgrep though (say for example slim files so that I could do rg -tslim PATTERN)? Last time I checked it was somehow possible but way too complicated for so common a task.

hathym 8 years ago |

cached version: https://webcache.googleusercontent.com/search?q=cache:JQ7qBa...

yellowapple 8 years ago |

You'll definitely want to check `free -m` before calling `swapoff` to make sure you really have enough memory for everything in swap, lest you want to invoke the wrath of the OOM killer.

Pete_D 8 years ago |

Not a command as such, but I recommend skimming the readline man page[0] and trying some of the bindings out to see if you're missing out on anything. I went years without knowing about ctrl+R (search command history).

[0] http://man7.org/linux/man-pages/man3/readline.3.html#EDITING...

vram22 8 years ago |

Not really a single command, it's a one-liner - a pipeline, but might be interesting, not only for the one-liner but for the comments about Unix processes that ensued on my blog:

UNIX one-liner to kill a hanging Firefox process:

https://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-ha...

sillysaurus3 8 years ago |

Throw this in your ~/bin as a script named math:

  #!/bin/sh
  scale=4 # results will print to the 4th decimal
  echo "scale=$scale; $@" | bc -l

Now you can do math.

  $ math '1+1'
  2
  $ math '2/3'
  .6666

This is especially useful in shell scripts with interpolated variables:

  x=10
  x=`math $x - 1`

xelxebar 8 years ago | |

Alternatively, you could use an alias:

    # .bashrc
    alias bc='bc --mathlib'

and a .bcrc file:

    # .bcrc
    scale = 4

Actually, this is what my .bcrc looks like:

    scale = 39

    k_c = 299792458                   # Speed of Light
    k_g = 6.67384 * 10^-11            # Gravitation
    k_atm = 100325                    # Atmospheric pressure
    k_h = 6.62606957 * 10^-34         # Planck's constant
    k_hbar = 1.054571726 * 10^-34     # H Bar
    k_mu = 1.256637061 * 10^-6        # Vacuum permeability
    k_ep = 8.854187817 * 10^-12       # Vacuum permittivity
    k_epsilon = 8.854187817 * 10^-12  # Vacuum permittivity
    k_e = 1.602176565 * 10^-19        # Elementary charge
    k_coulomb = 8.987551787 * 10^9    # Coulomb's constant
    k_me = 9.10938294 * 10^-31        # Rest mass of an electron
    k_mp = 1.672621777 * 10^-27       # Rest mass of a proton
    k_n = 6.02214129 * 10^23          # Avogadro's number
    k_b = 1.3806488 * 10^-23          # Boltzmann's constant
    k_r = 8.3144621                   # Ideal gas constant
    k_si = 5.670373 * 10^-8           # Stefan-Boltzmann constant
    k_sigma = 5.670373 * 10^-8        # Stefan-Boltzmann constant
    k_mt = 5.97219^24                 # Mass of Earth (Tierra)
    k_rt = 6.371 * 10^6               # Mean radius of Earth (Tierra)

    pi = 3.1415926535897932384626433832795028841968

    # requires --mathlib
    define t(x) { return s(x)/c(x); }
    define as(x) { return 2*a(x/(1+sqrt(1-x^2))); }
    define ac(x) { return 2*a(sqrt(1-x^2)/(1+x)); }
    define at(x) { return a(x); }
    define csc(x) { return 1/s(x); }
    define sec(x) { return 1/c(x); }
    define cot(x) { return c(x)/s(x); }

ghotli 8 years ago |

ctrl-r for fuzzy searching your history is all you really need to know.

husamia 8 years ago |

cutler 8 years ago |

CentOS 7 specifies `man rename` : rename [options] expression replacement file...

The rename example fails with "rename: not enough arguments".

jwilk 8 years ago | |

Unfortunately, there are two incompatible rename programs in the wild. :(

Debian ships this: https://metacpan.org/pod/distribution/File-Rename/rename.PL

Most(?) other distros ship the one from util-linux: http://man7.org/linux/man-pages/man1/rename.1.html

loudmax 8 years ago | |

Don't read it like this: rename <expression> <replacement file> ...

Read it like this: rename <expression> <replacement> <file(s)...>

Argument 1 is a string (not a regex), argument 2 is string to replace the old string, and arguments 3+ is a file, or list of files, or a glob.

The man page includes a warning about the lack of safeguards. It is unfortunate that rename doesn't have an equivalent of mv or cp's -i flag because it's so easy to overwrite files if your expressions aren't exactly correct.

On some Ubuntu systems, I've seen another version of rename that's actually a Perl script that uses regular expressions. I think that version lets you set some safeguards, which are sorely lacking in the binary version of rename that ships with CentOS.

wyoh 8 years ago | |

It's available in Fedora under the name "prename". I've asked the maintener to make it available for EPEL 7, will this work for you?

sirwitti 8 years ago |

Thanks, every once in a while these lists are useful!

jheriko 8 years ago |

.... or just use a modern os where all this stuff is intuitive and easy :P