Nice article. Really easy to follow introduction.

I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit.

One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. '<()', nope, "command as file argument to other command unix," nope. The first couple of times I tried to use it, I knew it existed but struggled to find any documentation. "Damnit, I know it's something like that, how does it work again?..."

Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these things exist....

Anyone know a good resource I should be using when I find myself in a situation like that?

  $ unzip <(cat z)
  Archive:  /dev/fd/63
    End-of-central-directory signature not found.  Either this file is not
    a zipfile, or it constitutes one disk of a multi-part archive.  In the
    latter case the central directory and zipfile comment will be found on
    the last disk(s) of this archive.
  unzip:  cannot find zipfile directory in one of /dev/fd/63 or
        /dev/fd/63.zip, and cannot find /dev/fd/63.ZIP, period.

    cat z | unzip   # I know, uuoc, demo only

  $ man bash
  /<\(
  

    /<\(

Once you discover <() it's hard not to (ab)use it everywhere :-)

    # avoid temporary files when some program needs two inputs:
    join -e0 -o0,1.1,2.1 -a1 -a2 -j2 -t$'\t' \
      <(sort -k2,2 -t$'\t' freq/forms.${lang}) \
      <(sort -k2,2 -t$'\t' freq/lms.${lang})
    
    # gawk doesn't care if it's given a regular file or the output fd of some process:
    gawk -v dict=<(munge_dict) -f compound_translate.awk <in.txt
    
    # prepend a header:
    cat <(echo -e "${word}\t% ${lang}\tsum" | tr [:lower:] [:upper:]) \
        <(coverage ${lang})

    bash <(yes echo hello)

    echo hello
    echo hello
    echo hello
    ...

Pipes are probably the original instantiation of dataflow processing (dating back to the 1960s). I gave a tech talk on some of the frameworks: https://www.youtube.com/watch?v=3oaelUXh7sE

And my company creates a cool dataflow platform - https://composableanalytics.com

Vince Buffalo is author of the best book on bioinformatics: Bioinformatics Data Skills (O'Reilly). It's worth a read for learning unix/bash style data science of any flavour.

Or even if you think you know unix/bash and data there are new and unexpected snippets every few pages that surprise you.

In zsh, =(cmd) will create a temporary file, <(cmd) will create a named pipe, and $(cmd) creates a subshell. There are also fancy options that use MULTIOS. For example:

    paste <(cut -f1 file1) <(cut -f3 file2) | tee >(process1) >(process2) >/dev/null

can be re-written as:

    paste <(cut -f1 file1) <(cut -f3 file2) > >(process1) > >(process2)

http://zsh.sourceforge.net/Doc/Release/Expansion.html#Proces...

http://zsh.sourceforge.net/Doc/Release/Redirection.html#Redi...

If you like pipes, then you will love lazy evaluation. It is unfortunate, though, that Unix doesn't support that (operations can block when "writing" only, not when "nobody is reading").

     let a = "some_file_name".as_string();
     println!("Opening: {}", a);
     let path = std::path(a);
     let mut fd = std::io::open(path);

     let mut fd = std::io::open(std::path("some_file_name".as_string()));

    &> stdout

AFAIK process substitution is a bash-ism (not part of POSIX spec for /bin/sh). I recently had to go with the slightly less wieldy named pipes in a dash environment and put the pipe setup, command execution and teardown in a script.

I've used *nix for ~15 years and never used a named pipe or process substitution before. Great to know about!

  $ find . -name "*blarg*.cpp"
  # Some output that looks like what I'm looking for.
  
  # Run the same find again in a process, and grep for something.
  $ grep -i "blooey" $(find . -name "*blarg*.cpp")
  
  # Yep, those are the files I'm looking for, so dig in.
  # Note the additional -l in grep, and the nested processes.
  $ vim $(grep -il "blooey" $(find . -name "*blarg*.cpp"))

    $ find . -name "*blarg*.cpp"
    $ grep -i "blooey" $(!!)
    $ vim $(!! -l)

  $ grep -i "blooey" $(!!)

  $ foo myfile
  $ bla !$

    $ find . -name '*blarg*.cpp' | grep -li blooey | vi -

In fish shell the canonical example is this:

   diff (sort a.txt|psub) (sort b.txt|psub)

The psub command performs the process substitution.

Does anyone have a working link to Gary Bernhardt's The Unix Chainsaw, as mentioned in the article?

How does the > process substitution differ from simply piping the output with | ?

For example (from Wikipedia)

tee >(wc -l >&2) < bigfile | gzip > bigfile.gz

vs

tee < bigfile | wc -l | gzip > bigfile.gz

Anybody know of a way to increase the buffer size of pipes? I've experienced cases where piping a really fast program to a slow one caused them both to go slower as the OS pauses first program writing when pipe buffer is full. This seemed to ruin the caching for the first program and caused them both to be slower even though normally pipes are faster as you're not touching disk.

Is this guy a bioinformatician? I think he's a bioinformatician.

Can't be sure if he is a bioinformatician because he never really mentions that he is a bioinformatician.

moreutils [1] has some really cool programs for pipe handling.

pee: tee standard input to pipes sponge: soak up standard input and write to a file ts: timestamp standard input vipe: insert a text editor into a pipe

[1] https://joeyh.name/code/moreutils/

i heard somewhere that go follows unix pipe link interfaces.

Pipes are very cool and useful, but it's hard for me to understand this common worship of something like that. Yes, it's useful and elegant, but is it really the best thing since Jesus Christ?

Say that you have a program that splits its output into two files, each given by command line arguments. A normal run would be

    <input.txt munge-data-and-split -o1 out1.txt -o2 out2.txt

but since the output is huge and your disk is old and dying, you want to run xz on it before saving it to disk, so use >():

    <input.txt munge-data-and-split -o1 >(xz - > out1.txt) -o2 >(xz - > out2.txt)

If you want to do several things in there, I recommend defining a function for clarity:

    pp () { sort -k2,3 -t$'\t' | xz - ; }
    <input.txt munge-data-and-split -o1 >(pp > out1.txt) -o2 >(pp > out2.txt)

In the tee case the substitution is actually going somewhere different than standard out (that's what tee does).

so:

    cmd1 | tee out.txt | cmd2

So tee is splitting the stream into two outputs, one that carries on out stdout (into cmd2) and the other one that is redirected into out.txt.

With process substitution you can do extra stuff on the way out, I guess (I've never seen it used for output before).

It looks like in the example given they're writing wc stuff to stderr while zipping the content (over stdout).

Nice to see that example, I hadn't even thought about the usefulness of process substitution for outputting like this!

When you connect to processes in a pipe such as ...

    a | b

you connect stdout (fd #1) of a to stdin (fd #0) of b. Technically, the shell process will create a pipe, which is two filedescriptors connected back to back. It then will fork two times (create two copies of itself) where it replaces standard output (filedescriptor 1) of the first copy by one end of the pipe and replaces standard input (filedescriptor 0) of the second copy by the other end of the pipe. Then the first copy will replace itself (exec) by a, the second copy will replace itself (exec) by b. Everything that a writes to stdout will appear on stdin of b.

But nothing prevents the shell from replacing any other filedescriptor by pipes. And when you create a subprocess by writing "<(c)" in your commandline, it's just one additional fork for the shell, and one additional filedescriptor pair to be created. One side, as in the simple case, will replace stdin (fd #0) of "c"... and because the input side of this pipe doesn't have a predefined output of "a" (stdout is already taken by "|b") the shell will somehow have to tell "a" what filedescriptor the pipe uses. Under Linux one can refer to opened filedescriptors as "/dev/fd/<FDNUM>" (symlink to /proc/self/fd/<FDNUM> which itself is a symlik to /proc/<PID>/fd/<FDNUM>), so that's what's replaced as a "name" to refer to the substituted process on "a"'s command line:

Try this:

    $ echo $$
    12345  # <--- PID of your shell
    $ tee >( sort ) >( sort ) >( sort ) otherfile | sort

and in a second terminal

    $ pstree 12345 # <--- PID of your shell

    zsh,301
      ├─sort,3600 # <-- this one reads from the other end of the shell's fd #14
      ├─sort,3601 # <-- this one reads from the other end of the shell's fd #15
      ├─sort,3602 # <-- this one reads from the other end of the shell's fd #15
      ├─sort,3604 # <-- this one reads from stdout of tee
      └─tee,3603 /proc/self/fd/14 /proc/self/fd/15 /proc/self/fd/16 otherfile

If your system doesn't support the convenient /proc/self/fd/<NUM> shortcut, the shell might decide not to create a pipe, but rather create temporary fifos in /tmp and use those to connect the filedescriptors.

http://man7.org/linux/man-pages/man2/pipe.2.html

http://linux.die.net/man/2/dup

You can watch the syscalls as they are made:

    $ strace -fe fork,pipe,close,dup,dup2,execve bash -c 'tee <(sort) <(sort)'

It allows multiple, parallel pipes to each individual command, where the | allows just one.