Shell pipes and in-place insanity

Shell pipes and in-place insanity(fakedrake.github.io)

15 points by fakedrake 12 years ago | 14 comments

jamesdutc 12 years ago |

I think this behaviour is supposed to be a feature of shell. Pipelines are executed in parallel.

Also, see `sponge` from `moreutils` (http://kitenet.net/~joey/code/moreutils/):

    $ man sponge
    ...
    sponge  reads standard input and writes it out to the specified file. 
    Un‐like a shell redirect, sponge soaks up all its input before opening the output file. 
    This allows constructing pipelines that read from and write to the same file.

sikhnerd 12 years ago |

This is the expected behavior of pipes[0]. A quick look at Lists [1] clears up how you can control this a little better (+Job Control). jamesdutc also linked to sponge, which may suit your usecase as well.

  [0] https://www.gnu.org/software/bash/manual/bashref.html#Pipelines
  [1] https://www.gnu.org/software/bash/manual/bashref.html#Lists

mtdewcmu 12 years ago |

That was a contrived example.

The shell sets up the pipe and file redirections for each process in the pipeline, then lets them go. It has no control over when the child processes read or write. It could be a race condition, but it's written in such a way that it appears that the file will get truncated before it can be read virtually 100% of the time.

fakedrake 12 years ago | |

You are right I should have clarify that. thank you

mtdewcmu 12 years ago | | |

I'm not sure if it's actually a race; it probably depends on how bash is implemented. But this prints what is expected:

echo "1\n2\n3\n4\n" > numbers && cat numbers | { sleep 1 && cat > numbers && cat numbers ; }

That's not exactly the same (because the redirection for cat #2 now happens in a subshell), but it shows that this is basically racy. :)

dredmorbius 12 years ago |

NB: If you're going to use CSS styling for code examples, choose foreground and background colors, as well as text sizes, which are actually legible.

Monospace fonts, black on white, at default sizes, are your best bets.

simoncion 12 years ago | |

Counteranecdatum for the author: I find your choice of foreground and background colors, syntax highlighting colors, font style and size, and line length inside your code blocks to be quite legible and pleasant to read.

Please make these sorts of choices again in the future!

LukeShu 12 years ago | |

Um... I see almost-black Courier New on off-white, at 13px and a slightly increased line-height. That seems fine to me.

dredmorbius 12 years ago | | |

The body text is approximately legible size, but low contrast.

The code examples are all but unreadable.

http://i.imgur.com/ogS9yat.png

Here's a quick improvement for legibility. I'm not claiming a thing of beauty.

http://i.imgur.com/mW880fB.png

My general style guidelines are included here:

https://news.ycombinator.com/item?id=6996064#up_6998399

$ man sponge ... sponge reads standard input and writes it out to the specified file. Un‐like a shell redirect, sponge soaks up all its input before opening the output file. This allows constructing pipelines that read from and write to the same file.