Pipe Viewer(ivarch.com) |
Pipe Viewer(ivarch.com) |
PV (Pipe Viewer) – add a progress bar to most command-line programs - https://news.ycombinator.com/item?id=23826845 - July 2020 (2 comments)
A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=8761094 - Dec 2014 (1 comment)
Pipe Viewer - https://news.ycombinator.com/item?id=5942115 - June 2013 (1 comment)
Pipe Viewer - https://news.ycombinator.com/item?id=4020026 - May 2012 (26 comments)
A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=462244 - Feb 2009 (63 comments)
Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?
It also helps me optimize my time. If something is going to finish in a few minutes, I probably won't context switch to another major task. However, if something is going to take a few hours then I'll probably switch to work on something different knowing approximately when I can go back and check on results.
pv -L 200K < bigfile.iso | ssh somehost 'cat > bigfile.iso'
Complete with a progress bar, speed, and ETA.
from man page:
-l limit
Limits the used bandwidth, specified in Kbit/s.
tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'
1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput.
2. things where the program has some useful cleverness I rely on that requires being fed by a named file argument, but behaves a lot less intelligently when being fed from stdin — e.g. feeding SQL files into psql(1).
3. things where the program, even while writing to stdout, also produces useful "sampled progress" informational messages on stderr, which I'd like to see; where pv(1) and this output logging would fight each-other if both were running.
4. things where there's no clean place to insert pv(1) anyway — mostly, this comes up for any command that manages jobs itself in order to do things in parallel, e.g. any object-storage-client mass-copy, or any parallel-rsync script. (You'd think these programs would also report global progress, but they usually don't!)
I could see pv(1) being fixed to address case 3 (by e.g. drawing progress while streaming stderr-logged output below it, using a TUI); but the other cases seem to be fundamental limitations.
Personally, when I want to observe progress on some sort of operation that's creating files (rsync, tar/untar, etc), here's what I do instead: I run the command-line, and then, in a separate terminal connected to the machine the files are being written/unpacked onto, I run this:
# for files
watch -n 2 -- ls -lh $filepath
# for directories
watch -n 4 -- du -h -d 0 $dirpath
Doing things this way doesn't give you a percentage progress, but I find that with most operations I already know what the target's goal size is going to be, so all I really need to know is the size-so-far. (And pv(1) can't tell you the target size in many cases anyway.)
tail -f /some/log | grep something | pv -lr > /dev/null
or
tcpdump expression | pv -lr > /dev/nullAnother utility I found out about is “progress” available at least on debian systems. It can monitor stuff like cp and mv without actually being used in the command
Thanks!
1) this gets it out of the pipeline. 2) the program gets to have the named arguments. 3) pv's out put is on a separate terminal. 4) your job never needs to know.
Downside: it only sees the currently open files, so it doesn't work well for batch jobs. Still, it's handy to see which file it's on, and how fast the progress is.
Also, for rsync: "--info=progress2 --no-i-r" will show you the progress for a whole job.
You'd be surprised how cheap these du(1) can be when you're running the same du(1) command over and over. Think of it like running the same SQL query over and over — the first time you do it, the DBMS takes its time doing IO to pull the relevant disk pages into the disk cache; but the Nth≥2 time, the query is entirely over "hot" data. Hot filesystem metadata pages, in this case. (Plus, for the file(s) that were just written by your command, the query is hot because those pages are still in memory from being recently dirty.)
I regularly unpack tarballs containing 10 million+ files; and periodic du(1) over these takes only a few milliseconds of wall-clock time to complete.
(The other bottleneck with du(1), for deep file hierarchies, is printing all the subdirectory sizes. Which is why the `-d 0` — to only print the total.)
You might be worried about something else thrashing the disk cache, but in my experience I've never needed to run an ETL-like job on a system that's also running some other completely orthogonal IO-heavy prod workload. Usually such jobs are for restoring data onto new systems, migrating data between systems, etc.; where if there is any prod workload running on the box, it's one that's touching all the same data you're touching, and so keeping disk-cache coherency.
My main use-case is netcat (nc).
As an aside, I prefer the BSD version, which I find is superior (IPv6 support, SOCKS, etc). "GNU Netcat" isn't even part of the GNU project, AFAIK. I also discovered Ncat while writing this, from the Nmap project; I'll give it a try.
But pv(1) is just blindly attempting to emit "\r[progress bar ASCII-art]\n" (plus a few regular lines) to stderr every second; and interleaving that into your PTY buffer along with actual lines of stderr output from your producer command, will just result in mush — a barrage of new progress bars on new lines, overwriting any lines emitted directly before them.
Having two things both writing to stderr, where one's trying to do something TUI-ish, and the other is attempting to write regular text lines, is the problem statement of 3, not the solution to it.
A solution, AFAICT, would look more like: enabling pv(1) to (somehow) capture the stderr of the entire command-line, and manage it, along with drawing the progress bar. Probably by splitting pv(1) into two programs — one that goes inside the command-line, watches progress, and emits progress logs as specially-tagged little messages (think: the UUID-like heredoc tags used in MIME-email binary-embeds) without any ANSI escape codes; and another, which wraps your whole command line, parsing out the messages emitted by the inner pv(1) to render a progress bar on the top/bottom of the PTY buffer, while streaming the regular lines across the rest of the PTY buffer. (Probably all on the PTY secondary buffer, like less(1) or a text editor.)
Another, probably simpler, solution would be to have a flag that tells pv(1) to log progress "events" (as JSON or whatever) to a named-FIFO filepath it would create (and then delete when the pipeline is over) — or to a loopback-interface TCP port it would listen on — and otherwise be silent; and then to have another command you can run asynchronously to your command-line, to open that named FIFO/connect to that port, and consume the events from it, rendering them as a progress bar; which would also quit when the FIFO gets deleted / when the socket is closed by the remote. Then you could run that command, instead of watch(2), in another tmux(2) pane, or wherever you like.
rsync --bwlimit=2OOK bigfile.iso name@server.org:path/in/destination
sftp -l 200 bigfile.iso name@server.org:path/in/destination
https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-...
1.5.2 - 10 February 2014 allow --watchfd to look at block devices
You can see the position in /proc/<pid>/fdinfo/<num>
For a contrieved (but the first I can think of)
dd if=/dev/abc status=progress 3>/dev/null | pv img