How to Make Money Using Grep, Sed and Awk

How to Make Money Using Grep, Sed and Awk(openmonstervision.github.io)

72 points by bsdpunk 8 years ago | 40 comments

nrki 8 years ago |

"How to make money by posting a pretty boring article to HN and asking for book purchases at the end of it"

janvdberg 8 years ago | |

I didn't mind, I actually lolled when reading his other blogpost about how he botched the personal question part of an interview: https://openmonstervision.github.io/blog/posts/how-not-to-ge...

nrki 8 years ago | | |

That article /is/ funny.

It also doesn't beg for book purchases for books he didn't write... :P

mar77i 8 years ago |

Dude! You need help with using your tools' features to reduce your pipeline lengths.

- grep derp | sed whatever is the same as sed '/derp/ whatever' (you might be interested in sed -r for that matter)

- sed whatever | sed whatever\ else is the same as sed -e whatever;whatever\ else

This list is by no means complete but saves a lot of external processes already, and seriously: think about writing the whole thing in pure bash or awk. There might be just too little gain to justify including all these tools without composing the many features they provide.

CJefferson 8 years ago | |

I disagree, unless it's slow enough to effect performance, I prefer to combine simple pipes.

The only time I have to use more complicated constructions is to get around the stupid problem that passing filenames with pipes is almost impossible to do safely.

tslug 8 years ago | | |

I learned when writing bash scripts for cygwin under Windows that Windows doesn't support copy-on-write fork()'s. This means that any new process is extremely expensive to fork, because it copies the memory from the process it was forking from, even if it doesn't need it.

As a result, I did as much as I could using only bash internals, and very rarely did I use pipes, because they always fork()'d a bash subshell in addition to whatever the process was.

In about 7k lines of bash script written to completely automate the iterative development of two games with a shared game engine featuring dockerized server containers, I was able to avoid using sed in almost all of it. When I did use it (pretty sure only two places), it was basically for mass substitution of variables inside of text files, and the multi-line syntax worked very nicely for this, as I could form the sed line with a loop over all the variables I wanted to replace.

It turns out for most bash scripts, you can do the most common sed substitutions using bash's rich variable substitution expressions. For instance, you see in a lot of bash scripts calling commands like "dirname" and "basename" to get the directory and filename of paths. There's a much faster way to do this in bash:

  path=/tmp/my/path/to/stuff.txt
  dir="${path%/*}"
  file="${path##*/}"
  other="${path/stuff/other}"

That dir line means "delete the shortest string that matches /* from the end of the string". That file line means "delete the longest string matching */ from the start of the string." That other line means "replace the word stuff with the word other in this string".

mar77i 8 years ago | | |

there's always find -print0 and many core tools support -z (-0 for xargs). but you have a point of course, YMMV applies.

drinchev 8 years ago | |

There's always some reason for not to write "the perfect" command. Either being in a hurry or you just don't care about it.

Considering the commented out valid code, it looks like he was saving time by just adding another "|" and check the output, instead of improving the existing command and check the output. Something that I do very often as well.

jk563 8 years ago | | |

Glancing over this I thought it said: "Either being hungry or...". Nodding in agreement, I then realised my mistake.

yomansat 8 years ago |

I looked up the freelance job in question, it pays $36 for a 215 page publication from the 80s, including checking where automation has failed.

A question to those who do freelance work: sites like Upwork often have low budgets, where do you guys find better projects with better compensation?

iDemonix 8 years ago | |

I gave up with all of them. A friend of mine is using Upwork and Freelancer (awful) to find gigs, and he tried to get me involved so I could help him out with overspill. It seemed like you had to put ludicrously low rates to get terrible work, and despite paying peanuts, everyone expects amazing results.

I ditched it, joined some communities online, did a tiny bit of networking, and announced I was looking for work related to X - that was far more fruitful than sending 10 pitches a day for gigs on those sites and getting nothing in response because someone in India will do it for a dollar. Although I did love seeing the same projects re-appear a week later to fix the crap that their offshore team delivered.

xrisk 8 years ago | | |

I'm from India and I'm horrified that so many people are willing to do work for nothing or next to nothing...

yomansat 8 years ago | | |

I see... Thanks for sharing. I think what we seem to be after is contracting/consulting work rather than freelancing, the naming is what'll set the tone.

herbst 8 years ago | |

Haven't worked below 60$/ for fun things and 80$/h for boring things on Upwork without much reputation. Sure I could find a better paid job, but if you can sell yourself as skilled you don't have to be afraid of the averagely low numbers on there

bsdpunk 8 years ago | |

Actually, it was a project, I priced out at $100. Thinking I could rip through it easily. It took me longer than I thought I am sad to admit, even with my rough pipes, and multiple sed and greppings :)

edit: And the catalog was 1048 pages long not counting the index.

tragomaskhalos 8 years ago |

A man who eschews Perl, and all later and more usable tools, in favour of a bastard hodgepodge of shell and awk with a smattering of sed: It's like looking in the mirror.

angarg12 8 years ago |

At what point it pays off to switch to python/node instead of keep growing a bash script?

ggm 8 years ago |

The best way to make money with grep/sed/awk is to become sufficiently l33t in them, you can be gainfully employed by others to "do things" on "the goddam unixes"

hey.. we got more unixes here. can somebody grep them out the way?

zero_one_one 8 years ago | |

grep -v .*[nN].x will solve the infrastructure problem...

As you say, the way to make money (with any tool) is to get good enough at using them to be useful to someone that wants to pay you to do so

bsdpunk 8 years ago | |

Best Comment is Best

nunez 8 years ago |

If author used better variable names, then the code would not have been as hard to look at

bsdpunk 8 years ago | |

That's fair