Ask HN: How does the 10x coder overcome configuration headaches?

39 points by edgeztv 17 years ago | 39 comments

I find that I spend at least 50% of my hacking time not writing code but figuring out how to get the desired results from 3rd party software. Things like configuring web servers, using frameworks, using web APIs, etc. In this realm, things often don't work as you'd expect them to, and documentation is often lacking. There is a lot of trial and error, and I absolutely hate endless trial and error (especially with many unknown factors). It always feels like banging your head against a wall.

If the legend of the mythical coder who is 10 times faster than average were true, this person would have to overcome the same set of challenges. But how can a one do this 10 times faster than his colleagues?

How does one deal with the headaches resulting from using 3rd party software? For me this is by far the biggest obstacle to being happy as a coder.

nostrademons 17 years ago |

Most 10x coders deal with it by not using 3rd party libraries, except for ones that are very highly regarded and trusted. These tend to avoid most of the configuration headaches because everyone uses them, so the maintainers get lots of practice in making them easily installable.

It's funny how common wisdom in the software industry is "reuse, don't rebuild", but if you look at projects that are actually successful and developers that are highly regarded, they all have a massive case of NIH syndrome. Google and Microsoft both build all their own software. Mozilla completely reimplemented COM for Gecko. John Resig just got criticized recently because TestSwarm duplicates a lot of the functionality of Selenium Grid. Linus Torvalds wrote his own OS instead of hacking MINIX, and then wrote his own VCS rather than use Subversion or CVS. Chuck Moore wrote everything himself, down to the hardware, which he designed with CAD software he wrote himself in Forth, a language he wrote himself.

If you look at it by numbers, this makes perfect sense. The average library is, by definition, average, so a coder who is way above average can undoubtedly write something better given enough attention to the problem. Plus, many of these super-coders got their reputation by writing lots of code in the first place, which you don't do if you just reuse third-party libraries.

Unfortunately, everyone thinks they're above average, which is why we get such a profusion of libraries in the first place. ;-)

jerf 17 years ago | |

'It's funny how common wisdom in the software industry is "reuse, don't rebuild", but if you look at projects that are actually successful and developers that are highly regarded, they all have a massive case of NIH syndrome.'

Like all other absolute statements, it's wrong, just as the converse ("rebuild, don't reuse") is, and a balance is required.

For context, I do a lot of XMPP work, which is all XML. In the languages I work in, I've gone through all the XMPP libraries that I know of, and found them wanting. I now use some layers I wrote myself directly around Expat. Why? Because nobody but nobody seems to be able to write an XML library that correctly handles namespaces! Which is death when working with XMPP. Yeah, if you hack hard enough you can get around the problem, but especially given the thinness of the usual XMPP libraries and what little they do, it doesn't take much to flip over to negative value.

On the other hand, when I went to write a routine that takes a PNG file of arbitrary size and resizes it to be an avatar, I'd be a damn fool to yell "screw it!" and reimplement a PNG reader.

So, it depends. Factors to consider include the long-term price you'll pay for a sub-optimal library, how close to core it is to what you are doing, whether or not you've ever implemented anything close enough to the library in question that you are really sure you can build something better (sometimes libraries suck because they suck, but sometimes they suck because they solve a hard problem and the "suckage" is actually the hardness of the problem poking out) as it is so easy to end up creating something worse, and, since I deal almost exclusively in open source now, whether you can take most of an existing library and actually bring it up to spec more easily than starting from scratch.

One of my favorite maneuvers, not that I do it often, is to take some rather large library that does what I need but not terribly well, along with a lot of other stuff, and then tear it to shreds in the process of making it do what I want. A coworker of mine recently wanted to do zeroconf autodiscovery, so they started with dhcpcd and started tearing out everything except the use of the dhcp protocol, because it was easier than trying to start their own stuff from scratch. Worked great in less than an hour; I can't imagine what else they could have done in less time, even finding a conventional library, downloading it, and learning how to use it would have taken longer than that.

nostrademons 17 years ago | | |

I mentioned that in the first paragraph, but then later went on to hyperbolize in the quote you pulled out. "Most 10x coders deal with it by not using 3rd party libraries, except for ones that are very highly regarded and trusted."

I do use 3rd-party libraries. I'm a big fan of JQuery, for example - I can't use it in my day job because we literally count bytes when serving JavaScript, but I use it all the time for prototypes and personal projects. But I know that Resig is a top-notch coder, and more importantly, JQuery has had thousands of projects beat on it for a couple of years. If there're corner cases, they've been explored and uncovered, bug reports have been filed, and patches have been made.

Similarly, I use Apache/Lighttpd and MySQL and Django and PIL and BeautifulSoup (though I had some installation issues with PIL, so maybe that's not a great example of a well-tested, well-documented library). But these are pretty well-exercised projects with large userbases: other people have already done the dirty work of discovering all the bugs.

Makes me wonder why anyone would ever use new open source projects, which is essential to getting to the well-tested, well-exercised stage. I think it comes down to need: they'll reach for an imperfect library if the problem is so scary that they don't even want to approach it. So go work on hard problems, and you'll be popular. ;-)

xenophanes 17 years ago | |

I mostly agree but:

> The average library is, by definition, average

It's not that simple. The average programmer who releases a library is better than the average programmer.

Hexstream 17 years ago | | |

Yes, however a library you didn't write is likely less well-suited to the specific problem you're trying to solve than the one it was originally created for. Also, you won't understand it as well as its creator. And it's likely not written in the same style as you'd have written it.

The benefit of using a library written by an above-average 3rd party is often offset if not negated by the above concerns.

russell 17 years ago | |

/warning: rant ahead. I'm at work on a Saturday.

I really sympathize with the OP, because I inherited a web based enterprise system that was built using as many open source packages as possible. It appears to be something north of two dozen; some of which I have never heard of. Have you ever tried googling for "barbeque software"? (I found it, barcode scanner.)

I am a big believer in getting your core compancies down cold. I'm in the Java world, so that's a given. If you are serious into business, you must have a good understanding of SQL. I am now going through hell because the previous developer wrote his object hierarchy in java and XML and let Hibernate put his database together. Now I have serious performance problems, along with mysterious bugs.

I see too much of the attitude of adding layers of adding layers of abstraction to allow them to to use the Java mindset. God sakes, people. SQL is way more powerful than Java. Use it. The same thing with JavaScript libraries. Don't pull in a 100K library to get a couple of useful functions. Write them yourselves. Avoid getting caught in a maintenance trap.

andreyf 17 years ago | | |

with JavaScript libraries. Don't pull in a 100K library to get a couple of useful functions. Write them yourselves. Avoid getting caught in a maintenance trap.

While I agree with a lot of the rest, I'm not sure I understand this part... a 19k library (jQuery) that gives you a single DOM interface that fully takes advantage of JavaScript and works in every major browser is what will keep you out of the maintenance trap, not writing/maintaining that interop code yourself.

tonystubblebine 17 years ago | |

Are you sure these are examples of 10x programmers and not 100x programmers?

nostrademons 17 years ago | | |

Personally I think they're the same creature, it's just that 100x programmers happened to work on problems that are more visible to other programmers.

I've noticed quite a few programmers inside Google that are easily the caliber of Resig or Zawinski, but I'd never heard of them outside of Google because they work on closed-source software and don't go around taking credit for their work.

kragen 17 years ago | |

See "Code Reuse Considered Harmful": http://lists.canonical.org/pipermail/kragen-tol/1999-March/0...

Rereading it, I think it was mostly right, but partly wrong. The path to massive programming productivity isn't to rebuild everything from zero; it's to reuse only the right things.

ojbyrne 17 years ago | |

I don't claim to be a 10x programmer, rather I've thought a lot about productivity and programming. In my opinion it's better to develop deep knowledge of a few tools, than spending time learning and futzing with many tools. I've mostly developed this opinion from experience and a Philip Greenspun anecdote that I can't seem to find, that basically goes like "programmers like to grab a bunch of 'best of breed tools,' disappear into the server room to integrate them, only to emerge 2 years later to ask for a $2 million budget increase and another 2 years.

triplefox 17 years ago | |

I don't think it's NIH exactly, but a certain laziness and unwillingness to try anything but the simplest(even stupidest) solution. My observations and personal experience indicate that a 10x will reuse formats, languages, and protocols, and they'll use libraries for the purpose of taking advantage of those things.

What you won't see reused is more high-level functionality. That's where a lot of assumptions have to be made about the program's internal structure, and it's where getting too generic and open-ended makes things very complex very quickly.

jawngee 17 years ago | |

Right on the money.

I use 3rd party libraries until I hit a tipping point where either the effort to implement is looking to be greater than the effort to simply write the baseline functionality I need from it. The basic idea is to get it to fill in the functionality you need to prove your point and then rewrite it as your needs increase, or if you're like me, are OCD about a consistent code base.

Instead of writing from scratch, I tend to copy and paste whatever I can salvage and build from there, typically refactoring the original code into something new entirely.

raffi 17 years ago | |

I agree with this. I use third party libraries when its not inconvenient to do so. All the complexity and the kitchen sink that comes with a lot of stuff drives me crazy. For example, when I try to use any Java library from Apache I inevitably need to download three or four other libraries that depend on other libraries and so it goes.

Otherwise, I code in my own scripting language and I deploy to my own app server. I'm very productive this way and I'm able to leverage stuff that doesn't suck.

bprater 17 years ago | |

>> Chuck Moore wrote everything himself, down to the hardware, which he designed with CAD software he wrote himself in Forth, a language he wrote himself.

Whoa, I read a bit too fast there and thought for a second that it was true -- Chuck Norris could do anything!

moe 17 years ago |

I am a 10x coder, although I prefer when people just refer to me as someone who gets stuff done.

The answer to your question is simple: Yes, we are faster because we overcome these challenges, because we are as much sysadmins as we are developers.

The secret sauce is to realize that the world does not end at the borders of your VM.

If you refuse to acknowledge that your webapp needs an operating system, a webserver, a cache and a database to operate and if you push the lowly "grunt-work" of setting these up to someone else then you'll never be a 10x developer. Because in that case you're depriving yourself of acquiring the magic systems knowledge that leads to all the little "ah, this could be easier"-moments.

alnayyir 17 years ago | |

>I am a 10x coder

Hubris makes me sad. :(

>we are faster

That's unnecessarily inclusive and doesn't convey anything of substance.

>If you refuse to acknowledge that your webapp needs an operating system, a webserver, a cache and a database to operate

>depriving yourself of acquiring the magic systems knowledge

I smell a proud C/C++ programmer.

waves ASM coder over here, we (??) asm coders can write parsing libraries that spank yours seven ways to Sunday.

Does it matter? No. That's not how you "get things done (TM)".

moe 17 years ago | | |

Hubris makes me sad. :(

Sorry to hear that, I was merely trying to provide a frame of reference.

I smell a proud C/C++ programmer.

Way to read something into a post.

waves ASM coder over here, we (??) asm coders can write parsing libraries that spank yours seven ways to Sunday.

That's fine and may get you a nice job at a hardware company. I don't see how it's related, though.

Does it matter? No. That's not how you "get things done (TM)".

And where did I say anything to that tune?

The point I was trying to make is the opposite: As long as you consider yourself a programmer, or worse a programmer of a particular language, you're probably not a "10x coder" [who keeps coming up with these idiotic terms anyways]. It doesn't matter whether that language is C, asm, Java or Lisp.

bestes 17 years ago | | |

I've worked with a 10x coder and he does everything moe mentioned above, even including hardware. He is disdainful of those who program, but can't maintain their own systems. One guy is not much of a sample, but many times I've seen him find solutions in hardware, OS settings, network configuration, etc. that you would never have found if you focused only on your application.

Dilpil 17 years ago |

The 10x coder doesn't really have a magical trick for dealing with those kind of things. He simply maintains his focus, and makes steady progress until the problem has been solved.

However, the 1x coder ignores the problem, or declares it unsolvable. Inevitably, he waits for the 10x coder to fix it. The 10x rule holds.

10x coding is as much about willpower as it is about technical competence and experience.

sunkencity 17 years ago | |

>However, the 1x coder ignores the problem, or declares it unsolvable.

Haven't we seen that before? It's usually solvable within 5 minutes. And so true, if you want to become a good programmer, never give up trying to solve a problem, and don't wait for your seniors to solve it for you, because you will never develop the skills needed to solve the problem yourself that way.

mistermann 17 years ago | | |

Precisely. Some people never seem to get past a certain skill level because they don't have the ambition to get pissed off and figure out how something really works.

kragen 17 years ago | | |

Spending N hours on a problem that's very difficult to solve is often worse than deciding to find a way to avoid solving it. There are N₂ examples in my history where I solved a problem that was much harder than necessary — or spent a lot of time trying, then failed — when I should have just stepped back to ask if it was worth it.

DanielBMarkham 17 years ago |

Use 3rd party stuff cautiously. I think "never use 3rd party stuff" is a little extreme, but remember that when you pick up the tool, the tool picks you up too. (Sorry -- watched too much Yoda) That is, you may want the tool for 1 or 2 really cool things, but the tool may demand out of you a week of playing around, pain and suffering, and a dedication that your spouse doesn't get.

One rule I use is that when I pick something up, I make sure that I can google help whenever I need it. If Joe-Bob's awesome language or widget set has only 47 people using it, and you have to put up with abuse on an IRC channel or buy 15 books just to get respect from the community -- forget it.

Wise choice of tools also means your cognitive load decreases. You simply don't have to remember so much crap anymore. Problem remembering how to attach events to an element in Javascript? Ten seconds with Google and you've got your answer. There's a certain tipping point that products reach. Be careful using stuff before acceptance grows to that point.

That's _platform-ish_ stuff. Tools, widgets, languages and such. For whatever you're actually doing, consider rolling your own stuff. Most times you only need a small subset of functionality, and if you're a good coder it's as easy to write it as use somebody else's. Plus, you understand what's going on. Many times I've used stuff with open source code only to spend hours plugging through their code trying to figure out what the heck they were trying to do. That hurts. Don't do that if you can help it. Sure it's great to modify OSS, but it's probably just as easy to spend that time learning more about the technical problem domain.

For instance, I'm playing around with the idea of writing another web app. I took a look at YUI -- I really like it. But at the end of the day, so far at least, it does a whole lot of stuff that I don't need, and payload size is important to me. So I'm just going to roll my own stuff. You're always weighing future hassle against productivity. Sometimes, like with learning functional programming or picking up COM or something, the rewards may be much greater than the hassle. But for most stuff, like version control, project planning, configuration, deployment, setup, etc -- you don't have the luxury of taking 3 days to figure some of these tools out. So don't.

aristus 17 years ago |

I'm not a 10x coder, but over the years I've learned to deal with buggy 3PLs by avoiding them, or encapsulating them once and for all. The problem with buggy 3rd-party libs is that the bugs are random. They quite literally keep you dumb by forcing to you memorize random trivia instead of new concepts.

Rewriting the same functions yourself in a better and more efficient way teaches you a lot, especially about 1) how to spot good libraries, and 2) when not to write it yourself.

If you look at the code of really effective hackers, the one thing they have in common is how small the code is compared to what it does. Look at web.py, or bret taylor's datastore built on top of mysql: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

ghempton 17 years ago |

10x coders are also 10x system admins

sunkencity 17 years ago |

The trick is to be comfortable working with a system that you don't know much about. I personally love figuring out new systems, and starting to work in new unknown environments. Yes it takes a lot of time to figure things out, but when you have done it enough times (figured out 3rd party software), you get a feel for it, and can draw the right conclusions on how a program works fast.

Inexperienced troubleshooters often enter the work with pre-concieved notions of how the system works and hold on to them for too long. Always doubt your knowledge.

I think it usually takes 1-2 days with a system until you start to get our bearings. Before that, you are just collecting data, random tidbits of how the system works.

I second the recommendation to use recommended libraries, and I love github for the code network analysis. Back when I did a lot of java development there was often a java library available which solved the problem available on the internet even for a specialized problem, but more often than not it turned out to be a completely shit implementation that needed to be replaced. Of course there are good java libraries, the trick is to know someone you can trust who can recommend libraries.

OTOH for some rails tasks that can be resolved by just adding a plugin the quality of the code doesn't matter much, since it's just a few lines of code configuration. Still a good library is much better than a bad. I wish there was some kind of leaderboard, or official plugin list for rails, it can be very hard to judge plugins for a newbie.

mahmud 17 years ago |

You deal with it by using said 3rd party libraries 100x before you embark on your "current" project. Know your tools well, inside and out, and you're 10x more productive than someone just starting out :-)

Also, there is something that has to be said for compromise. Get libraries for the language you know best and you will more productive than if you used a high-performance library for a language you don't know. Start small, and worry about scaling or "going big" later.

conanite 17 years ago |

The head-banging gets less and less over time. Or else I've developed a thick skull, not sure which.

Experience makes a huge difference - simply because you've seen this problem before, or at least seen this kind of problem before, or maybe you saw the solution to this problem a year ago when you were googling the solution to some other problem and tucked it away in a cavity in your brain.

With experience, you happen to know that there are certain kinds of many-to-many relationships Hibernate can't map in a legacy schema where the foreign key is a subset of a composite primary key. So when you encounter that on your next project you drop right down to SQL, to the wonder and bewilderment of your new colleagues whom you've just converted to an ORM tool because it "abstracts away all that sql stuff".

With experience, you know the kinds of ways certain systems go wrong, the kinds of solutions and workarounds to apply, and most importantly the kinds of things to google for when hunting for the solution.

Even scanning search results pages - to a novice they're homogeneous, but after some time you start ignoring whole classes of results. For example, you're trying to find what some annoying ActiveRecord exception is about, and you realise half of your search results are pointers to subversion log messages mirrored in a thousand places all over the internet. And they don't have the information you're looking for.

As some other commenters observed, you have to push and solve those problems that seem impossible, because the next time around, you'll find the answer a little faster. Keep up the head-banging. It eventually turns into supernatural powers. Just be careful with the wall.

ShabbyDoo 17 years ago |

w.r.t. 3pl stuff, the lack of good documentation can be annoying, but I find that my ability to use them quickly speeds up drastically as I use them in more places on more projects. I don't think this pattern is any different when I use a new language -- "What's that weird compiler error mean?" So, for me, it's a learning curve issue.

mattmcknight 17 years ago |

My approach to this is that my creativity and expertise lie in creating unique software. I am not out there to build a web server, so I look for the stuff that is commonly used, well documented, and has easily googlable problems. I try to learn them really well. I stick with the stuff that works. With small open source things, I have been known to pull in the code and get going with it myself...

Here are mistakes I see people make that I try to avoid: 1) Making decisions on technology before they have actually tried it. 2) Upgrading to the latest version of products too often. For every problem it fixes... 3) Fail to give developers system administrator access rights. Nothing worse than not being able to install stuff, but it happens all the time.

wheels 17 years ago |

You use what you know rather than switching to the language of the month every, well, month.

I've been using the same toolchain for a decade now. I don't have to futz with the configuration: I learned it 10 years ago.

There's a balance there, between keeping on top of technology and focusing on solving problems.

It's been interesting for me doing a little Ruby of late because coming from mostly C++ (with bits of Java and Perl) I find myself fighting with the interpreter a lot. It's not that C++ doesn't have its quirks, it's just that I've known them all for so long that I work around them intuitively.

mpk 17 years ago |

Well, experience goes a long way.

After having built software in different languages for different targets on different systems a few times, you start recognizing patterns.

One of the strategies I use to become comfortable with new environments is to build a piece of software (usually a prototype) using only the core language, standard library and maybe one or two 'mature' pieces of third-party library code.

This takes a bit of time and I usually end up with a codebase that is a bit crufty (especially when I'm learning the language as I go). However, this usually gets me to the prototype stage with a lot of newly acquired knowledge about performance of the language, best-practices, etc. As well as a lot of testing code.

After that I'll start looking into third-party libraries and start factoring out my own code with third-party libs where appropriate.

For Java, for example, I've found that a lot of the language helper/util code I write can be replaced with code from org.apache.commons.

For interacting with crufty stuff like webservices in all their non-standard glory, I usually build a few classes in Ruby with breakpoints so I can interactively analyze and poke at them. Much, much faster than going through the 'change code'-save-recompile-run cycle with breakpoints in C# and Java.

When dealing with loads of XML config files, I'll quickly do a git init and also keep all the files open in vim buffers. (Vim and Emacs are excellent tools for working your way through XML junk). Then I'll put print (or equivalent) statements in the code so I can run the app from the command line and filter the output through grep or even some custom scripts that'll filter the output for me and provide some basic analysis or correlation points.

The list goes on and on.

You have to realize that apart from the code that operates solely within your codebase (and not even then), you are always dealing with third-party code. The OS, compiler, the VM, the windowing routines, the threading model, etc.

As you go on you have to learn techniques to handle this. And one of the first things you already seem to have learned is that you should never be quick to voluntarily add to the pile of third-party software you're already dealing with.

And that 10x programmer working opposite you who generates mod_rewrite configs like there's no tomorrow? Yeah, that person spent about 3 days getting to the bottom of that engine and testing the hell out of various configs a few years ago. There's not much of a short-cut for these kind of things.

known 17 years ago |

I had been through this phase. Since most of the 3rd party software is closed source I could not debug their code. I ended up writing work-around code to circumvent the bugs in closed source 3rd party software. And the culprit here is MS Excel and VBA.

antirez 17 years ago |

My guess is that there are multiple factors:

* Not invented here syndrome: 10x coder may code small things himself that many times is faster then learning other stuff.

* Selection of the tools to use: it will use Ruby instead of Java, for example, and for a reason. In general he'll try to figure the fastest path to solve the problem so will try a lot less.