John Graham-Cumming (jgrahamc here) co-authored a piece on making scientific code open. It was received well-enough that Nature published it [0]. This approach has inspired others to do better work by describing a concrete problem, then outlining steps to fix it on an individual and institutional level.

When someone finds fault with the way a field conducts itself, I would implore them to constructively influence that field. You might be surprised how many are actually sympathetic to your concerns.

I'm not dismissing this author's concerns: to do that would really require knowing the molecular biology field (which is more than sequencing, it turns out). I do neuroscience right now, and programming can be a problem for some. But a constructive suggestion to change can have much more impact than a long rant.

[0] http://www.runmycode.org/data/MetaSite/upload/nature10836.pd...

> the software is written to be inefficient, to use memory poorly, and the cry goes up for bigger, faster machines! When the machines are procured, even larger hunks of data are indiscriminately shoved through black box implementations of algorithms in hopes that meaning will emerge on the far side. It never does, but maybe with a bigger machine…

I spent five years working in bioinformatics, and this is exactly the attitude of both the researchers and the other developers on the projects I worked on. It was very frustrating.

I have some experience working at a genomics research company and I'll broadly +1 Fred's experience about the industry, although in less negative terms. I got out before I got jaded, so my perspective is a bit more "oh, that's a shame" than his. I really like genetics, bioinformatics, hardware, deep-science, and all that but the timing and fit wasn't right.

The tools are written by (in my experience) very smart bioinformaticians who aren't taught much computer science in school (you get a smattering, but mostly it's biology, math, chemistry, etc.). Ex:

http://catalog.njit.edu/undergraduate/programs/bioinformatic...

http://www.bme.ucsc.edu/bioinformatics/curriculum#LowerDivis...

http://advanced.jhu.edu/academic/biotechnology/ms-in-bioinfo...

The tools themselves are written by smart non-programmers (a very dangerous combination) and so you get all sorts of unusual conventions that make sense only to the author or organization that wrote it, anti-patterns that would make a career programmer cringe, and a design that looks good to no one and is barely useable.

Then, as he said, they get grants to spend millions of dollars on giant clusters of computers to manage the data that is stored and queried in a really inefficient way.

There's really no incentive to make better software because that's not how the industry gets paid. You get a grant to sequence genome "X". After it's done? You publish your results and move on. Sure, you carve out a bit for overhead but most of it goes to new hardware (disk arrays, grid computing, oh my).

I often remarked that if I had enough money, there would be a killing to be made writing genome software with a proper visual and user experience design, combined with a deep computer science background. My perfect team would be a CS person, a geneticist, a UX designer, and a visual designer. Could crank out a really brilliant full-stack product that would blow away anything else out there (from sequencing to assembly to annotation and then cataloging/subsequent search and comparison).

Except, I realized that most folks using this software are in non-profits, research labs, and universities, so - no, there in fact is not a killing to be made. No one would buy it.

I sympathize with the author, but this piece fails because many of the specific criticisms are off-base, and he's not trying to be at all constructive.

For example, it isn't true at all that microarray data is worthless. The early data was bad, and it was very over-hyped, but with a decade of optimization of the measurement technologies, better experimental designs, and better statistical methods, genome-wide expression analysis became a routine and ubiquitous tool.

The claim that sequencing isn't important is ridiculous. It's the scaffold to which all of biological research can be attached.

However:

There is a great deal of obfuscation, and reinventing well-known algorithms under different names (perhaps often inadvertently). There's also a lot of low-quality drivel on tool implementations or complete nonsense. This is driven largely by the need in academia to publish.

The other side of this problem is that in general, CS and computer scientists don't get much respect in biology. People care about Nature/Science/Cell papers, not about CS conference abstracts. Despite bioinformatics/computational biology not really being a new field anymore, the cultures are still very different.

I agree with him, and have been complaining about the same shit for ages (I work in bioinformatics too). Sadly, biologists don't care. We're treated as the number crunchers. The real problem isn't that we waste computational resources, it's that many biologists download programs, run their data through it, and if it spits out an answer rather than an error, they trust it. Since that program probably has zero unit test coverage, and the results may be fed into pharmaceutical decisions, disease diagnostics, etc, you're basically fucked if something went wrong. Lots of us have said this[0].

Minor quibble: genome assembly is definitely still an open problem that's computationally difficult. So is robust high dimension inference, but that falls more under statistics.

I've wanted to leave at least a dozen times too, for the better pay, for working with programmers that can teach me something, and to not have my work be interrupted by academic politics. But the people pissed at the status quo are the ones that are smart enough to see it's broken and try to fix it, and if we all leave, science is really fucked.

[0] http://www.johndcook.com/blog/2010/10/19/buggy-simulation-co...

If you really want to get a feel for how deluted the Bioinformatics community is, look for a job in the field as an outsider. It's not uncommon to see requirements like:

"Must be an expert in 18 technologies" "Must have a PHD in Computer Science or Molecular Biology" "Must have 12 years experience and post doctoral training" "Pay: $30,000"

It's delusional because they apply the requirements it took for themselves to get a job in Molecular Biology (long PHD, post doc, very low pay for first jobs) and just apply it carte blanche to all fields that may be able to aid in their pursuits. Especially when it comes to software engineering where it can often be extremely difficult to explain why you did not pursue a PHD.

This is pretty hilarious, from my brief experience with bioinformatics I can very well imagine someone writing the opposite rant, about CS people getting into bioinformatics not knowing sh*t about biology. I mean, browse through bioinformatics textbooks, those are either written by computer scientists and those are little more than string algorithm textbooks or by biologists and then the layer of jargon for someone coming from CS is just impenetrable. Same with bioinformatics teachers, I come from a CS background, but spent one solid month seriously trying to understand the basics of molecular biology and my bioinformatics seminar instructor sometimes seemed to know less about it than me. Terrifying, no wonder nonsense results are produced.

I always feel awkward reading these rants, mainly because I've burned my bridges before and it really wasn't worth it. Even if it is true, it's better to leave it and move on.

If you really feel strongly about something, write it dispassionately (normally some time after the event) and treat it like a dissertation, backed with case studies and citations.

Basic science moves forward slowly limited by the pace of fortuitous discoveries. I have found that many people from the field of computer programming have unrealistic expectations of what can be done in biology and other sciences.

Sounds like a fed up academic with a stick up his backside.

Sh*tty data? Comes from the community. If the data and algorithms are so poor, and the author so superior, he should have been able to improve the circumstances.

This whole screed reads like an entitled individual who entered a profession, didn't get the glory, oh and yeah, academia doesn't pay well.

In the realm of bioinformatics, lets ignore the work done on the human genome and the like.

My experience working as a scientific programmer is this: my colleagues aren't forthcoming. I could list case after case of failure to document or communicate crucial details that cost me days, weeks and even months of effort. But I won't, until I have another job lined up. If I were in the author's position (I'm in another field), I would insist that my colleagues--all of them, in whatever field I ended up working, were forthcoming about their work. This is non-negotiable. Being over-busy is no excuse. (It may be an excuse for not being forthcoming, but right or wrong, I couldn't care less--I would not work with such people if I could avoid it, for whatever reason.)

Academia rewards journal publication and does not adequately reward programming and data collection and analysis, although these are indispensable activities that can be as difficult and profound as crafting a research paper. At least the National Science Foundation has done researchers a small favor by changing the NSF biosketch format in mid-January to better accommodate the contributions of programmers and "data scientists": the old category Publications has been replaced with Products.

Naming is important to administrators and bureaucrats. It can be easy to underestimate the extent to which names matter to them. Now there is a category under which the contribution of a programmer can be recognized for the purpose of academic advancement. Previously one had to force-fit programming under Synergistic Activities or otherwise stretch or violate the NSF biosketch format. This is a small step, but it does show some understanding that the increasingly necessary contributions of scientific programmers ought to be recognized. The alternative is attrition. Like the author of the article, programmers will go where their accomplishments are recognized.

Still, reforming old attitudes is like retraining Pavlov's dogs. Scientific programmers are lumped in with "IT guys." IT as in ITIL: the platitudinous, highly non-mathematical service as a service as a service Information Technocracy Indoctrination Library. There is little comprehension that computer science has specialized. For many academics, scientific programmers are interchangeable IT guys who do help desk work, system and network administration, build websites, run GIS analyses, write scientific software and get Gmail and Google Calendar synchronization running on Blackberries. It is as if scientists themselves could be satisfied if their colleagues were hired as "scientists" or "natural philosophers" with no further qualification, as opposed to "vulcanologist" or "meteorologist" (to a first order of approximation).

"I’m leaving bioinformatics to go work at a software company [...]"

"[bioinformatics] software is written to be inefficient, to use memory poorly, and the cry goes up for bigger, faster machines! [...]"

Well, the author is heading for a very bitter surprise...

You know, I'd be more inclined to listen to him if he didn't also completely decry almost all of modern biology, which (in my view) has been to the late 20th and early 21st centuries what physics was to the late 19th and early to mid 20th centuries.

I spent a year in a bioinformatics PhD program and got the feeling I was studying to be science's version of the business analyst. Not knowing enough about the biology or computation, but expected to speak the language of both. And what would my research consist of in such an applied science? Luckily I had another opportunity and became a software developer (which I'm happy with). The worst thing about the experience was listening to so many research presentations where I could tell the presenter didn't understand the science and could barely explain it.

Some thoughts on this article:

- This guy clearly has a limited understanding of the field. This quote is laughable: "There are only two computationally difficult problems in bioinformatics, sequence alignment and phylogenetic tree construction."

- As a bioinformatician, I feel sorry for this guy. Just like any other field, there are shitty places to work. If I was stuck in a lab where a demanding PI with no computer skills kept throwing the results of poorly designed experiments at me and asking for miracles, I'd be a little bitter too.

- Just like any other field, there are also lots of places that are great places to work and are churning out some pretty goddamn amazing code and science. I'm working in cancer genomics, and we've already done work where the results of our bioinformatic analyses have saved people's lives. Here's one high-profile example that got a lot of good press. (http://www.nytimes.com/2012/07/08/health/in-gene-sequencing-...)

- I'm in the field of bioinformatics to improve human health and understand deep biological questions. I care about reproducibility and accuracy in my code, but 90% of the time, I could give a rat's ass about performance. I'm trying to find the answer to a question, and if I can get that answer in a reasonable amount of time, then the code is good enough. This is especially true when you consider that 3/4 of the things I do are one-off analyses with code that will never be used again. (largely because 3/4 of experiments fail - science is messy and hard like that). If given a choice between dicking around for two weeks to make my code perfect, or cranking out something that works in 2 hours, I'll pretty much always choose the latter. ("Premature optimization is the root of all evil (or at least most of it) in programming." --Donald Knuth)

- That said, when we do come up with some useful and widely applicable code, we do our best to optimize it, put it into pipelines with robust testing, and open-source it, so that the community can use it. If his lab never did that, they're rapidly falling behind the rest of the field.

- As for his assertion that bad code and obscure file formats are job security through obscurity, I'm going to call bullshit. For many years, the field lacked people with real CS training, so you got a lot of biologists reading a perl book in their spare time and hacking together some ugly, but functional solutions. Sure, in some ways that was less than optimal, but hell, it got us the human genome. The field is beginning to mature, and you're starting to see better code and standard formats as more computationally-savvy people move in. No one will argue that things couldn't be improved, but attributing it to unethical behavior or malice is just ridiculous.

tl;dr: Bitter guy with some kind of bone to pick doesn't really understand or accurately depict the state of the field.

Really makes me want to learn more about molecular biology.

Any solid factual resources besides the references mentioned in this justified rant?

Also, yes molecular biologists with few exceptions know little more than fuck all about ecology. Hence the mostly gung-ho attitudes to GM of crop foods for example. Honestly. I've done real molecular biology work (simple commercial protein chemistry and molecular phylogenetics of mitochondrial DNA) and tried to start a PhD in ecology (failed due to funding issues and realising it was a dead end job wise).

There are a lot of problems in bioinformatics. Mainly, lack of reproducibility (ie "custom perl scripts"), poorly organized and characterized data and plenty of wheel reinvention (I heard Jim Kent, who first assembled the human genome, created his own version of wc [word of mouth, citation needed]).

The fact of the matter is that through high-throughput sequencing, microarrays, what have you, generation of biologically-meaningful results is possible.

There are a lot of problems in bioinformatics that need to be solved. Github has helped. More of bioinformaticians are learning about good software development practices, and journal reviewers are becoming more enlightened of the merits of sharing source code.

Also see the discussion at the bioinformatics subreddit: http://www.reddit.com/r/bioinformatics/comments/179e9k/a_far...

Interesting to read since I made the same career move last year. I agree with about half of it but don't see a lot of value or useful advice here.

I find it curious that he stops to salute ecologists, since I was in an ecology lab. I liked my labmates and our perspective, but we didn't have any magical ability to avoid the problems he aludes to here.

I think a lot of his frustration comes down to not being more involved in the planning process. That's not a new problem. R.A. Fisher put it this way in 1938: “To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.”

Perhaps the idea that we can have bioinformatics specialists who wait for data is just wrong. Should we blame PIs who don't want to give up control to their specialists, or the specialists who don't push harder, earlier? Ultimately the problem will only be solved as more people with these skills move up the ranks. But the whole idea that we need more specialists working on smaller chunks of the problem may be broken from the start (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1183512/).

OK, I agree that there are some shitty work on this field, but he can't think they we all in the same boat. For example "Irene Pepperberg’s work with Alex the parrot dwarfs the scientific contributions of all other sequencing to date put together." this is not true. Bioinformatics is not just blinding sequencing new DNA, but analyzing data and almost every new breakthrough in medicine is based in a direct (or indirect) bioinformatics analysis. I used to work in an agrobiotech company and the sequencer was the first source of data for any breeding program. Bioinformatics was used to design primers for PCR to find molecular markers. There is bad software out there? Yes, but I see this as an opportunity than a problem. And the cause is not the need to hide something, but the lack of ability of biologists with no CS background in the field.

Maybe overblown, but it echoes complaints I've heard from other bioinformatics people.

Surely this means there's a goldmine waiting there for someone to produce a non-broken toolchain for bioinformatics?

Or is it even possible to produce standard tools? Maybe all the labs are too bespoke?

i totally disagree on Fred's negative view of Bioinformatics. as "software is eating the world", it's actually bioinformatics is eating biology. today's main-stream biology is dealing with exploding amount of data from modern instruments, images or clinical data collected every day and mostly machine readable. to stay up-to-date a modern biologist / bioinformatist need to think biological problems in a "big-data" (i know, cliche) way, then try to gain some insight from the data with (computational) tools. today it's the algorithms, mathematical models and software packages on top of databases to pinpoint cancer SNPs and drive drug discovery. and today it's these same algorithms and math models driving how web bench works are designed. if you think biological data are "shitty", i guess you never see other kind of unstructured data out there. so many scholars in other fields envy biologist and medical scientists for something called "PubMed". on the other hand, for those purely wet bench "biologists" who think computers are magic boxes to give answers, insights, models with one push of the button, i do feel sorry for them. they are so last-gen as they just don't have the essential techniques nowadays (just like a molecular biologist not knowing pcr).

This is a little discouraging - BioInformatics was my top choice for a Master's program I'm planning to start this year. The program at Melbourne Uni looks really good (accepts from three streams, Math/Stats, Biology or Computing and tailors the course based on your background). Maybe I should go for a more generic Machine Learning one and try to apply that to healthcare in some other field if things are really this bad.

Some things are going to suck in academia, as this guy points out. But, its a necessary step and todays progress is almost always going to be tomorrows shit. So quit bitching.

Biologists are almost never good coders, if they can code at all. But thats not what they do, they signed up for pipettes, not python.

Its the programmers who wrote said shitty code that are to be blamed, but you can't hate under-paid and over-worked phd students who write this code even though it usually has nothing to do with their thesis (the math/algorithm is the main part, the deployable implementation is usually not the most important).

If you want good code and organized/accountable databases, go to industry. Theres nothing new about this transition. The IMPORTANT part, is that industry gives back to academia. So when you get an office with windows and a working coffee machine, remember to help make some phd student's life a little easier by making part of your code open source.

Hi, I'm a bioinformatics researcher. Apparently I work for this guy's ex(?)-employer although I have never heard of him before.

My single most limited resource is programmer time. My time and the time of other people who work with me. I have access to loads of computers that sit idle all the time, even if it is on nights and weekends. There is zero opportunity cost to me in using these computers more fully. I have enough human work to do that I can wait for the results without having any wait states.

There can be a big opportunity cost in trying to rework a workflow so that it is more efficient and then test it thoroughly ensure correctness. Doing this may seem more appealing to someone who is interested primarily in computational efficiency. But I am more interested in research efficiency, and so are my employers and funders.

Part of the problem is grant money. Sometimes it's faster to buy more machines and get more results as opposed to rewriting entire algorithms. But the author does correctly identify, I think, some tendencies of some academic bioinformaticists.

I dont think the problem is people (researchers, developers) but of the infrastructure for research. Researchers are constantly thinking about getting new grants and renewing old ones the way politicians are constantly worried about their corporate sponsors and getting reelected. The result is that we only get a little science and we only get a little good governance. The internal organizations that form as a result of this environment are artificial. In the lean times researchers make short term decisions aimed at generating marketing and taking mindshare. In the fat times researchers ensure that all computational and lab space are used and come up with new reasons for growth. A friend working in a large research institution once suggested a refactoring that would greatly improve efficiency of an application. Instead, she was handed back down a recommendation that would make the application less efficient with the same functionality. The reason was that the computational usage was about to be audited and the rule was that there would be no improvements in efficiency until after it was complete. The system is an old house with hundred year old plumbing. The people you pour through the system are going to flow through the pipes abiding by the laws of physics. Blaming them for a leak is about as useful as blaming water: while you may win the moral argument, you will not solve the problem. The best you can do is replace them with new people who will react largely in the same manner.

I live in this field, as a computer scientist learning the biology, and trying to make a living with a bootstrapped company.

I wrote a post about why GATK - one of the most popular bioinformatic tools in Next Generation Sequencing should not be put into a clinical pipeline:

http://blog.goldenhelix.com/?p=1534

In terms of your ideal software strategy, I can speak to that as well, as I am actually attempting to do almost exactly what you suggesting. My team is all masters in CS & Stats, with focus on kick-ass CG visualization and UX.

We released a free genome browser (visualization of NGS data and public annotations) that reflects this:

http://www.goldenhelix.com/GenomeBrowse/

But you're right, selling software in this field is a very weird thing. It's almost B2B, but academics are not businesses and their alternative is always to throw more Post-Doc man-power at the problem or slog it out with open source tools (which many do).

That said, we've been building our business (in Montana) over the last 10 years through the GWAS era selling statistical software and are looking optimistically into the era of sequencing having a huge impact on health care.

This is an old story. Every domain I've worked in featured a chasm between the domain experts and the software folks. Experts write terrible software that somehow mostly works. Software folks misunderstand the problem and create overwrought monstrosities.

In my experience, this applies to accounting software, sensor data, computer-aided design, print manufacturing, healthcare, etc.

I imagine there's phases of maturity, something akin to CMM/SEI. Eventually there's enough people with a foot on both sides to bridge the gap.

It just takes time.

Hrrm, I was in a genetics research lab myself and got annoyed at the inefficiencies myself. In particular, I got frustrating to write & use in-house scripts to run pipelines for compute clusters and then not know what the state of the execution is, where the files are, etc. It's sort of a meta-problem, but I decided to do a startup based on writing good software w/ a good UI to make the problem better (problem = running, monitoring, managing pipelines on clusters that have job schedulers like Grid Engine): http://www.palmyrasoftware.com/workflowcommander/

Maybe it's still in the early going, but I do see how it's going to be real difficult making a living doing this. OTOH, companies like CLC Bio seem like they're doing well for themselves...

Why wouldn't anyone buy your product? If it is easy to use, and SPEEDS UP RESEARCH TIME, your researcher/PI who is spending thousands on computing clusters will buy your software for their graduate students. Hell, my PI keeps asking me if I need a faster computer so I can run Matlab better/quicker. Really, if I had a software that helped me perform research faster/better/quicker and compare my results to ground truth or gold-standards, that is a much more useful tool than a bunch of hardware for my research. You push out papers fast.

So I disagree with you on your very last sentence (agree with the rest)

There are companies out there that offer commercial sequencing software. DNANexus is one.

As for whether there's "a killing to be made", it's kind of unclear so far.

As someone on a bioinformatics team in a public research institution, salaries range from $75k to $100K for developers on our team. This includes a number of people, including myself, who do primarily normal IT things (data management, small webapps for various clinical and research needs) and also for devs doing pipeline/workflow mgmt software, novel dev (e.g. new research code for sequencing), and variant calling work.

In my geographic area, this salary range is somewhat below corporate IT work (say 10% to 15%), but generally higher than the typical university software dev job listing. The university is really bad to list jobs and job requirements with laughable salaries. I have seen (in other departments) web app dev jobs that require significant front-end and back-end skillsets/experience and then pop a salary that is full 50% less than entry level jobs for CS undergrads.

One problem is that hiring departments in that position will find someone to hire at that rate, so they think it was correct. From personal experience, I can verify that "good on-paper" candidates with exceptional credentials (say MS in CS, bunch of experience) from other depts who look to join our team are unable to to write any code at the whiteboard at all (say a for loop in java to println something). But to be fair, a recent job interview cycle one of my teammates performed produced exactly two candidates out of 16 who could do this and only one of those could write a SQL statement that required a simple inner-join. Most of those folks were external, so it's not just a problem inside the institution.

I have a number of cynical and embarrassing opinions about this situation.

I assume these are separate requirements. I have not seen any doctoral-level positions advertised for a salary of $30,000. The minimum NIH salary for postdoctoral trainees is more than that.

It's only delusional if they can't find people to fill the jobs. The idea that, as an outsider, you know what requirements they should use in their hiring process better than they do is perhaps more delusional.

I have seen this in many bio fields. As the biology research becomes more of a computational problem, requiring unique solutions for bleeding edge research, I imagine the field is going to have huge pains before actually paying for the work vs. using seniority and degree level as the sole determinate of pay scale.

Right now experimentalists generate data and then try to find computer people to analyse their data. However, in the not too distant future computer models will drive experimental research as hypothesis generation tools. Then the computer people will be seeking biology people ( or robots) to run experiments to validate their hypothesis and there will be more respect for the field.

" I could give a rat's ass about performance. I'm trying to find the answer to a question, and if I can get that answer in a reasonable amount of time, then the code is good enough"

This is the only bad point that a lot of people are aligned with.

The more time a program needs to finish, the more time you will need to run it again with some other dataset, and in turn - more time to find the right answer.

I really feel that people with scientific and mathematics background should learn proper programming (not take a course in some language - but have actual experience). Design patterns, data structures, best practices, memory consumption, are all things that should be known before a person starts submitting code for this kind of projects.

What's the backstory on the author's tangent about the human genome? It sounded like the human genome project didn't actually do what the name implies.

The problem with bioinformatics is not "prematured optimization", but rather no optimization at all.

Out of curiosity, what other computationally difficult problems are there?

I'm very interested in bioinformatics, but sadly don't know as much about the field as I'd like.