I have been working in QA for a few years now and have made several observations.

* The people who are good at QA usually have the skills to work as a developer. QA is generally not as desirable as development to a young developer, so why do it?

* Many QA people are almost completely useless and exist just to generate paperwork so that the company can cover its ass. Warning signs for bad QA people seem to be certifications and fear of anything that does not strictly adhere to classic waterfall development. The bulk of all defects are found by a small number of people on my team.

* Most developers are horrible at testing and writing unit tests. We have thousands of unit tests and the main thing that they catch is that someone did not update the test after refactoring code.

On the other hand, I've seen QA groups who do no QA at all. Instead, they do frameworks.

"We need a test framework" are terrifying words to me. It means that I (the dev) will have no one at my back. It means that tests will be written by junior QA engineers (the senior ones are off writing and maintaining the framework, right?). It means that I am responsible for my own tests, because /no one/ else besides the customer will be testing my code.

That's generally all right; I think I do a good job. But not everyone does; some devs think they can toss code over the fence to a group that will tell them if it worked or not.

Everyone is responsible for quality; Devs must do unit tests, but they have blind spots and that is what QA is for, namely higher-level assurance that things are working as intended.

I could not agree more. And not only should all projects include QA, but they should include QA by an actual quality assurance analyst or team. QA performed by minimum wage (or low payed) temps or short-term services tends to be next to worthless. Also, don't even think about using a spreadsheet to track issues.

QA needs to be done, and it needs to be funded. Everything you put into will return in less development time (overall, not necessarily in the short term), and more satisfied users.

I mostly agree.

That MMO that is getting away with using customers as QA, and only fixing bugs that NEED fixing? I bet they're making a nice profit. I'm sure the customers are annoyed, but in my experience, gamers won't quit playing over bugs. Bad customer service, price hikes, etc... Sure. But not bugs. Even if they stick around for years.

However, business applications are a different story. Businesses can't afford to be stopped from doing what they need to. And a QA person makes all the difference in the world there.

For those that don't understand why, a programmer can't find their own bugs because they've already done their utmost to find bugs in the code. They've already stretched themselves to the limit before they sent the code to QA. You can train, coddle, beat, or otherwise influence them and only get a few more bugs found by programmers. It's much better for everyone's sanity and wallets if you just pay for proper QA.

One financial services company I worked at a long time ago was getting caught up in the Lean hype. We did process maps of nearly everything including our dev process. They put a big red circle around the QA piece noting that it was a candidate for elimination since it was not value-add.

That is parts like coding and deployment were necessary because they actually were important and helped move business ideas to the customer. QA to them was a redundant step. This coming from a shop that avoided automated testing because it was too expensive.

I was disappoint. QA to me is insurance against being human (which we all are). The ended up not getting rid of QA but the fact that they seriously considered it was shocking.

quoting Linus: "There really are only two acceptable models of development: 'think and analyze' or 'years and years of testing on thousands of machines'"[1]

i would never say that "we don't need no stinking QA", but i do believe that teams who live by "think and analyse" need QA a lot less than teams who don't.

[1] http://www.dustingetz.com/linus-think-and-analyze-motherfuck...

For startups you usually don't have enough people to have QA team. Therefore you have to take a more lean approach: Build Quality In.

On the other hand, when you are building your MVP, progress is not built features, but validated learning. So QA tasks takes more of an approach of building measuring tools (which have to be done by the people building the MVP).

Even on a later stage startup, QA takes a whole new role, specially when you start doing continuous deployment and start developing immune systems. These can get quite complicated (think of Netflix Chaos Monkey[1]), and having a dedicated team can be beneficial.

What we don't need is a 20th century waterfallish QA. But ensuring the quality of a product[2] has never been more important.

[1] http://www.codinghorror.com/blog/2011/04/working-with-the-ch...

[2] http://www.ashmaurya.com/2011/06/your-product-is-not-the-pro...

One thing also needs to be mentioned, is that supportability of features also needs to be something that has to be addressed early on. The unintended consequence of continuing to support every single combination of every single option makes your support matrix, as well as your QA effort, enormous, and your code incredible unmaintainable.

I work at a company where they have a 5 year old product, and the mentality at the time was "Yes I know it doesn't make sense but if the user does X, then it could cause the server to crash, so let's do Y." This grew to a series of matrices that has now turned the act of adding 1 single feature into a 3 pages worth of "If A then B, otherwise C". This has made the QA effort in terms of testing incredibly complex, and even developers get confused as to what is supported and what isn't.

For example, there is an instance where copying a directory takes a different code path than copying a single file. Why? Because they wanted to "special case" this situation. It makes the functionality and behavior utterly unpredictable.

The takeaway that comes with experience is don't spend too much time trying to prevent your users from doing stupid things. If they want to do stupid things and the system crashes, it's okay to say "Don't do it, otherwise it will crash." This keeps your feature set, your code, and your QA effort much cleaner and more maintainable over the long haul.

I worked with the Windows QA of XBox for a while and it was totally awesome to have super detailed information on bugs, where they occured and how often, with core dumps and stacks in case of crashes.

Users and bosses thinks they have done more than their duty when they said you that "it doesn't work", "nothing work" without more details. You have to run after them to get more context and detailed information.

I love QA.

I'm really glad to be working for a company that has the QA team involved from requirements to production (& maintenance thereafter).

One of the major problems can be keeping the newer people & graduates interested in the testing & QA process. There can be a lot of 'grinding' through test cases and I guess it takes a certain mentality to get through it on some days.

Sometime one must experience both side of the fences in the software development process.

I've been a QA, QA Developer, Developer, and Integrator during my internships.

QA is often the less honoured role of all. This creates a stigma that QA is some sort of crappy role to be in.

Being second class citizen requires more support than ever to work at the top day to day. Unfortunately, most people are not built that way so quality degrades because QA was treated like crap.

This leads to less advancement in the QA community because it's not a good role to be in. You only see a few books and a few noted experts in the QA community that continue to push better practices.

At the end of the day, it's a loss for all of us.

I wish there's none or minimum differences between QA and Developer as I agree with one of the comments here that if you do that, there will be more jobs created and IT professionals can experience different roles within our world.

My opinion is that QA should have a wide range of skills from load/performance testings (that means knowing the tools and statistics), automation testings (that means if DEV created hard-dependency issue that is not easily testable, QA should step in and teach DEV the patterns and best practices to avoid such situation), QA should understand business practices to beat up stupid Business Analysts.

More importantly, QA should keep track deficiencies and ineffectiveness during the development and potentially offer solution (or can be done through group discussions).

Unfortunately, that's not how the world works. Telling someone that he's not efficient or pointing out where the most bugs occurred when the team lead is the one whose responsible for the feature isn't desirable.

Not to mention that certain type of companies don't necessarily get the most ROI from having QA (if your typical Web 2.0 social networking tagging machine had a few issues, users probably complain but everyone will move on).

Ugh. I worked QA years ago and swore to never do it again. Management hates you because you're telling them the product is not ready to ship. Developers hate you because you're pointing out their errors. No one has your back, and everyone wants to blame you.

The company had literally fired the whole testing department previously for saying the product wasn't ready. With those annoying testers out of the way, out the door it went. And promptly had multiple emergency point releases the following week to fix showstoppers cropping up at the customer sites.

That got them to create a testing department again - but it was still without honor in the company. And firing the whole group again was often discussed, as they were _still_ delaying releases by finding bugs!

I departed. They got eaten in a merger several months later.

Relevant: why QA jobs are more fun than developer jobs http://blog.foundrs.com/2011/09/19/why-i-prefer-testing-over...

QA is the wrong solution to this particular problem. The problem is that in Java you can call methods on uninitialized objects. If Foo can "foo" and Bar can "bar", Foo can't "bar". But if you have "null", then suddenly it can "foo" and "bar" even though it's not a subtype of anything that can do that, and it doesn't implement those methods itself. Does that sound fucked up? Well, that's because it is.

QA is nice, but it's a finite resource that you should save for things that actually matter, not stupid things that a computer can fix for you in 30 seconds.

I would have liked to see a more realistic discussion of when QA helps the bottom line and when it doesn't. Plenty of software is sold and tolerated by users despite having bugs. If Microsoft waited until the bug count in Windows was 0, they would never ship, and would go out of business. In other words, this blog post is telling us that the correct amount of QA is higher than "no QA", but it is not helping us determine what the right amount of QA is.

While I agree we need QA (today), I would like to see different expectations.. Expectations that over time we will need less and less QA work.

I don't have much to add to the discussion. I just want to vent a bit of frustration. I came in this morning and found out that 7 of my co-workers were just laid off. We are a 30 people company so thats quite a big deal. And some of my good co-workers in QA were axed. I guess some management guy thought QA was unnecessary. Oh this seems to start to look like 2008 all over again!

I think the big thing that confuses people is that at a lot of companies with QA, QA is really "testing and QA" testing can (and IMO should) be done by the developers, and when people have seen QA as primarily being testing, they think they can do without if the developers take on testing.

Question:

When you're doing QA on a product, do you worry about border cases? I found quite a few bugs related to time syncing and inadequate safeguards in a project I worked on - but they were difficult to reproduce without a macro.

I'm still not sure whether people were happy or angry with me, for reporting all those bugs.

I used to run a QA department for a large e-commerce company and love what this article has to say. Nothing gets executive attention like putting a dollar value on downtime (like $1k per minute on a slow day).

I question how much good QA did for his project (Delta Graph), because it was so buggy it was nearly unusable.

A quibble: Although I have never doubted the importance of QA in the web development process (it can save your bacon), in most cases I do find it rather absurd and bit of a mockery when those who practice QA in a web environment are allowed to have the word 'engineer' in their job title. Does anyone else agree?

I worked at a QA-heavy financial organisation.

The only reason they needed QA is the software was a giant toilet-clogging turd. There were no formal unit or integration test cases so it was used to catch regressions against massive test plans knocked up in Excel, Word and god-knows what else.

"However, business applications are a different story."

You've sure got that right. Our application conducts real-time purchasing events (reverse auctions) where bidders are ranked based on a couple of different configurable algorithms. An error in calculation means incorrect ranking, which means the entire outcome is invalidated. Considering that it can take weeks to put together a purchasing event of this type means that there is a lot riding on that single series of calculations.

Simply releasing and finding bugs in production would be financial suicide. QA isn't even a question in our business. Everything is tested thoroughly when it goes out the door.

Lately my interest has been piqued by analytical code verification techniques and/or fuzzing (going beyond simple static verification or "checking the code style").

I think a lot of bugs made in day-to-day coding could be found given such methods. It would be best to integrate something like that into the compiler (and IDE) as it has the necessary information and structures. I believe the clang team is working on this (http://clang-analyzer.llvm.org/).

I guess it'd require defining "higher level" pre- and post conditions and assertions and such in the code...

Not a substitute for "proper QA", but I think sometimes that "software engineering" is simply dabbling in the dirt as a engineering discipline, a computer could probably run the function you are editing right now 1000000x per second and/or verify the hell out of it while you are thinking about the next letter to type :p

Actually, Linus seems to be saying: we have tested the crap out of this, and the patch broke something, and instead of playing guessing games, revert to the thoroughly tested version and think more closely about the problem before you submit stuff that hasn't been tested on thousands of machines, and will likely break one of them (because you didn't understand the whole problem). This exchange doesn't really cast any light on whether you need less QA. Instead, it seems to say: we rely on past tests to benchmark quality.

I have had my own problems with incompetent QA "engineers," but I do think an excellent QA team helps create an excellent product.

The Linux development/QA cycle is fairly interesting.

Greg Kroah-Hartman has given a few talks on this, and from the dev perspective a lot of it boils down to "make small, discrete changes, compile, and just f*cking ship it". Testing happens among users.

But, because this is free software, there are a substantial number of systematic testers among the users, whether this is commercial distros (Red Hat, Suse, Ubuntu), noncommercial ones (Debian), organizations with a substantial interest in the quality of Linux code (IBM, Intel, AMD, from which we've got some pretty extensive test suites), proprietary software developers (who hate it when a kernel change breaks their code), independent researchers (I know there's a lot of software quality research based on Linux), and even some performance bashing by Linux haters ... which results in better Linux code.

And there are a lot of users who end up discovering bugs as well.

But the coverage is pretty good. Even if it's not explicitly rolled into the dev process itself.

Similar situations apply to other projects, though Linux for its size, scope, and user base tends to do better than many.

The context of that is entirely different than a "do we need QA" context. "Think and Analyze" is an abstract philosophy, not a process for engineering.

I'd say that if you have a T&A team it would be great to measure and publish your defect rate. Of course, if you just have better than average coders, then naturally your defect rate can be low.

But even a low defect rate does not preclude having a formal QA process. Even the best coders in the world can ship the wrong product when the specifications are wrong.

Linus is in no way representative of the average developer, nor are most people on HN. Most of the developers in the industry are barely competent.