As I discussed in my talk "Testing is Overrated", programmer testing finds a different class of bugs than manual testing or code reviews.
One of the problems is that programmers are not very good at writing tests. They are much more likely to test "happy path" scenarios. Steve McConnell reports in Code Complete that mature testing organizations have 5 times more dirty tests than clean tests, while immature testing organizations write 5 clean tests for every dirty test.
Another big problem is that unit tests are never going to show you that your software blows. Only user testing can find that.
If you want to ship good software, I think you need to do a combination of code reviews, unit tests, system tests, user studies, and manual testing. Different organizations will find a balance that works for them.
That last is a key point. I like writing tests, but I hate the dogmatism that if you're not writing tests you're doing it wrong. Obviously, good software has been written without tests, and buggy crap has been fully TDD'ed. In the end, the team matters more than the methodology.
http://www.scribd.com/doc/8585284/Testing-is-Overrated-Hando... http://www.scribd.com/doc/8585444/Testing-is-Overrated
For example, TDD encourages code churn: you write tests for a lot of throwaway code.
Personally, it usually takes me two/three attempts before I'm satisfied with the version of the code that I have. Writing tests for these throwaway attempts is a waste of time.
As you become more experienced in writing tests and testable code, you realize that you no longer need TDD to write code that can be easily tested, so I see TDD more as "training wheels" for beginners.
Code should be developed with as large a view of the design as possible. TDD restricts development (by definition) to satisfying the tests. The effect is of 'moving the bugs around' since the existing tests rarely completely describe the whole design. You get into a loop of "oh, yeah, well we Really meant to write a test to cover that case too", repeated ad-nausuem.
Behavior Driven Development really shines in the aspect of "completely describing the design" as it focuses on driving development against a well defined behavior (using scenarios). BDD lets you know where the bug is: in the code or in the specifications (the behavior as specified by the customer). So there is a lot less "chasing around". If you are trying to cover behavior of a system only using TDD then I agree you can end up chasing around a bug.
However, both are necessary and highly effective when used together.
So yes, you record fewer bugs, fix those narrow cases quickly and pretend you're done.
Then it gets to the field, and whatever you didn't catch becomes a 3-alarm fire.
I vastly prefer shakedown tests of the sort where you launch a 'bot army' on cloud servers to thrash the app. Especially if there is Any network/client/server component.
"Dependent Types: A Look At The Other Side" by Larry Diehl
It talks about using Types for Driving your Development. So it is TDD, but without tests, without runtime and programmer's overhead of them.
When using TyDD, you isolate core that should go as bugless as possible and write DSeL for it in language with strong type system. The talk above tells about Agda, but we and some number of other companies in the world (Eaton http//eaton.com/, Echo http://aboutecho.com/ - from the top of my head) successfully employ Haskell for that task.
That way you shoot two birds with one stone: you get provably correct software and you get an ideal language to express your core problems.
Best of all, you get free "requirement tracking" - whenever your requirements change, you express them in types and then apply changes where they should, guided by friendly compiler. Still getting provably correct software (modulo types) at the end.
However, I actually came here to say something else about his material: if it is harder to decompose the business requirement into Haskell than it is to write monkey tests for one's Blub language, I am not certain anyone wins when going with a provably correct implementation.
My experience using advanced type system tells me that you can get away with functional tests. Types, also, takes much less code space-programmer's time than tests.
So I do not see how they could be harder to use. Certainly types will be unusual, but not much harder.
https://webcache.googleusercontent.com/search?q=cache:grokco...
I think that if you read the article linked, you'd find that the study didn't work that way. Any study that measures different things between experiment and control groups is not going to be sound.
Then it gets to the field, and whatever you didn't catch becomes a 3-alarm fire.
I've worked on TDD projects, and those are definitely are counted as bugs. Severity 1.
According to this article, the effect of TDD is to remove your ability to defer that decision until a time when you have more insight into which of those is better.
PS: I suspect, TDD ends up being something of a premature optimization on larger projects, you write tests before you know which direction you want to take the project in. Don't get me wrong Tests are great from a process improvement standpoint, but every major shift ends up breaking way to many tests that you either don’t do it, or you just wasted all that time writing tests.
This is why I'm a BDD guy, not a TDD guy. Seems like you're writing a lot of the wrong kind of tests...
TDD advocates make assertions based on a certain common sense ("tests are scaffolding") but I don't see the present evidence supporting such assertions.
The thing is that TDD also often calls for not spending a lot of time on "Big, upfront design". I could just as easily argue that without a coherent design, your next iteration is going to be longer and that the critical thing is to master good design principles. But that belief also needs evidence behind it.
Programming is ultimately a mysterious thing and so looking at real, long term data is good, especially after you become good enough that everything seems clear to you.
You could argue that without a coherent design, your next iteration is going to be longer. I'd argue the reverse - but it depends on what you mean by "big upfront design"
Every program, TDD or not, needs to get a basic architectural outline in place early on - e.g. decide if you're building a web app or a desktop client, which framework you'll use to build it. Usually you have a good idea of a lot of other major pieces of that you want in your architecture - they're there in "best practices" such as SOLID, repository pattern, CQRS, MVVM, etc. TDDs encourages "spikes" that prove the concept with a vertical slice through all layers of the app.
In my opinion and experience, "Big, upfront design" is a 1980s-1990s idea that all details of the program, not just the outlines, can be designed once before coding commences. And it is a completely false idea. TDD and agile say that you're going to redesign anyway, so you may as well accommodate that. This is where TDD speeds things up. Change is not just adding things to a program, it involves redesign on existing parts.
But I agree that an ounce of data is better than a pound of theory in this case. And number are far better than anecdote. That's what the article is for, I guess we should be reading it.
And if your product does not ship for 6 months, then your first few itterations even if faster will also be behind.
PS: (3 to 6) * (1.15 to 1.3) = (3 * 1.15 to 6 * 1.35) = (3.45 to 8.1).
Edit: No wait it's 15-35% longer so (3.45 to 8.1) months.
Anyway, I tend to build the happy path by any means necessary, verify that's what you want, then go for TQM style process improvement to add quality. I think of it like pulling a tiny string across the river before you build a rope bridge.
It’s not hard to add heaver gage rope to work your way up to a bridge. But, good luck crossing the chasm if you don’t know how to swim or where to find a boat. While that original string may have little to do with the finished bridge, but at least you have something to fall back on when you’re trying to get the GPU to do something etc.
PS: Ok, I think I took that analogy well past the breaking point.
Reminds me of the old joke "A: I'm looking for my wallet. B: where did you lose it? A: over there. B: then why look here? A: the light's better"