The evil unit test.(makinggoodsoftware.com) |
The evil unit test.(makinggoodsoftware.com) |
For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore.
The other issue is we had a lot of bugs around the integration of data. For instance if the DB class returned back an empty array vs null vs empty string. When we did the unit test we mocked it to return what we thought it should, which may not be the correct one. Integration tests better caught this, functional tests even more so.
It seems a lot of shops don't use tests at all, now working at places that have done both, I'd rather error on the side of fewer tests, and do it smart instead of hitting a metric and assuming your code falls into line.
If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.
This just means your tests are bad. Write good tests and you won't have this problem.
Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".
Most of the time, however, we are creating web apps and things that run at a much higher level of abstraction than the underlying algorithms. We should probably write tests at the same level of abstraction our solution is.
Although, I guess, when you're writing medical software or something for a bank, you should be a bit more strict than when you are making the next best social sharing app.
A guy I know once solved a bug by having a unit test for the md5 function from the standard library of the language at hand ... turns out a specific version was buggy.
Either way, I prefer using integration/functional test. What I care about is that the right output flies in for a certain input. What I don't care about is how. So I don't test for that.
Also, Unit tests must be created before the code they test.
yes they should. It's very helpful to think about how your code will be used before you write it, helps you think it through as well.
I'm a big fan now though. If I have written code that's hard to test, I re-factor the code so that it's testable. I've found that un-testable code is generally bad code (it works OK, but is hard to modify later). I also run my unit tests during the build (right after compilation), so I know right away if something fails.
I've come to really like unit testing. I'm less afraid to change things and feel better (in general) about the quality of my code.
Do test the API. Test the output.
Good, smart tests should be a pleasure to work with. They should give you the confidence to change the implementation without mercy. They make the feedback cycle faster, bugs more visible, and documentation more thorough.
The only time I've found unit tests to be a pain are when testing asynchronous, event-driven code. The tests aren't any more brittle, but they really need to be supported by good integration tests in order to find the kinds of bugs that can surface in such systems.
I think the point about unit tests is that they should be the default and not the exception. 100% coverage is an asymptotic value. But if you write the test first you can get pretty darn close.
I thought that was the point of unit tests...
> Do test the API. Test the output.
... and that this would be integration testing.
Am I drawing my mental lines separating the range of tests in the wrong places?
fwiw, what I mean when I say "don't test the implementation," is that your test should not test "how" a function or object performs its task when called.
tldr; unit tests should test units -- the smallest amount of functionality possible
A trivial example might be an object that retrieves some data from a persistent store and prints it out. It might have a "fetch" method whose argument is a key in the data store.
You could easily write a test case that checks when "fetch" is called that the data is written to the standard output stream. It would pass and you could go home after a hard day of work.
But if the requirements change and you need it to write to a socket instead and format the data for a binary stream -- you'll have to modify the "fetch" method and the tests.
If we were clever though we'd realize that when we wrote the test to see that "fetch" output its result to standard out we were actually asking the object to do too much and our test was testing several different units of work that could be broken down.
Ideally our hypothetical object would take a couple of callable objects that handle things like printing data to streams and formatting it. Then our test for fetch only has to test one small "unit" of functionality -- not several units (retrieving data, formatting it, printing it to a stream).
hth
The types cover more ground than typical unit tests -- and if you follow a few simple principles (avoid partial functions and some of the worse part of the libraries), you can trust the compiled code more than you trust UT'd code.
Production-quality Haskell does usually employ tests, and those are a joy to write (with QuickCheck), but there's little need to spend time writing tests at the early coding phases.
When doing large refactorings, you can also trust the type system to guide you and find virtually all bugs introduced.
However, I do not agree with the idea that tests are evil. If your tests are failing, or breaking regularly, or are hard to write, then your tests are trying to tell you something. Tests are a direct mirror of the state of your code. If your tests are brittle and hard to manage, don't delete the tests, fix your code. If you need to instantiate a tonne of objects in order to make your tests work, fix your methods dependencies and learn to test your methods in isolation of its dependencies.
Any production code needs tests, this is non-negotiable in today's development eco-system. But I get tired of people who do not understand the purpose of tests complaining that tests are evil. If there is a problem with your tests, then your tests are telling you there is a problem with your code. Clean up your code in order to clean up your tests. However, remember that as code changes, your tests will need to change as well. It is okay to edit, delete, change tests to reflect the current state of the application. Just change the tests first to express how you want to interact with a specific bit of code and then change the code, not the other way around.
I prefer fewer tests that try to touch most of the essential features in one go, rather than several low-level tests that test simple if not idiotically limited cases. I prefer fewer tests and high code coverage: by being smart you can do a lot of testing even with a single test if only it touches the strategically most sensitive parts of your program.
Also, most low-level stuff you will test implicitly by the rest of your code. Lots of your higher level stuff would fail to work if your lower level functionality was crappy. Thus, the basic utilities tend to straighten themselves (bug-wise) early on.
Lines of code have a cost and lots of tests drive up your lines. Less code, smarter code. Less tests, smarter tests.
This is, coincidentally, where property based testing like QuickCheck comes in.
Edit: Just pretend 2x3 has an asterisk instead of an x :)
In fact, we wrote about this before http://bitly.com/zdyJfl
We're hosting a webinar on Wednesday about different kinds of testing and when a unit test is appropriate and when another kind of test, like an integration test, may be a better choice - http://bitly.com/xeYSYg.
Unit tests have a time and place. There are times in which they are a must if you want to reduce technical debt and test your code - both if you're writing mission critical stuff or even when you're writing the latest social sharing app (remember the Fail Whale? We all hate that!). But, no, you don't need TDD on greenfield code all the time and you shouldn't have to write tests for logic.
I think anyone falling into these pitfalls just aren't writing good enough tests, don't dissuade them from furthering their testing skills, promote skillful testing. And change the title.
Good tests don't depend on implementation. Your tests should tests interfaces instead of objects and anything implementing the interface should pass the tests. If it doesn't, your interface is wrong or your test is wrong.
http://www.writemoretests.com/2011/09/test-driven-developmen...
Yes
>You will never access your database if you are not using an ORM.
Hell yes.
(Not necessarily disagreeing with his conclusions)
Works pretty well in Python.
Though it doesn't invalidate the point the author tries to make.
Blanket statements like 100% coverage, unit test per method are not too helpful, and it's important to understand the point where you will start to see diminishing returns with unit tests but that said well written unit tests with ~ 85-95 coverage by line free of conditional logic (use custom asserts) with meaningful test names/descriptions aren't exactly going to put you in league with beezlebub.we understand that the unit testing is trying to test the functionality and making sure that the acceptance criteria is also met at the same time. I would like to mention one more point is that we should also try to write unit test cases not just only test the functionality but also try to do a good code coverage. unit test cases are tools for: 1. functionality testing 2. comply with acceptance criteria 3. a tool to help good code coverage 4. be iterative. be backward compatible as long as it could. (anyway to add logic based on version of the code change? need to think on this). yes it makes it little bulky but will work and avoid needing code cleanup and redesigns.
let unit tests help come up with min 80% of code coverage.
Personally, I've given up on categorization entirely. It's just "automated tests". People seem to be very prone in getting caught up in the question of whether it's a "unit" or "acceptance" or "integration" test when that's a completely uninteresting question.
Even "Test what, not how" is only a rule of thumb; if I've got a complicated "what", I'll sometimes at least probe the implementation to be sure I'm on the right track during development. Of course, if I change the "what" I just nuke them. I say this having done this entire cycle over the last week. My "what" was too slow and I need a new "what", but unit testing the "what" helped me get it to the point where I could benchmark it much faster than if I had tried to "not test the what". (Sometimes, you can also nuke the "what" tests before putting them into long-term storage.)
If you can avoid getting caught up in attaching goodness and badness to the terms, unit/acceptance/integration can still be useful communication terms, but I resist using them for anything else, and I don't personally use them at all.
(Also, to be clear, I consider this amplification or reflection upon your post, not disagreement.)
"My "what" was too slow and I need a new "what", but unit testing the "what" helped..."
If only there were higher levels of abstractions to say what you mean.
I have also had times when I moved away from unit testing in favor of higher-level tests, but ultimately found that the quality of the code got worse and the build time became unacceptable. I'm not trying to generalize here, I'm only talking about my code on a particular project at a particular point in time.
I've had good luck attacking this problem directly. If your integration tests are sufficiently integration-y, often this slowness directly reflects the fact that your system is slow to the users as well!
Or, failing that, that you've got too many dependencies hardwired into your system, or some other real problem.
Following up on my other proximal post here, the question is ultimate not whether it's high or low level, it's whether it's fast or slow, period. I've written slow low-level tests because they fully tested a 10x10x10x10x10x10 state space accidentally, I've written (after some work on the underlying system) some very fast tests that literally simulate user input to the system via the web server at the highest possible level.
The real key is that you can't really follow a testing recipe, you always have to be looking out for the cost-benefits. I suspect the other key is that you must come to internalize the idea that automated testing is really important and if that means spending a man-week to improve your testing environment is what you need to do, then do it. It's really OK.
If you know ahead of time exactly what the inputs and outputs should be then sure, writing unit tests first is a good practice.
The example he works with is writing a driver for Tokyo Tyrant in java. He begins by writing a high level test of what he eventually wants to be able to do. Instead of implementing it though he comments it out and then starts to play around with potential implementation code inside the body of the test. Once he's finished playing he then gradually refactors code from the test body into the implementation until he's got everything he needs and uncomments the test.
It may well be the case that by the time you've finished exploring that you've realised the test you've written isn't very relevant. This is fine, the purpose of the writing the initial test is to guide your exploration in a productive direction. Once you have some implementation code you can always go back and reframe the test according to your new understanding.
The first problem is data consistency. Both require consistent data in whichever environment they are running. Not an insurmountable goal, but one that often leads to a lot of false failures. These lead to a lack of trust in the test feedback.
The second problem, as others have already pointed out, is that integration tests and functional tests don't tell you what is failing or why. Can I create a user? Nope. Why? 500 error. Time to fire up the debugger...
The third problem is that functional tests do not scale. Each functional test will take orders of magnitude longer to run than integration tests, which take orders of magnitude longer than unit tests. You very quickly end up in the situation where you have more hardware than people or you no longer get fast feedback.
Personally, feedback is the primary driver behind test automation. It is nice to know what I wrote will probably work and it certainly helps simplify the code. However, the biggest pay off comes from the fast and repeatable verification that what I changed has not impacted the code base negatively.
My explicit point in this case was about how the labels are actually bad, get in your way, and shouldn't be used because they hurt more than they help.
All my actual programming work here consists of adding printfs at strategic points to tease out exactly where I should be looking for the point data. Unit tests are completely useless for this process.
Come to think of it, even once I've got it figured out, it's very hard to see how to usefully use unit tests here...
http://paulgraham.com/arc0.html
I don't use Arc (or any kind of Lisp) but I understand what Paul Graham was trying to achieve.
Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation.
Remember you are testing what it does, not how it does it.
While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite.
You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html
> Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing.
Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises.
Of course YMMV.
We started using DI, and then had a rule that all unit tests need to be isolated. Which means mocking everything they touch.
For small methods it makes them very fragile, although the other way you end up getting a lot of intertwined tests that can wreak havoc after a lot of tests are made if you're not careful.
I've found that picking what you want to isolate seems to work the best. Our unit tests isolated everything, integration isolated any connections (i.e. DB/Services/etc) which made the tests much less fragile (and much more valuable), but made the errors harder to track down.