Generating unit tests in Go(magictests.com) |
Generating unit tests in Go(magictests.com) |
I opened up the demo page and generated the tests. On first glance, I see a bunch of tests that just verify that the given code does exactly what it was written to do. That misses the _entire_ point of tests. Oh, and only a single test case for each function.
Looking a bit more closely, the test cases wouldn't even pass. They're just filled with placeholder values. Okay, fine, the boilerplate is generated, but you have to fill in the expected returns.
What about mocks? Why can I only have one test case per function? Also, None of these test cases are documented. You're really still writing the bulk of the tests yourself anyway.
I'm not saying there is no value here, but if there is value, that demo shows none of it.
There is almost no use for unit tests, as they lock down function implementations without verifying functionality. They have some amount of use for a programmer to verify if what they wrote really is what they wrote, and maybe for data structure methods (not in Go of course), but that means maybe 1 in 50 methods justifies a unit test.
Everything else should be system tested, to see if the components fit together and if the interactions between various parts of the application really are what you think they are.
It's like javadoc back in the day. "Document all your function variables, you've only got them for x%", ... you write a parser that parses the app and simply adds javadoc everywhere, giving the obvious descriptions to obvious names. 2 hours of work for the generator, but actually interesting work, half an hour to read through it and change a few things, and boom, 30000 lines added in a day. And the worst part is, you've just made everything harder to read, but everybody's happy with you.
Unit testing is not an exercise in the mundane by making your manager happy, it is about validating that the thing you just wrote actually works, so that you can move on to the next thing, eventually aggregating all of that work together and constructing code that works!
As others have said, you test inputs and outputs (and maybe you need to test some failure modes depending on the complexity of the situation). When you're working on a parser, or lexical analyzer of any sort, wouldn't it be nice to know that it is capable of parsing what you thought it should?
System/BlackBox/Integration tests are too high-level and the more you rely on those as your sole testing, the more brittle and difficult you make it to track down the cause of test failures. It also means that you have to spend more time constructing test scenarios to testing the inner-workings of the system, things that are easier to do in unit tests. Bloated tests at that layer actually make it MORE difficult to refactor.
Decent unit tests help you define the inputs and outputs of your functions/classes/design. If it's hard to write a unit test on it, then you probably have side-effects in the code that you can't properly account for (bad code design). When it comes to a point that you need to refactor and the unit test is in your way, then guess what? refactor/delete the test...
This is...curious, given that every place I've ever been with any sort of testing culture tested contracts, not function implementations. They test the range and domain of the function, not the behavior of it.
Unit tests do not replace system-level/integration testing, but to dismiss them out of hand is manifestly unwise.
1) A smallish number of full end to end integration tests which actually make TCP requests to your app, and which actually have service talk to each other, etc.
2) A medium number of black box integration tests that just make HTTP requests to a single isolated service or component and verify that the response comes back out as expected (with no expectations on what happens in between input and output).
3) A much larger number of unit tests that are direct execution of business logic (just calling the code directly instead of calling the server externally) to verify various edge conditions in the inner underlying code.
To use a real world example you might have full end to end integration tests to ensure that your service allows the creation of a new account, and that when the account is created an email is dispatched, and that an authentication cookie is set which gives this new user the ability to make requests to the service.
This is just a high level test though. To back this test up you should have at least a couple hundred unit tests verifying that various edge cases with email formats, name formats, unicode characters in input, etc are handled correctly.
To speed up test suite execution time it doesn't make sense to run each of these hundreds of edge cases as full end to end integration tests, or you end up with a horrific test suite that will take minutes to run. So instead you should use unit tests to cover the underlying edge conditions. You can usually run a hundred unit tests in the amount of time that a single integration test will run.
I would disagree. You can change the implementation all you want, but you cannot change the functionality (contract). If your unit test verifies implementation rather than functionality, then it is a poor test.
I usually find such tests result from using mocks poorly.
Then i realised that this is a demo of a company that someone actually wants to make their fortune with:
All too rarely do I see people focusing testing on public APIs.
I don't generally write, nor really see, see a lot of internal contractual testing in unit testing (not least because it becomes very hard to do so if you actually use visibility in OOP).
That would be a system test. Nobody would write a parser in one function, so it's full functionality can't be tested in a unit test. This is also exactly the sort of test I'm advocating. The thing I'll guarantee you though : it won't provide 100% code coverage.
Unit tests are this kind of crap, testing trivial things 100 times. You immediately see why they provide 100% code coverage : https://en.wikipedia.org/wiki/Unit_testing#Design
> the more brittle and difficult you make it to track down the cause of test failures
Yes ! This of course happens because they find ever-more subtle bugs, not because there's something wrong with the test.
Yeah. 100% coverage is BS, again I go back to proof of quality.
> because the find ever-more subtle bugs
Well, yes and no. Yes integration tests are important for finding subtle bugs between different components of a larger system. No, bc an over-reliance on these instead of proper unit-testing, you end up with code bloat in your integration tests, and a maintenance nightmare because the tests are more complex. On top of that because you have so many integration tests It becomes unclear when the integration test is testing a valid API construct, vs something internal, making it very difficult to refactor code because you don't know what's expected API response vs some odd internal bug that you had to test for bc you didn't have a good unit test for the internals.
Since I tend to deal with distributed systems, I generally think in terms of blackbox tests as dealing with application APIs rather than function APIs, and integration tests as multiple system components being tested together.
Why?
I've heard this story a million times. Nobody's ever been able to robustly justify it to me. I don't believe it's true.
If you don't have a test for your subsystem that covers the more common use cases; how will you tell if it broke when you updated one of the functions/methods it uses?
If you don't have a test for individual flows at the level of the entire system how will you know when you did something that broke the build outright?
If you do none of that testing; you're handing off that function of your development team to the users of the system you're building.
Testing is an engineering practice that lets you make changes to complex systems with confidence that the system is still doing it's job after the changes you make.
It's not magic; it's engineering. As such, it's aim is to be boringly predictable.
And yeah, it's annoying; but it's a lot less annoying than facing an angry customer who is screaming about lost data, lost money and how they're going to sue you into a smoking hole in the ground for deliberately deceiving them about the issues with your software.
> If you don't have a test for a given item of functionality; how do you know it works?
This would be an integration test
> If you don't have a test for your subsystem that covers the more common use cases; how will you tell if it broke when you updated one of the functions/methods it uses?
That would be a system test
> If you don't have a test for individual flows at the level of the entire system how will you know when you did something that broke the build outright?
I may misunderstand the question, but ... you compile it ?
> Testing is an engineering practice that lets you make changes to complex systems with confidence that the system is still doing it's job after the changes you make.
And if you have unit tests ... doing those changes is twice as hard as when you don't have them. Also if you test the "job" of the complex system, that would definitely not be a unit test.
This is what unit tests look like in large companies. What you actually find in the field
def add1(a):
return a + 1
def TestAdd1(self):
self.assertEquals(add1(2), 3)
self.assertEquals(add1(-2), -1)
self.assertEquals(add1(8), 9)
self.assertEquals(add1(12), 13)
self.assertEquals(add1(5557), 5558)
None of your concerns apply to code like this. And nobody, ever, for any reason, should write code like this.
I personally don't get too hung up on whether a given test is a unit test, a system test or an integration test. For any reasonably sized project you're eventually going to have all of those. And sometimes something that was a unit test will suddenly become a test that crosses multiple units as requirements change and as dependencies get added to the project.
The viewpoint I take is that tests are an integral part of the software, they may not be part of the final delivered artifact but they are part of what defines the software.
You should be thinking as much about how to test that your code is doing what it should as to how to make it do so in the first place. If you do that you will have a far better understanding of the intent of any given piece of code and you will have done half the work of debugging before you write the bugs.
self.assertEquals(add1(Integer.MAX_VALUE), ???)