Their testing system depends on tests printing "OK" after every test. This means that in many cases, tests failing are indicated by the _absence_ of "OK" being printed.
(We've attempted to isolate those parts and write our own stuff testing against upstream in pytest. We once presented a proposal to move them to pytest, offering to do any work and even wrote pytest plugins to seamlessly integrate with their current system. We got a - literal - "Thanks, but no thanks.")
Oof. If they'd instead put "ok" before every test, they might have been accidentally compatible with TAP! https://testanything.org
It is nice to not have to depend on the language runtime to do the test.
You could even be a victim of "Embrace, Extend, Extinguish".
My advice is to consider forking it, and poaching contributors, in the interests of common good.
I’d love to investigate this further.
Experience has taught me that the “right” testing framework for a project is whatever the developers are happy and productive with.
If so that would appear common as came up right away.
Maybe unit tests need unit tests? (There’s probably a lint rule to catch what I describe above)
Yep - meta-testing (ensuring that every unit test that exists in a project adds unique coverage, remains valid, runs as expected, and I'm sure many other properties) could (and should!) definitely be automated.
Some more advanced meta-testing could involve tracking changes to a project's source history over time (in other words: tests that run with commit history). By that I'm thinking of situations like: "does this test genuinely still test what it used to, after the test and/or application code was modified?"
But yeah that would be a good thing for a linter to catch. I'm not aware if any do.
A reviewer should catch this error easily. I kind of think many don't give much attention to unittests when reviewing. Which is bad. Good unittests are far harder to write than good code.
There's much more subtle errors of this class (False Negatives / always pass).
The fix isn't to blame people for making mistakes. It's to figure out a design that doesn't allow this mistake to happen in the first place.
For example, the method could (today) require the second argument to be a keyword argument. This is also something a good linter should be able to warn on.
edit: rikatee and I wrote essentially the same reply at the same time. :-)
We do code review because we expect human error when the code was written by a human, but then we also expect not human error when the code is being read (reviewed) by a human? Any process that expects zero human error will always fail.
That's where linters add value: they allow devs to do what humans are good at (the creative complex and interesting stuff) while the bots do what bots are good at (the boring repetitive stuff)
Instead popular culture has decided that at best, this is what Schrödinger believed (Ha those crazy scientists) and at worst that somehow the cats being dead and not-dead at the same time is the core idea of quantum physics :/
> assertTrue also accepts a second argument, which is the custom error message to show if the first argument is not truthy. This call signature allows the mistake to be made and the test to pass and therefore possibly fail silently.
With modern features in python you could change the signature to
assertTrue(expr, *, msg=None)
assert expr, "custom message"
"assertTrue also accepts a second argument, which is the custom error message to show if the first argument is not truthy. This call signature allows the mistake to be made and the test to pass and therefore possibly fail silently."
Searching for announced breaking changes about arguments to Python included functions...
https://bugs.python.org/issue25628
https://bugs.python.org/issue29193
Bear in mind only 28% of codebases actually use built-in unittest package that this gotcha is affected by, so really it's 20 of 28% of 666 aka 10% ... but that claim would be hard to justify by folks that dig stats.
Yet people are keeping larger and larger objects in a coherent state. Probably nobody will ever do it with a cat, but quantum physics is keeping its tradition of taking anything people think as absurd and saying "well, not really, look at this".
Minimizing runtime dependencies is nice, but personally I couldn't care less about build/test time dependencies.
I don't touch Django so can't comment on that though.
Really? I've can't think of a "real" project that did not include 3rd party libs.
Reinventing wheels or struggling with poorer implementations (unittest vs pytest, http vs requests, etc) is huge drain for a project. A huge misstep.
Any one third-party library comes with these drawbacks, and each library must be evaluated individually. Some may be worth the pain (requests, etc.), but many are not. One should try to minimize the number of third-party dependencies one has, not necessarily eliminate them entirely. It’s simply that the lower number of third-party libraries you depend on, the less pain you get of the kinds I listed. Every individual third-party library will have to overcome that threshold by being useful enough. And I doubt that, for most people, pytest is that much better to work with than the built-in unittest is.
> While pytest is very popular, 28% of codebases still use the built-in unittest package.
> And I doubt that, for most people, pytest is that much better to work with than the built-in unittest is.
What've I missed?
This sounds more like you're saying that pytest wouldn't overcome that threshold; I was saying that has already happened, at least according to the article's statement.