Test Failures Should Be Actionable

Test Failures Should Be Actionable(testing.googleblog.com)

25 points by mike-the-brain 2 years ago | 15 comments

jchw 2 years ago |

This one is pretty basic, but it's still very important, and even though it is basic, many tests fail this basic rule all the time. Sometimes it's not really the test's fault: the fact that it's hard to make the test failure actionable actually reveals a design failure in the code. Protip: very seldom does it make sense for a function to return a boolean value indicating success or failure. (If you're a C++ person who hates or for other reasons, can't use exceptions, I hear you: my personal preference is something like expected<T> or StatusOr. I am a bit sad that C++ error handling still remains as fragmented as ever...)

TotT is one of those kinds of things that I really miss from Google. None of the TotTs were exactly groundbreaking, but they almost all made for excellent rules of thumb that would rarely ever be bad advice, and often times even if they seemed simple and obvious, you could still find places to apply them in your actual codebase. (TotT definitely nudged me into making some test improvements when I worked at Google.)

neallindsay 2 years ago |

This is the main reason we have assertion libraries and don't just use boolean expressions.

My favorite test framework of all time, rspec-given[1] (by the late, great Jim Weirich) solved this a different way by using AST introspection to extract both sides of a boolean expression and give a helpful message even with an assertion like: `Then { result == 7 }`

[1]: https://github.com/jimweirich/rspec-given

gavmor 2 years ago | |

Thank you! It feels a bit crazy-making to see a post like this published with no mention of eg rspec "matchers", ubiquitous in Ruby land for decades. I'm somewhat dismayed that this even needs to be said in 2024; I'll bet there's an entry in Ward Cunningham's wiki imploring devs to write tests that fail descriptively but—as evidenced by my lack of citations—it's often easier to write a post than search for a post. (And that's part of Joe Armstrong's "The Mess We're In")

Edit: to make a non-meta point, these sorts of appeals to "please exhibit good behavior" can't stand against economic forces, eg time constraints and KPIs, so I wonder if there isn't a ToolsNotRules[0] implementation which might enforce more introspective assertions.

Edit2: Oh, duh—TDD is the practice which enforces more actionable test failures! After all, an unactionable test failure drives no development. So if you're really keen on actionable failures, try TDD.

0. https://benchristel.github.io/process-to-processes/Fundament...

burnte 2 years ago |

With the rapidly increasing number of websites and applications that feel "an error occurred" is an acceptable error message, I'm glad someone is pointing out that failures should contain actionable information, but I'm also greatly saddened this is needed. I truly do not understand how anyone, much less developers, needs to be told that the content of an error message is important. It really boggles my mind. How is it not obvious to the point of pain that you need to know WHY something failed, not only that it failed?

simiones 2 years ago | |

For a website, if the cause of the error is something internal to the server (a bug or other temporary condition), what could possibly be gained by telling the user the reason why this failed? "An error occured, try again later" is the only actionable information you can possibly give them, if they have no access on your server to fix your bug.

burnte 2 years ago | | |

How do I know if it's an internal error or if I did something wrong? And yes, showing users the internal errors is useful as they can report the issue with the error message, saving developers time in searching.

tichiian 2 years ago | |

There are reasons for not reporting detailed errors to the user.

For websites, security is a big one. Detailed error messages can lead to information exfiltration, exploits, fingerprinting and generally bad things. Long error messages in some network protocols enable amplification attacks.

For general applications, there are commercial reasons. You don't want to enable your users too much, they should be incentivized to buy your pricey platinum premium package after all. Therefore an opaque error message, maybe with some error code but no text and no information, is necessary: The user will need to call support and get the error code decoded and be told in hour-long, tedious and expensive support calls how to fix the error. Actually actionable errors are what makes your company go bankcrupt, especially if you do freemium or OSS+support business models.

burnte 2 years ago | | |

While this is true, I don't think you truly missed my point. This is just being pedantic for the sake of it.

ltbarcly3 2 years ago |

certainly it is preferable to have actionable test failures, but also it should be very easy to rerun a failed test with additional prints or debug code if not. It is important that the way tests are run is not so heavy and inflexible that it takes more than 30s or so from identifying a failed test to rerunning the same test and seeing additional output. Putting effort to make all tests actionable is probably in reality a strategy to deal with terrible, inflexible, very high latency cloud CI setups.

iou 2 years ago |

If you like this blog post I think you’d like this book https://www.artofunittesting.com/

Test naming convention defined there of

[UnitOfWork_StateUnderTest_ExpectedBehavior]

Always resonated with me as from that you could also discern bugs in test code from developer’s intent.

Scarblac 2 years ago |

I feel this is pretty low on the list of things I want from a test.

Regressions should be rare; if you get a test failure when you break something's public API then that is already a success - we had the right test case!

That you then maybe need to spend a few minutes finding out exactly what is failing, I can live with that.

lolc 2 years ago |

I'm often left scratching my head about what behaviour a test was even supposed to test! One thing I find very helpful is to add my expectation as string message to every assertion in the test. When the test fails, it will tell me which expectation was violated.

xixixao 2 years ago |

This is why a powerful assertions library is so important.

(JS with Vitest, etc. is good, Rust - not so much)