Ask HN: What parts of software testing can realistically be autonomous today?

1 points by nishilpatel 177 days ago | 5 comments

Modern systems have good observability and better tooling than ever, yet testing often remains manual and slow. In your experience, what aspects of testing can actually run autonomously today, and what still seems fundamentally hard or unsolved?

stellarvore 177 days ago |

Historically speaking, if an autonomous system found a bug [monkeys and fuzzers have done that for years ;)], you'd just get a "test failed" log. You'd still need to manually dig through the stack traces to pinpoint the "why" of it. Well, an Ai app can tell you that the system crashed or a button moved 5px to the left. But it still can't tell you if that movement broke the UX without a human explicitely defining the heuristic first.

IMO, the hard parts that actually getting solved right now is the autonomous serialization of state; context capture and issue repro steps to be precise (and definitely not the "finding of bugs" part).

chaitanyya 176 days ago |

It's an interesting one. Every time I speak with engineering teams about reliability and correctness, they all want more of it, yet when it comes to investing in it, it's never really a priority.

More often than not, people test the wrong things; they struggle to even identify the right properties to test.

I question my worldview on this because I don't think it's a particularly difficult problem. There are companies like Antithesis that have done incredible work in this space.

I am building in automated property-based testing, and it's not an easy sell.

linkedinlobster 177 days ago |

We did not fail to automate testing. We automated the easy parts and kept calling it progress. The hard part was never running tests. It was deciding what actually matters. We cannot automatically detect risk. Risk lives in context, trade-offs, and user expectations...none of those are deterministic.

jeffreygoesto 177 days ago |

Testing is answering "does it do, what it is supposed to do?" and autonomous means "according to it's own law(s)". Sounds like a contradiction to me. I'd answer with "none".

pfdietz 177 days ago | |

One can define properties the software is supposed to have, then autonomously test for those properties (as in, initiate a process that spends arbitrary amounts of time running new tests to try to show the software fails to have those properties.)

Is this not autonomous because the properties weren't created without humans being involved? How could that even be possible?