Show HN: Open-source LLM-powered test automation library for mobile and web

Show HN: Open-source LLM-powered test automation library for mobile and web(github.com)

10 points by p0deje 1 year ago | 2 comments

Hey HN!

I'm Alex, a tech lead of the Selenium project and a QA engineer. For the last 10 months, I've been working on Alumnium - a test automation library that should make my life easier and let me write less code when automating the tests. The goal is to use LLM for _automating_ so I could focus more on _testing_.

Unlike other similar projects, Alumnium can be used in existing tests without changes to test runners, reporting tools, or any other test infrastructure. This allows me to gradually migrate my test suites (mostly Selenium) and revert whenever something goes wrong (this happens a lot, to be honest).

Other major differences:

- cheap (works on low-tier models like gpt-4o-mini, costs $20 per month for 1k+ tests)

- not an AI agent (dumb enough to fail the test rather than working around to make it pass)

- fast thanks to a built-in caching

- supports both mobile (Appium) and web (Playwright, Selenium)

- supports completely local execution (Ollama)

The project is still in a very much experimental phase, but I'd be happy to get any feedback.

pydry 1 year ago |

This looks like a recipe for making test flakiness even worse.

p0deje 1 year ago | |

It's quite stable on simple applications, but the flakiness definitely increases with more complex apps. I think there is a lot to do to fix it, one idea is to trim the accessibility tree to a narrow region that is relevant, then use it in a prompt. For example, if you want to "sort by price", the LLM doesn't really need to know about anything beyond the filters region of the app.