We've been trying to evaluate web scraping companies, but when you look at their benchmarks, you can't verify anything, and they mostly exist to prove the company is successful. They put somewhere between 98% and 100% because they pick their own urls, define success their own way, and don't publish the harness. We also saw companies like scrapfly astroturf websites like scrapeway and call them independent.

So, we built an open source benchmark that we want to represent the frontier of web data. We're trying to look across all major anti-bot providers and industries, to build a comprehensive hard-target test set. It's all in the repo, you can run it, and we pledge to maintain it monthly.

Curious what people think we could do to improve it? Would you change any of the conditions, add sites, etc. Our target list and pass criteria are something we're trying to improve, so would love to get feedback from the community on how we can actually provide something useful for people to test against!