Ask HN: Protecting your Sites/Services from Unwanted Traffic?

6 points by prologic 6 hours ago | 2 comments

So... If PoW (proof-of-work) schemes like Anubis ultimately don't' work in practise, because Bots/Crawlers are increasingly using headless browsers, are able to solve captures, proofs, etc;

What options do we have realistically to filter out or block unwanted traffic?

What clever schemes can we come up that don't rely on centralised serices like Cloudflare?

mstaoru 24 minutes ago |

Browsing is absolutely intolerable these days, with almost every website having a block screen, "we're checking your browser...", stuff like that. Lately I just close these websites immediately.

wanderingpixel 1 hour ago |

It depends on what you define as unwanted traffic.

Personally I do not have an issue with bots as long as they behave and are not straight up malicious, so I rely on a combination of rate limiting, a fine-tuned OWASP CRS ruleset and an aggressive Fail2ban enforcement (hit 2 triggers and you get a 24 hour ban, 2 bans and you get banned for 30 days).

My sites also make extensive use of static elements and caching.