Block AI bots crawlers on Nginx

Block AI bots crawlers on Nginx(github.com)

1 points by kurren 1 year ago | 12 comments

kurren 1 year ago |

I've been tinkering with a combined solution to block AI bots from crawling my website, it's a robots.txt file (I know, they don't care about it) and a Nginx directive (it works for Ngnix web servers, not if you are on Apache).

LinuxBender 1 year ago | |

An additional thing one can add is to block any connection that is not HTTP/2.0. Many bots still only use HTTP/1.1 or lower. I just drop with 444 but others may wish to give a friendly terse error message using 403.

    if ($server_protocol != HTTP/2.0) { return 444; }

Here's another one that will block most non-browsers (except for headless chrome of course)

    if ($http_sec_fetch_mode !~ (cors|no-cors|navigate) ) { return 444; }

These should be tested extensively on a non revenue impacting site.

Another bot-blocking method is to drop any TCP SYN packets with an MSS outside of a sensible range. Here is an example using netfilter on IPv4 in the "raw" table (to keep them out of the CPU impacting state table):

-A PREROUTING -i eth0 -p tcp -m tcp -d {your_wan_ip} --syn -m tcpmss ! --mss 1220:1460 -j DROP

kurren 1 year ago | | |

mmm, blocking everything not http/2.0 is also blocking legit browser, while blocking http/1.0 does not block bots (at least not ChatGPT); blocking non-browsers with $http_sec_fetch_mode works as expected.

kurren 1 year ago | | |

Thanks. I'll give it a try with the != HTTP/2.0 method.