Ask HN: Is there a robots.txt equivalent for LLMs? like LICENSEME.txt? If you run a blog and don't want to allow LLM crawlers to train on your content, do you have options? |
Ask HN: Is there a robots.txt equivalent for LLMs? like LICENSEME.txt? If you run a blog and don't want to allow LLM crawlers to train on your content, do you have options? |
Otherwise you need to selectively allow certain bots. However, as well as with web crawlers, respecting a robots.txt is optional.
Insidious with AI-models is that it is difficult or practicably impossible to prove that it trained on your data.
Difficult to establish a standard like robots.txt. There also was .well-known/security.txt that Google proposed. Some sites serve it, but it hasn't really become a standard.
But if you are concerned there's a good resource here for blocking them: https://darkvisitors.com/