If you were ever curious whether a site allows signed crawlers, you can use this tool. This tool respects the website's robots.txt directive.

You will be surprised how many sites allow all robots, but then block crawlers