BBC News Robots.txt | Dark Hacker News

Dark Hacker News

new|best|ask|show|jobs

Built with the HackerNews API. Developed by LJT.AI

Note: The HackerNews API provides up to 500 stories per category

BBC News Robots.txt(news.bbc.co.uk)

2 points by JKFSOM 317 days ago | 1 comment

BBC News Robots.txt | Dark Hacker News

JKFSOM 317 days ago |

The BBC News robots.txt file includes an interesting selection of disallowed pages:

- "Jailed refugee wins right to stay" (http://news.bbc.co.uk/1/hi/england/1678616.stm) - "Teacher's HIV sack claim rejected" (http://news.bbc.co.uk/1/hi/england/beds/bucks/herts/3663494....) - "Should the UK vote on the EU constitution?" subpage (http://news.bbc.co.uk/1/shared/spl/hi/pop_ups/05/uk_should_t...) - Retail students cash in their talent (http://news.bbc.co.uk/1/hi/education/3826037.stm)