Lessons Learned While Crawling the Web

Lessons Learned While Crawling the Web(moz.com)

33 points by jennita 12 years ago | 3 comments

alecsmart1 12 years ago |

The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"

jgmmo 12 years ago | |

I think it should be called 'web scraping 101'

Very thin article.

billrobertson42 12 years ago | | |

I found certain aspects of it really interesting. In particular, the way that the author interacts with the crawling software to gain insight into the structure of a website. Not just the process, but the way he interacts with it is interesting too.