If you were tasked with crawling the entire web, how would you go about doing it?
Ask HN: Crawl the entire web?2 points by nullundefined 8 years ago | 1 commentIf you were tasked with crawling the entire web, how would you go about doing it?