(Other co-founder of Neeva)

We wrote an op-ed last month in Fast Company re: the challenges of crawling at scale in today's world: https://www.fastcompany.com/90759792/with-google-dominating-...

tldr; there are two big challenges in crawling the web:

* quantity -- how do you crawl the web at O(B) pages per day

* quality -- how do you make sure you are crawling the high quality parts

On the quantity side, the web is a much trickier place to crawl than 10-15 years ago. Even if you build a system that is well behaved, respects rate limits and work w/ webmasters and CDNs to make sure you are treated as a good bot, the following things will bite you:

* some sites only allow googlebot/bingbot via robots

* even when sites allow all search crawlers, you'll see 429s, 503s, crawl delay directives that limit you to very low qps, and other mysterious errors. (most retailers and aggregators)

* working w/ webmasters works only if you manage to get a response from them (good luck)

* many sites require JS rendering (which is 100x more expensive in terms of the number of assets you are crawling)

(OTOH, Cloudflare is a force for good: https://radar.cloudflare.com/verified-bots)

On the quality side, crawl prioritization works best when you have click data, which most small search engines don't have enough of. In the absence of that, good seed sets and high quality inlinks are your friend.

In other news, eng blogs are being written up as we speak. Will post here on HN as they roll out. Also, like marginalia_nu pointed out, we share whatever we've been working on on a weekly basis on our Twitter account.

user:	vivekraghunatha
created:	June 14, 2020
karma:	12
about:	Co-founder Neeva. Neeva is ad-free, private search, deeply personal to you. We help our users find exactly what matters to them. Sign up at: www.neeva.co Contact me at: vivek.raghunathan@gmail.com or vivek@neeva.co Ex-VP of Eng, Google.