Ask HN: How would you build a custom web scraper?

2 points by yashbhatnagar 3 years ago | 2 comments

I'm trying to build a web scraper that can scrape similar websites. For example, if I wanted to scrape 10 different fast food sites and return the locations the stores are in, it would be able to identify different elements and do so. Any ideas on how to implement this?

sharemywin 3 years ago |

1. crawl pages looking for more pages on the site

2. scrape text

3. find address using something like this:

https://stackoverflow.com/questions/9397485/regex-street-add...

JoeyBananas 3 years ago |

Use Scrapy to get at least some measure of maintainability.