Ask HN: Can we make side projects by web scraping?

2 points by tpkahlon 5 years ago | 3 comments

pnwhyc 5 years ago |

Yes. So long as you do it morally. Your project must extract novel insights from the scraped content, not steal it. Some of my most successful projects have been a result of web scraping.

However, there are caveats. 1. If the scraped website receives frequent updates to its html structure, you will need to update your scraper frequently. This can turn your codeitonceandforgetit side project into a monthly maintenance headache. 2. You are entirely dependent on that site. If they shut their doors, or start blocking bots, you have to find a workaround or shut down your project completely

Ultimately, I would recommend it as a means for quickly attracting users, or as automation for part of a larger project. Ensure that the value is > the maintenance.

tpkahlon 5 years ago |

Recently, I created a project (https://cs1000.vercel.app) which is refined and ad free version of an article. Is it fine to build such side projects via web scraping? Do we need approval from site/entity involved?

codingdave 5 years ago | |

No, without permission, this is theft if IP, and not ethical. With permission it would be fine, but scraping content and reposting it as your own is fairly crass.