Ask HN: Aggregating authentic user reviews across platforms?

2 points by howardV 1 year ago | 1 comment

I'm exploring the technical feasibility of building a tool that aggregates genuine user reviews about websites from various sources (social media, forums, review platforms, etc.). The core challenge: How do you programmatically collect and verify authentic user sentiment about a website while respecting rate limits, ToS, and privacy concerns? Technical questions I'm grappling with:

Data sources: Which platforms actually allow review scraping legally? Authentication: How to handle platforms that require login for review access? Rate limiting: Best practices for respectful data collection across multiple APIs? Spam detection: How to filter out fake reviews and bot-generated content? Real-time updates: Efficient ways to keep review data current without overwhelming source platforms?

Broader questions:

Has anyone built something similar? What were the biggest technical hurdles? Are there existing APIs or datasets that make this more feasible? What legal/ethical considerations am I missing?

Currently researching this space and would love to hear from anyone who's tackled similar challenges in review aggregation, web scraping at scale, or sentiment analysis. Any insights on the technical architecture or cautionary tales would be incredibly valuable!

8organicbits 1 year ago |

Let's break it down:

Authentic user sentiment - this is an impossible problem. At scale, the best you could do is to ask an LLM to rate sentiment and authenticity. If you can tolerate inaccuracies, that may be viable.

Rate limits: web crawling frameworks do this out of the box via robots.txt, various headers, etc. Non-trivial to set up, but not novel.

ToS: You'll need a lawyer to advise here. Possibly by reading each ToS document individually, including every ToS update.

Legal/ethical: we'd need to know more about what you're doing to comment.

Generally: Retail websites want users to use their website to purchase products. If your scraping doesn't drive traffic to their sites, then they won't want your scraping. If users previously went directly to the retail website to view the reviews, but now used your site instead, then they'd fear loss of revenue. One way or another, they'll try to prevent this from happening.