Ask HN: Help! Tech stack for web scraping and simple frontend graphs

2 points by Spinosaurus 3 years ago | 1 comment

Hey all, starting to work on some of my own projects after a 5 year hiatus.

My personal projects in the past have often utilized a Node.js backend (Express) that served a React frontend, with a postgres db with Knex. I have a bit of familiarity with Rails.

I'm wondering how much things have changed in the past few years. I'm reading that React should probably be avoided, Vue seems popular, Svelte seems like the shiny new thing. I'm considering looking into Django as learning Python might be worthwhile.

My day job uses Kotlin, so Ktor or Spring seems like an option.

The project I'm building will basically scrape some data every day, store it in a database, and serve a front end primarily consisting of graphs. Is there a particular tech stack that is more suitable to that these days? Are there any stacks that should be avoided for such requirements? "Overengineering" is something I want to avoid.

rustdeveloper 3 years ago |

For web scraping, I recommend using a web scraping API, e.g. https://scrapingfish.com. This solves all potential problems with getting blocked and can make data extraction easier as well.

For the app, I've recently started using Remix (https://remix.run) and so far it seems to have been a good choice for me. There is a good integration with Remix in Mantine for front end: https://mantine.dev/guides/remix/. I think it's a good full stack choice if you just want to quickly build an app for your project/product.