Ask HN: How to Get Sports Data? I am thinking of creating a free REST API service that will allow users to list tournaments, schedules, match results etc - for basketball, cricket, badminton. How do I get the data into my database? Do I need to purchase this from someone or I simply do web scraping of different websites to get the data I want? |
In general, larger players have connections to private apis that come right from the field. Sports betting, espn, etc all pay for expensive connections to get live data.
The historical data will be time consuming but most likely legally safe.
Its the live match results data that you should not scrape in my opinion. Technically it will take some time to set up, but its possible. Its the legal part of all this that I would not recommend.
If i were you, id make two services: free historical data, and a paid live data api. I dont know much about sports data api in particular, but my question is why isnt this already out there? Probably because its very expensive to get enough feeds to have a live api anyone wants to use.
A personal solution I’ve been considering is a scraper that knows how to scrape across a few different sites (so you have fallbacks). This can be used to scrape real time data and distribute data fetching to avoid rate limits or reliance on one site (let’s say cbs sports).
For historical data, you’ll have to get attuned to wiki or specific sports sites that have archives (they are out there). Perfect job to outsource to anyone really if you don’t feel like doing the tedious work.
I’d say scraping is the way so long as your scraper knows how to get the same data from multiple sources. If you got the cash then just google a sports data api.
I also do some scraping in my spare time and i can tell you that large companies will have... precautions in place to prevent you from scraping their by-the-minute sports stats they pay millions for. Id also say right now scraping is a gray market, airlines in particular have proven that they are willing to fight for their (public) data, so tread carefully.
Scraping is a problematic way to do this too, because how can you verify what is up to date? You have to rely on one single authoritative source, otherwise youll be relying on some stat calc which “guesses” what the right stat is. Additionally, if you take live data that multiple large businesses feel they own, now they can split the lawyering fees on you and even if they lose, it will be cheaper for them together.
The historical content probably would be trivial to get, and its hosts most likely dont care if you have it. its the recent/live data which may be particularly difficult to take. In my professional opinion this is two separate projects- a paid up to date version with live stats and a free historical wiki.
There is no way they can trust scraped data. Especially when you can have different matches with slightly different names. For example U21 version of A vs B for a givens sport (learned that the hard way when arbing!).
Thats a start
I tried looking for an nba api but i couldnt find one- but i found nba api projects like https://pypi.org/project/nba-api/
I dont know espn gets their historical data but scraping it wouldnt be problematic. I would think the pro sports orgs would have an api for it
Looks like espn has pretty good historical data records and their api has some documentation. Using the espn api might be your best bet here