Foursquare Open Source Places: A new foundational dataset(simonwillison.net) |
Foursquare Open Source Places: A new foundational dataset(simonwillison.net) |
Foursquare at the time was an app for checking into places and becoming the mayor. Which was hugely popular with several of my colleagues that would obsessively checking to places to claim mayorship of places they liked. To ensure Foursquare had good content, they came up with a clever solution: just ask users that are there. This community driven curation was pretty smart. They got users to categorize and correct information.
Later when doing my first startup, I tried to work with open place data. It was a mess. There were bits and pieces available but nothing great. I actually crawled Foursquare and a few other websites in HTML form at some point to get at their ~30M places (at the time). In the end, we never launched anything and there were obvious IP issues with that data set that would have prevented us from using that commercially.
Anyway, Foursquare got all corporate and the whole checking in thing was spun off into a separate app and the main app became a glorified restaurant recommendation thingy which never really managed to stand out from the field of other recommendation thingies. And a data set + API you could license. Because that was still pretty good.
More recently, other companies partnered up to form Overture, which is a competing open data set that Bing, Amazon, and others are contributing to. Overture is kind of solving the same problem.
I'm guessing that's a big contributing reason for Foursquare to be open sourcing their data. It would be interesting to see if their data is going to be contributed to Overture or whether they are somehow against that. Last I checked, Overture still had lots of data issues. Combining the two data sets would probably help.
From a end-user's perspective, it was perfect! It was gamified just enough to be engaging but not so much that it would become annoying or a grind. It was tied to the real-world via restaurants and other places offering special discounts when you "checked-in" or became the mayor of a place. It had people contributing essentially-real-time updates to metadata of all places (opening hours, location, description, phone numbers, images, etc) - way before Google Maps user contributions were really a thing. It had a social network aspect to it that actually worked etc etc..
It pains me that Foursquare, the company, failed to figure out how to actually monetize any of this and keep the thing going that made them popular.. so they grasped as straws and did the whole app split thing (WTH, really) and the community just faded out as far as I can tell.
Dennis is trying something new now, which is true(to him) and to the original vision of Marauder's map and FSQ. He has been on the dogeball train since a long time and hats off to his commitment.
https://location.foursquare.com/resources/blog/products/four...
./clickhouse local -q "select count(*) from s3('s3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet');"
prints: 4180424
Foursquare could conceivably become a member and donate their data.
I'd imagine foursquare is much better on 'places' like this - overture has other things like building footprints etc.
Firstly, the published reason is they intend to let people contribute and maintain the dataset. This is a good open data mindset to allow a dataset to maintain its health and grow. It would make the dataset sustainable.
An unpublished reason is that it could be a sign that things within Foursquare are winding down. Making the dataset sustainable reduces internal costs and increases the value to themselves and others. (like the reasons to open source software)
Foursquare City Guide is shutting down, and recently laid off 105 (25%) employees. I would imagine releasing data before things stop is a good ethical move.
https://foursquare.com/city-guide-sunset/
https://techcrunch.com/2024/05/23/foursquare-just-laid-off-1...
"The letter does, however, address which units were impacted and, relatedly, the divisions that Foursquare plans to ditch, including Visits, OCF, and Foursquare City Guide.
According to Little’s letter, Foursquare is also pausing work on a number of other initiatives, including "Mobile Developers Tools, Geode, and the current version of FSQ Insights."
Not that I'm complaining; it's always nice to have more options.
As an aside a place in my (small, regional) city is listed in the sample gist. Feel like I should win something.
curl -s "https://fsq-os-places-us-east-1.s3.amazonaws.com/?list-type=2&prefix=release/dt=2024-11-19/places/parquet/&delimiter=/"
curl -O "https://fsq-os-places-us-east-1.s3.amazonaws.com/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet"