OpenFreeMap survived 100k requests per second(blog.hyperknot.com) |
OpenFreeMap survived 100k requests per second(blog.hyperknot.com) |
I think you perhaps underestimate just how big of a thing this became basically overnight. I mentioned a drawing over my house to a few people and literally everyone instantly knew what I meant without even saying the website. People love /r/place style things every few years, and this having such a big canvas and being on a world map means that there is a lot of space for everyone to draw literally where they live.
Placing pixels gives you points, which you can turn into more pixels or a bigger bag of pixels over time. I've seen people who have done enough pixel pushing that they get 3-4K pixels at a time.
They don’t need to estimate because in the article they talked to the site and got their traffic numbers: An estimated 2 million users.
That’s 1500 requests per user, which implies a lot of scripting is going on.
On the other hand, this is the first I've heard of this thing.
I'm sure there is some botting, it's basically guaranteed, but I wouldn't be surprised if nearly half the traffic was "legitimate". The bots don't normally need to reload (or even load) the map tiles anyways.
Oh absolutely not. I've seen so many autistic people literally just nolifing and also collaborating on huge arts on wplace. It is absolutely not just script kiddies.
> 3 billion requests / 2 million users is an average of 1,500 req/user. A normal user might make 10-20 requests when loading a map, so these are extremely high, scripted use cases.
I don't know about that either. Users don't just load a map, they look all around the place to search for and see a bunch of the art others have made. I don't know how many requests is typical for "exploring a map for hours on end" but I imagine a lot of people are doing just that.
I wouldn't completely discount automation but these usage patterns seem by far not impossible. Especially since wplace didn't expect sudden popularity so they may not have optimized their traffic patterns as much as they could have.
I'm curious what the peak req/s is like. I think it might be just barely within the range supported by benchmark-friendly web servers.
Unless there's some kind of order of magnitude slowdowns due to the nature of the application.
Edit: Looks like about 64 pixels per km (4096 per km^2). At full color uncompressed that's about 8TB to cover the entire earth (thinking long-term!). 10TB box is €20/month from Hetzner. You'd definitely want some caching though ;)
Edit 2: wplace uses 1000x1000 px pngs for the drawing layer. The drawings load instantly, while the map itself is currently very laggy, and some chunks permanently missing.
I managed a few Hetzner cloud instances, and some report perfect uptime for over a year. The ones that don't, I was the root cause.
What exactly leads you to make this sort of claim? Do you actually have any data or are you just running your mouth off?
It's impossible to predict that one's project may go viral.
>As a single user, you broke the service for everyone.
Or you did by not having a high enough fd limit. Blaming sites when using it too much when you advertise there is no limit is not cool. It's not like wplace themselves were maliciously hammering the API.
I understand that my popular service might bring your less popular one to the halt, but please configure it on your end so I know _programmatically_ what its capabilities are.
I host no API without rate-limiting. Additionally, clearly listing usage limits might be a good idea.
Quite entitled expectations for someone using a free and open service to underpin their project.
The requests were coming from distributed clients, not a central API gateway that could respond to rate limiting requests
> I host no API without rate-limiting. Additionally, clearly listing usage limits might be a good idea.
Again, this wasn’t a central, well-behaved client hitting the API from a couple of IPs or with a known API key.
They calculate that per every 1 user of the wlive.place website, they were getting 1500 requests. This implies a lot of botting and scripting.
This is basically load testing the web site at DDoS scale.
The block was done based on URL origin rather than client/token, why wouldn't a rate limiter solution consider the same? For this case (a site which uses the API) it would work perfectly fine. Especially since the bots don't even care about the information from this API so non-site based bots aren't even going to bother to pull the OpenFreeMap tiles.
If you are hitting anything free with more than 10rps (temporarily) you are an taking advantage in my opinion.
After on and off experimentation by various contributors, OSMF just released vector tiles as well: https://operations.osmfoundation.org/policies/vector/
Or, if it’s just a few bad actors, block based on JA4/JA3 fingerprint?
I think referer based limits are better, this way I can ask high users to please choose self-hosting instead of the public instance.
You want to track usage by the site, not the person, because you can ask a site to change usage patterns in a way you can't really ask a site's users. Maybe a per IP limit makes sense too, but you wouldn't want them low enough that it would be effective for something like this.
I guess they'd all rather their customers each upload the 120GB file and then charge them all individually.
If they're crafty they'll have their storage configured so there's only one actual copy on the underlying infra so every other shadow copy is pure profit.
https://github.com/hyperknot/openfreemap/blob/main/docs/asse...
Its a ddos attack, lucky you dont have to pay for the brandwidth, then its a denial of wallet
cloudflare would be pay per request, a hefty sum if ddos happens
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
Most stuff could run there now.
but interesting write-up. If I were a consumer of OpenFreeMap, I would be concerned that such an availability drop was only detected by user reports
Assuming it was close to 100% the rest of the year, that works out to 99.97% over 12 months.
Show us what you have done.
That's how agreements work. If someone says they will sell a hamburger for $5, and another person pays $5 for a hamburger, then they are entitled to a hamburger.
>On a free service.
It's up to the owner to price the service. Being overwhelmed by traffic when there are no limits is not a problem limited only to free services.
> Financially, the plan is to keep renting servers until they cover the bandwidth. I believe it can be self-sustainable if enough people subscribe to the support plans.
Especially since he said Cloudflare is providing the CDN for free... Yes, running the origins costs money, but in most cases, default fd limits are low, and you can push them a lot higher. At some point you'll run into i/o limits, but I think the I/O at the origin seems pretty managable if my napkin math was right.
If the files are all tiny, and the fd limit is the actual bottleneck, there's ways to make that work better too. IMHO, it doesn't make sense to accept a inbound connection if you can't get a fd to read a file for it, so better to limit the concurrent connections and let connections sit in the listen queue and have a short keepalive time out to make sure you're not wasting your fds on idle connections. With no other knowledge, I'd put the connection limit at half the FD limit, assuming the origin server is dedicated for this and serves static files exclusively. But, to be honest, if I set up something like this, I probably wouldn't have thought about FD limits until they got hit, so no big deal ... hopefully whatever I used to monitor would include available fds by default and I'd have noticed, but it's not a default output everywhere.
https://community.nginx.org/t/too-many-open-files-at-1000-re...
Also, the servers were doing 200 Mbps, so I couldn't have kept up _much_ longer, no matter the limits.
> Also, the servers were doing 200 Mbps, so I couldn't have kept up _much_ longer, no matter the limits.
For cost reasons or system overload?
If system overload ... What kind of storage? Are you monitoring disk i/o? What kind of CPU do you have in your system? I used to push almost 10GBps with https on dual E5-2690 [2], but it was a larger file. 2690s were high end, but something more modern will have much better AES acceleration and should do better than 200 Mbps almost regardless of what it is.
[1] to be honest, I'm not sure I understand the intent of open_file_cache... Opening files is usually not that expensive; maybe at hundreds of thousands of rps or if you have a very complex filesystem. PS don't put tens of thousands of files in a directory. Everything works better if you take your ten thousand files and put one hundred files into each of one hundred directories. You can experiment to see what works best with your load, but a tree where you've got N layers of M directories and the last layer has M files is a good plan, 64 <= M <= 256. The goal is keeping the directories compact so searching and editing is effective.
[2] https://www.intel.com/content/www/us/en/products/sku/64596/i...
NVMe disks are incredibly fast and 1k rps is not a lot (IIRC my n100 seems to be capable of ~40k if not for the 1 Gbit NIC bottlenecking). I'd try benchmarking without the tuning options you've got. Like do you actually get 40k concurrent connections from cloudflare? If you have connections to your upstream kept alive (so no constant slow starts), ideally you have numCores workers and they each do one thing at a time, and that's enough to max out your NIC. You only add concurrency if latency prevents you from maxing bandwidth.
Why would that kind of rate cause a problem over time?
> Using our public instance is completely free: there are no limits on the number of map views or requests. There’s no registration, no user database, no API keys, and no cookies. We aim to cover the running costs of our public instance through donations.
> Is commercial usage allowed?
> Yes.
IMHO, reading this and then just using it, makes a lot of sense. Yeah, you could put a cache infront of their CDN, but why, when they said it's all good, no limits, for free?
I might wonder a bit, if I knew the bandwidth it was using, but I might be busy with other stuff if my site went unexpectedly viral.
But it depends on the project architecture. If the tiles are needed only client-side, then there's really little reason to cache things on _my_ server. That would imply I'm better at caching openstreetmap tiles than... openstreetmap. Plus you're just making the system needlessly more complicated.
And there's little reason for openstreet map to be upset, since it's not like _I_ am making 2 million requests - there are 2 million separate users of osm+. It's aligned to their incentive of having more people use osm and derived works. All is well.
Seriously, whose first thought on reading that is “oh great, I can exploit this”.
You are not talking about an insane amount of data if it's 56 Gbit/s. Of course a caching server could handle that.
Source: Has written servers that saturated 40gig (with TLS) on an old quadcore.
Presumably a caching server would be 10GbE, 40GbE, or 100GbE
56Gbit/sec of pre-generated data is definitely something that you can handle from 1 or 2 decent servers, assuming each request doesn't generate a huge number of random disk reads or something
https://news.ycombinator.com/item?id=29651993
https://news.ycombinator.com/item?id=42365295
https://news.ycombinator.com/item?id=44038591
>are you just running your mouth off?
Don't be snarky. Edit out swipes.
(disclaimer/source: I am a CF employee)
And as soon as you have to do ant business / deal with the german side of the business expect everything to slow down to 2 weeks for response which will still be incorrect.
They are simply not worth the hassle. Go with a competent host.
I think that in both cases here (OpenFreeMap and wplace), Hetzner/OVH/Scaleway is the way to go. Depending on what we're doing, the cost savings can even allow us to have redundancy at another cheap provider just in case something goes wrong.
Since I never have to, that’s perfect isn’t it? If you need support from Hetzner you are using the wrong host.
They send nasty letters with pcap dumps of your "hacking" attempts and it doesn't matter how wrong you think they are.
If you need to go to disk to serve large parts of it, it's a different beast. But then again, Netflix was doing 800gig already three years ago (in large part from disk) and they are handicapping themselves by choosing an OS where they need to do significant amounts of the scaling work themselves.
What I've been referring to is the fact that even unlimited 1 Gbps connections can be quite expensive, now try to find a 2x40 gig connection for a reasonable money. That one user generated 200 TB in 24 hours! I have no idea about bandwidth pricing, but I bet it ain't cheap to serve that.
>
> At the moment, I don’t offer SLA guarantees or personalized support.
From the website.
Publicly attacking your users instead of celebrating their success and your new learnings is not what I would call handling it perfectly. I think going for a halo effect strategy where you celebrate how people are using your platform to accomplish their goals will help people understand how what is being done is valuable and want people to adopt it or financially support it. On the other hand attacking people who use your platform publicly can make people apprehensive in using it fearing that they will be criticized too.
This is just someone being not very specific in a text file on their computer. I have many such notes, some of them publicly viewable.
I am disappointed that they edited another guideline for the worse:
> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
It used to just say, don't complain about voting.
If the number of votes are so taboo, why do they even show us the number or user karma (and have a top list)?
They say they're receiving $500/mo in donos and that it's currently just enough to cover their infra costs. Given 300 million 70 KB files, R2 + high cache hit ratio would work out to about $300 in storage-months + request costs, or $600/mo with Cache Reserve and then they'd always hit cache if I understand the project right: meaning the costs shouldn't blow up beyond that, and that request count would essentially just not matter.
A dedicated server will always cost the same so you always know how much you pay.
It will cost 40 Euro/month to have 6 cores/12 threads,64gb of ram and 1Tb of ssd.
Dirt cheap compared to any other alternative
Not that, you know, I often take the time to do that, either - but it would improve the site and the discussions if we all did.
I got nasty letters during the letterbombing attack on Tor relays. They were nasty letters forwarded by Hetzner from the people who actually wrote them. Pointing Hetzner to the page about the attack made them go away.
https://en.wikipedia.org/wiki/False_equivalence
Here's one of several emails they've sent me over the years, each one happening right after running "ipfs daemon": https://0x0.st/Kryd.png
There's lots of other easily-searchable accounts of people having the exact same experience:
https://github.com/ipfs/kubo/issues/4343
https://discuss.ipfs.tech/t/moved-ipfs-node-result-netscan-d...
https://github.com/ipfs/js-ipfs/issues/3982
I may have a hint here - remember, that Nginx was created in the times of dialup was a thing yet and having single Pentium 3 server was a norm (I believe I've seen myself that wwwXXX machines in the Rambler DCs over that time).
So my a bit educated guess here, that saving every syscal was sorta ultimate goal and it was more efficient in terms of at least latency by that times. You may take a look how Nginx parses http methods (GET/POST) to save operations.
Myself I don't remember seeing large benefits of using open_file_cache, but I likely never did a proper perf test here. Say ensure use of sendfile/buffers/TLS termination made much more influence for me on modern (10-15 years old) HW.
Quoting:
> Traffic
>All root servers have a dedicated 1 GBit uplink by default and with it unlimited traffic. Inclusive monthly traffic for servers with 10G uplink is 20TB. There is no bandwidth limitation. We will charge € 1 ($1.20)/TB for overusage.
Huh, archive.org tells me its been unlimited since at least 5 years ago, so I guess I must’ve just seen someone mention 20TB and felt it was a reasonsble limit :)
