I made a simple geolocation service(maxkostinevich.com) |
I made a simple geolocation service(maxkostinevich.com) |
IP to country is fairly easy and I open sourced all the scripts and the database itself [0].
But IP to city is much harder, I'm not actually sure it's viable for anyone to do that without relying on some other 3rd party service.
I'd be very interested to hear if anyone knows how to pull that off in an open sourceable manner.
That law will affect open source solutions just as much.
> Aggregate and anonymous data is exempt from the CCPA, unless it is in any way re-identifiable. https://www.cookiebot.com/en/what-is-ccpa/
Would this be cheaper than running these services?
easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory.
Like if you know that 99% of all ips in a certain range are from US, then only store the 1% as a list.
I guess it’s ok for a basic consumer website, although it’s not exactly equivalent to ip geocoder databases/services - they allow passing ip addresses as part of geocoding request.
https://developers.cloudflare.com/workers/templates/pages/co...
AFAIK, an unresolved problem is a proper geolocation service - at least for city level resolution - for mobile IP addresses. There are some services in this field (digital element), but they are very unreliable.
It doesn't seem to have the same problem with Hawaii for some reason, they're all just US from what I can see.
Nginx + GeoIP2 module sets a HTTP request header and proxies the request to the golang app.
The go app does a lookup from redis, based a combination of country header and some url params and responds back with a redirect header.
Hosted it on a t2 medium in EC2 and I have seen it easily handle ~1500 requests / second without any issue.
"Application error An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command heroku logs --tail"
GCS for example and its the same with S3, costs space, egress and requests themself.
Data Storage 50 GB Standard Storage * $0.026 per GB $1.30 Network 10 GB egress * $0.12 per GB $1.20 Operations 10,000 Class A operations * $0.05 per 10,000 operations $0.05 Operations 50,000 Class B operations * $0.004 per 10,000 operations $0.02
I run ifconfig.io which now gets just over a billion hits a day. It is basically an echo service, and it just parrots back what cloudflare tells it. But since I run it on linode, it costs me a whopping $40 dollars a month for 35 billion returns a month. Using workers would cost me several thousand dollars.
It's a nice solution. Why log into your alt account to crap on it?
I'm trying to learn to take it as the highest form of compliment - when hackers care enough to hack on one of my ideas, that means it has to be cool.
https://blip.runway7.net https://github.com/runway7/blip
I lost track of the consumer apps I've used this on and still haven't received a $1+ bill.
"time_zone": {
"name": "America/Chicago",
"abbr": "CDT",
"offset": "-0500",
"is_dst": true,
"current_time": "2020-07-25T06:10:16.945136-05:00"
}Moment.js is probably the most prevalent time and date library.
Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config language is even simpler. It directly executes on every CDN edge node worldwide upon request.
For example, I just whipped this up on Fastly using VCL. It returns GeoIP as json data for your IP at the root path:
Or if you want a particular IP, just append it to the path:
http://geo.zombe.es/2a04:4e42:600::313
You could do the same via query params, headers, etc. Have URL endpoints that only return some of the data, and so forth.
The VCL syntax gets a little gross when you handle quoting strings and assembling json and testing if the string is empty, but it gets the job done.
Of course what you might want from GeoIP data may not be what you get. It's really kind of a useful kludge that gets treated sometimes as a panacea.
This dataset right now thinks that I'm about 5 miles east of my location, but when subnets are repurposed it could be much more significant. And the data sources are always changing, so who knows what it will think tomorrow.
It's very borderline from them (at least ethically)
The response time averages 10ms and is free, granted I'm not sure if there is a ToS or anything attached to this endpoint, I've only found non-conclusive discussion here https://community.cloudflare.com/t/what-are-the-terms-of-use...
I'd advise you to just create a Cloudflare Worker if you want do this. You'll get more reliability etc. and Workers have a very generous free tier.
It's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?
I got sent a (frustratingly incorrect) bot reply to a ticket and a reminder that Enterprise / Business / Pro customers are priority (in that order) even though I pay for Workers. It has been an uphill battle to get someone to take a look at the ticket so far. Thankfully, we haven't gone to production yet, but as a consequence, now need to plan to add mitigation in scenarios where Workers blacks-out our traffic (but Support can't be of immediate help because "free customer").
[0] https://community.cloudflare.com/t/workers-and-sub-requests/...
How is it that you prefer to engineer those workarounds instead of just paying $20/month? It surely can't be worth in engineering hours and ongoing maintenance of hacky workaround code?
If it isn't clear, there's an issue of trust here, not money. And frankly, a fallback shouldn't be much of an issue since fly.io, AWS Lambda@Edge are like-for-like swaps, so we'd still be "serverless" just on a different yet similar platform (albeit costlier).
If the exact location is important geoip is not accurate enough anyway. Forwarding to regional sites automatically is just annoying when it doesn't work properly or someone is traveling abroad.
https://arstechnica.com/tech-policy/2016/08/kansas-couple-su...
https://www.theguardian.com/technology/2016/aug/09/maxmind-m...
https://mashable.com/2016/08/11/ip-addresses-kansas/
ISP perspective here: Geolocation by granular /24 to /20 sized block of ipv4 space is often wildly inaccurate on a regional basis. It's entirely possible for the ARIN registration (used by maxmind) to be a street address in Seattle, but serving end user ISP customers a 4.5 hour drive away in a far eastern corner of WA state.
I've been meaning to make automatic Github releases for it.
https://www.united-coders.com/christian-harms/detailed-perfo...
It's also free and fast but with services like these I always wonder how long they manage to stay free.
I had good success with maxmind's free database. I can't find tooth results for accuracy but my sampling was pretty good. They also have an incentive to keep it.
6 million requests per month is only 2.2 requests per second, a raspberry PI could do this (technically).
Now I have been thinking about opening it up for more people because while there are a variety of these services out there, one more does not hurt — it already exists, and works, so who knows, maybe more people would like to use it. The code is open, and access logs go to /dev/null; I could probably add a read-only SSH user for people to confirm that for themselves.
Geolocation could easily be added to it, but then comes my question: is there any use-case for geolocation APIs that does not involve tracking users for shitty purposes?
I was excited about adding this feature because I am using pure NGINX for it, and it was a fun learning experience, but I asked myself that question when I started writing the documentation for the website, and I still do not have an answer. Marketing material for other APIs that offer geolocation usually have user tracking as a selling point.
Personally, I have no use for geolocation, and if all use-cases involve tracking without consent users, and breaking their privacy, I want no part in that.
[1] https://developers.cloudflare.com/workers/reference/apis/req...
[2] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
[3] https://cloud.google.com/appengine/docs/standard/go/referenc...
There's nothing stopping a script-kiddie from thundering away at the Serverless endpoint resulting in an unexpected and quite high billables [0].
As for Workers specifically, Cloudflare's rate-limiting plan makes the whole thing 10x expensive at $10 for 2,000,000 good-requests [1] + $1 for 2,000,000 Workers requests. Other cloud providers I don't think fare any different.
[0] https://community.cloudflare.com/t/how-to-protect-cloudflare...
[1] https://support.cloudflare.com/hc/en-us/articles/11500027224...
> on average the response took somewhere between from 200ms to 500ms
I'm getting latency of 66ms to 126ms with some simple Java code running on AWS Lambda using provisioned concurrency. I find the latency is just fine for most use cases.
What's the point of using Lambda if you have to provision capacity for it?
You pay a premium on the compute because it's able to scale down to zero (so in theory, it works out cheaper anyway), but in reality it only works well when you do everything in your power to have it not scale down to zero.
Furthermore, Maxmind adds non-factual entries to the database so they can identify it uniquely as their work. Map makers do something similar: https://en.wikipedia.org/wiki/Trap_street
I realise this doesn't directly answer your question but I guess one reason might be that they already provide a free version so releasing the paid version would just get you in strife?
"client": {
"conn_speed": "broadband",
"conn_type": "wifi",
"proxy_description": "vpn",
"proxy_type": "hosting"
conn_speed: https://developer.fastly.com/reference/vcl/variables/geoloca...
conn_type: https://developer.fastly.com/reference/vcl/variables/geoloca...
proxy_desc: https://developer.fastly.com/reference/vcl/variables/geoloca...
proxy_type: https://developer.fastly.com/reference/vcl/variables/geoloca...
conn_type is interesting to me, I'm not sure how you would distinguish wifi vs. wired based on HTTP header data.
vcl_recv and vcl_error are the important bits, the rest is VCL boilerplate from https://developer.fastly.com/learning/vcl/using/ for unused subroutines.
However, there's a lot of use cases that semi-accurate geo-locations make sense. The first of which is analytics. If I'm a marketing person at a SaaS company, I want to know where my customers physically are if possible for a variety of useful reasons.
Good quality estimates for location also help with security and compliance use cases as well. If a user logs in from a new country on the opposite of the world, you can flag that and take whatever action you want whether it's to block them, fire off an email to the person who owns the account or whatever.
I hope you're factoring in error rates. Because right now Google, with its billions of dollars to spend on geolocation, tells me that my laptop is in Albuquerque, and my phone is in Los Angeles. Neither are within 500 miles of either.
What reasons? What makes you feel entitled to that information if not volunteered?
Of course, you can just get around this with a VPN.
There are many many cases where it’s relevant. Others have mentioned prepopulating or ease of use aspects. But you’re making big assumptions that the client has an interactive user at all. It’s important where you don’t own the client or there is defined behavior, say at a protocol behavior, that must be respected. Moving earlier than that there are cases where you need to route traffic before any sort of connection is even established.
I suspect you’re also underestimating geolocation precision and accuracy. Free datasets will get you the right state or city say 90-95% of the time. Cheap datasets get you to the right town or post code 95-97% of the time. Expensive or bespoke multi source datasets will get you reasonable post code, neighborhood, or even household & address accuracy. Think of cross correlating order histories per physical address, device fingerprints, and IP addresses/triangulation.
I work with an event org with a fairly common name, and another city in our region has an event with the same name. For some years, we’ve outranked them for major keywords and getting their confused customers was becoming a headache for us.
So now if our website thinks you’re in the other city, a little banner appears above the content asking if you’re looking for the other site, and offers the link.
This has saved us ~10 phone calls per day in season. Before adding this link we all got to the point of just doing their customer service for them and helping people buy tickets because it was less drama than trying to get them to call the right people.
But in any case, your response does tell me that there are other things that can be done with that information, so maybe I should focus on the IP thing, and have the geolocation features kinda hidden away in the documentation pages.
Use my GEOIP and longitude and adapt your CSS to show me your content in nightmode when it is night time in my Longitude. That would be cool.
When I build websites, I usually prefer using `prefers-color-scheme: dark` in CSS, so the user can decide.
What would you cache it cheaper?
No matter what his worker needs the smallest time/memory slot there is. You cannot make it cheaper while still using workers.
the stuff in the "client.geo.<key>" space is from a newer/better/higher-tier service (they say, i forget the name). also I think some of it is mixed in with other sources and some of the info is self-sourced.
I should write a blog post about it. It has steadily gained more traffic over the years and hasn't need much care and feeding.
The go code can be found here if you're interested, though it some of the ugliest code I've written. I made it when ifconfig.me was having load issues many years ago.
I do have the service behind CloudFlare, which is essentially a reverse proxy. The reason is for CloudFlare is non-obvious; it is deserves a blog post, and that is connection pooling.
If I have all the requests go back to the origin, the bottle neck is not the go code, but Linux opening and closing all those single use TCP sessions. CloudFlare creates around 100k persistent connections to the backend, but then just keeps them open. This makes Linux much happier.
The high connection count actually made the service unstable once: https://github.com/georgyo/ifconfig.io/issues/2
With PaaS it's someone else's job who does this every day.
I’d like to set a system clock timezone via geoip only, without location lookup via wifi. On my more secure systems I have location services disabled and it’d be nice to have an accurate local time automatically.
So "cloud" is like renting a dynamically changing number of VPSs each month and which are usually more costly if your consumption is not spiky (eg. when a cheap VPS rented 24/7 is enough).
I need to be able to send a lat/long/timestamp to get tinezone.
I’ve seen timezone maps drawn, so this information does exist somewhere, but one would have to draw polygons for each timezone and do coverage checks, which seems complicated. The time zone lines zig and zag a lot.
If under 10 million requests per month, serverless will be cheaper and safer
Else if under 1 billion request per month, managed VPS will be cheaper
If over hundred of billions of requests per month, anything else than on premise is probably an error.
Then you have grey areas where you need to process.
But for most small project, you cannot beat the lambda prices. Especially if you expect bursts.
it sounds lot cooler to say that " i made geolocation service using serverless/lamba" than "i made geolocation service using boring monolithic technology". where the former takes 150 hours to get it right and painful to add/manage features. where the later is very easy to maintain and very predictable priced.
In my last gig, we ran plenty of services that would never need to scale and would likely never hit the point where Lambda became too expensive. Low-utilization services (Example for order submit, because we only ever got a couple hundred orders per day), Cron-type jobs (Admins could use Cloudwatch to monitor), Temporary fan-outs using queues. Anything that won't ever hit the millions-per-day level is a good candidate for something along these lines. Most of those services cost <$5/month.
[1] https://www.troyhunt.com/serverless-to-the-max-doing-big-thi...
32 million requests can be easily served with $50/month heroku node.
What I want to hear is how you argue serverless is “snake oil” when evidence says it’s massively cheaper even at scale. Less than $1/mo to serve 141m requests. How is that snake oil when even you say an equivalent “serverfull” solution costs 50x more?
1. You are saving $49 month by spending 250+ hours engineering hours to structure the application which is near impossible to extend/maintain.
2. The $49/month subsidy (free tier) you are getting is temporary and will eventually go away (think about google maps api price change).
3. 50x cost for serverfull solution is not a right number. lets say you have to handle 500M requests per month (around 200 QPS) the 50x becomes .000X.
4. If there is any sudden spike to popularity, you are screwed. you will end up paying several thousand $ per month.
I think I’ve heard enough to say you have no idea what you’re talking about and this conversation has become pointless.