Hey all, wanted to share my recent weekend hack project. I wanted to improve upon the existing space for looking up information about IP addresses and ASNs. The backend is written in Rust and pulls BGP data every 8 hours from the RIPE RIS project to build up a routing table and also adds geolocation information. The frontend is using React and Tailwind. Would love any feedback or suggestions on what to improve. |
I wanted to see if I could do the lookup work client-side, and also include some more metadata about cloud provider's IPs (region, service, etc), not that it's really better, just a toy idea I had.
I love the speed of the responses!
We categorize ASN and companies/organizations based on 4 categories: ISP, Education, Hosting and Business. This ASN level categorization are done mainly from WHOIS and other public internet records.
We don't sub-categorize by schools, university, public research institutions, k-12 etc. The reason is accuracy. Even though I can understand the possible methods for doing this, the issue is that it can not be done reliably at scale.
As a data provider, from our end we hope to provide the highest possible accuracy and vouch for the service we provide. For this level of classification, we will generally request users to say what data they need from us, and we try to help them come to a solution that they have to build on their own. They can do whatever classification they want to do based on their personal level of tolerance for accuracy.
Is there any way at all to do this for IPv6?
Additional info that would be helpful would be if this is hosted by one of the big tech companies (i.e. is it on AWS, Azure, etc), though maybe that will show up in the network info?
If the IP is associated with hostnames that would be nice to know. Not sure how much of a one-stop shop you want to be :)
That's a great idea. I'll add a tooltip over some of the less obvious fields describing what they are.
> Is there any way at all to do this for IPv6?
Indeed! It's supported natively, so if you have an IPv6 address you should see it automatically. Here's an example <https://ip.guide/2600::>
> If the IP is associated with hostnames that would be nice to know.
I dig that idea, I'll have to think of the best way to pull that data while keeping the app stateless and fast.
Reverse DNS lookups are usually pretty quick:
```shell
> time dig +short -x 2600:dead::beef
customer.my-isp.net.
dig +short -x 2600:dead::beef 0.00s user 0.01s system 1% cpu 0.882 total
Same, I've always wondered how do websites like this get their information.
At a 30,000ft view, the site works by building an in-memory routing table at boot time that has an entry for every route on the internet and which ASN (autonomous system number) announces it. From there, it stitches together data on the organization that the ASN belongs to, and geolocation data, and then exposes it in the API/UI. Under the hood it's using a treebitmap[1] data structure, which means that it can do _very_ fast lookups when given a particular ip address or ip address range.
Zooming in a little more, the main data source is MRT dumps from the RIPE RIS project[2]. There are a number of routers that RIPE (one of the 5 regional internet registries that hand out ASNs and IPs) that rely on network operators to share their view of every route on the internet. These routers are probably some of the most well-connected routers in the world. For the purposes of this service, MRT dumps are effectively a point-in-time snapshot of the routing table of that router. IP Guide parses that file, rebuilds it in memory, and attaches other relevant data to each IP range (also known as a "prefix").
This allows the service to stay fast and stateless, which was one of the main things I was trying to optimize for when building it.
[1]: https://blog.apnic.net/2021/06/04/storing-and-retrieving-ip-... [2]: https://ris.ripe.net/docs/10_routecollectors.html
I currently do "curl ipinfo.io" which returns json, and seems to use a more accurate geo lookup (based on the hotel wifi I'm currently on). It doesn't report the network though, and the ASN is a string of asn and name, no org or country, so that's a bonus.
However
curl ip.guide
>the ASN is a string of asn and name, no org or country
Yeah detailed ASN information on the API level is available on paid tiers.
However, we have a free IP to ASN Country database that is updated daily and provides full accuracy. That database provides ASN, AS organization, and AS domain information.
I think the reason we don't have more ASN details on the API even though we provide more information for free in the database is because of the target audience and usability of the IP metadata.
Our API allows for tokenless API access for up to 1,000 request/day. There is no compromise with data accuracy. The goal is to be generally useful and extremely fast. Detailed ASN information is still freely accessible from the website, though. So, we have to balance out free API, free website information and paid API/database.
I have been reaching out to folks to adopt the free IP to Country ASN database, as it is free to use and provides full accuracy. If anyone wants to build a public API on top of the IPinfo free database that return more ASN information available there, they are more than welcome to that!
Thank you for this! Much appreciated!
I am sure there are tons of similar programs, but here is one I wrote:
https://github.com/jftuga/ipinfo
I added on a distance column.
Just yesterday, I added this: brew install jftuga/tap/ipinfo
Is there any rate limit for queries?
For example in Germany some carriers just give you a random IP from a pool that is used for half the country, and will give you a new IP every 24 hours. How would you geolocate those with a downloadable database.
No rate limits or auth for requests, but as mentioned below the api starts to load shed at 2k concurrent requests.
Might you open-source this in the future?
Were you already comfortable in Rust or is there a reason you selected Rust for this project i.e performance and so plans to offer this as commercial product?
How often are RIPE info and MaxMind data updated these days or is that the 8 hours?
Thanks for sharing!
I don't think there's a strong reason to keep it closed-source.
> Were you already comfortable in Rust or is there a reason you selected Rust for this project?
This was a big learning opportunity for me to learn Rust. I had done two small projects in Rust before this one.
> How often are RIPE info and MaxMind data updated these days?
The MRT dumps where it gets its BGP data are updated every 8 hours, and MaxMind appears to be updated twice a week.
I provide a similar service but with more focus on bot/data center/VPN ip addresses.
Yours is very cool by pulling BGP data every 8 hours though!
I actually have a use case for this in my product Loginllama. I need to grab information about the IP addresses. I’m currently using a different API but don’t really like the product.
Is it rate limited or have any key authentication? My email is me at joshghent.com if you want to chat about this more.
LoginLlama looks super helpful, what else would you like to see from IP Guide to help? Would a boolean attribute on whether or not the IP is a known tor exit node be helpful?
curl -sS https://am.i.mullvad.net/json | jqalso s/Tailscale/Tailwind/
> (iwr ip.guide).Content | ConvertFrom-Json | Select-Object locationThe site is hosted on Neptune Networks, another side project of mine for people to learn and experiment with BGP.
Not affiliated, it’s just my current preferred.
You can try our ASN lookup tool: https://www.bigdatacloud.com/asn-lookup/
It provide wide variety of data objects, and also provide map view where possible to showcase the active region of operation ( its a unique feature of the API), plus heaps of data. You can test the API for free and access all the data objects.
Any doubts or queries, please let me know.
If you can download the free IPinfo IP to ASN database CSV file. You can run the following command to get the IP ranges of an ASN.
```
grep AS19551 asn.csv | ipinfo range2cidr | cut -f1 -d ','
```
You will need to install the IPinfo CLI to convert the IP range to their CIDR equivalent. I believe the range2cidr command is also available as a standalone binary as well.
After that, you might want to aggregate the IP ranges to bigger ranges. I have used this in the past for this:
curl -sL ip.guide/AS19551 | jq .routesI'm currently using ipset, that does aggregation if I remember correctly, but aggregate6 looks good!
There's even more under the hood, worth checking it out.
(Disclaimer: I work for the company that develops IPNetDB).
$ curl -sL ip.guide/bulk/networks.csv | head -n 2
prefix,asn,organization,country
1.0.0.0/24,13335,"Cloudflare, Inc.",USWould it be possible to allow access token usage in the tool, please?
https://news.ycombinator.com/formatdoc
"Text after a blank line that is indented by two or more spaces is reproduced verbatim. (This is intended for code.)"
I spend all day writing markdown and fenced blocks with type-hint is muscle memory at this point
I don't see any way to report it though.
I forgot to add the link in my original comment. But as you mentioned we are showing VPN as true, that indicates a very interesting situation. Can you ping our support, please?
If you can also mention that, you came from HN, that will be fantastic. We will open an engineering ticket first thing Monday and look closely into this. The situation is unusual, and I am sorry about the inconvenience.
https://dev.maxmind.com/geoip/geolite2-free-geolocation-data