AWS and their billions in IPv4 addresses(toonk.io) |
AWS and their billions in IPv4 addresses(toonk.io) |
There supposedly was an agreement that Amazon wouldn't start provisioning them for a certain amount of time, but whatever amount of time was specified, they either didn't honor it, or it wasn't enough time.
We were also moving assets over to AWS, and all of these things going on simultaneously caused what we called the three-pocalypse.
We would occasionally run across issues with external sites or newly provisioned lambdas who were on Amazon's new 3.0.0.0/8 block, but we couldn't reach them because internally that IP address didn't exist.
At the same time, they would open up a small block to allow access to those external sites, and then some internal service would no longer respond. Repeat ad nauseam. It was also compounded by the fact that there are countless teams in GE and not everyone would connect with who made what changes.
On a separate note I had hoped for some time that an AWS 18.x.x.x address would be useful to get access to journal papers but I tried and that sadly didn't work :(
Knowing what they say about both companies, it was probably both reasons at the same time :-))
Also, in year 2018 GE could have been ready for IPv6 but that would require not only existing IPAM but also some proper leadership, which based on your words GE doesn't seem to have.
It also made network architecture easy. Every MIT building got its own B class subnet. Hilariously, even the MIT boat house which couldn't have had more than a dozen computers online at once had 65,534 addresses. It made it very easy to find out which building someone was in based on their IP.
IPv4 scarcity only hits in certain segments. It doesn’t hit the major players.
The real problem is the hoarding by the major players. AWS will most likely never use that many IP addresses. I know that sounds like a stretch based on what we’re led to believe about running out of IPv4 addresses. I don’t know their IPv4 utilization, so it’s a guess. My guess is based on how we deploy internet facing services these days. It’s less dependent on needing lots of public IPs.
Besides the direct technical benefits of having so many addresses Amazon also potentially holds billions worth of assets that will eventually devalue and devalue quite extremely that they could write off for tax purposes.
IPv4 addresses will continue to appreciate in value until their collapse which means that Amazon might be able to write off tenfold than what they paid to acquire them in the first place.
Update: I just found http://www.circleid.com/posts/20180731_the_ipv4_market_2018_... which says
> As we head into the last half of 2018, small blocks are selling for just over $18/number, mid-blocks are in the $15-$18/number range, and large blocks have surpassed the $20/number threshold
so it seems there's a bit of an U shape, where mid-sized blocks have the lowest price per address.
AWS paid $108 million for this /10. That’s $25.74 per IP address.
AFAIU, IANA will only transfer blocks to a business entity, so if you're serious about it you might have to first start with that.
Here, this one's an easy one to remember, you can have it:
127.1.
Edit: it's also an easy way to outcompete others; IPv4 space has become a huge capex for starting a cloud provider.
Remember the IE6 banners? There will be a time in ten years (haha it's always ten years!) when you'll see "Our site runs with degraded performance over IPv4. Please contact your administrator."
For whom? There are lots of providers (for example Huawei) that offer turnkey solutions for ISPs to painlessly roll out CGNAT.
It also makes it much more difficult for customers to stress your upload bandwidth though.
So yes, Amazon hoard IPv4 with intention to maybe use them in the future. But hording it is, and in a non-hoard system they'd rent - not buy - IPv4 with a month notice or so.
Check out CIDR Blocks for a more complete answer: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing...
Checkout the table on the page (scroll down)
Can you expand on this a bit? I'm not a networking guru but it's my understanding that an IP address is an IP address, how could it be used more or less efficient?
Why would any significant fraction of these servers require public IPs? And services for that matter.
I wouldn't be surprised if most of them are in private VPCs with only very few endpoints exposed.
If you are referring specifically to AWS, having servers in public subnets is actually an anti-pattern. You may want to do so with bastion hosts and a handful of more specialized services. For everything else, put them behind a load balancer. A single NLB will take one IP per availability zone and will be able to service hundreds of servers, if not more.
Then you have things like Cloudfront and the like.
Not many IPs are needed, overall, compared to the number of actual servers.
My ISP only gives CGNAT or is happy to rent IPv4 for a premium, one address at a time. Keeping IPv6 out helps the IPv4 margins and staves off competition.
Are you sure about that? A comment here said that they had 1.4 million servers in 2014, and AWS at the time was probably 1/50th of what it is now. On top of that, I think they have some IoT offerings, and who knows what they're going to offer in the future? They're basically building an internet-sized platform to run half the internet on top of it.
But you'd have paid X for a reduction of X in tax liability, which, assuming you pay a sub-100% marginal tax rate, is a strictly dumb idea.
It’s ugly but it’s kind of the sort of system that everyone is incentivized to move to. Which kind of makes me sad.
IPv6 is backwards compatible so why would we "finally roll off IPv4"? IPv6 has been "here" in production for over a decade, and has seen slow adoption. I suspect it may be another 8-10 years until IPv6 gets close to a majority, but wouldn't IPv4 still be in-use?
What would force a roll off of IPv4?
Either way, I too would like to know more about how IPv4 space has appreciated/depreciated in the past decade
15-20 years ago IPv6 routing was often tunneled over v4, and it was noticeably less reliable than v4. Nowadays v6 is a touch faster/more reliable than v4, not very much. If that difference widens much and most users have v6 anyway, then v4 might get the axe. Otherwise... I don't think so.
FWIW I already operate some v6-only services. Not for the general audience. An SSH bastion host, a backup service, that kind of thing, not web sites for the general audience. I'm not in the least surprised if some warez topsite is v6-only already.
It would be really really great if true, unfortunately this is not correct.
IPv6 SHOULD have been this way, but The Powers That Be took it as an opportunity to "correct" other IPv4 issues and now we have what we have.
So long as the internet keeps working, my isp won’t care. I set up a HE ipv6 bridge, but it adds noticeable latency whenever it’s used, for sites like YouTube and Netflix.
I wonder if we need regulation to force the transition. The move to v6 might never complete otherwise.
Amazon could perhaps do with less IPv4 addresses, if people did misuse them. I work with a client who have a public IPv4 address associated with every single EC2 instance they have, despite only 5% of them have public facing services. They just got in the habit of assigning a public IP I guess.
- No outbound internet access
- IPv6-only outbound internet access
- NAT, for an addition monthly and per-GB fee
Given you can assign a public IPv4 address at no additional cost and have everything just work, there's little reason not to have one.
Also, NAT gateways cost money in AWS, so much that it is a running joke:
https://twitter.com/QuinnyPig/status/1294047698560012289
https://twitter.com/QuinnyPig/status/1293366642567651330
NAT doesn't add any additional security, Security Groups are fantastic at allowing you define your ingress/egress between instances and protecting them from harm.
All my instances get an IPv4 address an an IPv6 address by default so that there is parity. The fact that the IPv4 address still goes through some sort of NAT on AWS's side (1:1 but still NAT) kind of bothers me.
Google/Nest wifi did a good job of just making IPv6 enabled by default for all consumers.
Which one would you choose any why?
Especially if that one with 35% of my customers provides me with lower latency, higher throughput and costs me less in CPU time/power to run my traffic across.
The rest of the people I need to eat the cost for...
Also, with the increasing numbers of devices connected everyday, we're running out of IPv4. Think of the demand vs supply curve (demand high, supply low, result = higher price/ip)
You can check IPv6 adoption in each country here: https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...
[1] https://www.google.com/intl/en/ipv6/statistics.html#tab=ipv6...
More precisely, the repos; When you activate IPv6 on Debian, then apt-get (the package manager) is extremely slow. This is because it first tries to reach a repo in IPv6, then after 30s falls back to IPv4. If you disable IPv4, it is lightening fast. Many services behave the same way, to the point that computers are generally faster on IPv4.
Maybe it changed recently but it wasn’t the case for the last 10 years and I’ve quit trying, and I’m not knowledgeable enough to configure the Debian system far from the defaults.
Edit: Maybe it is my ISPs who don’t support IPv6, which makes it hard to improve because the problem is invisible for, for example, Debian developers who work on IPv6 support.
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1My local network and ISP are perfectly capable of using IPv6 but you have to call them to switch.
Devices, you will be surprised a lot of devices that does not have support for IPv6.
Used it at a previous org I worked for. Pretty nice - it can crawl your routers, build up your existing network allocations then help you analyze/optimize them.
Stupid powerful - we used it mostly for decentralized DNS management (!!) but after a while I finally got the network and security folks to realize what the system could do and the spreadsheets started to finally go away.
Speaking of decentralized management it has robust hierarchical role-based security - we had thousands of site admins managing DNS and eventually IP addresses for their individual sites, but also smoothly maintaining overall command-control. A very cool system.
We used that particular product extensively at my previous company. If you stole an ip without putting it into Orion there was a job that would enumerate/update info, if possible.
If the job failed, you and your boss would get an angry email.
This was in a company probably much smaller than F500.
The current situation is that the ISP can lessen load on their CGNAT solutions by providing IP6.
Disclaimer: I lived in the county over 30 years and still have strong links.
https://teamarin.net/2019/04/03/microsoft-works-toward-ipv6-...
But AFAIK that is still aspirational. I would be pleasantly shocked and impressed if they had already achieved that.
Things change...
Sure, if you have a tiny deployment you may not care (and the NAT fees may be a significant portion of that).
At some point, the NAT fees are noise - it amounts to ~ a dollar per day in us-west-2. Data processing charge is $0.045
It becomes way more valuable to ensure IT security, regulators and auditors that no, no inbound connections are allowed no matter what anyone does with the security group rules.
Also note that the AWS managed NAT gateways haven't been there forever. The option, before they were available, was to use one or more of your instances to NAT traffic. That's still available and could be an alternative, while reducing your potential footprint.
Also, I think the concept of a "private network" is inflexible and in some sense a premature optimization. If you use unique IPs you can decide on a subnet or even host basis what is exposed to the Internet and what isn't.
(Genuinely interested in hearing war stories from the front on this)
That's just for IPv6 support, which is beta, and took years to get there. Dual Stack is still in alpha, and that didn't appear until a really recent release (1.16).
I think routers and switches are probably the easier pieces of the puzzle, at least as far as managing them goes. The vendors have worked out all the kinks (hopefully?) before the equipment gets to you.
That's just to get the stuff to work. Then you have corporate/government policies and validation. Then you have to solve problems like "My VM resolves things to IPv6 by default, but I have no IPv6 gateway so everything times out". And then make sure that logic makes it up through your entire stack. Multicast? Not allowed on many networks.
Is it the end of the world? No. It's just a lot of extra work for everyone.
Someday, hopefully!
/Not working in networking
the high v6 penetration in mobile networks likewise suggests that smartphones are generally not a part of the network as much as dedicated leaf nodes that require no inbound connections.
I think Facebook is doing IPv4+IPv6 at the edge, and IPv6-only internally.
There's no translation layer needed.
Did you mean to compare Tor to the World Wide Web?
They are different protocols but they are not disjoint address spaces.
You add 96 bits to the address and boom it's converted.
Why does that bother you?
Also, if I have multiple IP's with EIP's attached so I can host multiple services (with unique IP's) I have to write automation to make sure I bind the service to the right internal private IP for the appropriate external IP address. It'd be much better if the IP address were routed directly to my EC2 instance.
Isn't that done in a more straightforward fashion by AWS loadbalancers? AWS load balancer IPs and ports on one side, listeners on the other side talking to your instances - if the instances are also in auto-scaling groups, there's zero automation needed after you set this up.
How would it have possibly been backwards compatible? Plenty of routers and ip-aware switches have, in hardware, specified that ips are 32-bits, so anything that added more bits would necessarily break existing hardware, and thus not be backwards compatible.
That's not to mention all the software that has similarly hardcoded the number of bytes in an ip.
How could we possibly have made ipv6 backwards compatible?
That isn't to say IPv6 couldn't have been done in a backward compatible way. I can think of ways to do that, and dozens of pros and cons - even though I haven't been in networking for 20 years and so I've forgotten a lot.
For hardware, that doesn't sound that different from not supporting it. Convincing end-users of internet hardware to update their switch's firmware is hard. Ubiquiti has done a pretty good job of making updates actually doable, but for most hardware I doubt it'd happen more quickly than the hardware itself would fail.
> That isn't to say IPv6 couldn't have been done in a backward compatible way. I can think of ways to do that
Can you explain any of those ways?
But it's technically quite ugly, and tunnelling/MTU-related problems would be more prevalent.
6rd works on similar principles, but in a way that plays more nicely with ISP routing and traffic management policies. The main user-visible change is that a variable-length, ISP-specific prefix is used instead of 2002::/16.
There are a few other transition mechanisms and embeddings of the IPv4 address space, for example the ::a.b.c.d address format. They all suffer from the fact that two-way communication without a proxy requires each endpoint to be capable of representing the full address of its peer. For a node which does not have a public IPv4 address to communicate with a node which does not understand larger addresses, some third node must exist in between to perform protocol translation.
Easy, just declare that the entire IPv4 address space is ::x.x.x.x in IPv6 and Bob’s your uncle. No idea why they didn’t do this other than sheer solipsism.
They did [0,1]:
The "IPv4-Compatible IPv6 address" was defined to assist in the IPv6
transition. The format of the "IPv4-Compatible IPv6 address" is as
follows:
| 80 bits | 16 | 32 bits |
+--------------------------------------+--------------------------+
|0000..............................0000|0000| IPv4 address |
+--------------------------------------+----+---------------------+
Note: The IPv4 address used in the "IPv4-Compatible IPv6 address"
must be a globally-unique IPv4 unicast address.
The "IPv4-Compatible IPv6 address" is now deprecated because the
current IPv6 transition mechanisms no longer use these addresses.
New or updated implementations are not required to support this
address type.
[0]: https://tools.ietf.org/html/rfc4291#section-2.5.5.1
[1]: https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresse...
HN discussion: https://news.ycombinator.com/item?id=10854570
A /24 has:
- .0 - network address - can't be used
- .255 - broadcast
On top of that you generally need at least 1 more IP address that is the gateway for that network.
- .1 - Usually
If you have a network with fail-over gateways generally you need to assign them individual IP's, so you end up with:
- .1 - floating IP
- .2 - router 1
- .3 - router 2
If you end up subnetting down into small subnets to give customers only let's say 16 IP's, (so a /28 (32 bits - 4 bits)) customers can only use 13 of those addresses (network/broadcast/gateway are already taken).
This gets worse as you go smaller, because each time you subnet you end up losing more IP's to the network/broadcast/gateway.
The rest of the network doesn't care what they look like at certain scopes. Those devices just know to route it to the next hop, and when that next-hop is your box with a bunch of interfaces/aliases/whatever configured, it'll just handle it.
Try it, you'll like it.
(Edit: IPv4 here, hence /32. No such foolery needed with v6.)
You can use /31's for point to point if both devices support it, but it's still hit or miss whether they do or not.
Even when getting a block of IP's from providers to a customer edge (think ISP's like Comcast or others) they tend to require using the .1 as the gateway, and the others are considered on-link and thus there is a network/broadcast address.
Your suggestion doesn't allow direct L2 communication between machines without sending traffic to the router, even if the two systems are L2 adjacent.
Even with v6 you'd want to assign multiple IP's to the same interface, except that instead of getting /128's routed to a machine you just a get a single /64 and you can use any of the addresses out of that range.
10.0.1.0/32
As a perfectly valid IP address.
Also, routing an entire block allows you to use all of the IP's in said block. So there are ways to do it efficiently, but with routing a block that block is not considered on-network and thus doesn't have a broadcast address nor can it use one.
Since this is about AWS... they don't do that. They will assign a random IP out of their available public IP for every resource that needs an IP. For many of these you can allocate an IP and move it around if you want (that's why they call it 'Elastic IPs').
Unless you have special needs (and actually own the IP ranges) your VPCs will all be in a private IP range. You can even have multiple VPCs with the same range with zero issues (unless you want to peer them).
Each network needs at least one gateway inside that network to reach the rest of the world, so that IP address is lost. Conventionally you also lose the lowest and highest IP address in a block to the network number and broadcast address (you can free those up, but it might confuse devices or admins)
So for every split or subdivision you make, you lose at least 3 addresses.
Conventionally you also split a network at a specific bit boundary (CIDR) so each network has 2^n IP addresses in it. If you went for n=4, you have 16 addresses and already lost 3 to the above factors, so now you have 13 addresses left.
If you happened to have 14 machines attached to that network, you need a network with 2^5 = 32 IP addresses, and you've got 15 unused IP addresses with no way to give them back to a higher level.
All quite similar to why a phone area code may not be able to make use of all phone numbers inside that area, but have no way real practical way to give them to other networks.
Pretty sure that we’ll never see routing tables with 4 billion entries. So it has to end somewhere.
Applications would still need to change, but apps change a lot faster than ISPs.
6to4 died because it was never the primary mode of addressing. Performance between 2002::/16 and the rest of IPv6 relied on anycast, which was unpredictable and therefore useless for real traffic.
6rd, fortunately, has none of these drawbacks, though it does require support from the ISPs.