Ask HN: Is Apple down? https://developer.apple.com doesn't work
App Store doesn't work
iMessage doesn't work.
Not just me - coworkers also struggling. Any idea what's going on? |
Ask HN: Is Apple down? https://developer.apple.com doesn't work
App Store doesn't work
iMessage doesn't work.
Not just me - coworkers also struggling. Any idea what's going on? |
Must be gravity. (Sorry, I had to.)
https://earthsky.org/sun/sun-activity-solar-flares-cme-week-...
LPDDR5 in the SoC doesn’t.
nslookup
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer.apple.com
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
More:
nslookup
> developer-cdn.apple.com.akadns.net
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
> server a.ns.apple.com
Default server: a.ns.apple.com
Address: 2620:149:ae0::53#53
Default server: a.ns.apple.com
Address: 17.253.200.1#53
> developer-cdn.apple.com.akadns.net
Server: a.ns.apple.com
Address: 2620:149:ae0::53#53
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.
Yes:
developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.
They fixed that and now it's back up.
This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).
> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.
Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.
It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.
The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.
$ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
Server: a.ns.apple.com
Address: 17.253.200.1#53
** server can't find developer-cdn.apple.com.akadns.net: REFUSED
$ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
Server: 1.1.1.1
Address: 1.1.1.1#53
Non-authoritative answer:
developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
Name: world-gen.g.aaplimg.com
Address: 17.253.121.201
Name: world-gen.g.aaplimg.com
Address: 17.253.121.202https://puck.nether.net/pipermail/outages-discussion/2022-Ma...
https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/
It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing.
lima has an anaylsis of the issue causing trouble:
Wife: My Apple Maps isn't working.
Me: Hmm, it's not working for me either. They must be having server problems. You should use Google Maps for now.
Wife: I can't download Google Maps either, the App Store doesn't seem to be working.
Looks like I really need to keep a 3rd party nav app installed just in case!
> App Store - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing intermittent issues with this service.
Apple Arcade - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Apple Music - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Apple TV+ - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing a problem with Apple TV+. We are investigating this issue.
iTunes Store - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Podcasts - Outage Today, 12:32 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating and will update the status as more information becomes available.
Radio - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.
Apple Business Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.
Apple School Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.
Device Enrollment Program - Outage Today, 1:14 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating this issue.
Schoolwork - Outage Today, 1:14 PM - ongoing Some users are affected This service may be slow or unavailable.
The link is currently not working...
- "Multiple Apple services are down such as: (Will be updating this list)"
https://old.reddit.com/r/apple/comments/tjg8tz/megathread_ap... ("[Megathread] Apple Outages")
Edit: It’s also refusing to download any apps, doesn’t even show the progress circle. Just a download icon next to the app name on the Home Screen and errors out when you click it.
Edit: Login and app downloads now working as of 6.00GMT
I often wondered how medieval the world would become if there was a huge sun flare ejection that breached the magnetic field and destroyed a bunch of data-centers. Think of the mess we'd be in!
$ dig -t NS developer.apple.com
[...]
apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.
$ dig @a.gslb.aaplimg.com developer.apple.com
[...]
;; ->>HEADER<<- opcode: QUERY, status: REFUSED
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; WARNING: recursion requested but not available
Unlikely to be BGP shenanigans as some people on Twitter claim. My network has direct peerings to Apple's AS714.
For example, when Facebook's services went down in October, people were reporting that AT&T and other cell carriers were down because they couldn't open the apps. As far as I know there wasn't an outage with any of the carriers that day.
https://www.macrumors.com/2022/03/21/icloud-and-apple-servic...
Big outage... is it some stupid DNS issue again?
A few minutes later it gave me another notification saying private relay was working again.
They work on iOS as well - so it seems to be a regional thing?
(Location: Germany)
Usually basemap because they are heavy are served through a separate CDN.
Guessing the issues are centered on North America.
Rough order of events:
1. Not working (could not find server)
2. Not working (request timeout)
3. Restart app
4. Working
Perhaps DNS was broken for awhile and restarting the app cleared the DNS cache and forced a fresh IP lookup?
In the future, if you want to check if something is DNSSEC-signed (things rarely are: DNSSEC is overwhelmingly not enabled on the commercial Internet), you can just `host -t ds <domain>`.
I tried several local utilities and options but couldn't find a reliable way to determine if a site would resolve under systemd-resolved with DNSSEC enabled other than using systemd-resolve with DNSSEC enabled. It seemed like any time dnsviz.net shows an error the domain will not resolve, but some things it shows as warnings also cause sites to not resolve while other warnings do not. My favorite is that Verisign's DNSSEC validator's domain fails to resolve with DNSSEC enabled.
Possibly some or all of this is systemd-resolved doing the wrong thing, however the errors and warnings on dnsviz.net make me think this is not the case. www.google.com, for example, does not show any warnings or errors.
Joking, but only somewhat. That's because the easy cases are handled by automation, etc. If you knew it could happen, you probably planned for it. Figuring out what the issue is, if there really is an issue, and the scope of the issue can take some time.
Yes, you would set up multiple hosts across the world polling that server, but that adds complexity. Maybe, those pollers decide the site is down because of a bug in your network setup, while the rest of the world happily uses your services.
I never claimed it’s impossible, just that it isn’t “quite easy”, especially to check that the “entire developer site is down”. The home page may be down, with the rest being up, the home page may be up, with the rest being down, etc.
https://github.com/systemd/systemd/issues/9867#issuecomment-...
It sounds like systemd-resolved has had a bunch of issues like that where it fails (or previously failed) on things that would be an issue if DNSSEC was enabled but shouldn't due to DNSSEC not being used. I'll stop blaming DNSSEC.