Personally, if DO don’t have anything new in a status post, I’d prefer seeing an update that says something like “We are continuing to work on the issue. Nothing new to report. Next update in X minutes.” That is a lot easier for me to parse than the text that someone seems to be copy/pasting in each update.

What unholy thing did they do that broke it across 12 different datacenters, good lord.

This is OT, but I have a droplet on DO and I'm amazed at the amount of malicious traffic it gets. Is it normal for a very private vps to receive thousands of ssh attempts per hour? I have fail2ban installed and the jail is so busy it's quite astounding. Anyone with more web hosting experience that can weigh in?

Not sure why the previous incident page got flagged. This is the new one.

It's affecting us for real. Making almost our whole service - serpapi.com - down. As we are storing database files on block storage volumes.

Isn't Digital Ocean running Ceph for their block storage?

I would wonder - as others suggested - that they may have stretched the cluster across datacenters ?!

Would be interested in the post-mortem.

Thank you Digital Ocean for once again proving that 'The Cloud' is not a backup.

This is your weekly reminder that anything you want to be reasonably “HA” should span multiple vendors in multiple DCs.

Anyone have a review of using DO k8s or DO managed DB in production?

DigitalOcean just posted a post-mortem on http://status.digitalocean.com/incidents/g76kgjxqrzxs

(the same url)

Higher latency (per status) is not end of world especially if it’s just “may experience” higher latency.

Hrm, Atlassian BitBucket is also down. Just a coincidence? Does BB use DO?

https://bitbucket.status.atlassian.com/incidents/4t1pkwrdtl8...

Their block storage is such a failure. I’m back and forth with support to automatically delete files with lifecycles for over 2 months now and it’s still not resolved.

It looks like they have just updated it as resolved and monitoring.

I was always wondering how I can get know proactively if something like this break or some service has an outage. As a result, I have built this tool( http://incidentok.com )

So that’s why bot attacks and spam traffic was lower.

This is really down for more than 2 hours!!!

DigitalOCean bad experience

Their ad was “you’ve been developing like a beast and your app is ready to go live”

DO is a nice thing to play around with and maybe launch something, but I wouldn’t run full production on it.

I work for a web hosting company in Texas, and this is ridiculously common. Any public IP with any public service at all will be poked, prodded, and generally made uncomfortable by every bot and crawler you can think of, trying common password combinations and scanning for common vulnerabilities in popular software. This catches so many of our customers by surprise, who tend to mistakenly believe they're being targeted in some kind of attack. Generally they're not, unless they're running something vulnerable and one of the bots noticed.

Fail2ban is great to at least stem the tide. It's good at slowing down SSH brute forcing, and can be set up to throttle poorly behaved scrapers so your site isn't getting hammered constantly. If you can deal with the inconvenience, it's even better to put services that don't need to be truly public behind an IP whitelist. That stops the vast majority of malicious traffic, most of which is going after the low hanging fruit anyway.

Otherwise, it's kinda just a fact of life. With the good traffic also comes the bad.

> Is it normal for a very private vps to receive thousands of ssh attempts per hour?

Yes. The thing is about the IPv4 space is that it’s really not that big (3,706,452,992 public addresses) so it’s pretty trivial to poke every single one esp if you fine tune your port list.

The most common advice is to hide your private services. Instead of using port 22 for ssh use 23231 instead. It’s a little more annoying but you can also use port knocking. So to open port 22 (or what ever port you like) first you got to poke port 23123 then 7654 then port 39212 within a short period of time then the port knocking software will open up port 22. (Or a combo of both change default port and port knocking)

It won’t stop people “ringing the door bell” to see if anyone is home, but it will help with the then trying to brute force pass the login prompt.

I work for a hosting company and this is totally normal. Digital Ocean and other VPS providers IP ranges are specifically targeted as many amateurs running servers there.

If you've disabled password logins then just don't worry about it. fail2ban is overkill you can rate limit with firewalld or iptables withou needing extra tools.

It is "normal". Even my home fix IP gets it without any service running on it other than ssh.

I had the same experience on DO as well as a few other providers.

Changing the ssh port to something in the 50_000 range drastically reduced the number of attempts and left my logs much cleaner :).

   Is it normal for a very private vps to receive thousands of ssh attempts per hour?

Well, I haven't bothered looking in a long time. But, back when I first got a cable modem back in the late 90's the malicious access attempts filled up my hard drive in just a couple of weeks. I don't remember the size of the HD, but I can only imagine the situation has gotten much, much worse since then.

Yup this is normal, when I can't change the SSH port (e.g for compatibility) then I switch f2b to permanently ban, which should reduce any incurred load by black-holing instead of attempting authentication as the list grows more comprehensive over time (this wont affect other services e.g apache, in-case a user is unwittingly part of a botnet).

Off topic reply, you should monitor the amount of malicious traffic coming from DO networks too. (I did for a few customers at different isps and its insane)