Ask HN: My VPS got hacked and now I'm facing a massive bill. What can I do?

110 points by Koekoeksklok 11 years ago | 77 comments

I've got a VPS which I use for small programming projects and college assignments. Two weeks ago I received an e-mail from my provider, stating that "your VPS has been transmitting a lot of outgoing traffic which results in a very large traffic usage bill". In September on my 500 GB data-limit VPS, it had been transmitting 27 TB of data traffic. This resulted in a € 3300 extra charge on my € 15 VPS. I'm expecting a similar bill for this month.

Of course I immediately shut down my VPS after the notice two weeks ago, but by then it had been using these amounts of traffic for a month and a half.

What are my options here? I can't afford to pay > € 5000 unfortunately. Does anyone have similar experiences?

patio11 11 years ago |

Incidentally, since many HNers probably come at this from a mental model of "Anything which appears on an invoice is non-negotiable and simply must be paid": a B2B service provider which collects payment after services are rendered is knowingly taking on credit risk and has already priced non-collectability of some accounts into their services. You may be overestimating how much drama is required for someone at their company to say "Wow, really? OK, sorry about that. I'll write it off."

This is one of many, many, many reasons why we don't generally do cost-based pricing and, when we do do cost-based pricing, the markup is absolutely phenomenal. It has to include risk premiums. As long as it do include risk premiums, you don't have to sweat the small stuff like e.g. an uncollectable $4k invoice. (n.b. Small stuff! $4k hiccups are utterly routine events and largely dealt with by processes rather than by treating them as sudden emergencies, even if they feel like that to natural humans.)

readme 11 years ago | |

Also OP, if they don't write it off make sure to tell us all who it was so we can avoid them.

The same thing happened to me with amazon. Amazon pid for it. It's highly unreasonable in my opinion to ask the customer of a VPS to pay for damages caused by a malicious attacker. It's tantamount to a landlord expecting you to pay after an arsonist comes along and burns down your apartment, just because you happened to be renting it at the time.

johnchristopher 11 years ago | | |

May I propose another analogy:

- a landlord* expecting you to pay after a squatter came along and opened a faucet in the basement (where OP rarely goes) to fill its own super tanker

* or the water company ?

13 11 years ago | |

I have had this experience. In my case the bill was altered to $0 before I'd even got to the point in the conversation where I intended to ask for some relief. It most certainly doesn't cost them that much, and it's within their interests to keep customers coming back for repeat payments than soured by being forced to pay an exorbitant fee for something which was obviously not their fault.

justizin 11 years ago | | |

Yeah, the $4k is kind of a manufactured figure, bandwidth expenses are particularly inflated, but it makes sense at smaller scale - if you have a $15/mo plan with 2TB of transfer and you go over 1TB, a few bucks penalty doesn't sound outrageous, and their cost is impacted by higher network management costs.

But, if you have a huge spike that wasn't really your fault, it doesn't cost them any more to write that off than it does the bandwidth that is consumed by a DDoS attack that is mitigated by a firewall.

When I worked at Rackspace, we hosted a very popular flash cartoon for one month after yahoo kicked them off the $5 hosting plan for pushing god knows how much bandwidth. They basically saturated a gigabit network port from a single server, and we sent them a bill for like a gajillion dollars. They went to another hosting company, of course, and I think got the bill down to something they managed to pay off.

Someone obviously had porsche-eyes, I thought it was kind of a shitty thing to do to the guys, who came to the office to make the voices of their characters for us and stuff.

switch007 11 years ago | |

I worked for a hosting company and we'd normally charge time spent handling the issue plus the upstream cost of bandwidth, depending on how nice the customer acted.

Blahah 11 years ago |

1. Report the incident to the police. Right now.

2. Report it to the VPS provider. Explain that you've reported it to the police. Ask for their cooperation in investigating the problem.

You do not have to pay. If they try to force you to pay, depending on your country, you'll probably end up in small claims court where you'll find judges are very reasonable people who usually side with the little guy. (IANAL)

BukhariH 11 years ago |

I've been in the exact same situation with AWS ( http://cl.ly/SHOu ).

It was a nerve recking couple of days but I contacted AWS support and they were extremely good. They helped me secure my machine and then cancelled the 1.4K payment they were going to take from my account.

In all the whole process took 2.5 weeks and I only had to pay $15 for the I/O requests.

The best thing I can recommend is to talk to your host and tell them honestly you can't pay that much and you weren't the cause of the charges either.

UnoriginalGuy 11 years ago | |

AWS also has billing alarms. If you're running an account which should cost $15/month then for the love of god set an alarms at e.g. $20/month.

I had a reserved instance for 12 months, forgot to renew it, and on the 13th month (when it was on-demand) the usage creeped over my cap and I started getting alarms allowing me to kill the instance, renew my reserved, and restart it. Saved me at least $10.

On a related topic, I wish VPS providers allowed you to pre-pay. With Microsoft's Azure I have an MSDN Ultimate account, which has $150 of pre-paid credit on Azure. When you go over the $150 they just shut your stuff down rather than charging you (in fact I don't have a CC on there at all). They don't even offer this kind of service to non-MSDN subscriptions which sucks, I'd love to just pre-pay $50/month to them and have everything shut off when I exceed it (so it becomes a "no risk" playground).

pmalynin 11 years ago | | |

You can prepay on Digital Ocean.

e40 11 years ago | |

I've seen AWS credit charges that people intended to make but didn't want. We have some public AMIs that include charges for our product. Twice AWS support contact us to ask if we would credit someone back on the order of $500 out of our pocket because their customer didn't realize running an AMI would incur charges. Yeah, I was pretty surprised by this, too. It seems AWS gives people one freebie, though.

thewhk 11 years ago |

I work for a VPS provider in the US. These situations are common and we usually just issue a credit and give a reminder to the customer to please secure their server

That brings me to my point. How did the hack occur? When you get a VPS you are fully responsible for what goes on in there. It is your responsibility to secure it and keep it updated. It's not the provider's fault you did not apply the latest security updates. It's not the provider's fault your Java application was using outdated and vulnerable libraries nor is it their fault you didn't set a CAPTCHA in front of your submission forms. Either hire a competent sysadmin if you can't take care of that yourself or find a provider that offeres managed hosting instead of a VPS, as that's what you'd most likely need.

There are some cases where it's the provider's fault such as the Linode BitCoin hack a few years back but mostly it's just poor server maintenance

waxjar 11 years ago | |

People that rent a $15/year VPS use it to run an IRC bouncer or a small web log, something you don't need to know a whole lot of sysadmin stuff for. They just need a machine that's always on.

It's hardly worth hiring a sysadmin for (I find that suggestion laughable, to be frank). Managed hosting doesn't allow you to do much else besides hosting a website in PHP, which is not enough for plenty of use-cases, including OPs.

lucb1e 11 years ago | | |

> $15/year VPS

Do tell, where do I get one of those? Cheapest I know of is $60 ($5 a month).

theonemind 11 years ago |

I work for a company that provides VPSes. In a situation like this, they can see the usage is aberrant and they can see it's not normal based on past bills. They'd likely offer a large credit if you say you didn't intend to do this, and it doesn't look like a fraudulent account. That being said, they themselves probably have bandwidth costs, and are not at all likely to forget all of the charge, perhaps half at best.

onestone 11 years ago |

Stop using providers which charge a ridiculous price for bandwidth (like AWS). There are many excellent alternatives where a TB costs only a few dollars/euros.

njsubedi 11 years ago | |

like?

notok22 11 years ago | | |

Hetzner has the first 20 TB free and then charges 2 euros per TB.

scandinavian 11 years ago | | |

Kimsufi (OVH) if you need really really cheap and beefy dedicated servers, with pretty much no support. They are great for hobby projects.

onestone 11 years ago | | |

Linode, DigitalOcean, OVH, Hetzner, and so on...

jacquesm 11 years ago | | |

1and1 leaseweb ovh

patio11 11 years ago |

I would begin by contacting your VPS provider, explaining the circumstances which caused the bill, and asking "What are our options?"

ColinCera 11 years ago |

Have you talked to your VPS provider? They should be able to cut you a break; after all, that 40TB of traffic cost them only a small fraction of what they're charging you, so if they're reasonable you should at least be able to get them to reduce the charges to their actual cost.

You might also offer to suggest writing up a post mortem for them, that they can provide to their customers as a lesson/tutorial on how to protect a VPS.

Finally, you can suggest that they might want to implement (and perhaps help them implement it) some kind of warning system, i.e., if a VPS suddenly begins using exorbitant amounts of bandwidth, and far more bandwidth than it ever has before, they really should email/text the owner an alert within 24 hours — not let it go on for 6 weeks. I'm surprised that they don't cap/throttle the bandwidth once you go over your plan's limit, to go along with sending you alerts. It borders on negligence on their part that they don't already have such a system in place.

MangoDiesel 11 years ago | |

In my opinion, it is negligence to an extent that OP should not have to pay for this, and he should find a new VPS provider.

Khaine 11 years ago | | |

Why? He failed to secure his server. Why is that the fault of the VPS Provider?

Jare 11 years ago |

Depends on your provider. Amazon AWS is known to have waived such bills in the past, see for example http://readwrite.com/2014/04/15/amazon-web-services-hack-bit...

zhovner 11 years ago |

To prevent such incidents Linode have alerts of traffic/cpu/disk thresholds. For example you can configure notification if your bandwidth utilization more than N Mbit/s in duration more than N minutes. Very useful for DDoS prevention.

bluedino 11 years ago | |

And look at your dashboards once in a while. I'd find it unusual if I saw my toy VPS cranking out 100mbs for a week straight!

emeraldd 11 years ago | |

Make sure you tune those alerts appropriately and know when your systems should be doing things that will trip them. We've had a couple of boxes that regularly trip cpu warnings during normal operation, but only inside specific time windows. Knowing what's not normal is vitally important with this stuff.

dangoldin 11 years ago |

I had something similar happen with AWS but the bill wasn't as high since they ended up flagging my box as spam-producing and shut off all outbound traffic. I'd just ask them and see if they can remove the charges, it worked in my case.

matthewarkin 11 years ago |

I had the same thing happen to me. I wrote about it on my blog http://mattarkin.com/protect-your-azure-linux-vm-aka-how-to-.... Basically I complained to Microsoft, they said they'd waive the charge but since it was for a linux vm they said they couldn't cover it. Then I complained to American Express claiming it was an unauthorized and fraudulent charge. Amex sent the dispute to Microsoft and they never responded so I wont the chargeback.

minopret 11 years ago |

I can understand how that could happen and what a problem it would be. I had an experience with a telephone bill myself, but the story is not going to help you.

I would suppose your first and best resort is to consult your lawyer, advocate, solicitor, barrister, Anwalt. I wonder what your relevant legal jurisdiction is.

I wonder whether it would help if you can account for your own whereabouts and your own usage of endpoint data services. I wonder if your method of payment to your VPS provider is mediated by a financial service that can help you dispute the bill.

I am not a lawyer.

jnardiello 11 years ago |

I assume you are in europe. I'd suggest simply talking with your provider, explaining the issue and asking them to investigate. I honestly expect them to cooperate and be understanding.

If they insist for you to pay: simply don't. State the truth: You can't afford it. Tell them the only way they will see this money is by taking legal action against you and even in that case you won't be able to comply - as you don't have the money.

Hope it helps :(

freshflowers 11 years ago |

Just in addition to some other helpful comments: based on posting I assume that your are Dutch or Belgian, located in Europe and are buying this VPS as a private consumer, not a company.

Which means your case is probably covered by consumer protection rules when it comes to informing you about data usage, and I seriously doubt a VPS provider has covered their ass as well as mobile providers tend to do.

tdicola 11 years ago |

Anyone have tips on how to secure their Linux VPS? I just set one up and disabled SSH password login, locked down all the ports with iptables (using ufw), and enabled fail2ban. Anything else I should install or configure to make myself a little more secure? Was considering tripwire but I dunno how much a headache it would be with false positives as I change things on the server.

MayIHaveAnother 11 years ago | |

A very common attack vector is through installed web applications. Especially if you run wordpress with a lot of plugins installed, be sure to enable correct read/write settings for /var/www, and update your application frequently.

Malicious entities runs 24/7 scans towards indexed URL's attempting to exploit various vulnerabilities, and many of the vulnerabilities allows remote code execution, upload of php files etc. This can be used to upload malicious code, simple php-webshells, and then your VPS is suddenly a part of a DDoS/Scanning network.

Exploited Wordpress sites are a problem, Zeus/Zbot-Trojan is often seen downloading updates/configs from these, and they are also often used to redirect users to Exploit Kits.

porker 11 years ago | | |

I came across these two tools recently that seem interesting: http://www.rfxn.com/projects/linux-malware-detect/ https://github.com/emposha/PHP-Shell-Detector

Not installed either yet (LMD could really use some .deb packages) but could be a useful alternative to Tripwire

Ded7xSEoPKYNsDd 11 years ago | |

On my personal machines, I also setup a cronjob to automatically install updates. There's a small risk of breakage (I had one in 5 years), but I prefer that over a bill like op's.

applecore 11 years ago |

PSA: Set up billing alerts! You should always have a notification sent to you when your monthly bill exceeds one or more dollar amounts. For example, if you're using AWS, Amazon CloudWatch lets you set an alarm on a billing metric to notify you automatically.

joshmn 11 years ago |

Post on WebHostingTalk.com - just do it. You'll get attention from the host, other hosts who will sympathize, and you'll see that they'll just write it off.

Post the link when you do and I'll be sure to comment on it (I'm somewhat very-active at WHT)

decisiveness 11 years ago |

I seem to be missing something. You knew it was happening when you got the first bill, but let it continue for another half month before shutting it down?

logn 11 years ago |

In addition to the other comments, make it absolutely clear to them (with proof if needed) that you're a student.

gregcmartin 11 years ago |

Make sure Elastic Search is not accessible from a public IP address (this is what likely got you in the mess to begin)

general_failure 11 years ago |

Do you know how you got hacked?

zack19 11 years ago |

cut your credit card report it as stolen tell the host that it wasn't u :p

ishener 11 years ago |

I feel really sorry for you situation. I first suggest talking to the hosting provider and explain what happened. Any decent service will give some discount in this case.

Unfortunately, I can't think of anything else. I wish it was realistic to tell you to go to the police.

Also, if you would give your email, I would definitely consider sending a donation through paypal... Hopefully other readers here will do the same.

patio11 11 years ago | |

There is one great reason to go to the police: it establishes a paper trail documenting that a crime was committed. It isn't necessary that the police catch the bad guys for that paper trail to be advantageous.

(Examples: police reports make CC disputes and legal declarations much easier and more likely to be given weight as other than self-serving explanations of a deadbeat. It may also trigger insurance policies either for you or for the VPS company.)