"Hetzner decided to cancel our account and terminate all servers"(mastodon.social) |
"Hetzner decided to cancel our account and terminate all servers"(mastodon.social) |
1) There is some fundamental data aspect Kiwix hasn't mentioned (or is entirely unaware of). I.e. CP or some other super illegal stuff.
2) Hetzner is profoundly incompetent, deleted production servers by accident, and the "But we sent you an email!" thing is a lie to cover up the mistake.
3) There is some kind of interaction that happened prior to this that we aren't privy to. Perhaps a series of late bills, legal threats, or some other inter-personal issue.
Predicted outcome:
I either expect Kiwix to get a knock by federal/national authorities. Or the more likely outcome in my opinion: some frustratingly vague statement by Hetzner PR about its customers being "mistaken" in regards as to why data-go-poof.
I mean seriously, let us assume it's something illegal: Sure, fine, whatever. Wouldn't it make more sense for that material to not be deleted, so whoever the guilty party is would be arrested for/prosecuted by it? Deleting the servers would be like police being informed about a murder weapon and asking the tipster to destroy the weapon before an arrest is even made. It doesn't make any sense to me. Surely if some bad thing were discovered, there would be some method to encrypt/restrict illicit material without destroying it.
Either bad blood, or unpaid bills, or simple incompetence seems like the most likely culprits to me.
As an example, you run any crypto related operation, even if it's a mere 5% of your workload, you will have this happen to you. You don't even have to be hosting anything at all.
OVH had a datacenter burn down a few years ago, so think about that what you will :)
Imo it makes sense to spread out, and have backups with a different provider than the one your main servers run on. They should all over S3, which is a standardized format for easy syncing of backups
There's also Webtropia I have used in the past, also German, without issues.
OVH would be the safest bet but their support is worst than Hetzner.
"""" We have detected that your server is using different MAC addresses from those allowed by your Robot account.
Please take all necessary measures to avoid this in the future and to solve the issue. We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it. In the event that the following steps are not completed successfully, your server can be locked at any time after DATEHERE.
How to proceed: - Solve the issue - Please note, in case you have fixed the problem, please wait at least 10 minutes before rechecking: https://abuse.hetzner.com/retries/?token=TOKENHERE - After successfully testing that the issue is resolved, send us a statement by using the following link: https://abuse.hetzner.com/statements/?token=TOKENHERE
Please visit our FAQ here, if you are unsure how to proceed: https://docs.hetzner.com/robot/dedicated-server/faq/error-fa... """
I was just using standard Docker to host a web app. No proxmox or KVM of any sort. I would just wait the 10 minutes, click their link https://abuse.hetzner.com/retries/?token=TOKENHERE, which would retry and would come back fine and my response would be "I changed nothing and the retry came back solved. I've done tcpdumps over a weeks time to see if any MAC addresses leak from the OS and none have while a similar ticket like this gets opened every couple days." The ticket would close shortly after I submitted.
I inquired to them at least twice about this and they just kept telling me I was leaking a MAC address that I wasn't allowed to even when I had proof of tcpdumps over a week time period. I found someone else who had this issue with them (most issues around this that I found were people hosting Proxmox) and they had Hetzner replace the NIC and it fixed the issue. Well, Hetzner wouldn't replace my NIC because "it was working" even though I referenced these abuse tickets. I ended up getting another dedicated server, migrated my app over there, and I haven't had issues since.
Their support is seriously not very good. Since that experience, I have had backups elsewhere and test restoring those backups regularly. The price to performance I get from them is unbeatable and like I said, I haven't had issues since getting a new machine. But, I'm definitely cautious and don't exactly trust things to not go sideways even though it's been 2 years since that experience.
Some of the files they host are pretty big, so maybe Hetzner just decided it wasn't worth hosting any more.
I've been using Hetzner for years though and never had an issue. But I don't get anywhere close to the 20TB traffic limit.
This reminds me that I should set up some backups though.
After all, Hetzner is now priotizing shareholder value and is removing smaller customers wasting their compute resources.
Crypto validators can be quite noisy neighbours which is a problem on fair use VPS
Dont think it relates to small or not
But I am not in a position to take those decisions anyway.
I have stopped relying on instances being secure and map out a just-in-case strategy (that I also regularly oversee or exercise) to quickly reset/restore and get back on track.
It does raise an interesting question of how to reliably contact a customer if email is broken?
aws, azure, and gcp aren’t cheap , but they offer better stability—both technically and operationally.
AWS has lots of problems. But they have a team of real humans that respond to tickets around the clock and actually understand stuff. To many businesses, that's worth the extra cost clone.
Even if you're a nobody spending $30 a month, AWS are extremely responsive and helpful.
https://www.theblock.co/post/182283/1000-solana-validators-g...
That said, I host on them too. But some stuff is on nearlyfreespeech.net.
I know a few others who have suffered the same from this company.
> 6. Termination
> The Provider may terminate this Agreement at any time, for any reason, with or without notice to the Adopter. Adopter may terminate the Services upon notice to the Provider.
Theres certainly other (German) legislation in place that might be relevant, but whats the point of speculating if we can’t blame the EU for it.
Every half decent switch made in the last 25 years can be configured to allowlist MAC addresses. Either that, or dropping customers onto their own VLANs is the standard way of managing this.
Unless they are a "GmbH & Co KG"?
But it's 1Gbit public and max 5Gbit (Plus double the price)
It's pretty clear they basically prohibit everything related to cryptocurrencies, even content, as you can see in their T&C.
You should have all you backups in a different location and terraform tested with a different cloud provider, otherwise you're risking the company.
[Edit] Where I come from: That doesn't say anything about Hetzner, I have been with them for 20+ years, they have stopped individual servers in that time frame, but haven't cancelled my whole account.
First they wanted us out of on-premise, and told us costs wouldn't matter.
Then they wanted us to be 'cloud agnostic', but when given deadlines changed to 'get it working in AWS ASAP, doesn't matter the tech debt'
Now they're freaking out about AWS costs, and we're back to juggling 'cloud agnostic' and 'reduce cost to serve in all clouds' priorities on top of features and maintenance, both of which are 10x slower due to tech debt and the plethora of bugs.
I really need to find a new job soon. Its insane how badly the execs and upper management are running this company. Every day is a knee jerk reaction from someone so detached from the reality of things or with so little understanding how it works, they do nothing but add process problems that barely address the issues they think they're solving.
There are so many providers, and therefore examples, of physical tin being accessible in under a minute with cost:hardware ratios that blow Cloud out if the sky (pun! ha!) OVH have a server for USD $95/month (with no commitments) that can be brought up and made available 120 _seconds_ that has six 3.8GHz cores, 32GB of RAM, 2x960GB NVMe SSDs, and 1Gbit/s of UNMETERED, guaranteed bandwidth... that's absolutely insane, and that's fully managed from the hardware down, so arguments like, "bUT yoU haVe to MAintAin hardWARE!" are just not true _at all_.
But also, the execs are the ones making the business risk decisions. Just make sure they have the correct info to make those decisions, the. Your responsibility is done.
When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago
> Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.
As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.
I’m doing that to linux, and then the Linux box is furthermore backed up with nakivo.
Not my favorite but the price was okay and I can run the whole director on Linux, unlike all their other competitors. [veeam’s next major release 13 or 14 should do this in the next year or so too.]
While nakivo backs up s3 buckets, NFS shares, and local file servers… to your point, I don’t trust it (or any other backup software I can’t extract and unpack the resulting backup by hand) as far as I can throw it. So I rsync or mirror it to a local Linux box with aws-cli and then back THAT up.
I think you can do all this with windows stuff too but I don’t know it that well
Additionally you can take servers that are linux vps’es and do the reverse: mirror THEIR content to an s3 bucket.
You can also run minio open source/free on your fileserver and set up s3 to s3 sync. Cloudflare for example will ingest and replicate your minio server automatically and you can firewall it all off to their address ranges. It’s not free but it actually prices out favorably compared to veeam and nakivo if that’s all you need backed up.
Of course it turns out that the restore can only happen to a _new database name_ not the original, and the code had in multiple places hardcoded the assumption of what the db was called.
So restoring also involved patching the code and rolling that out; you can't "roll back" because to roll back the db the code must roll forward.
Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.
1) Expensive
2) Not straightforward, e.g. is there a 1:1 setup in another cloud for your system?
3) Likely to go untested and be useless when you need it most
The most expensive part is going to be maintaining an up-to-date offsite data backup. Running a few VMs for a handful of hours is basically free.
https://rsync.net/pricing.html
That said I was once a CTO for a company with 10 photo studios and we had a large amount of new (raw, DSLR) photos per minute, so cost was an issue and also upload speed for offsite backups.
I manually pulled a backup of everything but jeez, not good.
What is the most reasonable point that meets the criteria of 'as soon as possible'?
Because I imagine out of the gate doing this could be a net negative, not a net positive.
On the other hand, I'm not sufficiently well versed enough on the absolute latest devops techniques that may make this whole thing trivial, but I thought all the major cloud providers had just enough quirks in their Terraform support you can't write once standup / deploy anywhere
It's very easy to do if you don't do the absolute latest devops techniques.
Of course an accident is different than just randomly terminating service.
If you had a bunch of retailers in a shared space (like a market), and one of them was setting off fireworks, using all the power/water in the space, and scaring away customers, I'd expect them to get kicked out pretty quickly.
Now it may be that this is a false positive, I'm sure they happen, but in the case where it's a legitimate bad actor that is actively harming both the company and other customers on those servers, what's the course of action the company should take?
I have never been a customer of google cloud for this reason and i sure as hell wont deploy new servers on hetzner until they provide a clear statement on what went wrong and what they will do to make sure they never screw up like this again.
Hetzner, the ball is in your court.
They're a budget host, you should always proceed with caution and never rely on for production. It's the same as buying a second hand eBay server to host users upon. I learnt that the hard way.
> Hetzner, the ball is in your court.
Not really. If you read the T&C, you'll find that they can do anything with the server.
From their T&C:
2.7. Furthermore, we reserve the right to terminate the contractual relationship without notice for good cause.
--
Any server with any company can do the same. There's been numerous stories of Amazon has doing the same. Same with Google.
Unless it's colocation or where you own the hardware you can be screwed in many ways.
I would never trust a dedicated server host.
Yes, but that doesn’t mean that they have to.
Did you miss this part?
Yeah, if. Why are you so certain that Hetzner "screwed up"?
But this seems to be about Kiwix (which in short is "offline Wikipedia" in various ways) and doesn't seem to be about questionable content in any way.
Eventually I guess we'll get Hetzner's perspective on this, as they tend to start writing publicly about issues once the other side starts writing publicly about it as well.
Personally I've been a happy user of Hetzner for many years, with no issues that weren't my own doing. But reading about people having their servers deleted in the middle of the night on a Sunday (Berlin time) and all data wiped immediately, with no recurse, does sound a bit aggressive. Luckily it seems like both me and Kiwix has mirrors for the data we care about.
That’s actually insane and business killing. Both for Hetzner’s reputation and potentially for their customer.
Hi,
I don’t have a mastodon to reply directly to you.
But i have had some issues with content being taken down by VPS providers as well.
What I’ve found works well is to use a VPS provider that the public is unaware of. And for some time I had used OVH based on the unlimited bandwidth and the reasoning that Wikipedia and Julian assange (who have far more enemies than I ever will) were using OVH.
I don’t know if that’s true any more because I subsequently moved my content to ENS and IPFS.
Anyway regardless of where your content is actually hosted or lives,
What I had done was turn my “real” servers into content origins , which were concealed form the rest of the world and lock it down in the firewall so it could only be reached by disposable squid proxy servers with a 10-liner config file
Then I pointed DNS , cloud flare etc at the squid nodes
And couldn’t care less if they were taken down.
Because I could deploy new ones in minutes elsewhere.
I didn’t have “bad content”, just ruthless business competition that kept coming at me like Tonya Harding.
And I’m sharing because your content didn’t seem too offensive either.
In the front end VPS nodes you’d just put the real address of your content as the remote origin.
And then nobody but you will ever know where it is.
Then generally your hosting company shouldn’t be aware of what it is either unless they’re snooping around in your files, and if they are, hell with them too.
You’re welcome to pass this along as a remark on avoiding censorship, or keep it to yourslelf as proprietary information I don’t mind. Let me know if you want or need an example squid conf. It’s seriously 10 lines at most and many examples found on google.
- Ask HN: Hetzner banned me with no explanation. What can I do? (https://news.ycombinator.com/item?id=32318524)
- Hetzner didn't even provide a detailed info on why they deactivated my account (https://news.ycombinator.com/item?id=40781617)> Dear Mr David Allison
> After reviewing your updated customer information, we have decided to deactivate your account because of some concerns we have regarding this information. Therefore, we have cancelled all your existing products and orders with us.
> Best regards
> Your Hetzner Online Team
And why would anyone need anything but code in monospace? Please don't do that.
i am always angry if i see articles about them here on HN because such a vendor should be blacklisted and not promoted.
Out of pure spite I built my own data center.
I would be curious about any details you can share.
Of course it wouldn't work for all cases but I find it beats having a vps somewhere that can be taken down for no reason at all.
Hardware: 4x Old decommed 19" dells on Ebay with plenty of DDR4 memory, HP Proliant G10+ are also good
Ups Eaton Pro
Gigabit Fiber Internet, which is more than enough. 10-50mbit can suffice for compute nodes too.
Bought ssds and m2 storage plus some spinning ols rust drives
Temp and humidity monitoring
Google Nest Protect smoke detector
TP link 16amp smart plugs on all, to have a control plane to turn it all off remotely
Workloads:
Most are LXC
Some Docker
KVM virtual machines
Zero trust: Some Cloudflare
Tailscale
Proxmox backup server to back it all up, lots of retention
Monitoring:
Deployed remote uptime monitoring on fly.io
Read and experiment a lot Hang out on /r/homelab /r/homedatacenter and /r/selfhosted for learning, community and inspiration
We cancelled our account last month because of that.
I cannot imagine the world of hurt that we'd be ushered in, had they actually dropped our data wholesale like they did for OP.
https://lowendspirit.com/discussion/comment/191966#Comment_1...
Would definitely be good to hear hetzners side of the story because all the cases I’ve seen thus far turned out to be a case of initial telling being understandably upset but leaving out crucial details.
They definitely are trigger happy with telling customers to find someone else & generally don’t elaborate on why
Last time I used them (pre-2020) they were going as far as requesting customer's ID and rejecting them on the basis of country of origin, and I assume this also includes facial features that may resemble "an average scammer". Obviously this did not happen to European/American IPs so they never faced such issues, and as such this practice was invisible to the world.
I can say for sure OVH and Scaleway would try to negotiate with you before erasing your data - this may have changed over the years.
Dodged a bullet.
I am always recommending to not build on Hetzner.
Ok but on topic, who is this guy and why did they do this to him?
The funny bit was I paid the invoice, and then my account remained suspended. When support finally got back to me a few days later, they said (and I quote)
Dear Client
We want to give you one last chance as a gesture of goodwill, so we revoked the cancellation for you.
Kind regards
Seeing it happen to a reputable project such as Kiwix [0] definitely damages my perception of Hetzner. I've read numerous complains on Reddit a few months ago but they mostly boiled down to breaching the ToS in obvious ways. Still, not giving a heads up before cancelling a service and no option to recover data is just bad business practice.
[0] (I've deployed Pi's with Kiwix in remote areas in Africa, it's an amazing project)
Same goes for having your domain with the compute provider.
If you're locked on some proprietary services like with AWS it is much bigger issue.
Given one of kiwix.org's goals on their homepage is "access to internet ... [due to] ... outright censorship" (which is legally required in some countries), likely they are in violation of T&C in
"8.1. The Customer is obligated to check and comply with the legal provisions arising from the use of the contractually agreed services"
and
"8.4. If we become aware of illegal activities, we are obligated under Art. 6 Abs. 1 DSA (Digital Services Act) to request that the Customer immediately removes the offending content and we are entitled to lock the Customer’s access to their Hetzner services or account."
Of course there is usually a bit of a chicken and egg issue with this sort of thing. Many companies only respond at all when complaints go viral on sites such as hn.
None of this is to say that Hetzner responded in an ideal manner, and whatever reasons exist for the termination are still not known, but it seems likely at least some of the OP's criticisms of the process are not valid.
Of course now it's too late as many will never come back to review this thread. As often happens on hn when somebody complains about something, the pitchforks come out before there is enough information to really understand the situation fully.
Over the past years there have been numerous people online which have claimed that Hetzner closes accounts without giving a reason. I'm sure most of those claims intentionally omit some details to make it look like they didn't infringe the T&C.
However as a Hetzner customer (a small one, to be fair), I'd still like to know that those complaints are baseless, and that I can still trust your company.
2.7. Furthermore, we reserve the right to terminate the contractual relationship without notice for good cause.
This below is where we got started - the ref number should make it easy for you to sort: > Procedure: L0020649F > Person: [redacted] / Kiwix > Cause: Hello, > > Starting this morning (December 1st at 00:00 UTC), our servers went down. > We received zero email nor notification of any kind from you. > Looking for a way to contact you, I looked into this Unlock tab that list an incident > that matches the time the problem started. > > It's been close to (12) hours already, without a single message from you. Our services > are down. > > In the Robot dashboard, there is no server listed. In the Traffic statistics page, it > says we have no IP. > In the Cloud dashboard, we cant even enter, it says Access Denied. > > What's going on? The billing page is reachable and it indicates we paid all our > invoices and the next one is to come in 5 days. So it's not a payment issue. > > I checked > https://docs.hetzner.com/robot/dedicated-server/troubleshoot... > > I am not sure if we're locked because the traceroute does not lead to > blocked.hetzner.com > Because the server is not listed, we cant use the whitelist or any other tool. > > Please restore the service immediately. > Please let us know what kind of issue there is if there is one. > > Only restoring SX65 #2453510 (135.181.224.247) is urgent. The two cloud ones can be > sorted out later.
We got two more emails from Hetzner the day after that (Monday 2) but none addressing the root issue. Our account access had been locked by then anyway so we had to call up Germany; you should be able to document that as well.
Not sure HN is the best place to compare notes but hey, happy to meet you where you feel comfortable responding.
What are you talking about? The comment you're replying to says:
> There was a notice of termination via email with a deadline in accordance with our T&C, on 30 October 2024.
> Our team has already been in contact with this customer several times and we also have the transmission protocol of the communication.
Or hosting content that Hetzner misclassified as against their ToC. Or that they decided was because of a string in a random file name. Or, in one Mastodon instances case recently, because Hetzner saw that users could upload their own images and decided that was risky (nevermind that this is common and they have moderation and a strategy for if someone tries to host anything illegal, but that one employee reviewing it was twitchy that day and there is no recourse), etc.
Or, in one Mastodon instances case recently,
because Hetzner saw that users could upload their own images
> Users first download Kiwix (or a browser extension), then download content for offline viewing with Kiwix. [1]
> Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong. [2]
Maybe some odd photos landed on WikiMedia which then got automatically synced to Hetzner's servers and then triggered some alarms.
I can't judge about Hetzner deleting the data, but them not attempting to really get in touch with the Kiwix team -- after all they should know that they are trying to do some good in this world -- is a really horrible move. In the same category as Google blocking access to user's accounts without any word, or German companies suing security researchers for notifying them about a security flaw in their systems.
Shame on Hetzner.
[0] https://en.wikipedia.org/wiki/Kiwix#Available_content
"the hoster deleted my stuff without warning" is up there with "the dog ate my homework"
On one hand you have these comments in this submission, saying Hetzner is too trigger-happy and takes down things too quickly. On the other hand, you have people like you using the process from the other side who feel like nothing is being done and it takes forever to get through them when needed.
I feel like it's very hard to have a balanced perspective unless you have experience of both sides of the process, which unfortunately I'm guessing most people are missing. I certainly am, as I've never tried to get someone else's servers taken down on Hetzner, so I have no idea how that process works, I've only ever been on the receiving side.
There are several comments under this thread from people reporting essentially that happening to them.
OP claims everything got immediately wiped without warning. That would be against Hetzner's own TOS.
OP also doesn't elaborate further, and is posting this in a position where he has to explain his own downtime of multiple days. Make your own judgmenent what is realistic here
I would not recommend them for a serious, money-on-the-table business.
Maybe they're cheap for a reason.
[citation needed]. Even when they shut down Russian customers they gave advance warning. This is the first time I have heard of service being shut off (and data deleted) without any warning.
That’s what the post said. But of course we have no idea if it’s true or not. No evidence was provided, and we are only hearing one side of the story.
And in that hackernews thread we have dozens of people relating similar stories.
The reality that they have this power, and that they'd delete data irretrievably, scares me.
Last year I had a misconfigured port on a Docker service, and someone was able to exploit it and run a port scanner. It was during a period that I was away from home, so if I hadn't seen their service abuse emails in time, I could have returned home after a few days to find all my data wiped out (or uptime monitors complaining).
as much as we like to hammer on EU (lack of) companies, one potential improvement point is customer service
German companies are awful at customer service. Even within the EU
one would have to reconsider a century of stereotypes if they weren't.
True also from my experience. I've noted several potential reasons why that is from my time in Germany.
Government provided customer protection laws are quite lax and disputes tricky to win and don't represent a big enough deterrent for the scammers when they're just a slap on the wrist and therefore part of the cost of doing business. Sure, you can get sued and you loose once, but if 80 of the 100 customers you scammed don't sue you or don't win, then you're still at a net positive and therefore it's profitable to keep doing that.
Also that Germany doesn't have common law, so lawsuits aren't arbitrated based on precedent, so customers who got screwed need to sue and win individually for the same issue which is favorable for the companies doing the screwing as without the precedent of common law that minimizes their risk of loosing by slam dunk every time. Also, some German judges art just tech illiterate boomers who will throw out a case they don't even understand unless you're Axel Springer.
(some) Rental agreements, internet, telco and gym memberships are my favorite infamous examples. They're almost universally regarded as anti-consumer, with tonnes of sketchy clauses, but German lawmakers do nothing to improve that for the consumer.
Secondly, Germans aren't used to being very demanding and lighting a brand on fire on social media the way Americans/Anglophones do on Twitter when they don't like something, partly because of cultural reasons where making a fuss in public is discouraged/shamed, partly because of legal reasons where a company can sue your or at least send you scary legal letters for libel if you damage their brand online like that in Germany. Or at lest, the company can simply demand the social media platform take down the offending posts, and by German law they have to comply which the likes of Google/Meta will comply automatically without any arbitration.
Also, culturally, the conservative Germans seem to have have gaslit themselves into believing everything "Made in Germany" is perfect without fault, while everything made abroad is of poor quality or at least worthy of scrutiny, so they just default to using German products without looking across the fence to check out the foreign competition. This way of thinking is more typical of manufactured goods but not sure how much it applies to SW products and services.
Couple these with the difficulty of starting and scaling a business in Germany as a small entrepreneur and with the legal and bureaucratic hoops designed to keep foreign competitors out, mean that German companies operating in Germany who became established players, have litte incentive to improve beyond the bare minimum, so they can keep providing poor quality services while still staying in business. It's classic of an economy of well connected dinosaurs sitting on old money.
For what now?
Great if it works for you, congrats. But I don't think this solves issues for many people, I doubt it solves an actual issue for you and it's basically the same as using cloudflare/akamai/similar but with a manually setup proxy on a VPS.
That’s not my experience. We get these emails about once every 6 months, we act and respond, and they don’t take anything down.
Is it possible that maybe others had a different experience than you, and those experiences are as valid as your own?
Besides, what was your website about? I've received notices I had to reply to within 24-hours, otherwise they delete the servers. But I've always replied and complied, so never had any servers deleted.
If it’s a single email - then even if it doesn’t get caught in a Spam filter that’s still a short period of time to notice and respond when the stakes are so high.
If that email goes to junk, or you’re unwell and not checking emails as frequently (given - I assume - that many of Hetzner’s customers are individuals) or any other number of reasonable situations, you’ve effectively had no warning before service termination and deletion of data.
I don’t mind cloud providers acting on suspicious usage patterns or abuse reports but there has to be some kind of due process or it just ends up unnecessarily destroying goodwill in a brand/provider.
some random app vendor didn't like the free promotion on our website https://macupdater.net/
we can delete any "offending" page within a few hours, but taking the whole server offline first and asking questions later is not OK by Hetzner.
others had better experiences and got a 24-hour timeframe. just asking but is this during business hours or can they send you a notice on saturday and you'll be offline by sunday? doesn't seem much better.
I was considering them for a small project, but as this project will be nobody's fulltime job, I can't guarantee that I or anybody else would necessarily see that email within 24 hours.
Even if most people will have no problem with them, I’d say that knowing how a company handles edge cases like this is much more valuable than knowing how the handle things when everything is fine.
They offered to "recover" the account, which was basically just an account shell with my info. All of the assets and backups had been permanently erased.
Now we need to know the full story. Did you have a public DMCA takedown link and actually handle requests and the complainers just ignored that and went over your head to Herzner? or did you just wing it running a server with UGC thinking it's surely gonna be OK?
I am not saying you were wrong but you only tell a small part of the story
However this is more related to EU regulation rather than Hetzner itself.
Hosting things within the EU has become really tough.
I, as a European, using mostly dedicated servers within the EU (including Hetzner) haven't noticed this at all. What are you referring to specifically?
Some "use cases" like building marketing profiles and alike certainly has gotten harder, but that's a feature so I'm guessing you're not referring to that. I don't think general "hosting things" has become any harder than before, assuming you're not trying to slurp up as much data as possible.
they did NOT give any 24 hours.
I have no idea if that was the reason, though.
f.ex. the situation with egress costing money in the US, but it's free on all EU location.s
Aren't you confusing Hetzner Cloud with Hetzner Robot (dedicated servers) here? AFAIK, Cloud has egress costs while Robot is usually unmetered.
They are exceptionally fast at detecting things like that though.
From their joint statement [0]:
> when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.
> an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.
> UniSuper had backups in place with an additional service provider. These backups have minimised data loss
Strangely enough on this last point a Google blog post [1] says:
> This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region.
[0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joi...
[1] https://cloud.google.com/blog/products/infrastructure/detail...
[0] https://news.ycombinator.com/item?id=42387842
[1] https://old.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_ca...
Given one of their goals on their homepage is "access to internet ... [due to] ... outright censorship" (which is legally required in some countries), likely they are in violation of
"8.1. The Customer is obligated to check and comply with the legal provisions arising from the use of the contractually agreed services"
and
"8.4. If we become aware of illegal activities, we are obligated under Art. 6 Abs. 1 DSA (Digital Services Act) to request that the Customer immediately removes the offending content and we are entitled to lock the Customer’s access to their Hetzner services or account."
Furthermore, we reserve the right to terminate the contractual relationship without notice for good cause. Such good cause is deemed to exist, among other reasons, if the Customer fails to meet its payment obligations or violates other important customer obligations. A further important reason which may result in us locking or terminating the Customer’s services or account without notice is if the Customer uses content that impairs the regular operating behavior or the security of our infrastructure or our product, or violates paragraphs 8.1. - 8.3. of these Terms and Conditions.
Was that on a phone call? Because if not, surely there is some record?
Yeah, I've found this to be the trick for ongoing hassle-free maintenance too. Make tearing stuff down and up frequent enough and you'll feel confident and safe when you're required to do so to recover from something.
Scariest are applications/services/servers that has been running for years but never restarted nor ever restored. Those scare me.
But that's one helluva "just", and also means that you can't use the platform-specific features that make life easier. In practice that's probably way more expensive than spinning some testing VPSes up and down.
If Amazon decides to throw the banhammer your way, how long will it take you to retool your stack onto another cloud platform? Will your company survive if all your services are offline for a few weeks?
And if you grow beyond the startup size, can you afford being locked to proprietary technology? What are you going to do if Amazon decides to increase your prices by 100%? How are you supposed to negotiate when Amazon knows you are unable to switch to another cloud provider?
You could leave a stack of HDDs and other consumables in your server cabinet for them.
"Such good cause is deemed to exist, among other reasons, if the Customer fails to meet its payment obligations or violates other important customer obligations."
But replying makes the story last longer.
And there's the saying, it's never confirmed until it's denied.
In the end, they did respond.
Also, don't forget about hardware acquisition costs, upgrades over time, replacement hardware and downtime due to outages, etc.
What size company would you have to be where a 24 hour notice would not be problematic? I'm actually curious as to opinions here, and understand that obviously part of it is how well managed are your employee leave messaging etc.
I know one company with a very good manager and I think they would have managed it with 5 people being in the group of people who would handle this kind of thing (keeping track of all services etc. Obviously only 1-2 person does this but redundancy so it falls back when they are on vacation), slightly over 30 people in company size altogether.
If you're a startup of 3 people for example 24 hours might be game over.
If you're a startup of 3 people for example 24 hours might be game over.
Realistically to have 24/365 email coverage you'd need like, full-time founders or at least a couple of paid employees.
For what I was considering, I will be a "founder" but I'll still be working my day job. So effectively that is > 16 hours per day (work + sleep) I need to dedicate to the day job. While I will generally be able to respond within 24 hours, I can't 100% guarantee it.
They do that even within EU, and even when your credit card passes 3D secure validation.
Or do you mean the backup process was fine, but they restored from the wrong media, a very old tape that was about to be overwritten, instead of retrieving the one with last-week's copy?
But it ends up costing a shitton of money to switch paradigms completely, and they don't switch paradigms completely for a number of years: If you're just migrating servers to ec2/vpc, you're doing cloud wrong.
Of course, there is the idea of cloud agnostic, or even multi region, which seems a challenge for most places.
At least with terraform, it is theoretically easier to swing configurations over to a different host.
In a world of scarcity, just keep communicating the tech debt. Maybe occasionally propose a project to address it.
they can easily go into the hundreds of TB, depending on your platform.
and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.
"daily test restore is infeasible for anything but toy projects. "
It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies.
"10+TB backup for example, and thats a _tiny_ backup size for a b2c product."
Sure.
If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.
> they were going as far as requesting customer's ID and rejecting them on the basis of country of origin
Unfortunately we're at stage they will outright ignore what they're told, and then blame engineers for not being able to do what they said they couldn't do from the start. They refuse to acknowledge their impact on creating the tech debt in the first place by poor planning and wishful but impractical timelines, so proving to them we need to tackle any part of it is a struggle without letting things degrade to the point a real customer with significant money on the line is upset enough by the state of things to tackle it.
Which ultimately means we're at the horribly dysfunctional stage of management/company growth, the question is does it continue to get worse or does the CEO eventually learn and seriously look at the effectiveness of the VP levels and make changes...
Those costs were moved and ended up higher than the capex costs were to begin with which everyone expected but the decision makers (they brushed it off every time they were asked in company Q&A's). Opex margins became a major issue and the company did performative layoffs and restructuring to appease the shareholders (then re-hired ~1/3 of the laid off staff within the next 8 months because they actually needed them)
The level of 'bad decision leading to bad decision' happening is somewhere between absurd and depressing at this point.
I think this all boils down to a knee-jerk reaction culture that doesn't think about the second or third degree consequences and/pr beyond the next 2-3 years.
They deleted all of my data a month in due to not beleiving my name was real, and without even bothering to contact me to verify anything. They deleted my backups as well because I was dumb enough to keep them under the same account.
I learned a valuable lesson the hard way and have improved my methods as a result, but sad that it cost me an entire month's work due to carelessness and recklessness on their part.
Sure, it's "cheap for a reason", but let's not pretend like this type of expectation is advertised, especially as many on HN tout them as a drop-in replacement for competitors.
Also shoutout to Cloudflare for showing off what a diverse company they are in this one /s
“Actually, they’re 1000 years old”
https://en.wikipedia.org/wiki/Network_Enforcement_Act
Unfortunately the act is designed to block vague categories like "hate speech" and "misinformation" and has huge fines attached, so it's designed to ensure that very trigger-happy enforcement is the only workable strategy. It was written to whack Facebook and Google primarily but it's possible that the wording also captures Hetzner, or they're worried that it might.
If they do feel they fall under it then they'd probably have to automate takedowns in response to abuse reports. As otherwise they'd need 24/7 on-call content reviewers, which goes against their low cost nature. So if this is the cause it's really an issue with German law being unfriendly to smaller/cheaper content hosters.
The questions here are rhetorical. It doesn't matter what we think the answers are. The penalties are so huge that if there's even a tiny chance of a judge disagreeing with you, then you have to take measures to avoid the risk.
* crypto mining (I used it when it wasn't causing much trouble but I noticed my nodes were constantly attacked at a ratio I newer saw for other servers); IIRC Hetzner's current ToS forbid crypto mining
* things in legally grey area which might be legal in some places but not so in others, especially in the EU
* protect your servers well; if you become a victim of an attack and your servers will start attacking other, Hetzner will isolate them and notify you so that you can solve the problem
Other than that, the only problems I had in the last 15 or so years are failing bare-metal components that they promptly replaced, that's all.
So beware of their ToS.
I disagree. It's not just the nuisance of wasted clock cycles. It also makes the network a juicy target for hackers. To anyone about to reply "you don't think people hack them now?", how do you think the correlation of attack sophistication and frequency looks for a network with/without a bunch of FREE MONEY inside? :)
> also some arbitrary financial technologies they don't like
I actually moved a business of ~100 FTEs from AWS to Hetzner once. Aside from the migration cost, the price was roughly 25% of AWS.
At the end, the biggest gain was not monetary, but human. For years, that business could retain skilled engineers who had the opportunity to work close to bare metal, caring about the nitty-gritty technical details of backups, failover and high availability.
And they did not even cost much. That they had so much leeway in designing the system instead of "relying on the cloud" was a major retainer.
I left many years ago, the business switched frameworks since then but they stayed on Hetzner.
P.S. Yes, that was before Hetzner Cloud became a thing )
My point is that even “something meaningful” comes with tech debt. It’s like that at my current place.
Too many people get “grass is greener” syndrome and think that there is some magical company somewhere which gives everyone plenty of time to refactor everything and fix all of the tech debt and execs make fantastic business risk decisions which always benefit the employee. In a world of scarcity, that practically never happens.
Just weigh your options in the market. If it’s worth staying where you are, just realize that the employee is not responsible for making business risk decisions, only responsible for sufficiently informing those who do of the facts.
But sure, maybe eating a broccoli will be construed as murder in the future, so best not eat anything at all.
But, backups will help if you replicated _bad data_, or more accurately _data changes_.
You can restore from backup if you accidentally ran `DELETE FROM foo;`, where replication will not help!
(Insert cryptolocker type viruses, bugs, human query mistakes, etc).
Backups are a specific point in time.
Are snapshots backups? snapshots on raid? snapshots on replicated disks?
Snapshots are not backups. Snapshots on RAID are not backups. Snapshots on replicated disks are probably backups, so long as the disks being replicated to are not inside the same case/building/city/continent (pick your risk suitably) and you're not able to delete the snapshots from the machine hosting the originals.
The second SIM in my phone provides a backup for my primary service provider, so long as I keep it activated. The torch in my pocket is a backup for the lighting in my house, so long as I keep it charged. My data in tarsnap is a backup, so long as I'm able to restore it. Which means data in tarsnap isn't a complete backup on its own: unless I'm able to recover the encryption key, I don't actually have a backup.
Snapshots can be backups depending on where they are stored usually not if stored locally. For example, RDS snapshot is backup for database going down but not account being deleted or region destroyed. Generally, snapshots are way to make backups to more durable medium.
But yes, if the problem is simply that the main setup is down, replication will often give you a more (or even completely) up-to-date copy than a daily backup will.
> And, does restoring from backups always mean losing more _recent_ data than replication?
This depends on the archiving technology and what you're archiving.
Our file and object stores take one full backup every day. This means, we could lose up to 24 hours of data changes on these stores if something happens within these 24 hours. If this is acceptable or not depends on the RPO - the recovery point objective, or the "maximum acceptable data loss". However, especially for documents, 24 hours can be acceptable, because users and customers do tend to have files they uploaded to the system around for a few days. Especially if you have a chance to identify the lost documents.
Both on MySQL with the InnoDB driver, as well as on postgres, you can use PITR backup solutions - point in time recovery. With this, pgbackrest or e.g. xtrabackup store a full backup of the database usually once a day at our place, and then keep archiving the WAL / transaction logs of the system. And we, in turn, archive snapshots of these into the longterm archiving once a day.
If we need a restore, we'd first restore a pgbackrest or xtrabackup state from the long term archiving onto a system. And then we can use the PITR recovery mechanisms to restore at a specific point in time.
Technically, we could precisely recover down to the last transaction before the disastrous transaction to minimize data loss. In fact, I've done so one or two times after some database migrations went haywire. That involved scrolling through transaction logs with a viewer to identify when the migration tool starts running, noting down the transaction ID of the transaction tool starting it's check and then restoring to the transaction before. Very cool tbh.
This is important for an RDBMS, because the data in the relational database tends to be much more volatile than the data in a file or object store. With a filestore, users upload a file and then move it to their recycling bin or their "done" folder on the local system and can easily drag it back out tomorrow. With the database, the user spent 30 minutes to an hour writing up some text or a comment and expects it to be saved and sound once they hit "Reply". Losing this kinda data creates a lot more work & effort for our customers, because then they have to figure out what state the data is in and what to redo. This may also cause their business processes to run haywire and... it's not great.
backup is a replacement of specified files required by a system recovery procedure. it may be a total image, or a collection of config, and dat files, that are daily bootup settings,
Besides, no mainstream crypto is mined anymore except Bitcoin.
So moot.
A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.
Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though.
Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.
A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions.
Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities).
In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that.
Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.
It is obviously a hostile language created by lawyers who did not spend much time researching the subject.
Of course it's unenforceable in practice and that is why a hefty chunk of Ethereum nodes are hosted on Hetzner for years and years with no problems.
A snapshot is a backup if a user deletes/edits their file and wants the old version. Raid is a backup if you're recovering from 1 disk failing
Wonder what's the algorithm they use to know a "real name"
On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus on their app and leverage the automated tools provided by the hardware and OS teams.
Cloud gives you at least half of that, or all of it depending on your approach, for a cost. There are points where the cost makes sense and times when it doesn't, and typically that changes through the life of a company. Unfortunately there is a not insignificant overhead even with current tools to maintaining a truly substrate agnostic infrastructure that can be deployed on top of multiple clouds, on-premise etc... so companies are locked in even when economics change.
You're assuming that "On premise" equates to "inside our building, in racks we've installed, using power and networking we have to manage." You're correct if that's the case for your business, but my argument is based around the idea that you can use _managed_ hosting providers of physical hardware that'll be either next door to you, in the same city, or close to your users (i.e, you're a business in Germany but your customer base is in London, so you host the servers using a London based provider.)
The idea that you have to manage hardware is greatly diminished when you consider the availability of managed providers that are dirt cheap.
Exactly. At which point, you’re essentially reinventing a cloud, usually not very well. If you have access to really good people you can pull this off, and that’s why you see so many people on HN doing the “who needs cloud” flex.
But the reality is that for most companies, managing non-trivial amounts of hardware is not a core competency, and they regularly shoot themselves in the foot by trying it.
Besides, no mainstream crypto is mined anymore except Bitcoin.
As I said, I haven't seen any actual responses here -- only non-response "responses". So are you saying they gave any different ones there? If so, where, specifically? Or if you're claiming any of the ones here are non-empty, which one(s)?
This is factually false.
> (some) Rental agreements, internet, telco and gym memberships are my favorite infamous examples. They're almost universally regarded as anti-consumer, with tonnes of sketchy clauses, but German lawmakers do nothing to improve that for the consumer.
Any examples here? The fact that contracts like these, if you forgot to cancel them, can only renew for one month is better than anything I've seen anywhere else. Also that you must be able to cancel anything online with the click of a button if the contract was made online. Add that to the fact that any clause is worthless if it includes something a reasonable person wouldn't expect. I don't know many countries that actually enforce this - Germany does all the time.
The problem is you always need to sue to get justice for that which means paying for lawyers and consuming time and money plus stress.
(Also in some cases, it's the other way around. If your landlord wants to increase the rent it's on them to sue you if they have a valid case.)
Do you have a source for this? (maybe it's a new thing) Because the subject of cancelling contracts is even a meme in the German (expat) community
(of course for your standard German you need to be able to plan your life years ahead)
Initial contract terms can be longer (up to 24 months) and as the site points out, the new rules only apply to new contracts, others can be up to annual.
I had Google take down my (negative but factual) review of a restaurant because of apparent "libel". There was basically no recourse (except "you can file a complaint but we'll probably ignore it"). I guess that explains why there are so many bad top rated restaurants.
I have the opposite perception. Most of the customer-screwing business practices I constantly see in other countries don't exist in Germany, because nobody even dares trying them.
And yet they're still put ion the rental agreement because the landlords know they can get away with it s it's a seller's market.
>I really have no idea what you're talking about here.
Google or look on reddit posts of foreigners getting screwed in Germany.
If your landlord puts something in the contract that is against the law you can sign it and simply ignore it.
It seems like many people/organizations belived that they would be rid of the whole "operations problem" once they shifted all their workloads from on-prem to cloud. They believed that they paid a full team for running cables and replacing broken fans/hard drives/PSU:s, when that aspect of on-prem is a tiny (but non-zero) amount of work.
OS level security? So, "apt update && apt upgrade", then? I mean, what else are you doing, writing patches for the kernel? Checking every line of code that runs? Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level? Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.
There's a Terraform provider for Proxmox, which is an excellent hypervisor. Making a template takes less than an hour with configuration.
You do need an Ops person for sure, but an entire _team_?
Across 10k-100k+ servers, all running services and needing to orchestrate restarting across the whole fleet, while providing 0 downtime or impact to thousands of clients with terabytes of data being processed and analyzed at any given time.
Sure whats so hard about changing a tire? Well try to do it on an 18-wheeler while its driving down the highway without any impact to its speed.
> Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level?
Part of a layered and in-depth system but one that introduces complexity.
>Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.
Tailscale in an enterprise production environment? Not going to pass any sort of security audit and probably violates a number of certifications customer require at the enterprise level for network access controls, visibility and auditing.
Just managing the git/jenkins/spinnaker/terraform infrastructure in dozens of locations deploying to and maintaining tens of thousands of servers/pods requires a 24x7 team on top of the hundreds of teams and tens of thousands of devs using it.
If you're small enough that doesn't make sense, then you might be small enough one Ops person can handle the load (One is never enough if you're smart but...), but you are dealing with a very small amount of infrastructure and services at this point.
If you "need" that many servers (and aren't Google), you've built your systems massively wrong.
My issue is really on the other end of that scale, where getting C-suites to recognize when owning that core competency is actually beneficial to the company even if its not the focus of the company.
I grew up around companies leveraging vertical integration at the right scales to improve costs, seeing companies go the opposite direction trading all those advantages for often never-materializing benefits is... frustrating.
> Regulations for fairer consumer contracts are on the 1. March 2022 came into force and ensure that you can terminate automatic contract renewals for contracts for regular goods deliveries and services (such as streaming services or magazine subscriptions) more quickly.
> In addition, the 1st was founded. July 2022 a termination button duty introduced to simplify termination processes.
Or simply because the alternative is being homeless?
And why should the default for foreigners be getting screwed?
As I already mentioned, you can simply sign a contract and then proceed to ignore all the illegal clauses. They're not binding.
> And why should the default for foreigners be getting screwed?
People getting screwed because of them not knowing their rights is basically something that can happen in every legal system, and if people come from other countries without certain legal protections, they're more likely to not know about them. That's just a reality of life.
And if it's about the apartment, you've probably paid a deposit. And if you ignore some of the clauses, they will probably try to get back at you and punish you by not returning the deposit (or part of it). From here - GOTO 1 ;(
This is total nonsense.