This whole post is basically bullshit, "secure" transporation meant taking a random electric car (renault zoe) at the company headquarters or at DC5 and throwing hardware in the trunk.
No security in place whatsoever, servers laying in datacenter hallways fully loaded with disks, accessible to anyone.
Most of the company had access to the back office where they could just access customers' data without any kind of logging. (Internally this was called sudo mode on the online console, we had basically to click on a button to log as any customer).
The "funniest" was the corporate vpn network being shared by the internal datacenter network which meant any employee had, for instance, full access to all the home made switches management interfaces which had no access control whatsoever, it happened that employees stumbled accross this and wondering what it was (LOL), there was even a button to upload and flash (!) a firmware.
The upper-management was terribly incompetent and would discard any kind of issue that was not affecting sales in the immediate term as not important (security or otherwise).
I remember this because I used fscrypt after seeing the post, Scaleway seems to have removed the entire user forum sometime last year.
I stopped using their service after they removed their ARM severs with short notice, But this is more scary.
Unfortunately this is the reality when it comes to AWS[1] or other top-tier cloud providers vs inexpensive ones. The cost saving has to happen somewhere.
[1] https://aws.amazon.com/blogs/security/importance-of-encrypti...
What about their access control policy and network access control policy?
Do you know who their certifier is, as they don’t say, so I can ensure I never, ever trust them?
The whole ISO27001 thing is false advertisement, the scope is very restricted and scaleway, online.net, dedibox or whatever are NOT ISO27001 certified.
The certification only applies to a very restricted range of products which are not even pubicly available.
> What about their access control policy and network access control policy?
A joke, many critical services were only firewalled to the outgoing ip address of their corporate wifi network which practically anyone entering the building was given the password.
The private network on online.net dedibox offering was just ACLs that were applied with a loop on every switch without much error handling it happened that what was displayed on the console was not the real configuration applied to a switch. What a joke.
And this is just a starter, this kind of things goes on and on. Scaleway is a terrible company.
> Lechelle said Scaleway worked with the YouTuber to recover the disk. The French-language video creator has written to Scaleway with assurances they have not copied the information contained on the disk. It is said some customer data was on the drive, unencrypted, including the source code and SSH keys of an Italian VPS provider.
The YouTube video is in French without English CC subtitles, but there's screenshots of exploring what I assume would be one of the qcow images: [2]
* 03:59 `/root/.ssh` with SSH id_rsa (private) keys, (public) authorized_keys and (hashed) known_hosts
* 07:14 server binaries
* 08:42 server logs
* 12:59 AWS/S3 secret access keys
* 13:33 Source code... in QBASIC ?! must be very valuable legacy code :D
* 17:30 brute-forcing the ssh known_hosts
* etc
[1] https://mobile.twitter.com/Micode/status/1395640486715662336
are there OSes that perform quick format without issuing TRIM command to the SSD? TRIM would vanish the data in less than 15 minutes.
"Requirement for CISP:
(a) Security measures
The CISP will implement and maintain appropriate technical and organisational measures for the CISP’s data centre facilities, servers, networking equipment and host software systems that are within the CISP’s control and are used to provide the CISP’s service (the CISP Network). Those technical and organisational measures should (a) be designed *to help customers secure personal data against unauthorised processing and accidental or unlawful loss, access or disclosure*, and (b) address the security responsibilities of the CISP as set out in Annex A (Security Responsibilities)." [1]
I do not know about others certifications, but this situation seems to be a clear violation of one of the requirement for CISP. Answer to this requirement is disk encryption. Moreover they are authorized to store medical records and data. I can't imagine that they do this without providing proper disk encryption. in the light of this event, I'm not sure they qualify for all these ceertifications.
It’s possible scaleway aren’t certified, and are lying about it. Maybe they had an auditor in, maybe not, maybe they have a boatload of nonconformities that they’re not documenting (which you must, if you have them).
Aka they could no longer sweep it under the rug.
Drive encryption is pretty standard these days, isn't it?
Can anyone speak to the effort in doing it at the OS level?
A lot of hosting providers actually say that their data centers are ISO 27001 compliant, but that doesn't make the provider itself compliant
(and it seems scaleway is still doing that too on https://www.scaleway.com/en/datacenter/)
- Why was the disk being replaced (SMART warnings? HW upgrade? It failed in some way)
- Why was the quick format recoverable? Was TRIM issued or the HW didn't support it?
- As a follow up, I'm guessing the HW didn't support HW encryption?
- Why are they relying on 3rd parties for the destruction of data?
Meanwhile, it seems that HDs don't leave Google datacenters in one piece
> Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
- Andrew S. Tanenbaum
If the SSD was stolen over one year ago why do they only acknowledge it now?
In March 2021 [1] they were writing about how great their security was:
> We are proud of our data centers and their security. We consider that we have implemented the best solutions to protect your most valuable asset: your data. We are well aware of the huge responsibility this represents. There can be no compromises when it comes to your data.
Why were customers whose data was leaked only informed in June 2021? The delay of over 1 year is a huge GDPR issue.
The timeline of the incident from public sources:
21 May 2021: Micode tweets a screenshot of the directory listing [2]
26 May 2021: First video on the subject [3]
6 June 2021: Scaleway customer is notified their data was leaked [4]
21 July 2021: Second video on the subject [5]
24 July 2021: Third video on the subject [6]
24 July 2021: Scaleway releases a French blog post stating "Over a year ago, an SSD was stolen" [7]
Storing the data unencrypted is bad, but IMHO Scaleway's handling of the incident creates much bigger questions about their credibility.
[1] https://blog.scaleway.com/how-we-protect-your-data/
[2] https://mobile.twitter.com/Micode/status/1395640486715662336
[3] https://www.youtube.com/watch?v=vt8PyQ2PGxI
[4] https://www.lowendtalk.com/discussion/comment/3258386/#Comme...
[5] https://www.youtube.com/watch?v=aOBVZUL1iBA
[6] https://www.youtube.com/watch?v=xf_cKTlOYLo
[7] https://blog.scaleway.com/incident-securitaire-video-youtube...
Previous discussion: https://news.ycombinator.com/item?id=27957471
If just their data centres are compliant, I guess there’s nothing to stop their staff or contractors just stealing data, as they have here - as compliance purely around physical security for a data centre is 100% meaningless if you’re just ignoring cryptographic controls and human security controls.
I’m guessing their certification body isn’t themselves certified. Actually, the fact that they neither say who their certifying body is nor include a link to their certificate, suggests to me that they are lying - which is far more common than you would think.
I don't think many of the competitors in this space (eg Linode and DigitalOocean) own their datacenters either.
As Linode put it: Linode as an entity is not ISO27001 and we rely on the data centers we colocate with to obtain and maintain these certifications
Or DigitalOcean in 2017: Our FRA1 facility is ISO9001:2008, ISO27001:2005, and ISO22301:2012 certified.
(DigitalOcean later got their own certification)
There should be NO DATA on the drive if it was really formatted on OS supporting TRIM.
Anyway, what they should at least is just issue `blkdiscard /dev/sdb`, or betterly Secure Erase (on SATA) or Sanitize (on NVMe).
Entering (and managing) that key is the hard part.
Disclaimer: I still didn't try it myself yet, but it's on my (too long) list of things to do.
While I'm not going to defend moving around unencrypted disks it is not as simple as that.
The difficult part is not the encryption. The difficult part is the key management of the encryption. You need to solve issue like ensuring you don't lose access to the keys (and thereby access to the data), securely providing keys to people and machines that need access, dealing with key revocation when people leave the company, etc.
Though I guess in this case using a simple (long enough) passphrase instead of some form of keys could have saved them from this particular threat.
Each system could reach out to other storage systems to ask them to use their private key to decrypt the header (for example if their hardware security module had failed), but in some configurations this would require an operator to intervene to enter in passphrase to unlock the hardware security module to authorize the action.
This means that:
1. The symmetric cipher key could always be recovered, even from paper backup
2. Having physical access to a disk or any set of disks did not allow you to read the data
3. Having physical access to a disk and a hardware security module did not allow you to read the disk (unless you knew the passphrase, which was always present, and user set)
4. Having physical access to disks, servers, and hardware security module may not allow you to read the data (in the more secure configuration where passphrases were not cached on disk -- but this meant that rebooting required an operator to manually enter the password at boot)
5. The set of valid public keys could be changed frequently (this was indeed automated and only the set of currently active hardware security modules could decrypt the current disk header)
Of course you could always short circuit this by making a copy of the symmetric key when you did have access (e.g., `dmsetup table --showkeys`) but without putting some more hardware in the fast/hot path that was unavoidable. The symmetric key could not easily be changed, without rewriting the entire disk (though since it was a storage system designed to accommodate multiple failures, this wasn't that hard, but we didn't do it automatically)
The hardware security modules we used were FIPS 140-2 validated and physically connected to the racks (though it would be possible to cut them away). It would also be possible to spy on the APDUs sent to the modules to capture the decrypted data, since there was not mutual authentication (it was in the works though).
This is the level of redundancy/backups/processes the large cloud providers have. It's not just "hey is this encrypted" – it's "what happens, how do you retire a drive, are the modules FIPS 140-2 compliant, how do we physically secure our HSMs?" etc
It's just so much more than "hey Digital Ocean is 50% cheaper"
AWS's egress bandwidth charges still feel criminal though.
Yet another reason to just stick with the big providers. They seem to have pretty serious compliance needs and encrypt no matter what. (Yet you should still encrypt your assets, but I believe they all encrypt their volumes anyway and have pretty strict policies on how to dispose this kind of hardware)
However for a server, the chance of a thief getting a disk and the HSM at the same time is low; and you really don't want to have to enter a password every boot. So the HSM design delivers some benefits worth having.
Whereas for a laptop, you can easily type in a password; and the chance of a thief snatching the disk and TPM at the same time is basically 100%. So there hasn't been a big push among Linux users to start using the TPM.
I also used the same software on my laptop to encrypt it. It's nothing fancy, just using the dm-crypt kernel module and any kind of hardware security module that talks PKCS#11 (which is all of them); On my laptop I just used my existing smartcard (which I used to login to the system and remote systems).
The disk header was just text occupying the first 4MiB of the disk in 2 circular buffers similar to LVM (though 2 copies for redundancy).