Ask HN: We just lost 100s of files on GCP, is there any way to recover them?

12 points by OtmaneBenazzou 3 years ago | 5 comments

Hi, due to an (yet) unidentified bug on our Firebase storage/Google Cloud infra/buckets, we've lost hundreds of files in the last few days(it was spotted just a few hours ago).

There is no backups of all the files(I know...I did not decide that)

I'd guess Google would probably be legally required to not wipe everything instantly for legal/forensics purposes, right? So is there any way to somehow recover those files?

Does any Googler or GCP user here knows if there is any kind of way of recovering those files? They are pretty critical to our core product.

Thanks in advance

albert_e 3 years ago |

Not familiar with GCP but in AWS S3, if the bucket has versioning enabled -- any simple delete operation only places a "delete marker" on that object (soft delete).

Details for S3 -- https://repost.aws/knowledge-center/s3-undelete-configuratio...

If there is an analogous feature on GCP -- it might come in handy?

Does GCP support / account team for your organization have any suggestions?

If versioning is not enabled, even Amazon S3 does not offer a way of recovery in most cases. If it was a malicious attack or slip-up on AWS side -- there may be a possibility of at least escalating it via account management to get some engineering teams to investigate it deeper and attempt a recovery.

Someone 3 years ago | |

GCP has versioning, too (https://cloud.google.com/storage/docs/object-versioning)

Because you still pay for storage of ‘deleted, but recoverable’ files, it isn’t enabled by default, though.

pyeri 3 years ago |

The importance of regular backups on production couldn't be stressed enough. Even in the old days of XAMPP and millennial tech, folks used to take regular mysql backups because they knew that even if the tech stack was robust, the hosting providers can raise their hands any moment.

With Google and Amazon AWS nowadays, we have this increased sense of carefree attitude thinking that these deities will take care of us and our data whatever may come. But remember that deities can fail too at times, they are just another technology company in the end, only with a bit more stock value and narrative power than others.

CharlieDigital 3 years ago |

Interesting. I had this same issue occur a few weeks ago and chalked it up to user error. An entire folder (subpath) in a bucket got wiped out. Code review did not turn up anything that would cause a full path wipe.

Also using Firebase APIs for Google storage.

I really like Firebase, but this really made me wonder weather I should switch over to AWS S3 for the storage.

temikus 3 years ago |

Depends on whether they threw out the key and crypto shredded. Contact support immediately and keep escalating like your life depends on it. You _might_ be able to recover it but chances are slim.