Fixing Ext4 Under Pressure

102 points by vitplister 2 years ago | 14 comments

mike_hock 2 years ago |

Let me guess, the server has been running with metadata checksumming (and the mystery feature) disabled since that incident and it still hasn't been switched on again.

umanwizard 2 years ago |

0x10000 in the mentioned bitfield is EXT4_FEATURE_RO_COMPAT_ORPHAN_PRESENT which is documented here: https://www.kernel.org/doc/Documentation/filesystems/ext4/or...

If I’m understanding this correctly, probably the only effect of having accidentally turned this off is leaking some blocks.

captn3m0 2 years ago |

I would have considered re-compiling tune2fs with the check disabled, but this is a cool hack.

fragmede 2 years ago |

And people say "just do it on-prem, it's cheaper than cloud! It's not all that hard." Work like this is impressive as hell, but finding wizards who can do this kind of work is difficult and expensive. I've sat there are poured over hexdumps trying to get it to reveal it's secrets (Kaitai is pretty cool). But this kind of nitty gritty went away with cloud, to be replaced with a different kind of problem, distributed systems. My point is, that's pretty cool, but there's less demand for that kind of work these days, which is unfortunate, because I kind of liked it.

lathiat 2 years ago | |

Except the same thing might happen to your ext4 cloud instance and they’ll just tell you it’s lost.

awskinda 2 years ago | | |

As with most things, I think the answer is “it depends.” I have seen some very efficient storage systems designed around S3/R2 object storage where applications run in ephemeral containers, and storage is abstracted.

I have also played a role in running an on-prem SAN that worked well mostly.

fragmede 2 years ago | | |

The RDS database in the cloud that is backed by ext4 is trivially backed up (at great cost) to S3 which isn't going anywhere. S3 has 99.999999999% of durability, that is to say is eleven nines. They're not going to tell you it's lost.

chgs 2 years ago | |

Or he could have made backups like we’ve been doing for decades

> I didn't have any recent PostgreSQL backups.

LordN00b 2 years ago | | |

That would have been the easiest/best solution here. However...haven't they uncovered a limitation in the filesystem here? The superblock data WAS fine, only the checksum was at fault. They found a away around the issue, wrote up their findings, suspicion about the flushing after a resize, and asked for more tooling support. This is classically a good blog post.

mistrial9 2 years ago | |

management systematically replaced people like that as a) expensive b) rivals in power. The idea to build replaceable-parts operations by workers, instead of University skill level super-admins, was by design and implemented with plenty of capital behind it.

applied_heat 2 years ago | | |

This is what any business does and what you would do with your own growing business that you one day will no longer own too. You need defined roles with responsibilities and required skills that you can hire people to fill as people come and go in their own lives. Sucks to be a cog but cogs we are