Same Job, Two Engines: InnoDB and Iceberg Internals(marcpaul.io) |
Same Job, Two Engines: InnoDB and Iceberg Internals(marcpaul.io) |
Rows store all columns together
Not guaranteed. The DYNAMIC (default) and COMPRESSED row formats [0] can shift larger values of variable-length columns in off-page records. In the example given, this wouldn’t occur, but it isn’t a guarantee per se. This is roughly equivalent to Postgres’ TOAST, though Postgres’ default threshold for TOAST is 1/4 of a page (2 KiB with its default 8 KiB page size), whereas InnoDB is 1/2 of a page (8 KiB with its default 16 KiB page size).
0: https://dev.mysql.com/doc/refman/9.7/en/innodb-row-format.ht...
Suggested next steps would be trying out DuckDB instead. My guess is it would smoke spark etc at this scale. Always good to learn different corners of the data landscape.
Everything I've read says you're absolutely right about the performance boost lol. It very well may be the next piece I check out, thank you.
I swapped out some data analytics processing that was originally implemented with Python & Postgres to DuckDB and have been immensely pleased with the outcome. Stuff that would take 6+ hours to do with Python/Postgres now takes me around half an hour to complete.
It's a great addition to an existing Postgres database as it has very tight integration with it's native Postgres extension, which allows you to do things like select data from your Postgres database, process it locally with DuckDB, then insert back into your Postgres database.
I'd highly recommend giving it a try.
One of the more interesting talks I read about was how someone used MySQL's bin log to mirror a lakehouse different versions of data and used DuckDB as the compute layer of a lakehouse. I think the talk was held in South America, so I didn't get to go.
Postgres is just useful for pretty much everything it seems like lol. Knowing they work together so well makes for a side project to learn both.