Lightning Memory-Mapped Database Manager (LMDB) 1.0

Lightning Memory-Mapped Database Manager (LMDB) 1.0(lmdb.tech)

45 points by radiator 2 hours ago | 17 comments

hmry 1 hour ago |

Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class.

I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author also shared some unconventional opinions about undefined behavior (like "Compilers are deterministic, if I know what platform I'm compiling to then no behavior is undefined. And if compiler authors disagree, they are morons.")

But presumably it's thoroughly tested, so those aren't problems in practice? Would be really interested to hear from people who've actually used it. I've mainly stuck to SQLite instead.

markasoftware 57 minutes ago | |

Not amazing. In certain workloads I ran, once the db reached several hundred gb, writes would hang for longer and longer periods of time, eventually hours, while the db grew drastically in the background. https://news.ycombinator.com/item?id=30023623 seems to be the same issue, and it was serious enough that Shopify decided not to use lmdb.

And yes, I ensured there were no outstanding long lived readers, verified with mdb_stat -r. My workload used one transaction per read/write anyway (never needed larger atomicity). Once the db got into the bad state, running my program on it would almost immediately run into the issue again, so I really think the db is in a bad state such that most writes would cause it to hang, not related to how I do transactions. This workload would pretty consistently hit the issue once the db got to several hundred gb.

Issue #10236 on the OpenLDAP bug tracker might be the root cause, who knows. It's been marked CONFIRMED for years without a fix, while other similar issues are created.

This is extremely annoying. It seems workload dependent (other workloads I've run create absolutely massive lmdb dbs without this issue) and once it happens your only recourse is to make a new db and copy the contents over (thankfully reads still work fine on these borked dbs).

Other than that, though, it's great. Never in any case had actual data corruption, and reads and writes are extremely fast (until this issue happens)

Edit: fun fact, since shopify may have created Bolt in response to this bug, and then Bolt was the root cause of the 73-hour Roblox downtime in 2021, this bug may indirectly have caused one of the worst outages ever!

ChrisTrenkamp 55 minutes ago | |

I can't go into specifics, but I use LMDB for the commandline application I maintain for my employer. I also extended it into a web service for internal use. As long as you stick to the safe LMDB options, which are the default options, it's reliable. The documentation clearly outlines what safety guarantees you lose when you enable/disable certain options: http://www.lmdb.tech/doc/group__mdb.html#ga32a193c6bf4d7d5c5...

I had a situation where the web service's writes were slowing down to an unbearable crawl because the number of entries in the database were reaching tens of billions of entries. Thankfully, the users never experienced the slowness. The website stayed nice and fast, even though the background updates were extraordinarily slow. The issue was fixed by sharding the databases.

thombles 1 hour ago | |

Be cautious if you're using large databases on iOS. At least until fairly recently, iOS doesn't page dirty mmaped pages back to disk and after enough churn the app will OOM.

radiator 1 hour ago | |

It has been used successfully as the backend for OpenLDAP and Monero, at least.

packetlost 1 hour ago | |

I believe at least one of the two official Minecraft implementations use it for their map/save format.

ozgrakkurt 59 minutes ago | |

It is a small amount of code so easy to integrate into an application.

It is really reliable except write performance in my experience.

Author of it writes very spicy stuff and sounds pretty rude.

I would recommend doing a prototype with real data scale and testing if it meets your requirements. The write performance can be really atrocious and It doesn't have a high performance potential because it is based on memmap.

hilariously 1 hour ago |

Maybe rephrase this part - "It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database."

I generally do think read-write mode would offer higher write performance than read only as well :)

radiator 2 hours ago |

New features in LMDB 1.0 include:

- support for incremental backup

- support for page-level checksums and encryption

- support for DB on raw block devices

- support for 2-phase commit

- support for page sizes up to 64KB

plus other minor additions to the API.

paveworld 1 hour ago |

HTTP ?? Com’on man

radiator 1 hour ago | |

it is just a link to documentation

jayct 48 minutes ago | | |

that could easily be trojan-horsed with links to malware if you are viewing it in a poorly secured setting (like public wifi), because you can't verify the origin. so the best we can say about the author is that we are getting inconsistent signals on how seriously they understand and implement security concerns. so better review that code carefully before use, rather than assuming their expertise from release notes.

Retr0id 1 hour ago | | |

TLS certs are freeeeee

quotemstr 14 minutes ago |

I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd.

You can seal memfds too, which means that the "read-only" mode is easy to implement: just map your memfd for write, apply F_SEAL_FUTURE_WRITE, and share the memfd to anyone you want to have read-only access.

By doing your own O_DIRECT IO instead of relying on the kernel's defaults, you get a lot more control. You choose how much readahead to do; you choose your read-cluster size. You choose your cache eviction strategy. You choose when to write back.

BTW: O_DIRECT can also be done asynchronously using aio or io_uring. There's no such thing as an asynchronous page fault. And IO errors? Would you rather deal with EIO or SIGBUS?

Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.

And it's not any faster either. O_DIRECT is DMA. A page cache fill is also DMA. It's the same operation, spelled differently.