SQLite should have (Rust-style) editions

SQLite should have (Rust-style) editions(mort.coffee)

84 points by gnyeki 2 hours ago | 35 comments

sethev 1 hour ago |

Interesting idea - I like seeing a list of pet-peeves followed by a proposal for a straightforward way to have a set of 'alternative defaults' that remains backwards compatible. If you don't want to opt in, don't run the new PRAGMA edition = 2026.

Too often it's just a list of issues and a wish that everyone else will change.

In (mild) defense of SQLITE_BUSY - busy_timeout just tells sqlite to sleep and retry up to the timeout when it receives SQLITE_BUSY. It seems like a sensible default for a library to leave that up the calling code - which may have something else it could do while it waits. However, that logic often gets missed!

tptacek 1 hour ago | |

This isn't so much a list of pet peeves as it is the almost universal way people that work seriously with SQLite configure the database. It's reasonable to suggest that the alternative settings for each of these suggestions is probably the wrong default for 2026.

sethev 1 hour ago | | |

Yes, agree. These are very sane defaults and match what I use..

groundzeros2015 31 minutes ago | |

Yep. The whole locking database thing is this persistent myth about SQLite. All databases lock on write, it’s a question of the granularity of the lock. Multiple writers simply take turns.

kccqzy 2 hours ago |

SQLite is slightly different from Rust in that it is a data container. It’s somewhat more common for people to move SQLite database files from one machine to another and then inspect using the command line tool. And it is often the case that the embedded SQLite version in your app is a newer version than whatever version /usr/bin/sqlite3 happens to be. Adding editions to your SQLite file will probably break this use case of using an older version to read a database written by a newer version because it does not know what has changed in a new edition.

Not a big deal though. Probably just need better ops to bundle the command-line utility that’s the same version as what’s used in your app.

Rendello 1 hour ago | |

> SQLite is slightly different from Rust in that it is a data container.

I think this is the key.

From sqlite.org [1]:

> [Since 2004], the file format has been fully backwards compatible.

> By "backwards compatible" we mean that newer versions of SQLite can always read and write database files created by older versions of SQLite. It is often also the case that SQLite is "forwards compatible", that older versions of SQLite can read and write database files created by newer versions of SQLite. But there are sometimes forward compatibility breaks. Sometimes new features are added to the file format

---

Given editions (A) and (B), what does backwards compatibility look like? Must (B) be backwards compatible with (A)?

If yes -> editions are backwards compatible but not necessarily forwards compatible, which is the current status quo:

    -------(A)----(B)--

If no -> editions are not backwards compatible, the edition space is bifurcated:

    ---+----(A)--------
        \
         \--(B)--------

Now you may have to worry about backwards compatibility with (A)..(Z). What happens when you import a file from edition (Y)?

1. https://www.sqlite.org/formatchng.html

---

Interesting PS, grepping sqlite.org for "backwards compat": https://pastebin.com/Q7b7h4eM

mort96 57 minutes ago | | |

The answer to the question "What happens when you import a file from edition (Y)" should, ideally, be exactly the same as the answer to the question "What happens when you import a file created with parameters foo, bar and baz set to values a, b and c". There's really nothing new here.

Some parameters are properties of the database file itself, such as (I believe) journal_mode. These parameters probably result in issues with earlier versions of sqlite; if you create a file with journal_mode WAL, you're not gonna be able to open it in a version of sqlite without WAL support. Same with strict tables; if you have a database which contains strict tables, I assume you're going to encounter issues if you try to open it in a version of sqlite too old to support strict tables. But anything new enough to support the individual database file features shouldn't have any trouble. This is true whether we're talking about an edition pragma or individual parameters set the old way.

Some parameters are properties of the connection, like the foreign_keys parameter. If you're trying to open a database file in a version of sqlite which doesn't support foreign key validation at all, just don't set the foreign_keys parameter.

tptacek 1 hour ago | |

For some of these pragmas you have the same issue with or without "editions", right? Busy timeout is per-connection. And then: if you're running in WAL mode, you, the user, have to know that, or risk messing up the database by copying just the .db file rather than vacuuming-into.

kccqzy 1 hour ago | | |

Editions make the problem worse by requiring the version not only to support the underlying pragmas but also to understand the edition mapping.

Example: PRAGMA foo=1 is introduced in 2027. PRAGMA edition=2030 implies this foo pragma. Now you unnecessarily lock out three years worth of releases.

ijustlovemath 1 hour ago | |

editions could be self-describing as to certain semantic changes, and you could embed that with the file. older versions could safely ignore it and newer versions parse and run it. you could also force things like "editions must be declared early", "editions are one way only" etc to get some level of security in the adoption of the change

arijun 1 hour ago | |

I think you can maintain full backward compatibility with all these changes. SQLite has a bunch of meta-data that you can use to see that no earlier editions have modified your database since you last wrote. You make the new editions follow these strict rules, but if an old edition modifies the DB in the interim, you fall back to the original rules, until it’s verified compliant again.

appplication 1 hour ago | |

It seems SQLite could be evolve to solve this by just bundling itself entirely in the data files? After all, the binary is less than 1MB, anywhere you’re putting a database surely has at least that much overhead, for most applications it’s less than a drop in the bucket.

I’d be interested to learn if there are any db implementations that take this approach, or reasons this wouldn’t work.

mort96 1 hour ago | | |

Well you'd have the problem that an sqlite database file created on a Linux AMD64 box could be copied to an AArch64 macOS machine to be read there. And quite a lot of (cross platform) software build their file formats on top of SQLite.

arijun 1 hour ago | | |

So for every new SQLite db you want to access would have to run a new untrusted executable? That seems… hilariously bad for security.

andai 1 hour ago |

The "use strict" thing is interesting. I often hear people say, well we can't fix absurd behavior in JS because backwards compatibility! Well, we already did, and we can do it again!

ShinyLeftPad 1 hour ago | |

"use stricter"

chillfox 10 minutes ago |

you can also set those defaults with compile time flags, that's what I have been doing.

Thaxll 1 hour ago |

SQLite gets so much praise here but when you start using it, you realize quickly how bad it is, the type system is by default very limited and dangerous.

It's like comparing old php with a strongly typed language.

There is not even a date type...

groundzeros2015 30 minutes ago | |

SQLite competes with fopen. Not Postgres

kccqzy 38 minutes ago | |

It’s not as bad since you can always use a powerful programming language with a good type system that avoids type errors at the SQL level. You can build good abstractions in your programming language.

souvlakee 59 minutes ago |

Sadly, the ORM layer lags here: Drizzle has no way to declare STRICT tables. The request has been open since March 2023 (issue #202, now discussion #2435) and didn't make the v1.0 beta either. The only workaround is hand-appending STRICT to generated migration SQL, which doesn't work at all if you use `drizzle-kit push`.

https://github.com/drizzle-team/drizzle-orm/discussions/2435

mort96 2 hours ago |

Oh hi, author here. Fun to see this make it to HN.

simonw 1 hour ago | |

Suggestion: post this on https://sqlite.org/forum/forum - the SQLite team monitor that forum closely and I've had some really great answers from them to questions or suggestions in the past.

mort96 1 hour ago | | |

I haven't really ever participated in that forum before, but it's an interesting idea. I don't know if I'm going to do it, I might. Though I also wouldn't mind if someone else posted about it there. I might even make a forum account and participate in the discussion.

quadhome 1 hour ago | | |

https://lobste.rs/c/kzln1c

andai 1 hour ago |

In the first example, there's a a second thing that surprised me: you delete an entity and it's unique ID gets reused? Is that a good idea?

I guess if foreign keys are handled properly then that's not a problem by definition? But it sounds wrong somehow.

deepsun 12 minutes ago | |

I think that's a security vulnerability.

If a parent table ID gets reused, then it's a potential to expose data to a wrong user -- security broked.

Rendello 1 hour ago |

It might be worth bringing this up on the forum [1]. The developers are quite active there, and it's possible they've never considered this option, or they have considered it and have reasons to not go for it. The original design followed Postel's Law (see my comment from the other say [2]), it would (theoretically) be nice if that mess could be avoided by specifying an edition.

Today I noticed I could do `pragma foreign_key = ON`, and despite the pragma being wrong (it should be foreign_keys, plural), it reported nothing. In fact, it reports nothing with the correct pragma either. So check your pragmas!

1. https://sqlite.org/forum/forum

2. https://news.ycombinator.com/item?id=48900625

Polizeiposaune 1 hour ago | |

The Postfix mailer has allowed recommended default behavior to evolve using its "compatibility_level" parameter:

https://www.postfix.org/postconf.5.html#compatibility_level

https://www.postfix.org/COMPATIBILITY_README.html

You get a warning whenever you depend on the deprecated old default until you either move forward or specifically commit to the old behavior.