Full-Text Search with DuckDB(peterdohertys.website) |
Full-Text Search with DuckDB(peterdohertys.website) |
I build a whole LLM benchmark system around it that lets you run the whole benchmark in your browser: https://sql-benchmark.nicklothian.com/#sample-queries-and-sq...
Click on a cell and you can run the SQL the LLM generated vs what the solution is: https://sql-benchmark.nicklothian.com/?highlight=ggml-org_ge...
The hardest part is getting people to understand that it is interactive! People expect a document-looking webpage to be static, but we can do so much better!
There is a V2 branch with some minor fixes etc I'm working on.
You can point DuckDB to almost any data source and boom, you get an SQL table that you can search, sum, or join to any other data. Or you can attach existing databases from completely independent db systems, and query and join them as one, without having to first importing anything.
It feels exhilarating (if you're into that sort of thing!)
Replying to myself to add this caveat: as I found out the hard way, the limit of such a setup is that query performance depend on the performance and indexing choices of the underlying data stores.
If you have no control over those, and you don't have real time requirements, you're usually better off first importing everything into duckdb (which is usually quite fast) and doing your queries from there.
EDIT: "drop-in replacement like SQLite", not "for SQLite".
While SQLite is often used for comparison (“SQLite for OLAP”), I’ve never seen DuckDB market itself as a “drop-in” replacement. Where did you see that?
It’s not THAT hard to do if you’re not afraid of compiling in the C/C++ ecosystem. But people should be aware that a lot of stuff is still really nascent and in active development in the extension ecosystem. I wish more extension authors made at least nightly builds available, that would save a ton of pain and drive a lot more adoption IMO
https://duckdb.org/docs/current/extensions/overview#autoload...
I would love to have more detail on this mechanism.
all are by the DuckDB team except three third-party owners. I’m unfamiliar with Vortex, but presume it’s like LanceDB and MotherDuck with a serious company behind it. and presumably the DuckDB team trusts them not to ship malware in their extension
I think it’s a UX trade off that benefits users with minimal security downsides. and you can configure this behavior. some docs here: https://duckdb.org/docs/current/operations_manual/securing_d...
Also creates fun situations like getting on a plane then realizing that your extension isn't available!
It seems that nixpkgs at least fails to run the extension but more by luck than design. I hope they find a way to vendor the extensions locally.
I'm aware of jmail.world, but they haven't (yet?) published the source code.
I had Claude hack something together recently: https://healdsburg-youcubed-emails.vercel.app/
It works fine for this small set of emails, although the search isn't great, and there was more preprocessing that I would have liked. (I would prefer to be able to point a single binary at a pst or mbox file, and have it magically serve it like this, even if it means I need a VPS to serve it.)
msgvault seems really good. The tui is fast and FTS5 search works well.
I will definitely use it.
But it doesn't allow me to make a mailbox accessible to a wide audience, because:
- AFAICT there's no web version
- inline images don't show up
https://github.com/nraynaud/sojourner/blob/main/PDS/db_creat...
Example: https://nraynaud.github.io/sojourner/PDS/explore.html
Clickhouse seems less marketed, but seems quite similar.
These emails aren't published by default but email archives are often included in responses to public record requests.
Ideally anyone who receives one of these archives would be able easily inspect it themselves, and also make it available to others.
It's WASM plus client-side React components in a static site.
I'm hosting of Cloudflare Pages (without their serverside function hosting), so yes GH pages will work fine.