Queryable Executables(fzakaria.com) |
Queryable Executables(fzakaria.com) |
...basically `s/zip/sqlar/g` and you're not that far off.
SQLite as an application file format - https://sqlite.org/appfileformat.html
...basically instead of `*.docx`, you'd do something like `UPDATE pages SET content='...' WHERE number=1` (or something). It's a compelling idea!
CouchDB/CouchApp - https://couchapp.readthedocs.io/en/latest/intro/what-is-couc...
...about 2 decades before its time. Think mongo/redis document store with a built-in concept of offline sync + replication. Survived a bit by `pouchdb` which was slightly simplified and focused on mobile apps.
Views and indexes were `*.js` functions run on every insert/update, and "apps" were basically `*.html` + `*.js` served out of the database directly. Think `address-book.couchdb` which contained `index.html` and `index.js` (and `/admin/...` routes) which "knew" how to render + edit the data itself.
Prior to that was the glorious moment when PHP began including sqlite drivers by default and you could basically do: `php something.php mydatabase.db` and have a full/hermetic browser-app experience without needing to stand up an apache server or mysql database to connect to.
"Data that knows how to edit itself" (what was that weird new-fangled parquet data encoding format or whatever) is an AWESOME idea! Maybe Mr. McCarthy was on to something when he kept bringing up "homoiconic" (or is this "monomorphism"...)
If you're listening, I propose: `*.sqlitexe`
Native applications ship as executable files. These files are basically a combination of machine code instructions (the program logic to be run) + a bunch of extra data that needs to be loaded into memory for the program to run + metadata so the operating system knows how to combine it all.
The first article noted that the file format for this can be thought of as a very specialised, antiquated database format. The author then managed to convert some real applications of theirs into sqlite databases of the necessary program data, and then taught the operating system how to treat those sqlite databases as programs to run.
The second article builds on this, by creating a program shipped as an sqlite database, and then making that program read and write itself (through sqlite code) to store its application state. So instead of having a web server application that loads an sqlite db table, it is just a database file that the operating system can run as a native application, that also stores user data in itself.
perhaps is more correct to say "all data, including code, is table-representable, even though being a graph" or "everything falls back to tables" or even "relational algebra is all u need", but I strongly disagree SQL being a domain on its all, and that it (all) collapses into such domain.
One can collapse segment tables likewise into DATALOG, which is also a PROLOG-derivate. So then the thing demonstrated here is - "all collapses into grammars perhaps". which is not new, but there are plenty of engineering details, and whatnots to consider, to make such model viable for large-scale deployment. And trouble is it is not so easy to infer stuff about grammars before you expound/infer on them.
don't get me wrong - I love SQL, and respect SQLite and DuckDB for what they are. what we see here is one very curious approach and great demonstration.
And for me, tables are just trees, but for them, trees are just tables.
Is there a higher-level abstraction than parametric grammars nagivating program states...?
As a sidenote, it seems think Stephen Wolfram has similar take on things, but like taken to extremes and applied to dynamic systems. His idea is that all that we experience is a byproduct of some ever-unfolding energies governed by parametric rules. Thus Ruliad. Though his thing sounds less plausible than what we (already) have (applied and working) in classic computing.
idk...
btw, table rows ar facts from PROLOG perspective and much of what SQL stands on is PROLOG. the parametric rules with conjunctions part is in the WHERE clause if one ever wonders... took a while for me to figure it out.
p.s. graphs are represented very often by means of tables, relations if u want. trees are graphs. then, on the other hand, we use trees to index table-storages. so the connection/mapping is there all the time.
Whereas a relation ("table" in its crude form) can be queried and projected into many forms (including as a tree) but with multiple paths to the same data and ensure future uses are not trapped in a mandatory hierarchy. This was Codd's entire discovery and it still holds 50 years later.
On the one hand: I don't think I want that. Including static content with the binary makes sense, certainly. However, storing writable run-time data there feels messy; I prefer a read only binary which is handed a writable state directory (it is worth saying that I've spent a lot of time with nix and other immutable distros).
On the other hand: This is the coolest, most fun thing I've seen in a good while, and I absolutely want to see it taken 1000% further. Who cares about perfectly operationalized immutable deployments when the hacker spirit is in the air?
I'll bet you could use this to run with another thing APE does: fat binaries. If program text lives in a database, what's one more row? Just
SELECT text FROM executable WHERE arch = $(uname -m)
Edit: actually on further consideration this feels perfect for smalltalk; you can put the VM and image in a single file
Absolutely wonderful stuff.
The SELF upgrade (heh, self upgrade) and rollback processes could benefit from some... fancier... footwork.
Your example has a new binary copying old data into it, but then you have to move the new binary to the deployed location. Which means an outage through stop service, data migration, replace file, start service.
What if the upgrade process was more like... write the new SELF data into the old binary, send SIGHUP, and then the service fork+execs itself, while doing haproxy-like zero downtime FD handover?
Replacing the SELF data in the existing file is safe right now, because you can't mmap segments into memory. But if you do end up figuring out some clever BLOB alignment mmap stuff, you could do the SELF upgrade like a data migration! INSERT segments/symbols, fork+exec, and the data migration cleans out the old code. :-D
Updating the SELF schema to allow multiple sets of segments and symbols would allow for this upgrade trick, but could do other fancy things... thin multi-arch binaries where only the code segments differ.
BLOB alignment should also mean more efficient static asset serving and a bunch of other niceties... definitely worthy of investigation.
However -- very strong however -- as fun as this is, I would never, ever, ever allow an internet-facing service binary to be self-writable. :-)
Everything old is new again. And I don't mean it in a disparaging way. There's lots of "old" ideas that are simply great ideas that did not win on their own time but might come back with force in the future.
Incredibly impressive on a conceptual level though. If someone proposed doing this while having the executable only grab a read-only reference to itself, I think that would be a legitimately solid idea.
https://dl.packetstormsecurity.net/mag/pocgtfo/pocorgtfo14.p...
Good for deployment but an accidental deletion of the binary can cause loss of both data and code. Good approach for AI harness and agents though.
And now all SQL injection bugs are RCE!
It's a fun idea, though.
[1] https://en.wikipedia.org/wiki/Metakit - base tech
[2] https://wiki.tcl-lang.org/page/Starkit
[3] https://wiki.tcl-lang.org/page/Tclkit
[4] https://wiki.tcl-lang.org/page/Starkit+Meet+ZipCould the webserver receive a code segment from the web and add it to itself (like a plugin upload)?
And yes you can let authorized users add custom code, make it run in the database like a micro-micro-vm and do stuff on triggers etc in the database as well.
This makes it even more powerful, because users can switch code during runtime and you can have multiple users "programming" against the same database at the same time working against the same data.
Vendors who have done this like IBM with the AS/400 have typically put the database APIs front and center in their sales pitches, and POSIX on AS/400 is more akin to WSL than something well-integrated into the traditional single-level store (or at least was a few years ago when I last took a look at it).
But if a “legacy” application is only interacting with the filesystem through filesystem APIs, but you as a developer can interact with it as if it were a database (with transactional semantics that extend beyond the virtual inode boundary), I think it can still be very valuable.
This would be a super pragmatic approach for a modern compiler suite. This is the brilliant part of the whole idea.
The biggest concern for me would be the security angle - if there would be some way to seal the executable itself and descriptor tables so that an application can be guaranteed to never touch that and only ever modify the other “runtime” tables. Not doable with raw sqlite though since it has no kind of ACL mechanism, but would be a neat extension so that the interpreter handed the handle to the process directly with the privileged tables cordoned off from writing.
SQLite's unix VFS is actually using a mixture of mmap and write() by default[1] and you'd need to combine that with mseal() and some more pieces to actually pull it off. It would probably be possible.
(There's prior art here; although done differently: https://sqlite.org/src/file/ext/misc/appendvfs.c).
(Or computer mad scientist? Whatever (mad scientist) && (computer scientist) is)
The whole thing is brilliant anyway. Up with this sort of thing!
Everything old is new again.
One of my favourite ways to develop apps is to use Lua - for everything. Construct an efficient core application framework, then embed the Lua VM in it, and then do all program logic and control flow in Lua, then put the Lua bytecode in the binary with luastatic, and off we go.
So this technique could be used to tack on state as Lua bytecode, meaning I could then accomplish something I’ve wanted for my Lua apps for decades - migration. I could have the app save its state safely, then simply transfer the binary itself to another machine, and recover gracefully.
Of course this could be used with any tools - not just Lua - but the idea of having the entire runtime binary included in the Lua state table is just so delicious I wanna try it ..
Okay, you have to expand on this. Do you have a blog post or anything that goes into more detail?
On the other hand, sqlite supports concurrency and ACID transactions, so it might actually be better at handling concurrent access to shared state
Makes sense to me. Also for production, I would have the app change its name by adding a datetime to the filename, indicating when it was last synchronized.
This is a very appealing idea, I too hope it gets 1000% more attention and adoption.
I also found comments about replacing identifiers in SQL (used to implement ALTER statements probably?) which say that it won't work right if two things in the database have the same name and it'll get caught by a sanity check later.
That's just the parts I looked at.
[1]: https://en.wikipedia.org/wiki/B-tree#B-tree_usage_in_databas...
Or think of an editor or something like Excel. You probably don’t want the files you edit to all be stored in the binary and require separate export/import functions for sharing, or for editing using alternative applications.
And in the end, that goes for pretty much all data. The applications are the tools you use to view or modify the data, but the data remains independent.
local appConfig[myUniqueArgv0] = { .. some config data .. }
I'm thinking now of the hoops you have to jump through to edit a package.json file to update your dependencies, and thinking yeah, what if you could do: "UPDATE dependencies SET version='1.2' where name='madlib';"
I have this problem on my computers already, because configuration files and other program data is spewn across all kinds of directories and files all over the place.
The only two options I have is a) do a factory reset and reinstall software from scratch and spend time configuring things again, or b) live with all of the extra garbage that has been excreted by the software on my systems just so that I can keep the data that I actually want to keep.
Whereas to pick up all of my data that I do care about and copy only that and nothing extra to another place, is extremely time consuming to the point that it’s not realistically doable :(
Of course, cramming absolutely everything including user documents into the program won’t solve that either.
My ideal system would separate data that was written because of me (non-default config values, bookmarks, etc) from documents etc made by me, from garbage that the program wants to write that I absolutely do not care about and do not want to know about.
> The only two options I have is a) do a factory reset and reinstall software from scratch and spend time configuring things again, or b) live with all of the extra garbage that has been excreted by the software on my systems just so that I can keep the data that I actually want to keep.
NixOS solves this.
> My ideal system would separate data that was written because of me (non-default config values, bookmarks, etc) from documents etc made by me, from garbage that the program wants to write that I absolutely do not care about and do not want to know about.
And NixOS with https://github.com/nix-community/impermanence does this, forcefully. Actually normal nix might solve it depending on how far you want to go.
Again, I wouldn't usually try to evangelize it, but you have described your ideal system, and it exists.
Spamming my home folder on the other hand is criminal..
Basically, take your application framework of choice - in my case its JUCE (C/C++) - and embed a Lua VM in it. Put the UI in the framework, and then encode all of the program logic and application control flow in Lua, which marshals the UI and the rest of the system through the VM interface.
Sure, you can just 'do everything in C/C++', but I've found that for long-term projects, doing app logic in Lua, a simpler language to wrangle, has been very fruitful. And, it puts the framework in an interesting context - serving the VM - which reduces bugs and keeps things pretty tidy. Most of the C/C++ code is just UI and system-level scaffolding - which I then write Lua tests for - and then once this is in place, the app logic is very productively developed and shipped as byte code. The App becomes a host for the logic.
My favourite other example is turboLua:
I think there are false negatives in some of the lesser-maintained antivirus tools - but I have never had issues with it.
$ luastatic —help