A future for SQL on the web(jlongster.com) |
A future for SQL on the web(jlongster.com) |
that would solve the safari indexeddb 7 day ttl issue to start with.
and if replication could be made to work on top of something like webrtc we're looking at a great foundation to start building distributed, decentralized browser apps.
i must say that the experience is quite horrible - that torture of having to write map/reduce functions, added with some erratic behavior in regards to data integrity (inserted entries silently discarded, sync to the remote couchdb instance working somewhat whimsically). as soon as your dataset is sizable in any regard (tens of thousands of records in a collection, if i recall the terminology) it begins to just break apart.
was writing a browser extension, and used pouch with the hope of keeping its persistence local and avoid needing a server. seeing that it leaks tried to trade it for a couchdb server. seeing how bad sync is, and that couch is not very comfortable to work with either ended up throwing the thing in favor of a postgresql+postgrest backend.
So it sounds like IndexedDB was the right abstraction all along.
Another spec that failed from a practical perspective because of Mozilla's reluctance to implement key aspects of it is web components.
So far I've found IndexedDB to be outright depressing in it's limitations.
"If you are writing a web app today, you’ll probably choose IndexedDB to store data. It’s the only option for something database-like that works across all browsers."
RDBMS all the way baby
It's a fun article, I'd recommend trying reading all of it.
It was torpedoed because it was SQL-based (and not trendy "key value" and "web scale").
There was the whole excuse that the specification was "whatever SQLite does" and, therefore, not suitable for being a standard. There would be worse things than SQLite upon which to base a standard, all things considered. I still believe it was torpedoed because of lack of trendiness and "not invented here".
Hindsight shows that was entirely correct, as SQLite bugs were then found that could be exploited directly via WebSQL, Firefox of course was not vunerable. (https://hub.packtpub.com/an-sqlite-magellan-rce-vulnerabilit...)
As a sidenote, I worked a lot with the WebSQL API and it was not a very good API in the slightest, immaturity may excuse some of its flaws, and it isnt like Safari did a much better job with IndexedDB, its just a buggy browser and thats where WebSQL was used most, but a large part of the problem is that it was bolting an API that assumed a single threaded client when that is not the reality with web pages where multiple tabs exist
Looks at Chrome
And yet we're are now at a point where Chrome rams its own APIs through standards bodies, and there are no (and often won't be) any independent competing implementations.
That way you never have to deal with browser incompatibility or unchangeable specifications.
For those who are able to stomach an uncomfortable political history instead of an easy, technical answer, you can take a look at [2]. It's interesting that 7 years later, many the folks who pushed hard to get rid of SQL in favor of NoSQL seem to no longer occupy positions of prominence in the industry.
[1] https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_A...
[2] https://nolanlawson.com/2014/04/26/web-sql-database-in-memor...
The issue is not just that there wasn't a competing implementation at the time, it is that you could not feasibly create a competing implementation. The set of features supported by SQLite is _massive_. And even you stick with SQLite, which version? And which extensions? Which features do you remove because they're not safe or not applicable for web?
Alternatively you could start from the ground up and create a new database spec that is similar to SQLite, but more limited in scope. But then it will not be directly compatible with SQLite, and need a translation layer. The browser would likely have to re-implement large parts of SQLite for this to work. And you'd miss out on so much of the functionality that makes SQLite attractive to use.
Arguably that would still have been better than IndexedDB. All that IndexedDB has going for it, is that it is simple to implement. But even then Safari manages to get the implementation horribly broken so often.
What we need instead is to have a low-level storage api that can be used to implement other databases as libraries, including SQLite. The Storage Foundation API, mentioned in the post, might just give us that.
>The issue is not just that there wasn't a competing implementation at the time, it is that you could not feasibly create a competing implementation. The set of features supported by SQLite is _massive_. And even you stick with SQLite, which version? And which extensions? Which features do you remove because they're not safe or not applicable for web?
This seems like a case of perfect being the enemy of good.
Otherwise you'd have to deal with different versions of SQLite in different browsers, most likely outdated, with many options turned off. SQLite is full of quirks and gotchas, so it's safest to ship your version in your app.
I would have loved WebSQL, but it is reasonable to require multiple implementations for full standardization.
One could also have embedded a trimmed-down PostgreSQL or MariaDB into browsers.
Although I do empathize with browers vendors. I worked at Mozilla at the time and was aware that this is a lot of things to think about when integrating something onto the web. I get why it happened, but practically speaking maybe it should have won. It's not like Chrome seems to care much about cross-browser standards these days.
I'm hopeful for a storage layer like this though: https://web.dev/storage-foundation/
It might actually be a better outcome if we get a storage layer with close to native performance, and then you can compile and db/lib/etc and it gets to use it.
I think devs should be happy, not sad. It looks like we’re finally getting it right, and will have a feasible way to add the incredible sqlite to the set tools we have available to make web apps.
Hopefully absurd-sql keeps going and browsers adopt a good storage standard.
This is almost certainly not even close to correct. There are substantial reasons why it wouldn't be a good idea, but this might be the biggest one: it's very hard to adequately sandbox an external C library.
(... and, also, Apple probably would prefer that the web didn't exist at all, but that's a different pandora's box...)
This is also how I justify my opinion that the "no independent implementations" was more of an afterthought excuse, and less of a primary motivation on Mozilla's part.
(I'm a little pressed on time here, but I could probably find more examples if I had more time to search... Sorry!)
Quoting Jonas Sicking from a W3C IRC log[0]:
> we've talked to a lot of developers
...
> the feedback we got is that we really don't want SQL
Quoting Maciej Stachowiak from the "public-webapps@w3.org" list[1]:
> Hixie has said before he's willing to fully spec the SQL dialect used by Web Database. But since Mozilla categorically refuses to implement the spec (apparently regardless of whether the SQL dialect is specified),
...
> At the face-to-face, Mozilla representatives said that most if not all of the developers they spoke to said they wanted "anything but SQL" in a storage solution.
The comments relating to JOIN in the comments on Mozilla's blog post comparing IndexedDB and WebSQL[2] betray the anti-SQL (and, arguably, anti-relational database) stance at Mozilla.
Mozilla's people didn't like SQL, so any excuse for dismissing WebSQL (or even a simplified SQL dialect that didn't have the "quirks" of SQLite) was a foregone conclusion.
And now here we are shipping >1MB transpiled WASM payloads around when we could have agreed on a query language feature set implemented in native code using a consistent back-end standard storage format.
[0] https://www.w3.org/2009/11/02-webapps-irc
[1] https://lists.w3.org/Archives/Public/public-webapps/2009OctD...
[2] https://hacks.mozilla.org/2010/06/comparing-indexeddb-and-we...
I feel like "just sqlite" was really a very practical idea and now, because it has already failed once, we can‘t really try it again, or can we?
Having the spec basically be "bundle one particular piece of software into the browser" might not be in the spirit of web standards, but on the other hand SQLite is so widely bundled into everything, does it really matter?
If you look at the recent StackOverflow survey, the majority of the developers only have around 5 years of professional experience in the industry.
SQL has been around since the 1970s and is still around in force for reason. There's a good chance a lot of developers, especially the enormous number on the front end, do not have experience with SQL or are just getting into back-end work where they are exposed to it.
There have been a lot of SQLite articles recently, as well as node.js libraries to query SQL. I have a hunch this may be why.
Which version do you want to bundle? With which compile flags and which extensions? Do you have a checklist for bug-for-bug compatibility? Because you'll definitely need one when a future SQLite releases a security patch and breaks a million webpages.
overall, if you looked at HN like five years ago, every DB headline was key/value, mongodb, maybe some cassandra / couchdb, links to the "web scale" cartoon.
these days, it's SQL SQL SQL, with a heavy dose of SQLite and PostgreSQL. SQL survived the key/value fad despite the nebulousness of a workable "standard" (yes there's a SQL standard but no vendor DB implements all of it or doesn't add many many features, syntaxes, and behaviors on top of it). In particular SQLite recently seems to look to Postgresql for guidance on new syntaxes such as how it implemented upsert, it's RETURNING syntax is explicitly from PostgreSQL, and it interestingly uses the same "VACUUM" term for db utility cleanup.
SQL has survived every fad since the 1970s:
Stonebraker "What Goes Around Comes Around"
https://people.cs.umass.edu/~yanlei/courses/CS691LL-f06/pape...
Whenever I need simple (but indexed) key-value (unless that's a hi-load server-side) I always just use SQLite anyway. I really don't understand why do we need any data storage other than SQLite (and HDF5 perhaps) to exist on the client side.
You can put anything in an SQLite record so it can certainly be used as a key-value table. Where at least the values can be arbitrary binary blobs.
Using SQLite results in a new, built-in reliance on a C library. Even though most languages can use C library bindings, it does present some issues in some build scenarios, such as static builds, and, given the enormous scope of SQLite, would prevent anyone from ever achieving a 100% rust browser.
At any rate, if that really is your goal for some reason, rendering engine is going to be your first problem anyway. Then a JS runtime.
Just a heads-up, the File System Access API[1] is underway in Chrome, which potentially removes nearly all of the absurdity here. It has other benefits too. A web page using this could write a .sql file on to your drive, that other programs could then access. One of the other bright stars in my world is Karli Koss, who has an extensive personal data-extraction setup for a ridiculously colossal variety of services & devices[2]. A vast amount of this massive massive data-gathering framework is just reading sqlite databases of the various devices and apps. If the web can help participate more actively, can let apps write sql files to store state: so much the better I say. Help externalize your state beyond the browser, please!
[1] https://wicg.github.io/file-system-access/#api-filesystemwri... https://caniuse.com/native-filesystem-api
This really is quite incredible. Same idea extends to your filesystem too. Tracking millions of 1KB objects on disk? You could load the whole set into memory substantially faster from SQLite using the same disk. If WAL is enabled with reasonable sync flags, the same applies going back out to disk as well.
SQLite is the most powerful dependency that our product uses today. We have been using it in production as the sole persistence mechanism for 100% of our data for the last 5-6 years now. Recently, we have started leveraging the actual SQL engine to process all of our business logic as well.
So many apps could have just saved to a .db file instead of coming up with their own binary format. So many others could have just embedded a tiny, easily sandboxed, Lua library and added scripting languages, rather than embedding larger ones (or worse yet, implementing their own scripts).
I ended up switching to https://github.com/agershun/alasql which could handle up to 80MB of data or so. (I haven't yet tested on larger datasets so I don't know the actual limits.)
I don't think this is a fundamental limitation of sql.js as the linked article proves that you can implement custom paging for sql.js. But unless you do that (which I haven't spent the time to figure out how to do) then sql.js will run out of memory very quickly.
Just something to be aware of if you're investigating it.
If there's a high-level library that makes more effective use of memory with sql.js under the hood let me know.
Unlike absurd-sql I don't need the results to be permanent. I just wanted an in-memory SQL for joining, filtering, grouping data.
Raw sql.js is limited by the browser's wasm memory limit, but 80Mb should not cause an issue...
Another use is privacy centric apps that send nothing to the server, using the web as a kind of “install” platform but nothing else.
Seconded – I was pretty dismayed when I saw the IndexedDB helper library landscape.
I ended up making https://github.com/TehShrike/small-indexeddb which is ~50 lines to make it less onerous to work directly with the IDBObjectStore.
https://github.com/jakearchibald/idb
It's basically a promise-based version of the standard API.
Safari will happily delete your IndexedDB database after 7 days of inactivity.
It deletes "all of a website’s script-writable storage after seven days of Safari use without user interaction on the site". That includes:
- Indexed DB
- LocalStorage
- Media keys
- SessionStorage
- Service Worker registrations and cache
Source: https://webkit.org/blog/10218/full-third-party-cookie-blocki...
Found via: The pain and anguish of using IndexedDB: problems, bugs and oddities - https://gist.github.com/pesterhazy/4de96193af89a6dd5ce682ce2...
The design is very much a 00s throwback with a lot of density and power compared to more modern apps in the same category, so Mac users are unlikely to like it anyway. I was just tired of the older apps lacking modern features I like while the apps with the modern features were clearly mobile/touch first and so much slower for bulk operations than their older competitors as a consequence.
Having said that, I do like the idea of Sqlite in the front end for localstorage.
I was profiling on an older computer. On my newer one, summing 100 items takes ~8ms (use the raw idb mode). When I said "simple operations" I meant simple queries that you'd expect apps to write, not just 1 single read/write. It is a little faster for each read/write, but there seems to be a bottom floor. Even if reading an item itself is fast, opening a transaction is slow. So any query, even if it only reads one item, is going to suffer the perf hit of opening a transaction.
It's only twice as fast as Firefox, so overall IDB is still super slow when compared to running the same queries with native SQLite. We're talking summing 100 items taking ~.01ms or less. I have no idea why it's so slow.
@jlongster I have a question about this:
> The backend calls it [Atomics.wait] to wait on the result from the worker and blocks until it’s done.
Does this mean the main (UI) thread is blocked during queries? Or are there more threads, like UI <- async messages -> SQLite main <- Atomics blocking -> SQLite FS backend?
————
At Notion, we’ve used IndexedDB for two purposes: (1) to durably persist a queue of changes to send to our backend, and (2) in the desktop app, to LRU cache the page data we read from the server to accelerate reads. Both of these used localStorage years ago, but we ported to IndexedDB because of data loss on localStorage. Porting was fine for the write queue, but we really noticed the slow when we tried porting the data cache. To get close to the original performance we coalesce reads, and we delay writes to the cache significantly so they can batch more effectively into a single readwrite transaction that we send after the reads for the current page load are complete.
That worked okay, but it was annoying to maintain the IDB cache code because our Android and iOS apps used SQLite for their caches, and it’s so much easier to add new queries using SQL compared to writing IDB iterations - and it’s faster. So we switched to using native SQLite via a bridge to a Node process. Now with absurd-sql, maybe we could bring the same caching logic to browsers.
The thing stopping me is how unreliable we’ve found IndexedDB to be - aside from the optimization work. We notice a lot of bugs in IDB implementations on different browsers. In Safari (especially on iOS) there’s a bunch of spooky issues that have caused stalls or spurious errors, sometimes requiring an app restart before the IDB database can be re-opened. Forget it on Android - weird vendor webview patches mean your storage might get cleared out from under you. On Firefox, we notice that sometimes the IndexedDB database doesn’t create all the object stores we request for some reason. Even on Chrome, IndexedDB can suddenly start refusing writes in the middle of a session with no clear explanation, and on Windows restarting the computer is sometimes the only fix.
If we can share SQLite queries with our native apps then maybe it’s worth wading deeper into these issues… but it really does feel like building on quicksand.
While in-memory databases have their uses, it kneecaps
SQLite into something far less useful. To build any
kind of app with it, we need the ability to write and
persist.
This could be done via the File System Access API:
https://developer.mozilla.org/en-US/docs/Web/API/File_System...
The API already works nicely in Desktop Chrome:
On a similar note, have this nagging feeling that we used to have this ability to use SQL in client-side applications. I just can't recall how?
/s
The Internet is a wild place...
Only mad because it’s so much better in every way.
1. People often reset the cache on their browser, after all it is just a cache. SO a big benefit of databases which is persistent data is kind of not there. Juts rest your cache.
2. The second great benefit of databases is that they are multi-user. There can be in fact millions of users. But this benefit would not be there if the database lives and executes in the browser.
It is a great project and I hope one day we will be able to use it in production.
I'd like to add one drawback with this solution: Complexity.
From my experience, complexity can easily lead to more problems than what it solves. With all the "blackboxes", e.g., WASM, JS, SQLite, IndexDB, ..., it might be hard to find bugs. Most of the tools used are somewhat stable and mature tough. SQLite, for example, has a whooping 100% test coverage (line coverage at least).
this may be the most performant/secure/cheapest b2b saas stack ever
every customer gets own sqlite database, downloads it to their browser on first load, each db gets synced to s3
everything is served statically from s3 as well
This is really cool, I wonder if it could be built into something like https://datasette.io/ - without the need for a python runtime.
Attack vector might be to register 1000 domains then get a page to load each of those to fill up its 2Gb quota? Just guessing….
You can explore it with the mouse cursor.
I recommend downloading sample DB, writing some dummy query like "SELECT BILLINGCOUNTRY, COUNT(INVOICEID) FROM INVOICE GROUP BY 1 ORDER BY 2 DESC" and then pressing Execute.
I've been planning to write an extensive article about it and open sourcing the solution cleaning up the code a little bit, but still haven't got much time to do so.
gotta keep ourselves fully looped! ⥀
(is there perchance a repo available with your work? that'd be lovely to see.)
It doesn't. Because there are now 4 different file access api proposals. At least one of them is already in Chrome (File System Access). [1]
Both Safari and Mozilla are unwilling to implement any of the four until the mess is cleaned up, and there's a single coherent proposal.
Chrome, of course, doesn't care. Storage Foundation API in addition to File System Access they already implemented.
[1] https://github.com/WICG/storage-foundation-api-explainer/iss...
https://archive.jlongster.com/Open-Sourcing-My-Gambit-Scheme...
(And also, I've observed that webapps that notice they're on an iOS device can insist on being "added to the home screen" before they'll do anything, and so ensure their data stays around.)
The problematic aspect of this, is that there's no equivalent of this "exemption by user explicitly expressing the desire to keep the app around" for non-mobile Safari.
Maybe for desktop Safari, the data should be kept around if the web app's rel="canonical" URL is bookmarked in the browser?
Or desktop Safari could just support regular Progressive Web App standards, and so show an "Install App" indicator (like e.g. desktop Chrome does for this webapp: https://www.soundslice.com/). But I have a feeling Apple will never support this on desktop...
Edit: In order for this to be true, Apple (at the very least) needs to enable push notifications and an install prompt for progressive web apps.
Although I agree that it's a fairly minor win in the grand scheme of how handicapped PWA's are on iOS. And like you said, the lack of install prompts and tucking away the PWA "installation" option in the share menu makes it less than intuitive and requires manual awareness efforts by devs[2].
[1] Last section of https://webkit.org/tracking-prevention/
This makes total sense to me, and I’m happy Facebook can’t store tracking data in my Safari for longer than 7 days.
Can you run a PWA on an iPhone? Yes, you can even pin them to the start screen to keep data long term, as others have pointed out.
Are you able to use all features other browsers enable? No, only the ones Apple allows.
Nevermind that there's nothing strictly stopping GM from changing the designs for Buicks to include a vehicle capable of a fifth wheel mount.
Also in my opinion websites have no business storing data on my device through a browser. If I want that behavior I would use a native app.
We will provide a new backend for the Storage Foundation API when it's available.
As a bonus point this effectively adds yet another level of "Yo, Dawg" which I can't not love just as a matter of principle.
Webkit says in your source:
> It is not the intention of Intelligent Tracking Prevention to delete website data for first parties in web applications.
The latter! Your app running queries must be on a worker, and then the IDB backend will spawn another worker. `Atomics.wait` is not even available on the main thread.
Ideally in the future, there will be a better storage API that we don't even need all the Atomics silly-ness (hopefully it provides Sync methods)
That's really cool re: Notion! That's exactly the kind of thing I want too: a way to just build apps the same way everywhere, on mobile/desktop/web.
You are right about various issues, and I personally don't have to worry much about it on my app because I have native mobile apps and I don't support the web version on mobile. I intentionally do that -- the mobile web is just too broken in too many ways. My impression is the IDB is more stable on desktop, but because mobile is more memory sensitive there are more issues there.
However, you should try it out! I definitely discovered a lot of weird things; I definitely was able to get Safari into a weird state the required a complete app restart. Here's the thing though: I found ways around them. If you do a lot of read requests in a certain way, Safari will lock up permanently. However, if you make sure to wait until the `readonly` transaction is finished before starting a new one, the problem goes away. I was able to reliably reproduce that problem and it went away with that fix.
I think absurd-sql is so promising because it normalizes the patterns of how IDB is accessed, and it already includes fixes for a bunch of edge cases. There are probably more, but try it out! If you run into an edge case, we can tweak the IDB backend until it works. We can paper over these issues in the underlying backend and you don't have to worry about it because you aren't directly managing IDB read/writes.
[0] https://en.wikipedia.org/wiki/LevelDB#Usage
Edit: Sorry, just re-read the article. The author does mention that Chrome's IndexedDB isn't implemented in SQLite.
https://thegraph.com/docs/indexing
> Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. They also earn from a Rebate Pool that is shared with all network contributors proportional to their work, following the Cobbs-Douglas Rebate Function.
> GRT that is staked in the protocol is subject to a thawing period and can be slashed if Indexers are malicious and serve incorrect data to applications or if they index incorrectly. Indexers can also be delegated stake from Delegators, to contribute to the network.
> Indexers select subgraphs to index based on the subgraph’s curation signal, where Curators stake GRT in order to indicate which subgraphs are high-quality and should be prioritized. Consumers (eg. applications) can also set parameters for which Indexers process queries for their subgraphs and set preferences for query fee pricing.
It's Ethereum though, so it's LevelDB, not SQLite on IndexedDB on SQLite.
I think the NoSQL and MongoDB folks are worried for their careers.
Then that adapter is carbon copied by everyone, because again, nobody is going to reimplement an 80kLOC SQL database as well as whatever 10k LOC parser/lexer/typechecking adapter someone wrote, if they can avoid it. Then everyone just uses that forever, and you're back to square 1, using one implementation everywhere, which is the exact situation standards are supposed to avoid anyway.
The working group was correct to reject a "compromise" like that because that's never how it works out in practice; it's a submarine suggestion from the start. And a big part of this is all because, as evidenced by numerous responses in this thread, modern computer programmers seem to value their own immediate satisfaction and time over literally every other potential concern, no matter how significant.
> Implementation experience is required to show that a specification is sufficiently clear, complete, and relevant to market needs, to ensure that independent interoperable implementations of each feature of the specification will be realized.
From https://www.w3.org/2015/Process-20150901/#implementation-exp...
A 100% memory-safe browser is definitely something I'm clamouring for.
> At any rate, if that really is your goal for some reason, rendering engine is going to be your first problem anyway. Then a JS runtime.
Mozilla is working on the rendering engine. Memory-safe JS runtimes already exist (e.g. Nashorn).
Regarding the SQL dialect - that could have been handled by specifying a standard SQL dialect that's then dynamically translated to the target engine's dialetc.
An example is that Telegram maintains a local db which stores the most recently accessed content so you can scroll through past messages and media without having to request them every time you view them.
If, on the other hand, you're explicitly uploading or entering data on a web site, well then yes, the default assumption would certainly be that the data is getting saved on the web site's servers (if it's being saved at all). That's how the web has worked for a very long time, with the exception of cookies and a few other more recent but not widely known storage mechanisms which in most cases aren't considered sufficiently robust for persistent on-device storage.
And the majority of uses for local data is to cache fetched data or store temporary preferences not worthy of being in the server side database.
They’re storing it somewhere, no? I feel like the benefit here is more so the browser file system doesn’t get bloated
Oddly enough the handful of web apps I’ve known to use local storage don’t communicate this at all. Developers seemingly treat data in browser local storage as existing indefinitely and unlikely to be deleted, when in fact it’s probably the most easily accidentally lost form of storage an app can have.
Why? Because it's a very common pattern. Almost fundamental. Take any project more complex than a toy calculator, and you'll quickly find places where the authors are hand-rolling relational operations.
Does your app have an array of records in memory, which it then searches for values, filters by conditions, and/or sorts? I'd give it a 50/50 chance that if you replaced that array with an in-memory SQLite table, and all operations on it with SQL queries on it, the result would be less code, more readable code and better performance[0].
It's not just the web - I'd argue that programming languages in general should all embed first-class in-memory relational database engines. It should be a part of the language standard - if it's easy to write this:
struct Foo {
int bar;
date baz;
string quux;
};
//somewhere in something
Array<Foo> foos;
table<Foo, index=[bar, quux]> foos;
and get a thing that can be efficiently queried, filtered and transformed along dimensions specified, with the compiler turning your request into efficient bytecode/nativecode. Doesn't mean you have to write queries in vanilla SQL - there are more composable syntaxes out there. But arbitrary loops and map-reduces aren't that good either.
When you start looking at your data processing code as database operations, you'll see it everywhere. That map/reduce/zip blob? That's a JOIN query. That struct you're keeping in your event loop, that looks like:
struct Schedule {
Array<Task> tasks;
PQueue<TaskId> toRun;
};
--
[0] - It's trivial to add indexes to SQL table. The engine will maintain them for you. Nobody habitually adds indexes to their own regular variables. They, along with the code to maintain them, would manifest as extreme code bloat. Meanwhile, in-memory SQLite is freakishly fast. If you're measuring performance in Python/Ruby units, you won't even notice the FFI overhead.
Sure, there could be cases where you have absolutely no alternatives to do something really specific, but in those cases I'd first invite you to reconsider whether what you're attempting to do actually needs to be a web app. Of course, people's thoughts will probably be split there.
However, the bottom line is still this - large sites load slower and cost more to load, they consume more battery and CPU, which on lower end devices could lead to system instability. If users don't outright leave your bloated solution (i.e. if they're forced to use it because of network effect) then in the case of any non-optional conditions (slow or unstable networks, low end devices, expensive data) their experience will be miserable.
Maybe have your IM client be a downloadable app instead? Better yet, why not have it be a native app instead of a bundled browser app, for excellent file size, small attack surface and great performance, while conforming to the OS look and feel?
Alternatively, why not make your IM client have a lightweight browser version that doesn't try to do everything under the sun? Personally, i feel like the answer is not to make browsers do more, but to try to do less yourself.
To that end, utilize server side rendering and pre-rendering with hydration when needed. Split some bundles and shake some trees. Compress as much as possible and use boring, common fonts while preferring local versions if available. Avoid videos in most cases and use smaller images, or vector graphics. The web can be a simple, beautiful and fast place if we make it one.
I suppose the problem is, on the web, you're redownloading the whole bundle each time you run. There are layers of caching in the browser that are supposed to help, but they're almost always defeated by a combination of app misconfiguration and developers' reluctance to use the cache given the ridiculous rate of redeployment of web software.
I'd still argue that SQL API is a prime candidate for a shared system-level library - AKA browser built-in. The concepts behind it have been worked out in the 1970s and honed through decades since. It's not experimental tech. The API won't be changing every month.
WebDatabase, with or without a spec, boiled down to shipping SQLite. We didn't want the Web to depend on that. It might even have required shipping a specific version of SQLite to make sure that query planning matches other browsers.
It seems that IndexedDB has turned out poorly for various reasons but one of those reasons is that Chrome has botched their implementation and that's not Mozilla's fault.
Ironically, all the popular browsers already ship SQLite - they just not expose it directly. So, in a way, the Web already depends on SQLite. So does a lot of other technology - we're talking about one of the, if not the, most widely distributed pieces of software on the planet.
The "no independent implementations" was certainly a convenient way to dismiss away the already-implemented SQLlite-based solution. It's a valid argument for rejecting SQLlite as the spec, for sure. Implementations notwithstanding it seems like Mozilla was unwilling to even consider anything SQL-based.
The same tone comes out in the "Beyond HTML5: Database APIs and the Road to IndexedDB" Mozilla blog post[0]:
> In order to really get Web SQL Database right, we’d have to first start with defining a meaningful subset of SQL for web applications. Why define a whole other language, when more elegant solutions exist within JavaScript itself?
...
> We were resolved that using strings representing SQL commands lacked the elegance of a "web native" JavaScript API ...
That sounds like the same rhetoric used in the early MongoDB / NoSQL craze days to dismiss RDMS, ACID, JOINs, etc. Never mind that SQL is a battle-hardened DSL that has proven itself to be suitable to the task-- it wasn't "elegant" enough for the front-end devs Mozilla surveyed.
As an aside, I also find the quote from that post very amusing:
> In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter.
The example for a JOIN using IndexedDB in the referenced article[1] looks strained. It really doesn't look like it has "syntactic simplicity" compared to the WebSQL version above it.
[0] https://hacks.mozilla.org/2010/06/beyond-html5-database-apis...
[1] https://hacks.mozilla.org/2010/06/comparing-indexeddb-and-we...
Very few people are using Chrome-only APIs which are not in the standards yet. So it's not really a concern.
But otherwise, Chrome really has pushed the web forward more than any other browser. If it hadn't, native (and walled garden style) app stores would have totally taken over.
The web is in business (and thriving) as an application platform because it's being pushed forward relentlessly. The Storage Foundation API discussed in TFA is a good example.
Ah yes. But SQlite not having competing independent implementations somehow is?
Also, "not many people using something" is not as great an argument as you think it is. See, for example, the latest problem with browsers deciding to remove alert/prompt/confirm: https://dev.to/richharris/stay-alert-d
> The web is in business (and thriving) as an application platform because it's being pushed forward relentlessly.
A quote from the same article: "An ad company shouldn't have this much influence over something that belongs to all of us".
So somehow non-standards that Chrome pushes (many of which will never get a different implementation because both Safari and Mozilla consider them harmful) are good, and push the web forward. But SQlite is bad because there are no independent competing implementations.
Got it.
> The Safari team did quite a lot of work re-styling their alerts and dialogs to be within the context of the webpage, yet phishing and scams that utilize this still run rampant on iOS. Repeated alerts are used to lock up the browser and make it unusable, forcing non-technical users to call a scam telephone number because they think their device was hacked.
A few bad actors ruining it even for totally different usecases.
Except that's not why it was culled. The reasons are discussed in great detail on this thread and elsewhere.
> non-standards that Chrome pushes (many of which will never get a different implementation because both Safari and Mozilla consider them harmful) are good
I'm saying nobody uses those Chrome only APIs, until they become standards (which requires acceptance by other vendors). Or are you against experimentation?
> It’s important to ensure that our iOS visitor is using Safari because iOS doesn’t permit other browsers to install our awesome PWA’s!
So if the user is using Firefox or Chrome etc browser, they can’t do it. And since all browsers have to use WKWebView, the limitation applies to all browsers. Apple has basically crippled other browsers with this anti competitive behaviour.
Like wish granted! You can now use Blink and V8 on iPhones! But…
* They still are required to delete website data after 7 days.
* They must distribute add-ones through the App Store.
* They still can’t run workers in the background because the browser is slept like any other app.
* They have to use Apple’s notification service like any other app.
* They can’t install things to the Home Screen because no other app can.
But yayyyy. Blink!
Same thing happened to Do Not Track... that was used for fingerprinting
I don't think this web app is going to do as well on Apple devices because of the lack of notifications and also the difficulty to add to home screen.
BTW, you have to give the web app permission to notify. It's not like native apps.
Apple has clearly crippled PWAs to maintain their walled garden.
I'm sure there are always going to be individuals that will claim that they're better at everything, but that doesn't really mean anything. There are people that believe in a flat earth as well after all
This reminded me of the writings of a german Philsopher, Heidegger maybe? The proposition was, that development happens in cycles similar to a pendulum swinging while moving upwards. Thesis and Antithesis leading to Synthesis.
That argument was more usually a straw man than a good faith belief.
On the other hand the belief that SQL will solve all our problems seems to be legitimately held by some people.
Regardless of how you view it, the benefit of hindsight shows the exact thing that people warned would happen did in fact happen (a widespread venerability in SQLite exposed across various browsers). Its also a fairly strange point to be personally insulting people involved in the process whose careers are doing perfectly well.
Back in the day, there were people who strongly suggested MongoDB and IndexedDB were the future, and that PostgreSQL, MySQL, SQLite were trash. I've noticed the folks who rode that hype-train moved into other kinds of occupations that aren't exactly engineering-focused anymore.
The biggest problem is that yeah, WebSQL tends to be faster than IndexedDB. Or at least it was back when I was working on PouchDB. Biggest issue IIRC was that joins were faster in SQLite than implementing the same thing in userland on top of IndexedDB. Browsers eventually shipped getAll/getAllKeys which also helped with cursor slowness.
I haven't looked much at the Storage Foundation API [1], but it seems like a more reasonable approach moving forward. Just give developers the low-level tools and let them build SQLite on top of it. Also the Chromium devs have been working on relaxed durability, which apparently improves IDB perf in some scenarios [2] (although still not as fast as Firefox it seems [3]).
[1]: https://github.com/WICG/storage-foundation-api-explainer
[2]: https://www.chromestatus.com/feature/5730701489995776
[3]: https://bugs.chromium.org/p/chromium/issues/detail?id=102545...
1. The API is the dogshit hot mess you'd expect for a pre promise/async API.
2. The lack of partial/computed secondary indexes.
3. Apple/Safari does EVERYTHING in their power to break it. I refuse to believe it's incompetence at this point, it's actively malicious.
Six houses, all alike in dignity, In fair IRC, where we lay our scene, From ancient grudge to new mutiny, Where civil blood makes civil hands unclean. From Oracle, that SQL seer of IndexedDB, To Google, the stronghold of search, We add Mozilla, the Web SQL killa, And Apple, peering from its mobile perch. Here, a storage war would set keys to clack, Tongues to wag, and specs to shatter, There was also Microsoft and Opera, Who don't really seem to matter.
THE PLAYERS
NIKUNJ MEHTA, of House ORACLE, an instigator JONAS SICKING, of House MOZILLA, an assassin MACIEJ STACHOWIAK, of House APPLE, a pugilist IAN FETTE, of House GOOGLE, a pleader CHARLES MCCATHIENEVILE, of House OPERA, a peacemaker
ACT 1
SCENE: A dark and gloomy day in Mountain View, or perhaps a bright and cheery one, depending on your IRC client's color scheme.
> Firefox: Negative [1]
> Safari: Negative [2]
However, I fully expect Chrome to ship it in stable sometime soon, like they do with dozens of other APIs.
There are now four different half-baked storage/file api proposals, at least one of them is already in stable Chrome... it's a mess.
[1] https://github.com/mozilla/standards-positions/issues/481
[2] https://lists.webkit.org/pipermail/webkit-dev/2021-February/...
There are other valid arguments, but "use a native app for smaller file size" specifically doesn't make sense at all - the opposite is true.
While webapps may be larger than simple web pages, they're still far far far smaller than native apps, and using service workers etc, you do only pay the download size for a web app once, just like a native app.
For example: the Twitter Android app is a 21MB download. I just tested loading the Twitter PWA (https://mobile.twitter.com) logged in but with an cleared cache: it's a <3MB, including all the content visible in my feed on the first load.
The native app is _7+ times larger_.
I suspect the same or worse is true for the vast majority of other apps, e.g. Uber on Android is a 50MB download, while m.uber.com (reloaded after login with a cleared cache) gives you the same features with <1MB.
Using native UI frameworks will almost always yield smaller file sizes than the equivalent implementation in any of the web technologies, regardless of whether served as a webpage, or bundled in a WebView or something similar (with few exceptions, one of which is listed below). Just consider a WinForms/GTK/Qt or a similar application in contrast to using React/Angular/Vue and including those within Electron, or even loading them in a web browser.
Many of the mobile applications out there actually include hybrid technologies (like Ionic, React Native, Xamarin or anything else of the sort), since nowadays telling a company that you'll have separate codebases with OS native widgets for iOS, Android and others would not be met positively. Therefore, concessions are made to speed up development, at the expense of having more abstraction layers and larger bundles.
One aspect in which you're probably correct, however, is that technically the web indeed affords you the possibility of having pretty good download space savings, if you choose which forms to make user download, as opposed to including all of them in the application. For example, if you have a billing form that actually needs 50 different variations based on countries and regions, and include them bundled into a native app, then on the web you'll instead be able to just download the one that you need dynamically. That said, none of this is alway easy, hence Twitter and Uber both employ a large number of engineers that work on screens like that, even if many people won't even see them.
My point here is that there are many different approaches and technologies that can be used, "native" in the context of my message meaning an application that uses the OS UI and widgets directly, without reinventing the wheel or attempting to encapsulate cross platform or web technologies within it. Of course, it's perfectly understandable and valid why companies that are highly focused on their own brands and shipping quickly don't necessarily adopt that approach.
Being able to reach all devices and have a super fast experience without consuming bandwidth is always good, but not always a high priority .
There are also aspects which are probably entirely overlooked because of that approach, for example, consider:
- https://whatdoesmysitecost.com/test/210813_AiDcEB_e62852d10cb795009ccd61f1dd2d8623#gniCost
- https://whatdoesmysitecost.com/test/210813_AiDcPZ_dd4b52c10c37e4c7483ab391015a00cb#gniCost
If the target audience of your products and sites are wealthy and able bodied individuals in 1st world countries, then by all means go ahead. However, in certain other pursuits inclusivity is definitely an important goal as well and one that i personally value a bit higher than fancy looking sites with rich functionality.
That said, isn't 20 seconds a bit much? Sure, i get that it's just an example, but i think even heavier pages like GMail load in approximately half of that. I think that bundle splitting and some optimization are probably a good idea even in heavier pieces of software!
Ah yes. Once Chrome releases something in stable, literally no one ever uses those APIs, no one. And even Google's propaganda machine doesn't tell you to use them (example: https://web.dev/usb/)
> Or are you against experimentation?
I'm not. What Google does isn't experimentation.
> I'm not. What Google does isn't experimentation.
So the way this is done now (with origin trials to collect feedback from developers) is not good enough? What exactly is your definition of experimentation?
It doesn't really matter how many. It means that people are already using this functionality, and depend on it. I didn't link to an article on removing alert just for the fun of it.
> So the way this is done now (with origin trials to collect feedback from developers) is not good enough? What exactly is your definition of experimentation?
Running this as origin trial is a very good way of doing it. What's not a good way is:
- having an unrealistic timeline.
Example: Mozilla was asked for position on WebHID three months before Chrome released in stable. The "standard" was so badly written that Mozilla engineers couldn't understand it. Chrome still released it, and updated the "standard" two months after it was already in the wild.
- ignoring any and all objections, and favoring own needs only
Example: Constructible Stylesheets. The spec has a trivially reproducible problem. Several developers (not only from Mozilla and Safari) pointed this out, and proposed several changes to the spec that would get rid of the problem. However, Google's own lit-html was interested in the spec. So they released it in stable, said that "0.8% of pageviews now use this" (reality: only lit-html used it at the time), and refused to hide it back under a flag. Safari simply said they are not going to implement the spec in this shape.
- completely ignoring the realities and needs of the web
See https://dev.to/richharris/stay-alert-d and my comment here: https://dev.to/dmitriid/comment/1h5bh
- presenting these "standards" as fait accompli even if other browser vendors will never ever implement them
They have co-opted web.dev as a propaganda machine. The site presents itself as a neutral resource for web develoeprs. It's not, it's a Chrome-only vehicle. They present various specs as already available and never specify issues with those specs. For example, WebUSB: https://web.dev/usb/
See, e.g., Mozilla's position on various specs. Scroll down to "harmful": https://mozilla.github.io/standards-positions/ I'll let you guess how many of those have already been shipped in Chrome. Then you can find what web.dev says about it
- gaslighting other browser vendors
I won't link to specific tweets because I don't need to be angry. But almost every single of Google's high visibility "standards people" and "developer advocates" and "community managers" will always, and I mean absolutely always paint all other browsers in as negative light as possible (as lagging behind, as hurting the web, as damaging the web, as being a detriment to the web etc.)
Unfortunately, the term has been co-opted to mean "website I can install/pin as an app". Again, Google is probably to blame for this, but AFAICT it was never meant to be the meaning of the term. What it does do is create misunderstandings, like a sibling thread claiming that (desktop) Firefox doesn't support PWAs because you can't install anything.
[1] https://developer.mozilla.org/en-US/docs/Web/Progressive_web...
And "progressive enhancement" just means "being written in such a way that you'll work with a core set of APIs — all the way down to 'no javascript support' — but will take advantage of further APIs if they're available." Where the app forms a sort of Russian nesting doll of functionality, where it "launches" with just the innermost layer (e.g. just server-rendered HTML), and then in turn each new layer is enabled if possible (i.e. if Javascript is enabled, the browser will load some JS from a <script> tag that will then take over the DOM and turns the page into a SPA; etc.)
It's just a further refinement of the older concept of "graceful degradation", where the gracefully-degraded experience is delivered at the start, and then bootstrapped out into the non-degraded version.
Under this definition, the "further layers of functionality" provided to "installed" webapps in some browsers are just a specific kind of progressive enhancement. The app detects that it's being run "installed", and takes advantage of that. (And before that, the browser detects that the app has such a layer, and therefore treats the webapp as "installable.")
Thus, all "installable" apps are PWAs. (But not all PWAs are "installable"!)
I would note that webapps like Slack — i.e. webapps that have both web and "native" versions, where the "native" version is just the same remote webapp from the web version, loaded in a special browser that exposes extra capabilities that the webapp then takes advantage of — are also PWAs by this definition.
Why the non-relational databases keep getting reinvented has been a bit of a mystery when there is already a rich history of development to look to. I get the feeling a lot of the industry isn't much into history, especially of pre-micro computer systems.
Network databases, key-value stores, graph databases, commercial offerings like Pick... you'd think the NoSQL people would have looked into it all before proclaiming their new found solutions, but apparently not.
He is paraphrasing: "Those who cannot remember the past are condemned to repeat it." - George Santayana, The Life of Reason, 1905.
That inspired René Magritte to paint this: https://americanart.si.edu/artwork/those-who-cannot-remember...
Making the connection to events in our lifetime: https://hankeringforhistory.com/those-who-cannot-remember-th...
(edit) Some more reflections on it: https://blog.rtcx.net/remember