Local-first software (2019)(inkandswitch.com) |
Local-first software (2019)(inkandswitch.com) |
Unfortunately, if you go to ChatGPT and ask it to build a website/app, it immediately points the unknowing user towards a bunch of cloud-based tools like Fly.io, Firebase, Supabase, etc.
Getting a user to install a local DB and a service to run their app (god forbid, updating said service), is a challenge that’s complex, even for developers (hence the prevalence of containers).
It will take some time (i.e. pre-training runs), but this is a future I believe is worth fighting for.
Not sure where your experience is coming from but when I asked an LLM, Claude to be more precise, it referred me to local options first, such as SQLite. It didn't consider cloud platforms at all until I had asked, presumably because it can understand local code and data (it can query it directly and get back results) but cannot understand the context of what's in the cloud unless you configure it properly and give it the env variables to query said data.
In my experience it’s great at utilizing local storage and SQLite, if you ask it to.
I just asked the ChatGPT web client (4o, as that’s what most non-developers might default to):
> Can you build me a website for my photos
And it immediately started suggesting Wordpress, Wix, Squarespace, etc.
Specifically, this was section 4 of the various questions it asked me:
> 4. Tech Preference (optional)
> - Do you want this as a static HTML site, WordPress, or built with something like React, Next.js, or Wix/Squarespace? > - Do you need help hosting it (e.g., using Netlify, Vercel, or shared hosting)?
As a non-programmer, I likely wouldn’t understand half those words, and the section is marked optional.
If I follow the “default path” I’m quickly forking over a credit card and uploading my pictures of dogs/family/rocks to the cloud.
I would be surprised if in a decade we won't have local models that are an order of magnitude better than current cloud offerings while being smaller and faster, and affordable ASICs to run them. That'll be the first real challenger to the internet's current position as "the" place for everything. The more the web gets enshittified and commercialized and ad-ridden, the more people will flock to this sort of option.
This is nuts. Computers are supposed to enhance and enable thinking, not make you stupid. In this sense, cloud software is possibly the biggest fraud ever perpetrated on the paying, computer-using public.
For the love of God, please bring back my late 1990s and early 2000s brain-boosting computer experience.
Essentially antithetical to capitalism, especially America's toxic late stage subscription based enshittification.
Which means its typically a labor of love or a government org has a long term understanding of Software as a Infrastructure (as opposed to SaaS)
I would sooner trust a GPL-licensed remote software program than store a kilobyte of personally identifying information in a proprietary "local first" system.
[0] https://www.macrumors.com/2023/12/06/apple-governments-surve...
Conflict-of-interest transactions have hidden or coercive impact, lined up in favor of the party with stronger leverage. Examples include un-asked and unwanted surveillance of data or activity, coercive use of information, vendor lock in, unwanted artificial product/service dependencies, insertion of unwanted interaction (ads), ...
None of that is inherent to capitalism. They clearly violate the spirit of capitalism, free trade, etc.
It is providers taking advantage of customer lack of leverage and knowledge to extract value that does not reflect the plain transaction actually desired by customers. Done legally but often with surreptitious disclosure or dark pattern permissions, border line legally where customers would incur great costs identify and protest, or plain old illegally but in a hidden manner with a massive legal budget to provide a moat against accountability.
It is tragic that the current generation of Silicon Valley and VC firms have embraced conflict of interest based business models. Due to the amounts of money that scaling "small" conflicts can make. Despite the great damage that we now know scaling up "small" conflicts can do.
That was not always the case.
If we want a functioning market based system then we need to explicitly correct for this by aggressively taxing the wealthiest entities (individuals and companies) in our society to bring things closer to a level playing field.
Free trade is antithetical to capitalism. Free trade means everyone is on a level playing field, but capitalism means those with more capital are above the rest. These are obviously not compatible.
Not for a PWA.
The article posits it as though subscription software is something which has been sneaked in on us. But users today expect things like instant updates, sync across devices, collaboration, and constant bug fixes and patches - none of which come easily if you're only willing to pay for the system once.
Oh but it has (IMO).
> users today expect things like instant updates [...] constant bug fixes and patches
Nah, this is in reverse. With boxed software, the developer had to deliver an essentially bug-free product. Now, with easy updates technically possible, the developers have gone complacent, and deliver shit. That is why users expect bugfixes instantly. (And any enlightened user abhors unrequested features, as there are no features without regressions, and who wants regressions in any serious application?) The only tolerable online updates are security fixes.
> sync across devices, collaboration
This is a valid expectation, but its execution has been a train-wreck. Research, design and implementation should start with end-to-end encryption; the network architecture should be peer-to-peer (mesh, not centralized). What do we get instead? More centralization of control than ever, and less privacy and ownership than ever.
Good point. Governments would do this if they really worked "for the people"
The backend for my personal notes, tasks, bookmarks, calendar and feeds are files in directories synced with Syncthing across devices.
I ended there after going from one app to another and being tired of all this.
It is self hosted with no server backend (beyond a Syncthing on a NAS or VPS, optional). It is very reliable and works without Internet connection.
I could have put everything in sqlite too and sync it one way or another, but it seemed already too complicated for my requirements.
I can't share it beyond my close relatives but I had the same problem with people using Google or Microsoft before.
We do use online services like firebase for auth, and some service to fetch commodity prices etc, but rest of the data is stored in browser storage (sqlite) and backed to local disk (and soon dropbox). We also syncs data across devices, always encrypting data in transit.
I think it's the way to go, for most personal data applications.
That said, the app is currently designed for desktop use—mobile UX is still on our roadmap.
As for signup: it helps us track and bill users. Building our own auth system for local-first would’ve been a full project on its own. Until better options exist for authorization and billing in local-first apps, we’ll stick with a cloud signup flow.
- because I could not find something similar that doesn't milk and own my data
- to never lose a bookmark again
- to have my bookmark data encrypted in the cloud
- to have private history
- to have some extra time saving features in the extension that are for unknown reason rare to find
- more learning and experience (it's acutally quite complex to build this)
Whether you need a spinner or not should be decided by the User Experience (e.g., when the user has to wait for more than 100ms, show a spinner), and not by the location of the data. I am a big fan of local-first apps and enjoy building them myself. However, sometimes your app takes a moment to load. With local-first, you eliminate the network as a source of delays, but there are other factors as well, such as large data sets or complex algorithms.
For example, when you have a project planning software and want to plan 100 work packages with multiple resource combinations in an optimal way, depending on the algorithm, this can take some time. In that case, a spinner or a progress bar is a good thing.
Spinners should not exist in a local first app.
I mean, I did it, I built an app with a transparent background sync. Then I added a special page, 'sync center'.
In reality, mobile devices don't always have perfect network connections. Therefore, when the user is unsure whether the device is in sync or if the sync is in progress but encounters an obstacle, they might perceive the app as unreliable.
Banning spinners is dogmatic, not user-centric.
> Cloud apps like Google Docs and Trello are popular because they enable real-time collaboration with colleagues, and they make it easy for us to access our work from all of our devices. However, by centralizing data storage on servers, cloud apps also take away ownership and agency from users. If a service shuts down, the software stops functioning, and data created with that software is lost.
"Apple pie might be tasty and nutritious and exactly what you want, but, theoretically, apple pie could burst into flames someday, and take your favorite pie-eating bib with it.
If there is anyone interested in working on such projects - let's talk! We can't leave our future to greedy surveillance zealots.
I love this article, but the section on security raised a lot of questions. What's the model for authorizing access to documents for collaboration? How do you managed keys safely for encrypted data? How do users recover "lost" keys?
Cloud computing models have a lot of security mechanisms built-in. You might not like the model (AWS IAM for example) but at least there's a foundation already in place.
There are other options for key storage, revoking group privileges, etc. It's an extensive topic, but the foundation is there, it just depends on your network and use cases.
NC itself gets you file sync and webdav etc. An add on gets you the webby version of LibreOffice. You can bolt on AI addons to classify and tag your images/photos and with a bit more effort, your docs too.
It's properly local first.
Its not local-first or some sort of cloud diet trend, it should be the norm.
"Sharing models" are totally irrelevant until the concept of self-determination is tolerated by the powerful (and they will never tolerate it). Accessing my data from multiple devices is totally secondary; I don't trust mobile endpoints to access my remote data in the first place.
This is the model I want from 90% of the software out there, just give me a reasonable price to buy it, make the product good, and don't marry it to the cloud so much that its unusable w/out it.
There are also a lot of added benefits to this model in general beyond the data privacy (most are mentioned in the article), but not all the problems are solved here. This is a big space that still needs a lot of tooling to make things really easy going but the tech to do it is there.
Finally, the best part (IMHO) about local-first software is it brings back a much healthier incentive structure - you're not monetizing via ads or tracking users or maxing "engagement" - you're just building a product and getting paid for how good it is. To me it feels like its software that actually serves the user.
What is sad is that they used to be local files first note app and then they moved to sqlite citing some sync and performance issues.
While this sounds good deal, with this approach
- You have to charge total cost of subscription at once (1y or 2y),
- Still have to keep servers running for syncing, also you have think about cases where user syncing 1y of data in a single day.
- Have to keep people on the payroll for future developments.
(You are here thinking only in developer perspective.)
Asking for a friend . . .
Yes, you are. You can find tons of purely local apps that monetize themselves with ads.
How do they do that without hitting the internet?
A notepad also isn't enough to correlate heart rate etc to specific exercises and plotting over time
AI photo and video generation is impractical to run locally.
ComfyUI and Flux exist, but they serve a tiny sliver of the market with very expensive gamer GPUs. And if you wanted to cater to that market, you'd have to support dozens of different SKUs and deal with Python dependency hell. And even then, proficient ComfyUI users are spending hours experimenting and waiting for renders - it's really only a tool for niche artists with extreme patience, such as the ones who build shows for the Las Vegas Sphere. Not your average graphics designers and filmmakers.
I've been wanting local apps and local compute for a long time, but AI at the edge is just so immature and underpowered that we might see the next category of apps only being available via the cloud. And I suspect that these apps will start taking over and dominating much of software, especially if they save time.
Previously I'd only want to edit photos and videos locally, but the cloud offerings are just too powerful. Local cannot seriously compete.
You think it always will be? What can the new iPhone chips do locally?
There was a great panel discussion this year from a number of the co-authors of the the paper linked, discussing what is Local-first software in the context of dev tools and what they have learnt since the original paper. It's very much worth watching: https://youtu.be/86NmEerklTs?si=Kodd7kD39337CTbf
The community are very much settling on "Sync" being a component of local first, but applicable so much wider. Along with local first software being a characteristic of end user software, with dev tools - such as sync engines - being an enabling tool but not "local first" in as much themselves.
The full set of talks from the last couple of years are online here: https://youtube.com/@localfirstconf?si=uHHi5Tsy60ewhQTQ
It's an exciting time for the local-first / sync engine community, we've been working on tools that enable realtime collaborative and async collaborative experiences, and now with the onset of AI the market for this is exploring. Every AI app is inherently multi user collaborative with the agents as actors within the system. This requires the tech that the sync engine community has been working on.
https://news.ycombinator.com/item?id=19804478 - May 2019, 191 comments
https://news.ycombinator.com/item?id=21581444 - Nov 2019, 241 comments
https://news.ycombinator.com/item?id=23985816 - Jul 2020, 9 comments
https://news.ycombinator.com/item?id=24027663 - Aug 2020, 134 comments
https://news.ycombinator.com/item?id=26266881 - Feb 2021, 90 comments
https://news.ycombinator.com/item?id=31594613 - Jun 2022, 30 comments
https://news.ycombinator.com/item?id=37743517 - Oct 2023, 50 comments
Connected appliances and cars have got to be the stupidest bit of engineering from a practical standpoint.
I’ve been building the offline-first (or local-first) app Brisqi[0] for a while now, it was designed from the ground up with the offline-first philosophy.
In my view, a local-first app is designed to function completely offline for an indefinite period. The local experience is the foundation, not a fallback and cloud syncing should be a secondary enhancement, not a requirement.
I also don’t consider apps that rely on temporary cache to be offline-first. A true offline-first app should use a local database to persist data. Many apps labeled as “offline-first” are actually just offline-tolerant, they offer limited offline functionality but ultimately depend on reconnecting to the internet.
Building an offline-first app is certainly more challenging than creating an online-only web app. The syncing mechanism must be reliable enough to handle transitions between offline and online states, ensuring that data syncs to the cloud consistently and without loss. I’ve written more about how I approached this in my blog post[1].
[1] https://blog.brisqi.com/posts/how-i-designed-an-offline-firs...
The business is doing alright, but what really keeps me going is that the app is something I genuinely wanted for myself. I use it all the time, it's always open in the background, so that keeps me motivated.
I looked over your careers page and see all of your positions are non-remote. Is this because of limitations of working on local-first software require you to be in-person? Or is this primarily a management issue?
The problems with closed-source software (lack of control, lack of reliability) were solved with a new business model: open source development, which came with new licenses and new ways of getting revenue (maintenance contracts instead of license fees).
In the same way, we need a business model solution to cloud-vendor ills.
Imagine we create standard contracts/licenses that define rights so that users can be confident of their relationship with cloud-vendors. Over time, maybe users would only deal with vendors that had these licenses. The rights would be something like:
* End-of-life contracts: cloud-vendors should contractually spell out what happens if they can't afford to keep the servers running.
* Data portability guarantees: Vendors must spell out how data gets migrated out, and all formats must be either open or (at minimum) fully documented.
* Data privacy transparency: Vendors must track/audit all data access and report to the user who/what read their data and when.
I'm sure you can think of a dozen other clauses.
The tricky part is, of course, adoption. What's in it for the cloud-vendors? Why would they adopt this? The major fear of cloud-vendors is, I think, churn. If you're paying lots of money to get people to try your service, you have to make sure they don't churn out, or you'll lose money. Maybe these contracts come only with annual subscription terms. Or maybe the appeal of these contracts is enough for vendors to charge more.
It’s based on NixOS to provide as much as possible out of the box and declaratively: https, SSO, LDAP, backups, ZFS w/ snapshots, etc.
It’s a competitor to cloud hosting because it packages Vaultwarden and Nextcloud to store most of your data. It does provide more services than that though, home assistant for example.
It’s a competitor to YUNoHost but IMO better (or aims to be) because you can use the building blocks provided by SelfHostBlocks to self-host any packages you want. It’s more of a library than a framework.
It’s a competitor to NAS but better because everything is open source.
It still requires the user to be technical but I’m working on removing that caveat. One of my goals is to allow to install it on your hardware without needing nix or touching the command line.
If you really embrace "local first" just use the file system, and the user can choose from many solutions like git, box, etc.
I hate signing up for your sync just as much as any other SAAS, but it's even more opaque and likely to break.
First is that yeah, local first, but I also want concurrency. If it's just local first, you're right, any old sync will do. But I want more than that. I want to not have to think (a la dropbox, being slick). I want my wife and I to be able to make separate edits on our phones when we're in a dead zone.
Second is that sync works a lot better when it has deep knowledge of the data structure and semantics. Git and box both have significant shortcomings, but both exacerbated by the concurrency desire.
If files are insufficient, what data-structure would make modular sync possible for multiple applications in an OS?
And I’m not suggesting one doesn’t exist, I’m challenging to present a comprehensive solution, that probably involved operating systems.
> I want my wife and I to be able to make separate edits on our phones when we're in a dead zone.
Files do this.
In practice, it’s hard! You’re effectively responsible for building a sync engine, handling conflict resolution, managing schema migration, etc.
This said, tools for local-first software development seem to have improved in the past couple years. I keep my eye on jazz.tools, electric-sql, and Rocicorp’s Zero. Are there others?
But once you get past toy examples, you start wanting to support operations like "edit", and there generally isn't a way to infer the user's intent there. Like, if my cookie recipe starts with 100g of sugar, and I modify it on my phone to use 200g of sugar, and I modify it on my desktop to use 150g of honey instead of 100g of sugar, there are a bunch of ways to reconcile that:
1. Stick with 200g of sugar, drop the 1.5x honey substitution.
2. Stick with 150g of honey, drop the 2x.
3. Merge them - 300g of honey.
4. Merge them - 150g of honey and 50g of sugar.
There's no way for any automated system to infer my intent there. So you've got to either:
1. Ask the user to resolve the conflict. This means you have to build out the whole "resolve this merge conflict for me" UI and the promise of "conflict-free" has not been fulfilled.
2. Arbitrarily choose an option and silently merge. This risks badly surprising the user and losing changes.
3. Arbitrarily choose an option, but expose the fact that you've auto-resolved a conflict and allow the user to manually re-resolve. This requires even more UI work than option 1.
4. Constrain your data model to only allow representing intents that can be deterministically resolved. In practice I think this is too severe of a constraint to allow building anything other than toy apps.
IMO #1 and #3 are the least-bad options, but I don't think they're consistent with the expectations you'd have for CRDTs after reading this article.
(FWIW, https://automerge.org/docs/reference/documents/conflicts/ is the relevant documentation for their Automerge library. It looks like they've chosen option 3.)
But it is a nightmare when it goes wrong: the conclusion I've reached is that it is out of reach to regular people who don't want the Byzantine support load that could accompany something going wrong. They want turnkey. They want simple. They aren't interested in operating services, they're interested in using them.
The FLOSS model of self hosting doesn't really offer a reliable way of getting this: most businesses operating this way are undercapitalised and have little hope of ever being any other way. Many are just hobbies. There are a few exceptions, but they're rare and fundamentally the possibility of needing support still exists.
What is needed, imo, is to leverage the power of centralised, professional operations and development, but to govern it democratically. This means cooperatives where users are active participants in governance alongside employees.
I've done a little work towards this myself, in the form of a not-yet-seen-the-light-of-day project.
What I'd love to see is a set of developers and operators actually getting paid for their work and users getting a better deal in terms of cost, service, and privacy, on their own (aggregate) terms. Honestly, I'd love to be one of them.
Does anyone think this has legs to the same extent as local-first or self hosting? Curious to know people's responses.
I've been working on Relay [0] (realtime multiplayer for Obsidian) and we're trying to follow tailscale's approach by separating out the compute/document sync from our auth control plane.
This means thats users still subscribe to our service (and help fund development) and do authn/authz through our service, but we can keep their data entirely private (we can't access it).
[0] https://relay.md
Originally the idea was to keep everything within the Obsidian UI so things like username/password didn't make sense (no password managers in Obsidian).
We initiate the OAuth2 login flow from within Obsidian. I guess we could add an extra click that takes you to our website first and then add support more auth methods from there. I don't really want it to feel like a web app though.
I'd love to hear your take. Which login method do you think is both simple and could be coherently used within Obsidian on all platforms?
The user-friendliness challenge is real though. Setting up Audiobookshelf [1] is more work than "just sign up," but once you have it running, the local-first client becomes much cleaner to build. No user accounts, no subscription billing, no scaling concerns. Simple pricing too: buy once, own forever. No monthly fees to access your own audiobooks.
[1] https://tiamat.tsotech.com/pao (2012)
Moreover, local-first —at least in theory— enables less infrastructure, which could reignite new indie open source software with less vendor lock-in.
However, despite all my excitement about embracing these ideas in the pursuit of better software, there's one hurdle that preventing more wide spread adoption amongst developers, and that is the Web platform.
The Web platform lacks building blocks for distributing hashed and/or signed software that isn't tied to origins. In other words, it's hard to decouple web-apps from the same-origin model which requires you set up a domain and serve requests dynamically.
Service Workers and PWAs do help a bit in terms of building offline experiences, but if you want users to download once, and upgrade when they want (and internet is available), you can't use the Web. So you end up breaking out of the browser, and start using Web technologies outside of the browser with better OS functionality, like Electron, React Native, Tauri et al (the https://userandagents.com/ community is doing some cool experiments in this space).
I view everyone flocking around Electron as proof of a failure on this front.
I held off on playing with Typst for years because I was under the (incorrect) impression that the only way to use it was with their web editor. I'm sure that their editor is completely fine, but I am pretty entrenched in Neovim and Pandoc had been serving me well.
Once I found out that Typst has a command line version that I can use directly, it became more appealing, because I'm pretty sick of cloud shit.
I'm personally very against vendor lock in, but there is some value to them.
This seems like a bold claim, but IMHO Ink & Switch have earned their solid reputation and it wouldn't surprise me if it's true. I agree w/ their analysis and am philosophically aligned w/ their user-centric worldview. So who's going to build "Firebase for CRDTs"?
Do you actually need anything special for CRDTs over a normal database? My understanding is the actual CRDT part is done "client side"
* Download all your data from Microsoft's "OneDrive" cloud storage, which if not disabled, is the default storage method in a new Windows install.
* Verify that all your files are now stored locally.
* Click the gear icon, go to "Settings -> "Account" -> "Unlink this PC," right-click, "Unlink account".
* Remove Microsoft's OneDrive app from your system -- full removal is the only way to prevent perpetual harassment and reactivation. Go to "Apps" -> "Apps & features" (or "Installed apps" on Windows 11) -> "Microsoft OneDrive", right-click, "Uninstall."
* Optional extra step: cancel your Microsoft 365 subscription and install LibreOffice (free, open-source).
Wanting to be able to run AI fully privately, and offline is the reason we created Cactus:
https://github.com/cactus-compute/cactus
Fully open-source, cross-platform & blazing-fast; lets you plug in private AI into any app on your phone.
- Password manager: KeyPassXC
- Notes: Logseq
- Analytics: Plausible
- Media: Jeyllyfin
- Uptime kuma
- Finance tracker: Actual Budget etc is too heavy so I built this. https://github.com/neberej/freemycash/
- Search: Whoogle? is kinda dead. Need alternative.
An OS agnostic apps, meaning web apps is such a killer feature. You can use the same app on: Linux, Mac OS, Windows, Android, iOS
Even more, developing for web is typically faster. You made the change in the code => you see the result on the screen. For example: phone apps written in swift could be faster than ones written in react-native, but it is so annoying waiting for the compilation to finish after every small change.
----
When I worked on imagery data in an autonomous vehicles company product managers pushed us to explore the data in the cloud and it was soooo inconvenient.
As the result, PMs were ignored and everyone had a personal desktop with GPUs and fast SSDs that had the local copy of the data, so that debugging, prototyping would be fast.
As lag that one gets working with a heavy data remotely reminded moving back from SSD to slow HDD, where you needed to wait some time to see the result on the screen.
It was only half a second every time, but felt ultra annoying.
Since it needed to access users' local photo libraries, I didn't want the app to connect to the internet under any circumstances. So I made it a paid app instead of the usual free+in-app purchases model, since the latter requires calling StoreKit which goes online. But because the app had to run the CLIP model, it would crash on lower-performance phones like the iPhone X. Users who paid for it couldn't use it and felt scammed, leading to tons of one-star reviews and angry complaints about their photos being stolen. Eventually I decided to open-source the app, though it never brought me much revenue anyway.
Two years later, Apple started announcing they'd be integrating this exact feature into Apple Intelligence : )
It’s self reinforcing because those companies that get subscription revenue have both more revenue and higher valuations enabling more fund raising, causing them to beat out companies that do not follow this model. This is why local first software died.
It's sad because the dynamics and incentives around clear, up-front prices seem generally better than SaaS (more user control, less lock-in), but almost all commercial software morphs into SaaS thanks to a mix of psychology, culture and market dynamics.
There are other advantages to having your software and data managed by somebody else, but they are far less determinative than structural and pricing factors. In a slightly different world, it's not hard to imagine relatively expensive software up-front that comes with a smaller, optional (perhaps even third-party!) subscription service for data storage and syncing. It's a shame that we do not live in that world.
Example: I made a firefox extension that automatically fills forms using LLM. It's fully offline (except OPTIONALLY) the LLM part, optionally because it also supports Ollama locally.
Now the issue is that it's way too hard for most people to use: find the LLM to run, acquire it somehow (pay to run it online or download it to run in Ollama) gotta configure your API url, enter API key, save all of your details for form fulling locally in text files which you then have to backup and synchronize to other devices yourself.
The alternative would be: create account, give money, enter details and all is synced and backedup automatically accross devices, online LLM pre-selected and configured. Ready to go. No messing around with Ollama or openrouter, just go.
I don't know how to solve it in a local way that would be as user friendly as the subscription way would be.
Now things like cars and washing machines are a different story :p
When Apple joined the madness, all hopes where lost (that was a long time ago now, sight)
So you can run 1000 local first app that syncs to a Dropbox for that 10/m in storage. And that storage is full B2C level ready to go not some low level s3 like primitive. Has auth, has supported has programs to sync.
Really most of the cloud cost is not needed.
Nobody is forcing anybody to make their games rely solely on online services. It's not a legal requirement, regulatory requirement, or anything else. It is a choice, like most things in software. To make the choice to rely on online services and then say "we'll have to spend money later to unfuck this!" is honestly short sided, pathetic, and nobody should accept it.
Whenever it's possible to solve a business problem or political problem with a technical solution, that's usually a strong approach, because those problems are caused by an adversarial entity and the technical solution is to eliminate the adversarial entity's ability to defect.
Encryption is a great example of this if you are going to use a cloud service. Trying to protect your data with privacy policies and bureaucratic rules is a fool's errand because there are too many perverse incentives. The data is valuable, neither the customer nor the government can easily tell if the company is selling it behind their backs, it's also hard to tell if he provider has cheaped out on security until it's too late, etc.
But if it's encrypted on the client device and you can prove with math that the server has no access to the plaintext, you don't have to worry about any of that.
The trouble is sometimes you want the server to process the data and not just store it, and then the technical solution becomes, use your own servers.
For something like data portability--being able to take my data to a different provider--that probably requires a technical solution.
But other problems, like enshittification, can't be solved technically. How do you technically prevent a cloud vendor from changing their pricing?
And you're right that the solution space is constrained by technical limits. If you want to share data with another user, you either need to trust a central authority or use a distributed protocol like blockchain. The former means you need to trust the central provider; the latter means you have to do your own key-management (how much money has been lost by people forgetting the keys to their wallet?)
There is no technical solution that gets you all the benefits of central plus all the benefits of local-first. There will always be trade-offs.
What if cloud platforms were more like brokerage firms? I can move my stocks from UBS to Fidelity by filling out a few forms and everything moves (somewhat) seamlessly.
My data should be the same way. I should be able to move all my data out of Google and move it to Microsoft with a few clicks without losing any documents or even my folder hierarchy. [Disclaimer: Maybe this is possible already and I'm just out of the loop. If so, though, extend to all SaaS vendors and all data.]
This calling back might amount to taking delivery. In a banking context, that is where the user takes delivery of whatever money and other property is in the account. In the cloud vendor case, this would be the user receiving a big Zip file with all the contents of the account.
Taking delivery is not always practical and is also not always desirable. Another option in a financial context is transferring accounts from one vendor to another: this can take the form of wiring money or sometimes involves a specialized transfer process. Transferring the account is probably way more useful for many cloud services.
This leads us to a hard thing about these services, though: portability. Say we delineate a clear property interest for user's in their cloud accounts and we delineate all of their rights. We have some good interests and some good rights; but what does it mean to take delivery of your Facebook friends? What does it mean to transfer your Facebook account from one place to another?
I don't think that's quite correct. I think the authors fully acknowledge that the business case for local-first is not complexly solved and is a closely related problem. These issues need both a business and technical solution, and the paper proposes a set of characteristics of what a solution could look like.
It's also incorrect to suggest that local-first is an argument for decentralisation - Martin Kleppmann has explicitly stated that he doesn't think decentralised tech solves these issues in a way that could become mass market. He is a proponent of centralised standardised sync engines that enable the ideals of local-first. See his talk from Local-first conf last year: https://youtu.be/NMq0vncHJvU?si=ilsQqIAncq0sBW95
And while they spend a lot of time on CRDTs as a technical solution, I didn't see any suggestions for business model solutions.
In fact, if we had a business model solution--particularly one where your data is not tied to a specific cloud-vendor--then decentralization would not be needed.
I get that they are trying to solve multiple problems with CDRTs (such a latency and offline support) but in my experience (we did this with Groove in the early 2000s) the trade-offs are too big for average users.
Tech has improved since then, of course, so maybe it will work this time.
This is not practical for data of any size. Prod migrations to a new database take months or even years if you want things to go smoothly. In a crisis you can do it in weeks but it can be really ugly, That applies even when moving between the same version of open source database, because there's a lot of variation between the cloud services themselves.
The best solution is to have the data in your own environment to begin with and just unplug. It's possible with bring-your-own-cloud management combined with open source.
My company operates a BYOC data product which means I have an economic interest in this approach. On the other hand I've seen it work, so I know it's possible.
It seems like you'd need the latter to truly be immune to cloud-vendor problems. [But I may not understand how it works.]
It is not only a business problem. I stay away from cloud based services not only because of subscription model, but also because I want my data to be safe.
When you send data to a cloud service, and that data is not encrypted locally before being sent to the cloud (a rare feature), it is not a question of if but when that data will be pwned.
The experience has made me a big fan of self hosting.
And the biggest advantage I see of this perspective over the "technical problem" perspective is that assigning responsibility completely covers the problem space, while "hope that some clever math formula can magic the problem away" does not.
I'm trying to imagine how this would be enforced when a company shutters and it's principals walk away.
My company does that with a few small vendors we've got for the source code we depend on.
But that's the point of contracts, right? When a company shuts down, the contracts become part of the liabilities. E.g., if the contract says "you must pay each customer $1000 if we shut down" then the customers become creditors in a bankruptcy proceeding. It doesn't guarantee that they get all (or any) money, but their interests are negotiated by the bankruptcy judge.
Similarly, I can imagine a contract that says, "if the company shuts down, all our software becomes open source." Again, this would be managed by a bankruptcy judge who would mandate a release instead of allowing the creditors to gain the IP.
Another possibility is for the company to create a legal trust that is funded to keep the servers running (at a minimal level) for some specified amount of time.
Anecdotally, I’ve never worked anywhere where the data formats are documented in any way other than a schema in code,
Not necessarily. I like local-first due to robust syncing via CRDTs, not because I somehow want to avoid cloud providers.
"Mechanically preventing wrongdoing from happening" can be a bit of a Shangri-La. What Tech can mechanically do is increase the cost of wrongdoing, or temporarily deflect attempts towards easier targets. But that by definition cannot "solve the problem for everyone" as there will always be a lowest hanging fruit remaining somewhere.
What contracts can do is help to reduce the demand for wrongdoing.
It is like that we must protect users privacy from greedy websites so we will make the bad ones spell out that they use cookies to spy on users - and the result is what we have now with the banners.
Just as with the open source movement, there needs to be a business model (and don't forget that OSS is a business model, not a technology) that competes with the old way of doing things.
Getting that new business model to work is the hard part, but we did it once with open source and I think we can do it again with cloud infrastructure. But I don't think local-first is the answer--that's just a dead end because normal users will never go with it.
It's ok to just solve the problem and let the businesses fail. Predation is healthy for the herd. Capitalism finds a way, we don't have to protect it.
Also, so many of these selfhostable apps are web applications with a db, server and frontend, but for a lot of use cases (at least for me personally) you just use it on one machine and don't even need a "hosted" version or any kind of sync to another device. A completely local desktop program would suffice. For example I do personal accounting once a month on my computer – no need to have a web app running 24/7 somewhere else. I want to turn on the program, do my work, and then turn it off. While I can achieve that easily as a developer, most of the people can't. There seems to be a huge misalignment (for lack of a better word) between the amount of high-quality selfhostable FOSS alternatives and the amount of people that can actually use them. I think we need more projects like yours, where the goal is to close that gap.
I will definitely try to use selfhostblocks for a few things and try to contribute, keep it up!
Feel free to give it a try though, I’d love that! Also feel free to join the matrix channel UF you have any questions or just to get some updates.
Also some more pondering on local-first application development from a "few" (~10) years back can be found here: https://unhosted.org/
I should probably write a blog post, but I will say that I investigated power sync, electricSQL, livestore and powersync before. I briefly looked at jazz tools but wanted something a bit more structured.
I'm pretty impressed this far. I've actually been writing it with Vue and a community library. Permissions were a bit tricky, but once I figured it out it was simple. And I like their magic email login. And I like their dashboard/reply, but there are a few big changes I would make there to make it less fiddly.
I love that it's open source, and that if I want to, I could self host it.
As for the other options:
- jazz wasn't structured enough
- livestore came off as too fiddly with the event store, but it was appealing. That the dev tools are payealled was disappointing, but understandable
- electriSQL really only provided half a solution (read, not the write model
- couchDB / pouchDB wasn't structured enough for me, and I wanted better cross document support than was obvious / baked in.
- did not investigate zero really
Brightened reading this. If you have any feedback please let us know! We on the discord, and answer over on founders@instantdb
EDIT: actually I wanted to point to the "landscape" link (in the top menu) but that URL is quite unergonomic.
Rust and JavaScript implementations, a handful of network strategies. It doesn't come with the free or paid offering that jazz.tools does, but it's pretty nice.
It’s a local-first platform that supports real-time sync with CRDTs at its core, making conflict resolution much easier to manage. Ditto is designed to handle offline-first use cases and peer-to-peer sync out of the box, so you don’t have to build a custom sync engine from scratch.
It supports a wide range of platforms including Swift, Kotlin (Android), Flutter/Dart, React Native, JavaScript (Web/Node), .NET (C#), C++, Java, and Rust. You can dive deeper into what it offers from the docs site: https://docs.ditto.live/home/about-ditto
Here is a good recap of the current players. https://www.localfirst.fm/landscape
It's also open source and has bindings for Dart, JS, Swift, C#, Kotlin, etc
This is what every "cloud file sharing" provider like Dropbox is doing. If there is a conflict, the version on the server is "the right one", and your locally conflicted file is copied on the side with some annotation in the file name.
That being said, yes, i do believe *in the near/upcoming future* local-first, self-hosting and i will add more fair open source vendors will work! Well, at least, i hope so! I say that because Europe's recent desire to pivot away from the big U.s. tech companies, and towards more digital sovereignty - in my opinion - begins the foundational dependency for an ecosystem that will/could sustain self hosting, etc. The more that europe is able to pivot away from big tech, the more possibilty exists for more and varied non-big tech vendors manifest...and the more that Europe adopts open source, the more the possibility that usage and expertise of self-hosting grows....plus, for those who do not know how to, or simply do not wish to manage services themselves...well, in time i think Europe will have fostered a vast array of vendors who can provide such open source, digital services, but get paid a fair cost for providing fair value/services, etc. ...and, by the way, i say this all as a biased person in favor of open source AS WELL AS being an American. :-)
Utopia. Unattainable. Self-determination of the individual has been consistently persecuted under all societal arrangements; communism and capitalism equally hate a citizen that wants to remain independent and self-sufficient.
Yes, there might be a breakthrough or a bug in encryption, and jnless you've been targetted, you can respond. But we've seen and experienced breakdowns in human character (employees spying on customers, stealing data...), government policies and company behaviour to trust the complexity and cost (lawyers) of enforcing accountability through policy.
In general, you do need both, but if you've got one, to engineers, technical solution is usually more appealing.
The entire point of Chapter 11 (and similar bankruptcy legislation internationally) is to allow companies to get out of contracts, so that they can restructure the business to hopefully continue on as a going concern.
The asset in the contract is their customer's data; it is becoming stale by the minute. It could be residing in debtor-owned hardware and/or in data centers that are no longer getting their bills paid.
It takes time to get a trustee assigned and I think we need an immediate response - like same day. (NAL but prep'd 7s & 13s)
The big problem comes into play for new, or more custom types of applications. It takes a while for something to become ubiquitous enough that standard formats are developed to support them.
I suspect we're a decade off from being able to generate Veo 3, Seedance, or Kling 2.1 videos directly on our phones.
This is going to require both new compute paradigms and massively more capable hardware. And by that time who knows what we'll be doing in the data center.
Perhaps the demands of generating real time fully explorable worlds will push more investment into local compute for consumers. Robotics will demand tremendous low latency edge compute, and NVidia has already highlighted it as a major growth and investment opportunity.
But it's even worse in that case, because that can also happen if they mess something up. Your email account got banned by some capricious bot, or the provider abruptly decided to stop providing the service, and then the service tied to it decided to send you a verification code to the email you don't have access to anymore -- even though you didn't forget your password for either of them. So now you have even more ways to lose all your stuff.
Meanwhile if you were willing to trust some email provider to not screw you and you only needed some way to recover your keys if your computer falls into the sea, you could just email a copy of them to yourself. And then you wouldn't be relying on that provider to have the only means of recovery, because they're still on your device too.
Through regulating markets to ensure fierce competition - including things like portability, standard APIs, banning egress fees and similar lock in techniques, breaking up infrastructure (DCs and networking) from service providers. In cloud we have 3 vertically integrated mega oligopolies. That’s not a healthy market.
> data portability […] probably requires a technical solution
Yes, formats and APIs are needed for technical reasons, but it already exists (or fairly trivial to implement) and is not provided – sometimes actively obstructed – for business reasons. Imo interop is predominantly bottlenecked by social/business concerns.
This is different from what the local-first article is describing, which addresses data for individuals. That's a much harder problem to solve at scale.
That, and also there are real benefits to the end user of having everything persisted in the cloud by default.
But by then my extension probably also won't be needed anymore as it would likely be integrated in the OS.
That plus web dev is trendy and everybody is learning it. I wouldn't know how to code a proper desktop app right now, I've not done it in years. I don't want to criticize that or the centralization aspect – there will still be ways to put these centralized things on a PC for example.
Only useful thing from Garmin app has been comparing heart rates to a year ago.
My understanding is maybe use an open source provider to mitigate vendor lock in?
>3. The network is optional
Ad SDKs usually allow caching ads for a period of time so that ads can still be shown while the device is temporarily offline.
IMHO, a fully local app is an app that can run locally with all the functionality, not that it's isolated from everything else.
Browser, email client (running locally on your device such as Mail.app, mutt, Outlook,...), Zed (text editor, runs locally but can check for updates... as can many other modern apps)...
there are different kinds of ads, but lets be clear that even a Show HN is a form of ad. Some forms of ads are just more appreciated than others.
Pen and paper is severely underrated today.
Note that this does not even need two users: I hit this problem with a desktop and laptop and self-hosted NextCloud myself.
In general, a filesystem that actually stored both raw data (to fail-over to), but also a per-format event log, and maybe even app specific events (imagine a PNG changes, we could have any change recorded as raw bytes, generic bitmap image operation like "modify pixels at x,y to ..." and app-specific log like "gimp: apply sharpen filter on polygon area ...").
This would allow the other side to attempt to do the smartest sync it has (if it has a compatible version of gimp, it could decide to apply the filter, otherwise fall back to raw pixel changes if no conflicts, and then fall back to full file contents reconciliation).
Just like MIME handlers get registered, if file systems provided such change logs, some could have very advanced sync systems with this support from "filesystems".
I’m also suspect of logs as a general form of conflict resolution as you are just hoping the two edits don’t touch the same area. And if they do then you are left in an invalid state.
You brought up zips. Pile of files seems like a way you can divide up data so it can have more pieces that are mergable/diffable.
For example “the folder can contain N files” or “there must be exactly 1 of this file”.
For zip and other container-type files, you'd have log entries to the tune of "changed contained file foo.png: ...".
Operating systems would need to support some basic operations: container file operations like zip files, basic bitmap image editing, basic text document diffing, structured text diffing (XML, JSON, YAML...), etc...
Applications would provide OS-registered services (like MIME handlers are registered today) that can interpret and produce even more semantic events on top of the existing ones.
The environment could offer an interface during "syncing" when it detects a conflict to resolve it using one of the generic (or not) conflict resolution mechanisms (use local or remote version completely; use incremental delta if there is some generic semantic diff; app-provided capability if present on both sides).
Now, you are right that this can be implemented with this log being a regular file next to the file itself and completely user-space, but you will hit issues if you are not able to tie it nicely to things like fwrite/fsync and similar syscalls.
Obviously, for it to make sense, it needs to be widely accepted as the approach, which is what the local-first movement is trying to achieve with CRDTs.
I think it boils down to provenance and concurrency. If we edit the same line a file, that's ba merge conflict when it really should be simple and something I shouldn't have to bother with. And when we do do the same line edit, I'd love to have provenance on that data.
Granted, those aren't local first thing exactly, but I think there will be apps that want all of that.
If that's true, it's a substantial upset to the old paradigms of data and computing.
I still purport that in the future, most applications and screen time will fall into a generative AI bucket: creating media, writing code, watching videos, playing games, searching for information, etc. I wouldn't even be surprised if our personal images and videos get somehow subsumed and "enriched" with AI.
[1] Well, email might fall into a non-generative bucket. There are already tools that purport to both read and write your emails for you. I'm not quite sure what to make of those.
Good thing I'm not talking about data consumption apps then, as I mentioned in my comment above. Local-first apps specifically are not amenable to data consumption purposes so while you are right on the generative AI part, it's unrelated to the topic of this post.
Related: I've been incubating an idea for a while that open source, as it presently stands, is largely an ecosystem that exists in support of cloud SaaS. This is quite paradoxical because cloud SaaS is by far the least free model for software -- far, far less free than closed source commercial local software.
It's the same thing as the subscriptions for movies like Netflix, except at least in the last case we can fight back with various means (and it's not a necessity).
The SaaS model is basically a perfect racketeering setup, I think it should be outlawed at least philosophically. There is no way business is not going to abuse that power and they have already shown as much...
I agree with your sentiment on Open Source. I think like many of these types of things, it lives in contradictions. In any case, Linux as it is today, couldn't exist without the big commercial players paying quite a bit to get it going.
No, what we need is a way for people to not starve so that they don't have to make money at all and can focus instead on their passion project(s). Cough UBI cough
If we really wanted a system where we deem certain items essential and went everyone to have access to them, it makes no sense to pay for them. Money may still make sense for nonessential or luxury items, but it just gets in the way if the government has to give me money so I can go spend it on the food they actually want me to have.
Sure there is: “$500 upfront or $21/mo for 24 months *”
* if you don’t complete you 24 payments, we freeze your license.
Obsidian is doing a pretty good job selling sync functionality to their free client. Because the have a really good markdown editor implementation IMHO with community plug-in support that IMHO beats every PKM cloud tool out there that competes with them.
They're still a local-first note application. It's just slightly harder for you to edit your notes externally, and not even by that much - it's very easy to directly query (read and write) SQLite databases, and if you really cared you could have made a script to grab a note, export it to a temporary text file, allow you to edit it, then update the SQLite database.
> I can't version controlled backup and sync those files
You absolutely can - you can dump SQLite databases to text files that contain SQL queries that will restore the database that you can then backup and sync: https://stackoverflow.com/questions/75675/how-to-dump-the-da...
> then they moved to sqlite citing some sync and performance issues
Yes, that's because "plain text" files are bad for performance and harder to sync correctly. For people who (1) have over a hundred thousand notes they want to keep (like me) and (2) want maximum confidence that they're not going to lose years worth of work, that's incredibly important.
The devs made the right choice. You can always write scripts to interface with a SQLite database with an external editor. You can't take plain text files and magically make them as fast and durable as a database.
Bear devs advise against doing that: "Generally speaking, it is safe to access the database for reading only"
It's still worth pointing out that this design decision is orthogonal to the decision to use SQLite, though. The Bear devs could have designed it such that you could write to the database directly, or they could have kept the text-file-based design, but still told the user that they can't modify the files directly (as one of my personal projects does). The assignment of blame to SQLite specifically is misguided.
> It's just slightly harder for you to edit your notes externally
Yup, just slightly harder! Very slightly. A difference of 3.75 picoseconds. Couldn't agree more.
> it's very easy to directly query
Right!
> and if you really cared..have made a script
And designed a nuclear reactor while I was at it, isn't it?
> The devs made the right choice
Yessss!! Finally.
Ffs!!!
Also interesting to note that you couldn't actually muster a coherent response to any of my points and just had to make ad hominem attacks and emotional outbursts.
Claude can one shot this.
And as for sqlite being local first - it's not the same. It's the nature of the data a plain text note taking app deals with that neuters that very idea of those plain text notes by storing in a local sqlite db.
It's possible most ubiquitous open source software ever (far more common than markdown) and your notes exist in fully readable text form inside it
Bear being proprietary is the real threat of lock in
sqlite3 data.db 'select text from notes'
How would you design a system that uses SQLite as primary storage for notes (rather than just as a search index or metadata store) while still letting people edit those notes in any text editor, sync via any cloud and make them accessible to any AI or other third party software that knows how to use a file system?
Exporting and re-importing every single time you want to edit a note is impractical for tech folks and impossible for regular users.
I can imagine a partial solution using FUSE, FileProvider and whatever the respective file system abstractions are on other platforms. But that would be a huge amount of work.
This is just unbelievable! Fucking pitchforks are out literally!
I am dealing with plain text notes and you all want me to write sql queries and scripts to access those fucking text files?
Are you all (these few people who just jumped in the subthread) pulling some sort of prank of so?
I don't want you to do anything. Use whatever you want! But if you love your tool except for the fact it uses an open database format instead of text files, then lucky you, there's a solution, and it takes all of five minutes.
I'll admit that I was unclear because I used the word “starve” in my message. Obviously this choice of word was borne out of the principle that food is among the most vital basic needs, and that depriving people of it is one prominent way in which our current society is cruel. Nevertheless, UBI is not about just food, but more generally about basic needs.
Surely it wouldn't just be food, but regardless the government would be coming up with a list of items and basing the UBI on that. If those items are deemed necessary enough that everyone should have access, why not make them freely available rather than abstracting through money first?
For a UBI to exist the government has to define some kind of system for how to set the basic income level, I assume they would define a basket of goods (and services) similar to how CPI is calculated and define a formula for how the UBI is calculated from there. One way or another, someone in the government would be coming up with a list of what is "essential" and deciding how much to subsidize.
They would also need to similarly define how the UBI is adjusted over time. Prices change over time, and there would likely be a pretty quick jump in prices when the UBI is first put into effect.
All that to say, we will never have a UBI without it being centrally planned from the top. Whether it is authoritarian or not would be up for debate, it can be centrally planned and pushed from the top down without being authoritarian - unless one thinks we're already authoritarian, most of our current federal programs would fit into that category.
It's not about telling you what you should need or live without. It's about providing a service.