Datastore API (beta)

207 points by gluegadget 12 years ago | 84 comments

peterkelly 12 years ago |

The biggest problem right now with online file storage services (sorry, I meant "The Cloud") is that while they all do basically the same stuff, there's no standard protocol for accessing them. It's a bit like if every website used a custom protocol instead of HTTP, and web browser vendors had to explicitly collaborate with each web site (or use the website's SDK) in order to be able to retrieve content.

Imagine this: All the major players in the space (Dropbox, Box, Google Drive, Skydrive, Sugarsync etc.) as well as app developers got together and came up with a standard way of doing this that makes it possible to use a single protocol & API for all online storage providers. I see this being a good thing for everyone - users can choose whatever service they want, app developers can easily provide their customers with a ton of options, and storage providers don't have to explicitly produce their own SDK or court developers to do integration with their service. We did have WebDAV, but that doesn't seem to be widely adopted by any of the major players.

I've been facing this problem myself with my own app; it's a lot of work to support all the major services and this is really just plumbing stuff that everyone would rather be taken care of so they can focus on the more important stuff. I had a meeting with Box recently where I proposed the idea of an open source framework for iOS which would provide a common native API that abstracts over all the different REST APIs used by different storage providers. The people I spoke to seemed quite open to the idea and agreed that it would be useful. I don't think a library like this would necessary be considered a bad thing by any of the major players, as it evens out the playing field and makes it easy to support different services.

If you're interested in taking part and/or using such an API in your iOS app, I'd love to hear your thoughts on the matter. I have some code already (which I haven't released yet) and am looking to clean up the API a bit, but I think it would be useful to a lot of developers.

davidjgraph 12 years ago | |

That's exactly the idea of Remote Storage [0]. The problem is here that the lowest common denominator of functionality isn't actually very much.

Anything but fairly trivial apps quickly run into problems with application data (how relevant), file patching, file intents (Google have this),

Google have, IMO, the most advanced solution in the space, so you could argue have the most the lose from standardising. But in fairness, some of their stuff (I'm thinking the real-time collaboration part of the Drive API) is so complex, there's no way you'd ever see it in a common standard.

That said, it'd be really nice to see a common API over the lower stuff.

[0] http://remotestorage.io/

peterkelly 12 years ago | | |

Thanks for the link, this does look very interesting.

I believe filepicker.io have something along these lines as well, but haven't had a chance yet to look into the details of it.

icebraining 12 years ago | |

Big players aren't interested in anything that reduces lock-in. Seamless integration is one of the keypoints they have to get you to use their services.

Obligatory presentation by Bruce Schneier on the online feudalism: http://www.infoq.com/presentations/cloud-security

peterkelly 12 years ago | | |

I found Box to be the opposite of this. While I obviously can't speak on their behalf, and I was discussing the issue with their developer relations people (not senior executives), they said they didn't see a problem with this idea.

DTE 12 years ago | | |

exactly...it is unlikely they would be interested in data storage being treated as the commodity it is.

nly 12 years ago | |

By Jove, you're right! Perhaps what we need is some kind of universal protocol for transferring files. We could call it the 'File Transfer Protocol' or something.

roc 12 years ago | | |

As it turns out, the protocol for actually transferring the files is utterly irrelevant compared to the protocol for getting updates, keeping various local stores synchronized, retaining versions, managing conflicts, handling client-side bugs and surprises, etc.

peterkelly 12 years ago | | |

LOL yes ;)

What happened to FTP anyway? I know there were a number of issues with it (security, working through NAT, lack of a standard directory listing format etc); but these were all problems that people managed to solve in different ways.

It just goes to show how far how our industry is willing to embrace change for the sake of change (with the results sometimes being inferior) every time a new buzzord appears. I reckon a modernised version of FTP would do the job pretty well.

What I'm mainly concerned with at the moment is building something that recognises a widely-adopted protocol is going to be very difficult politically to achieve (and get implemented), but a library which abstracts over the different protocols would relieve most of the pain.

taskstrike 12 years ago | |

Hey, this is exactly what NimbusBase (http://nimbusbase.com) wants to do. It's a work in progress to integrate all the services, but we got the major 2 Dropbox and Google Drive.

brettcvz 12 years ago | |

Check out https://www.inkfilepicker.com/

munificent 12 years ago |

I'd like to coin a modern update of Zawinski's Law[1]:

    Every software company attempts to expand until it
    offers a cloud-based platform API. Those companies
    which cannot do so are replaced by ones which do.

[1]: http://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski.27s_law...

mmahemoff 12 years ago |

Something I don't get about Dropbox API: Why can't I just throw URLs at it?

If a user wants to save <thing at URL>, I have to pull it to my server and push it into Dropbox. Dropbox could just have grabbed it directly and in many cases, won't even have to thanks, to system-wide caching.

(This is based on the last time I looked at the API; maybe this is supported now, but I don't think so.)

timdorr 12 years ago | |

You're describing the Saver Drop-In: https://www.dropbox.com/developers/dropins/saver

natch 12 years ago | | |

Can the Saver Drop-In be used with arbitrary third-party URLs? To me this is what was being described, so if not, then no, it's not describing the Saver Drop-In.

mmahemoff 12 years ago | | |

Great to know they got around to this and did so in a way that leverages the existing HTTP caching protocols. Thanks!

JosephRedfern 12 years ago | |

How would system-wide caching work unless the HTTP server dropbox requested the file from sent a checksum of the file being requested? Just because something has the same URL doesn't mean that it's the same file at the other end (http://wordpress.org/latest.tar.gz for instance)

varenc 12 years ago | | |

It obey's HTTP caching controls. Checkout the end of the docs: https://www.dropbox.com/developers/dropins/saver

oellegaard 12 years ago |

Such an awesome idea. However, in light of the recent events the first thing to come up in my mind when I see "datastore" and a US corporation is "PRISM integration". I'd like to see proper encryption of Dropbox as the next feature, so that we - innocent users- are not uploading our data to the US government.

Afforess 12 years ago | |

Can't you just encrypt the data before storing it in your app? Why does Dropbox need to do this for you? Would you even trust Dropbox if they did add it themselves?

peterkelly 12 years ago | |

Well, the nice thing about PRISM is that it's API is so simple that you don't even need to muck around with SDKs, REST, JSON encoding etc. Integration is completely straightforward.

lovemenot 12 years ago | | |

Expect to see the NSA as a back-up-of-last-resort cloud solution provider any time soon?

redblacktree 12 years ago | |

I combine dropbox with TrueCrypt for sensitive stuff. For me, that's just digital copies of important documents.

aasarava 12 years ago | | |

So are you just syncing entire truecrypt .tc files to Dropbox? Does that work incrementally or does the entire .tc file have to get resent every time you modify a file within?

philip1209 12 years ago | | |

Yes - in my dropbox public folder, I keep a truecrypt bucket with scans of everything in my wallet, should I lose it.

hcarvalhoalves 12 years ago | |

The right approach is to encrypt before pushing down the wire - it's useless if Dropbox just encrypts on their end.

r00fus 12 years ago | |

Sadly, you're going to need to encrypt before uploading. Just as the webserver should never trust the client, the user should never trust the server with sensitive data.

Try boxcryptor - works with more than just Dropbox.

duaneb 12 years ago | |

Proper encryption meaning metadata is encrypted as well? Isn't that the whole appeal of http://mega.co.nz/?

jonaldomo 12 years ago |

After quickly clicking through the api documentation I don't understand why I would use this over a hosted NoSql database like mongohq. I still like having additional options though.

forgotAgain 12 years ago | |

Off-line access.

icefox 12 years ago | |

Perhaps you don't want to host a db just for user data.

jimbokun 12 years ago |

"List – A special value that can contain other values, though not other lists."

https://www.dropbox.com/developers/datastore/tutorial/ios#re...

Seems like a very significant limitation. No mechanism for storing hierarchical data structures. For example, this is certainly not capable of storing arbitrary JSON data as a Record.

est 12 years ago | |

There's also limit on Postgres hstore and arrays. No nested sturcture.

makeramen 12 years ago |

As someone who has been playing around with Parse and Firebase SDKs (on Android), this seems to be the first one that offers full 2-way sync. I haven't actually used it yet but it seems that way due to the presence of conflict resolution [1]. I'm personally really excited about that for mobile side-projects and MVPs.

Parse has an offline cache, that only caches query results and not actual objects. So any local saves/updates are not reflected until you get network access again, unless you manually implement your own cache.

Firebase is better at handling realtime changes, but is lacking in any realtime capabilities at the moment. Also Firebase's mobile authentication SDK still isn't released yet so no way to handle users yet either.

[1]: https://www.dropbox.com/developers/datastore/tutorial/androi...

pavpanchekha 12 years ago | |

Yeah, the Datastores API does two-way synchronization. Also, offline caches are done for you and offline changes are propagated (with conflict resolution) when the internet comes back up.

makeramen 12 years ago | |

Correction: "lacking in any realtime capabilities" should read "offline capabilities". And by that I mean offline without manually managing cache.

Also by 2-way sync I mean 2-way cache sync. Firebase is actually really good with 2-way sync, but without cache/offline (yet) it's kind of a dealbreaker for me.

obilgic 12 years ago |

What app.net wanted to accomplish.

WalterGR 12 years ago | |

At first glance it also looks like WinFS for the Cloud.

jimsilverman 12 years ago |

this is basically what iCloud was supposed to be.

sigil 12 years ago |

So now app developers have a choice: they can use iCloud or Dropbox to manage distributed app state. Any iOS app developers (I'm not one) care to compare & contrast the two options?

It looks like Dropbox may be using Operational Transformation [0] to merge changes to lists:

Lists are special field values. They contain an ordered list of other values, though not other lists. Lists can be manipulated via four list operations: put (i.e. replace), move, insert, and delete. These list operations allow Dropbox to handle merging changes to the structure of the list automatically. [1]

iCloud, on the other hand, uses the coarser, state-centric approach of whole document versioning. [2] (Edit: okay, key-value property versioning is also available.)

But iCloud lets the app control conflict resolution, while the Dropbox Datastore API only lets you choose from a predefined set of conflict resolution strategies. [3]

Which approach provides greater value to the app developer? Has Dropbox realized here that developers are just not comfortable implementing a custom merge strategy for entire documents, and not having to worry about this is worth some loss of control?

[0] http://en.wikipedia.org/wiki/Operational_transformation

[1] https://www.dropbox.com/developers/datastore/tutorial/ios#re...

[2] http://developer.apple.com/library/ios/#documentation/DataMa...

[3] https://www.dropbox.com/developers/datastore/tutorial/ios#co...

hboon 12 years ago | |

Not answering your question, but developers can also use http://parse.com or similar.

alexobenauer 12 years ago |

For the iOS library specifically, I wonder if it would be possible to build a library that would (nearly) automatically map a core data store in to this Datastore API. If so, including such functionality would make the library a no-brainer for many.

ZitchDog 12 years ago | |

I think with NSIncrementalStore it would be totally doable, and a great alternative to Core Data Sync.

fireboi 12 years ago |

are there any kind souls that could help please explain the difference between dropbox datastore, firebase and parse? thank you.

sankethkatta 12 years ago | |

In comparison to firebase, this solves the offline sync problem, if your user is offline, you can still save to dropbox datastore locally, when you get a connection, it will automatically sync and take care of conflict resolution.

mikelehen 12 years ago | | |

Firebase dev here. :-) Firebase does handle offline syncing / conflict resolution.

If you have a Firebase app open and you lose network connectivity, the app still continues to work fine and any modifications you make will be synced back to Firebase when you regain connectivity.

What we don't (quite yet) do is handle the case where you make changes, kill the app (without regaining network connectivity), restart the app, and then get network connectivity, but that's coming soon.

As for other differences, Dropbox Datastore seems to be tightly tied to Dropbox, so the end user for your app must have a Dropbox account (and if they want to collaborate with other users, those users must also have Dropbox accounts, etc.). So it's really for building apps on top of Dropbox.

Firebase and Parse are just generic backends, with no ties to other services, so your end users don't need anything to use your app.

As for differentiating Parse and Firebase, Firebase deals with data in real-time, pushing updates to apps as soon as data changes. Parse is a traditional request/response model, where your app has to explicitly "refresh" to get new data.

joeblau 12 years ago |

Does anyone know of an OSS alternative to this similar to http://helios.io/ that supports Android/iOS?

jchrisa 12 years ago | |

Couchbase Lite is an embedded JSON database with sync capability. It's Apache licensed and available for iOS and Android:

https://github.com/couchbase/couchbase-lite-ios

https://github.com/couchbase/couchbase-lite-android

joeblau 12 years ago | | |

Thanks for the link to this library. It seems like I would still need to implement a lot of features like login, but it's a great start.

BadassFractal 12 years ago |

What is the long term vision here? Is the idea that consumers will have little apps here and there that will procedurally interact with the owner's data on Dropbox? Or, is Dropbox making a product along the lines of S3 so that backends can consider using it too? I guess given how expensive space is at this point, that's not really an option.

chacham15 12 years ago | |

dhouston has said more than once that he wants dropbox to be the filesystem of the internet.

frankcaron 12 years ago |

I wonder how Apple will react to @dropbox's new Datastore sync in the long run. Hopefully it won't pull the rug out a la IAP, UDID, OF, ...

jemeshsu 12 years ago | |

I would rather they react positively by opening up their iCloud API for web application for third parties. Apple already has such in place as can be seen from their cloud apps at icloud.com.

duaneb 12 years ago | | |

Apple is a hardware company.

callumjones 12 years ago | |

I highly doubt they would do anything to stifle Dropbox's syncing functionality in iOS. Apple is still focused on selling iOS devices and ensuring people keep buying iOS devices, they will do anything in their power to keep you happy.

Limiting Dropbox functionality on iOS would be met my swift user opposition, something that doesn't guarantee recurring sales.

joejohnson 12 years ago |

So Dropbox is becoming a more-direct competitor with iCloud. Meh.

I'd rather they work on an end-to-end encryption system with a focus on ease of use.

Afforess 12 years ago | |

Hacker News is a tiny audience. I don't see any business sense in focusing on features that 0.1% of users want.

albertyw 12 years ago | | |

Not to sound self-righteous, but the HN audience (or more plausibly, some subset within) has a high influence on what the rest of the world sees and wants.

Any Dropbox employees read HN?

iamclovin 12 years ago | |

I think this is exactly the direction they should take.

afarrell 12 years ago | |

If they did want to serve this audience, they would actually want to make it easy for someone (possibly the employees themselves) to build a Free encryption layer on top of them.

johnrob 12 years ago |

Grammar check: "These days, your app need to store and sync more than just files."

modarts 12 years ago |

Why would I use this over something like Parse?

joeblau 12 years ago | |

I haven't looked into it but it may be cheaper. That being said, both Parse and Azure support more platforms than this offering. I think that even Firebase has an service in this space as well.