Show HN: ObjectiveFS – Shared file system for EC2 and laptop

Show HN: ObjectiveFS – Shared file system for EC2 and laptop(objectivefs.com)

27 points by objectivefs 10 years ago | 21 comments

mieses 10 years ago |

How is different from s3ql? https://github.com/s3ql/main

Edit: Support for concurrent mounts seems to be the main difference https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!can-i-access-a...

objectivefs 10 years ago | |

S3QL has a lot of great features. You are right that support for concurrent mounts is one of the main differences. ObjectiveFS also keeps the file system data consistent and stores it in S3 asap, for sharing and to handle EC2 unexpected termination. S3QL uploads all metadata changes at a regular interval (by default every 24h) or during unmount.

hedgehog 10 years ago | | |

Rough guess on the design: The main metadata history is in a primary log, then each client extends that with its own branch of the history for filesystem operations. Logs are probably some kind of Merkle tree. Bulk data is content addressed and referenced from the logs, maybe with large objects split using a rolling checksum like Adler-32 to reduce object sizes and allow for partial updates to large objects. Someone (you guys?) runs a coordination service routes notifications between clients and manages leader election. The elected client reads all of the logs, does conflict resolution locally, and then updates the primary log and collects garbage.

One drawback to this design would be that many small files in the filesystem would translate to many small objects in S3 (with associated operations). One solution would be to put small objects right into the metadata log. Alternately they (or maybe all objects) could be put into a log-structured merge tree.

Another problem with this design is that S3 doesn't support append operations so sync latency would be bounded by client log flush intervals, again creating lots of small objects. Maybe the coordination service routes some of the data to manage this?

Anyway, really interesting design problem. Is this close?

NeutronBoy 10 years ago |

Awesome project!

It might just be a release timing thing, but what's the advantage of this over Amazon Elastic File System [1]? I see you've got a tagline on your signup No need to wait for EFS, are you planning a feature comparison or similar?

[1] https://aws.amazon.com/efs/

edit: derp. Turns out I should have read further!

How is ObjectiveFS different from Amazon EFS?

ObjectiveFS uses highly durable object stores (Amazon S3 and Google Cloud Storage) to store your files. Your data is accessible from all regions, your office and your laptops. Performance scales with your object store.

ObjectiveFS supports Linux and OS X and is running in production today.

Amazon EFS is NFS-based, with access to each file system limited to EC2 instances in the same region. It is not available today and will only be in limited preview for some customers in summer 2015. It will cost $0.30/GB and performance will scale with the number of GB stored.

https://objectivefs.com/faq#how-is-objectivefs-different-fro...

graceofs 10 years ago | |

Thanks! We plan to publish a feature comparison table soon. A couple more differences:

1. We have strong end-to-end integrity checks and encryption on by default.

2. Amazon EFS's performance is dependent on the filesystem size: larger filesystems include more iops.

objectivefs 10 years ago |

We built a shared file system with an S3 backend. We've been running our own servers for many years, and wanted something low maintenance, reliable, scalable and secure. So, we built this POSIX-compliant shared filesystem for Linux and Mac.

We love to hear your feedback. Feel free to try it out. We will also be here to answer questions.

bio4m 10 years ago |

How is this different from say mounting s3 as a volume with fuse ?

graceofs 10 years ago | |

It's a complete POSIX filesystem, and we implement a log-structured filesystem on top of S3. So, you can use it with your regular programs expecting POSIX semantics.

Mounting S3 lets you view the objects in a bucket, but it doesn't behave like a regular file system.

x5n1 10 years ago | | |

https://github.com/s3fs-fuse/s3fs-fuse

"large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes"

benwilber0 10 years ago | |

judging from https://objectivefs.com/features

sounds like maybe they're just packaging s3fs-fuse and then charging for it "as a service"? they're extremely light on technical details.

graceofs 10 years ago | | |

Hi Ben, thanks for your feedback. We will add more technical details and clarify the differences between objectivefs and s3fs-fuse.

ObjectiveFS is a log structured filesystem that we implemented completely from scratch. It is a POSIX filesystem and works with the existing Linux and OS X tools and software.

FullyFunctional 10 years ago |

This would have been nearly perfect for me if it had supported disconnected operation with later synchronization (~ Dropbox, CODA, ..). Alas, it's a live fs. (Dropbox is great except it doesn't encrypt locally and has to have a local copy of everything, unlike a cache).

hrez 10 years ago |

No compression?

objectivefs 10 years ago | |

ObjectiveFS comes with fast compression that is always enabled, it works great for log files and other compressible data. Let us know what you think if you give it a try.

hrez 10 years ago | | |

Thanks. It would be nice to put on a feature list along with some details.

benwilber0 10 years ago |

website is slow as shit

graceofs 10 years ago | |

Thanks for letting me know. We are getting a lot of traffic, and working on fixing it.

UPDATE: the website should be faster now.