Membase a new NoSQL

58 points by leej 15 years ago | 44 comments

wvenable 15 years ago |

"For those familiar with memcached, membase provides on-the-wire protocol compatibility, but adds disk persistence; hierarchical storage management; data replication; live cluster reconfiguration and rebalancing; and secure multi-tenancy with data partitioning. Like memcached, membase is simple, fast and elastic."

I bet a lot of large sites using multiple memcached servers would find this interesting.

sqrt17 15 years ago | |

... as in, those people who use memcachedb but would like to have something bigger?

pcarmichael 15 years ago |

The timing for me is interesting as I was just about to start up a cassandra setup here. This might be a better fit for my particular needs if I could get it running. Unfortunately, I'm having a difficult time figuring out where to start. A general getting started guide would be very helpful.

- The FAQ on membase.org is empty and the wiki does not appear to be populated with much content (i.e. 'Directions for working with the membase source and descriptions of the components in the source will be available soon').

- The google group does not appear to have significant content besides a 'Where's the ruby driver?' post.

- My work blocks IRC, so jumping on freenode isn't an option.

- I tried downloading from the northscale.com site, but for whatever reason I cannot enter the required fields on the download form (in Firefox 3.6.4).

- I downloaded the source, which is broken out into a number of directories - figuring out what the various subsystems are and what needs to be done is going to take some digging.

dlsspy 15 years ago | |

Thanks for reporting the FF issue. The site's been updated and tested again.

We've obviously got a bit to go from our internal build farms to things that are easier for other contributors to play with. Documentation for developers is still being written.

Forgetting everything we know about the things we've been doing for a while so we can explain it to people who haven't been doing it has proven to be a bit harder than we initially thought. :)

mileszs 15 years ago | |

I offer no counter or dissent to your post. However, apparently, the binaries have not yet been posted. Perhaps once they get that crucial detail finished, things will be clear. It appears they haven't even reached an actual beta stage. It might have been better for them to wait until everything was ready to go before launching the site publicly.

Considering the 'front-end' of membase is memcached code, you should be able to use any library that interfaces with memcached to interface with membase.

Out of curiosity, what drew you towards Cassandra over, say, Riak or MongoDB?

pcarmichael 15 years ago | | |

MongoDB was ruled out because auto-sharding is still alpha (though it should be out of alpha soon, I'd prefer to wait and see what the community experience is with it first.)

Between Cassandra and Riak, I chose Cassandra because it can run on windows while Riak cannot. I needed it for a data store for some cluster computing work I'm doing, but for various reasons needed my test environment to be windows only. I think I can get rid of that restriction; perhaps benchmarking Riak vs. Cassandra for my particular use case is in order.

DennisP 15 years ago | |

Linux 64-bit binary is available without filling out any forms at http://labs.northscale.com/membase/

mahmud 15 years ago | |

My work blocks IRC, so jumping on freenode isn't an option

Get yourself a shell account and run irssi under screen.

weixiyen 15 years ago |

Sounds a lot like what Redis is doing with Redis Cluster, except this one is already in production.

p3ll0n 15 years ago |

The key/value model embraced by NoSQL databases (Scalaris, Voldemort, Tokyo Cabinet, etc) is the simplest and easiest to implement but inefficient when you are only interested in querying or updating part of a value.

This article (http://seattleweb.intel-research.net/people/lamarca/pubs/pap...) coming out of Intel argues that it is also difficult to implement more sophisticated structures on top of a distributed key/value. The author's main point is that a few specialized applications can and have been built on a plain distributed key/value store, but most applications have ended up having to customize the key/value store's internals to achieve their functional or performance goals.

From the little bit I have read about Membase it looks well positioned to bring simple distributed key/values stores to the next level and back into the lime light.

sqrt17 15 years ago | |

I for one like the concept of storing JSON-like data structures and using javascript-based indices on top of that, like you can do with CouchDB or MongoDB. For the greater part, having something where you can store big amorphous blobs is much less useful than something where you can (a) work on a standard format (usable at least from Java and Python) and (b) use some indices to speed up access via selected attributes.

For a project of mine, I've setup something where the frontend (i.e. javascript) munches on some data structures, which are then passed on to the backend (some Python code for business logic and authentification) which sticks them into the database more or less directly. Works really well, although MongoDB still has this "embrace of the exotic stranger" feeling (i.e., you wonder when exactly your database will stop to like you and just crash), whereas CouchDB is just too slow for my purposes.

mkramlich 15 years ago |

if Zynga is already using it behind say Farmville that will give it some cred out of the gate

ryanwaggoner 15 years ago | |

The About page lists "Major Deployments" and seems to indicate that both Farmville and Cafe World are using it, which is impressive.

zppx 15 years ago |

Had not the time to search through the codebase, but the source include 10 folders, a sysadmin's hell, some with software in C, some with software in C++ and some with python code, one of them is a folder named 'mencached'.

mahmud 15 years ago | |

I worked on a project that had a directory named "manuel" and contained 8 C files and a file named "make.sh". I always thought it was some guy's private playground and shouldn't be touched. But once I dug into it I discovered a NIH implementation of Docbook + Doxygen + Markdown + grep + strings, and make.sh did what Makefile would have done, use make. And 'manuel' never existed, the French comments alerted me to guess what it really meant: Manual.

silentbicycle 15 years ago | | |

And of course, "manual" means, "by hand". :)

agotterer 15 years ago |

This looks similar to http://www.gear6.com/memcached-product/memcached.

agotterer 15 years ago | |

Wonder why I got down voted for an informative link to a similar project that I have no involvement with?

alexpopescu 15 years ago | |

Interesting! I've concluded the same thing based on the very scarce info they're offering.

leej 15 years ago |

* there are alternatives but the backers give this project one up. * web site needs a significant overhaul. * this was a commercial product, if i am not mistaken, and just recently opened up. * is this forked off memcached or just another NoSQL with memcached compatibility is still a mystery.

dlsspy 15 years ago | |

It's not a fork. We've done a lot of work on memcached to allow it to support multiple backends on the same network interface. Part of membase is a new memcached backend.

tobyhede 15 years ago |

Does anyone know what kind of durability Membase offers? One of my ongoing concerns with MongoDB is the lack of single-server durability.

Lorin 15 years ago |

That is one ugly mascot.

ergo98 15 years ago |

It's unfortunate there are no Windows binaries (well apparently there are no binaries yet period, but I mean they don't plan on binaries until some time out). We could host Linux VMs, but ideally we just run Windows binaries and get rid of the abstraction.

That is the case for a large number of these products. At best offering a terrible Cygwin port. In the Windows world the premiere product right now is the beta of AppFabric, but almost no one uses that and it has an absurd list of dependencies that preclude many uses.

siculars 15 years ago | |

Your post is just another nail in the MS coffin. The simple fact is that the latest, greatest hotness is just not playing in the MS ecosystem. Virtually all NoSQL systems support windows as an afterthought if at all. Not to mention hotness like nodejs.

The bottom line is that the people who make the new shinnies just don't play on MS boxes and could care less if their toys ever play on MS boxes. No, I'm not talking 'enterprise'.

wmf 15 years ago | |

Porting software to a completely different culture isn't cheap and there's little incentive for volunteer open source projects to do it. If enough software that you want isn't available on Windows, it's probably cheaper for you to learn Unix.

henrikschroder 15 years ago | | |

There's plenty of people that want to volunteer porting projects to Windows, there are other hurdles such as Microsoft refusing to ship a C99 compiler, and getting the Windows port accepted into the main repo for the project.

ergo98 15 years ago | | |

Where did I say I didn't know Unix? The problem is that for some deployed solutions adding in Linux boxes simply because they host one piece of software isn't optimal, so "native" solutions are chosen instead. We aren't talking about a product that is tightly coupled with the OS (or at least it shouldn't be).

peter123 15 years ago |

The site membase.org seems to be really slow now. Not a good showcase of the technology it's trying to preach.

cmelbye 15 years ago | |

Why would a static, informational web site be using a database?

Xurinos 15 years ago | | |

There is an argument in favor of marketing to the perception, though. I am sure he/she is not the only viewer that might assume they are eating their own dog food and using it behind their website. We have countless tales on HN about increasing customer interest by speeding up your site. If you want to talk about how cool your tech is, make sure your website reflects a similar attention to the important details. I think the comment is legitimate for the perspective even if the technological reality is that the slowness is completely unrelated to the database.

Also, given the popularity of CRMs and so forth, it might not be good to assume that the pages are really "static". For all we know, their site is using their database to provide content -- edited by documentation/marketing folks -- to page templates.