Redis Loadable Modules System

Redis Loadable Modules System(antirez.com)

297 points by djanowski 10 years ago | 60 comments

educar 10 years ago |

The docs explain the module system (https://github.com/antirez/redis/blob/unstable/src/modules/I...) but it doesn't quiet explain what the system is for. I couldn't quite get that from reading the blog post or the doc page. Specifically:

What kind of modules did you have in mind when designing this? Are any of the below possible?

* ldap backend login system

* multi-tenant redis

* alternate db format/file system.

* implement a new redis data type. say built-in date support

* can i implement new commands and operators on said data types

An of course, thanks for this fantastic software. We use it everyday. Will read in more detail about this over the weekend :-)

dvirsky 10 years ago | |

Actually the post explains the motivation well. The basic idea is to add capabilities to redis at native speeds, without having to either convince antirez that they're necessary or forking redis.

I work for Redis Labs, and over the course of the past few weeks we've experimented with some modules using the API. Some of the things we did that might answer some of your questions:

a PAM based authentication module, which IIRC can plug into LDAP potentially.

A bloom filter implementation on top of bitmaps.

An inverted index that doesn't use existing data types but rather creates compressed indexes directly using redis strings (I wrote that one personally BTW)

An API to store images on redis and retrieve them resized or cropped.

So right now the only way to create new data types is to piggyback strings as raw memory, but the final version of the API will include actually registering new data types.

You can see the modules we've released here: https://github.com/RedisLabsModules/

vkjv 10 years ago | | |

Maybe you can relay the information to the appropriate person, but on http://redismodules.com, I can neither right click "open in new tab" or ctrl+click to open in a new tab.

This breaks my normal browsing habit of opening new tabs as I see something that interests me, but waiting until I finish skimming the list to read.

creshal 10 years ago | | |

> a PAM based authentication module, which IIRC can plug into LDAP potentially.

Yes, pam-ldap / pam-sssd / pam-winbind plug into various LDAP implementations (plain LDAP / ActiveDirectory) and work transparently with all PAM clients.

ecesena 10 years ago | |

I'm not related to redis devel, but I happened to be at the conference where antirez announced the modules.

My understanding from the talk was:

* already available: strings DMA, that you can use for new data types

* in progress but soon (like, a few weeks): commands pre-hooks, that you can use e.g. for auth

* I assume soon (but I don't recall, it could be already available): operations on new data types

* I don't think I've heard anything about alternate db format, but maybe you can think of creative ways of mixing pre-hooks and strings DMA

* I'm not sure what you have in mind about multi-tenant, but I guess you can implement some primitives in a way similar to auth

educar 10 years ago | | |

What I meant by multi-tenant is a multi-user redis (just like multi-user mysql/postgresql. (ie) i can create users and have them use redis completely separately on a single instances on redis.

astrodust 10 years ago |

There's been a lot of unusual features I've been hoping Redis will support but obviously given their edge-case nature it's unlikely that they'd ever be implemented.

This is great to see. Instead of cluttering up the core engine with junk people can mix in their own functionality.

aaw 10 years ago |

This looks interesting. I've only glanced at the docs, but it looks like this module system lets you define new commands but not necessarily new data structures (since you'd have to implement RDB/AOF serialization for those).

Are there plans for that? It's probably possible to do most of that with this module system but you'd have to serialize the data structure into a string and "type X" wouldn't return your custom data structure type.

antirez 10 years ago | |

Definitely plans to register new data types with serialize/deserialize callbacks! Thanks.

maerF0x0 10 years ago | |

He mentioned in the keynote being able to do your own datastructures by raw access to the data and primitives like seeks and next/prev . Sounds like it's at least planned, if not roughed in.

BadgerEmeritus 10 years ago |

This is great and long awaited, please consider some kind of a namespacing scheme for module-defined commands though, or things can easily become messy. In the module example the "hello." prefix is specified for all defined commands - it would probably be better to specify it once on module initialization and have all commands defined by the module prefixed with it automatically, the module registries could then manage the prefix registration/allocation to module authors.

avar 10 years ago |

Since Redis is BSD licensed I wouldn't be surprised if this is the start of popular paid-for proprietary third party modules.

byroot 10 years ago | |

Even if redis was GPL licensed you still could do that. Linux is GPL and there is proprietary driver modules.

ploxiln 10 years ago | | |

There are proprietary modules for linux, and they are distributed and used, but there isn't really a full consensus on whether they are legal. You can certainly do it and get away with it though.

http://yarchive.net/comp/linux/gpl_modules.html

spullara 10 years ago |

I once built something like this for Redis but never really thought he would do it!

https://github.com/spullara/redis/tree/plugins

glenda 10 years ago | |

I've written something similar too. Adding plugins/modules was like a rite of passage for anyone digging into the redis source. I'm definitely happy to see modules getting official support now though.

spullara 10 years ago | | |

I was blown away with how easy it was and with the code quality.

derefr 10 years ago |

If these modules are going to be as fast as the native commands, is there any reason to not lift the entire native command set out into a set of "base" modules?

dvirsky 10 years ago | |

They're not going to be as fast, but almost as fast. You could do that in theory but IMHO there's not much point

shawn-butler 10 years ago |

How people can code on airplanes amazes me. I have tried but the cramped space, the constant interruptions.. I never get much done.

akbar501 10 years ago |

@antirez, is Redis 4.0 going to be part of the more rapid release schedule (will we see it in 3, 6 months) or are major upgrades still annual releases (we'll see 4.0 in 12 months)?

antirez 10 years ago | |

Definitely part of the short release cycle, we'll probably focus on getting modules done and the features already pending into "unstable", and a few more minor stuff, and release it.

manigandham 10 years ago |

https://github.com/RedisLabsModules/redex

Shows some sample modules from Redis Labs called RedisEx, pretty useful stuff (although most of these should really be in Redis by now but at least it's an option to include as modules).

dvirsky 10 years ago | |

> most of these should really be in Redis by now

This exactly the point of the module system. You don't need to either convince antirez to accept/implement a new feature, or fork redis if he doesn't agree. It will make the dynamics of redis much more interesting IMO

maerF0x0 10 years ago |

they also announced a couple of "hubs" for hosting/discovering these plugins, plus a vetted one by redislabs

itamarhaber 10 years ago | |

Indeed - http://modules.redis.io will be the open source place and the Redis Labs' Module Hub is at https://redismodules.com

evantahler 10 years ago |

So who is adding node.js support?!

MasterScrat 10 years ago |

Someone should really implement an interval module.

This custom implementation was pretty good: http://blog.togo.io/how-to/adding-interval-sets-to-redis/

halayli 10 years ago |

I bet a package manager will follow it. redis install moduleX and then we'll have lib and target dependencies.

antirez 10 years ago | |

But everything will depend on the padding module.

halayli 10 years ago | | |

and then the author will remove it one day. :)

lsiebert 10 years ago |

Super excited to hear about this at Redis Conf, looking forward to implementing a few ideas.

therein 10 years ago |

Sounds exciting. Hopefully it won't be as under-appreciated as Apache Traffic Server's plugin system.

awinter-py 10 years ago |

databases (redis included) are awesome. It's always surprising to me that we don't run web apps directly on the database; they're high-performance, durable and parallel, and for some apps are already the perf bottleneck.

I think the only reason we don't is that deployment and testing isn't straightforward. Fingers crossed antirez will get this right.

TheAceOfHearts 10 years ago | |

You might find CouchDB pretty interesting, then! [0]

[0] https://couchdb.apache.org/

awinter-py 10 years ago | | |

I know them for data sync. is there a stored proc deployment feature as well?

angelbob 10 years ago |

I'm hoping there's an easy way to turn this off for the foreseeable future, to avoid all the obvious security holes this is likely to introduce.

dvirsky 10 years ago | |

You don't have to use it, and you should not use modules you don't trust.

angelbob 10 years ago | | |

In an ops-type capacity, I support developers. I keep up with news like this specifically to avoid winding up with modules somebody else trusts, but didn't inspect.

MBCook 10 years ago |

I tried to read the article but I couldn't. It's all one big <pre> block. The word wrapping is wrong on input! Individual paragraphs are groups of individual lines and extra long lines that the browser wraps for you. So you end up with lines in the middle of paragraphs that only have one or two words.

Net result is it's very hard to read.

I tried Safari's reader mode, but that was even worse. Because all formatting comes from happenstance in the <pre> tag reader mode loses all formatting. The entire article (and code!) becomes one long block of text.

I'm continually amazed at people's attempts to reinvent text layout in the browser.

incepted 10 years ago | |

> I'm continually amazed at people's attempts to reinvent text layout in the browser.

Agreed.

I don't recommend you try to read pg's essays, they will make you very angry.

manigandham 10 years ago | |

Why is this downvoted? It's true that this blog has bad formatting considering how simple it is, there's no reason to use <pre> for the entire text.

Just because the author created Redis doesn't mean the site isn't badly designed.

MBCook 10 years ago | | |

I make these kind of comments every once in a while. They're usually downvoted, I assume, for being off-topic or unnecessarily harsh ('We might have cured cancer!' vs 'I can't read the article because of the video behind the text.').

But I hope some people see them and work on the issue.

Sometimes, if enough other people agree, these kind of comments get voted up. I guess this issue didn't annoy most people enough to get the up votes.