Using C for a specialized data store(pixenomics.tumblr.com) |
Using C for a specialized data store(pixenomics.tumblr.com) |
It's quite fast, you can use a Redis string as a random-access array up to 512MB/each, and there are several good ways to handle persistence/backup. I don't think there was a need for them to write any C themselves.
> We were reluctant to use a NoSQL solution as this would require retrieving the pixels through a socket, storing it in memory and then processing them. It makes more sense to process it where it’s stored.
The pulling shouldn't be an issue -- I don't know about PHP, but in pure Python, I can pull an arbitrary 10MB string from Redis in ~85-90ms. With hiredis (C extension), that falls to about 47ms.
I can't speak to processing, since I don't know exactly what transformations you're performing.
We will probably head towards redis in the future when precise backups are essential. Undecided what will do this processing though.
Use APU / HPU, if PCIe latency is a problem.
I understood that they running something like convolution (I.e, each pixel calculated from surrounding pixels) - this will be fast using OpenCL model).