Have a look at the WCAG contrast ratio and luminance functions to gauge similarities/differences between colors. If you are looking to extract a theme those formulas are gold!
What sort of hashing/tolerance are you using for grouping? Using the above you can specify a tolerance level, and use sqrt((r1-r2)^2 + (g1-g2)^2 + (b1-b2)^2) to get a diff and then compare that with the tolerance.
We blogged a bit about how it works too: http://99designs.com/tech-blog/blog/2012/05/11/color-analysi...
But it seems to need improvement to support Pillow, latest version of Python2, etc. I still didn't try but I'll examine it. Thanks.