Z-order curve usage to decrease dimensionality to 1(ssahinkoc.blogspot.com) |
Z-order curve usage to decrease dimensionality to 1(ssahinkoc.blogspot.com) |
http://blog.christianperone.com/2015/08/googles-s2-geometry-...
And 2015 HN thread: https://news.ycombinator.com/item?id=10066616
In a small number of dimensions, without knowing what search algorithm is being used, this is just more work than comparing the original values. It doesn't mention what "k-NN algorithm" is being used, beyond brute force search.
Lossily compressing N-dimensional data (from 2 to 1000s of dimensions) into a representation that requires fewer bits can be done via quantization as well, either scalar quantization, vector quantization (aka k-means) or product quantization, if your data has known statistics.
It also matters if you are building a static data structure that is queried many times, versus one that needs continual updating.
And since modern spatial database architectures don't sequentialize storage along the curve (because it doesn't make sense as a matter of engineering), the sole selling point of Hilbert curves is moot. You shouldn't design most systems in a way that could exploit the benefits of a Hilbert curve.
I've read your blog entries on SpaceCurve (http://www.jandrewrogers.com/2015/10/08/spacecurve/), found them very interesting but also just whetting the appetite. Are there no public reviews or papers covering discrete topology/sharding?
https://github.com/leni536/fast_hilbert_curve
I only implemented the index->XY calculation yet. It compiles to 36 instructions without any branches and takes up 86 bytes.
https://github.com/leni536/fast_hilbert_curve/wiki/How-effic...
I think I can apply the same tricks for the inverse function too.
http://www.akt.tu-berlin.de/fileadmin/fg34/publications-akt/...
https://www.reddit.com/r/ProgrammerHumor/comments/4xzi9a/oh_...
QUILTS: Multidimensional Data Partitioning Framework Based on Query-Aware and Skew-Tolerant Space-Filling Curves Shoji Nishimura (NEC Corporation); Haruo Yokota (Tokyo Institute of Technology)
It discusses C-Curve, Z-Curve, and Hilbert curves.
From what I can tell, it's the exact same algorithm used by Geohash.
2. It won't be better than a k-d tree. Dimensionality reduction is usually done when you have really truly huge numbers of dimensions that are sparsely populated and you don't care much about some information loss (e.g., for machine learning) or, in this case, when you have an easy way to create a single dimensional index and you want to force multi-dimensional data into it. In the general case a k-d tree would be objectively better in terms of performance.
As a couple of people have mentioned in this thread, space filling curves aren't great at preserving locality (i.e., two points that are "close together" in two dimensional space might end up being "far apart" in one dimensional space, and vice versa). A k-d tree is easy to code up and, in general, will be more efficient for queries like k-NN than dimensionality reduction because it's better at preserving locality.
There are also good libraries for multi-dimensional data structures for pretty much any mainstream language.
Most people know you can use Z/C-curve encodings to dynamically content address point data. There is a (very useful) generalization to hyper-rectangle types, perfect for content-addressing non-point geometries etc, but those types can't be meaningfully sequentialized at all in big systems. Most non-trivial spatial analytics involve non-point geometries, so being able to sequentialize points has limited utility.
Second, the computational cost of sorting along the curve, assuming you are using only points, is prohibitively high for negligible benefit. Modern storage engines use small shards, which are adaptively re-sharded as needed, and medium-sized pages. For insert, the content-addressing mechanic gets you to a single page; it would be significantly more expensive if you were sorting along the curve. For query, the typical selectivity on a shard is so high due to small adaptive shards, that you are better off treating it as an unsorted vector anyway. In short, much slower inserts and few (if any) query benefits.
As an optimization, it tends to only be applicable in cases where the architecture is significantly suboptimal anyway e.g. the use of gigantic shards. You'd get more benefit by fixing the architecture than trying to optimize poor architecture if at all possible.
And I'm sure that constructing H-curves doesn't need to be as complicated as that linked academic code makes it look, but for now I don't know of any implementation.
Thanks for the link though :)
zorder64_inv:
movabsq $0x5555555555555555, %rax
pextq %rax, %rcx, %rdx
shrq %rcx
pextq %rax, %rcx, %rcx
shlq $32, %rcx
movl %edx, %eax
orq %rcx, %rax
retq
zorder64:
movl %ecx, %eax
movabsq $0x5555555555555555, %r8
pdepq %r8, %rax, %rcx
movl %edx, %eax
pdepq %r8, %rax, %rax
addq %rax, %rax
orq %rcx, %rax
retqI haven’t ever seen any convincing benchmarks or other analysis where the Hilbert curve created any notable performance advantage vs. Z order; the only time you really need it is if moving along the linearized coordinate must never have jumps in the multidimensional coordinates, but I’m not convinced there are many if any real-world cases where that is important (note that in either case small movements in the multidimensional coordinates are associated with large jumps in the linearized coordinate). If the only goal is to minimize memory fetches, etc. then the Z ordering works just fine.
(If you know any good comparisons where the Hilbert curve comes out ahead, I’d be curious to read them.)
https://www.compuphase.com/riemer.htm
I suspect that error diffusion along the long jumps of a z-order curve could create strange undesired artifacts.