Gsplat: CUDA accelerated rasterization of gaussians with Python bindings(docs.gsplat.studio) |
Gsplat: CUDA accelerated rasterization of gaussians with Python bindings(docs.gsplat.studio) |
>Gaussian splats are 3d, but currently rendered as 2d cards ordered on centre, not per pixel like they should be.
>So as centres reorder, they "shuffle". This is a showstopper for general adoption in VR
>The solution? Ordering per pixel will solve this, as would stochastic methods which write depth correctly (though those have other problems for VR).
>For it to be viable you need to cull much more aggressively but also can't have any warp divergence when processing samples.
>Unforunately those things are in direct conflict. Paper's renderer is at extreme end of "weak cull, perfect warp convergence".
Apparently need some new Order-Independent Transparency algorithms that handle depth layers.
[1] https://x.com/charshenton/status/1710207169407447396?s=20
The next step I wanted to try was to super-sample (eg render at double resolution) and linearly down sample. I imagine this would give a slightly noisy result that I think would be pretty visually pleasing.
* https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pr...
* https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-s...
* https://aras-p.info/blog/2023/09/27/Making-Gaussian-Splats-m...
Gsplat gets a mention:
Aside: the excellent gsplat.tech by Jakub Červený (@jakub_c5y) seems to also be using some sort of VQ/Clustering for the data. Seriously, check it out, it’s probably be nicest gaussian splatting thing right now w.r.t. usability – very intuitive camera controls, nicely presented file sizes, and works on WebGL2. Craftsmanship!https://www.youtube.com/watch?v=kShNYOuDnlI&t=224s
I got it recommended because I watched the following video:
https://www.youtube.com/watch?v=HVv_IQKlafQ&t=64s
Both are worth watching if one didn't know it exists.
You can manipulate the splats in viewer, kind of similar to a pointcloud, something to the effect of https://twitter.com/parazar/status/1713618668788109357 but there are a bunch around social media.
This has an entrypoint to the research and various projects so far: https://github.com/MrNeRF/awesome-3D-gaussian-splatting
The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion".
Voxels are bad as they take too much memory and are uniform (each voxel is eg 5x5x5 cm).
Gausians are kinda like "variable sized voxels" that also have orientation, color based on angle viewing and can stretch.
Imagine 3D blobs basicaly (3D capsules or more like 3D density bubbles with transparency).
So the scene can be represented in 3D much more efficiently using gausians splats, which is why they are used for "3d stable diffusion".
So now that this has come into use we also need efficient way or rendering them.
> It's for deep learning, not that much "for graphics".
No, while it is true that there is some overlap between the techniques and concepts used, gaussian splatting isn't necessarily "for deep learning". The library provides a differentiable rasterizer for gaussian splats. This basically means that you can ask it "if I want my output image to change in this and this direction, in what direction should I change the position / orientation / color / ... of my splats?". This enables users to plug it into other software (that is also commonly used for deep learning) and to optimize the parameters of the splats to represent a particular scene.
Since it's primarily a differentiable rasterizer for splats I think it's fair to say that it is "for graphics".
> The problem is "how do you do 3D deep learning 3D scene reconstruction" aka "how to make 3d equivalent of stable diffusion".
That it uses gradient descent doesn't mean that it is "deep learning". There are no neural networks or layers here.
It's not an "equivalent of stable diffusion". The way it's used now is to learn a representation of a single scene, not unlike photogrammetry. Sure, there may be other use cases for this library, but this is primarily what gaussian splatting is about.
And like another person explained, there’s no deep learning. There’s not even a neural net. In the other NeRF literature there are neural nets but they usually aren’t deep. RegNeRF uses a deep neural network along side a shallow net for regularization.
It seems like it took off after this SIGGRAPH paper: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Anyway, this doesn't seem to allow any sort of dynamic lighting whatsoever, but for scans of existing places, such as house tours, street views and so on, this seems super promising.
So this method can only really be used for static scenes made from a point cloud?
I remember 10 years ago, a company called Euclideon revealed a demo[1] of a rendering technique (they called it "Unlimited Detail Real-Time Rendering") that put nearly every other rasterization technique of the time to shame. However, the major issue with the demo was that it only showed a static scene with zero dynamic lighting. I don't think they ever revealed the "secret sauce" of how it works, but it seems to have similar limitations to this new process, or at least that's what I expect considering it can't handle something as simple as a dynamic mesh.
It was extremely hyped up in certain areas, but over a decade later this technology isn't really used in any mainstream rendering engines because it only works for static scenes.
I hate to disparage new rendering techniques, but this really feels like a repeat of what happened back then. What exactly is different in 3D Guassian Splatting? Can it be used in any dynamic scenes in real-time at all? Does it provide any significant advantages from this older system?
The new thing is fast regression of a real light field into Gaussian splats, which can then be rendered at reasonable rates. Dynamic lighting requires full inverse rendering as a preprocess, which is way beyond the scope of the technique. Technically Gaussian splats could form part of an inverse rendering pipeline, and also be the final target representation, but I'm not sure there would be any benefit over alternative rendering primitives.
What I mean is could you make a version of stable diffusion that uses splats instead of pixels?
Think of them as point clouds + lighting. Relighting is viable. Procedural deformation is viable. Animation is just a case of transforming sub-groupings.
Collisions might be a bit trickier but you can always derive a low-res collision mesh from the input data and use that as a proxy.
It's early days at the moment and people are still exploring the possibilities.
You'd have to take multiple pictures of the scene, then move some light source around, take another set of pictures etc. And in a similar sense to the irradiance volumes[1], instead of encoding just the gaussian parameters, encode them using something that lets you reconstruct the gaussian parameters based on the position of the primary light source for example. I know estimating light position and such from images has been worked on for image-based BRDF extraction for a long time[2].
Of course it'll require a lot more images and compute, but that's the nature of the dynamic beast.
Again not really thought this through and it's not really my field, though I was into physically-based rendering a decade ago. Just seems like this is something that seems like it would be solved by natural progression in a not too distant future.
[1]: https://chrisoat.com/papers/Oat_GDC2005_IrradianceVolumesFor...
[2]: https://graphics.stanford.edu/~lensch/Papers/TOG03BRDF.pdf (random example)
I wouldn't be surprised if gaussian splats make it into AAA games, though. Not as the main rendering primitive, but for specific things like vegetation where they really kick ass.