Show HN: Gaussian Splat of a Strawberry(superspl.at) |
Show HN: Gaussian Splat of a Strawberry(superspl.at) |
Yes. I knoweth what "splats" are: They are splats of fuzzy blobs on the display surface.
What's the matter?
https://www.youtube.com/watch?v=k0MGoFaIiWs - WARNING - heavily adult lyrics and a loud starting "MOTHERFU*AAAAAAS" right off the top.
If my old phone can render it with fairly high frame rate, I’m sure a full game with realistic graphics would work well on a mid/high end PC.
1. Lighting is baked in so they don’t fit well in dynamically lit environments
2. Splats aren’t as suitable for bodies you’d animate, like meshes are. They also don’t work well for very thin or sharp surfaces in general
3. Splats are pretty memory-hungry so large scenes would require better hardware than consoles can offer
These might not be as true today, and I can’t recall exactly when I read this. Probably 3 years ago or so. I recently read they’re making their way into games, so it’ll happen as the kinks are worked out.
I was really pulling for OTOY to keep making light field capture and display technology...
Anyway, very cool splat, fair play
You’ll also see the same type of problems in any incomplete part of any scene. Go poke around any of them and you’ll find places where there wasn’t enough information to make a satisfying reconstruction. For example, the back of the moon <https://superspl.at/scene/2ac8f423>, the fridge in the coffee shop <https://superspl.at/scene/6a0c3ccf>, underneath the footbridge in the forest <https://superspl.at/scene/23ebe85c> (or indeed anything in the distance), etc, etc.
(Can we do a Gaussian Splat of the setup of the photograph for the Gaussian Splat of the Strawberry?)
> Oops!
> navigator.sendBeacon is not a function
And it shouldn't be one.
Last lines:
HTTPS-First Mode: Upgrading insecure speculative TCP connection “http://superspl.at/” to use “https”.
HTTPS-First Mode: Upgrading insecure request “http://superspl.at/” to use “https”.
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://sentry.sc-prod.net/api/302/envelope/?sentry_version=7&sentry_key=d107d06d48c2e0f20554f721048e3faa&sentry_client=sentry.javascript.react-router%2F10.50.0. (Reason: CORS request did not succeed). Status code: (null). 2
can't access property "messageManager", target is null ExtensionParent.sys.mjs:75:42
verifyActorForContext resource://gre/modules/ExtensionParent.sys.mjs:75
recvAPICall resource://gre/modules/ExtensionParent.sys.mjs:1186
_recv resource://gre/modules/ConduitsChild.sys.mjs:90
receiveMessage resource://gre/modules/ConduitsParent.sys.mjs:474
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://sentry.sc-prod.net/api/302/envelope/?sentry_version=7&sentry_key=d107d06d48c2e0f20554f721048e3faa&sentry_client=sentry.javascript.react-router%2F10.50.0. (Reason: CORS request did not succeed). Status code: (null). 2
Missing resource in locale fr: devtools/client/toolbox.ftl
can't access property "messageManager", target is null 3 ExtensionParent.sys.mjs:75:42
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://sentry.sc-prod.net/api/302/envelope/?sentry_version=7&sentry_key=d107d06d48c2e0f20554f721048e3faa&sentry_client=sentry.javascript.react-router%2F10.50.0. (Reason: CORS request did not succeed). Status code: (null).
TypeError: Response constructor: Response body is given with a null body status. Utils.sys.mjs:279:17Edit. TIL Poe's law
But I'm always experimenting with the mounting, thanks for the inputs.
That is indeed a very cool scene being about to wander around and still have decent resolution
Seriously though - it's breathtaking.
The first guy who figures out the bridge between splats and dynamism - animation, editing, responsiveness - is going to be one of the immortals of 3d design.
The scene desperately needs some clipping on the boundaries. If you use an app like Scaniverse, you can add a bounding box to cull far away points which are often poorly reconstructed.
If you have a newer iPhone with a LIDAR scanner, highly recommended. You can make dolls-house renderings of your house or garden which is surprisingly useful for planning and measuring walls/features.
There is no mesh or model. The visual surface of the strawberry could be made up of blobs spaced far apart physically and not where the surface appears to be.
This is why they are called radiance fields, they model the light not the geometry.
Practically the blobs positions/rotations can be constrained to better physically match the geometry of a strawberry.
Otherwise the splat would fall apart as soon as the viewing angle is changed slightly (Which it absolutely does in many examples on supersplat, you cannot really create an out of distribution view with 3GS, it's not magic)
I took maybe 10 pictures of a model I built and threw it at my 3060 during dinner and it came out quite nice.
What I love about gaussian splats is the way they degrade - instead of a hard cutoff or LoD changing spheres into cubes etc., they get increasingly "dreamy" - the basic idea is still there, just less detailed.
Take for example this scene:
https://superspl.at/scene/e721ea7c
If you navigate closer to the trees, things around you become blurry - as if the very fabric of reality unraveled.
https://bayardrandel.com/gaussographs
More recent work on my instagram
Apples model to generate Gaussian splats from a single image. Takes about 30 seconds on an M1 Pro.
It falls apart once you move too much, but for a little side-wiggling or a second-eye view for VR, it's great. And looks a lot better than the old approach of depth map + vertex shaders that I use in https://github.com/combatwombat/tiefling. But ml-sharp has 2.6 GB weights, a bit too big to run in the browser.
It seems to work a lot quicker than 30s now on iDevices and Macs.
AI-based relighting will no doubt start working soon.
it could look like the real-time lighting of an old game engine on rather modern assets. (quake 2-3 era)
or perhaps some "occlusion pre-pass" could be done to create a voxelized sparse volume from the splats that set a "voxel opacity value" for each to absorb light? (not far from how prebaked GI works nowadays)
note: not an expert on rendering, just a nutjob that did stuff in opengl in the old days.
As I scrolled through the website, I was even more impressed with this one though!
If I'm reading Chrome right, that's 171 MB, for the website, and the data.
If I'm doing the math right, that's 40 seconds worth of the bandwidth from Netflix, at its highest rate.
But something looks off: the red area around the "seeds" is pushed towards the center of the strawberry - or that at least the outer most layer is somewhat transparent and some deeper layers are visible.
I have a question about perhaps the most boring aspect of this strawberry: the license. you write, "You can download it under CC BY license, but attribution is appreciated rather than required." IANAL, but I'm pretty sure you can't license your work CC-BY and then waive the BY requirement in the description. Rather, you'd have to license it with something more permissive like CC0 and request attribution if you want attribution to be optional. Is that right?
Sort of. There are countries like France where attribution rights are fundamentally inalienable, and the author can always demand attribution, even after forever waiving the right to attribution. But in the US, the party who benefits from a contractual provision can generally choose to ignore or forfeit that provision. Whether it's still called the "CC BY" after that is debatable, since the whole point is attribution, but licensing by the CC BY and immediately waiving that requirement seems legitimate to me.
The filesize of a 3d animated splat is seemingly very small, and the method enables ~arbitrary FPS. But it seems the setup required to record it is still huge and expensive, which limits its usefulness.
Even with that there are some interesting use cases, eg. I'd love to be able to watch concerts this way, and freely move around the stage and crowd from any angle.
(I'm ... still not sure what I'm looking at on TFA, and whether or not my browser configuration fails to fully present the site as intended.... OK, if all you're seeing is a blurred image of a strawberry, yes, you'll want to enable a bunch of JS resources. I'm using uMatrix, several hosts must be enabled.)
I'm wondering if the splat community has decided this paper is valuable -- https://github.com/fraunhoferhhi/Self-Organizing-Gaussians -- looking at all the detail in the strawberry splat made me wonder how small one can get the download, and what the current state of the art is for compression.
Focus stacking generally is not perfect process and can lead to artifacts/errors and I'd imagine those can then compound when stacked images are used for 3dgs. Also the image focus actually provides some depth data in itself that could be useful?
My take.. at a macro scale, the dof is usually so small, that it's hard to get a reliably track. So you'd need some sort of way to tell that these stacked photos belong into a series, and then you sort of are doing focus stacking :-) I do think the alignment algorithm could be improved. Maybe the approaches I linked could be used to make a much more robust focus stacking algorithm, that also corrects for 3D geometry. That would be really cool!
You would have to mask out the blurry areas for each image. I guess one could just implement a feature where the optimizer only optimizes gaussians within the sharp distances relative to the camera.
Another way would be some kind of 4d GS where one dimension is the focus distance. But I'd guess the renders would inherently have shallow depth then, which is less useful usually.
For example this bumblebee: https://superspl.at/scene/cf6ac78e
Edit: I completely missed that this was posted by him (:
An input of a kind of schematic representation of what the designer wants would be better. It may resemble a storyboard or a collection of organised notes that large projects tend to already use.
Fully generative could probably do some cool things, but people will still want to bring their peronal vision to life.
vs
48meg
Splats make pretty pictures but they are 10x the data for pretty much anything I've seen.
3gig for this one: https://superspl.at/scene/8429e5e2
I guess the number of samples required to generate a GS is the constraint now, but maybe that will get solved.
Once they get quicker to train, I expect this to be a popular use of them
It kinda splatted my mind.
As I understand it, Gussian Splats aren’t a method of constructing missing data in the sense you’re asking here.
But there are other, well established, methods of generating missing data in convincing manner, that are beyond my field of expertise or interest to be able to repeat here.
With the added benefit that if I’m wrong, being wrong can be a great way to spur discussion 8)
This feels so much closer to how minds must store and process spatial information than the usual 3D models do.
It has a mean, and that mean doesn’t have to lie on the surface, consider the case where the mean is deep inside the strawberry but its spike contributes to the surface appearance (e.g a seed could be represented this way, or it could be represented by a small well-oriented blob on the surface, the optimiser doesn’t care)
That seems like the way to do it, someone has to be working on that.
searching...
Since you can view Gaussian splats from any POV you end up generating an output that is closer to the representation of the world instead of a projection that a single observer sees.
A diffusion style process generating gausians instead of pixels. You could possibly do nerfs that way, but it would be effectively generating a trained network. If you managed to do that it would have broad application throughout the field of AI.
That's assuming you could even reasonably train a neural net to output viable weights, of course.
Remembering how we used to cheat by projecting hi-res texture maps onto simple geometry to save modeling background objects with the trade-off of constraints on size, viewing angle, shadows, etc. With the relative ease of capturing these splats at high-fidelity, can they be used in a scene as projection-map-like 'fakes' except with far fewer constraints?
We've been leaning away from pure polygons for decades, anyway. Vertex skinning, SDFs, volumetrics, simulation, and a lot more.
The meshes in a From Software game are for exmple hilariously simple, most of the animation is force simulation to make the famous "frizzles" that they like.
If you can sample points inside a volume, in theory you could do that with splat geometry. If someone figures out a way to pass in animation time to a sampler, sample along geometry/wireframe or something else, and keep it from overly twinkling it might change everything.
I’m hand waving all the complexity into “if done one figures out”, of course.
I just don’t see why this method can’t evolve in the way diffusion models have evolved (knowing very little of the geberative mechanics of splats).
Not sure that's what you mean, but there was recently a paper where they put meshless (e.g. voxel or SDF) geometry in an animated tetrahedral mesh "cage" and then animate the meshless model by animating the mesh cage:
https://diglib.eg.org/server/api/core/bitstreams/bd94e19b-98...
https://youtube.com/watch?v=6lKAvxV2mno
https://youtube.com/watch?v=3c3-ue-fd88
Though this currently isn't compatible with 3DGS if I understand the limitations section correctly.
> Finally, our method operates unordered, limiting its suitability for complex volumetric effects. However, a potential solution lies in sorting the generated intervals for proper blending. This enhancement could improve our approach’s compatibility with various meshless representations, such as radiance fields and volumetric lighting.
The "vertex" in "vertex skinning" is really just "a thing with a position that gets moved."
p' = Σ wᵢ · Mᵢ · p
It's just a position. Triangles can come along for the ride downstream, but they're not essential, which is one of the reasons it's so efficient for some stuff. Polygons are the optimal surface - but surfaces are often extraneous.
Take all this a few hefty grains of salt, I'm an amateur in the field. My 3d/CAD work is strictly in support of my enterprise stuff. And making wicked battlemaps for gaming VTTs, natch.
But I will stand by the overarching statement that polygons are in fact an abstraction, and bridging that abstraction with whatever is in splats would be wicked awesome.