Show HN: SuperSplat – open-source 3D Gaussian Splat Editor(playcanvas.com) |
Show HN: SuperSplat – open-source 3D Gaussian Splat Editor(playcanvas.com) |
Splats are sort of like byte code, they are the compiled and optimized representation of reflected light as semi-transparent guassians.
Or you can think of them as the PDF equivalent of a Google or Word Doc. All the logic is gone, and you just have final optimized results.
Generally when you edit PDFs, the results are not great and you cannot make major edits because the layout won't reflow, etc.
So while this is cool, I don't think it will take off unless there is another innovation in terms of either using AI to "reflow" the lighting and surfaces after an edit, or inferring more directly the underlying representations (true surface properties and the light sources.)
https://playcanv.as/e/p/cLkf99ZV/
Integrating AI is an interesting topic and something that certainly has potential.
- cleaning up noisy GuassianSplats is useful. There are often stragglers floating around in space that need to get deleted.
- compression/optimizing them is useful.
This being a cleanup and compression tool makes sense, but I guess I don't call that an "editor."
I guess I was more arguing against the idea that this is a viable "editor" where one can combine and manipulate in more radical ways Gaussian Splats. The current technological approach doesn't make this a feasible use case.
In that model, we don't compile them, we train them; we don't run them, we sample/rasterize them.
This link came up on HN before and was a great refresher/expander on the math of Guassians which allow all this. [1].
Since Gaussians can be estimated, neural networks can model/generate them. Researchers are using this for 4D work and mesh extraction. The NNs run at lower frame rate informing the 3DGS running at interactive rates.
You are right that it is ephemeral and really a weird trick of the eye and we need new ways to edit/create it. Vectors/pixels have had a lot more time to grow tooling. People are working on it, just the toolbox is different. Very cool stuff will be coming up, I bet!
[1] https://news.ycombinator.com/item?id=41912160 I've also re-learned Fourier transforms to appreciate similar concepts.
- SuperSplat dev :)
- https://www.meta.com/en-gb/experiences/gracia/25784099001234...
EDIT: yep - https://gsgen3d.github.io/
Supersplat is actually a game engine (but for the web)
Individualkex also has a couple videos on the high level ideas: https://youtu.be/GQXDjzNWuPc?si=zlAN7dO9STGATKad
[1] https://github.com/playcanvas/supersplat
I remember a time when it was considered unpolite to ask a question without googling first. Is it still the case?
Any tips for an app to use on iOS to capture the necessary .ply data?
Scaniverse is a great app by Niantic that can do this on-device, but it isn't very customizable and can't export its raw scanning data (exported .plys do not have the data this editor requires).
Let's start from a single guitar string. If you pluck it, it makes a sound. It's because the string vibrates. In sound processing (a.k.a. "signal processing"), it is said we can express any complex vibration of a string (or, a sound wave) as a sum of increasingly compressed ("higher frequency") sin/cos waves (called "higher harmonics"), each of them multiplied by its "contribution" (some frequencies, a.k.a. harmonics, are more present, others less). (This is also called more generally a "Taylor polynomial" IIRC/IIUC, or a "Fourier transform" in the particular case of a wave.) Notably, a .MP3 file format takes this sum, and cuts it off at some point - assuming that if we keep only a bunch of the "most strong" harmonics, and cut away the remaining waves that are less "contributing", the audible difference won't be noticeable. Also, a guitar string has a very tiny amplitude of those vibrations compared to its length, so they are barely visible. If you take a friend and start waving a piece of rope between you, you can get bigger waves, making the amplitudes much more visible.
Now, a guitar string is a 1-dimensional wave. If we go to 2D, we get a membrane of a drum. When you hit a drum to make a sound, it will start vibrating. In the same way, the shape the membrane takes in those vibrations, can be expressed as a sum of "simpler" 2D vibrations - presumably mathy/physicsy people call them "circular harmonics" or something. Again, on a drum the vibrations aren't really visible to naked eye, but if you instead took a floppy rubber circle loosely stretched on a metal ring, and start shaking it, probably you'd get bigger waves. Interestingly, IIUC, a JPEG image is basically "MP3 but in 2D case".
Now, back to Gaussian Splats and Spherical Harmonics - I assume that "spherical harmonics" are the same thing but done to a balloon. If you pump up a ("perfectly spherical") balloon, and then hit it, presumably the vibrations of its surface can also be expressed as a sum of increasingly more wrinkled ("higher harmonics") sphere-like shapes, each one multiplied by its factor/contribution/strength/presence in the actual vibration. Again, on a balloon the deformations from ideal shape are super small; but if you imagine some really floppy balloon-like sphere floating in the air, you could imagine the wrinkles being much deeper.
I assume in case of gaussian splats, apart from storing factors of each of the spherical harmonics contributing to the final "distorted blobby balloon shape", you also probably store a color of this contribution. This way, from some angles the dominating color of the blobby balloon would look more green, from others more yellow, etc.
Interestingly and coincidentally, a similar thing happens in an atom. The various "contributions" to the "blobby balloon" shape are called "electron energy levels" (or "orbitals") IIRC (https://en.wikipedia.org/wiki/Atomic_orbital). And the actual "blobby balloon" shape is probably called an "electron cloud" IIRC. I'm super grateful you pointed me in the direction of trying to understand Spherical Harmonics, because when I saw those shapes of atomic orbitals in the past, they always seemed weird to me, and confusing. Now it seems I understand where they came from, that's super exciting!
Found a decent video series about this on YT, showing vibrations of a plucked string, etc.: https://youtube.com/playlist?list=PLpBx-1imHuxISNflNHo0Qr4mQ...
Eheh, found one more video - building up the shape of the surface of the Earth from a sum/superimposition of increasing number of Spherical Harmonics - https://youtu.be/dDQTHFeJf5M - again roughly what an MP3 or JPEG algorithm does, depending on how much "fidelity" you choose, i.e. how many more precise harmonics you keep :)
Ah, and also - IIUC, in some other domains of math, those "harmonics" can also be said to be "eigenvalues" (https://en.wikipedia.org/wiki/Eigenvalue), and in somewhat more familiar territory, they could be called "orthogonal" meaning that a sum of them can allow to represent any shape in some space - in a similar way as orthogonal vectors of a cartesian coordinate system (i.e. the "1"s on XY axes in 2D, or on XYZ in 3D - or your green/blue/red arrows in Blender) allow to represent any point in that coordinate system.
- Copy & Paste: e.g. delete a tree and fill the hole with a copied patch of grass
- Color Adjustments: tinting, brightness, etc.
If these aren't editing ops, I don't know what is. :) Sure, you _could_ go back and recapture photogrammetry or rerun training, but that's super costly in terms of time. SuperSplat lets you make simple edits quickly and easily.
Yet this comment tree thinks it’s a novel observation that makes the tool useless.
First, Kagi might give the same results today, but what about tomorrow or a year from now? Will Kagi still exist a year from now or will Kagi links all be broken?
A better idea is to follow this HN guideline here and everywhere:
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
The kind, non-snarky response to something that could be searched for is to simply answer the question.
When you share from Kagi you share the actual results, not the search that led to them. I believe they don't change over time. If they ever disappear, the next search engine will be just as good for that same question.
> The kind, non-snarky response to something that could be searched for is to simply answer the question.
A few times, sure. There are things that cross the line in my mind though. You do occasionally run into questions which took longer to write than it takes to check. I think it's ok to discourage them slightly while still providing the obvious answer. Lgmtfy worked great for those because you know what is and always will be the first answer for them.