Spherical harmonics support

[AntonioMacaronio]

Overview of Changes
Inside gaussian_splats.py’s load_ply_file() function:

• processing the spherical harmonic data inside sh_coeffs as a numpy array each row is a gaussian with 48 coefficients (in this case, it is 447703, 48);
  0 mod 3 => red
  1 mod 3 => green
  2 mod 3 => blue???

Inside _scene_api.py’s _add_gaussian_splats() :

• we create a new buffer by making the numpy array a float16, but then view it as int32 so it is now shape (447703, 24) where each cell now contains 2 spherical coefficients
  then send this buffer through the websocket
  Updated _messages.py’s GaussianSplatsMessage to include the sh_buffer
  Updated MessageHandler.tsx ’s “GaussianSplatsMessage” case to pass the sh_buffer into the SplatObject as a prop

Inside GaussianSplats.tsx:

• Updated SplatObject prop to now upon mounting to the DOM, it will set a new buffer inside the GaussianSplatsContext state (mainly the variable groupBufferFromId) called the sh_buffer_name, likewise the cleanup will also remove this new sh_buffer_name
• Since SplatObject is a forwardRef, it means that the parent node can access it (which is SplatRenderContext) and its state
• Since SplatRenderer is a child of SplatRenderContext, it can access the context that surounds it, which is GaussianSplatsContext, which contains the groupBufferFromId variable
• SplatRenderer calls mergeGaussianGroups and useGaussianMeshProps , which have been updated to support this
  mergeGaussianGroups now also merges the sh_buffer into one big buffer of spherical harmonic coefficients called combinedSHBuffer. It keeps the original gaussianBuffer by filtering out all spherical harmonic buffers from groupBufferFromId based on whether the key associated with the buffer has “sh_buffer_” as a prefix
  useGaussianMeshProps gets this combinedShBuffer and creates a texture buffer to pass into the shader.
• Now, since each channel of the textureBuffer’s RGBA channels is of size int32, this means each texel has 4 int32s worth of storage. Each gaussian has 48 float 16s, or 24 32bit spherical harmonics worth of data. This means each gaussian will need 6 texels of space to store its spherical harmonic data
• We pass this textureBuffer into the shader code, where I’ve copied gsplat's python code.

[Crezle]

Looking for this support as well to accurately represent trained 3DGS models from for example, nerfstudio!

[Crezle]

Perhaps you could try to merge changes from main to see if it works with the current version of Viser? I have been testing some of this on the current version of Viser on my own fork as well if I can aid in development of this @AntonioMacaronio.

[AntonioMacaronio]

@Crezle Yes! can definitely do that soon - will try to get back to you in 1 or 2 days on that

[Crezle]

Nice @AntonioMacaronio! I have an attempt to replicate what you've done in the newer version of Viser in #399 if you're interested in checking out before having to handle merge conflicts:)