SH color handling question / clarification

[ckolve]

From my understanding we have 2 components to the SH data in the original INRIA style 3dgs data - f_dc_ (direct SH component, 0-order, ie base colour) and then f_rest_ (higher order SH coefficients, view dependent color data). When loading the data, f_dc is transformed into Cd, f_rest ends up in the sh_coefficients array.
Subsequent operations on the SH data seem to only operate on the view dependent part, ignoring the direct component. An example would be 'gaussian splat attribute adjust' - if you do an hsv adjust, it only modifies f_rest, but ignores f_dc (or color).
Is this behavior intentional?

[david-rhodes]

The behavior is "experimental." To be completely transparent, I don't believe the math is accurate either way. f_dc_* is decoded (to Cd) before color adjustments happen, but f_rest_* attributes have no such decoding. I assume HSV adjustments for view dependency "kinda work" because the operations are linear.

Anyway, I think it's better to keep view dependent color and base color separate for now (e.g., it can be useful to desaturate spherical harmonics in isolation). If you want to constrain them, use channel expressions from a color adjust SOP for Cd to constrain HSV adjust parms in the gaussian splat attribute adjust SOP.

[ckolve]

agreed on keeping f_dc and f_rest separate, but it seems keeping f_dc around is useful for regenerating Cd at later steps in a network might be useful...

as a data point:

o3-mini thinks HSV adjustments directly on the coefficients is wrong ("because the SH expansion is a linear projection of a spatial function, and nonlinear color operations (like hue rotation) do not translate simply into linear operations on the coefficients"), instead should be done as a postprocess after the color is determined based in the complete set of SH coefficients. But it would be possible to generate new coefficients based on a complete set of color corrected outputs (via sampling the new color sphere)

detailed reasoning here:
https://www.perplexity.ai/search/read-this-doc-https-3dvar-com-owx6tRz8QYyjlCC5zpvVAQ

[david-rhodes]

Thank you for sharing this.

Yeah, the UX isn't quite ironed out yet. Ideally, we could fetch the rasterized splats directly in Copernicus, but this isn't yet possible.

The gaussian_splats_evaluate_spherical_harmonics SOP was originally developed to bake the evaluated SH from a camera view into the Cd attribute via VEX, but this has not been updated to handle SH rotation (or the sh_coefficients refactor).

cc @rubendhz what do you think? I remember us discussing including f_dc in the sh_coefficients, but I didn't consider downstream edits like this.

[rubendhz]

If I understood correctly, the main question here is whether we should always keep the original/unmodified color attribute alongside the "main" one (that might have been modified, including with potentially incorrect operations). Also, I see this question as independent from whether we use Inria's naming convention or Houdini's, or stored in SH (f_dc) or RGB (Cd) form, since going back and forth is a trivial operation which could even be provided as a standalone utility.

f_dc_* is decoded (to Cd)...

^ This is literally this operation

When a splat model is rotated, there's no point in keeping the original SHs around, only the rotated ones. Ideally we think of color adjustments the same way, if an operator has done something to color (assuming operation is correct), original value becomes irrelevant. If user really wants to keep original color value (prior to the operation), it'd be on their plate to duplicate/rename the original attribute. I often do this to compare results or other purposes.

My preference is not to duplicate color values, and only have Cd + sh_coefficients to avoid giving color a special treatment (we wouldn't be keeping original values opacity/scale/etc) and also avoiding having to manage two data representations in every utility moving forward. If a utility requires the sh form for color, it can convert to that and back in a self-contained manner.

[david-rhodes]

If a utility requires the sh form for color, it can convert to that and back in a self-contained manner.

Sounds good--thanks for your input!

[ckolve]

going back and forth is a trivial operation which could even be provided as a standalone utility.

Makes sense, agree that these kind of transformation would be handy to be encapsulated in a dedicated node/utility. I see similar transformation being done to generate LAB_* , HSV_* , RGB_* representations upon import.

If user really wants to keep original color value (prior to the operation), it'd be on their plate to duplicate/rename the original attribute. I often do this to compare results or other purposes.

This also makes sense. I think to facilitate this, a change on the UX side might be nice - currently these transformation happen inside the import node. So the user never gets direct access to the raw un-transformed data. Of course it is not hard to load the ply directly and merge those datastreams, but additional import options could be handy.

[david-rhodes]

Yes, good call! I've been wanting to refactor feature_attributes into their own digital asset for a while now.

As for preserving original ply attributes, that's likely just a simple toggle/switch on the import SOP. Want to make a PR? 😄

Are you able to discuss what you're building, by the way? Keen to know your use case(s). Cheers!

[ckolve]

Not at liberty to talk about specific projects, but I think it's fair to say that Houdini with GSOPs makes for a nice prototyping environment to test ideas related to working with splatting data. It has replaced / enhanced many of my personal splat manipulation setups.

In the context of this discussion and interchange of splatting data in general, these efforts by the AOUSD group members might be interesting to keep an eye on:
https://forum.aousd.org/t/3d-gaussian-splatting-in-openusd/2232/2

[david-rhodes]

Thank you for sharing.

A USD schema for splats is quite interesting . . .