The ANARI_KHR_GEOMETRY_GLYPH extension for glyphs/oriented shapes

[kvankooten]

Usecase is visualization of a set of (typically unconnected) points with arbitrary shapes of arbitrary non-uniform scale (typicaly called glyphs).

Problem with the current spec is: line-segment representation for cylinders/cones whereas spheres are point-based, constant conversion for user (and arbitrary interpolation of data for all arrays) if they only want to change shape of the representing glyphs around point samples. Need for a point-based representation that allows for orientations (quaternions) and non-uniform scaling, so the various attribute arrays and shapes can be provided/changed/removed orthogonally from eachother without the need for the app to re-process any existing data - ie. a natural glyph interface.

A new geometry type is preferable to amending the sphere geometry type, as this allows for points to be represented in a form fully optimized for points only (think of a custom point splatting/neural representation).

The proposal is as follows:

...

The extension introduces geometry type glyph/orientedShape, with the same parameters (and definitions) as the sphere geometry, except:

• instead of (vertex.)radius, a (vertex.)scale of type ANARI_(FLOAT32/)FLOAT32_VEC3
• orientations are introduced using the parameter (vertex.)orientation of type ANARI_FLOAT32_QUAT_IJKW.
• Additionally, there are the shapeType and shapeGeometry parameters of type ANARI_STRING and ANARI_GEOMETRY respectively.
• The shapeTransform parameter of ANARI_FLOAT32_MAT4, as part of the definition of the prototype shape.

Parameter shapeType supports types up to sphere/cylinder/cone if ANARI_KHR_GEOMETRY_SPHERE/CYLINDER/CONE are supported, whereas shapeGeometry supports up to triangle/quad if ANARI_KHR_GEOMETRY_TRIANGLE/QUAD are supported. For further shape types, extensions ANARI_KHR_GEOMETRY_GLYPH_SHAPETYPE_<TYPE> and ANARI_KHR_GEOMETRY_GLYPH_SHAPEGEOMETRY_<TYPE> will have to be introduced.

If both shapeType and shapeGeometry are set, the shapeGeometry is used over the shapeType. If none are set, a sphere is chosen as default shape.
In case of shapeType being set to cylinder/cone, a (vertex.)cap parameter is supported, to denote cap information similar to the cylinder and cone geometry types.

In absence of a scale and shapeTransform parameter, the bounds of a shapeType are always of length 2 in every dimension, centered around the origin, so [min,max]=[-1.0, 1.0]. A shapeGeometry retains its bounds as the extent of the geometry. The scale multiplies these bounds in every dimension (thereby being equivalent to the radius parameter, but over multiple dimensions). The shapeTransform is applied before the scale.

In absence of an orientation and shapeTransform parameter, cones and cylinders are aligned along the Z-axis, with the apex of the cone on the positive side. The shapeTransform is applied before the orientation.

Arrays of the shapeGeometry set for a particular attribute parameter take precendence over those set for that same parameter on the glyph geometry.

Name of the extension is either ANARI_KHR_GEOMETRY_GLYPH or ANARI_KHR_GEOMETRY_ORIENTED_SHAPE.

...

Open questions:

• There is some overloading by having the all of the parameters shapeTransform, scale and orientation. The first parameter is required in those cases where a user wants to compose more complex shapes out of primitive shapes without having to modify any of the vertex arrays, so that they can be shared between the various glyph geometries. An example is the case of arrows made out of a cone glyph and a cylinder glyph. In an implementation, shapeTransform is therefore an intrinsic part of the shape definition (ie. only changing when the shape changes), where an implementation can choose to premultiply the transform into the shape's vertices. The scale and orientation parameters are - similar to all other per-vertex parameters- simply defaults in those cases where an array is not available, can change at will regardless of shape and ignored when a per-vertex array is set - in contrast to the shapeTransform, which always applies. An example usecase could be to have the scale/orientation change over time based on some global parameter, where the shape stays the same.
• shapeType and shapeGeometry could possibly be merged into one parameter, although they can be quite different from an implementation standpoint, as per the supported shape types for either of the two: instanced cylinders is different from an instanced geometry of cylinder primitives. Also, they exhibit subtly different rules for 'trickle down parameters', which is highlighted in the next point.
• Consider an example where curve geometries would be supported on the shapeGeometry parameter - the radius parameter would have to be introduced on the glyph geometry for per-instance control. Could there be a general rule for different types of geometries (especially those bound to shapeGeometry), automatically defining particular parameters to be supported per-instance? This seems easy at first; just ignore the already existing parameters of position/color/attribute/radius/index and create a vertex.<parametername> on the glyph geometry for what remains. But this already fails for the normal parameter of a triangle geometry. Also, this rule seems again slightly different for the glyph's shapeType, where in case of a sphere the radius parameter should be ignored, whereas for spheres on the glyph's shapeGeometry there is an argument to be made for the radius parameter to be reintroduced (since we'd be instancing collections of spheres).
• one consideration is to rename the glyph's vertex. prefix to instance. on parameters; this would be to allow per-instance arrays to be identified differently from possible per-vertex arrays provided by a child geometry in the shapeGeometry/Type parameter. By keeping this distinction explicit, it may seem like a better choice in case where materials explicitly specify the level (instance/primitive/vertex) of an attribute to bind to. However, the renaming may cause confusion, since the glyph also holds primitive attributes and there is hardly a usecase for both vertex and primitive attributes to be provided for glyphs at the same time. In fact, the glyph's own primitive arrays are equivalent to vertex arrays if there is no primitive.index parameter on the glyph, and in case there is a primitive.index, one would hardly ever need two shapes around the same point to be rendered with different attributes (reuse). Therefore, all arrays on the glyph can be considered to be 'per-instance' from the point of binding them to a material parameter, with the existing precedence rules being more than sufficient.

[kvankooten]

An example extensions json is provided by the usd device, which already implements the extension: https://github.com/NVIDIA-Omniverse/AnariUsdDevice/blob/main/khr_geometry_glyph.json

The repo also provides tests under https://github.com/NVIDIA-Omniverse/AnariUsdDevice/blob/main/examples/anariTutorial_usd.c#L437C6-L437C6

[johguenther]

I wonder whether the discussed array instance sub-type will cover this glyph usecase well, while being more generic.

[kvankooten]

The glyph proposal is created especially because something like the array instance subtype is too generic: it's critical that the various aspects of the transform are separated, and the fixed shape transform is separated from the per-instance transform parameters as well - otherwise it is not a natural glyph interface.

It is critical for the USD device to not have this information flattened into a single matrix, and it dramatically simplifies ANARI's integration into apps like ParaView in general, where glyphs are front and center. Bugs due to mistakes in the transform logic already exist even without the flattening and simply switching back and forth between certain different shapes should not result in dramatically different codepaths being executed, which would be the case if you would compare using a sphere geometry with an instance array of arrows.

I wouldn't be principally opposed to having the glyph construct exists elsewhere in the ANARI object hierarchy since there are some drawbacks/ambiguities of the current approach, namely essentially introducing two-level transforms - in the instance, and then again in the shapeTransform param. But it is definitely most similar to a mesh object in its interface, so I'm not sure where it should then move.