Cascaded shadow map support

[swiftcoder]

I'm adding cascaded shadow maps to my own project, and I'd be open to contribute an implementation back to three-d, but I'm not spotting a great way to plug new shadow map backends into the existing lighting system.

Since cascaded shadow maps require changes to both shadow map generation and sampling, we'd need a way to provide replacements for both Light.shader_source() and Light.generate_shadow_map(). The most strait forward is to add a ShadowMapper trait (name needs workshopping), but that would require Light to become Light<S> where S: ShadowMapper, which has knock-on effects up and down the API. Or maybe alternately Light could contain a shadow_mapper: Box<dyn ShadowMapper>. Or we could make the caller explicitly pass an Option<&ShadowMapper> into the two functions that require it...

Do you have a preferred approach here? Is adding fancier shadow algorithms a good fit with the goals of three-d?

[asny]

Sorry for the late reply 😬 Hope you're still up for the task, it sounds like a really nice addition! It's definitely aligned with the goals of three-d.

I think being able to swap out the shadow algorithm of each light is a bit overkill. If someone really wants to implement their own shadow algorithm and don't want to contribute to three-d, it's possible to implement a new Light type.

I think the best approach to add cascaded shadow map support in three-d is changing the existing DirectionalLight to support both normal and cascaded shadow maps internally. The only addition to the API as far as I can tell, is a function generate_cascaded_shadow_map that people can choose to call instead of generate_shadow_map. generate_cascaded_shadow_map should take the camera as well as any additional necessary parameters. Internally, the lighting calculations would be different whether or not generate_shadow_map or generate_cascaded_shadow_map was called last. Does that makes sense?

[BonsaiDen]

The way I did Cascading Shadow Maps (using Variance Shadow Maps) was to:

1. Implement the Light trait for a new struct
2. Provide a custom shader for calculating the light via the existing calculate_light() and multiplying it with the shadow sample from the cascade, this is a bit "ugly" since there is no global way to inject additional fragment source that shared across lights, so all functions need to postfix with the light ID in order to avoid complications, however the rest is relatively straight foward
3. Have a method on the new DirectionCSMLight to compute the cascades
4. In here we encounter a few more troubles:

• We need to compute the individual cascade frustums and view projections, that's not too hard, however we cannot create a Camera directly from a view and the custom projection that is needed, so we need inject an additional uniform for the cascadeViewProjection
• Now of course, this will not be picked up by the Geometries that we render into the cascades, so we stick those into a wrapper impl of the trait and do a lot of dirty string replace magic to get things working:

struct CascadeDepthGeometry<'a, T: Geometry> {
    inner: &'a T,
    cascade_matrix: Mat4
}

impl<'a, T: Geometry> Geometry for CascadeDepthGeometry<'a, T> {
    fn id(&self, required_attributes: FragmentAttributes) -> u16 {
        self.inner.id(required_attributes) | 1 << 14
    }

    fn aabb(&self) -> three_d::AxisAlignedBoundingBox {
        self.inner.aabb()
    }

    fn draw(
        &self,
        camera: &Camera,
        program: &Program,
        render_states: RenderStates,
        attributes: FragmentAttributes,
    ) {
        program.use_uniform("cascadeMatrix", self.cascade_matrix);
        self.inner.draw(camera, program, render_states, attributes);
    }

    fn vertex_shader_source(&self, required_attributes: FragmentAttributes) -> String {
        // Emulate GL_DEPTH_CLAMP
        let source = self.inner.vertex_shader_source(required_attributes);
        let mut patched = String::with_capacity(source.len());
        patched.push_str("uniform mat4 cascadeMatrix;\n");
        patched.push_str("out float cascadeDepth;\n");
        for l in source.lines() {
            if l.contains("uniform") {
                patched.push_str(l);

            } else {
                patched.push_str(&l.replace("viewProjection", "cascadeMatrix"));
            }
            if l.contains("gl_Position = ") || l.contains("gl_Position=") {
                patched.push('\n');
                patched.push_str("cascadeDepth = gl_Position.z / gl_Position.w;\n");
                patched.push_str("cascadeDepth = (gl_DepthRange.diff * cascadeDepth + gl_DepthRange.near + gl_DepthRange.far) * 0.5;\n");
                patched.push_str("gl_Position.z = 0.0;\n");
                // FIXME Need to make sure viewProjection is still used, otherwise Program.use_uniform will panic!
                patched.push_str("cascadeDepth *= viewProjection[3][3];\n");

            } else {
                patched.push('\n');
            }
        }
        log::info!("Vertex shader #{} patched for CSM", self.id(FragmentAttributes::NONE));
        patched
    }
}

In the end we needed to do this anyway though, as we need to emulate GL_DEPTH_CLAMP (not available in WebGL) to avoid artifacts during rendering (such as large object getting cut of by the frustum near plane)

in float cascadeDepth;

layout (location = 0) out vec2 outColor;

void main() {
    // Emulate GL_DEPTH_CLAMP
    float depth = clamp(cascadeDepth, 0.0, 1.0);
    gl_FragDepth = depth;

    // bias second moment based on viewing angle
    float dx = dFdx(depth);
    float dy = dFdy(depth);
    vec2 moments = vec2(depth, depth * depth);
    moments.y += 0.25 * (dx * dx + dy * dy);

    // Optimization for 2 moments proposed in
    // http://momentsingraphics.de/Media/I3D2015/MomentShadowMapping.pdf
    moments.y = 4.0 * (moments.x - moments.y);
    outColor = moments;
}

However currently when writing custom shaders that end up replace certain parts of their inner fragment / vertex shader, running into issues with the panic behaviour of Program::use_uniform() often requires some workarounds to ensure that the uniforms are still used but don't affect anything.

One additional complication I've run into is that Variance Shadow Mapping and other techniques greatly benefit from mip maps, however there is currently no way to limit the number of mip levels when creating textures and updating all 10+ levels for multiple cascades per frame is rather slow (hence why my implementation is only using two levels).

TL;DR; a list of small things that would be nice to have:

• Switch all internal meshes and material to always use Program::use_uniform_if_required() to make custom shader work easier, or have a method on Program to disable the panic from within custom code (e.g. program.ignore_unused_uniforms(true))
• A way to configure the maximum number of mip levels for textures, this would most likely need a rework of the texture APIs since they already have a lot of parameters (while we're at it we could also add support for anisotropic filtering configuration if the necessary extension is present)
• Built in support for GL_DEPTH_CLAMP emulation activated via a new flag in the FragmentAttributes

[asny]

Thanks for the effort @BonsaiDen! I'm not sure what makes this so much different from a normal shadow map, it's basically just more maps 🤔 I think it makes more sense to support it in each of the existing light types instead of implementing a new one. But of course, that requires changing three-d, if you don't want that, then yeah, you need to copy quite a lot of code. I'm not sure I want to change anything so it's easier to not contribute to three-d. There's always the option of forking the project and do whatever you like.

[BonsaiDen]

Most of the troubles come from edge cases when rendering geometry into the cascades (e.g. the clamping I mentioned), there's an endless amount of tradeoffs and config options to make it look good for different scenes.

I guess splitting the Light and Shadow logic would a be a good first step, i.e. keep the Light trait, but move the shadow rendering out in it's own ShadowPass trait, then you'd pass both a slice of Light as well as Shadow instances into the render pipeline.

Each ShadowPass instance would then generate shader code to handle all the passed in lights and have a draw/update method to handle the shadow map.

btw: The recent introduction of the Viewer trait (removes the shader rewrite hack for the projection matrix above since I can now supply an impl with the per-cascade projection) and the removal of the FragmentAttributes definitely made things easier to tweak / customize! 👍