Feature request: `--vpp-msharpen / --vpp-msmooth` - Edge-masked sharpening and smoothing filters

[teaching-droid]

Hi @rigaya

Thank you, as always, for the fantastic work on NVEncC/QSVEncC/VCEEncC.

I want to suggest two complementary filters from Donald A. Graft (both GPL v2+, source available at http://www.rationalqm.us/mine.html):

MSharpen - edge-selective sharpening
MSmooth - edge-preserving spatial smoothing

What makes these different from existing filters?

The biggest complaint about the unsharp mask is that setting the strength high enough to sharpen important edges also amplifies noise and small details. MSharpen solves this by first detecting important edges, then applying sharpening only to those areas. You can set very high sharpening strengths without amplifying noise because the edge mask protects non-edge areas.

Similarly, the biggest complaint about typical threshold-based smoothers is that setting the threshold high enough for good smoothing also destroys detail. MSmooth solves this by decoupling the two controls: threshold controls what is detected as detail, and strength controls how much smoothing is applied. You tune them independently.

Both filters share the same edge detection (3×3 box blur → threshold comparison → binary mask). The only difference is what happens after:

• MSharpen applies unsharp-mask sharpening at edge pixels, leaves flat areas alone
• MSmooth applies spatial averaging in flat areas, leaves edge pixels alone

How the algorithm works

Edge detection is quite simple: blur the source, compare neighbouring pixel values against a threshold, and build a binary mask. There's a "highq" mode that checks all four directions (diagonal + vertical + horizontal) instead of just diagonal.

For MSharpen, the sharpening formula at edge pixels is 4·src − 3·blur, blended by strength. Single-pass after edge detection.

For MSmooth, non-edge pixels are averaged with their cardinal neighbours (but only the neighbours that are also non-edge). This smoothing pass is repeated strength times (1-25) for progressively stronger smoothing, similar to how --vpp-warpsharp handles its iterative blur.

Proposed syntax

--vpp-msharpen [threshold=<int>],[strength=<int>],[highq=<bool>],[mask=<bool>]

• threshold (0-255, default 15): edge detection sensitivity
• strength (0-255, default 100): sharpening intensity
• highq (default true): full 4-direction edge detection
• mask (default false): show edge mask for threshold tuning

--vpp-msmooth [threshold=<int>],[strength=<int>],[highq=<bool>],[mask=<bool>],[chroma=<bool>]

• threshold (0-255, default 15): edge detection sensitivity
• strength (1-25, default 3): smoothing iterations
• highq (default true): full 4-direction edge detection
• mask (default false): show edge mask for threshold tuning
• chroma (default false): also smooth chroma planes

Code

I've taken a look at the original source code and prepared some OpenCL kernel drafts and C++ skeleton headers following the rgy_filter_* conventions, in case they might be useful as a starting point. The kernels use the standard read_imagef / write_imagef pattern. Since both filters share the blur and edge-mask kernels, implementing the second one after the first would mostly just mean adding one more kernel.

I'm happy to attach these files if you're interested.

Source material

• MSharpen 1.10 by Donald A. Graft (GPL v2+): http://www.rationalqm.us/mine.html
• MSmooth 2.02 by Donald A. Graft (GPL v2+): same site

Even implementing just MSharpen alone would already be very valuable.
I suggest these improvements to contribute to the project's continued excellence.

Thank you for considering this.

[teaching-droid]

Hi @rigaya

I've had a closer look at the actual QSVEnc source code (specifically at how rgy_filter_edgelevel and rgy_filter_warpsharp are structured) and I want to correct a couple of things I said in my original request.

When I mentioned that the kernels use the standard read_imagef / write_imagef pattern, that was not accurate. Looking at the actual codebase, the kernels read from image2d_t but write to __global uchar * raw buffers with pitch and Type casting, write_imagef is not used at all. The intermediate buffers between passes also use raw buffers, not image2d_t. I also didn't account for the Type / bit_depth compile-time defines that are needed for 8-bit/10-bit support.

I've gone back and reworked the OpenCL kernel drafts and skeleton headers to actually match these patterns:

• Output uses __global uchar *pDst with (Type)(clamp(val, 0.0f, 1.0f - RGY_FLT_EPS) * PIXEL_MAX)
• Compiled with -D Type=uchar/ushort -D bit_depth=8/10
• Source input wrapped as image2d_t via createImageFromFrameBuffer
• Intermediate buffers (blur, mask, ping-pong) as raw __global uchar * with pitch
• MSmooth iterative smoothing follows the same std::swap pattern as warpsharp's blur iterations
• Headers use RGYCLFramePool for image pools and std::array<std::unique_ptr<RGYCLFrame>, 2> for ping-pong buffers

I'm attaching the updated files in case they're useful. These are still just drafts of course. I'm sure there are things I've missed or got wrong, but they should be closer to the actual codebase now.

Files:

• rgy_filter_msharpen.cl — 2 kernels (blur + sharpen)
• rgy_filter_msharpen.h — skeleton header
• rgy_filter_msmooth.cl — 3 kernels (blur + edge mask + smooth)
• rgy_filter_msmooth.h — skeleton header

Thank you.

msharpen_msmooth_opencl_draft.zip