Moving Background Blur to GPU

[rapterjet2004]

🖼️ Screenshots

Before	After

Demo (foreground original / background blurred)	Mask (foreground white / background black)

This PR addresses an issue with the previous implementation of Background Blurring. It used too much memory space, as openCV contains huge builds for every possible memory architecture supported on Android. This leads to duplicate files, large amounts of unnecessary code, and a larger download size. The MediaPipe library and the .tflite file used for foreground-background detection was trivial in comparison.

This is fixed by moving the entire blurring operation to openGL ES, a graphics framework that is built into Android, cutting out 100 MB from the build size while also increasing performance. This is done by the implementing Gaussian Blur algorithm from scratch in GLSL, the openGL Shader language.

This adds

• BackgroundBlurGPUProcessor a file that handles the openGL logic, and acts as an abstraction over the inner workings of the image processing. OpenGL's logic is ... very unclear. I tried my best to document as much as possible, including any linear algebra that I had to perform.
• background_blur_vertex the required vertex shader, needed to identify the position of each pixel on the screen, before the geometry, rasterization, and fragment shader steps of the openGL pipeline.
• gaussian_blur_frag_shader the Gaussian Blur implementation, this is what actually performs the logic on the image pixels. The code is also somewhat unclear if you're not familiar with shaders or gaussian blur. It helps to read the technical article I commented from Intel.
• seg_mask_frag_shader the masking operation implementation, very simple, just checking if the pixel should be blurred or retain it's original unblurred value
• testing_frag_shader a simple fragment shader used for quick testing
• libyuv_android a light weight helper lib for converting between image formats without the overhead of OpenCV

Copied over from old PR

• BackgroundBlurFrameProcessor A WebRTC VideoProcessor object, this handles the video frame conversions and processing. Holds references to the ImageSegmenterHelper and the BackgrounBlurGPUProcessor. It now holds it's own dedicated Thread and Handler for running OpenGL's GPU operations. It also takes in a reference to the surfaceTextureHelper from CallActivity to send the VideoFrame objects to the WebRTC VideoSink on the appropriate capture thread. Failing to do both of these causes a crash.
• BlurBackgroundViewModel
• ImageSegmenterHelper uses MediaPipe to perform the segmentation mask (foreground / background separation). Really just a wrapper over the MediaPipe API
• background_replace_24px
• 2 new strings

🚧 TODO

• rebase + solve merge conflicts
• analyze for memory leaks
• linter

🏁 Checklist

• ⛑️ Tests (unit and/or integration) are included or not needed
• 🔖 Capability is checked or not needed
• 🔙 Backport requests are created or not needed: /backport to stable-xx.x
• 📅 Milestone is set
• 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

[github-actions]

APK file: https://www.kaminsky.me/nc-dev/android-artifacts/5891.apk



To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.

[github-actions]

Codacy

Lint

Type	master	PR
Warnings	98	99
Errors	0	0

SpotBugs

Category	Base	New
Bad practice	6	6
Correctness	10	10
Dodgy code	54	54
Internationalization	3	3
Malicious code vulnerability	3	3
Performance	4	4
Security	1	1
Total	81	81

Lint increased!