Godot 4.0 does not have 16-bit viewports yet, which means I cannot calculate flowmaps and various other textures correctly.
There is a PR working on it, but it is stale. godotengine/godot#61667
EDIT: It turned out I had some bugs in my code that made the problem worse, fixing those issues made it doable to generate the flowmaps, thought they will be better with 16 bit viewports. I might also move away from using Viewports and use Render Device directly, but that will require quite a bit of work.
Keeping the bug open now to track the issue.