FR: GPU acceleration

[NicholasFlamy]

I see that this uses ONNX but I haven't looked much at the code yet. I'm wondering how difficult implementing other execution providers might be.

I'd be interested in initially implementing OpenVino, as it can run (officially) on Intel and (unofficially) on AMD CPUs and performs better than the default ONNX CPU execution provider.

Then I'd also be interested in ROCm just for the fun of it and since all of my machines have AMD GPUs.

[abb128]

I'm currently switching away from ONNX to GGML at the moment, which should enable Vulkan but IIRC there were some problems with Vulkan backend. I did confirm that ROCm works but it may not be possible to ship that in Flatpak without shipping massive ROCm dependencies

[NicholasFlamy]

I'm currently switching away from ONNX to GGML at the moment, which should enable Vulkan but IIRC there were some problems with Vulkan backend.

If it ends up working that'd be really nice.

I did confirm that ROCm works but it may not be possible to ship that in Flatpak without shipping massive ROCm dependencies

I wonder if the dependencies could be optional. Maybe give instructions for passing through the host ROCm? (I know that goes against the idea of Flatpaks but as an AMD user who started on Windows with an RX 5700 XT I'm used to jank and workarounds lol.)

[origintopleft]

I did confirm that ROCm works

Hello! Is there a way I can build this project locally with ROCm support? My 9th gen i7 is only just barely keeping up with caption generation, but I've got a very beefy RX 6700 XT that would make short work of it. I understand that doing this is probably still considered experimental, and I'm willing to accept the disclaimer that bugs that come up shouldn't be reported unless I can reproduce them in the regular builds

EDIT: On further investigation, it turns out my distro packages the ROCm-enabled version of the ONNX runtime separately from the CPU bound one; switching ONNX packages out worked just fine without rebuilding this project specifically. Thanks for the good work!