Add support for Apple Silicon Metal

[cicorias]

Are there any plans for Metal support on Apple Silicon (M1…4)

[samuel100]

Thanks for raising an issue @cicorias ! I'll move this to a discussion (under feature requests). What is your motivation for wanting Metal support - performance? If so, what level of performance over WebGpu?

[shawncicoria]

Yes, inference performance is the desire.

Ideally for Apple Silicon it uses the GPU -- since I can't see the source code for the foundry service Inference.Service.Agent -- just not sure if Metal is equivalent to webgpu.

For example, in Candle it's clear that Metal (GPU) is used.

Would the best place to look is in the ONNX Runtime Repo?

[samuel100]

On Apple Silicon we leverage the GPU (not cpu) via the ONNX Runtime WebGpuExecutionProvider. When you run the model you should see that it is pulling model-generic-gpu - what this means is that the ONNX graph has been optimized for any GPU. For some additional providers like CudaExecutionProvider the ONNX graph has some additional optimizations for CUDA that can be leveraged in the ONNX Runtime.

The WebGPU Execution Provider uses a library called Dawn that converts from the WebGPU shader language to Metal. Therefore, to directly answer the question.... "Foundry Local already supports Metal".

[cicorias]

Sweet

[skottmckay]

WebGPU EP uses a library called Dawn that converts from the WebGPU shader language to Metal.

[cicorias]

Sweet x2!