Metal: Add deformable conv2d and performance optimizations #3355
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements deformable conv2d (DCNv2) for Metal/Apple Silicon: - deformable_im2col: forward pass with bilinear sampling at learned offsets - deformable_col2im: backward pass for input gradients - deformable_col2im_coord: backward pass for offset and mask gradients Features: - Support for modulated deformable convolution (optional mask) - Configurable stride, padding, dilation, and offset groups - Float32 and Float16 variants - Atomic add for thread-safe gradient accumulation Ported from mps-deform-conv (https://github.com/mpsops/mps-deform-conv). Enables models like DCNv2, Deformable DETR, and BiRefNet on Apple Silicon.
- Use MPSGraph for conv2d (7x faster for 3x3+ kernels) - Add fast broadcast kernel for inner-dim pattern [B,N,C]+[C] (2.7x faster Linear) - Add fast transpose_last2 kernel for attention K^T pattern (14x faster contiguous)
imperatormk
changed the title
- Add graph caching to avoid recompilation on each call - Use MPSGraphExecutable for pre-compiled graph execution - Use MPSCommandBuffer for async GPU execution (non-blocking) - Use direct MTLBuffer for zero-copy tensor data - Add command_queue() getter to Commands struct Benchmarks (M3 Max): - 3x3 conv 256x256: 12.5ms -> 2.5ms (5x faster, matches PyTorch) - 7x7 conv 256x256: 21ms -> 11ms (2x faster, matches PyTorch)
Cast MTLBuffer to AnyObject pointer type for correct Objective-C message passing.
|
the linked mps-deform-conv does not seem to exist. |
Author
✅ |
|
i still think you're causing problems by vibe-coding so many extensions and then going around Github to push for their inclusion before they're a) reviewed, b) vetted for inclusion. you did this on so many repositories so far i'm really wondering what is your motivation? why are you doing this? |
Author
|
I partially validate your point but this is not the place for this |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds Metal GPU kernels for deformable convolution 2D (DCNv2) on Apple Silicon, plus several performance optimizations for the Metal backend.
Deformable Conv2D Kernels
deformable_im2col: forward pass with bilinear sampling at learned offsetsdeformable_col2im: backward pass for input gradientsdeformable_col2im_coord: backward pass for offset and mask gradientsPerformance Optimizations
[B, N, C] + [C](~2.7x faster Linear layers)transpose_last2kernel for attention K^T pattern (~14x faster contiguous)Features
Benchmarks
Use Cases
Enables models that rely on deformable convolutions to run on Apple Silicon:
Origin
Deformable conv ported from mps-deform-conv, a standalone PyTorch MPS extension.
Test Plan
candle-metal-kernels/src/tests.rs