feat: Implement native Apple Silicon transcription with MLX-Whisper or whisper.cpp

[davidamacey]

Summary

OpenTranscribe currently uses WhisperX for transcription, which doesn't fully utilize Apple Silicon's GPU capabilities (falls back to CPU on MPS devices). Implementing a native Apple Silicon solution using MLX-Whisper or whisper.cpp would provide significant performance improvements for macOS users with M1/M2/M3 chips.

Current State

• WhisperX is configured to use CPU on Apple Silicon (MPS) devices due to compatibility issues
• Performance is suboptimal compared to native implementations
• Users with expensive Apple Silicon hardware aren't getting full value from their GPU

Proposed Solutions

Option 1: MLX-Whisper (Recommended)

Pros:

• 30-40% faster than whisper.cpp on Apple Silicon
• Native MLX framework designed specifically for Apple Silicon
• Excellent GPU utilization on M1/M2/M3 chips
• Python-based, easier integration with existing codebase
• Active development and Apple support

Cons:

• Only works on Apple Silicon (need to maintain WhisperX for other platforms)
• Smaller community compared to whisper.cpp
• May require significant refactoring of transcription service

Option 2: Lightning-Whisper-MLX

Pros:

• Claims 10x faster than whisper.cpp
• 4x faster than standard MLX-Whisper
• Optimized specifically for Apple Silicon
• Best-in-class performance for macOS

Cons:

• Very new project, stability concerns
• Limited documentation
• May lack features compared to WhisperX

Option 3: whisper.cpp

Pros:

• Cross-platform (works on Apple Silicon, CUDA, CPU)
• Very mature and stable
• 6-7x faster than vanilla Whisper on CPU
• Good Apple Silicon support via Metal
• Could potentially replace WhisperX entirely

Cons:

• Requires C++ integration (more complex)
• 30-40% slower than MLX-Whisper on Apple Silicon
• Would need Python bindings or subprocess calls

Performance Benchmarks (2024)

Based on recent benchmarks:

• MLX-Whisper: ~50% faster than vanilla Whisper on Apple Silicon
• Lightning-Whisper-MLX: 10x faster than whisper.cpp (claimed)
• whisper.cpp: 6-7x faster than vanilla Whisper on CPU, good Metal support

Implementation Plan

Phase 1: Research & Prototype

• Benchmark MLX-Whisper vs whisper.cpp on M1/M2/M3 hardware
• Test feature parity with WhisperX (timestamps, speaker alignment)
• Evaluate integration complexity for each option
• Create proof-of-concept implementation

Phase 2: Architecture Design

• Design abstraction layer for multiple transcription backends
• Create platform detection logic (Apple Silicon vs CUDA vs CPU)
• Plan migration strategy from WhisperX
• Design configuration system for backend selection

Phase 3: Implementation

• Implement chosen solution (likely MLX-Whisper)
• Create fallback mechanism to WhisperX for non-Apple platforms
• Update Docker configurations for Apple Silicon
• Implement proper error handling and logging

Phase 4: Testing & Optimization

• Comprehensive testing on M1, M2, M3 hardware
• Performance benchmarking vs current WhisperX implementation
• Memory usage optimization
• Edge case testing (long files, multiple speakers)

Phase 5: Documentation & Deployment

• Update documentation for Apple Silicon users
• Create migration guide
• Update setup scripts for macOS
• Release with clear performance expectations

Technical Requirements

Core Features to Maintain

• Word-level timestamps
• Speaker diarization compatibility
• Multiple language support
• Batch processing capability
• Progress callbacks for UI updates

New Requirements

• Automatic backend selection based on hardware
• Configurable backend via environment variables
• Performance metrics logging
• Graceful fallback on errors

Code Changes Required

Backend Changes

backend/app/tasks/transcription/
├── base_transcriber.py          # New: Abstract base class
├── whisperx_transcriber.py      # Refactored from whisperx_service.py
├── mlx_transcriber.py           # New: MLX-Whisper implementation
├── whisper_cpp_transcriber.py   # New: Optional whisper.cpp implementation
└── transcriber_factory.py       # New: Factory for backend selection

Configuration Updates

• Add TRANSCRIPTION_BACKEND environment variable
• Add APPLE_SILICON_OPTIMIZATION flag
• Update hardware detection to identify MLX availability
• Add backend-specific configuration options

Docker Updates

• Create Apple Silicon specific Dockerfile variant
• Update docker-compose with platform-specific service definitions
• Ensure proper MLX installation in containers

Acceptance Criteria

• 2x or better performance improvement on Apple Silicon vs current implementation
• No regression in transcription quality
• Seamless fallback to WhisperX on non-Apple hardware
• All existing features continue to work
• Clear documentation for users
• Automated backend selection based on hardware

Performance Targets

• M1 Pro: < 5 minutes for 1-hour audio (currently ~10 minutes)
• M2/M3: < 4 minutes for 1-hour audio
• Memory usage: < 8GB for large model
• GPU utilization: > 80% during transcription

Related Issues

• bug: Docker containers fail to start on ARM64/Apple Silicon Macs #38 - Docker containers fail to start on ARM64/Apple Silicon Macs (resolved)
• [FEATURE] Cross-Platform PyTorch Model Support for CUDA, MPS, and CPU #25 - Cross-Platform PyTorch Model Support

References

• MLX-Whisper GitHub
• Lightning-Whisper-MLX
• whisper.cpp
• Performance comparison article

Labels

enhancement, performance, macos, apple-silicon, transcription, mlx

Priority

High - Significant performance improvement for growing Apple Silicon user base

[TheFutureIsVoluntary]

@davidamacey May I suggest Parakeet MLX as well? I've been getting under 2 minutes for 1 hour audio on M1 Pro with it: https://github.com/senstella/parakeet-mlx

[Edit: Or, add functionality for calling an OpenAI compatible API endpoint - ie, allow changing the URL for the OpenAI API endpoint in the ASR Provider setup. I assume this would give users LOTS of options of what they can use, as many projects are supporting OpenAI compatible API endpoints. I was able to get an API endpoint for parakeet running using mlx-audio from https://github.com/Blaizzy/mlx-audio . I could post audio to transcribe like this:

curl -X POST http://localhost:8000/v1/audio/transcriptions
-F "file=@audio_file.opus"
-F "model=mlx-community/parakeet-tdt-0.6b-v2"
]

[m13v]

for macOS native transcription on Apple Silicon, WhisperKit has been really solid in production for us. the CoreML backend means you get proper GPU scheduling through the ANE (Apple Neural Engine) without fighting Metal directly. one gotcha we hit: the model download and CoreML compilation on first run can take a while, especially for larger models. worth caching the compiled model artifacts so subsequent launches are instant. also if you're doing long recordings, chunked processing is essential because loading the entire audio into memory for a 2 hour file will blow up. we split at silence boundaries detected by a simple energy threshold, not fixed time windows, so you don't cut mid-word.

[m13v]

our TranscriptionService that handles WhisperKit inference on Apple Silicon with the silence-boundary chunking and model caching I mentioned: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift