feat: add Pyannote speaker diarization support

[mhenrichsen]

🎯 Overview

This PR integrates Pyannote speaker diarization as a toggleable feature for the /v1/audio/transcriptions endpoint, similar to the existing VAD functionality.

✨ Features Added

• 🎯 Speaker Diarization: Integrates pyannote/speaker-diarization-3.1 model
• 🔧 Toggleable: Add diarization=true/false parameter (default: false)
• 🏗️ Model Management: Full PyannoteModelManager with lifecycle management
• 📊 API Extension: Speaker fields added to TranscriptionSegment and TranscriptionWord
• ⚙️ Configuration: PyannoteConfig with device and TTL settings
• ✅ Backward Compatible: Existing API calls work unchanged

🧪 Testing

Successfully tested with real audio file:

• ✅ Basic transcription: Works correctly
• ✅ Diarization enabled: Identifies 2 speakers (SPEAKER_00, SPEAKER_01)
• ✅ Speaker assignment: Correctly maps speakers to transcript segments
• ✅ API compatibility: No breaking changes to existing functionality

💡 Usage Example

```bash
curl -X POST "http://localhost:8000/v1/audio/transcriptions" \
-H "Content-Type: multipart/form-data" \
-F "file=@audio.wav" \
-F "model=guillaumekln/faster-whisper-tiny" \
-F "diarization=true" \
-F "response_format=verbose_json"
```

Response includes speaker information:
```json
{
"segments": [
{
"text": "Hello, this is speaker one.",
"start": 0.0,
"end": 2.5,
"speaker": "SPEAKER_00"
},
{
"text": "And this is speaker two responding.",
"start": 3.0,
"end": 5.5,
"speaker": "SPEAKER_01"
}
]
}
```

📁 Implementation Details

• Dependencies: Added pyannote-audio>=3.1.1
• Model Manager: Created PyannoteModelManager following existing patterns
• Configuration: Added PyannoteConfig and _unstable_diarization setting
• API Types: Extended with speaker fields maintaining backward compatibility
• Endpoints: Updated /v1/audio/transcriptions and /v1/audio/translations
• Tests: Comprehensive test coverage for all scenarios

🔧 Configuration

Set via environment variables:

• _UNSTABLE_DIARIZATION=true - Enable diarization by default
• PYANNOTE__INFERENCE_DEVICE=cuda - Use GPU for diarization
• PYANNOTE__TTL=600 - Model cache time (seconds)

The feature is ready for review and integration! 🚀

[dotmobo]

amazing, please integrate this devs :)

[sebastianelsner]

I tested this PR today and it worked well for me. The issues I had:

1. lots and lots of UserWarnings, probably one for each chunk of audio to be diaritized:
   torchaudio/_backend/utils.py:213: UserWarning: In 2.9, this function's implementation will be changed to use torchaudio.load_with_torchcodec` under the hood. Some parameters like ``normalize``, ``format``, ``buffer_size``, and ``backend`` will be ignored. We recommend that you port your code to rely directly on TorchCodec's decoder instead: https://docs.pytorch.org/torchcodec/stable/generated/torchcodec.decoders.AudioDecoder.html#torchcodec.decoders.AudioDecoder.
2. The uv.lock file needs updating with pyannote-audio uv add pyannote-audio otherwise the docker containers do not build
3. Add a section to the documentation, that the user needs to setup the HF_TOKEN and accept the EULA for the gated models pyannote/segmentation-3.0 and pyannote/speaker-diarization-3.1
4. When I ran pyannote with cuda, the transcription with the curl command given aboce would relieably crash the container. When running with PYANNOTE__INFERENCE_DEVICE=cpu, so that pyannote ran with cpu and transcription with gpu, the container did not crash.

[mhenrichsen]

@sebastianelsner please give it another go

[sebastianelsner]

The code changes did not help.

Concerning:

The performance impact on diarization is typically minimal compared to the stability benefit. If you need maximum performance for both models, consider:
- Running them in separate containers with dedicated GPU memory allocations
- Increasing GPU memory if available
- Running operations sequentially instead of concurrently

Literally none of this AI generated text applies here. The impact is huge, its like 5x slower. this cant be run in separate containers, ram increase does not help and nothing runs concurrently.

[flefevre]

This feature will be great to help scientists team to make their meetings notes.
Hope you will integrate it

[grungkers]

How about realtime integration?

[flefevre]

@mhenrichsen have you time to look at this ? it will be key for laboratory researchers that want to meeting note very quickly
how can we help

[flefevre]

It seems that last version is compatible.
Could you check with speaker diarization #582 Thanks and happy new year