Feature Request: Live Transcription with Real-time Speaker Identification

[davidamacey]

Feature Request: Live Transcription with Real-time Speaker Identification

Overview

Implement real-time live transcription capabilities during audio recording sessions with automatic speaker identification using existing speaker profiles in the database.

Core Requirements

1. Live Transcription Engine

• Real-time speech-to-text during active recording sessions
• Configurable accuracy vs speed trade-offs:
  • Fast mode: Lower latency (~100-200ms), acceptable accuracy for live viewing
  • Accurate mode: Higher latency (~500ms-1s), maximum accuracy
• Streaming audio processing with WebRTC or WebSockets
• Progressive transcript building with word-level confidence scores

2. Real-time Speaker Identification

• Live speaker matching against existing speaker profiles in database
• Voice fingerprint comparison using speaker embedding models
• Dynamic speaker labeling as conversation progresses
• Confidence thresholds for automatic vs manual speaker assignment
• Unknown speaker detection with option to create new profiles

3. User Interface Components

• Live transcript viewer - clean, scrollable interface
• Speaker identification panel - visual indicators for matched speakers
• Real-time controls - start/stop live transcription, accuracy mode toggle
• Navigation options - modal, sidebar, or dedicated page view
• Export capabilities - save live transcript for review/editing

Technical Architecture

Option A: Client-side Processing

Pros:

• Lower server load
• Reduced latency
• Privacy benefits
• Offline capability

Cons:

• Device performance dependency
• Battery drain on mobile
• Limited model accuracy
• Browser compatibility issues

Implementation:

• WebAssembly-based Whisper models
• Web Audio API for microphone access
• IndexedDB for speaker profile caching
• Service Workers for background processing

Option B: Server-side Processing

Pros:

• Consistent performance
• Access to full AI models
• Centralized speaker database
• Better accuracy

Cons:

• Network dependency
• Server resource intensive
• Higher latency
• Bandwidth requirements

Implementation:

• WebSocket streaming to backend
• Celery workers for real-time processing
• Redis for session state management
• FastAPI streaming endpoints

Option C: Hybrid Approach (Recommended)

• Client-side pre-processing for immediate feedback
• Server-side refinement for accuracy and speaker ID
• Progressive enhancement - starts fast, improves over time
• Fallback mechanisms when network/server unavailable

Industry Standards & Best Practices

WebRTC Standards

• MediaStream API for audio capture
• RTCPeerConnection for real-time streaming
• Opus codec for efficient audio compression
• DTLS encryption for secure transmission

Speech Recognition Standards

• Web Speech API as fallback option
• ONNX models for cross-platform compatibility
• Voice Activity Detection (VAD) to reduce processing
• Buffered streaming with overlap for continuity

Accessibility Standards

• WCAG 2.1 AA compliance for transcript display
• Keyboard navigation support
• Screen reader compatibility
• High contrast mode support
• Adjustable text size options

Implementation Plan

Phase 1: Foundation (Sprint 1-2)

• Audio capture infrastructure

  • WebRTC MediaStream integration
  • Audio quality preprocessing (noise reduction, normalization)
  • Buffered streaming with configurable chunk sizes
  • Error handling and reconnection logic

• Backend streaming architecture

  • WebSocket endpoints for real-time audio
  • Celery task queue for processing pipeline
  • Redis session management
  • Audio format standardization

Phase 2: Basic Live Transcription (Sprint 3-4)

• Real-time speech processing

  • Streaming Whisper integration (OpenAI Whisper-streaming or faster-whisper)
  • Configurable model sizes (tiny, base, small for speed vs accuracy)
  • Word-level timestamps and confidence scores
  • Progressive transcript assembly

• Frontend live viewer

  • Real-time transcript display component
  • Auto-scrolling with manual override
  • Word highlighting as spoken
  • Confidence indicators

Phase 3: Speaker Identification (Sprint 5-6)

• Speaker embedding pipeline

  • Real-time voice fingerprint extraction
  • Comparison against existing speaker profiles
  • Confidence scoring and threshold management
  • Unknown speaker detection

• Speaker UI integration

  • Visual speaker indicators
  • Confidence meters
  • Manual override controls
  • Speaker profile quick-assign

Phase 4: Advanced Features (Sprint 7-8)

• Performance optimization

  • Client-side preprocessing options
  • Adaptive quality based on network conditions
  • Model caching strategies
  • Memory usage optimization

• User experience enhancements

  • Multiple view modes (modal, sidebar, fullscreen)
  • Export and save functionality
  • Integration with existing transcript editor
  • Mobile-responsive design

Phase 5: Production Readiness (Sprint 9-10)

• Quality assurance

  • Cross-browser testing
  • Mobile device testing
  • Network condition testing
  • Load testing with multiple concurrent sessions

• Documentation and deployment

  • User guides and help documentation
  • Admin configuration options
  • Monitoring and alerting
  • Performance metrics dashboard

Technical Specifications

Audio Processing

• Sample Rate: 16kHz for transcription, 44.1kHz for speaker ID
• Bit Depth: 16-bit minimum
• Channels: Mono for processing, stereo support
• Formats: WAV, WebM, OGG support
• Compression: Opus for streaming, FLAC for storage

Performance Targets

• Transcription Latency: <500ms end-to-end
• Speaker ID Latency: <200ms additional
• Accuracy: >95% for clear speech
• Concurrent Sessions: 10+ simultaneous users
• Memory Usage: <500MB per session

Security Considerations

• End-to-end encryption for audio streams
• RBAC integration with existing auth system
• Data retention policies for temporary audio
• Privacy controls for speaker profiles
• Audit logging for compliance

Dependencies & Integration

New Dependencies

• WebRTC libraries: aiortc, pywebrtc, or similar
• Streaming models: faster-whisper, whisper-streaming
• Real-time frameworks: Socket.IO or native WebSockets
• Audio processing: librosa, soundfile for backend

Existing System Integration

• Speaker profiles: Extend current speaker management
• Authentication: Integrate with JWT/RBAC system
• Database: New tables for live sessions and temporary data
• Storage: MinIO for temporary audio chunks
• Monitoring: Flower dashboard for real-time tasks

Success Metrics

• User adoption: >50% of users try live transcription
• Accuracy satisfaction: >4/5 user rating
• Performance: <1s average transcription delay
• Reliability: >99% session completion rate
• Speaker ID accuracy: >90% correct identification

Risk Mitigation

• Progressive enhancement: Feature works without live transcription
• Graceful degradation: Fallback to post-processing mode
• Resource limits: Configurable session timeouts and limits
• Quality controls: Minimum audio quality requirements
• User controls: Easy disable/enable options

Future Enhancements

• Multi-language support for live transcription
• Real-time translation capabilities
• Automated meeting minutes generation
• Integration with calendar systems
• Voice command recognition for meeting control

---

Priority: High
Effort: Large (8-10 sprints)
Impact: High - transforms user experience for live meetings
Technical Risk: Medium - complex real-time processing requirements

🤖 Generated with Claude Code

[davidamacey]

Update: Live Transcription Development

We've decided to implement live transcription as a separate companion project called OpenCompanion.

Why a Separate Project?

Live transcription with real-time speaker identification has distinct architectural requirements (streaming audio, GPU worker pools, WebSocket orchestration) that warrant its own codebase. This allows us to:

• Keep OpenTranscribe focused on file-based transcription workflows
• Develop specialized real-time infrastructure without impacting existing features
• Enable both standalone use and integration with OpenTranscribe

OpenCompanion Features

The new project addresses all requirements from this issue and more:

Feature	Status
Real-time STT	✅ Streaming with faster-whisper (large-v3)
Speaker Diarization	✅ Real-time segmentation with pyannote
Speaker Identification	✅ Voice matching using OpenSearch k-NN embeddings
Live Insights	✅ Real-time action items, decisions, questions extraction
Session Summaries	✅ AI-generated meeting summaries with export
Streaming TTS	✅ VibeVoice-Realtime with streaming LLM input
Multi-user Sessions	✅ Supports 10+ concurrent users

Coming Soon

The code is under active development and will be published to: github.com/davidamacey/opencompanion

OpenCompanion can operate in:

• Standalone mode: Full Svelte 5 dashboard for independent use
• Integration mode: Backend service for OpenTranscribe's frontend

Stay tuned for the v0.1.0 release!

[davidamacey]

Update: Detailed Implementation Plan

A comprehensive implementation plan has been developed for live streaming transcription & real-time speaker identification. The full plan is available as a gist for future reference:

Live Streaming Transcription — Implementation Plan (Gist)

Architecture Overview

The plan covers a dual-mode streaming architecture:

Mode	Processing	Latency	GPU Required
Mode A (Server-side)	Backend GPU via WebSocket	~500ms	Server GPU
Mode B (Client-side)	Browser via WebGPU/ONNX	~200ms	Client WebGPU
Hybrid	Client STT + Server speaker ID	~300ms	Client WebGPU + Server GPU

Plan Sections

1. ONNX Export Pipeline — Export WeSpeaker ResNet34-LM to ONNX for browser inference with fp16 quantization (~25MB), bit-for-bit embedding compatibility with PyTorch
2. Backend: Server-Side Streaming (Mode A) — WebSocket endpoint (/api/ws/stream), AudioRingBuffer, StreamingTranscriptionService (faster-whisper), StreamingSpeakerService (live k-NN matching against OpenSearch profiles)
3. Frontend: Client-Side Streaming with WebGPU (Mode B) — @huggingface/transformers with WebGPU backend, AudioWorklet capture, Web Worker inference, IndexedDB speaker profile caching
4. Frontend: Shared UI Components — StreamingTranscript, StreamingControls, StreamingSettings (shared across modes)
5. Backend: API Endpoints for Client Mode — REST endpoints for session save, speaker identification, profile sync
6. Settings & Configuration — Mode selection, VAD sensitivity, chunk size, speaker ID thresholds
7. Database & Migration — source_type column for live_recording vs upload, streaming session metadata
8. Testing Strategy — Unit tests, integration tests, E2E browser tests, speaker ID accuracy validation

Implementation Phases

Phase	Scope	Estimate
1. Foundation	Backend infrastructure, audio buffer, schemas, migration	3-4 days
2. Server-Side Streaming	Mode A core — WebSocket, transcription, speaker services	5-6 days
3. Backend REST API	Client mode endpoints (save, identify, profiles)	2-3 days
4. Frontend Streaming UI	Svelte components, stores, AudioWorklet, i18n	4-5 days
5. ONNX + Client Inference	Mode B — ONNX export, WebGPU detection, Web Worker	5-6 days
6. Hybrid Mode + Polish	Auto-detection, hybrid wiring, E2E tests, docs	2-3 days

Total estimated effort: 21-27 days

Key Design Decisions

• Separate WebSocket endpoint (/api/ws/stream) — binary PCM frames at ~320KB/s need dedicated handling, not multiplexed with existing JSON notification WS
• WeSpeaker ONNX for browser — 256-dim embeddings directly comparable to batch-processed profiles in OpenSearch (same model as PyAnnote v4)
• AudioWorklet + Web Worker — AudioWorklet captures PCM on audio thread, Web Worker runs inference off main thread, zero jank
• Progressive speaker ID — initial match after 2s of speech, confidence improves over session duration with running centroid updates

Related Plans

A separate Performance Improvements Plan covers optimization items that complement live transcription (pipeline retries, benchmark tooling, Celery chord-based enrichment).

---

This issue remains open for tracking. Implementation will proceed under the OpenCompanion companion project as noted in the previous comment.

[m13v]

We've been working on real-time transcription in our macOS AI agent (Fazm) with WhisperKit, and speaker identification is the hardest part. Some thoughts:

For the fast vs accurate mode tradeoff, we found you can get surprisingly good results by using a smaller Whisper model (tiny/base) for the live preview and then re-transcribing completed segments with the larger model. Users see instant feedback and get corrected text within a few seconds.

For speaker embeddings, the key challenge is that real-time comparison against a database of voice prints needs to be fast. We'd suggest pre-computing speaker embeddings at profile creation time and using cosine similarity at inference. Keep the embedding model separate from the transcription model so they can run in parallel.

One practical tip: for the WebSocket streaming, send audio in 16kHz mono format from the client rather than converting server-side. Cuts the data transfer in half and reduces server-side processing. The Web Audio API's AudioContext can do the resampling before streaming.

For the confidence thresholds on speaker assignment, we found 0.7 cosine similarity works as a reasonable default - above that, auto-assign. Below that, label as "Unknown Speaker" and let the user correct it, which then improves the profile for future sessions.

[m13v]

We open-sourced our transcription pipeline here: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

Shows the WhisperKit integration, chunked audio processing, and how we handle real-time partial results. The speaker identification piece is something we're still iterating on but the audio pipeline architecture might be a useful reference.

[davidamacey]

@m13v Wow this is excellent feedback and confirmation of the pipeline and workflow. This is definitely emerging capability to do real-time identification. Thank you for sharing your progress.

Great idea for doing the real-time display with a smaller faster model and then cleaning up with a larger model on the backend.

Agreed the live identification is quite tricky!

I am open to collaboration if you want to see what you can implement. The current latest code I am developing on is feat/search-and-rag, I am getting close to 0.4.0 release. It's taking longer due to integration testing with signification backend improvements.

[m13v]

Really appreciate the openness to collaboration! The two-model approach (fast model for real-time display, larger model for cleanup) has been working well for us. We're using WhisperKit for the local inference piece. I'll take a look at your feature branch and see where we can contribute. What's your current approach for speaker embedding generation - are you doing it on-device or sending audio segments to an API?

[davidamacey]

@m13v, thank you for the quick reply. Open to learning more and collaborating where we can.

I am currently using my forked version of pyannote-audio, where I optimized the batches and throughput for stable gpu memory.

The basis for OpenTranscribe is to bring any audio file and process it on the backed from S3 compatible storage, then run through the preprocessing, databasing, transcription, diarization, then post process the results and database, indexing text, then gender ML model, and then store the results, triggering LLM tasks for summary and topic extraction. It's a full pipeline!

Related to the speaker diarization its using the clustering with pyannote-audio with the community-1 model that has overlap detection and labeling for segments. Internally it uses the wespeaker-resnet-24 model for the embeddings. So for the Cloud ASR models I embed the timestamped labeled speaker segments and it provides the same embedding results as if I ran it locally. Allowing for matching of speaker profiles. <-- This feature is what I just added and tested this evening in the branch.

• Additionally I created an AHC clustering algorithm in the backed to assist in labeling speakers in the GUI for pre clustered unlabeled entities.

Currently this is all on the backend processing. But it would be ideal for the microphone recording feature on the frontend that I have, to display the words and speaker name on the page as they are talking. While I have Claude agents I don't have enough hands to trigger all the tasks at once, lol.

Hopefully that gives more context.

[m13v]

Your pipeline sounds really well thought out. The pyannote-audio fork with optimized batching for stable GPU memory is smart - that's one of the biggest pain points with diarization at scale.

The S3-based processing architecture with preprocessing, transcription, diarization, then post-processing is solid for batch workloads. For the live transcription side, the main challenge we hit was keeping diarization accurate in real-time when speakers overlap or switch quickly. We ended up using a sliding window approach with speaker embedding caching that helped a lot.

The gender ML model and text indexing post-processing layers are interesting additions. Would love to hear more about how the gender classification fits into your workflow. Are you using it for speaker labeling or something else?

If you want to compare approaches on the real-time side, our implementation is at https://fazm.ai/gh - specifically the audio pipeline and speaker tracking components.