feat: Intelligent Backchanneling Detection - Madhav Kapila #385
+977
−21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add BackchannelingConfig for configurable word lists - Implement _is_backchanneling_only() method in agent_activity.py - Filter backchanneling only when agent is speaking (state-aware) - Support semantic interruption (mixed inputs like 'yeah but wait') - No VAD modification - logic layer only - Add README.md with usage documentation - Add unit tests and example agent
There was a problem hiding this comment.
Pull request overview
This pull request adds intelligent backchanneling detection to the LiveKit Agents framework, enabling voice agents to distinguish between passive acknowledgments (like "yeah", "ok") and actual interruptions (like "wait", "stop"). The implementation introduces a new configuration system with customizable word lists and integrates backchanneling filtering into the agent's speech interruption logic.
Changes:
- Added BackchannelingConfig dataclass with configurable backchanneling and interrupt word lists supporting multiple languages
- Integrated backchanneling detection into agent_activity.py to filter out passive acknowledgments during agent speech
- Updated AgentSession to accept backchanneling_config parameter with sensible defaults
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| livekit-agents/livekit/agents/voice/backchanneling_config.py | New configuration module defining word lists and factory functions for backchanneling detection |
| livekit-agents/livekit/agents/voice/agent_activity.py | Added _is_backchanneling_only() method and integrated backchanneling checks into interruption handlers |
| livekit-agents/livekit/agents/voice/agent_session.py | Added backchanneling_config parameter and passed it through to options |
| livekit-agents/livekit/agents/voice/init.py | Exported BackchannelingConfig for public API |
| livekit-agents/livekit/agents/voice/README.md | Comprehensive documentation with usage examples and configuration details |
| examples/voice_agents/test_backchanneling.py | Example agent implementation demonstrating backchanneling feature |
| tests/test_backchanneling.py | Unit tests for configuration and detection logic |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Updated _is_backchanneling_only() in both test classes to match agent_activity.py - Changed from word-level matching to SUBSTRING matching for interrupt words - Added sentence splitting logic using re.split() for multi-sentence handling - Added FrozenInstanceError import for proper frozen dataclass testing - Fixed test_config_is_frozen to use proper dataclass frozen attribute assignment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem Statement
Voice agents currently interrupt their speech whenever VAD detects user audio, even for passive acknowledgments like "yeah", "ok", or "hmm". This creates an unnatural conversation flow where the agent stops mid-sentence unnecessarily.
Solution
Implemented intelligent backchanneling detection that distinguishes between:
How It Works
Implementation Details
New Files
livekit-agents/livekit/agents/voice/backchanneling_config.pylivekit-agents/livekit/agents/voice/README.mdexamples/voice_agents/test_backchanneling.pytests/test_backchanneling.pyModified Files
livekit-agents/livekit/agents/voice/agent_activity.py_is_backchanneling_only()method and filtering logiclivekit-agents/livekit/agents/voice/agent_session.pybackchanneling_configparameterlivekit-agents/livekit/agents/voice/__init__.pyBackchannelingConfigKey Design Decisions
Testing Instructions