An emotionally intelligent voice assistant with real-time emotion detection and visual feedback
VISU is an advanced voice assistant that combines conversational AI with emotional intelligence. It detects user emotions, responds empathetically, and provides real-time visual feedback through a modern web interface.
- Real-time emotion detection - Analyzes user sentiment and responds appropriately
- Empathetic responses - Matches user's emotional state (sad → empathetic, happy → excited)
- 8 emotion types - Happy, curious, empathetic, neutral, excited, concerned, supportive, playful
- Visual feedback - Live emotion display with colors and animations
- Natural conversation - Powered by advanced language models (GPT/Cerebras)
- High-quality TTS - Cartesia voice synthesis
- Accurate STT - Deepgram speech recognition
- Voice activity detection - Silero VAD for seamless interaction
- Live emotion display - WebSocket-powered real-time updates
- Modern UI - Glassmorphism design with smooth animations
- Responsive design - Works on desktop and mobile
- Connection resilience - Auto-reconnection and error handling
- Modular design - Separate agent, context, and frontend components
- Configuration management - Environment-based settings
- Error handling - Robust fallback mechanisms
- Extensible - Easy to add new emotions, tools, and features
- Python 3.8+
- Node.js (optional, for advanced frontend development)
- API keys for:
- OpenAI/Cerebras (LLM)
- Deepgram (Speech-to-Text)
- Cartesia (Text-to-Speech)
- LiveKit (Real-time communication)
-
Clone the repository
git clone https://github.com/AbhiramVSA/VISU-Reborn.git cd VISU-Reborn -
Set up Python environment
# Using uv (recommended) uv sync # Or using pip pip install -r requirements.txt
-
Configure environment variables
cp .env.example .env # Edit .env with your API keysRequired environment variables:
OPENAI_API_KEY=your_openai_key DEEPGRAM_API_KEY=your_deepgram_key CARTESIA_API_KEY=your_cartesia_key LIVEKIT_API_KEY=your_livekit_key LIVEKIT_API_SECRET=your_livekit_secret LIVEKIT_URL=your_livekit_url
-
Start the emotion frontend
cd frontend python server.py -
Run the voice assistant
# In a new terminal uv run main.py -
Open the emotion display
- Visit http://localhost:8000 in your browser
- You'll see real-time emotion updates as you interact with VISU
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Voice Input │───▶│ VISU Agent │───▶│ Emotion API │
│ (Microphone) │ │ - LLM Processing│ │ (HTTP POST) │
└─────────────────┘ │ - Emotion Det. │ └─────────────────┘
│ - Response Gen. │ │
┌─────────────────┐ │ - TTS Output │ ▼
│ Voice Output │◀───│ │ ┌─────────────────┐
│ (Speakers) │ └──────────────────┘ │ Web Frontend │
└─────────────────┘ │ - Live Display │
│ - WebSocket │
│ - Animations │
└─────────────────┘
agent/visu.py- Main voice assistant with emotion detectionfrontend/server.py- FastAPI server for emotion visualizationcontext/- Context loading and managementconfig/- Configuration and environment managementprompts/- AI personality and behavior rules
VISU recognizes and responds to user emotions with appropriate emotional states:
| User Emotion | VISU Response | Visual Color | Emoji |
|---|---|---|---|
| Sad | Empathetic, Concerned | Purple, Orange | 🤗 😟 |
| Happy | Excited, Joyful | Gold, Red | 😊 🤩 |
| Angry | Calming, Supportive | Green | 🫂 |
| Confused | Patient, Helpful | Blue | 🤔 |
| Neutral | Friendly | Light Gray | 😐 |
Create a .env file in the root directory:
# Required API Keys
OPENAI_API_KEY=sk-...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=sk_car_...
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
LIVEKIT_URL=wss://...Edit the prompt files to customize VISU's behavior:
prompts/prompt.txt- Core personality and speaking styleprompts/rules.txt- Operational rules and constraints
Place text files in the context/ directory to give VISU additional knowledge.
VISU-Reborn/
├── agent/ # Voice assistant core
│ ├── visu.py # Main agent implementation
│ └── __init__.py
├── frontend/ # Emotion visualization
│ ├── server.py # FastAPI backend
│ ├── requirements.txt
│ └── README.md
├── context/ # Knowledge base
│ ├── context.py # Context loader
│ └── *.txt # Context files
├── config/ # Configuration
│ └── settings.py # Environment management
├── prompts/ # AI personality
│ ├── prompt.txt # Core personality
│ └── rules.txt # Behavioral rules
├── main.py # Entry point
├── pyproject.toml # Python dependencies
└── .env # Environment variables
# Run frontend tests
cd frontend
python -m pytest
# Run agent tests
python -m pytest tests/# Format code
black .
isort .
# Lint code
flake8 .
pylint agent/ frontend/We welcome contributions! Here's how to get started:
-
Fork the repository
git fork https://github.com/AbhiramVSA/VISU-Reborn.git
-
Create a feature branch
git checkout -b feature/amazing-feature
-
Make your changes
- Follow the existing code style
- Add tests for new features
- Update documentation as needed
-
Test your changes
# Test the agent uv run main.py # Test the frontend cd frontend && python server.py
-
Commit and push
git commit -m "feat: add amazing feature" git push origin feature/amazing-feature -
Create a Pull Request
- Use the PR template
- Describe your changes clearly
- Link related issues
- Add new emotion types
- Improve emotion detection accuracy
- Create emotion transition animations
- Add support for different languages
- Improve voice recognition accuracy
- Add voice cloning features
- Create mobile app version
- Add emotion history graphs
- Implement theme customization
- Add comprehensive testing
- Improve error handling
- Optimize performance
- Add Docker support
- Write tutorials
- Create API documentation
- Add video demos
- Use Black for Python formatting
- Follow PEP 8 conventions
- Write descriptive commit messages
- Add docstrings to all functions
- Write unit tests for new features
- Test voice interaction manually
- Verify frontend functionality
- Check emotion detection accuracy
- Update README for new features
- Add inline code comments
- Write clear function docstrings
- Update API documentation
Found a bug? Please create an issue with:
- Clear title describing the problem
- Steps to reproduce the issue
- Expected vs actual behavior
- System information (OS, Python version, etc.)
- Logs or error messages
Have an idea? Create an issue with:
- Clear description of the feature
- Use case explaining why it's needed
- Proposed implementation (if you have ideas)
- Examples of similar features
Contributors will be:
- Listed in the contributors section
- Mentioned in release notes
- Given credit in documentation
- Invited to the contributors team
- Multi-language support
- Voice cloning capabilities
- Mobile app version
- Docker containerization
- Video emotion detection
- Multi-user conversations
- Plugin system
- Cloud deployment options
- AR/VR integration
- Advanced emotion analytics
- AI model fine-tuning
- Enterprise features
This project is licensed under the MIT License - see the LICENSE file for details.
- LiveKit for real-time communication infrastructure
- OpenAI for advanced language model capabilities
- Deepgram for accurate speech recognition
- Cartesia for high-quality text-to-speech
- FastAPI for the modern web framework
- Contributors who make this project better
- Documentation: Wiki
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Email: Support
If you find VISU helpful, please consider:
- ⭐ Starring the repository
- 🍴 Forking to contribute
- 📢 Sharing with others
- 💝 Sponsoring the project
Built with ❤️ by the VISU Community
Making AI more emotionally intelligent, one conversation at a time
Making AI more emotionally intelligent, one conversation at a time