Enterprise-grade AI platform for real-time multimodal sentiment analysis combining video, audio, text, and document processing with live emotion detection, summary generation, and word cloud visualization.
🚀 Quick Start • 📖 Documentation • 🎯 Use Cases • 🐛 Report Bug • 💡 Request Feature
🏠 Landing Page
🔐 Authentication
📊 Dashboard & Analysis Results
📹 Live Detection & Real-time Results
- 🌟 Overview
- 🎯 Real-World Use Case
- ✨ Complete Feature Set
- 🛠️ Tech Stack
- 🏗️ System Architecture
- ⚡ Quick Start
- 📖 Complete Setup Guide
- 🔌 API Documentation
- 🎯 Feature Deep Dive
- 🔧 Troubleshooting
- 🚀 Deployment
- 🛣️ Roadmap
- 🤝 Contributing
- 📄 License
The Multimodal Sentiment Analysis Platform is a comprehensive AI-powered solution designed to analyze sentiments, emotions, and feedback from multiple modalities including:
- 📄 Documents (PDF, Excel) - Stakeholder feedback, comments, surveys
- 📹 Video - Facial expressions, body language, visual cues
- 🎙️ Audio - Voice tone, prosodic patterns, emotional speech
- 📝 Text - Written comments, suggestions, observations
Traditional sentiment analysis tools only process text, missing crucial emotional cues from voice tone, facial expressions, and body language. Large volumes of feedback (such as public consultation comments) often risk being inadequately analyzed or overlooked.
This platform bridges this gap by:
| Challenge | Our Solution |
|---|---|
| Analyzing large volumes of stakeholder comments | Batch PDF/Excel processing with individual line analysis |
| Understanding emotional context beyond text | Multimodal fusion (video + audio + text) |
| Summarizing lengthy feedback | AI-powered abstractive summarization |
| Identifying key themes | Word cloud visualization & entity extraction |
| Processing in real-time | WebSocket-based live emotion detection |
| Metric | Score |
|---|---|
| Emotion Recognition Accuracy | 78.4% |
| Sentiment Classification | 85.2% |
| Real-time Latency | < 1 second |
| Supported Emotions | 7 Classes |
| Document Processing | PDF, XLSX, XLS |
Government Use Case: Ministry of Corporate Affairs (MoCA)
The eConsultation module is an online platform where proposed amendments and draft legislations are posted on the MCA website. External users can submit their comments and suggestions through the MCA21 portal.
When a substantial volume of comments is received on draft legislation:
- ❌ Certain observations may be inadvertently overlooked
- ❌ Feedback may be inadequately analyzed
- ❌ Manual review of thousands of comments is time-consuming
- ❌ Identifying the overall sentiment across all feedback is difficult
- ❌ Understanding key themes and frequently mentioned topics is challenging
┌─────────────────────────────────────────────────────────────────────────────┐
│ eConsultation Feedback Analysis Flow │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ 📥 Upload Comments → 🤖 AI Analysis → 📊 Results │
│ (PDF/Excel file with (Automated processing (Dashboard with │
│ stakeholder feedback) of each comment) insights) │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ OUTPUT INCLUDES: │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ 📈 Sentiment │ │ 📝 Summaries │ │ ☁️ Word Cloud │ │
│ │ Analysis │ │ Per Comment │ │ Keywords │ │
│ │ │ │ │ │ │ │
│ │ • Positive │ │ • Concise │ │ • Visual │ │
│ │ • Negative │ │ meaning │ │ representation│ │
│ │ • Neutral │ │ • Key points │ │ • Density map │ │
│ │ for EACH line │ │ extracted │ │ of themes │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ 🏷️ Named Entity │ │ 📊 Overall │ │ 📁 Export │ │
│ │ Recognition │ │ Statistics │ │ Options │ │
│ │ │ │ │ │ │ │
│ │ • Organizations │ │ • Total comments│ │ • JSON │ │
│ │ • Persons │ │ • Distribution │ │ • Download │ │
│ │ • Locations │ │ • Page-by-page │ │ results │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
| Requirement | Platform Feature |
|---|---|
| Sentiment Analysis | ✅ Legal-BERT based zero-shot classification for positive/negative/neutral sentiment on each comment |
| Summary Generation | ✅ DistilBART abstractive summarization to convey meaning precisely |
| Word Cloud | ✅ Visual density representation of keywords used by stakeholders |
| Reduce Analysis Effort | ✅ Automated batch processing of entire documents |
| Individual & Overall | ✅ Both per-line analysis AND combined document analysis |
- Upload the Excel/PDF export of eConsultation comments
- Automatic Processing - Each line is analyzed individually for:
- Sentiment (positive/negative/neutral)
- AI-generated summary
- Named entities (organizations, persons, locations mentioned)
- Combined Analysis - Overall document analysis generates:
- Sentiment distribution chart
- Word cloud of all keywords
- Top mentioned entities
- Export Results - Download complete analysis as JSON
|
|
|
|
┌──────────────────────────────────────────────────────────────┐
│ Live Detection System │
├──────────────────────────────────────────────────────────────┤
│ │
│ 📹 Camera Input ──→ 🔄 WebSocket Server ──→ 📊 Results │
│ (Port 8080) │
│ │
│ Features: │
│ • Real-time emotion detection │
│ • JWT-authenticated connections │
│ • Video frame processing │
│ • Audio chunk analysis │
│ • Text input sentiment │
│ • Periodic analysis updates (every 2 seconds) │
│ • Upload recorded sessions for batch analysis │
│ │
└──────────────────────────────────────────────────────────────┘
| Feature | Description |
|---|---|
| 🔐 Authentication | NextAuth.js with secure password hashing (BCrypt) |
| 🔑 API Key Management | Generate, revoke, and manage API keys (sa_live_* format) |
| 📊 Quota System | Track usage with per-operation costs (2 points each for sentiment, live detection, PDF analysis) |
| 💳 Stripe Payments | Subscription management with quota stacking |
| 📈 Usage Analytics | Real-time quota tracking and billing |
| 🔄 Auto-Upload | Automatic session upload after live detection |
| Plan | Price | Quota | Features |
|---|---|---|---|
| Basic | $9.99/mo | 30 requests | Standard processing, Email support |
| Pro | $29.99/mo | 100 requests | Priority processing, API access |
| Premium | $99.99/mo | 1000 requests | Unlimited features, Priority support |
| Enterprise | Contact Sales | Custom | SLA, Custom integrations, Dedicated manager |
- Responsive Design - Mobile-first with Tailwind CSS
- Animated UI - Smooth transitions with Framer Motion
- Real-time Dashboard - Live analytics and processing status
- Progress Tracking - Visual feedback for long-running processes
- Interactive Results - Expandable analysis cards, sentiment charts
| 🎨 Frontend | ⚙️ Backend | 🤖 AI/ML | ☁️ Infrastructure |
|
|
|
|
flowchart TB
subgraph Client["🖥️ Client Layer"]
Browser[Web Browser]
Camera[📹 Camera]
Mic[🎙️ Microphone]
end
subgraph NextJS["⚡ Next.js Application (Port 3000)"]
Pages[React Pages & Components]
API[API Routes]
Auth[NextAuth.js]
end
subgraph WebSocket["🔄 WebSocket Server (Port 8080)"]
LiveServer[Live Analysis Server]
JWT[JWT Authentication]
end
subgraph Python["🐍 FastAPI Service (Port 8001)"]
PDFAnalyzer[PDF/Excel Analyzer]
LegalBERT[Legal-BERT Classifier]
Summarizer[DistilBART Summarizer]
NER[spaCy NER]
WordCloud[Word Cloud Generator]
end
subgraph Database["🗄️ Database Layer"]
Prisma[(Prisma ORM)]
PostgreSQL[(PostgreSQL/SQLite)]
end
subgraph External["☁️ External Services"]
S3[(AWS S3)]
SageMaker[AWS SageMaker]
Stripe[Stripe API]
end
Browser --> Pages
Camera --> LiveServer
Mic --> LiveServer
Pages --> API
API --> Auth
API --> Prisma
API --> PDFAnalyzer
API --> S3
API --> Stripe
LiveServer --> JWT
LiveServer --> SageMaker
PDFAnalyzer --> LegalBERT
PDFAnalyzer --> Summarizer
PDFAnalyzer --> NER
PDFAnalyzer --> WordCloud
Prisma --> PostgreSQL
| Service | Port | Description |
|---|---|---|
| Next.js Frontend + API | 3000 |
Marketing site, dashboard, REST APIs, NextAuth, Prisma |
| WebSocket Live Analysis | 8080 |
Real-time emotion streaming (JWT protected) |
| FastAPI PDF Analyzer | 8001 |
Document extraction, sentiment, summarization, NER, word cloud |
| Stripe Webhooks | 3000 |
Subscription events at /api/stripe/webhooks |
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ User │ │ ApiQuota │ │ Purchase │
├──────────────┤ ├──────────────┤ ├──────────────┤
│ id │◄──│ userId │ │ id │
│ name │ │ requestsUsed │ │ userId │
│ email │ │ maxRequests │ │ stripeSession│
│ password │ │ resetDate │ │ amount │
│ stripeId │ │ secretKey │ │ planType │
│ isActive │ └──────────────┘ │ requestsGrant│
│ subscriptionId └──────────────┘
└──────────────┘
│
▼
┌──────────────┐ ┌──────────────┐
│ VideoFile │ │ Session │
├──────────────┤ ├──────────────┤
│ id │ │ id │
│ userId │ │ sessionToken │
│ key │ │ userId │
│ analyzed │ │ expires │
└──────────────┘ └──────────────┘
# Verify installations
node --version # v18+ required
python3 --version # v3.10+ required
npm --version # v10.2.4+# Clone and enter directory
git clone https://github.com/UtkarsHMer05/sentiment-analysis.git && cd sentiment-analysis
# Install dependencies
npm install
# Configure environment
cp .env.example .env.local
echo "AUTH_SECRET=\"$(openssl rand -base64 32)\"" >> .env.local
echo "NEXTAUTH_URL=\"http://localhost:3000\"" >> .env.local
echo "DATABASE_URL=\"file:./dev.db\"" >> .env.local
# Setup database
npm run db:push
# Setup Python service
cd pdf-analyzer-service && chmod +x setup.sh && ./setup.sh && cd ..
# Start everything
npm run dev:full🎉 Open http://localhost:3000
- Open http://localhost:3000
- Click Sign Up → Create account
- Navigate to Dashboard
- Test PDF Analysis or Live Detection
| Software | Version | Download |
|---|---|---|
| Node.js | v18+ | nodejs.org |
| Python | v3.10+ | python.org |
| Git | Latest | git-scm.com |
Step 1: Clone & Install Node Dependencies
git clone https://github.com/UtkarsHMer05/sentiment-analysis.git
cd sentiment-analysis
npm installExpected output:
added 500+ packages
✔ Generated Prisma Client
Step 2: Configure Environment Variables
cp .env.example .env.localEdit .env.local:
# Database (SQLite for dev, PostgreSQL for production)
DATABASE_URL="file:./dev.db"
# Authentication (generate with: openssl rand -base64 32)
AUTH_SECRET="your-secret-here"
NEXTAUTH_URL="http://localhost:3000"
# AWS (Optional - for video analysis)
AWS_REGION="us-east-1"
AWS_ACCESS_KEY_ID="your-key"
AWS_SECRET_ACCESS_KEY="your-secret"
AWS_INFERENCE_BUCKET="your-bucket"
AWS_ENDPOINT_NAME="your-endpoint"
# Stripe (Optional - for payments)
STRIPE_PUBLIC_KEY="pk_test_..."
STRIPE_SECRET_KEY="sk_test_..."
STRIPE_WEBHOOK_SECRET="whsec_..."
# Python Backend
PYTHON_BACKEND_URL="http://localhost:8001"Step 3: Setup Database
npm run db:push
# View database (optional)
npm run db:studioStep 4: Setup Python Backend
cd pdf-analyzer-service
chmod +x setup.sh
./setup.sh
cd ..This installs:
- FastAPI & Uvicorn
- PyTorch & Transformers
- Legal-BERT, DistilBART models
- spaCy with en_core_web_sm
- WordCloud, pdfplumber, openpyxl
Step 5: Start All Services
# Full development mode (Next.js + WebSocket)
npm run dev:full
# Or run separately:
npm run dev # Terminal 1: Next.js (port 3000)
npm run websocket # Terminal 2: WebSocket (port 8080)
cd pdf-analyzer-service && source venv/bin/activate && python main.py # Terminal 3: Python (port 8001)All protected endpoints require:
Authorization: Bearer sa_live_your_api_key_here
| Route | Method | Description | Auth |
|---|---|---|---|
/api/upload-url |
POST | Generate S3 presigned URL for video upload | API Key |
/api/sentiment-inference |
POST | Trigger SageMaker video analysis | API Key |
/api/pdf-analysis |
POST | Analyze PDF/Excel document | Session |
/api/live-emotion |
POST | Process live recording session | Session |
/api/live-recording-upload |
POST | Upload live-recorded media chunks | API Key |
/api/user/api-key |
POST/DELETE | Create or revoke API keys | Session |
/api/user/quota |
GET | Get quota status | Session |
/api/stripe/checkout |
POST | Create Stripe checkout session | Session |
/api/stripe/webhooks |
POST | Handle Stripe subscription events | Stripe |
# Analyze PDF document
curl -X POST "http://localhost:8001/analyze-document" \
-F "file=@feedback.pdf" \
-F "analysis_type=both"
# Response includes:
# - individual_analysis: Array of per-line results
# - combined_analysis: Overall document analysis with word cloudConnect: ws://localhost:8080?token=<jwt>
Send:
{
"type": "start_analysis" | "stop_analysis" | "video_frame" | "audio_chunk" | "text_input",
"data": {},
"timestamp": 1734882712345
}Receive:
{
"type": "video_analysis",
"data": {
"emotion": "joy",
"sentiment": "positive",
"confidence": 0.92,
"keywords": ["happy", "excited"]
},
"timestamp": 1734882712345
} PDF/Excel Upload
│
▼
┌───────────────────────────────┐
│ Text Extraction │
│ (pdfplumber / openpyxl) │
└───────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Legal-BERT │ │ DistilBART │ │ spaCy NER │
│ Zero-Shot │ │ Summarization │ │ Entities │
│ Classification │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ ┌─────────────────┐ │
│ │ Word Cloud │ │
│ │ Generation │ │
│ └─────────────────┘ │
│ │ │
└─────────────────────┼─────────────────────┘
▼
Analysis Results JSON
| Operation | Cost | Description |
|---|---|---|
sentiment_analysis |
2 points | Video sentiment analysis |
live_detection |
2 points | Live emotion detection session |
pdf_analysis |
2 points | PDF/Excel document analysis |
Database Issues
# Reset database
rm prisma/dev.db
npm run db:push
# Regenerate Prisma client
npx prisma generatePort Conflicts
# Kill process on port
lsof -ti:3000 | xargs kill -9
lsof -ti:8001 | xargs kill -9
lsof -ti:8080 | xargs kill -9Python Backend Issues
cd pdf-analyzer-service
source venv/bin/activate
pip install -r requirements.txt
python -m spacy download en_core_web_smReset Everything
rm -rf node_modules .next prisma/dev.db
npm install
npm run db:push
npm run dev:full- Connect repository to Vercel
- Set environment variables
- Use Vercel Postgres or Neon for database
- Deploy WebSocket server separately (AWS ECS, Cloud Run, or Render)
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]docker build -t sentiment-app .
docker run -p 3000:3000 sentiment-app# Development
npm run dev:full # Start all services
npm run dev # Next.js only
npm run websocket # WebSocket only
# Database
npm run db:push # Apply schema
npm run db:studio # Visual editor
# Code Quality
npm run lint # Check linting
npm run lint:fix # Fix issues
npm run typecheck # Type checking
npm run format:write # Format code
# Production
npm run build # Build app
npm run start # Start production| Service | Port | URL |
|---|---|---|
| Next.js | 3000 | http://localhost:3000 |
| WebSocket | 8080 | ws://localhost:8080 |
| Python API | 8001 | http://localhost:8001 |
| Prisma Studio | 5555 | http://localhost:5555 |
sentiment-analysis/
├── src/
│ ├── app/
│ │ ├── api/ # API routes
│ │ ├── dashboard/ # Main dashboard
│ │ ├── live-detection/ # Live emotion capture
│ │ ├── live-results/ # Live analysis results
│ │ └── pdf-analysis/ # Document analysis
│ ├── components/ # React components
│ │ ├── LiveEmotionCapture.tsx
│ │ ├── PdfAnalysisCapture.tsx
│ │ ├── PdfAnalysisResults.tsx
│ │ └── AnalysisResultsModal.tsx
│ ├── lib/
│ │ ├── quota.ts # Quota management
│ │ └── stripe.ts # Stripe config
│ ├── server/
│ │ ├── auth/ # NextAuth config
│ │ └── websocket/ # Live analysis server
│ └── sections/ # Landing page sections
├── pdf-analyzer-service/ # Python FastAPI backend
│ ├── main.py # API & ML models
│ ├── requirements.txt
│ └── setup.sh
├── prisma/
│ └── schema.prisma # Database schema
└── package.json
- Modern dashboard UI
- Live emotion detection
- PDF/Excel sentiment analysis
- Word cloud generation
- Summary generation
- Named entity recognition
- Quota & billing system
- API key management
- Mobile app (React Native)
- GraphQL API
- Multi-language support
- Video conference integration
- Batch processing queue
- Advanced analytics dashboard
- Fork the repository
- Create a feature branch:
git checkout -b feat/amazing-feature - Commit changes:
git commit -m "feat: add amazing feature" - Run checks:
npm run check - Submit a pull request
This project is licensed under the MIT License - see the LICENSE file for details.
- GitHub Issues
- Email: utkarshkhajuria7@gmail.com
- ⭐ Star us on GitHub!
Made with ❤️ by Utkarsh Khajuria
A comprehensive AI solution for analyzing stakeholder feedback, public consultations, and multimodal sentiment detection.