RAG server

[yuqiannemo]

API Comparison: Unified Server vs Old Servers

Overview

The new unified_server.py combines the best of both main.py and rag_server.py, removing duplicates and adding Colinear Query Expansion.

---

🔄 Endpoint Mapping

OLD: main.py → NEW: unified_server.py

Old Endpoint	New Endpoint	Status	Notes
GET /files/search	GET /search/keyword	✅ Kept	Wildcard filename search
GET /files/{file_id}	GET /files/{file_id}	✅ Kept	Same functionality
POST /files/register	POST /files/register	✅ Enhanced	Now returns action field
DELETE /files/{file_id}	DELETE /files/{file_id}	✅ Enhanced	Also deletes embeddings

OLD: rag_server.py → NEW: unified_server.py

Old Endpoint	New Endpoint	Status	Notes
POST /upload	POST /files/upload	✅ Enhanced	Better naming, update support
POST /search	POST /search/semantic	✅ UPGRADED	Now with Query Expansion!
GET /files	GET /files	✅ Kept	Same functionality
GET /files/{id}	GET /files/{file_id}	✅ Merged	Combined with main.py version
GET /stats	GET /stats	✅ Enhanced	More detailed stats
GET /	GET /	✅ Enhanced	Better health check

---

🆕 What's New: Colinear Query Expansion

How It Works

When you search with POST /search/semantic:

{
  "query": "machine learning tutorial",
  "top_k": 5,
  "use_query_expansion": true,
  "expansion_count": 3
}

Behind the scenes:

1. Original Query: "machine learning tutorial"
2. Expanded Query 1: "document about machine learning tutorial"
3. Expanded Query 2: "file containing machine learning tutorial"
4. Expanded Query 3: "information regarding machine learning tutorial"

Each variant is:

• Converted to an embedding
• Searched against the vector database
• Results are combined and deduplicated
• Best match for each file is kept
• Final results are ranked by similarity

Response Format

[
  {
    "file_id": 42,
    "filename": "ml_guide.pdf",
    "path": "/home/user/Documents/ml_guide.pdf",
    "device": "laptop",
    "device_ip": "192.168.1.10",
    "device_user": "john",
    "size": 1048576,
    "file_type": ".pdf",
    "similarity_score": 0.89,  // 0-1, higher = better match
    "last_modified_time": "2024-10-15T10:30:00",
    "matched_via": "expanded_query_2"  // Shows which query variant matched
  }
]

---

📊 Key Improvements

1. Unified API

• Single server instead of two separate ones
• Consistent naming conventions
• No duplicate endpoints

2. Query Expansion

# OLD (rag_server.py):
POST /search → basic semantic search

# NEW (unified_server.py):
POST /search/semantic → semantic search + query expansion
  - Generates multiple query variants
  - Searches with each variant
  - Combines and ranks results
  - Shows which variant matched

3. Better Search Options

• Keyword Search (GET /search/keyword): Fast filename matching
• Semantic Search (POST /search/semantic): Content-aware with query expansion

4. Enhanced File Operations

• Register (metadata only): POST /files/register
• Upload (with embedding): POST /files/upload
• Both support create and update operations
• Both return action: "created" or "updated"

5. Improved Responses

• All search results include similarity_score (0-1)
• Semantic search shows matched_via field
• Stats endpoint shows embedding model info

---

🚀 Migration Guide

For Existing Clients

1. Keyword Search (no changes needed):

   # OLD: GET /files/search?query=*.txt
   # NEW: GET /search/keyword?query=*.txt

2. Semantic Search (with new features):

   # OLD:
   curl -X POST http://localhost:8000/search \
     -H 'Content-Type: application/json' \
     -d '{"query": "python code", "top_k": 5}'
   
   # NEW (with query expansion):
   curl -X POST http://localhost:8000/search/semantic \
     -H 'Content-Type: application/json' \
     -d '{
       "query": "python code",
       "top_k": 5,
       "use_query_expansion": true,
       "expansion_count": 3
     }'

3. File Upload (new path):

   # OLD: POST /upload
   # NEW: POST /files/upload

---

🎯 Performance Comparison

Feature	main.py	rag_server.py	unified_server.py
Keyword Search	✅	❌	✅
Semantic Search	❌	✅	✅ + Query Expansion
File Registration (metadata only)	✅	❌	✅
File Upload (with embedding)	❌	✅	✅
Update Support	✅	❌	✅
Embedding Deletion	❌	❌	✅
Query Expansion	❌	❌	✅ NEW!
Local Model (no API key)	N/A	❌ (was using APIs)	✅

---

💡 Usage Examples

Example 1: Register File Metadata (No Upload)

curl -X POST http://localhost:8000/files/register \
  -H 'Content-Type: application/json' \
  -d '{
    "file_name": "report.pdf",
    "absolute_path": "/home/user/report.pdf",
    "device": "laptop",
    "device_ip": "192.168.1.10",
    "device_user": "john",
    "last_modified_time": "2024-10-19T12:00:00",
    "size": 2048000,
    "file_type": ".pdf"
  }'

Example 2: Upload File with Embedding

curl -X POST http://localhost:8000/files/upload \
  -F "file=@document.txt" \
  -F "device=laptop" \
  -F "device_ip=192.168.1.10" \
  -F "device_user=john" \
  -F "absolute_path=/home/user/document.txt"

Example 3: Semantic Search with Query Expansion

curl -X POST http://localhost:8000/search/semantic \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "budget analysis",
    "top_k": 5,
    "use_query_expansion": true,
    "expansion_count": 3
  }'

Example 4: Keyword Search (Fast)

curl -X GET 'http://localhost:8000/search/keyword?query=*.xlsx'

---

🔧 Configuration

All settings in unified_server.py:

EMBEDDING_DIM = 384  # all-MiniLM-L6-v2 (local model)
DB_PATH = "file_records.db"  # SQLite database
VECTOR_DB_PATH = "vectors.db"  # Vector embeddings
FAISS_INDEX_PATH = "faiss.index"  # FAISS index

---

✅ Next Steps

1. Stop old servers (if running):

   # Stop main.py or rag_server.py

2. Start unified server:

   cd /data/qyu/projects/dubhacks1/backend
   python unified_server.py

3. Test query expansion:

   curl -X POST http://localhost:8000/search/semantic \
     -H 'Content-Type: application/json' \
     -d '{"query": "test", "top_k": 5, "use_query_expansion": true}'

4. Check stats:

   curl http://localhost:8000/stats

[AndyJLi0]

Closing, changes already in master