LangChain NLP Workflow

A comprehensive enterprise-ready NLP workflow system that chains together document processing, summarization, and question-answering using LangChain and Large Language Models.

🎯 Project Overview

This project demonstrates a complete NLP pipeline that:

1. Ingests and processes documents from various formats (TXT, PDF, DOCX)
2. Generates intelligent summaries using LLMs
3. Enables question answering with source attribution
4. Orchestrates workflows for enterprise applications

Perfect for demonstrating advanced LangChain skills for the Techolution AI Intern (Gen AI) position.

🌟 Key Features

Core Capabilities

• Multi-format Document Processing: Supports TXT, PDF, and DOCX files
• Intelligent Text Chunking: Configurable chunk sizes and overlap
• Vector Database Integration: Uses ChromaDB for efficient similarity search
• LLM-powered Summarization: Map-reduce and other summarization strategies
• Contextual Q&A: Question answering with source document attribution
• Workflow Orchestration: Seamless integration of multiple NLP tasks

Enterprise-Ready Features

• Configurable Settings: YAML-based configuration management
• Multiple Interfaces: CLI, Web UI (Streamlit), and Jupyter notebooks
• Comprehensive Logging: Detailed logging for debugging and monitoring
• Error Handling: Robust error handling with meaningful messages
• Testing Suite: Unit tests for core components
• Modular Architecture: Easy to extend and customize

🏗️ Project Structure

langchain-nlp-workflow/
├── src/
│   ├── workflows/
│   │   ├── __init__.py
│   │   └── nlp_workflow.py          # Main workflow orchestrator
│   └── utils/
│       ├── __init__.py
│       ├── config.py                # Configuration management
│       ├── document_processor.py    # Document loading and processing
│       └── vector_store.py          # Vector database operations
├── config/
│   └── config.yaml                  # Main configuration file
├── data/
│   └── sample_docs/                 # Sample documents for testing
│       ├── ai_healthcare.txt
│       ├── renewable_energy.txt
│       └── ml_finance.txt
├── tests/
│   └── test_workflow.py             # Unit tests
├── notebooks/
│   └── demo_workflow.ipynb          # Interactive Jupyter demo
├── docs/                            # Documentation
├── main.py                          # CLI interface
├── streamlit_app.py                 # Web interface
├── requirements.txt                 # Python dependencies
├── .env.template                    # Environment variables template
└── README.md                        # This file

🚀 Quick Start

1. Installation

# Clone or extract the project
cd langchain-nlp-workflow

# Install dependencies
pip install -r requirements.txt

2. Configuration

# Copy environment template
cp .env.template .env

# Edit .env file and add your OpenAI API key
OPENAI_API_KEY=your-openai-api-key-here

3. Run the Workflow

Option A: Command Line Interface

# Basic usage with sample documents
python main.py -s data/sample_docs

# With custom questions
python main.py -s data/sample_docs -q "What are the main topics?" "What are the key challenges?"

# Summarization only
python main.py -s data/sample_docs --summarize-only

Option B: Web Interface

# Launch Streamlit app
streamlit run streamlit_app.py

Option C: Jupyter Notebook

# Start Jupyter and open the demo notebook
jupyter notebook notebooks/demo_workflow.ipynb

📋 Usage Examples

Programmatic Usage

from workflows import NLPWorkflowOrchestrator

# Initialize workflow
workflow = NLPWorkflowOrchestrator()

# Run complete workflow
questions = [
    "What are the main topics discussed?",
    "What are the key challenges mentioned?"
]

results = workflow.run_complete_workflow(
    source_path="data/sample_docs",
    questions=questions
)

# Access results
print("Summary:", results['steps']['summarization']['summary'])
for qa in results['steps']['qa_results']:
    print(f"Q: {qa['question']}")
    print(f"A: {qa['answer']}")

Individual Components

# Document processing only
from utils import DocumentProcessor

processor = DocumentProcessor(chunk_size=1000, chunk_overlap=200)
documents = processor.process_directory("data/sample_docs")

# Vector store operations
from utils import VectorStoreManager

vector_manager = VectorStoreManager()
vector_store = vector_manager.create_vector_store(documents)
similar_docs = vector_manager.similarity_search("AI in healthcare")

⚙️ Configuration

Main Configuration (config/config.yaml)

models:
  default_llm: "gpt-3.5-turbo"
  embedding_model: "text-embedding-ada-002"

workflows:
  document_processing:
    chunk_size: 1000
    chunk_overlap: 200

  summarization:
    max_tokens: 500
    temperature: 0.3

  question_answering:
    max_tokens: 300
    temperature: 0.1
    top_k: 5

vector_store:
  type: "chroma"
  persist_directory: "./chroma_db"

Environment Variables

OPENAI_API_KEY=your-openai-api-key-here
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=your-langsmith-api-key-here

🧪 Testing

Run the test suite:

# Run all tests
python -m pytest tests/

# Run specific test file
python tests/test_workflow.py

# Run with verbose output
python -m pytest tests/ -v

🌐 Web Interface Features

The Streamlit web interface provides:

• File Upload: Drag-and-drop document upload
• Real-time Processing: Live progress updates
• Interactive Q&A: Custom question input
• Results Visualization: Formatted results display
• Export Options: Download results as JSON or TXT

Access at: http://localhost:8501

📊 Key Metrics & Performance

• Document Processing: Handles 100+ documents efficiently
• Chunking Strategy: Optimized for 1000-token chunks with 200-token overlap
• Response Time: Sub-second query responses with proper caching
• Memory Usage: Efficient vector storage with ChromaDB
• Scalability: Modular design supports horizontal scaling

🔧 Customization Options

Custom LLM Models

# Use different models
workflow = NLPWorkflowOrchestrator()
workflow.llm = ChatOpenAI(model_name="gpt-4", temperature=0.2)

Custom Embeddings

# Use custom embedding models
from langchain.embeddings import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vector_manager = VectorStoreManager(embeddings=embeddings)

Custom Prompts

# Customize summarization prompts
from langchain.prompts import PromptTemplate

custom_prompt = PromptTemplate(
    template="Summarize this text focusing on technical details: {text}",
    input_variables=["text"]
)

🚀 Enterprise Integration

API Deployment

# FastAPI wrapper example
from fastapi import FastAPI
import uvicorn

app = FastAPI()
workflow = NLPWorkflowOrchestrator()

@app.post("/process")
async def process_documents(request: ProcessRequest):
    return workflow.run_complete_workflow(
        request.source_path, 
        request.questions
    )

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Docker Deployment

FROM python:3.9-slim
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["streamlit", "run", "streamlit_app.py"]

📈 Skills Demonstrated

This project showcases key skills relevant to the Techolution AI Intern position:

Technical Skills

• LangChain Framework: Advanced usage of chains, agents, and tools
• LLM Integration: OpenAI GPT models, prompt engineering
• Vector Databases: ChromaDB for similarity search and retrieval
• Document Processing: Multi-format file handling and text chunking
• Python Development: Clean, modular, well-documented code
• Testing: Unit tests and integration testing

Enterprise Skills

• Configuration Management: YAML-based settings, environment variables
• Error Handling: Comprehensive error handling and logging
• User Interfaces: CLI, web interface, and notebook interfaces
• Documentation: Comprehensive README and inline documentation
• Workflow Orchestration: Chaining multiple NLP tasks seamlessly

GenAI Skills

• Fine-tuning Concepts: Architecture ready for model fine-tuning
• RAG Implementation: Retrieval-Augmented Generation patterns
• Prompt Engineering: Optimized prompts for different tasks
• Model Versioning: Structure supports model versioning workflows

🤝 Contributing

This project follows enterprise development practices:

1. Code Style: PEP 8 compliant with type hints
2. Documentation: Comprehensive docstrings and comments
3. Testing: Unit tests for core functionality
4. Modularity: Loosely coupled, highly cohesive design
5. Configuration: Externalized configuration management

📝 License

This project is created for demonstration purposes for the Techolution AI Intern (Gen AI) position.

🙋‍♂️ Support

For questions about this project or the implementation:

1. Check the inline documentation and docstrings
2. Review the Jupyter notebook for detailed examples
3. Run the test suite to understand component behavior
4. Explore the configuration options for customization

---

🎯 Alignment with Techolution Requirements

This project directly addresses the key requirements from the job description:

Mandatory Skills ✅

• Fine-tuning LLMs: Architecture supports fine-tuning workflows
• NLP & LLMs: Comprehensive use of Falcon, LLama compatible frameworks
• LangChain: Advanced LangChain implementation with chains and agents
• Vector Databases: ChromaDB integration with similarity search
• PyTorch/TensorFlow: Ready for deep learning model integration
• Python Programming: Clean, professional Python codebase

Preferred Skills ✅

• Model Versioning & Deployment: Structured for enterprise deployment
• Cloud Deployment: Ready for containerization and cloud deployment
• Machine Learning: ML workflow orchestration and management

Job Responsibilities ✅

• Workflow Orchestration: Complete NLP pipeline management
• Cross-functional Collaboration: Modular design for team integration
• Technical Communication: Comprehensive documentation and examples
• Quality Standards: Professional code with testing and error handling

This project demonstrates the ability to build production-ready AI systems that align with Techolution's vision of "Real World AI" and "innovation done right."