AI Document Processor - Enterprise Edition

🎯 Overview

The AI Document Processor is an enterprise-grade, cloud-native document processing platform that combines Azure's AI services with intelligent caching, analytics, and a modern web interface. It transforms documents into structured data using Azure Document Intelligence and OpenAI GPT models.

✨ Key Features

🚀 Live & Batch Processing - Real-time and queue-based document processing
🧠 AI-Powered Analysis - Azure Document Intelligence + OpenAI GPT integration
📊 Enterprise Analytics - Comprehensive tracking, metrics, and cost analysis
🔄 Smart Caching - Multi-tier caching system reducing costs by up to 80%
🎨 Modern Web Interface - Advanced HTML5 interface with real-time monitoring
🔐 Enterprise Security - Managed identity, IP restrictions, and secure authentication
📈 Dynamic Prompts - JSON-based prompt configuration system
🏗️ Azure Native - Fully integrated with Azure Container Apps and services

🏗️ Architecture

graph TB
    subgraph "Frontend Layer"
        UI[Modern Web Interface]
        API_CLIENT[API Client]
    end
    
    subgraph "API Layer"
        FASTAPI[FastAPI Server]
        ROUTES[API Routes]
        MIDDLEWARE[Security Middleware]
    end
    
    subgraph "Processing Layer"
        DOC_PROC[Document Processor]
        PROMPT_MGR[Prompt Manager]
        CACHE[Smart Cache]
        ANALYTICS[Analytics Manager]
    end
    
    subgraph "Azure Services"
        CONTAINER_APP[Container Apps]
        BLOB[Blob Storage]
        DI[Document Intelligence]
        OPENAI[Azure OpenAI]
        ACR[Container Registry]
    end
    
    subgraph "Storage Containers"
        BRONZE[Bronze - Raw Files]
        SILVER[Silver - Processed]
        GOLD[Gold - Archive]
        CACHE_STORE[Cache Storage]
        ANALYTICS_STORE[Analytics Data]
    end
    
    UI --> API_CLIENT
    API_CLIENT --> FASTAPI
    FASTAPI --> ROUTES
    ROUTES --> MIDDLEWARE
    MIDDLEWARE --> DOC_PROC
    DOC_PROC --> PROMPT_MGR
    DOC_PROC --> CACHE
    DOC_PROC --> ANALYTICS
    
    CONTAINER_APP --> FASTAPI
    DOC_PROC --> DI
    DOC_PROC --> OPENAI
    DOC_PROC --> BLOB
    
    BLOB --> BRONZE
    BLOB --> SILVER
    BLOB --> GOLD
    CACHE --> CACHE_STORE
    ANALYTICS --> ANALYTICS_STORE

🚀 Quick Start

Prerequisites

Azure Subscription with appropriate permissions
Azure CLI installed and configured
PowerShell 5.1+ (for deployment scripts)
Git for repository management

1. Clone Repository

git clone <your-repository-url>
cd ai-document-processor-personal-main

2. Configure Environment

# Copy environment template
Copy-Item env.example .env

# Edit .env with your Azure details
# Key variables:
# - RESOURCE_GROUP
# - AZURE_OPENAI_NAME
# - AZURE_DOCUMENT_INTELLIGENCE_NAME
# - AZURE_STORAGE_ACCOUNT

3. Deploy Infrastructure (Phase 1)

# Full infrastructure deployment - creates all Azure resources
.\infrastructure\provision-env-modular.ps1 -ResourceGroupName "your-rg-name"

# Or deploy step by step if needed
.\infrastructure\provision-env-modular.ps1 -ResourceGroupName "your-rg-name" -StartFromStep 1

4. Deploy Application (Phase 2)

# Build and deploy the actual application container
.\deploy-container-app-final.ps1 -ResourceGroupName "your-rg-name"

# Or start from specific step if needed
.\deploy-container-app-final.ps1 -ResourceGroupName "your-rg-name" -StartFromStep 3

5. Access Application

Your application will be available at:

https://app-<your-resource-name>.agreeablerock-<hash>.eastus2.azurecontainerapps.io

📋 Features Deep Dive

🔄 Processing Workflows

Live Processing

Real-time document processing
Immediate results via REST API
Job tracking with unique job IDs
Progress monitoring through web interface

# Example: Process document via API
curl -X POST "https://your-app-url/process-optimized" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@document.pdf" \
  -F "prompt=Extract key information from this document"

Batch Processing

Queue-based processing for multiple documents
Background processing with status tracking
Bulk operations with progress monitoring
Result aggregation and reporting

🧠 AI Integration

Azure Document Intelligence

Multi-format support (PDF, DOC, DOCX, images)
Text extraction with layout preservation
Table and form recognition
Custom model support

Azure OpenAI Integration

GPT-4 and GPT-3.5-turbo support
Custom prompts with dynamic configuration
Token management and cost optimization
Structured output generation

📊 Enterprise Analytics

Real-time Metrics

Processing statistics (documents, tokens, costs)
Cache performance (hit rates, savings)
System health monitoring
Active job tracking

Historical Analytics

Cost analysis over time
Performance trends and optimization insights
Usage patterns and capacity planning
Compliance reporting

🔄 Smart Caching System

Multi-tier Architecture

In-memory cache (fastest access)
Blob storage cache (persistent, low-cost)
LRU eviction policy
TTL-based expiration

Cost Optimization

Up to 80% reduction in API calls
Automatic cache warming
Intelligent cache invalidation
Performance metrics tracking

🎨 Modern Web Interface

Dashboard Features

Live Processing tab with drag-drop upload
Batch Processing queue management
Monitoring with real-time metrics
Settings and configuration management
API Integration testing tools

User Experience

Responsive design for all devices
Real-time updates via WebSocket-like polling
Progress indicators and status tracking
Error handling with user-friendly messages

🔧 API Reference

Core Endpoints

Document Processing

POST /process-optimized
Content-Type: multipart/form-data

Parameters:
- file: Document file (PDF, DOC, DOCX, TXT, images)
- prompt: Processing instructions (optional)
- workflow: "live" or "batch" (default: "live")

Batch Operations

POST /batch/queue-document
GET /batch/queue-summary
GET /batch/job-status/{job_id}
DELETE /batch/cancel-job/{job_id}

Analytics

GET /analytics/real-time
GET /analytics/cache-performance  
GET /analytics/historical
GET /analytics/cost-analysis

System

GET /health
GET /cache/stats
GET /cache/clear

Response Format

{
  "success": true,
  "job_id": "uuid-string",
  "processing_time_ms": 1500,
  "extracted_text": "Document content...",
  "analysis": "AI-generated analysis...",
  "metadata": {
    "file_size": 1024000,
    "pages": 5,
    "tokens_used": 150,
    "cache_hit": false,
    "cost_estimate": 0.0023
  },
  "storage_info": {
    "bronze_container": "bronze/uuid/filename.pdf",
    "silver_container": "silver/uuid/analysis.json",
    "gold_container": "gold/uuid/filename.pdf"
  }
}

🚀 Two-Phase Deployment Architecture

The deployment process is split into two distinct phases for better modularity and troubleshooting:

Phase 1: Infrastructure Provisioning (`provision-env-modular.ps1`)

Purpose: Creates all Azure resources and configures the foundation Location: infrastructure/provision-env-modular.ps1

What it does:

✅ Creates Resource Group
✅ Deploys Storage Account with containers (bronze, silver, gold)
✅ Creates Cognitive Services (OpenAI + Document Intelligence)
✅ Deploys GPT-4.1-mini model
✅ Configures custom subdomains for managed identity
✅ Sets up Key Vault with secure API key storage
✅ Creates Container Registry (ACR)
✅ Creates Container App Environment with placeholder app

Phase 2: Application Deployment (`deploy-container-app-final.ps1`)

Purpose: Builds and deploys your actual application code Location: Root directory deploy-container-app-final.ps1

What it does:

✅ Builds Docker image using ACR build service (no Docker Desktop needed)
✅ Updates Container App with your application image
✅ Configures managed identity authentication
✅ Sets up role assignments for Azure services
✅ Creates required blob storage containers
✅ Verifies deployment health and provides URLs

Why Two Scripts?

Aspect	Phase 1 (Infrastructure)	Phase 2 (Application)
Frequency	Once per environment	Every code update
Duration	10-15 minutes	3-5 minutes
Purpose	Foundation setup	Code deployment
Failures	Resource conflicts, permissions	Build issues, config errors
Restart	Modular (any step 1-8)	Modular (any step 1-6)

🏗️ Infrastructure Components

Azure Resources Created (Phase 1)

Resource	Purpose	Configuration
Container App Environment	Hosting platform	Auto-scaling, HTTPS
Container App	Application runtime	0-3 replicas, managed identity
Container Registry	Image storage	Basic tier, admin enabled
Storage Account	Document & cache storage	5 containers, lifecycle policies
Azure OpenAI	LLM processing	GPT-4.1-mini, custom subdomain
Document Intelligence	Text extraction	Custom subdomain
Key Vault	Secure secret storage	API keys, access policies
Managed Identity	Secure authentication	Cognitive Services & Storage access

Storage Container Structure

Storage Account
├── bronze/          # Raw uploaded documents
├── silver/          # Processed results (JSON)
├── gold/            # Final archived documents
├── cache/           # Smart cache storage
└── analytics/       # Analytics and metrics data

🔐 Security Features

Authentication & Authorization

Managed Identity for Azure service access
IP-based restrictions for enhanced security
API key authentication for external access
Azure AD integration ready

Data Protection

Secure file upload with validation
Encrypted storage in Azure Blob
SAS token based access control
No sensitive data in logs

Network Security

Container App with HTTPS only
Private networking options available
CORS configuration for web security
Input sanitization and validation

🚀 Deployment Guide

Detailed Deployment Guide

Phase 1: Infrastructure Provisioning (`provision-env-modular.ps1`)

Run once per environment - Sets up all Azure resources:

# Full infrastructure deployment
.\infrastructure\provision-env-modular.ps1 -ResourceGroupName "RG-DOC-Process-WB-P"

# Or step-by-step deployment
.\infrastructure\provision-env-modular.ps1 -ResourceGroupName "RG-DOC-Process-WB-P" -StartFromStep 1

Steps performed:

Resource Group - Creates or validates resource group
Storage Account - Creates storage with bronze/silver/gold containers
Cognitive Services - Deploys OpenAI and Document Intelligence
GPT Model - Deploys GPT-4.1-mini model
Service Endpoints - Retrieves service endpoints securely
Key Vault - Creates Key Vault with API keys (secure storage)
Container Registry - Creates ACR for Docker images
Container App - Creates Container App Environment with placeholder

Phase 2: Application Deployment (`deploy-container-app-final.ps1`)

Run for every code update - Builds and deploys your application:

# Full application deployment
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P"

# Skip Docker build if unchanged
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P" -SkipImageBuild

# Start from specific step
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P" -StartFromStep 3

Steps performed:

Container Environment - Validates Container App Environment exists
Docker Build - Builds image using ACR build service (no Docker Desktop)
ACR Credentials - Retrieves registry credentials
Container App - Updates app with new image and environment variables
Storage Containers - Creates required blob containers (bronze, silver, gold, prompts)
Health Check - Verifies deployment and provides URLs

🔄 Deployment Workflows

First-Time Deployment

# Step 1: Infrastructure (run once)
.\infrastructure\provision-env-modular.ps1 -ResourceGroupName "RG-DOC-Process-WB-P"

# Step 2: Application (initial deployment)
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P"

Code Updates (Daily Development)

# Only run Phase 2 for code changes
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P"

# Skip Docker build if no code changes
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P" -SkipImageBuild

Troubleshooting Deployments

# Phase 1: Restart from specific infrastructure step
.\infrastructure\provision-env-modular.ps1 -ResourceGroupName "RG-DOC-Process-WB-P" -StartFromStep 5

# Phase 2: Restart from specific application step  
.\deploy-container-app-final.ps1 -ResourceGroupName "RG-DOC-Process-WB-P" -StartFromStep 4

When to Use Each Script

Scenario	Script to Use	Reason
New Environment	Phase 1 → Phase 2	Need all Azure resources
Code Changes	Phase 2 only	Infrastructure already exists
Config Changes	Phase 2 only	Environment variables update
Resource Issues	Phase 1 (specific step)	Fix infrastructure problems
App Not Starting	Phase 2 (specific step)	Fix deployment issues
New Features	Phase 2 only	Code deployment sufficient

Troubleshooting Deployment

Common Issues

Resource Group Not Found

# Create resource group first
az group create --name "your-rg" --location "eastus2"

Managed Identity Permissions

# Check role assignments
az role assignment list --assignee <managed-identity-id>

Container App Not Starting

# Check logs
az containerapp logs show --name "your-app" --resource-group "your-rg"

Custom Subdomain Issues

# Update cognitive services
az cognitiveservices account update --name "your-openai" --resource-group "your-rg" --custom-domain "your-openai"

📊 Monitoring & Operations

Health Monitoring

Health check endpoint (/health)
System metrics tracking
Performance monitoring built-in
Error logging with structured format

Analytics Dashboard

Real-time processing metrics
Cost tracking and optimization
Cache performance analysis
Historical trends and reporting

Maintenance

Automatic scaling based on demand
Cache cleanup and optimization
Log rotation and archival
Performance tuning recommendations

🔧 Configuration

Environment Variables

# Azure Services
AZURE_OPENAI_ENDPOINT=https://your-openai.openai.azure.com/
AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT=https://your-di.cognitiveservices.azure.com/
AZURE_STORAGE_ACCOUNT=yourstorageaccount

# Application Settings
API_DEBUG=false
LOG_LEVEL=INFO
MAX_FILE_SIZE_MB=50
CACHE_TTL_SECONDS=3600

# Resource Names (for deployment)
RESOURCE_GROUP=ai-doc-proc-rg
AZURE_CONTAINER_APP=app-ai-doc-proc-202510
AZURE_CONTAINER_REGISTRY=acraidocproc202510

Prompt Configuration

The system uses a dynamic JSON-based prompt configuration system:

{
  "version": "1.0.0",
  "workflow_types": {
    "live": {
      "name": "Live Processing",
      "max_tokens": 2000
    },
    "batch": {
      "name": "Batch Processing",
      "max_tokens": 8000
    }
  },
  "document_prompts": {
    "invoice": {
      "live": {
        "minimal": "Extract key invoice data...",
        "general": "Extract invoice details...",
        "detailed": "Extract comprehensive invoice information..."
      }
    }
  }
}

🧪 Testing

API Testing

# Health check
curl https://your-app-url/health

# Process document
curl -X POST "https://your-app-url/process-optimized" \
  -F "file=@test-document.pdf" \
  -F "prompt=Analyze this document"

# Get analytics
curl https://your-app-url/analytics/real-time

Web Interface Testing

Navigate to your Container App URL
Use the Live Processing tab for immediate testing
Try Batch Processing for multiple documents
Monitor results in the Monitoring tab

Load Testing

# Use the provided test scripts
python test_api_comprehensive.py
python test_batch_processing.py

📈 Performance & Scaling

Auto-scaling Configuration

Minimum replicas: 0 (cost-effective)
Maximum replicas: 3
Scale trigger: HTTP concurrency > 10
Scale down: Automatic when idle

Performance Optimization

Smart caching reduces API calls by 80%
Async processing for better throughput
Connection pooling for Azure services
Efficient memory management

Cost Optimization

Pay-per-use scaling (scales to zero)
Cache-first strategy for repeated requests
Token optimization for LLM calls
Storage lifecycle policies

🔄 CI/CD Pipeline

GitHub Actions (Optional)

name: Deploy AI Document Processor
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Deploy to Azure
        run: |
          az acr build --registry $ACR_NAME --image ai-document-processor:$GITHUB_SHA .
          az containerapp update --name $APP_NAME --image $ACR_NAME.azurecr.io/ai-document-processor:$GITHUB_SHA

Manual Deployment

# Standard deployment process
.\deploy-container-app-final.ps1 -ResourceGroupName "your-rg"

🤝 Contributing

Development Setup

Fork the repository
Create feature branch
Install dependencies: pip install -r requirements.txt
Run locally: python -m api.server
Test changes thoroughly
Submit pull request

Code Standards

Python 3.9+ compatibility
Type hints for all functions
Async/await for I/O operations
Comprehensive error handling
Unit tests for new features

📚 Documentation

Customer Demo Guide - Complete demo walkthrough
Project Summary - Technical architecture overview
Infrastructure Guide - Deployment details
API Documentation - Interactive API docs

🆘 Support & Troubleshooting

Common Issues

1. Container App Not Accessible

Check if custom domain is configured
Verify managed identity permissions
Review container logs for errors

2. Processing Errors

Validate input file formats
Check Azure service quotas
Monitor token usage limits

3. Cache Issues

Clear cache via /cache/clear endpoint
Check blob storage permissions
Review cache configuration

Getting Help

Check logs in Azure Portal
Review troubleshooting section above
Test with provided sample documents
Create issue with detailed error information

Performance Monitoring

# Monitor system health
curl https://your-app-url/health

# Check cache performance
curl https://your-app-url/cache/stats

# View analytics
curl https://your-app-url/analytics/real-time

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🏆 Enterprise Features Summary

✅ Production Ready - Full enterprise deployment with monitoring
✅ Cost Optimized - Smart caching and auto-scaling
✅ Secure - Managed identity and IP restrictions
✅ Scalable - Container Apps with automatic scaling
✅ Observable - Comprehensive analytics and monitoring
✅ Maintainable - Modular architecture and documentation
✅ User Friendly - Modern web interface with real-time updates
✅ API First - Complete REST API with OpenAPI documentation

🚀 Ready for Enterprise Deployment!

This AI Document Processor represents a complete, production-ready solution for enterprise document processing with Azure's most advanced AI services.

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
Installation		Installation
api		api
config		config
core		core
infrastructure		infrastructure
scripts		scripts
static		static
tests		tests
CUSTOMER_DEMO_GUIDE.md		CUSTOMER_DEMO_GUIDE.md
Dockerfile		Dockerfile
PROJECT_SUMMARY.md		PROJECT_SUMMARY.md
README.md		README.md
azure-pipelines.yml		azure-pipelines.yml
comprehensive_test_document.pdf		comprehensive_test_document.pdf
deploy-container-app-final.ps1		deploy-container-app-final.ps1
enterprise-prompt-config.json		enterprise-prompt-config.json
env.example		env.example
env.local		env.local
requirements.txt		requirements.txt
startup.sh		startup.sh
upload-prompt-config.py		upload-prompt-config.py

markremmey/ai-doc-processor

Folders and files

Latest commit

History

Repository files navigation

AI Document Processor - Enterprise Edition

🎯 Overview

✨ Key Features

🏗️ Architecture

🚀 Quick Start

Prerequisites

1. Clone Repository

2. Configure Environment

3. Deploy Infrastructure (Phase 1)

4. Deploy Application (Phase 2)

5. Access Application

📋 Features Deep Dive

🔄 Processing Workflows

Live Processing

Batch Processing

🧠 AI Integration

Azure Document Intelligence

Azure OpenAI Integration

📊 Enterprise Analytics

Real-time Metrics

Historical Analytics

🔄 Smart Caching System

Multi-tier Architecture

Cost Optimization

🎨 Modern Web Interface

Dashboard Features

User Experience

🔧 API Reference

Core Endpoints

Document Processing

Batch Operations

Analytics

System

Response Format

🚀 Two-Phase Deployment Architecture

Phase 1: Infrastructure Provisioning (provision-env-modular.ps1)

Phase 2: Application Deployment (deploy-container-app-final.ps1)

Why Two Scripts?

🏗️ Infrastructure Components

Azure Resources Created (Phase 1)

Storage Container Structure

🔐 Security Features

Authentication & Authorization

Data Protection

Network Security

🚀 Deployment Guide

Detailed Deployment Guide

Phase 1: Infrastructure Provisioning (provision-env-modular.ps1)

Phase 2: Application Deployment (deploy-container-app-final.ps1)

🔄 Deployment Workflows

First-Time Deployment

Code Updates (Daily Development)

Troubleshooting Deployments

When to Use Each Script

Troubleshooting Deployment

Common Issues

📊 Monitoring & Operations

Health Monitoring

Analytics Dashboard

Maintenance

🔧 Configuration

Environment Variables

Prompt Configuration

🧪 Testing

API Testing

Web Interface Testing

Load Testing

📈 Performance & Scaling

Auto-scaling Configuration

Performance Optimization

Cost Optimization

🔄 CI/CD Pipeline

GitHub Actions (Optional)

Manual Deployment

🤝 Contributing

Phase 1: Infrastructure Provisioning (`provision-env-modular.ps1`)

Phase 2: Application Deployment (`deploy-container-app-final.ps1`)

Phase 1: Infrastructure Provisioning (`provision-env-modular.ps1`)

Phase 2: Application Deployment (`deploy-container-app-final.ps1`)

Packages