Add comprehensive integration tests and improve project structure

[CameronScarpati]

Summary

This PR adds a complete end-to-end integration test suite covering the full volatility surface pipeline, improves project documentation and structure, and refactors dashboard components for better code reuse.

Key Changes

Testing & Quality Assurance

• New integration test suite (tests/test_integration.py): 358 lines covering the complete pipeline from synthetic data generation through IV extraction, SVI calibration, arbitrage diagnostics, and surface queries
  • Tests IV recovery accuracy from Black-Scholes prices
  • Validates SVI fit quality (R² > 0.95, RMSE < 0.01)
  • Verifies arbitrage-free fitting via Durrleman conditions
  • End-to-end build_surface() pipeline validation
  • Dashboard data structure compatibility checks
  • BS ↔ IV roundtrip consistency across strikes and expiries
• Shared test fixtures (tests/conftest.py): Centralized synthetic data generation and market parameters (SPOT, RISK_FREE, DIV_YIELD) used across all test modules
• CI/CD pipeline (.github/workflows/ci.yml): GitHub Actions workflow testing Python 3.10–3.12 with linting and caching

Documentation

• Comprehensive README.md: 177 lines covering architecture diagram, methodology (IV extraction, SVI parameterization, no-arbitrage enforcement), quick start guide, project structure, and references to Gatheral (2004) and Durrleman (2005)
• Docstring improvements: Added module-level docstrings and __all__ exports to all source modules for clarity

Code Organization & Refactoring

• Dashboard helper module (dashboard/components/helpers.py): Consolidates repeated computation patterns (forward price, log-moneyness, fitted IV from SVI) to eliminate duplication across components
• Component imports (dashboard/components/__init__.py): Centralized exports for cleaner imports in app.py
• Module exports: Added __all__ lists to src/ modules (iv_engine.py, svi_fitter.py, arbitrage.py, data_loader.py, surface.py) and src/__init__.py for public API clarity
• Dashboard refactoring: Updated residual_heatmap.py and term_structure.py to use the new compute_chain_fitted_iv() helper, reducing code duplication by ~30 lines
• Import ordering: Fixed import order consistency across modules (stdlib → third-party → local)

Data & Scripts

• Enhanced download script (data/download.py): Added --synthetic flag, improved output formatting, integrated full pipeline execution with surface building and diagnostics
• Synthetic data caching: Added data/spy_options.parquet to support fast dashboard reload
• Script improvements (scripts/generate_synthetic_data.py): Fixed import ordering

Configuration

• Enhanced pyproject.toml:
  • Added metadata (readme, license, authors, keywords, classifiers)
  • Defined optional dev dependencies (pytest, ruff)
  • Added pytest markers for slow tests
  • Configured ruff linter with comprehensive rules and isort settings
  • Added project URLs

Minor Fixes

• Fixed type hints and linting issues across multiple files
• Removed unused imports
• Improved code consistency (e.g., int(round(...)) → round(...))
• Updated .gitignore to allow parquet caching while excluding CSVs

Notable Implementation Details

• Synthetic IV model in conftest.py matches the realistic SPY-like characteristics used in generate_synthetic_data.py, ensuring test data consistency
• Integration tests use @pytest.mark.slow to allow selective execution with -m "not slow"
• Dashboard components now share a single source of truth for fitted IV computation via helpers.py, reducing maintenance burden
• Full pipeline validation includes checks for data structure compatibility with all dashboard components (3D surface, smile slices, residual heatmap, arbitrage diagnostics, term structure)

https://claude.ai/code/session_014Cap2tLzWHDwrx9GrfGJE3