Refactor 3D surface rendering and improve IV filtering robustness#11
Merged
CameronScarpati merged 17 commits intomainfrom Mar 13, 2026
Merged
Refactor 3D surface rendering and improve IV filtering robustness#11CameronScarpati merged 17 commits intomainfrom
CameronScarpati merged 17 commits intomainfrom
Conversation
- Add centered full-page screenshot section at the top (docs/screenshot.png) - Add Highlights summary table for quick scanning - Add Tech Stack section listing core technologies - Tighten language for a more professional tone throughout - Reorganize sections for better readability https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- Clamp local volatility values above 300% to NaN in the Dupire surface to eliminate numerical spikes from near-zero denominators at boundaries - Handle None spot price from yfinance fast_info for index tickers (SPX) by falling back to the last close from recent price history https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- Raise denominator threshold from 1e-8 to 0.01 to reject near-singular points where the Durrleman condition is barely satisfied - Lower local vol cap from 300% to 150% (equity local vol is typically 10-80%; anything above 150% is numerical noise) - Narrow k_grid from +/-0.35 to +/-0.20 log-moneyness to avoid deep OTM wing artifacts where SVI derivatives are unreliable https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
yfinance requires a caret prefix for index tickers. Users naturally type SPX, NDX, RUT, etc. without the caret, so map these automatically. https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- Apply normalized-convolution Gaussian smoothing (sigma 1,2) to remove jagged ridges from unevenly spaced finite differences across expiries - Raise denominator threshold from 0.01 to 0.05 for stricter rejection - Lower local vol cap from 150% to 80% (realistic equity range) - Narrow k_grid to +/-0.15 and resolution to 80 points - Fix z-axis range to 0-80% so the plot doesn't auto-scale to outliers https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- Add _select_expiries() that drops short-dated slices (< 22 days) and enforces minimum spacing between expiries to avoid noisy FD from closely-spaced weekly expiries with independently-calibrated SVI params - Raise denominator threshold to 0.1 for aggressive rejection of near-singular Durrleman points - Reject negative dw/dT (calendar-spread violation residuals) - Cap at 60% before and after smoothing (realistic equity range) - Heavier Gaussian smoothing sigma=(2,3) with stricter weight threshold - Narrow k_grid to +/-0.10 and fix z-axis range to 0-60% - Fix y-axis range on slice plot to match https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- Lower min expiry from 22 days to 10 days to include more short-dated - Reduce min expiry gap from 11 days to 7 days for denser T sampling - Lower denominator threshold from 0.1 to 0.05 to retain more valid pts - Raise pre-smooth cap from 60% to 100% (post-smooth still caps at 100%) - Reduce smoothing sigma from (2,3) to (1.5,2.5) and weight threshold from 0.5 to 0.3 so edges aren't aggressively NaN-ed - Widen k_grid back to +/-0.15 from +/-0.10 - Auto-scale z-axis to 98th percentile of data instead of fixed 60% https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
Replace noisy finite-difference dw/dT with analytical derivatives from cubic splines fitted to each SVI parameter across T. This eliminates the root cause of spikes and jagged ridges: independently-calibrated SVI params jumping between closely-spaced weekly expiries. - Add _smooth_svi_params() that fits UnivariateSpline to each of the 5 SVI parameters (a, b, rho, m, sigma) as smooth functions of T - Compute dw/dT analytically by differentiating the SVI formula w.r.t. T using the spline derivatives da/dT, db/dT, drho/dT, dm/dT, dsigma/dT - Evaluate on a regular 30-point T grid instead of raw expiry dates - Reduce Gaussian post-smoothing to light polish sigma=(0.8, 1.2) since the upstream signal is now clean - Auto-scale z-axis to 97th percentile of computed values https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
The core issue: live options chains include deep OTM options with extreme IVs (200%+) that blow up auto-scaled z-axes, and the full strike range creates sparse grids with tiny fragmented surface patches. surface_3d.py: - Narrow strike grid to +/-25% log-moneyness around spot instead of full chain range (400-900 for SPY) - Filter market IVs > 200% and < 1% as outliers - Cap fitted IVs from SVI wing extrapolation at 200% - Auto-scale z-axis to 2nd-98th percentile of data - Format IV colorbar as percentage residual_heatmap.py: - Filter out outlier IVs > 200% and residuals > 50 vol points - Restrict strikes to +/-25% log-moneyness around spot local_vol.py: - Already rewritten with spline-smoothed SVI parameters (previous commit) https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- Lower fitted/market IV cap from 200% to 80% — equity IV at ±15% moneyness should never exceed this; anything higher is SVI wing noise - Narrow strike grid from ±25% to ±15% log-moneyness around spot - Tighten residual filter from 50 to 10 vol points - Format z-axis as percentages for IV views https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
The visualization changes to surface_3d.py and residual_heatmap.py were breaking the synthetic data display. Reverted both to their original working state. Instead, fix the root cause: deep OTM options produce extreme IVs (200%+) from Newton-Raphson, which poison the SVI fit and make all downstream visualizations look bad. Added IV sanity filter in build_surface() that discards IVs > 150% before SVI fitting. https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
- local_vol.py: Replace spline-based approach with FD-based Dupire that works correctly for both synthetic (100% valid, 24-34% range) and live-like data (99%+ valid, 13-39% range). Add expiry selection to skip closely-spaced slices and light Gaussian post-smoothing. - surface_3d.py: Limit strike grid to ±30% log-moneyness around median forward to prevent SVI wing extrapolation blowup. Cap fitted IV at 150%. - residual_heatmap.py: Adaptive strike bucket sizing for different underlying price scales (SPY vs SPX). Verified programmatically on synthetic, live-like (27 weekly expiries), and wide-strike scenarios. All 130 existing tests pass. https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
Pipeline (surface.py): - Add moneyness filter: calls restricted to k > -0.15, puts to k < 0.15, both capped at |k| < 0.30. Removes deep ITM options with unreliable IVs. - Add per-slice MAD-based outlier removal to catch stale quotes and data errors that pass the moneyness filter. Display (surface_3d.py): - Tighten fitted IV cap from 1.5 to 0.80 to prevent SVI wing blowup. - Add z-axis clamping based on data percentiles for all view modes. Tested on real SPY data (241 options, 18 expiries) and synthetic data. Synthetic: all 9 slices preserved, 100% local vol valid. Real SPY: 4 clean slices, fitted IV capped at 0.77, residuals std=0.05. All 130 tests pass. https://claude.ai/code/session_01NaRAUNdn9rRfssmMNLL87z
…data The Market IV 3D surface was empty because sparse market IV points were scattered onto a dense grid (mostly NaN), which Plotly cannot render as a continuous surface. Now uses Scatter3d markers overlaid on a semi- transparent fitted surface, so every valid market IV point is visible. The Residual view similarly uses scatter points with a zero-plane reference instead of trying to surface-interpolate sparse residuals. Additional fixes: - Adaptive IV cap replaces hardcoded 0.80 ceiling (uses 95th pct × 1.5) - Adaptive z-axis range for Market IV includes actual market data range - Residual heatmap uses adaptive bucket count for sparse strike data - Moneyness filter preserves minimum 5 points per slice for SVI fitting - Outlier filter skips removal when it would leave < 5 points per slice - Lazy import for data_loader avoids requiring yfinance at import time All 118 existing tests pass. Validated with both synthetic placeholder data (280 market IV points, 280 residuals) and real SPY/SPX options data (38 market IV points, 38 residuals across 6 expiry slices). https://claude.ai/code/session_01ATapVU79KVy5UgFXtcnFUM
Select only out-of-the-money options (calls K>F, puts K<F) before IV extraction — standard practice on derivatives desks that significantly improves IV quality for live data. Also strengthen Dupire local vol Gaussian smoothing and widen the moneyness grid for better coverage. https://claude.ai/code/session_01ATapVU79KVy5UgFXtcnFUM
CameronScarpati
force-pushed
the
claude/fix-market-iv-graphs-xFrAk
branch
from
689182d to
9a7d1bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR refactors the 3D surface visualization to separate concerns between fitted and market IV rendering, improves IV filtering logic to be more adaptive, and enhances data validation throughout the pipeline.
Key Changes
Surface Rendering (
dashboard/components/surface_3d.py)_build_surface_grid()into_build_fitted_surface()(for SVI-fitted IV grid) and_get_market_iv_points()(for extracting market observations)_adaptive_iv_cap()to compute IV caps based on the 95th percentile of fitted values rather than a hard 0.80 cap, preventing extreme wing artifacts while preserving legitimate high-IV regimesrender_surface_3d()into three specialized functions:_render_fitted_surface(): Shows the SVI-fitted surface alone_render_market_iv(): Overlays market IV scatter points on a semi-transparent fitted surface_render_residual(): Displays residuals with a zero-plane referenceIV Filtering (
src/surface.py)Data Validation & Robustness
src/__init__.pyto lazy-loaddata_loadermodule, avoiding yfinance dependency at import timesrc/iv_engine.pyto use ASCII-compatible notation (e.g.,sqrtinstead of√,dC/dsigmainstead of∂C/∂σ)Implementation Details
min(max_strike × 1.05, F × e^0.35)to respect both data extent and moneyness boundshttps://claude.ai/code/session_01ATapVU79KVy5UgFXtcnFUM