Add attention token visualization and manifold exploration

[architehc]

Summary

This PR adds comprehensive attention token visualization and manifold exploration capabilities to SGLang, enabling real-time analysis of attention patterns during inference.

Key Features

• Top-K Attention Capture: Server-side capture of top-k attended positions per token
• Attention Fingerprinting: 20-dimensional fingerprint vectors summarizing attention patterns
• Manifold Discovery: UMAP/HDBSCAN clustering for discovering attention pattern zones
• Real-time Visualization: React UI for exploring attention patterns and manifold structure
• Zone Classification: Automatic classification into syntax_floor, semantic_bridge, structure_ripple zones

Exploration Results: Qwen3 80B vs 4B

We conducted a 5-hour exploration comparing attention patterns between Qwen3-80B (MoE) and Qwen3-4B (Dense):

Zone	80B (MoE)	4B (Dense)	Difference
structure_ripple	64.0%	84.3%	+20.3%
semantic_bridge	35.3%	15.5%	-19.8%
syntax_floor	0.7%	0.1%	-0.6%

Key Finding: Smaller dense models rely ~20% more on structural attention patterns, while larger MoE models achieve more balanced semantic/structural attention through expert routing.

New Components

• examples/attention_explorer/ - Full-stack attention exploration application
  • ui/ - React frontend with manifold visualization
  • discovery/ - Manifold discovery pipeline with checkpointing
  • schemas/ - Database schemas for fingerprint storage
  • exploration_outputs/ - Reports and visualizations from model explorations

API Extensions

# Enable attention capture
response = client.chat.completions.create(
    model="model-name",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"return_attention_tokens": True}
)

Server Arguments

• --return-attention-tokens: Enable attention token capture
• --attention-tokens-top-k: Number of top attended positions to capture (default: 32)

Test Plan

• Unit tests for attention modules
• Integration tests for fingerprinting pipeline
• E2E tests for UI features
• Manual testing with Qwen3-80B and Qwen3-4B models

Documentation

• Exploration One-Pager
• 80B Exploration Report
• 80B vs 4B Comparison

🤖 Generated with Claude Code