[P1][Phase-3][experimental] Beijing Baseline Validation & 3×3 Model Grid Completion

[matercomus]

Metadata

• Priority: P1-Major
• Phase: Phase 3 (Before Publication)
• Feasibility: experimental
• Category: validation
• Effort: 60-70 hours (3 models × 25 epochs × ~1 hour)
• Dependencies: None (can run immediately)

Problem Statement

Beijing dataset currently has incomplete model coverage compared to Porto, and the vanilla baseline needs validation with additional random seeds to ensure reproducibility.

Current Model Coverage

Beijing (6 models - incomplete):

• vanilla: 1 seed (seed42) ⚠️ Only 1 seed - needs validation
• distilled: 2 seeds (seed42, seed44) - Missing seed43

Porto (9 models - complete):

• vanilla: 3 seeds (42, 43, 44) ✅
• distill_phase1: 3 seeds (42, 43, 44) ✅
• distill_phase2: 3 seeds (42, 43, 44) ✅

Evidence of Problem

Concern: Beijing vanilla baseline (seed42) may have anomalies

• Only 1 vanilla seed makes it impossible to verify consistency
• Cannot compute cross-seed statistics (mean, std, confidence intervals)
• Porto has 3 seeds for robustness - Beijing should match

Asymmetry: Beijing has 2 distilled but only 1 vanilla

• Unbalanced comparison (2-seed distilled vs 1-seed vanilla)
• Cannot apply cross-seed analysis tools (designed for N=3)
• Porto's 3×3 grid enables robust statistical analysis

Required Changes

Option A: Minimal Validation (2 models, ~50 hours)

Add 2 vanilla models to verify baseline:

• vanilla_25epoch_seed43.pth
• vanilla_25epoch_seed44.pth

Result: 3 vanilla + 2 distilled (sufficient for validation)

Runtime: 2 × 25 epochs × ~1 hour = 50 hours (~2.1 days)

Option B: Complete 3×3 Grid (3 models, ~75 hours) ✅ Recommended

Add 2 vanilla + 1 distilled to match Porto:

• vanilla_25epoch_seed43.pth
• vanilla_25epoch_seed44.pth
• distilled_25epoch_seed43.pth

Result: 3 vanilla + 3 distilled (matches Porto's structure)

Runtime: 3 × 25 epochs × ~1 hour = 75 hours (~3.1 days)

Option C: Just One More Vanilla (1 model, ~25 hours)

Minimal risk mitigation:

• vanilla_25epoch_seed43.pth

Result: 2 vanilla + 2 distilled (at least one validation point)

Runtime: 1 × 25 epochs × ~1 hour = 25 hours (~1 day)

Recommended Approach: Option B (Complete 3×3 Grid)

Why This Matters

Statistical Rigor:

• Cross-seed analysis requires N≥3 for meaningful statistics
• Confidence intervals need multiple seeds
• Standard deviation meaningless with N=1 or N=2

Publication Quality:

• Reviewers expect multiple random seeds for reproducibility
• Porto has 3 seeds - Beijing should match for consistency
• Asymmetric coverage raises questions

Risk Mitigation:

• If vanilla seed42 has issues, we have seeds 43, 44 to validate
• 3 seeds reveal if distillation benefits are seed-dependent
• Enables robust paired statistical tests (Issue Integrate paired statistical tests into evaluation pipeline #51)

Benefits of 3×3 Grid

Enables existing analysis tools:

• scripts/analysis/cross_seed_analysis.py designed for N=3
• Paired tests more robust with matched 3×3 coverage
• Consistent methodology across datasets

Cross-dataset comparison:

• Beijing 3×3 vs Porto 3×3 (apples-to-apples)
• Same seed robustness analysis for both
• Fair comparison of distillation benefits

Implementation

Training Configuration

Use existing Beijing config with different seeds:

# Vanilla seed43
uv run python train_with_distill.py \
  --dataset Beijing \
  --config config/Beijing.yaml \
  --seed 43 \
  --cuda 0 \
  --data_dir /home/matt/Dev/HOSER-dataset

# Vanilla seed44
uv run python train_with_distill.py \
  --dataset Beijing \
  --config config/Beijing.yaml \
  --seed 44 \
  --cuda 0 \
  --data_dir /home/matt/Dev/HOSER-dataset

# Distilled seed43 (if Option B)
uv run python train_with_distill.py \
  --dataset Beijing \
  --config config/Beijing.yaml \
  --seed 43 \
  --cuda 0 \
  --data_dir /home/matt/Dev/HOSER-dataset
  # (with distillation enabled in config)

Parameters:

• Epochs: 25 (same as existing models)
• Config: Use existing config/Beijing.yaml
• Seeds: 43, 44 (to complement existing seed42)

Output Files

New model files (in hoser-distill-optuna-6/models/):

• vanilla_25epoch_seed43.pth
• vanilla_25epoch_seed44.pth
• distilled_25epoch_seed43.pth (if Option B)

Evaluation After Training

After training new models, run full evaluation (can use tonight's A* eval scripts):

cd hoser-distill-optuna-6
uv run python ../python_pipeline.py --use-astar

This will:

• Generate trajectories for new models
• Evaluate all metrics (including trajectory-level)
• Run paired statistical analysis
• Update cross-seed analysis

Timeline

Option A (2 vanilla models)

• Training: 50 hours (~2.1 days)
• Evaluation: 8-10 hours
• Analysis: 2-3 hours
• Total: ~2.5 days

Option B (3 models - 3×3 grid) ✅ Recommended

• Training: 75 hours (~3.1 days)
• Evaluation: 12-15 hours
• Analysis: 3-4 hours
• Total: ~4 days

Option C (1 vanilla model)

• Training: 25 hours (~1 day)
• Evaluation: 4-5 hours
• Analysis: 1-2 hours
• Total: ~1.5 days

Validation Steps

• Train new vanilla models (seed43, seed44)
• Train new distilled model (seed43) if Option B
• Run full evaluation pipeline with A* search
• Verify vanilla baseline consistency across seeds
• Update cross-seed analysis
• Validate paired statistical tests work with 3×3 grid
• Document any seed-dependent effects

Expected Outcomes

If Vanilla Baseline is Consistent

• Seeds 42, 43, 44 show similar performance (±5% variance)
• Confirms original vanilla seed42 is valid
• Strengthens publication claims

If Vanilla Baseline Has Issues

• Discover seed-dependent behavior
• Identify which seed is anomalous
• Report mean ± std across 3 seeds (more robust)
• Avoids publishing potentially flawed baseline

Cross-Dataset Comparison

• Beijing 3×3 vs Porto 3×3 (symmetric comparison)
• Consistent statistical analysis methodology
• Robust conclusions about distillation benefits

Success Criteria

• Beijing has 3 vanilla models (seeds 42, 43, 44)
• Beijing has 3 distilled models (seeds 42, 43, 44) if Option B
• Cross-seed variance within acceptable range (<10%)
• Vanilla baseline validated as consistent
• Cross-seed analysis updated for Beijing
• Paired tests run successfully with 3×3 grid

Files to Generate

Models:

• models/vanilla_25epoch_seed43.pth
• models/vanilla_25epoch_seed44.pth
• models/distilled_25epoch_seed43.pth (Option B)

Evaluations (auto-generated by pipeline):

• eval/*/results.json for new models
• eval/*/trajectory_metrics.json for new models
• paired_analysis/*/ updated with new comparisons

Documentation:

• Update cross-seed analysis results
• Document seed consistency findings
• Note any seed-dependent effects discovered

Priority Justification

Why P1-Major:

1. Validation critical: Only 1 vanilla seed is insufficient
2. Pre-publication: Must verify before submission
3. Blocks confidence: Cannot trust results without seed validation
4. Relatively quick: 3-4 days vs weeks for other tasks
5. Prevents embarrassment: Better to discover issues now than in reviews

Coordination with Other Issues

Related:

• Issue [P2][Phase-3][document] Issue 3.1: Sample Size Not Justified #19: Sample size justification (benefits from 3×3 grid)
• Issue [P2][Phase-3][experimental] Porto Dataset Beam Search Ablation Study #44: Porto ablation (enables symmetric Beijing-Porto comparison)
• Issue Integrate paired statistical tests into evaluation pipeline #51: Paired statistical tests (more robust with 3×3 grid)

Timeline:

• Can run immediately (no dependencies)
• Or run after tonight's A evals* complete (~2 days from now)
• Should complete before paper submission

Recommendation

Pursue Option B (Complete 3×3 Grid):

• 75 hours (~3 days) is manageable
• Matches Porto's structure exactly
• Enables all cross-seed analysis tools
• Provides maximum confidence in results
• Publication-ready model coverage

Alternative: If time-constrained, Option A (2 vanilla) validates baseline in ~2 days

---

Status: Pending decision
Next Steps: Approve option and schedule training runs

[matercomus]

Execution Plan: Sequential Training → A* Evaluation

Decision

Proceeding with Option B (Complete 3×3 Grid) using sequential execution.

Rationale:

• Porto A* evaluation already running (no interference)
• Beijing needs baseline validation before expensive evaluation
• Sequential training safer (no GPU resource conflicts)
• Single A* eval run after all models complete

Execution Sequence

Phase 1: Model Training (75 hours, ~3.1 days)

1. vanilla_25epoch_seed43.pth - 25 hours
2. vanilla_25epoch_seed44.pth - 25 hours
3. distilled_25epoch_seed43.pth - 25 hours

Phase 2: A Evaluation (12 hours, ~0.5 days)*

• All 6 Beijing models (3 vanilla + 3 distilled)
• 2 OD sources (train + test)
• Includes paired statistical analysis

Total Timeline: ~87 hours (~3.6 days)

Automated Execution

Created wrapper script: scripts/train_then_eval_beijing.sh

• Trains all 3 models sequentially
• Automatically starts A* evaluation after training
• Comprehensive logging
• Can be run in tmux and detached

Commands

# Stop current Beijing eval if running
tmux kill-session -t beijing-astar 2>/dev/null || true

# Start training → evaluation sequence
tmux new-session -s beijing-train-eval
cd /home/matt/Dev/HOSER && ./scripts/train_then_eval_beijing.sh
# Ctrl+b, d to detach

Monitoring

# View progress
tmux attach -t beijing-train-eval

# Check logs
tail -f /home/matt/Dev/HOSER/logs/train_then_eval_beijing_*.log

# Monitor WandB for training metrics

Expected Completion

Start: Tonight (Nov 6, 2025)
Training complete: ~Nov 9, evening
Evaluation complete: ~Nov 10, morning
Results ready: ~Nov 10, afternoon

Deliverables

Models:

• hoser-distill-optuna-6/models/vanilla_25epoch_seed43.pth
• hoser-distill-optuna-6/models/vanilla_25epoch_seed44.pth
• hoser-distill-optuna-6/models/distilled_25epoch_seed43.pth

Evaluation Results:

• A* trajectories for all 6 models
• eval/*/results.json with normalized metrics
• eval/*/trajectory_metrics.json for paired tests
• paired_analysis/*/ comparison results

Analysis:

• Cross-seed variance for vanilla (seeds 42, 43, 44)
• Cross-seed variance for distilled (seeds 42, 43, 44)
• Validation of baseline consistency
• Complete 3×3 grid for statistical analysis

---

Status: Ready to execute
Next: Run wrapper script in tmux tonight

[matercomus]

Execution Started - Nov 6, 2025 20:33

Status: IN PROGRESS

Phase 1: Beijing Model Training - Started

• Model 1/3: vanilla_25epoch_seed43 - Training now (6% cache loaded)
• Model 2/3: vanilla_25epoch_seed44 - Pending
• Model 3/3: distilled_25epoch_seed43 - Pending

Phase 2: A Evaluation* - Not started (will begin after Phase 1)

Parallel Operations

Porto A Evaluation* (separate, running in parallel):

• Status: 8% complete (422/5000 trajectories)
• Model: distilled_25epoch_seed42
• Speed: 5.45s/trajectory
• Independent from Beijing training

Timeline Update

Started: Nov 6, 2025 20:33
Expected Model 1 complete: Nov 7, ~21:30 (25 hours)
Expected Model 2 complete: Nov 8, ~22:30 (50 hours)
Expected Model 3 complete: Nov 9, ~23:30 (75 hours)
Expected A eval complete*: Nov 10, ~11:30 (87 hours)

Monitoring

Beijing training logs:

tmux attach -t beijing-train-eval
tail -f /home/matt/Dev/HOSER/logs/train_then_eval_beijing_*.log

Porto evaluation (parallel):

• Running independently in separate tmux pane
• No interference with Beijing training

---

Next update: When Model 1/3 completes (~25 hours)

[matercomus]

✅ Implementation Update

Status: Implementation complete - ready to run

Changes Made

1. Added --no-distill flag to train_with_distill.py

   • Forces vanilla training even when config has distill.enable: true
   • Ensures robust vanilla training without config manipulation
   • Commit: 4c5dcfb

2. Updated config/Beijing.yaml with tuned hyperparameters

   • Updated to use best hyperparameters from Optuna study (hoser_tuning_20251003_162916)
   • lambda: 0.01 → 0.00142
   • temperature: 2.0 → 4.37
   • window: 4 → 7
   • Ensures new models match existing Beijing models
   • Commit: cdd692e

3. Updated scripts/train_then_eval_beijing.sh for full 3×3 grid

   • Now trains all 6 models (Option B - Complete 3×3 Grid):
     • 3 vanilla models (seeds 42, 43, 44)
     • 3 distilled models (seeds 42, 43, 44)
   • Added backup logic for all seeds before training
   • Updated timeline: ~162 hours (~6.75 days) total
   • Commit: 4c5dcfb

Implementation Details

Training Script: scripts/train_then_eval_beijing.sh

• Automatically backs up existing checkpoints for seeds 42, 43, 44
• Uses --no-distill flag for vanilla training
• Uses tuned hyperparameters from config for distilled training
• Sequential training (one model at a time) to avoid resource contention

Usage:

cd /home/matt/Dev/HOSER
./scripts/train_then_eval_beijing.sh

What it does:

1. Backs up existing checkpoints (if any)
2. Trains 3 vanilla models sequentially (seeds 42, 43, 44)
3. Trains 3 distilled models sequentially (seeds 42, 43, 44)
4. Runs A* evaluation on all 6 models
5. Generates trajectory-level metrics for paired statistical tests

Next Steps

• Run the training script (will take ~6.75 days)
• Verify all 6 models train successfully
• Run evaluation pipeline after training completes
• Update cross-seed analysis with new results
• Document seed consistency findings

Timeline

• Training: ~150 hours (6 × 25 hours)
• Evaluation: ~12 hours
• Total: ~162 hours (~6.75 days)

All models will use the same tuned hyperparameters from the Optuna study, ensuring consistency with existing Beijing models.

[matercomus]

🚀 Training Started

Status: Currently running

Started: 2025-11-07 14:35:59 CST

Current Progress:

• ✅ All prerequisites verified
• ✅ Backups completed for seeds 42, 43, 44
• 🔄 Currently training: Model 1/6 - vanilla_25epoch_seed42
  • Epoch 1 in progress (~13% complete)
  • Distillation correctly disabled via --no-distill flag
  • WandB tracking active

Log File: /home/matt/Dev/HOSER/logs/train_then_eval_beijing_20251107_143558.log

Expected Timeline:

• Model 1 (vanilla_seed42): ~25 hours → completes ~Nov 8, 15:35
• Model 2 (vanilla_seed43): ~25 hours → completes ~Nov 9, 16:35
• Model 3 (vanilla_seed44): ~25 hours → completes ~Nov 10, 17:35
• Model 4 (distilled_seed42): ~25 hours → completes ~Nov 11, 18:35
• Model 5 (distilled_seed43): ~25 hours → completes ~Nov 12, 19:35
• Model 6 (distilled_seed44): ~25 hours → completes ~Nov 13, 20:35
• Evaluation: ~12 hours → completes ~Nov 14, 08:35

Total: ~6.75 days (completes ~Nov 14, 2025)

The script is running in a tmux session and will train all 6 models sequentially. All models are using the tuned hyperparameters from the Optuna study.

[matercomus]

Training Status Update

Previous attempt: Training had to be stopped because it was crashing when running concurrently with the A* Porto re-evaluation.

Current status: Training restarted sequentially at Mon Nov 10 15:35:34 CST 2025

Training Plan

Running train_then_eval_beijing.sh to train all 6 Beijing models:

Phase 1: Training (150 hours)

• Vanilla models: seeds 42, 43, 44 (~25h each)
• Distilled models: seeds 42, 43, 44 (~25h each)

Phase 2: A Evaluation (12 hours)*

• All 6 models evaluated on train + test OD sources

Total estimated time: ~162 hours (~6.75 days)

Current Progress

• ✅ Started Model 1/6: vanilla_25epoch_seed42
• 📊 Epoch 1 in progress (~42% complete)
• 📈 WandB run: Beijing_b128_acc8

Training is now running in isolation to avoid resource conflicts.