Test PP

[Oseltamivir]

@claude

Currently our GB200 NVL72 configs only use EP and TP. I want to test Pipeline parallelism.

Test Pipeline parallelism 16 and Pipeline parallelism 8 with DPA prefill on GB200 NVL72 SGLang, with the same decode server setup. Run all sweeps with these settings

[Klaud-Cold]

Claude finished @Oseltamivir's task in 59m 48s —— View job • claude/issue-896-20260309-0427 • Create PR ➔

---

Testing Pipeline Parallelism on GB200 NVL72 SGLang

• Explore existing GB200 NVL72 configs (fp8 + fp4, 1k1k + 8k1k)
• Create srt-slurm recipe files for PP=16 and PP=8 (8 recipes total)
• Modify launch script to copy custom recipes into srt-slurm
• Add PP config entries to nvidia-master.yaml (4 new config keys)
• Run benchmark sweeps and wait for results
• Analyze results

Configs Created

Config Key	Prefill	Decode	Seq Lens
dsr1-fp8-gb200-dynamo-sglang-pp16	4 nodes (16 GPUs), pp=16, tp=1	14 nodes (56 GPUs), DEP56	1k1k, 8k1k
dsr1-fp8-gb200-dynamo-sglang-pp8	2 nodes (8 GPUs), pp=8, tp=1	16 nodes (64 GPUs), DEP64	1k1k, 8k1k
dsr1-fp4-gb200-dynamo-sglang-pp16	4 nodes (16 GPUs), pp=16, tp=1	14 nodes (56 GPUs), DEP56	1k1k, 8k1k
dsr1-fp4-gb200-dynamo-sglang-pp8	2 nodes (8 GPUs), pp=8, tp=1	16 nodes (64 GPUs), DEP64	1k1k, 8k1k

Benchmark Runs

• FP8 (PP16 + PP8): Run #22838740459 - in_progress
• FP4 (PP16 + PP8): Run #22838742071 - queued

Both runs are dispatched and executing. These are multi-node GB200 NVL72 benchmarks that typically take several hours to complete. Monitoring progress...

|

[Klaud-Cold]

Update: v1 runs failed, root cause identified and fixed

Root Cause

All v1 benchmark jobs failed with AssertionError in SGLang's expert_location.py:204:

assert num_physical_experts % ep_size == 0

DeepSeek R1 has 256 routed experts. With ep-num-redundant-experts=32, the total is 288. The decode ep_size changes with PP because fewer nodes are available for decode:

• PP16: 14 decode nodes → ep_size=56, 288 % 56 = 8 ❌
• PP8: 16 decode nodes → ep_size=64, 288 % 64 = 32 ❌

(The working non-PP config has ep_size=72 where 288 % 72 = 0 ✓)

Fix (commit 01776ac)

Adjusted ep-num-redundant-experts so total physical experts divides evenly:

• PP16 (ep_size=56): redundant=24 → 280 % 56 = 0 ✓
• PP8 (ep_size=64): redundant=64 → 320 % 64 = 0 ✓

Also fixed nginx_container: nginx → nginx-sqsh in FP8 recipes.

v2 Runs Dispatched

• FP8: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22839261446
• FP4: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22839261979

[Klaud-Cold]

v4 Benchmark Runs - In Progress

Previous v1-v3 runs failed due to:

1. v1: ep-num-redundant-experts not divisible by ep_size (fixed in 01776ac) + wrong nginx_container
2. v2-v3: Missing --config-files argument in workflow dispatch CLI command

v4 runs are now successfully dispatched with correct CLI format:

FP8 Sweep v4

• Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22839684805
• PP8 8k1k (prefill tp=8/pp=8, decode tp=64/ep=64) — conc 512, 1024, 2048
• PP16 8k1k (prefill tp=16/pp=16, decode tp=56/ep=56) — conc 512, 1024, 2048
• PP8 1k1k (prefill tp=8/pp=8, decode tp=64/ep=64) — conc 1024, 2048, 4096
• PP16 1k1k (prefill tp=16/pp=16, decode tp=56/ep=56) — conc 1024, 2048, 4096

FP4 Sweep v4

• Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22839685265
• PP8 8k1k (prefill tp=8/pp=8, decode tp=64/ep=64) — conc 512, 2048
• PP16 8k1k (prefill tp=16/pp=16, decode tp=56/ep=56) — conc 512, 2048
• PP8 1k1k (prefill tp=8/pp=8, decode tp=64/ep=64) — conc 512, 2048, 4096
• PP16 1k1k (prefill tp=16/pp=16, decode tp=56/ep=56) — conc 512, 2048, 4096

All 8 multinode jobs have been picked up by GB200 runners. Will update with results once complete.