Skip to content

Commit 2b35007

Browse files
committed
feat: add ep/sp FSDP MoE finetuning entry and update script
- Add new entry for ep/sp FSDP MoE finetuning in README table - Update ep_fsdp_qwen3_moe.py script to include ulysses_size parameter for enhanced parallelism configuration
1 parent 1a6d0ca commit 2b35007

File tree

2 files changed

+3
-0
lines changed

2 files changed

+3
-0
lines changed

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ pip install -e .
6969
| --------------------------------- | --------------- | ------------------------------------------------- |
7070
| FSDP finetuning | transformers | [Script](cookbook/transformers/fsdp2.py) |
7171
| FSDP MoE finetuning | transformers | [Script](cookbook/transformers/fsdp2_moe.py) |
72+
| ep/sp FSDP MoE finetuning | transformers | [Script](cookbook/transformers/ep_fsdp_qwen3_moe.py) |
7273
| EP MoE finetuning | transformers | [Script](cookbook/transformers/ep_fsdp_qwen3_moe.py) |
7374
| pp/tp/cp finetuning | megatron | [Script](cookbook/megatron/tp.py) |
7475
| pp/tp/cp MoE finetuning | megatron | [Script](cookbook/megatron/tp_moe.py) |

cookbook/transformers/ep_fsdp_qwen3_moe.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,13 @@
2121
# 4 gpus, dp=2, ep=2
2222
dp_size = 2
2323
ep_size = 2
24+
ulysses_size = 2
2425

2526
device_mesh = DeviceMesh(
2627
device_type=Platform.get_platform().device_prefix(),
2728
mesh=np.arange(dp_size * ep_size).reshape(dp_size, ep_size),
2829
mesh_dim_names=('dp', 'ep'),
30+
ulysses_size=ulysses_size, # enable sp
2931
)
3032

3133
twinkle.initialize(

0 commit comments

Comments
 (0)