allreduce仿真问题咨询

[ykk199]

我按照示例中的命令运行，可以得到和tutorial中相同的结果。
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -g 32 -gt H100 -bw 400Gbps -nvbw 1440Gbps
AS_SEND_LAT=2 AS_PXN_ENABLE=1 ./bin/SimAI_simulator -t 8 -w ./example/microAllReduce.txt -n ./Rail_Opti_SingleToR_32g_8gps_400Gbps_H100 -c ./astra-sim-alibabacloud/inputs/config/SimAI.conf

但是当我用如下命令增加GPU数量，重新运行，得出的结果没有任何改变。请帮忙看一下什么原因，谢谢！
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -g 64 -gt H100 -bw 400Gbps -nvbw 1440Gbps
AS_SEND_LAT=2 AS_PXN_ENABLE=1 ./bin/SimAI_simulator -t 8 -w ./example/microAllReduce.txt -n ./Rail_Opti_SingleToR_64g_8gps_400Gbps_H100 -c ./astra-sim-alibabacloud/inputs/config/SimAI.conf

[tianhao909]

AS_SEND_LAT=2 AS_PXN_ENABLE=1 ./bin/SimAI_simulator -t 8 -w ./example/microAllReduce.txt -n ./Rail_Opti_SingleToR_64g_8gps_400Gbps_H100 -c ./astra-sim-alibabacloud/inputs/config/SimAI.conf

check 一下上面命令中的 microAllReduce.txt
比如如果都是tp8的ar， 32卡 和 64卡 可能 tp8的ar端到端时间是一样的？

[ykk199]

microAllReduce.txt是示例中的，未做任何改变。第一行的信息为：
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 64 checkpoints: 0 checkpoint_initiates: 0

我想用simAI仿真32和64GPU的allreduce跟nccl-test实测做个对比，请问这样做是否正确，如不正确，还请帮忙指导下如何修改。感谢！

[tianhao909]

model_parallel_NPU_group: 8 是 tp 的ar， tp的ar一般不跨机

========
也可以先熟悉下这些文档，主要改aicb的workload应该

aicb paper链接：
https://www.sciencedirect.com/science/article/pii/S2772485925000250?via%3Dihub

simai paper链接：
https://www.usenix.org/conference/nsdi25/presentation/wang-xizheng-simai

aicb和simai 的仓库 tutorial，
https://github.com/aliyun/SimAI/blob/master/docs/Tutorial.md
https://github.com/aliyun/aicb/blob/master/training/tutorial.md

========
具体workload的细节答疑也可以参考这个issue的讨论～
aliyun/aicb#59

[ykk199]

改成dp后确实有效果了，但是测试2048张H100的allreduce时，会异常停止，ncclFlowModel_EndToEnd.csv是空的。还请帮忙看一下是哪里的问题。

./example/microAllReduce.txt中的内容为：
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 2048 checkpoints: 0 checkpoint_initiates: 0
1
embedding_layer -1 556000 NONE 0 1 NONE 0 1 ALLREDUCE 1024 1

Rail_Opti_SingleToR_2048g_8gps_400Gbps_H100是通过如下命令生成的：
python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py --ro -gps 8 -g 2048 -gt H100 -bw 400Gbps -nvbw 1440Gbps -asn 32 -psn 64

运行命令为：
AS_SEND_LAT=2 AS_PXN_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAlltoall.txt -n ./Rail_Opti_SingleToR_2048g_8gps_400Gbps_H100 -c ./astra-sim-alibabacloud/inputs/config/SimAI.conf

输出的log为：
maxRtt=4720 maxBdp=236000
Running Simulation.
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 2304 index in ring: 0 offset: 1total nodes in ring: 2304
ring of node 0, id: 0 dimension: local total nodes in ring: 2304 index in ring: 0 offset: 1total nodes in ring: 2304
ring of node 0, id: 0 dimension: local total nodes in ring: 2304 index in ring: 0 offset: 1total nodes in ring: 2304
ring of node 0, id: 0 dimension: local total nodes in ring: 2304 index in ring: 0 offset: 1total nodes in ring: 2304
total nodes: 2304
Success in opening workload file
model_parallel_NPU_group is 8
checkpoints layers are:
layers initiating fwd_in_bckwd are:
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 288 index in ring: 0 offset: 8total nodes in ring: 288
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 288 index in ring: 0 offset: 8total nodes in ring: 288
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 288 index in ring: 0 offset: 8total nodes in ring: 288
ring of node 0, id: 0 dimension: local total nodes in ring: 8 index in ring: 0 offset: 1total nodes in ring: 8
ring of node 0, id: 0 dimension: local total nodes in ring: 288 index in ring: 0 offset: 8total nodes in ring: 288
pp_commize:0
id: embedding_layer , depen: -1 , wg_comp_time: 1
type: HYBRID_TRANSFORMER_FWD_IN_BCKWD ,num passes: 1 ,lines: 1 compute scale: 1 ,comm scale: 1
stat path: ./ncclFlowModel_ ,total rows: 1 ,stat row: 0
CSV path and filename: ./ncclFlowModel_detailed_2304.csv
CSV path and filename: ./ncclFlowModel_EndToEnd.csv
simulator run
info: no forward pass collective for layer: embedding_layer
info: no input grad collective for layer: embedding_layer
chunk size is: 1024 , size is: 1024 , layer_num is: 0 , node: 0
info: all-reduce weight grad collective issued for layer: embedding_layer with size: 1024, involved dimensions: 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
pass: 0 finished at time: 556004
无其他信息输出

[tianhao909]

hello， 您试试把h100换成a100也会报同样的错误么

然后之前有些issue也可以参考参考～

#193

#139

#142

[ykk199]

Hi，换成A100也会有同样的问题。另外，我注意到astra-sim ns3在千卡集群仿真上也会有类似的异常停止或卡住的现象。请问您使用astra-sim ns3和simAI仿真千卡集群规模有遇到类似问题吗？

[tianhao909]

hi, 你可以搜一搜之前的 issue，
有类似的问题， 可能是 ns3 多线程 发流或者完成流的blocking和nonblocking ， CollectiveBarrier::Blocking CollectiveBarrier::NonBlocking ，
或者 指针double free (多线程调度可能导致同一条stream执行了两次exit(),从而引起double free, 可能需要确保每条流只执行一次exit())

================

或者得debug

比如 改了 workload 改了 topo ， 可能可以试试原来默认的workload 和 topo 能不能过， 改了哪个一层层调用栈去对一下， 类似消融实验，

改动引起的bug

============

具体debug方法

运行命令前面加AS_LOG_LEVEL=DEBUG

AS_LOG_LEVEL=DEBUG AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/xxx.txt -n ./xxxtopo -c astra-sim-alibabacloud/inputs/config/SimAI.conf运行之后

从 /etc/astra-sim/SimAI.log
这个文件里面查看

如果想修改这个路径， 改这个文件
/SimAI/astra-sim-alibabacloud/astra-sim/system/MockNcclLog.h
#define LOG_PATH "/etc/astra-sim/"

源码里可以搜索NcclLog->writeLog，用全局的单例维护，可以修改需要的flow和trace采集信息和采集点

[ykk199]

好的，我再调试看看，有问题再跟您请教。感谢！