[Issue]: 2GPU per node hangs with rccl-tests's alltoall_perf and aws-ofi-rccl plugin

[tmh97]

Problem Description

When running the alltoall_perf test with openmpi 5.0.3, the aws-ofi-rccl plugin, and OPX libfabric provider using (-np 4 -N 2) to specify two processes per node and using the -g 1 argument to specify 1 gpu per process, we are experiencing infrequent hangs at message sizes 8MB and 16MB.

It seems that RCCL plugin calls fi_endpoint to establish connection, which makes OPX open endoints and generate rx and tx flowkeys. When we hit the hang, and check the flowkeys, it seems they were overridden during runtime by RCCL plugin calling fi_endpoint again, even though connections were already established.

Operating System

RHEL 9.4

CPU

AMD Milan

GPU

AMD Instinct MI210

ROCm Version

ROCm 6.2.0

ROCm Component

rccl

Steps to Reproduce

Run alltoall_perf from rccl tests on two nodes, each node has 2 gpus.

Run command to reproduce:

mpirun -np 4 -N 2 -x MPIR_CVAR_CH4_OFI_ENABLE_RMA=1 -x MPIR_CVAR_ENABLE_GPU=1 -x MPIR_CVAR_CH4_OFI_ENABLE_HMEM=1 -mca mtl ofi -x FI_PROVIDER=opx -mca btl self,vader alltoall_perf -b 4 -e 268435456 -f 2 -g 1

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[ppanchad-amd]

Hi @tmh97. Internal ticket has been created to investigate your issue. Thanks!

[huanrwan-amd]

Hi @tmh97 Sorry for the late reply.
Can you please provide more info while running the test. E.g set NCCL_DEBUG=TRACE and provide the logs? and if possible, can you also try to logging around fi_endpoint?
I tried the test with OpenMPI 5.0.6, ROCm6.2.1, lastest aws-ofi-libfabric and libfabric 1.20.1 cannot reproduce the issue.

[tmh97]

@ppanchad-amd thanks for opening a ticket to track this!

@huanrwan-amd I will retry with the same software stack as you and report back here! If I am still hitting issues, I'll provide some NCCL_DEBUG and FI_LOG_LEVEL logs.

[tmh97]

Hey @huanrwan-amd and @ppanchad-amd, can we re-open this issue? I have recently reproduced this again and would like to dig into the root cause.

[huanrwan-amd]

@tmh97 Thanks for the reply, can you provide NCCL/RCCL related logs on the issue?

[tmh97]

@huanrwan-amd Hey so the issue is, when running with 2 processes per node, 1 gpu per process, two nodes, it seems that 2 of the 4 endpoints created are hanging with bytes_outstanding. It is also reproducible with nccl.

Which logs would be most helpful, just NCCL_DEBUG=info?

[huanrwan-amd]

Hi @tmh97, Yes, let's start with NCCL_DEBUG=info. And Can you please provide other ROCm software/driver version info as well? And are you still using AMD Instinct MI210?

[tmh97]

rccl_debug_logs.txt

So i'm actually using MI300x now!

My hunch is the hang could be due to:

• NCCL/RCCL waiting on a completion that never got signaled even though we issued it.

• A bug in the NCCL/RCCL OFI plugin not tracking or handling CQEs properly.

• A race condition where a CQE is injected, but the app doesn’t “see” it.

Thanks @huanrwan-amd :)

[huanrwan-amd]

Hi @tmh97,

1. Can you also provide the driver version, using amd-smi version?
2. in the logs:
   host-A:35116:35116 [0] /long_pathname_so_that_rpms_can_package_the_debug_info/src/rccl/build/hipify/src/init.cc:142 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0"
   Can you try disable kernel.numa_balancing as https://rocm.docs.amd.com/projects/rccl/en/develop/how-to/troubleshooting-rccl.html#collect-the-rccl-microbenchmark-data and check the output?

[tmh97]

@huanrwan-amd

Here's the driver version. It is the same on both of the 2 nodes I am using.

============================== Version of System Component ===============================
Driver version: 6.10.5

I will try disable kernel.numa_balancing this afternoon and let you know how that goes!

[tmh97]

Hey @huanrwan-amd, no effect when I set kernel.numa_balancing=0 on both nodes :(

[huanrwan-amd]

Hi @tmh97, thanks for the update. Can you please also provide the logs?

1. In the logs, it shows you are using 4 GPUs across two nodes. Since you had updated your hardware, can you please describe your new setup and the command you are using?

#   Rank  0 Pid  35116 on host-A device  0 [0000:05:00.0] AMD Instinct MI300X
#   Rank  1 Pid  35117 on host-A device  1 [0000:26:00.0] AMD Instinct MI300X
#   Rank  2 Pid   5897 on host-B device  0 [0000:05:00.0] AMD Instinct MI300X
#   Rank  3 Pid   5898 on host-B device  1 [0000:26:00.0] AMD Instinct MI300X

2. Have you had a chance to try with ROCm 6.4? which just released few days ago. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html