[Issue]: rccl tests on two Mi300x nodes gives me poor performance using mpirun

[rgee18]

Problem Description

echo "OS:" && cat /etc/os-release | grep -E "^(NAME=|VERSION=)";
OS:
NAME="Ubuntu"
VERSION="22.04.5 LTS (Jammy Jellyfish)"

echo "CPU: " && cat /proc/cpuinfo | grep "model name" | sort --unique;
CPU:
model name	: AMD EPYC 9534 64-Core Processor

echo "GPU:" && /opt/rocm/bin/rocminfo | grep -E "^\s*(Name|Marketing Name)";
GPU:
  Name:                    AMD EPYC 9534 64-Core Processor
  Marketing Name:          AMD EPYC 9534 64-Core Processor
  Name:                    AMD EPYC 9534 64-Core Processor
  Marketing Name:          AMD EPYC 9534 64-Core Processor
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
  Name:                    gfx942
  Marketing Name:          AMD Instinct MI300X
      Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-

Operating System

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

CPU

AMD EPYC 9534 64-Core Processor

GPU

AMD EPYC 9534 64-Core Processor

ROCm Version

ROCm version: 6.2.0

ROCm Component

No response

Steps to Reproduce

I followed the steps here to compile the rccl-tests https://github.com/ROCm/rccl-tests

I used MPI=1 and added a hostfile with my private IP addresses along with slots=8.

The command i'm using to run mpi job is HSA_NO_SCRATCH_RECLAIM=1 mpirun -np 16 --hostfile hostfile.txt --bind-to numa ./all_reduce_perf -b 8 -e 128M -f 2 -g 8

And i'm getting the following results attached.

all_reduce_test.txt

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[nileshnegi]

The attached logs suggest that rccl-tests was not built correctly. Please check if your MPI installation is uniform across both nodes and is correctly picked up by the RCCL-Tests binaries (e.g. ldd ./all_reduce_perf).

Please use -g 1 if you are launching 1 MPI process per GPU, i.e. HSA_NO_SCRATCH_RECLAIM=1 mpirun -np 16 --hostfile hostfile.txt --bind-to numa ./all_reduce_perf -b 8 -e 128M -f 2 -g 1

[rgee18]

all_reduce_test_2.txt

I'm not receiving Avg bus bandwidth when running HSA_NO_SCRATCH_RECLAIM=1 mpirun -np 16 --hostfile hostfile.txt --bind-to numa ./all_reduce_perf -b 8 -e 128M -f 2 -g 1

[rgee18]

Also, when running ./all_reduce_perf -b 128M -e 2048M -f 2 -g 1 on one host, I get the results:

# nThread 1 nGpus 1 minBytes 134217728 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
rccl-tests: Version develop:645be0e
# Using devices
#  Rank  0 Group  0 Pid 598224 on rgee-mi300x device  0 [0002:00:01] AMD Instinct MI300X
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
   134217728      33554432     float     sum      -1    66.81  2009.04    0.00      0     0.09  1488826.71    0.00      0
   268435456      67108864     float     sum      -1    134.0  2002.50    0.00      0     0.09  3080154.40    0.00      0
   536870912     134217728     float     sum      -1    267.8  2004.80    0.00      0     0.09  5702293.28    0.00      0
  1073741824     268435456     float     sum      -1    542.6  1978.94    0.00      0     0.09  12044215.64    0.00      0
  2147483648     536870912     float     sum      -1   1132.0  1897.07    0.00      0     0.10  21230683.62    0.00      0
# Errors with asterisks indicate errors that have exceeded the maximum threshold.
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0
#

[nileshnegi]

For single-node, if you are using mpirun, and you have built RCCL-Tests with MPI=1, pls use HSA_NO_SCRATCH_RECLAIM=1 mpirun -np 8 --bind-to numa ./all_reduce_perf -b 8 -e 128M -f 2 -g 1

For single-node, if you are not using mpirun, and you have built RCCL-Tests with MPI=0, pls use HSA_NO_SCRATCH_RECLAIM=1 ./all_reduce_perf -b 8 -e 128M -f 2 -g 8

[nileshnegi]

all_reduce_test_2.txt

I'm not receiving Avg bus bandwidth when running HSA_NO_SCRATCH_RECLAIM=1 mpirun -np 16 --hostfile hostfile.txt --bind-to numa ./all_reduce_perf -b 8 -e 128M -f 2 -g 1

In this case, i would advise checking your MPI installation, and if RCCL is picking the correct MPI installation.

The attached logs suggest that rccl-tests was not built correctly.

Please check if:

• your MPI installation is uniform across both nodes (e.g., which mpirun shows the same output on both nodes).
• is correctly picked up by the RCCL-Tests binaries (e.g. ldd ./all_reduce_perf points to the same libmpi.so on both nodes).
• if MPI is installed in a non-standard location, pls set PATH and LD_LIBRARY_PATH accordingly.