MANA (installed in a shared directory) crashes when launching MPI app

[ggboom12138]

If I install MANA in a shared directory, it won't function properly.

Command:
mpirun -np 4 /home/Share/mana/bin/mana_launch /home/Share/mana_test

Error info:

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 642345 RUNNING AT worker1
= EXIT CODE: 11
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)

Environment:

• 3.10.0-1160.118.1.el7.x86_64
• MPICH-3.3.2
• MANA-1.2.0

Any similar issues before?

[xuyao0127]

@ggboom12138 Thank you for your feedback. You mentioned MANA installed in a shared directory crashes. Does MANA crash when installed in a directory that is not shared? What is the testing program you used? And what is the Linux distribution you used? We'd like to setup a similar environment and reproduce this bug.

[ggboom12138]

If MANA is installed in a non-shared directory, it won't crash, but it cannot perform checkpointing properly.

Question:
Can you roughly explain how mana is typically tested in a multi-node environment? Does each node need to start a coordinator? Do the checkpointing commands need to be executed on each node?

testing program:
mpi_hello_world.mana.exe

Linux distribution:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"

[xuyao0127]

You only need to run the mana_coordinator on one node. The mana_coordinator will create a temporary .mana.rc file in your home directory that includes the hostname and port number of the coordinator. Each MANA processes (on whichever nodes) will use the content read from the .mana.rc file to establish the connection with the coordinator.

The checkpoint command also needs to communicate. So you can either use mana_command -c on the same node as the coordinator, or manually include the hostname and port number when sending the checkpoint command from a different node.

Are you using SLURM? How do you run the coordinator and the test program?

[ggboom12138]

I only use mpirun to run ( not using SLURM ). If I run the mana_coordinator on only one node, there will be an ERROR "No MANA coordinator detected". It seems that the coordinator has not been started on another node.

COMMAND:
mpirun -np 2 -f hosts /home/share/mana/bin/mana_launch /home/share/mana/mpi-proxy-split/test/mpi_hello_world.mana.exe

[xuyao0127]

Can you check if there's a .mana.rc file in your home directory after run the mana_coordinator on one node?

[ggboom12138]

There's a .mana.rc file in my home directory.

[xuyao0127]

Can you add a --verbose flag to the mana_launch command, for example: mpirun -np 2 -f hosts /home/share/mana/bin/mana_launch --verbose /home/share/mana/mpi-proxy-split/test/mpi_hello_world.mana.exe?

mana_launch is a wrapper for dmtcp_launch. The --verbose flag will print the actual command executed, including the host and port number read from the .mana.rc file (-h and -p flags). If those values are invalid, we can look deeper into the issue.

[ggboom12138]

After updating the mana_v1.2.0 code, I can now run mana_launch successfully, but it requires running mana_coordinator on two nodes simultaneously. However, this also leads to issues with checkpointing, possibly due to having two coordinators.

[xuyao0127]

You only need one coordinator. I suspect your process on the other node doesn't have the correct hostname or port number from .mana.rc. It could be a bug in these scripts, or related to your environment setup. If you can provide the output of the --verbose flat, we can further narrow down the problem.

[ggboom12138]

I plan to run a 4-process MPI program on two nodes (node A & node B) using MANA.

step1: I run mana_coordinator --verbose on node A.
output:

• coordinator_found=0
• '[' '!' -z '' ']'
• MANA_RC=/home/share/.mana.rc
• /home/share/mana/bin/dmtcp_coordinator --exit-on-last -q --daemon --status-file /home/share/.mana.rc
• coordinator_found=1
• set +x
  *** Coordinator/job information written to /home/share/.mana.rc

step2: execute mpirun -np 4 -f hosts /home/share/mana/bin/mana_launch --verbose /home/share/mana/mpi-proxy-split/test/mpi_hello_world.mana.exe on node A。
output:
No MANA coordinator detected. Try:
/home/share/mana/bin/../bin/dmtcp_coordinator
Or:
/home/share/mana/bin/../bin/dmtcp_coordinator --exit-on-last -q --daemon
No MANA coordinator detected. Try:
/home/share/mana/bin/../bin/dmtcp_coordinator
Or:
/home/share/mana/bin/../bin/dmtcp_coordinator --exit-on-last -q --daemon

[ggboom12138]

Hello,
I have resolved this issue. Previously, the contents of the.mana.rc file on node B were different from those on node A.

Now I have another problem. I tried running WRF (Weather Research and Forecasting Model) using mana, but the process seems to be stuck. It looks like there might be an issue with the "jump operation" in lower-half.cpp. Do you know how to fix it? Or how to debug it?
#if defined(__i386__) || defined(__x86_64__)
asm volatile (CLEAN_FOR_64_BIT(mov %0, %%esp; )
: : "g" (dest_stack) : "memory");
asm volatile ("jmp *%0" : : "g" (ld_so_entry) : "memory");

[xuyao0127]

Thank you for reporting this. We will take a look on this bug.

[xuyao0127]

@ggboom12138 To understand the problem better here are some questions we want to ask:

1. Have you tested simple MPI programs in the test directory on single node?
2. For the WRF task, did you run on one node or multiple nodes?
3. When the process was stuck, is it during runtime or at checkpoint time?
4. Normally assembly instructions won't "stuck". Did you attach GDB to one of the process to check where did the program stuck? If call frames in GDB's backtrace is full of question marks, you can load GDB utils provided by MANA to load symbols for those call frame. To do that, you first need to run source mana/dmtcp/util/gdb-dmtcp-utils.py to load the utilization functions. Then run load-symbol-library to load missing symbols.

[ggboom12138]

1. Yes, I have tested simple MPI programs in the test directory on single node. These programs run successfully.
2. I ran the WRF task on one node.
3. When the process was stuck, it was during runtime. In fact, I observed that the actual WRF process was not running.
4. I used gdb for debugging and found that after asm volatile ("jmp *%0" : : "g" (ld_so_entry) : "memory");, the WRF process did not start running.

[xuyao0127]

@ggboom12138 Could you provide the version number of WRF so we can try to reproduce the bug?