ERROR(123): MPI exchange of gas species densities (CODE=1) timed out for MPI process 28 running on ******* will abort.

[gaurav24khantwal-bit]

Describe the bug
I am getting an error while running the simulation.

ERROR(123): MPI exchange of gas species densities (CODE=1) timed out for MPI process 22 running on ****. FDS will abort.

ERROR(123): MPI exchange of gas species densities (CODE=1) timed out for MPI process 17 running on ****. FDS will abort.

Abort(1) on node 17 (rank 17 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 17
Abort(1) on node 22 (rank 22 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 22

ERROR(123): MPI exchange of gas species densities (CODE=1) timed out for MPI process 19 running on ****. FDS will abort.

Abort(1) on node 19 (rank 19 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 19

ERROR(123): MPI exchange of gas species densities (CODE=1) timed out for MPI process 14 running on ****. FDS will abort.

Abort(1) on node 14 (rank 14 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 14

The simulation was setup for 1200 seconds but it got terminated at 54 seconds. The fds file is very large as I am simulating using PyroSim 2025.

[gaurav24khantwal-bit]

please tell me what could be the possible reasons for this error. I am simulating for smoke movement in case of vehicle fire in a basement. The domain contains sprinklers; series of nozzles representing water curtains; and multiple vents for smoke extraction and fresh air supply. I have considered the slices for CO concentration, visibility, temperature along the z plane; and gas phase devices for CO sensors at multiple locations.

[marcosvanella]

Post the input file and I'll run the case here.

[gaurav24khantwal-bit]

Hi @marcosvanella thanks for the prompt response. The work I am doing is part of a consultancy project for which I have signed a non-disclosure agreement. Therefore, I cannot share the FDS file with you. I can provide details of the devices and objects I am using, as mentioned in my previous comment. Could you please tell me the possible reasons that may cause this error (Code = 1)?

[mcgratta]

The error message says that the exchange of gas temperatures and species at mesh boundaries during the first half of the time step failed. We cannot tell why because that information is not returned by the computer.

One thing to consider. Are your meshes all about the same size? If not, it is possible that the large mesh took much longer to process than the small ones leading to a bottleneck. Also, are there sprinklers activated such that one or two meshes are slower than the rest? Does the error repeat if you run the case again?

[drjfloyd]

If you are modeling underground parking, unless your building is very unique, underground parking looks all the same. If I just gave you a few plan drawings of underground parking with no identifying information, it is unlikely you would be able to identify the building. The FDS model doesn't have framing structural details, concrete strength, part numbers for equipiment, etc. It is just the basic geometry and boundary conditions. If Kevin's questions do not lead to success, you may want to check if an anonymized input file could be shared.

[marcosvanella]

@gaurav24khantwal-bit, as replied before. Check that your meshes have about the same number of cells, and that that number stays below ~200k. Ideally you want your meshes 40^3 to 50^3. This way you will have the parallel work done more or less uniformly across your MPI processes.
Without an input file there is not much we can do. So if the problem persists, see if you can send in a similar input file that demonstrates the problem as Jason suggested.

[gaurav24khantwal-bit]

The error message says that the exchange of gas temperatures and species at mesh boundaries during the first half of the time step failed. We cannot tell why because that information is not returned by the computer.

One thing to consider. Are your meshes all about the same size? If not, it is possible that the large mesh took much longer to process than the small ones leading to a bottleneck. Also, are there sprinklers activated such that one or two meshes are slower than the rest? Does the error repeat if you run the case again?

The error repeats at the same time- step again. Also the error is coming before the sprinkler activation.

[mcgratta]

Can you run the case with the latest nightly build of FDS ?

[gaurav24khantwal-bit]

@gaurav24khantwal-bit, as replied before. Check that your meshes have about the same number of cells, and that that number stays below ~200k. Ideally you want your meshes 40^3 to 50^3. This way you will have the parallel work done more or less uniformly across your MPI processes. Without an input file there is not much we can do. So if the problem persists, see if you can send in a similar input file that demonstrates the problem as Jason suggested.

Okay. I'll try this. If I didn't find the solution, I'll share the fds file. What do you mean by 40^3 to 50^3? Which number is it- total number of mesh?

[mcgratta]

IJK=40,40,40 or IJK=50,50,50

[gaurav24khantwal-bit]

Can you run the case with the latest nightly build of FDS ?

I am using PyroSim to run my simulations. I am new in this field.

[gaurav24khantwal-bit]

IJK=40,40,40 or IJK=50,50,50

Okay.. In my case the height of the basement is about 6 m but its floor is about 400 by 200 m. It is not possible to keep IJK in this range.

[marcosvanella]

What mesh sizes are you using now? what is the max IJK you see in your &MESH lines?