SWAN+WRF problem when SWAN use multiple processors

[nuo-231]

Hello, John

When I use wrf+swan coupling, there is no problem when the number of processor cores on a node is 36, but an error will occur when more than 36 cores are used

This is my error report

[nuo-231]

PRINT01-001.txt

[nuo-231]

slurm-2442105.txt

[NicJeffress]

Hi,

I don’t work directly with WRF, but in SWAN I’ve hit memory limits when running large grids with MPI. One quick diagnostic could be to try a SWAN-only run using your target processor count to isolate whether memory is the limiting factor. For reference, a while back I tested a 705×380 grid with 36 directional bins and 20 frequency bins. My “limits” weren’t very precise—I found them by trial and error, bumping the requested memory each time a run failed.

cpus | Mem
2 | <20G
4 | <50G
8 | <20
10 | <50 G
20 | <50 G
30 | <100
40 | 50G-100G
50 | <100

When I bumped the domain to 1121×921x36x20, 68 CPUs was the max I could use on a ~350 GB node. Even though there were more cpus available, runs with more CPUs failed due to memory issues.

cpus | Memory
12 | <300 G
34 | <350 G (max)
68 | <350 G (max)

I’m not sure which compilers you’re using; when I used GNU compilers, SWAN hung at the first compute statement when it ran out of memory, whereas with Intel compilers it produced a memory-error message.

Hope this helps—good luck with the modeling!

[nuo-231]

Thank you for your reply. I'm using an intel compiler. My swan grid size is 288*377. Currently, I've tested that it can run with 45 cores. Is the cause of this issue due to the swan compilation problem

[nuo-231]

@jcwarner-usgs

[jcwarner-usgs]

Sorry for the delay. We are back to work now.
For the tiling, SWAN will take the longest dimension (377) and divide by the number of cores. 377/80 = 4.7. so it will try to use 4 columns for each tile.
Each tile must have some active points in it (some points that are not land masked).
Are the edges of your grid land masked?

[nuo-231]

Yes, my simulation area has a large land portion. Now I can run it with fewer cores, but I just can't use more than a certain number of cores. My problem is temporarily solved.

[nuo-231]

I have another question: if my SWAN uses a regular input grid, how should I go about creating the scrip file when coupling it with WRF?

[jcwarner-usgs]

the Sandy case is an example of coupling roms - wrf - swan. There are example scrip input files in Lib/SCRIP_COAWST and the user manual has documentation to set this up.