Labelling grid decomposition by MPI rank

[johnomotani]

For code coupling, it may be useful to record the MPI domain decomposition in the grid file. It should be a relatively simple addition to do this in the same way that BOUT++ does.

Specifying the domain decomposition at the grid generation stage is a bit of a restriction, but probably an acceptable one in many use cases. In principle one could list all the allowed MPI decompositions - there will not usually be a ridiculous number of them, although there is the risk of a combinatorial explosion for very large grids...

Proposal

Add NPES (total number of MPI ranks), NXPE (number of ranks in the radial, $x$-direction), and NYPE (number of ranks in the poloidal/parallel, $y$-direction) options to the BoutMesh class here (see for example the Mesh class for examples of the syntax here. These options are sufficient to determine the domain decomposition according to BOUT++. Possible settings:

• "none" (default) the current behaviour which does not write any domain decomposition information
• Specified values for any NPES or two of NPES, NXPE, NYPE. If only NPES is given, choose NXPE in the same way as BOUT++. If all three are given, error if NPES == NXPE * NYPE is not satisfied.
• (Optional extension) "all" given the grid settings, generate all allowed values for NYPE (which depend on the numbers of grid points in each poloidal section of the grid) and NXPE (which are any factors of nx) and give the outputs as 1d arrays rather than single integers.

Calculate the MPI rank corresponding to each grid point/cell and save in an array (e.g. MYPE to match BOUT++ terminology). MYPE will be a 2d array, or maybe 3d if the "all" option is supported - also write NPES, NXPE, NYPE for convenience. If only NPES is given, follow the logic in https://github.com/boutproject/BOUT-dev/blob/9b6ce3ac6d159307094571fc61a62757ff3d95a0/src/mesh/impls/bout/boutmesh.cxx#L282-L315. If NXPE or NYPE are also given, follow https://github.com/boutproject/BOUT-dev/blob/9b6ce3ac6d159307094571fc61a62757ff3d95a0/src/mesh/impls/bout/boutmesh.cxx#L241-L280. The check for whether a certain pair of NXPE and NYPE values is valid is https://github.com/boutproject/BOUT-dev/blob/9b6ce3ac6d159307094571fc61a62757ff3d95a0/src/mesh/impls/bout/boutmesh.cxx#L165-L238. See also the (maybe more human-readable) description in the docs https://bout-dev.readthedocs.io/en/stable/user_docs/bout_options.html#grids. Note that hypnotoad already calculates the BOUT++ topology indices ixseps1, jyseps1_1, etc., so suggest to add the features suggested in this issue after the topology indices are calculated.

I guess one could also modify BOUT++ to read NPES, NXPE, NYPE from the grid file if they are present, presumably throwing an exception if the NPES does not match the actual number of MPI ranks used, and falling back on settings from BOUT.inp (the current implementation) when they are not present. If this is not done, it would be up to the user to make sure their BOUT.inp is consistent with the grid file. An intermediate step might be for PhysicsModel codes where this feature is needed (e.g. Hermes-3 when coupled to some other code) to do a consistency check that BOUT++'s NPES, NXPE, NYPE match the values in the grid file - this would allow the values in the grid file to be ignored if the grid file is used by a code that does not need them (not sure that is a particularly useful feature though!).

Ping @mikekryjak @mrhardman @will-saunders-ukaea @jwscook.