MPI task scheduling

[nguu0123]

Currently we support MPI optask through srun but we still consider this as only 1 optask like following (controller:308) but in fact it should be consider as nodes * cpus_per_rank tasks and we need to find placement for all of them

        while not stop_event.is_set():
            try:
                logger.info("main loop")
                time.sleep(self.scheduling_interval)
                self.check_running_tasks()

                resources = self.resources_available()

                queue_dict = self.queue.dict()
                task_placement = self.algorithm.next_tasks(queue_dict, resources)

[rantahar]

The self.algorithm.next_tasks function should allocate the task on multiple nodes and take the requested number of ranks and CPUs into account. It will also allocate multiple tasks at a time, if the algorithm is designed that way.

There is an issue with slurm, though: if the number of nodes is not specified, the algorithms will assign only one rank per node. This is to make sure we reserve the same number of CPUs on each node. Setting he number of get's around this, but the whole process could be more general.

[maishivamhoo123]

@team can you please assign this issue to me?