Respect the cgroup when setting affinity

[rantahar]

Slurm (and probably other schedulers) set a cgroup for the job. There is no need to look at monitoring data outside this cgroup.

Use the affinity list to remove CPUs from the free_cpus list.

[nguu0123]

There is an allowed cpu list from the obs for each task in the node, it's in the metadata:
metadata.allowed_cpu_list,"[49, 50, 51, 52, 53, 54, 55]"

[rantahar]

How is that discovered? Juhana suggested reading the cgroup files from /sys/fs/cgroup. I wrote a quick example in odop/engine/cgroup.py

[rantahar]

Cgroups are not specific to slurm, We need to respect them on any Unix system.

[nguu0123]

for each process, I check os.sched_getaffinity(pid) to get the cpu affinity, the cgroup for the whole job will be the aggregation of those cpu affinity and the second thread of such core (core x and core x+64)