Skip to content

Feature request: CPUDistributingLocalScheduler with separate processes per hyperparam. run #79

@Flunzmas

Description

@Flunzmas

Disclaimer: requires a feature only available from python>=3.11.

What: Initialize the concurrent.futures.ProcessPoolExecutor used in CPUDistributingLocalScheduler with the new parameter max_tasks_per_child to control the amount of tasks (~hyperparameter runs) per process:

with concurrent.futures.ProcessPoolExecutor(
            max_workers=num_parallel,
            max_tasks_per_child=1 if self.one_process_per_run else None
        ) as pool:
...

If max_tasks_per_child is None, the scheduler behaves as usual. If provided a number, each child process will terminate after the specified amount of runs. If this number is 1, each hyperparam. run will run in its own process.

Why: This feature is useful for programs that you'd like to distribute over CPUs, but need to be in separate processes each (i.e. you can't reuse processes for running multiple runs sequentially).

How: Add a yaml config bool attribute one_process_per_run that, when using scheduler: "cpu_distribute", is passed to the CPUDistributingLocalScheduler and handed on to the concurrent.futures.ProcessPoolExecutor as indicated above (link to location in code).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions