Worker ID functionality

[pmla]

Hello loky team.

I would like to implement a "worker ID" function for loky. I originally wrote an issue and a pull request on the joblib github page. @tomMoral has kindly pointed me to the correct place for this PR.

@tomMoral: I have stripped this PR down so that it is relevant to loky and added a unit test. The worker ID assignment now accounts for available IDs with the process_worker_id variable. I'm not sure if I have accounted for all mechanisms by which processes can be ended though.

[tomMoral]

Thanks for the PR! I did a first pass with a bunch of comments.

I think the only thing that is not accounted for is the resize of the executor for smaller number of workers but I am not sure how reliable we can make this.
I think if it is too complex, we can have a go with this implem and document the fact that downsizing the size of the executor will leave you with unreliable worker_id.

One thing that would be great would be to add a docstring to get_worker_id explaining the rational and how to use it.

[tomMoral]

It would be nice to add a test where you make sure all ID in [0, max_worker) are given in an executor. You can use jobs that do not return before an event is set such as in tests/_test_process_executor.py-540:test_timeout.

[danielyan86129]

Any progress? Very useful feature!

[scottgigante-immunai]

Would love to see this merged!

[qianyizhang]

any progress of getting merged??

[tomMoral]

Thanks for the ping on this feature.
To make progress, the best would be to have people testing this feature extensively to see that it does not break.
One would need to rebase this one and see how it goes.

[tomMoral]

Thanks for the ping on this feature.
To make progress, the best would be to have people testing this feature extensively to see that it does not break.
I can try to rebase this one and see how it goes.

Bote that another implementation of this feature is in #285 (IIRC, I forgot this PR existed and give a try to make this feature, sorry for the duplication).

[qianyizhang]

Thanks for the ping on this feature. To make progress, the best would be to have people testing this feature extensively to see that it does not break. I can try to rebase this one and see how it goes.

Bote that another implementation of this feature is in #285 (IIRC, I forgot this PR existed and give a try to make this feature, sorry for the duplication).

Either implementation appears to be satisfactory, but I prefer this one as it introduces an environment variable rather than an attribute, which seems less complex. This feature does not alter any existing functionality, so please proceed with a release if feasible.