Resource manager changing resources when not needed. #209
Description
Right now, when a worker requests a resource, it's completely first come first serve, which can result in a worker changing the resource it uses between tasks. Which this is not directly a bug, it can cause undesired behavior with OTHER libraries that don't handle this well, such as torch keeping a memory reservation on a previous GPU, wasting GPU memory.
The current work around is to have a task run in a subprocess, but a better solution would be to use TLS/Process Local Storage to prefer to use the last used resource first, before picking the next available.