Documentation on interaction of tensorflow with LAW.

[acampove]

Hi,

I tried to run our code with a law workflow and I oberved that, it got stuck with:

3 running tasks, waiting for next task to finish

It seems that there are issues between tensorflow and how this package handles parallelization through processes. The import of tensorflow needs to be done in the run method, i.e.:

    def run(self):
        """Run the `compare` command for one set of wildcards."""
        from rx_plotter_scripts.compare import main      as compare 

        cfg = self._get_config() 
        compare(settings=cfg)

where rx_plotter_scripts imports tensorflow. I cannot find this sort of stuff anywhere and it seems that the issue stems ultimately from tensorflow, because if the parts that use tensorflow are turned off, the process just works.

Is this known? Could this be bypassed to simplify things for the user? If not, could this be documented better?

Cheers.

[riga]

Hi @acampove ,

thanks for opening the issue.

The situation you describe sounds indeed like an interference between tensorflow's (quite spendy) multi-processsing settings (inter- and intra-op) and luigi's scheduling.

It reads quite like an edge case, so documenting this is of course not as simple.

However, maybe the force_multiprocessing setting in the worker section does the trick. I added that a while ago to luigi to enforce multi-processing to be used by luigi independent of the number of --workers you pass, i.e., even for a single worker, luigi would use mp to encapsulate the processing of each task. This could fence tensorflow and hopefully also handle it's unloading at the end of the run process. You just need to make sure that you import tensorflow inside the run method.

Could you give that a spin? If you're using a combined law/luigi config, you just need to add

[luigi_worker]
force_multiprocessing: True

to your law.cfg file.