Nested concurrency

[aliabbasjp]

Is it possible to have a @Synchronised function whose body calls another function which is decorated with @Synchronised decorator?

from deco import *
from time import sleep
@concurrent
def bconc():
    sleep(3)
    return 1

@concurrent
def fconc():
    sleep(10)
    return 2
@synchronized
def bar():
    bd={}
    for i in range(5):
        bd[i]=bconc()
    return bd



@synchronized
def foo():
    bd=bar()
    fd={}
    for i in range(5):
        fd[i]=fconc()
    return fd,bd


print foo()

site-packages\deco\conc.py", line 57, in __call__
    out = compile(self.ast, "<string>", "exec")
TypeError: required field "lineno" missing from stmt

[alex-sherman]

Yeah this should definitely work. The error you're getting is from a bug in the latest release. I've just pushed the fix to pip, so if you pip install deco --upgrade everything should work fine.

[JackKenney]

Similarly, is there a way to have @Concurrent functions act like @synchronized functions and call @Concurrent functions?
In a triple-nested for loop situation (e.g. comparing L samples against M data points for N features)
I understand if it's a limitation of the parallel pooling, but if it's not that would be great.

Also I realized that nested for loops and filling in a 2d/3d/Nd matrix would work but the idea of child processes forking child processes is fairly common.

[alex-sherman]

I made a change to allow callables besides just functions as arguments to @concurrent. I tried the following:

from deco import *
import time

@concurrent
def inner(i):
    time.sleep(0.1)
    return i

@concurrent
@synchronized
def middle(j):
    results = {}
    for i in range(10):
        results[i] = inner(i * j)
    return sum([v for _, v in results.items()])

@synchronized
def outer():
    results = {}
    for j in range(10):
        results[j] = middle(j)
    return sum([v for _, v in results.items()])


if __name__ == "__main__":
    start = time.time()
    print(outer())
    print(time.time() - start)

Which produced the following multiprocessing error:

AssertionError: daemonic processes are not allowed to have children

I think I agree in the abstract that it's common for child executions to create more executions but I think it's generally not supported by spawning additionally processes.

There is probably a way for attempted concurrent calls in child processes to be instead communicated back to the main process and executed in the same pool, but it seems pretty complicated. I think deco is good only at making simple concurrency tasks very easy, and nested concurrency IMO beginsto leave the "simple" category anyway where you may just have to bite the bullet and use multiprocessing.pool directly.

I'll leave this open as a blue sky feature, but can't commit any time to it in the near term.

[JackKenney]

That all makes sense - I agree it starts to become exceedingly complicated.
FWIW, in multiprocessing you can start child processes that start mapped pools, but pools can't start other pools they have to be fully-fledged processes to do that and at that point the OS will start to complain.
Don't think it's worth a ton of time exploring tbh. But thanks for looking into it!