@concurrent

The first and most important decorator is @concurrent. This will get applied to the functions you want to be run in parallel, in separate processes or threads.

Basic usage

Using @concurrent on its own wraps the function it's applied to with calls to pool.apply_async. It changes the return value of the function to be a concurrent_result. These results have a .get() method that will return the real result of the function as soon as its ready. Using this decorator on its own could look something like this:

@concurrent
def process(data):
    ...

def process_data_set(data_set):
  concurrent_results = [process(data) for data in data_set]
  results = [result.result() for result in concurrent_results]
  return results

It's important to notice here that in the first line of process_data_set we don't call .result() on the results. It's critical that we initiate all the calls to process first, rather than waiting on each result one at a time.

Simplified Usage

If dealing with conccurent_results seems to complicated continue reading on the side bar section for @sychronized which can greatly simplify concurrent programming with deco.

Custom parameters

Any arguments passed to the @concurrent constructor will be passed to the constructor of the concurrency backend (usually Pool, whose documentation is here)

@concurrent(processes = 2)
def work():
    ...

Using threads

By default @concurrent will use multiple processes through the Pool API. It can also uses threads through the multiprocessing.threadpool API. The same rules for passing parameters applies, and ThreadPool mimic's the constructor of Pool.

@concurrent.threaded(processes = 2)
def work():
    ...