Parallel processing notes

Notes

Using celery and boto

• waiting() and complete_count() method calls can be expensive
• group methods can't track progress if ignore_result=True...

Running workers

• Set the prefetch multiplier (CELERYD_PREFETCH_MULTIPLIER) to 1 to prohibit many tasks from being reserved by a single worker
• Queue up tasks, then launch workers: better chance of rapidly consuming tasks sitting on queue
• Setting the broker visibility timeout to a low value seems to ensure messages are rapidly consumed ({'visibility_timeout' : 10})
• Use celery --autoscale=max,min to limit concurrent jobs (best with concurrency at least # processors)
• Running multiple workers on one machine doesn't seem to make a difference in how jobs are consumed

Observations...

• Run time constrained by I/O: Pickling/un-pickling
• Run time constrained by CPU: Matching template
• Memory not a limiting factor for t2.medium or better (maybe t2.small too?)
• t2.micro instances run out of memory on 8 MB Carrizo test case (duh)