CLICEO provides robust handling of worker pools from the standart library's multiprocessing module. It was written with
two use cases in mind:
- Having workers launch potentially long-running command line processes using the standard library's
subprocessmodule, and - Running workers in an interactive shell (e.g.
ipython)
This project was borne out of frustration with constant manual cleanup of zombie processes and the clutter of files they created when they
terminated incorrectly -- particularly after ctrl-C / KeyboardInterrupt. As such, the main components are:
- The
CLIcontextManagerclass, which can create temporary files and directories which are cleaned up automatically - The
CommandLineCallerclass, a generic CLI controller which runs a command in the CLI context and handles output capture - The
WorkerandPoolManagerclasses which combine to track and, when necessary, kill PIDs, and return tracebacks from workers to the parent process to simplify debugging
CLICEO hasn't been updated from Python 2 because it's been a very long time since I've needed to run hundreds sequence alignments and phylogenetic reconstructions in parallel. It was a life saver while I was developing ASPEN.