[important for pyafq] Call a function when jobs are complete

[36000]

This function could run on its own ec2 instance. This would be useful with pyAFQ, because we want to combine the tract profiles using the individual tract profiles stored on s3 after a cloudknot run is done.

[arokem]

Because the return value of the knot.map is a concurrent futures object, it has an add_done_callback method, which I believe does something very similar to what you want here, although, unless you rig this up to be more sophisticated, this would run on the calling machine, I believe.

There's an example usage here: https://github.com/nrdg/cloudknot/blob/master/examples/scipy-2018-paper-examples/monte_pi.ipynb

[richford]

Thanks @arokem for highlighting that functionality. @36000 if you want any help or pair programming for putting your combine csv function in the callback, I'm happy to help.

[36000]

I will put this on my todo list for this week. I think it is fine to happen locally although ec2 would be ideal cost-wise.

[richford]

I agree that it'd be nice to put the callback computation on the cloud as well. This is in keeping with "moving compute to data" rather than vice versa and is considerate to orgs like NDA and CMI that host a lot of neuroimaging data on the cloud.

I wonder if we could instantiate another knot in the global namespace, say knot2, and then put the knot2.map() call in the callback function. It's worth trying.

On the other hand, this is starting to look an awful lot like a workflow system, which I think is beyond the scope of cloudknot.

[36000]

I think the knot2 idea would work perfectly, actually. Is there syntax for only running one job instead of a list of jobs?

[richford]

Use map(*args, job_type="independent"). Although, really we should change cloudknot to look at the length of the args and automatically switch job_type to "independent" if len(args) == 1.