scipy, numpy and other modules in pre/post processing scripts

[gfursin]

Hi all!
Here are my thoughts about TensorFlow problem with postprocessing scripts which has scipy/numpy ...

Once again it's a tricky situation since we call pre/post-processing scripts from the CK python installation, not from TensorFlow, etc. And we can't just mix CK python with other pythons from packages, since they can be different (!).

My general view on CK modules and pre/post-processing scripts within CK workflow space is that they should be very simple and portable without many dependencies on external modules, i.e. mainly just APIs.

However, if this is strictly necessary (we have a few cases already in stats), then this should be installed together with the CK, i.e. where we describe how to install python and python-pip, we should also ask to install extra dependencies needed for CK modules: scipy, numpy.

ON THE OTHER HAND, if we require complex functionality in the pre/post processing scripts from installed packages, we then use pre/post-processing scripts again just as simple and portable wrappers (we have deps resolved there usually) where we set up require deps and call a new and complex sub-script with all modules from an installed package via system call.

IN FACT, we can create a simple API in ck-env:module:os which will do it (and maybe pass input/output via some tmp JSON file) ...

AS A CONCLUSION:

1. for now we can just install missing scipy/numpy in Travis at the same time as we install python; we also note in ReadMe that users need to install them!
2. when someone has time, please try to create a function in ck-env:module:os (or env) to set up deps and call python module - in such case, we can move scipy, numpy from pre/post processing TF modules to new ones, and call them from pre/post processing modules via above API - it will set up env for installed TF, will call complex sub-module, and will collect all necessary info ...

If it's not clear, please tell me and we can discuss it further! Thanks!