Installing svae: any instructions?

[finmod]

Any instructions on how to install svae? A similar to autograd and simple install like

pip install svae or pip install https://github.com/mattjj/svae/archive/master.zip

would be appreciated.

Thanks

[mattjj]

Hey, thanks for your interest.

This code is a work-in-progress research project and it's probably not going to be clear or easy to use for a while. That said, you can build the code by running this in the git root:

python setup.py build_ext --inplace

You'll need cython and an OpenMP-compatible compiler (i.e. GCC, ICC, or a very recent Clang).

There may also be some dependencies on my other libraries, most of which are pip installable from pypi. It also depends on autograd (and autograd internals).

So it'll probably be easier to poke around here later, but in the meantime I can try to answer any questions.

[finmod]

Thank you for the prompt reply. For now, I will wait until the dust settles. It is clear that svae has a vocation to be integrated as an autograd example/model when debugged.

I have raised HIPS/autograd#97 about the deepGP example in autograd and I would be glad to solve #97 first as a preliminary step to svae.

[wuaalb]

FWIW, I could install svae without much problems on my system (Anaconda/Windows), although I haven't tried running all the experiments yet..
I just had to add #include <cmath> before Python.h in the .cpp files of the pyhsmm dependency (to avoid 'hypot not defined' errors).

[mattjj]

Great to hear!

Things are still a bit barebones here, but in the near future I hope to make the examples more interesting and easily runnable. At the moment the code is also not very approachable, but we have plans to improve it.

As a teaser, here are a couple animations of SVAE training: a GMM SVAE and an LDS SVAE. The LDS example is explained in the current arXiv version of the paper, and the GMM example will probably appear in an updated version.

[wuaalb]

Thanks for taking the time to provide the source code; and those animations are looking great.

I ran the lds_svae_dots.py experiment and the running time was higher than I expected.
I terminated after running for some 12+ hours and it got up to producing dots_289.png (which looked like it would take a while longer to converge). I also noticed the process was using 7+ GB of memory (although this wasn't increasing significantly over time).

Is this normal? I guess it could be related to using Windows; I'm not at all familiar with autograd either..

Out of curiosity; would it be feasible (and sensible) to implement something like this in Theano for instance? I'm not totally sure what the training procedure entails precisely (this kind of paper takes me a while to get through), but is there anything fundamental that make it unsuitable for such a framework (missing ops, prohibitively long compile times, ..)?

[mattjj]

That's not normal; that example only takes a few minutes on my machine, and doesn't take much memory.

There may be something up with building on Windows (or your specific platform), or I may have left the demo in a broken state, or maybe it's something else. It might have even been the drawing code that was slow for you. The demos will be more runnable in the future, but they're not there yet.

Re Theano, I actually started with Theano but it has some disadvantages for this kind of code, which uses a lot of linear algebra and also has long chain computation graphs. For example, LDS message passing uses a bunch of Cholesky factorizations. Search for the string 'dpotrf' in here:

http://developer.download.nvidia.com/compute/cuda/compute-docs/cuda-performance-report.pdf

Even on huge 4096 x 4096 matrices, the optimized cuSolverDN Cholesky implementation is only marginally faster than a CPU MKL version. With this SVAE code we need to compute a lot of small (say 10 x 10 or 20 x 20) factorizations, and I suspect the MKL version (or an Eigen version) on the CPU would excel there. The story with the other operations is similar, though the dpotrf and a few dgemm's are the only operations with complexity that is cubic in the state dimension.

Theano doesn't generate calls to cuSolverDN on the GPU yet, as far as I can tell, and its gradient computation relies on Theano-compiled explicit loops (which again I suspect MKL would easily outperform, though I haven't actually tested it):

https://github.com/Theano/Theano/blob/e7f6b32e08075a88f66748c89be79bf2264fe3d0/theano/tensor/slinalg.py#L71-L133

I suspect the structure of the algorithms might lead to long compile times as well.

That said, I haven't really tried a complete Theano implementation; an end-to-end Theano version could work well, and maybe it's already gotten better at this stuff since I experimented with it.

The approach I took instead was to write some highly optimized Cython code for message passing (and differentiating through message passing). Read through the several .pxd and .pyx files to get a sense for it. I also used some autograd internals and added a half-baked feature to them.

EDIT: this tweet suggests that TF Cholesky-like operations on the GPU might cross over for ~100x100 matrices.

[wuaalb]

Thank you for your detailed answer.

I'll wait until the repo matures a little.
Just FYI I'll post the results of running the other experiments on my machine here:

• gmm_svae_pinwheel OK
• gmm_svae_synth OK, took around 1h to complete; plots look similar to the animation you posted
• lds_svae_dots see above; plots do not look like the animation you posted at all
• lds_svae_synth shape mismatch error (1800,2) (900,2) in h = J * mu recognition_models.py#L29
• slds_svae_synth OK, took < 10 mins to complete
• vanilla_vae_mnist OK, but I killed it after generating > 10k pngs, couldn't easily tell from code when it was supposed to stop

[pauloabelha]

Hi all, nice to find this thread,

I'm struggling to install and use the package. All I managed was to run (after messing around with paths and imports) was:
python experiments/gmm_svae_synth.py

How can I install SVAE and run all experiments? I have been trying to install as an egg after running build and install in setup.

I could only find two experiments inside the experiments folder:
gmm_svae_synth and gmm_svae_synth_plot_pdfs.py

Are those other experiments mentioned in this thread in older versions of the repo?

Thank you and kind regards,