a problem may exist in BPPCA

[likang7]

I've looked through your code and the paper Variational principal components.
And I found you compute exception such as <W.T.dot(W)> by tr(q.w_mean).dot(q.w_mean).
However, for gaussian distribution, there is <x.dot(x.T)> = cov + mu.dot(mu.T). So I think you should add the cov term?
Am I right?
I am doing the same job right row, but I found the variational lower bound decrease sometimes, hope that you can have a try. ^_^

[cangermueller]

To which line to you refer? Can you update the relevant code and let me know if you observe any difference?

[likang7]

for example. in update w(line 75), you compute E_q[x.dot(x.T))] by
x_cov += x[:, np.newaxis].dot(np.array([x])), in which x = q.x_mean[:, n]
I think the true form should be
x_cov += x[:, np.newaxis].dot(np.array([x])) + q.x_cov

Similar situation happens in update_x(line 66), update_alpha(line 93), and update_gamma(line 100, 104).

The mse score seems to be ok, but if you try to calculate the variational lower bound L(Q), you will find it decrease sometimes.
I test it by using the first 30 lines data of 'digits' in scikit learn datasets, and set q = 16.

By the way, I think you forget to multiply 0.5 at the end of update_gamma when updating q.gamma_b

[likang7]

Hi, I found the lower bound didn't decrease after adding the cov term, but the ARD fails now...
Do you have any idea? I'm confused now...

[cangermueller]

How to you evaluate ARD? Are you sure that you implemented all updates and the lower bound correctly? Do you have a link to your repository?

[likang7]

I evaluate ARD through constructing data using this function:

def simulate():
    N = 50
    P = 10
    K = 3

    x = np.random.normal(0, 1, size=(N, K))
    alpha = np.random.gamma(1, 1, size=(K))
    W = np.empty((P, K))
    for k in xrange(K):
        W[:, k] = np.random.multivariate_normal(np.zeros(P), (alpha[k]**-1) * np.eye(P))
    y = x.dot(W.T)

    return y

Then I set q = 9, and draw the Hinton graph.(using the code described here)
There should be 3 column show but:

By the way, I'm not clearly how to construct the dataset describing in the paper...Do you have any idea?

You can check my repository through https://github.com/likang7/vppca
The main difference between our implementation is that I add the cov term describe above, and multiply 0.5 when updating b_psi(gamma_b in you version).

I derive the lower bound with the help of the following thesis:
Variational Approach to Factor Analysis and Related Models (Page 31~32)

[cangermueller]

Thanks for the reference, this is really useful! Did you try different update orders? Unfortunatly, variational inference is often quite sensitive with respect to the update order and initialization. Did you play around with the hyperparameters?

[likang7]

I try all these but it seems that they're not the problem.
I can't find out what's wrong with the program, and I think I must misunderstand about something...
If you have further discovery, please tell me.

[jaydangar]

This Code is giving me this error :

Traceback (most recent call last):
File "New.py", line 68, in
ppca = pca.ppca.PPCA()
AttributeError: 'module' object has no attribute 'PPCA'

I have imported pca but it is still giving me this error.

[cangermueller]

I updated src/pca/__init__.py to use relative imports. Try it again. Note that you have to add src/pca to you path, e.g. using `os.sys.path.append('./src/') as in the notebook. Also note that the code only works with python2.