SVD did not converge error when running fit_gcate

[KKKrick]

Hii,

I encountered an error when running fit_gcate on my dataset. Is there a recommended way to handle or avoid this issue? Thanks!
Here is the output and traceback:

'Estimating dispersion parameter...'
'Fitting poisson GLM with offset...'
{'d': 21, 'n': 14247, 'p': 3352, 'r': 10}
'Estimating initial latent variables with GLMs...'
'Fitting nb GLM with offset...'
100%|██████████████████████████████████████████████████████████████| 3352/3352 [00:18<00:00, 184.85it/s]
'Fitting GLM done.'
Traceback (most recent call last):
File "/home/kkk/R1/causarray_method.py", line 31, in
res_1, res_2 = fit_gcate(Y, X, A, r, verbose=True)
File "/home/kkk/anaconda3/lib/python3.9/site-packages/causarray/gcate.py", line 92, in fit_gcate
res_1, res_2 = estimate(Y, X, r, a,
File "/home/kkk/anaconda3/lib/python3.9/site-packages/causarray/gcate.py", line 142, in estimate
res_1 = alter_min(
File "/home/kkk/anaconda3/lib/python3.9/site-packages/causarray/gcate_opt.py", line 215, in alter_min
u, s, vt = svds(res_glm[-1], k=r)
File "/home/kkk/anaconda3/lib/python3.9/site-packages/scipy/sparse/linalg/_eigen/_svds.py", line 349, in svds
u, s, vh = svd(Av, full_matrices=False, overwrite_a=True)
File "/home/kkk/anaconda3/lib/python3.9/site-packages/scipy/linalg/_decomp_svd.py", line 131, in svd
raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge

[jaydu1]

Hi,

Thank you for your questions. Here are two possible reasons for the error:

• Have you tried filtering out low-quality cells and genes? Near-zero vectors can lead to numerical instability.
• Regarding the covariates, observed confounders, or the design matrix, could you check if the matrix is nearly singular?

Also, it would be helpful if you could share the data that is causing the issue, if it is publicly available.

[KKKrick]

Thank you for your suggestions. I have removed lowly expressed cells and genes, and also checked the condition numbers of both Y and A. They appear to be well-conditioned. It seems that the error is due to the SVD not converging when applied to the result of the GLM fit.

Here is one dataset that reproduces the error. Thank you for looking into this!

subset_example.zip

[KKKrick]

Below is the code I used to analyze this dataset:

import numpy as np
import pandas as pd
import scanpy as sc
import pickle
from causarray import prep_causarray_data, fit_gcate, LFC
import time

start_time = time.time()
adata_counts = sc.read_h5ad('data/subset_example.h5ad')
Y = pd.DataFrame(adata_counts.X.A.copy(), columns=adata_counts.var.index)

remove low-quality genes

Y = Y.loc[:, ~(Y == 0).all()]
col_sums = Y.sum(axis=0)
mask = (col_sums > 100) & (col_sums < 19500)
Y = Y.loc[:, mask]

A = pd.get_dummies(adata_counts.obs['condition'], columns=['condition'], drop_first=False).drop(columns=['control'])
Y, A, X, X_A = prep_causarray_data(Y, A)
Y = Y.astype(float)
A = A.astype(float)

#from causarray import estimate_r, plot_r
#df_r = estimate_r(Y, X, A, np.arange(5,20,5))
#df_r.to_csv('perturbseq_r.csv', index=False)
#df_r = pd.read_csv('perturbseq_r.csv')
#fig = plot_r(df_r)

r = 10
res_1, res_2 = fit_gcate(Y, X, A, r, verbose=True)
U = res_2['U']

save

with open("causarray_res1.pkl", "wb") as f:
pickle.dump(res_1, f)
with open("causarray_res2.pkl", "wb") as f:
pickle.dump(res_2, f)

offsets = np.log(res_2['kwargs_glm']['size_factor']) # use the precomputed size factors
df_res, estimation = LFC(Y, np.c_[X, U], A, np.c_[X_A, U], offset=offsets, verbose=True)

df_res.to_csv('causarray_df_res.csv')
estimation.to_csv('causarray_estimation.csv')

end_time = time.time()
print("Running time: {:.2f} seconds".format(end_time - start_time))

[jaydu1]

Hi, could you please let me know which version of the package and platform (windows/linux/macos) you are using? I tried running the code on my end, and everything seems to be working fine. I just want to see if this issue is related to the package version or if it is specific to your platform.

[KKKrick]

Hi,

Thank you for your reply. Here are the details of my environment:

causarray: 0.0.5
joblib: 1.1.0
numba: 0.60.0
numpy: 1.24.4
pandas: 2.2.3
scikit-learn: 1.0.2
scipy: 1.9.1
sklearn-ensemble-cv: 0.2.5
statsmodels: 0.13.2
tqdm: 4.64.1
Python version: 3.9.13
Platform: Linux Ubuntu 22.04

[KKKrick]

Hi! I upgraded my scipy package from version 1.9.1 to 1.12.0, and after this upgrade, the SVD problem was resolved. It seems likely that the issue was related to a bug or limitation in the older scipy version. Could you please let me know which scipy version you are using?

[jaydu1]

The version of scipy I am using is 1.15.0. I will need to include the version requirement for scipy in future releases. Thank you for reporting the issue!