Bug in uwot::umap

[Famingzhao]

Strange bugs occur when I use a big number cells (about 70k). But using a smaller cluster seems to be good running.

[parkjooyoung99]

Have you solved this problem??
I am having the same issue

[Laraine-Z]

Me too

[jlmelville]

uwot maintainer here. The only way I can generate that error message in uwot::umap is if the neighbor data in bbknn_out$nn.idx as passed here contains missing (NA) data:

bbknnR/R/bbknn.R

Line 218 in 83e9308

nn_method = list(idx = bbknn_out$nn.idx, dist = bbknn_out$nn.dist),

If that's the case, then something must have gone wrong when generating the nearest neighbor data in the run_bbknn function: most likely that the neighbor search is failing to find a sufficient number of neighbors for each item (cell). This can happen with functions in rnndescent with some choices of parameters, although it should return an index of 0 and not NA in that case (this would still be disastrous for UMAP though).

There is a bug in uwot in the sense that there is no helpful error message saying what has gone wrong. The next CRAN release of uwot will fix this. But I can't fix the underlying issue: with the same input data and parameters, you will still get an error at this point (it will just be more obvious why there has been a failure). The dense neighbor list-of-matrices format does not allow for missing values, so an error should happen here. For a better fix, @ycli1995 will need to make some changes to the code of this package.

My suggestions for fixing this are: if it's ok to have some missing values in bbknn_out$nn.idx (e.g. not every cell needs to have the same number of neighbors) , then the dense matrix format is the wrong data structure for that. uwot now supports sparse distance matrices to represent entries where the number of neighbors can vary, but uwot does not provide functions to convert between sparse and dense formats. For an example function that might help here, see the nng_to_sp function I defined in this article in uwot's documentation. You would then do something like:

dense_nn <- list(idx = bbknn_out$nn.idx, dist = bbknn_out$nn.dist)
sparse_nn <- nng_to_sp(dense_nn)
# in the call to `umap`:
umap <- umap( 
    nn_method = sparse_nn,
    # etc...
 )

Otherwise this requires fixing the neighbor search code to ensure the full neighbor graph is returned (or to stop early). Not something that I can fix in uwot, unfortunately.

[ycli1995]

Hi, @Famingzhao, @parkjooyoung99 and @Laraine-Z. Sorry for the delayed reply. Currently I cannot reproduce the issue with NA in the nn.idx, even if using the full pancreas dataset in Seurat website. Therefore the real cause for this issue is still unclear. Would you mind sharing your Seurat object so that I can try to figure out which part that inroduces the NA for nn.idx?