Variables in .GlobaEnv are copied to each thread when using the Extract function #3
Description
R version 3.6.1
To reproduce, create a large matrix (this one has about 4GB):
large_matrix<-matrix(1:000000000,1:10000000000)
Then try to use the Extract function with many cores:
Extract(FASTA_PATH="GE.fasta",CPU=3)
In a clean R session, after using devtools::load_all("GeneEssentiality"), each thread uses about 600 MB.
However, after creating the matrix and running the Extract(...), each thread will use about 4 GB.
Tried lots of things to solve this with no success:
Changed the enviroment of the function to base.env()
Changed the parallel::parLapply to use a function like parLapply(cl,SEQ,Calc_feats)
Tried with %dopar%
Tried changing to parLapply to parSapply
No matter what is done, it keeps copying something that is not even named in the functions. It simply imports all things from ancestor enviroments, it seems.
The workaround is to extract features in a clean session and save them. Just do not try to use Extract with many cores if the session has large objects
________Session Info
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /mnt/DATABASES/anaconda3/envs/giovannimc/lib/R/lib/libRblas.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] GeneEssentiality_0.0.1.0000
loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 lubridate_1.7.9 lattice_0.20-38
[4] prettyunits_1.1.1 class_7.3-15 ps_1.3.3
[7] assertthat_0.2.1 rprojroot_1.3-2 digest_0.6.25
[10] ipred_0.9-9 foreach_1.5.0 R6_2.4.1
[13] plyr_1.8.6 backports_1.1.8 stats4_3.6.1
[16] ggplot2_3.3.2 pillar_1.4.6 rlang_0.4.7
[19] caret_6.0-86 rstudioapi_0.11 data.table_1.13.0
[22] callr_3.4.3 rpart_4.1-15 Matrix_1.2-17
[25] desc_1.2.0 devtools_2.3.1 splines_3.6.1
[28] protr_1.6-2 gower_0.2.2 stringr_1.4.0
[31] munsell_0.5.0 compiler_3.6.1 pkgconfig_2.0.3
[34] pkgbuild_1.1.0 nnet_7.3-12 tidyselect_1.1.0
[37] tibble_3.0.3 prodlim_2019.11.13 codetools_0.2-16
[40] randomForest_4.6-14 fansi_0.4.1 crayon_1.3.4
[43] dplyr_1.0.0 withr_2.2.0 MASS_7.3-51.3
[46] recipes_0.1.13 ModelMetrics_1.2.2.2 grid_3.6.1
[49] nlme_3.1-139 gtable_0.3.0 lifecycle_0.2.0
[52] magrittr_1.5 pROC_1.16.2 scales_1.1.1
[55] cli_2.0.2 stringi_1.4.6 reshape2_1.4.4
[58] fs_1.5.0 remotes_2.2.0 doParallel_1.0.15
[61] testthat_2.3.2 timeDate_3043.102 seqinr_3.6-1
[64] ellipsis_0.3.1 generics_0.0.2 vctrs_0.3.2
[67] lava_1.6.7 iterators_1.0.12 tools_3.6.1
[70] ade4_1.7-15 glue_1.4.1 purrr_0.3.4
[73] parallel_3.6.1 processx_3.4.3 pkgload_1.1.0
[76] survival_2.44-1.1 colorspace_1.4-1 sessioninfo_1.1.1
[79] memoise_1.1.0 usethis_1.6.1