segfault error, cause "unknown"

[sevachkaa12]

Hello! I am trying to run the sliding window chromunity pipeline on Pore-C data in Drosophila, but I am not super familiar with computational analysis. I am running the pipeline on our university's HPC through a singularity container that contains the core R packages and dependencies. I wrap the Rscript in an sbatch script as follows:

module load singularity

singularity exec \
  --bind /sandbox:/sandbox \
  [path/to/.sif image] \
  Rscript [path/to/Rscript/chromunity_test_merge.R]

The R script is as follows:

suppressPackageStartupMessages(library(rtracklayer))
suppressPackageStartupMessages(library(GenomeInfoDb))
suppressPackageStartupMessages(library(skitools))
suppressPackageStartupMessages(library(chromunity))
suppressPackageStartupMessages(library(MASS))
suppressPackageStartupMessages(library(gUtils))

parquet_gr <- readRDS("[path/to/RDS/merged_gr.RDS]")
parquet_gr$cid <- parquet_gr$read_idx

this_sliding_chrom = sliding_window_chromunity(concatemers = parquet_gr, resolution = 5e4, window.size = 2e6, 
                                               take_sub_sample = FALSE, chr  = "DAWXEX010000003.1", subsample.frac = NULL, mc.cores = 1, genome = NULL)

head(this_sliding_chrom$concatemers)
head(this_sliding_chrom$binsets)

The out file shows it working until it catches the segfault error:

Warning message:
In valid.GenomicRanges.seqinfo(x, suggest.trim = TRUE) :
  GRanges object contains 29964 out-of-bound ranges located on sequences
  DAWXEX010000096.1, DAWXEX010000056.1, DAWXEX010000097.1, DAWXEX010000069.1,
  DAWXEX010000130.1, DAWXEX010000226.1, DAWXEX010000026.1, DAWXEX010000012.1,
  DAWXEX010000098.1, DAWXEX010000143.1, DAWXEX010000137.1, DAWXEX010000002.1,
  DAWXEX010000145.1, DAWXEX010000055.1, DAWXEX010000160.1, DAWXEX010000135.1,
  DAWXEX010000114.1, DAWXEX010000059.1, DAWXEX010000161.1, DAWXEX010000209.1,
  DAWXEX010000185.1, DAWXEX010000188.1, DAWXEX010000151.1, DAWXEX010000025.1,
  DAWXEX010000029.1, DAWXEX010000054.1, DAWXEX010000016.1, DAWXEX010000043.1,
  DAWXEX010000167.1, DAWXEX010000032.1, DAWXEX010000046.1, DAWXEX010000088.1,
  DAWXEX010000083.1, DAWXEX010000122.1, DAWXEX010000169.1, DAWXEX010000216.1,
  DAWXEX010000006.1, DAWXEX010000074.1, DAWXEX010000007.1, DAWXEX010000200.1,
  DAWXEX010000044.1, DAWXEX010000076.1, DAWXEX010000110.1, DAWXEX010000118.1,
  DAWXEX010000078.1, DAWXEX010000080.1, DAWXEX010000091.1, DAWXEX010000 [... truncated]
Chromunity 2026-01-15 21:09:25.613025: Generated 3068 bins across 26 windows
Processing 1 to 1000000
Processing 1000001 to 2000000
Processing 2000001 to 3000000
Processing 3000001 to 4000000
Processing 4000001 to 5000000
Processing 5000001 to 6000000
Processing 6000001 to 7000000
Processing 7000001 to 8000000
Processing 8000001 to 9000000
Processing 9000001 to 10000000
Processing 10000001 to 11000000
Processing 11000001 to 12000000
Processing 12000001 to 13000000
Processing 13000001 to 14000000
Processing 14000001 to 15000000
Processing 15000001 to 16000000
Processing 16000001 to 17000000
Processing 17000001 to 18000000
Processing 18000001 to 19000000
Processing 19000001 to 20000000
Processing 20000001 to 21000000
Processing 21000001 to 22000000
Processing 22000001 to 23000000
Processing 23000001 to 24000000
Processing 24000001 to 25000000
Processing 25000001 to 26000000
Processing 26000001 to 27000000
Processing 27000001 to 28000000
Processing 28000001 to 29000000
Processing 29000001 to 30000000
Processing 30000001 to 31000000
Processing 31000001 to 32000000
Processing 32000001 to 33000000
Processing 33000001 to 34000000
Processing 34000001 to 35000000
Processing 35000001 to 36000000
Processing 36000001 to 37000000
Processing 37000001 to 38000000
Processing 38000001 to 39000000
Processing 39000001 to 40000000
Processing 40000001 to 41000000
Processing 41000001 to 42000000
Processing 42000001 to 43000000
Processing 43000001 to 44000000
Processing 44000001 to 45000000
Processing 45000001 to 46000000
Processing 46000001 to 47000000
Processing 47000001 to 48000000
Processing 48000001 to 49000000
Processing 49000001 to 50000000
Processing 50000001 to 51000000
Processing 51000001 to 52000000
Processing 52000001 to 53000000
Processing 53000001 to 54000000
Processing 54000001 to 55000000
Processing 55000001 to 56000000
Processing 56000001 to 57000000
Processing 57000001 to 58000000
Processing 58000001 to 59000000
Processing 59000001 to 60000000
Processing 60000001 to 61000000
Processing 61000001 to 62000000
Processing 62000001 to 63000000
Processing 63000001 to 64000000
Processing 64000001 to 65000000
Processing 65000001 to 66000000
Processing 66000001 to 67000000
Processing 67000001 to 68000000
Processing 68000001 to 69000000
Processing 69000001 to 70000000
Processing 70000001 to 71000000
Processing 71000001 to 72000000
Processing 72000001 to 73000000
Processing 73000001 to 74000000
Processing 74000001 to 75000000
Processing 75000001 to 76000000
Processing 76000001 to 77000000
Processing 77000001 to 78000000
Processing 78000001 to 79000000
Processing 79000001 to 80000000
Processing 80000001 to 81000000
Processing 81000001 to 82000000
Processing 82000001 to 83000000
Processing 83000001 to 84000000
Processing 84000001 to 85000000
Processing 85000001 to 86000000
Processing 86000001 to 87000000
Processing 87000001 to 88000000
Processing 88000001 to 89000000
Processing 89000001 to 90000000
Processing 90000001 to 91000000
Processing 91000001 to 92000000
Processing 92000001 to 93000000
Processing 93000001 to 94000000
Processing 94000001 to 95000000
Processing 95000001 to 96000000
Processing 96000001 to 97000000
Processing 97000001 to 98000000
Processing 98000001 to 99000000
Processing 99000001 to 100000000
Processing 100000001 to 101000000
Processing 101000001 to 102000000
Processing 102000001 to 103000000
Processing 103000001 to 104000000
Processing 104000001 to 105000000
Processing 105000001 to 106000000
Processing 106000001 to 107000000
Processing 107000001 to 108000000
Processing 108000001 to 109000000
Processing 109000001 to 110000000
Processing 110000001 to 111000000
Processing 111000001 to 112000000
Processing 112000001 to 113000000
Processing 113000001 to 114000000
Processing 114000001 to 115000000
Processing 115000001 to 116000000
Processing 116000001 to 117000000
Processing 117000001 to 118000000
Processing 118000001 to 119000000
Processing 119000001 to 120000000
Processing 120000001 to 121000000
Processing 121000001 to 122000000
Processing 122000001 to 123000000
Processing 123000001 to 124000000
Processing 124000001 to 125000000
Processing 125000001 to 126000000
Processing 126000001 to 127000000
Processing 127000001 to 128000000
Processing 128000001 to 129000000
Processing 129000001 to 130000000
Processing 130000001 to 131000000
Processing 131000001 to 132000000
Processing 132000001 to 133000000
Processing 133000001 to 134000000
Processing 134000001 to 135000000
Processing 135000001 to 136000000
Processing 136000001 to 137000000
Processing 137000001 to 138000000
Processing 138000001 to 139000000
Processing 139000001 to 140000000
Processing 140000001 to 141000000
Processing 141000001 to 142000000
Processing 142000001 to 143000000
Processing 143000001 to 144000000
Processing 144000001 to 145000000
Processing 145000001 to 146000000
Processing 146000001 to 147000000
Processing 147000001 to 148000000
Processing 148000001 to 149000000
Processing 149000001 to 150000000
Processing 150000001 to 151000000
Processing 151000001 to 152000000
Processing 152000001 to 153000000
Processing 153000001 to 154000000
Processing 154000001 to 155000000
Processing 155000001 to 156000000
Processing 156000001 to 157000000
Processing 157000001 to 158000000
Processing 158000001 to 159000000
Processing 159000001 to 160000000
Processing 160000001 to 161000000
Processing 161000001 to 162000000
Processing 162000001 to 163000000
Processing 163000001 to 164000000
Processing 164000001 to 165000000
Processing 166000001 to 167000000
Processing 165000001 to 166000000
Processing 167000001 to 168000000
Processing 168000001 to 169000000
Processing 169000001 to 170000000
Processing 170000001 to 171000000
Processing 171000001 to 172000000
Processing 172000001 to 173000000
Processing 173000001 to 174000000
Processing 174000001 to 175000000
Processing 175000001 to 176000000
Processing 176000001 to 177000000
Processing 177000001 to 178000000
Processing 178000001 to 179000000
Processing 180000001 to 181000000
Processing 179000001 to 180000000
Processing 181000001 to 182000000
Processing 182000001 to 183000000
Processing 183000001 to 184000000
Processing 184000001 to 185000000
Processing 185000001 to 186000000
Processing 186000001 to 187000000
Processing 188000001 to 189000000
Processing 187000001 to 188000000
Processing 190000001 to 191000000
Processing 189000001 to 190000000
Processing 192000001 to 193000000
Processing 191000001 to 192000000
Processing 194000001 to 195000000
Processing 193000001 to 194000000
Processing 196000001 to 197000000
Processing 195000001 to 196000000
Processing 198000001 to 199000000
Processing 197000001 to 198000000
Processing 199000001 to 200000000
Processing 200000001 to 201000000
Processing 202000001 to 203000000
Processing 201000001 to 202000000
Processing 204000001 to 205000000
Processing 203000001 to 204000000
Processing 206000001 to 207000000
Processing 205000001 to 206000000
Processing 208000001 to 209000000
Processing 207000001 to 208000000
Processing 210000001 to 211000000
Processing 209000001 to 210000000
Processing 212000001 to 213000000
Processing 211000001 to 212000000
Processing 214000001 to 215000000
Processing 213000001 to 214000000
Processing 216000001 to 217000000
Processing 215000001 to 216000000
Processing 218000001 to 219000000
Processing 217000001 to 218000000
Processing 220000001 to 221000000
Processing 219000001 to 220000000
Processing 222000001 to 223000000
Processing 221000001 to 222000000
Processing 224000001 to 225000000
Processing 223000001 to 224000000
Processing 226000001 to 227000000
Processing 225000001 to 226000000
Processing 228000001 to 229000000
Processing 227000001 to 228000000
Processing 230000001 to 231000000
Processing 229000001 to 230000000
Processing 232000001 to 233000000
Processing 231000001 to 232000000
Processing 234000001 to 235000000
Processing 233000001 to 234000000
Processing 236000001 to 237000000
Processing 235000001 to 236000000
Processing 237000001 to 238000000
Processing 238000001 to 239000000
Processing 240000001 to 241000000
Processing 239000001 to 240000000
Processing 242000001 to 243000000
Processing 241000001 to 242000000
Processing 244000001 to 245000000
Processing 243000001 to 244000000
Processing 246000001 to 247000000
Processing 245000001 to 246000000
Processing 248000001 to 249000000
Processing 247000001 to 248000000
Processing 250000001 to 251000000
Processing 249000001 to 250000000
Processing 252000001 to 253000000
Processing 251000001 to 252000000
Processing 254000001 to 255000000
Processing 253000001 to 254000000
Processing 256000001 to 257000000
Processing 255000001 to 256000000
Processing 258000001 to 259000000
Processing 257000001 to 258000000
Processing 260000001 to 261000000
Processing 259000001 to 260000000
Processing 262000001 to 263000000
Processing 261000001 to 262000000
Processing 264000001 to 265000000
Processing 263000001 to 264000000
Processing 266000001 to 267000000
Processing 265000001 to 266000000
Processing 268000001 to 269000000
Processing 267000001 to 268000000
Processing 270000001 to 271000000
Processing 269000001 to 270000000
Processing 272000001 to 273000000
Processing 271000001 to 272000000
Processing 274000001 to 275000000
Processing 273000001 to 274000000
Processing 275000001 to 276000000
Processing 276000001 to 277000000
Processing 278000001 to 279000000
Processing 277000001 to 278000000
Processing 280000001 to 281000000
Processing 279000001 to 280000000
Processing 282000001 to 283000000
Processing 281000001 to 282000000
Processing 284000001 to 285000000
Processing 283000001 to 284000000
Processing 286000001 to 287000000
Processing 285000001 to 286000000
Processing 288000001 to 289000000
Processing 287000001 to 288000000
Processing 290000001 to 291000000
Processing 289000001 to 290000000
Processing 292000001 to 293000000
Processing 291000001 to 292000000
Processing 294000001 to 295000000
Processing 293000001 to 294000000
Processing 296000001 to 297000000
Processing 295000001 to 296000000
Processing 298000001 to 299000000
Processing 297000001 to 298000000
Processing 300000001 to 301000000
Processing 299000001 to 300000000
Processing 302000001 to 303000000
Processing 301000001 to 302000000
Processing 304000001 to 305000000
Processing 303000001 to 304000000
Processing 306000001 to 307000000
Processing 305000001 to 306000000
Processing 308000001 to 309000000
Processing 307000001 to 308000000
Processing 310000001 to 311000000
Processing 309000001 to 310000000
Processing 312000001 to 313000000
Processing 311000001 to 312000000
Processing 313000001 to 314000000
Processing 314000001 to 315000000
Processing 316000001 to 317000000
Processing 315000001 to 316000000
Processing 318000001 to 319000000
Processing 317000001 to 318000000
Processing 320000001 to 321000000
Processing 319000001 to 320000000
Processing 322000001 to 323000000
Processing 321000001 to 322000000
Processing 324000001 to 325000000
Processing 323000001 to 324000000
Processing 326000001 to 327000000
Processing 325000001 to 326000000
Processing 328000001 to 329000000
Processing 327000001 to 328000000
Processing 330000001 to 331000000
Processing 329000001 to 330000000
Processing 332000001 to 333000000
Processing 331000001 to 332000000
Processing 334000001 to 335000000
Processing 333000001 to 334000000
Processing 336000001 to 337000000
Processing 335000001 to 336000000
Processing 338000001 to 339000000
Processing 337000001 to 338000000
Processing 340000001 to 341000000
Processing 339000001 to 340000000
Processing 342000001 to 343000000
Processing 341000001 to 342000000
Processing 344000001 to 345000000
Processing 343000001 to 344000000
Processing 346000001 to 347000000
Processing 345000001 to 346000000
Processing 348000001 to 349000000
Processing 347000001 to 348000000
Processing 349000001 to 350000000
Processing 350000001 to 351000000
Processing 352000001 to 353000000
Processing 351000001 to 352000000
Processing 354000001 to 355000000
Processing 353000001 to 354000000
Processing 356000001 to 357000000
Processing 355000001 to 356000000
Processing 358000001 to 359000000
Processing 357000001 to 358000000
Processing 360000001 to 361000000
Processing 359000001 to 360000000
Processing 362000001 to 363000000
Processing 361000001 to 362000000
Processing 364000001 to 365000000
Processing 363000001 to 364000000
Processing 366000001 to 367000000
Processing 365000001 to 366000000
Processing 368000001 to 369000000
Processing 367000001 to 368000000
Processing 370000001 to 371000000
Processing 369000001 to 370000000
Processing 372000001 to 373000000
Processing 371000001 to 372000000
Processing 374000001 to 375000000
Processing 373000001 to 374000000
Processing 376000001 to 377000000
Processing 375000001 to 376000000
Processing 378000001 to 379000000
Processing 377000001 to 378000000
Processing 380000001 to 381000000
Processing 379000001 to 380000000
Processing 382000001 to 383000000
Processing 381000001 to 382000000
Processing 384000001 to 385000000
Processing 383000001 to 384000000
Processing 385000001 to 386000000
Processing 386000001 to 387000000
Processing 388000001 to 389000000
Processing 387000001 to 388000000
Processing 390000001 to 391000000
Processing 389000001 to 390000000
Processing 392000001 to 393000000
Processing 391000001 to 392000000
Processing 394000001 to 395000000
Processing 393000001 to 394000000
Processing 396000001 to 397000000
Processing 395000001 to 396000000
Processing 398000001 to 399000000
Processing 397000001 to 398000000
Processing 400000001 to 401000000
Processing 399000001 to 400000000
Processing 402000001 to 403000000
Processing 401000001 to 402000000
Processing 404000001 to 405000000
Processing 403000001 to 404000000
Processing 406000001 to 407000000
Processing 405000001 to 406000000
Processing 408000001 to 409000000
Processing 407000001 to 408000000
Processing 410000001 to 411000000
Processing 409000001 to 410000000
Processing 412000001 to 413000000
Processing 411000001 to 412000000
Processing 414000001 to 415000000
Processing 413000001 to 414000000
Processing 416000001 to 417000000
Processing 415000001 to 416000000
Processing 418000001 to 419000000
Processing 417000001 to 418000000
Processing 420000001 to 421000000
Processing 419000001 to 420000000
Processing 421000001 to 422000000
Processing 422000001 to 423000000
Processing 424000001 to 425000000
Processing 423000001 to 424000000
Processing 426000001 to 427000000
Processing 425000001 to 426000000
Processing 428000001 to 429000000
Processing 427000001 to 428000000
Processing 430000001 to 431000000
Processing 429000001 to 430000000
Processing 432000001 to 433000000
Processing 431000001 to 432000000
Processing 434000001 to 435000000
Processing 433000001 to 434000000
Processing 436000001 to 437000000
Processing 435000001 to 436000000
Processing 438000001 to 439000000
Processing 437000001 to 438000000
Processing 440000001 to 441000000
Processing 439000001 to 440000000
Processing 442000001 to 443000000
Processing 441000001 to 442000000
Processing 444000001 to 445000000
Processing 443000001 to 444000000
Processing 446000001 to 447000000
Processing 445000001 to 446000000
Processing 448000001 to 449000000
Processing 447000001 to 448000000
Processing 450000001 to 451000000
Processing 449000001 to 450000000
Processing 452000001 to 453000000
Processing 451000001 to 452000000
Processing 454000001 to 455000000
Processing 453000001 to 454000000
Processing 455000001 to 456000000
Processing 457000001 to 458000000
Processing 456000001 to 457000000
Processing 458000001 to 459000000
Processing 460000001 to 461000000
Processing 459000001 to 460000000
Processing 462000001 to 463000000
Processing 461000001 to 462000000
Processing 464000001 to 465000000
Processing 463000001 to 464000000
Processing 466000001 to 467000000
Processing 465000001 to 466000000
Processing 468000001 to 469000000
Processing 467000001 to 468000000
Processing 470000001 to 471000000
Processing 469000001 to 470000000
Processing 472000001 to 473000000
Processing 471000001 to 472000000
Processing 474000001 to 475000000
Processing 473000001 to 474000000
Processing 476000001 to 477000000
Processing 475000001 to 476000000
Processing 478000001 to 479000000
Processing 477000001 to 478000000
Processing 480000001 to 481000000
Processing 479000001 to 480000000
Processing 482000001 to 483000000
Processing 481000001 to 482000000
Processing 484000001 to 485000000
Processing 483000001 to 484000000
Processing 486000001 to 487000000
Processing 485000001 to 486000000
Processing 488000001 to 489000000
Processing 487000001 to 488000000
Processing 489000001 to 490000000
Processing 491000001 to 492000000
Processing 490000001 to 491000000
Processing 492000001 to 493000000
Processing 494000001 to 495000000
Processing 493000001 to 494000000
Processing 496000001 to 497000000
Processing 495000001 to 496000000
Processing 498000001 to 499000000
Processing 497000001 to 498000000
Processing 500000001 to 501000000
Processing 499000001 to 500000000
Processing 502000001 to 503000000
Processing 501000001 to 502000000
Processing 504000001 to 505000000
Processing 503000001 to 504000000
Processing 506000001 to 507000000
Processing 505000001 to 506000000
Processing 508000001 to 509000000
Processing 507000001 to 508000000
Processing 510000001 to 511000000
Processing 509000001 to 510000000
Processing 512000001 to 513000000
Processing 511000001 to 512000000
Processing 514000001 to 515000000
Processing 513000001 to 514000000
Processing 516000001 to 517000000
Processing 515000001 to 516000000
Processing 518000001 to 519000000
Processing 517000001 to 518000000
Processing 520000001 to 521000000
Processing 519000001 to 520000000
Processing 522000001 to 523000000
Processing 521000001 to 522000000
Processing 523000001 to 524000000
Processing 525000001 to 526000000
Processing 524000001 to 525000000
Processing 526000001 to 527000000
Processing 528000001 to 529000000
Processing 527000001 to 528000000
Processing 530000001 to 531000000
Processing 529000001 to 530000000
Processing 532000001 to 533000000
Processing 531000001 to 532000000
Processing 534000001 to 535000000
Processing 533000001 to 534000000
Processing 536000001 to 537000000
Processing 535000001 to 536000000
Processing 538000001 to 539000000
Processing 537000001 to 538000000
Processing 540000001 to 541000000
Processing 539000001 to 540000000
Processing 542000001 to 543000000
Processing 541000001 to 542000000
Processing 544000001 to 545000000
Processing 543000001 to 544000000
Processing 546000001 to 546304043
Processing 545000001 to 546000000

 *** caught segfault ***
address (nil), cause 'unknown'
/var/spool/slurm/slurmd/job1289938/slurm_script: line 17: 2763111 Segmentation fault      (core dumped) singularity exec --bind /sandbox:[path/to/.sif image] Rscript [path/to/Rscript/chromunity_test_merge.R]

Setting mc.cores to >1, triggers a memory leak:

$ seff 1288563 | grep "State\|Memory"
State: OUT_OF_MEMORY (exit code 0)
Memory Utilized: 2.04 TB
Memory Efficiency: 464.97% of 450.00 GB

If mc.core = 1, I still catch a segfault even though memory is being underutilized:

seff 1289938 | grep "State\|Memory"
State: FAILED (exit code 139)
Memory Utilized: 166.99 GB
Memory Efficiency: 37.11% of 450.00 GB

Is there anything I am doing incorrectly? Is it a package/dependency incompatibility? Perhaps memory is not being allocated or cleaned up properly?

Thank you for the help!!

[imielinski]

My guess is you're running out of memory? These analyses can get very memory intensive, particularly with a large dataset. How big is the Pore-C data (how many concatemers). Maybe try subsampling reads or running on a small genomic region. Or try a higher memory server .. Under the hood chromunity is just using R data.table, so maybe update the package or try a different version. Hope this helps

[sevachkaa12]

@imielinski, I appreciate these thoughts! I have subsampled my data and am working with a fairly small genome (at least compared to humans). The second error message with mc.cores=1, seems to imply that it simply fails after catching the segfault while only using ~37% of the memory it was given. I am using the most recent package versions too!

Based on my inspection of the parquet data, we have roughly 144 million reads with 96.3 million concatemers (of restriction fragments >= 2). Is there anything else you might suggest to try? Thank you all!