Slow cache recovery after sample removal and resume

[NureSharafNowerSamia]

Describe the bug
When running the pipeline, if there is an issue with one of the samples, the entire pipeline fails. After removing the problematic sample and resuming the run, the pipeline takes a significantly longer time to recover and reuse cached results compared to earlier versions. Although the cache is flagged, it appears that the pipeline is redoing parts of the analysis instead of properly using the cache.

Impact
This issue is causing much longer run times and inefficient resource usage when recovering from an error. It slows down the workflow considerably, especially when working with large datasets.

To Reproduce

Environment: Local (Docker profile used)

Pipeline version: cdcgov/phoenix v2.1.1

Command run: nextflow run main.nf -profile docker -entry PHOENIX --input <samplesheet.csv> --kraken2db

Custom config file: Not used

Text of the error:
No specific error text upon resuming. After removing the faulty sample, the resumption process is significantly slower, and re-analysis of already completed parts occurs even though the cache is flagged.

Files that caused the error:
Available upon request if needed.

Expected behavior
After removing the problematic sample and resuming, the pipeline should quickly reuse the cached results and only rerun necessary steps, similar to behavior in earlier versions.

Screenshots
N/A

Logs
Logs can be provided if needed. (.nextflow.log and relevant .command.err / .command.out files)

Additional context
In previous versions of the pipeline, resuming after error recovery was much faster and efficient. This change in behavior might be related to recent updates in the pipeline structure or Nextflow version handling.

[NoForkJustCommit]

Hi, thanks for your interest in phx. I would need the troublesome sample to figure out why the failure is happening. We have tried to build in checks for most failures to keep this from happening, but there is always a new edge case. We are aware there was some issues with -resume working as expected and have made some improvements to help with that in the forthcoming v2.2.0 version. Is the sample public on NCBI where I could pull it?

[NureSharafNowerSamia]

No, the samples aren't from NCBI; they're ones we sequence regularly. Regarding the -resume option in the Phoenix workflow, it currently caches only the initial two steps. This means that if the pipeline is interrupted, it doesn't resume from the exact point of failure but rather restarts from the beginning of the third step.

[NoForkJustCommit]

yes, the fixes for improved -resume functioning are in the next release not v2.1.1 so that isn't a surprise. Please email HAISeq@cdc.gov to set up a way to get the sequence files to us so we can figure out the cause of the failure and address it. And it would be helpful to have a look at the .nextflow.log for the run with the problematic sample. Thanks!

[jvhagey]

Hi @NureSharafNowerSamia we refactored how failures are handled to fix -resume can you confirm that -resume is working as expected with the latest release v2.2.0 ? thank you!