polars.exceptions.ComputeError: could not parse `-` as dtype `i64` at column 'score' (column number 6)

[jazsakr]

Hello! I am running the latest version of the FIRE pipeline with ONT data. I have not had any issues before, but after the update I got the following error when running a new sample:

localrule fdr_track_chromosome:
    input: temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz, results/260Fiber/additional-outputs-v0.1.1/fire-peaks/260Fiber-v0.1.1-fire-score-to-fdr.tbl
    output: temp/260Fiber/fire-peaks/v0.1.1-GL456354.1-FDR.track.bed
    jobid: 0
    reason: Forced execution
    wildcards: sm=260Fiber, v=v0.1.1, chrom=GL456354.1
    threads: 4
    resources: mem_mb=94208, mem_mib=89844, disk_mb=4096, disk_mib=3907, tmpdir=/tmp/jsakr/36408502, runtime=120, slurm_partition=standard, slurm_account=seyedam_lab

Activating conda environment: ../../../../../../../data/homezvol3/jsakr/snakemake-conda-envs/fc5018ac87025e0ae5ebc0630d3241a7_
[INFO][Time elapsed (ms) 874]: Read FDR table:
      threshold       FDR  shuffled_bp      real_bp
0         -1.00  1.000000          1.0          1.0
1          4.69  0.996092  192991831.0  193748972.0
2          4.70  0.996064  192981454.0  193744101.0
3          4.71  0.991812  191478713.0  193059441.0
4          4.72  0.991793  191464091.0  193048458.0
...         ...       ...          ...          ...
2147      45.11  0.000611        336.0     550139.0
2148      45.76  0.000419        219.0     522586.0
2149      47.61  0.000379        168.0     443844.0
2150      54.92  0.000219         46.0     209682.0
2151      55.13  0.000000          0.0     205122.0

[2152 rows x 4 columns]
Traceback (most recent call last):
  File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 333, in <module>
    defopt.run(main, show_types=True, version="0.0.1")
  File "/data/homezvol3/jsakr/snakemake-conda-envs/fc5018ac87025e0ae5ebc0630d3241a7_/lib/python3.10/site-packages/defopt.py", line 356, in run
    return call()
  File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 324, in main
    apply_fdr_table(infile, outfile, fdr_table, nrows)
  File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 205, in apply_fdr_table
    pileup = read_pileup_file(infile, nrows)
  File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 48, in read_pileup_file
    header = my_read_csv(infile, separator="\t", n_rows=1).columns
  File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 31, in my_read_csv
    result = pl.read_csv(*args, **kwargs)
  File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 135, in wrapper
    return function(*args, **kwargs)
  File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 135, in wrapper
    return function(*args, **kwargs)
  File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 135, in wrapper
    return function(*args, **kwargs)
  [Previous line repeated 1 more time]
  File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/io/csv/functions.py", line 422, in read_csv
    df = _read_csv_impl(
  File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/io/csv/functions.py", line 568, in _read_csv_impl
    pydf = PyDataFrame.read_csv(
polars.exceptions.ComputeError: could not parse `-` as dtype `i64` at column 'score' (column number 6)

The current offset in the file is 8191 bytes.

You might want to try:
- increasing `infer_schema_length` (e.g. `infer_schema_length=10000`),
- specifying correct dtype with the `dtypes` argument
- setting `ignore_errors` to `True`,
- adding `-` to the `null_values` list.

Original error: ```remaining bytes non-empty```
[Tue Mar 11 18:22:31 2025]
Error in rule fdr_track_chromosome:
    jobid: 0
    input: temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz, results/260Fiber/additional-outputs-v0.1.1/fire-peaks/260Fiber-v0.1.1-fire-score-to-fdr.tbl
    output: temp/260Fiber/fire-peaks/v0.1.1-GL456354.1-FDR.track.bed
    conda-env: /data/homezvol3/jsakr/snakemake-conda-envs/fc5018ac87025e0ae5ebc0630d3241a7_
    shell:
        
        python /data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py -v 1             --fdr-table results/260Fiber/additional-outputs-v0.1.1/fire-peaks/260Fiber-v0.1.1-fire-score-to-fdr.tbl             temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz temp/260Fiber/fire-peaks/v0.1.1-GL456354.1-FDR.track.bed
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

Any suggestions? Thank you!

[mrvollger]

Odd, Not sure what would cause this. Sorry! Any chance this is data you can share, or a test case you can make with a subset of the data that I can play with?

[jazsakr]

Yes! I added reads from the chromosome that threw the error here. Thank you!

[mrvollger]

Thanks! Can you also share the reference or relevant contig so the I can run the whole pipeline?

[jazsakr]

Okie, I think I extracted that particular chromosome correctly and uploaded it to the same folder. If not, you can download the reference with wget https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M36/GRCm39.genome.fa.gz

[mrvollger]

I am unable to replicate a - in column 6 of temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz. Can you confirm you see it there in your file?

[jazsakr]

I don't see it in my file either (I uploaded it into the same folder). Did it not error out for you? What else should I check?

[mrvollger]

Without the full dataset I am not able to run the pipeline to completely recreate this so I was checking the pileup tbl generated from the bam file you uploaded. If you also upload the FDR tbl I can try this one python command in isolation and I'll see if that recreates it.

[jazsakr]

Uploaded!

[mrvollger]

I am not able to recreate this with the single line command. If you remove the inputs to this rule and rerun do you have the same issue?

[jazsakr]

I removed temp/260Fiber/* and results/260Fiber/additional-outputs-v0.1.1/ before rerunning and I still got the same error. I should also mention that this happens to multiple chromosomes, not just this one.

[mrvollger]

Hmm, I seem unable to replicate this with the smaller fileset. Do you have a way to share the full one?

[jazsakr]

We are planning on publicly releasing the data sometime next week. I can share it with you once it is uploaded, I will let you know. Is there anything else I can try in the meantime?

[mrvollger]

You could try altering the polars version in the environment yaml in case it is a bug in a particular patch of polars. Another idea would be to arbitrarily rename the contigs and realign incase there is some weird corner case being hit with contig names. Though I have a hard time imagining what that could be.

I'll let you know if I have any other one off ideas.

[dverac]

I am having a similar error, also with a Nanopore data with good coverage