ValueError: The specified BB file does not exist!

[rtasakis]

Hi! I'm using HATCHet for the first time and I'm encountering the following error that my specified BB file does not exist. I'm not sure what is the issue, but from my understanding this is something that upstream steps of running HATCHet should produce? Any help would be greatly appreciated!

Here's the content of my .ini file

# What individual steps of HATCHet should we run in the pipeline?
# Valid values are True or False
download_panel = False
count_reads = True
genotype_snps = True
phase_snps = False
fixed_width = False  # True uses older fixed-width versions of some commands
count_alleles = True
combine_counts = False
cluster_bins = True
loc_clust = True  # True uses new locality-aware clustering
plot_bins = True
compute_cn = True
plot_cn = True

# What chromosome(s) do we wish to process in the pipeline? Leave unspecified to process
# all chromosomes found in the normal/tumor bam files
chromosomes = "22"

# Path to reference genome
# Make sure you have also generated the reference dictionary as /path/to/reference.dict
reference = /path/to/ref/human_g1k_v37.fasta

# Make sure you have generated the .bam.bai files at the same locations as these bam files
normal = /path/to/normal/Normal.dedup.recal.bam

# Space-delimited list of tumor BAM locations
bams = /path/to/tumor/Tumor.dedup.recal.bam

# Space-delimited list of tumor names
samples = TumorTest

# Output path of the run script
output = /path/to/HatchetOut/TumorTest_out

# How many cores to use for the end-end pipeline?
# This parameter, if specified, will override corresponding 'processes' parameters in individual <step> sections below.
processes = 2

[genotype_snps]
# Reference version used to select list of known germline SNPs;
# Possible values are "hg19" or "hg38", or leave blank "" if you wish for all positions to be genotyped by bcftools
reference_version = hg19
# Does your reference name chromosomes with "chr" prefix?; True or False
chr_notation = False

# Use 8 for WGS with >30x and 20 for WES with ~100x
mincov = 8
# Use 300 for WGS with >30x and Use 1000 for WES with ~100x
maxcov = 300
# Path to SNP list
#   If unspecified, HATCHet selects a list of known germline SNPs based on <run.reference_version> and <run.chr_notation>
#   If not, please provide full path to a locally stored list (.vcf.gz) here.
snps = /path/to/vcf/tmpTestHatchet/00-All.vcf.gz

[combine_counts]
# Minimum number of SNP-covering reads per bin and sample
msr = 5000
# Minimum number of total reads per bin and sample
mtr = 5000

[cluster_bins]
diploidbaf = 0.08
# Minimum and maximum number of clusters to infer
# (using silhouette score for model selection)
minK = 2
maxK = 30
# You can instead specify an exact number of clusters:
# exactK = 15

[plot_bins]
sizethreshold = 0.01
figsize = "6,3"

[compute_cn]
clones = 2,6
seeds = 400
minprop = 0.03
diploidcmax = 6
tetraploidcmax = 12
ghostprop = 0.35
limitinc = 0.6

I run it with:

hatchet run new_hatchet.ini 

And the output is:

[2025-Feb-18 14:12:57]('# Parsing the input arguments, checking the consistency of given files, and extracting required ', 'information\n')
        normal: ('/path/to/normal/Normal.dedup.recal.bam', 'Normal')
        chromosomes: ['22']
        samtools: /opt/samtools-1.7/samtools
        bcftools: /opt/bcftools-1.7/bcftools
        snps: /path/to/vcf/tmpTestHatchet/00-All.vcf.gz
        reference: /path/to/ref/human_g1k_v37.fasta
        j: 2
        q: 0
        Q: 11
        E: False
        mincov: 8
        maxcov: 300
        outputsnps: /path/to/HatchetOut/TumorTest_out/snps
        verbose: False
[2025-Feb-18 14:13:25]# Inferring SNPs from the normal sample
Progress: |----------------------------------------| 0.0% Complete [[2025-Feb-18 14:13:25]Caller-1 starts on Normal for 22)]
Progress: |████████████████████████████████████████| 100.0% Complete [[2025-Feb-18 14:21:44]Caller-1 ends on Normal for 22)]
[2025-Feb-18 14:21:44]# Counting number of identified SNPs
3945 SNPs have been identified in total
[2025-Feb-18 14:21:44]# SNP Calling is concluded
## Called SNPs have been written per chromosome in:
/path/to/HatchetOut/TumorTest_out/snps/22.vcf.gz
[2025-Feb-18 14:21:44]# Parsing the input arguments, checking the consistency of given files, and extracting required information

        normal: ('/path/to/normal/Normal.dedup.recal.bam', 'normal')
        samples: {('/path/to/tumor/Tumor.dedup.recal.bam', 'TumorTest')}
        chromosomes: ['22']
        samtools: /opt/samtools-1.7/samtools
        bcftools: /opt/bcftools-1.7/bcftools
        snps: {'22': ' /path/to/HatchetOut/TumorTest_out/snps/22.vcf.gz'}
        reference: /path/to/ref/human_g1k_v37.fasta
        j: 2
        q: 0
        Q: 11
        qual: 11
        E: False
        gamma: 0.05
        maxshift: 0.5
        mincov: 0
        maxcov: 1000
        outputNormal: /path/to/HatchetOut/TumorTest_out/baf/normal.1bed
        outputTumors: /path/to/HatchetOut/TumorTest_out/baf/tumor.1bed
        outputSnps: /path/to/HatchetOut/TumorTest_out
        verbose: False
[2025-Feb-18 14:22:10]# Counting SNPs alleles from the matched-normal sample
[2025-Feb-18 14:27:29]
Progress: |████████████████████████████████████████| 100.0% Complete
[2025-Feb-18 14:27:29]# Selecting heterozygous SNPs
[2025-Feb-18 14:27:30]# Writing the list of selected SNPs, covered and heterozygous in the normal sample
[2025-Feb-18 14:27:30]# Writing the allele counts of the normal sample for selected SNPs
[2025-Feb-18 14:27:30]# Counting SNPs alleles from tumour samples
[2025-Feb-18 14:40:58]
Progress: |████████████████████████████████████████| 100.0% Complete
[2025-Feb-18 14:40:58]# Writing the allele counts of tumor samples for selected SNPs
[2025-Feb-18 14:40:58]# Parsing and checking input arguments

        bams: ['/path/to/normal/Normal.dedup.recal.bam', '/path/to/tumor/Tumor.dedup.recal.bam']
        names: ['normal', 'TumorTest']
        chromosomes: ['22']
        samtools: /opt/samtools-1.7/samtools
        mosdepth: mosdepth
        tabix: tabix
        j: 2
        outdir: /path/to/HatchetOut/TumorTest_out/rdr
        use_chr: False
        refversion: hg19
        baf_file: /path/to/HatchetOut/TumorTest_out/baf/tumor.1bed
        readquality: 11
[2025-Feb-18 14:40:58]Sample normal -- Starting chromosome 22
[2025-Feb-18 14:40:58]Sample TumorTest -- Starting chromosome 22
[2025-Feb-18 14:41:18]Sample normal -- Done chromosome 22
[2025-Feb-18 14:41:39]Sample TumorTest -- Done chromosome 22
[2025-Feb-18 14:41:39]Starting mosdepth on sample normal with 1 threads
[2025-Feb-18 14:41:39]Starting mosdepth on sample TumorTest with 1 threads
[2025-Feb-18 14:48:49]Done mosdepth on sample normal
[2025-Feb-18 14:56:49]Done mosdepth on sample TumorTest
Loading chromosome 22
Reading SNPs file for chromosome 22
Loading counts for chromosome 22
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?
Done chromosome 22
[2025-Feb-18 14:58:40]# Array forming completed successfully, removing intermediate count files. 
[2025-Feb-18 14:58:41]# Counting total number of reads for normal and tumor samples
Progress: |----------------------------------------| 0.0% Complete [[2025-Feb-18 14:58:41
Progress: |----------------------------------------| 0.0% Complete [[2025-Feb-18 14:58:41
Progress: |████████████████████--------------------| 50.0% Complete [[2025-Feb-18 14:58:4
Progress: |████████████████████████████████████████| 100.0% Complete [[2025-Feb-18 14:58:58]TotalCounter-10 ends on TumorTest for 22]
[2025-Feb-18 14:58:58]# Writing the total read counts for all samples in /path/to/HatchetOut/TumorTest_out/rdr/total.tsv
[2025-Feb-18 14:58:58]# Parsing and checking input arguments
The specified BB file does not exist!Traceback (most recent call last):
  File "/usr/local/bin/hatchet", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/hatchet/__main__.py", line 60, in main
    globals()[command](args)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/run.py", line 327, in main
    cluster_bins(
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/cluster_bins.py", line 17, in main
    args = parse_cluster_bins_args(args)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/ArgParsing.py", line 382, in parse_cluster_bins_args
    ensure(isfile(args.BBFILE), "The specified BB file does not exist!")
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/Supporting.py", line 147, in ensure
    return error(msg, raise_exception=True, exception_class=exception_class)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/Supporting.py", line 137, in error
    return log(
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/Supporting.py", line 124, in log
    raise exception_class(msg)
ValueError: The specified BB file does not exist!

[RunpengLuo]

Hi Rafail,

If this is your first run, you should set combine_counts = True, which will generate the BB file and it's compulsory for all subsequent steps like cluster_bins. Thanks.

[rtasakis]

Thanks, I appreciate the help! It worked and the run moved on, but I stumbled upon another error at the GC bias correction step. The output continued from my comment above was:

NO PHASING FILE FOUND at /path/to/HatchetOut/TumorTest_out/phase/phased.vcf.gz. Not including phasing in binning process.
[2025-Feb-19 14:46:46]# Parsing and checking input arguments
Identified 1 chromosomes.

        baffile: /path/to/HatchetOut/TumorTest_out/baf/tumor.1bed
        outfile:  /path/to/HatchetOut/TumorTest_out/bb/bulk.bb
        referencefasta: /path/to/ref/human_g1k_v37.fasta
        sample_names: ['normal', 'TumorTest']
        min_snp_reads: 5000
        min_total_reads: 5000
        use_chr: False
        processes: 2
        chromosomes: ['22']
        array: /path/to/HatchetOut/TumorTest_out/rdr
        totalcounts: /path/to/HatchetOut/TumorTest_out/rdr/total.tsv
        phase: None
        blocksize: 25000
        max_snps_per_block: 10
        test_alpha: 0.1
        multisample: True
        ref_version: hg19
Loading intermediate files for chromosome 22
Binning p arm of chromosome 22
No SNPs found in p arm for 22
Binning q arm of chromosome 22
[2025-Feb-19 14:46:47]Correcting haplotype switches on q arm...
Found haplotype switches in  0.00% of bins
Insufficient haplotype switching detected (<[0.01]), skipping correction.
Done chromosome 22
[2025-Feb-19 14:46:47]# Merging per-chromosome bb files and correcting read counts
[2025-Feb-19 14:46:47]# Performing GC bias correction on read depth signal
Traceback (most recent call last):
  File "/usr/local/bin/hatchet", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/hatchet/__main__.py", line 60, in main
    globals()[command](args)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/run.py", line 262, in main
    combine_counts(args)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/combine_counts.py", line 154, in main
    autosomal_bb = rd_gccorrect(autosomal_bb, referencefasta)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/rd_gccorrect.py", line 20, in rd_gccorrect
    bb = bb.merge(
  File "/usr/local/lib/python3.8/site-packages/pandas/core/frame.py", line 9843, in merge
    return merge(
  File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 148, in merge
    op = _MergeOperation(
  File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 741, in __init__
    self._maybe_coerce_merge_keys()
  File "/usr/local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1401, in _maybe_coerce_merge_keys
    raise ValueError(msg)
ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat

Is this error because the phasing of snps file is required for the GC correction and then it fails to merge with upstream output? I used the phase_snps = False option, but changed and I ran it with phase_snps = True (and set the required arguments in the [download pane] section of the ini file). I got the following:

�[1m�[95m[2025-Feb-19 19:13:32]# log notes
�[0m�[92m
	refpanel: 1000GP_Phase3
	refpaneldir: /path/to/refPanel/phase_test
�[0mTraceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 199, in _new_conn
    sock = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.8/socket.py", line 930, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 441, in request
    self.endheaders()
  File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 279, in connect
    self.sock = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 206, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x7718f80e9130>: Failed to resolve 'hgdownload.cse.ucsc.edu' ([Errno -2] Name or service not known)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='hgdownload.cse.ucsc.edu', port=80): Max retries exceeded with url: /goldenPath/hg19/bigZips/hg19.fa.gz (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7718f80e9130>: Failed to resolve 'hgdownload.cse.ucsc.edu' ([Errno -2] Name or service not known)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/hatchet", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/hatchet/__main__.py", line 60, in main
    globals()[command](args)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/run.py", line 64, in main
    download_panel(
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/download_panel.py", line 36, in main
    dwnld_refpanel_genome(path=args["refpaneldir"])
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/download_panel.py", line 104, in dwnld_refpanel_genome
    out = download(config.urls.refpanel_genome, dirpath=path, extract=False)
  File "/usr/local/lib/python3.8/site-packages/hatchet/utils/Supporting.py", line 212, in download
    with requests.get(url, stream=True) as r:
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='hgdownload.cse.ucsc.edu', port=80): Max retries exceeded with url: /goldenPath/hg19/bigZips/hg19.fa.gz (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7718f80e9130>: Failed to resolve 'hgdownload.cse.ucsc.edu' ([Errno -2] Name or service not known)"))

I'd appreciate the help to get this going! Thanks :)

[RunpengLuo]

Hi Rafail,

1. The first issue comes from whether your reference/BAM file uses chr notation or not (like chr1 in .fasta file), if yes, please set chr_notation = True and rerun all previous steps (remove intermediate files first).
2. It looks like the second issue is from download_panel step, which is related to internet connection. Can you check if the running environment has internet connection? e.g., if you are running it in a compute cluster, some cluster may not support internet connection and you have to run the download_panel step locally before proceeding the following steps (set other steps to False first) on a cluster. Please check the detailed documentation here.

Feel free to let me know how it goes.

[rtasakis]

Hi Runpeng,

Thanks for your response - appreciate the help.
About the chr_notation: the reference genome and my bam files do not have the "chr" in the chromosome numbering, so I set the chr_notation = False and rerun after removing the intermediate files. The error about object and int64 columns merging persisted and also tried the chr_notation = True and I got to the same point. I used this SNP resource for the snp calling step (https://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh37p13/VCF/00-All.vcf.gz) and I checked that there were no "chr" in the chromosome entries either.

The error occurs pretty downstream in the pipeline at the GC bias and read count correction step, if the chr notation was the issue wouldn't it be upstream in the snp calling? I also checked the produced intermediate files up until that point and there's no "chr" in the chromosome entries. So I am not sure how it reaches to the object vs int64 merging error. Any chance it's the pandas version?

This is the python and pandas version in my system:

Python 3.8.20 (default, Sep 27 2024, 06:05:08) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'2.0.3'
>>> 

About the download_panel step: do you have a recommendation for the source I should choose? I am using the reference genome HumanG1Kv37 from GATK (see here). My bam files are aligned against the same reference.

Thanks again!

[RunpengLuo]

Hi Rafail,

Indeed it is a bug in our program related to type conversion issue, sorry for the inconvenience and thanks a lot for pointing it out! I've created a pull request and will update you once it is completed.

For the reference panel, please set ref_panel=1000GP_Phase3 (by default) in download_panel section and set reference_version=hg19 in genotype_snps section, which should be compatible with your reference file, also leave chr_notation=False. You can also refer to the template we provided here.

[rtasakis]

@RunpengLuo, I see the pull request as merged, do you think I am able to run a test with the updated script? Thanks!

[RunpengLuo]

Hi @rtasakis, in case you did manual installation, you can feel free to try it. Otherwise if you're using Bioconda install, you may wait for a few days as I'm updating the Bioconda at the moment. Thank you.

[RunpengLuo]

Hi @rtasakis Please feel free to try the Bioconda version, it is up now, thank you.

[rtasakis]

Hi @RunpengLuo, I wanted to let you know that the update (v 2.1.2) has fixed the error I reported above. I will close this issue. Thanks a lot for looking into it!

Also, a minor side note, when I am calling hatchet it still gives the v2.1.0. Just wanted to let you know to avoid any future confusions with users.

rafail@server:/home/rafail# hatchet

HATCHet v2.1.0
Usage: hatchet <command> <arguments ..>

The following commands are supported:
 count-reads
 count-reads-fw
 genotype-snps
 count-alleles
 combine-counts
 combine-counts-fw
 cluster-bins
 plot-bins
 compute-cn
 plot-cn
 run
 check
 download-panel
 phase-snps
 cluster-bins-gmm
 plot-cn-1d2d
 plot-bins-1d2d