Failed to convert gnomAD v4.1 VCFs to CRISPRme-required format

[schuroff]

Describe the bug
Failed to convert gnomAD v4.1 VCFs to CRISPRme-required format.

To Reproduce

We installed CRISPRme v2.1.7 using mamba.
We also tried with conda, but the same issue occurred.
The installation completed successfully and:
crisprme.py --version #worked as expected.

We are using gnomAD v4.1 joint VCFs.

When running the gnomAD-converter command as described in the documentation, we got the following error:

Command-line:
~/crisprme$ crisprme.py gnomAD-converter --gnomAD_VCFdir ~/crisprme/VCFs/hg38_gnomad.v41/ --samplesID ~/crisprme/sampleIDs/samplesIDs.gnomad.v41.txt --joint --thread 20 --multiallelic --debug
DEBUG MODE
python: can't open file '/home/tiaf/crisprme/PostProcess/convert_gnomAD_vcfs.py': [Errno 2] No such file or directory
Traceback (most recent call last):
File "/home/tiaf/miniconda3/envs/crisprme/bin/crisprme.py", line 1319, in
gnomAD_converter()
File "/home/tiaf/miniconda3/envs/crisprme/bin/crisprme.py", line 1101, in gnomAD_converter
raise subprocess.SubprocessError(
subprocess.SubprocessError: An error occurred while converting gnomAD VCFs in /home/tiaf/crisprme/VCFs/hg38_gnomad.v41

After checking, we found the convert_gnomAD_vcfs.py file in the following locations:
/home/tiaf/miniconda3/envs/crisprme/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py
/home/tiaf/miniconda3/pkgs/crisprme-2.1.7-hdfd78af_0/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py
/home/tiaf/crisprme/CRISPRme/src/convert_gnomAD_vcfs.py
/home/tiaf/PostProcess/convert_gnomAD_vcfs.py

Then, we copied the folder /home/tiaf/miniconda3/pkgs/crisprme-2.1.7-hdfd78af_0/opt/crisprme/PostProcess/ to /home/tiaf/crisprme/
and re-ran the command. This time, we got a different error:

~/crisprme$ crisprme.py gnomAD-converter --gnomAD_VCFdir ~/crisprme/VCFs/hg38_gnomad.v41/ --samplesID ~/crisprme/sampleIDs/samplesIDs.gnomad.v41.txt --joint --thread 20 --multiallelic --debug
DEBUG MODE
Traceback (most recent call last):
File "/home/tiaf/crisprme/PostProcess/convert_gnomAD_vcfs.py", line 27, in
from utils import remove
ModuleNotFoundError: No module named 'utils'
Traceback (most recent call last):
File "/home/tiaf/miniconda3/envs/crisprme/bin/crisprme.py", line 1319, in
gnomAD_converter()
File "/home/tiaf/miniconda3/envs/crisprme/bin/crisprme.py", line 1101, in gnomAD_converter
raise subprocess.SubprocessError(
subprocess.SubprocessError: An error occurred while converting gnomAD VCFs in /home/tiaf/crisprme/VCFs/hg38_gnomad.v41

We highlight the following error: ModuleNotFoundError: No module named 'utils'. What is it?
In addition, we tried to run the complete search below:

crisprme.py complete-search --genome Genomes/hg38 --vcf vcf_gnomad_config22.txt --guide HBG-175.txt --pam PAMs/20bp-NG-SpCas9.txt --samplesID ~/crisprme/sampleIDs/samplesIDs.gnomad.v41.txt --be-window 4,8 --be-base A,G --mm 3 --bDNA 1 --bRNA 1 --merge 3 --output chr22_search --thread 20

and got the following errors:
mv: cannot stat '/home/tiaf/crisprme/Genomes/variants_genome/SNPs_genome/hg38_enriched/': No such file or di
rectory
mv: cannot stat '/home/tiaf/crisprme/Genomes/variants_genome/SNPs_genome/.json': No such file or directory
mv: cannot stat '/home/tiaf/crisprme/Genomes/variants_genome/SNPs_genome/log.txt': No such file or director
y
Traceback (most recent call last):
File "/home/tiaf/miniconda3/envs/crisprme/opt/crisprme/PostProcess/./pool_index_indels.py", line 25, in
for f in os.listdir(indels_folder):
FileNotFoundError: [Errno 2] No such file or directory: '/home/tiaf/crisprme/Genomes/variants_genome/'
ERROR: indels indexing failed on hg38_gnomad.v41

We believe this error is due to the failed VCF conversion.

Additional test

We also tested a smaller dataset (chr22) using:

~/crisprme$ crisprme.py complete-test --chrom chr22 --vcf_dataset 1000G
Downloading fasta file for chromosome(s) chr22
Downloading VCF data for chromsome(s) chr22
Creating VCF config file for dataset(s) 1000G
Downloading sample ids for dataset(s) 1000G
Creating samples config file for dataset(s) 1000G
Downloading ENCODE and GENCODE annotation data
Creating PAM file
Creating guide file
python: can't open file '/home/tiaf/crisprme/crisprme.py': [Errno 2] No such file or directory

Thank you for your help!
Best,
Paulo Schuroff

[ManuelTgn]

Hi @schuroff,

Thank you for your interest in using CRISPRme, and for the detailed report!

The first error you encountered during the gnomAD VCF conversion is caused by the --debug flag. This flag is intended only for development or debugging when running CRISPRme from the source code, not when it’s installed via conda or mamba. If you remove the --debug option and rerun the command, the conversion should proceed normally.

The subsequent ModuleNotFoundError: No module named 'utils' occurred because the convert_gnomAD_vcfs.py script was manually moved outside its original installation directory. When CRISPRme is installed through mamba/conda, its internal modules (like utils) are stored and referenced within the package’s own directory structure. Moving individual files outside that environment prevents Python from finding these dependencies.

Regarding the third error you reported: since one of the previous runs didn’t complete successfully, it’s possible that some intermediate files or directories were left in a corrupt state. I recommend cleaning up your current working directory and recreate the required folder structure automatically before running the next command.

As for the error in the complete-test functionality, this is due to a small bug in the complete_test.py script. At line 464-470, the code:

crisprme_cmd = (
        f"python crisprme.py complete-search --genome {CRISPRME_DIRS[0]}/hg38 "
        f"--thread 4 --bmax 1 --mm 4 --bDNA 1 --bRNA 1 --merge 3 --pam {pam} "
        f"--guide {guide} --vcf {vcf} --samplesID {samplesids} --annotation {encode} "
        f"--gene_annotation {gencode} --output crisprme-test-out --thread {threads} "
        f"{debug_arg}"
)

should be replaced with:

crisprme_cmd = (
        f"crisprme.py complete-search --genome {CRISPRME_DIRS[0]}/hg38 "
        f"--thread 4 --bmax 1 --mm 4 --bDNA 1 --bRNA 1 --merge 3 --pam {pam} "
        f"--guide {guide} --vcf {vcf} --samplesID {samplesids} --annotation {encode} "
        f"--gene_annotation {gencode} --output crisprme-test-out --thread {threads} "
        f"{debug_arg}"
    )

The affected file should be located at /home/tiaf/miniconda3/pkgs/crisprme-2.1.7-hdfd78af_0/opt/crisprme/PostProcess/complete_test.py

We’re aware of this issue and have already fixed it in our development branch. The fix, along with several new functionalities, including a more comprehensive off-target search and improved VCF conversion handling, will be available in the upcoming CRISPRme v2.1.8, which we plan to release next week.

Thank you again for bringing this to our attention and for testing CRISPRme so thoroughly! Please don’t hesitate to reach out if you encounter any further issues.

Best,
Manuel

[schuroff]

Hi Manuel,

Thanks very much for your quick and detailed reply — really appreciate it.
A short update on what I tried after your suggestions:
I reinstalled CRISPRme using conda (fresh environment) and applied the change you suggested in complete_test.py. After that, the command below worked perfectly!
However, when I tried to run the gnomAD-converter on a gnomAD v4.1 joint VCF, I got the following error:

crisprme.py gnomAD-converter --gnomAD_VCFdir VCFs/hg38_HGDP/ --samplesID samplesIDs/samplesIDs.gnomad.v41.txt --joint --thread 20 --multiallelic

Error:
Traceback (most recent call last):
File "/home/omicas/miniconda3/envs/crisprme/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py", line 36, in
import pysam
ModuleNotFoundError: No module named 'pysam'
Traceback (most recent call last):
File "/home/omicas/miniconda3/envs/crisprme/bin/crisprme.py", line 1319, in
gnomAD_converter()
File "/home/omicas/miniconda3/envs/crisprme/bin/crisprme.py", line 1101, in gnomAD_converter
raise subprocess.SubprocessError(
subprocess.SubprocessError: An error occurred while converting gnomAD VCFs in /home/omicas/teste2/VCFs/hg38_HGDP

I then installed pysam into the same environment:

conda install -c bioconda pysam

But after that, I entered a loop of different dependency-related errors, and the conversion still failed.
I also tested this workflow under WSL, Debian, and Ubuntu, on multiple machines — all showed the same behavior.

Thanks again for the support — I can test any proposed fixes and report back.

[ManuelTgn]

Hi @schuroff,

Thank you for reporting this unexpected behavior during the CRISPRme installation in a fresh environment, and for the detailed follow-up tests you performed.

I was able to reproduce the issue you described. It seems that, last week, the dependency solver in mamba/conda was temporarily malfunctioning, which caused inconsistent environment resolution. Fortunately, this was not an issue with CRISPRme itself.

I re-tested the installation this morning, and the process worked correctly again. Below are the exact steps I followed to successfully install CRISPRme v2.1.7 together with pysam:

mamba create -n crisprme-217 python=3.8 -y
mamba activate crisprme-217
mamba install crisprme=2.1.7 -y
mamba install pysam -y  # this should install pysam v0.22.1
crisprme.py  # should print general help
crisprme.py --version  # v2.1.7

Please try following the steps above and let me know if the installation now works correctly on your side.

Best,
Manuel

[schuroff]

Hi Manuel,

Sorry for the delay in replying — I was away for a few days and could not follow your recommendations earlier. I tried reinstalling everything as you suggested:

mamba create -n crisprme-217 python=3.8 -y
mamba activate crisprme-217
mamba install crisprme=2.1.7 -y

When installing pysam mamba install pysam -y # this should install pysam v0.22.1. I encountered the following error:

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.

warning  libmamba Please make sure you understand Anaconda Terms of Services.

warning  libmamba See: https://legal.anaconda.com/policies/en/
bioconda/linux-64                                           Using cache
bioconda/noarch                                             Using cache

Pinned packages:

  - python=3.8

warning  libmamba The specification of the environment does not seem solvable in your current setup.
warning  libmamba For instance, packages from different channels might be specified,
warning  libmamba whilst your current configuration might not allow their resolution.
warning  libmamba
warning  libmamba If it is the case, you need to either:
warning  libmamba  - adapt the channel ordering (e.g. by reordering the `-c` flags in your command line)
warning  libmamba  - use the flexible channel priority (e.g. using `--channel-priority flexible` in your command line)
warning  libmamba
warning  libmamba For reference, see this piece of documentation on channel priority:
warning  libmamba https://docs.conda.io/projects/conda/en/stable/user-guide/tasks/manage-channels.html#strict-channel-priority
error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ pin on python =3.8 * is installable and it requires
    │  └─ python =3.8 *, which can be installed;
    └─ pysam =* * is not installable because there are no viable options
       ├─ pysam 0.23.3 would require
       │  └─ python >=3.10,<3.11.0a0 *, which conflicts with any installable versions previously reported;
       ├─ pysam 0.23.3 would require
       │  └─ python >=3.11,<3.12.0a0 *, which conflicts with any installable versions previously reported;
       ├─ pysam 0.23.3 would require
       │  └─ python >=3.12,<3.13.0a0 *, which conflicts with any installable versions previously reported;
       ├─ pysam 0.23.3 would require
       │  └─ python >=3.13,<3.14.0a0 *, which conflicts with any installable versions previously reported;
       ├─ pysam 0.23.3 would require
       │  └─ python >=3.14,<3.15.0a0 *, which conflicts with any installable versions previously reported;
       ├─ pysam 0.23.3 would require
       │  └─ python >=3.9,<3.10.0a0 *, which conflicts with any installable versions previously reported;
       └─ pysam [0.10.0|0.11.0|...|0.9.1.4] conflicts with any installable versions previously reported.
critical libmamba Could not solve for environment specs

Since conda/mamba could not resolve dependencies, I installed pysam via pip: pip install pysam. That worked, but when I ran the GNOMAD-converter crisprme.py gnomAD-converter --gnomAD_VCFdir VCFs/ --samplesID samplesIDs/samplesIDs.gnomad.v41.txt --joint --thread 20 --multiallelic I got the following error:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/omicas/miniconda3/envs/crisprme-217/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/omicas/miniconda3/envs/crisprme-217/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/omicas/miniconda3/envs/crisprme-217/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py", line 338, in run_conversion_pipeline
    vcf_merged = bcftools_merge(vcf_genotypes, multiallelic)
  File "/home/omicas/miniconda3/envs/crisprme-217/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py", line 308, in bcftools_merge
    raise subprocess.SubprocessError(
subprocess.SubprocessError: An error occurred while merging alleles in /home/omicas/teste2/VCFs/gnomad.joint.v4.1.sites.chr22.vcf.gz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/omicas/miniconda3/envs/crisprme-217/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py", line 385, in <module>
    convert_gnomad_vcfs()
  File "/home/omicas/miniconda3/envs/crisprme-217/opt/crisprme/PostProcess/convert_gnomAD_vcfs.py", line 376, in convert_gnomad_vcfs
    pool.map(partial_run_conversion_pipeline, gnomad_vcfs)
  File "/home/omicas/miniconda3/envs/crisprme-217/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/omicas/miniconda3/envs/crisprme-217/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
subprocess.SubprocessError: An error occurred while merging alleles in /home/omicas/teste2/VCFs/gnomad.joint.v4.1.sites.chr22.vcf.gz
Traceback (most recent call last):
  File "/home/omicas/miniconda3/envs/crisprme-217/bin/crisprme.py", line 1319, in <module>
    gnomAD_converter()
  File "/home/omicas/miniconda3/envs/crisprme-217/bin/crisprme.py", line 1101, in gnomAD_converter
    raise subprocess.SubprocessError(
subprocess.SubprocessError: An error occurred while converting gnomAD VCFs in /home/omicas/teste2/VCFs

Observed behavior:

• I tested only with chromosome 22.
• The converter starts generating the file gnomad.joint.v4.1.sites.chr22.vcf.gz.
• When the file reaches ~45 MB in size, the process fails with the error above.

Should I avoid installing pysam via pip and instead use a specific conda channel/version? What could it be?

Thanks in advance for your help!
Paulo Schuroff