MissingOutputException error: apparently latency time is not enough #568

MauriAndresMU1313 · 2025-12-26T15:16:15Z

MauriAndresMU1313
Dec 26, 2025

Hi there! Hope all the team enjoy Christmas time!

I'm trying to perform an analysis with the following proteins:

$ cat config/sample_sheet.csv 
features_input/anp32a_human_P39687.fasta
features_input/anp32b_human_Q92688.fasta
features_input/calr_human_P30988.fasta
features_input/np_goose_kansas_EPI3222725.fasta
features_input/pb2_goose_kansas.fasta
features_input/pb1_goose_kansas.fasta
features_input/pa_goose_kansas.fasta

I used the following command, changing in multiple times the arguments --jobs --restart-times (from 45 to 70, and 5 to 10, respectively):

snakemake --executor slurm --profile config/profiles/slurm --jobs 45 --restart-times 5

I'm having the following error in the rule_create_features (I will add the complete error at the end of the post).

MissingOutputException in rule create_features in file "https://raw.githubusercontent.com/KosinskiLab/AlphaPulldownSnakemake/main/workflow/Snakefile", line 183:
Job 0 completed successfully, but some output files are missing. Missing files after 600 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait
...

Each time that I increase the restart times, the number of files increased:

Click to expand

P_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz
NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz
NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz
NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz
NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz
NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-24.json.xz
NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz
PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz
PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz
PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz
PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz
PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz
PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz
PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz
PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz
PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz
PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz
PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz
PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-24.json.xz
PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4.pkl.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-18.json.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-19.json.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-22.json.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-23.json.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-24.json.xz
sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-25.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1.pkl.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-18.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-19.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-22.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-23.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-24.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-25.json.xz
sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-26.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1.pkl.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-18.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-19.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-22.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-23.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-24.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-25.json.xz
sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-26.json.xz

I checked the .snakemake/logs/, I'm sure that is because I increased too much the amount of jobs:

    message: SLURM sbatch failed. The error message was 'sbatch: error: Your job is submitted to default account: general
sbatch: error: Not GPU job!
sbatch: error: slurm_job_submit: initialized
sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)'.
    sbatch call:
        sbatch --parsable --job-name f5f9025f-3e6d-40ae-b06d-1ff33c54dace --output slurm-logs/rule_create_features/anp32b_human_Q92688/%j.log --export=ALL --comment rule_create_features_wildcards_anp32b_human_Q92688   -p c64-m512 -t 30240 --qos=normal --mem 473615 --ntasks=1 --cpus-per-task=8 -D /scratch/mmora30/workspace/AlphaPulldownSnakemake --wrap="/users/mmora30/.conda/envs/snake/bin/python3.12 -m snakemake --snakefile '/scratch/mmora30/workspace/AlphaPulldownSnakemake/workflow/Snakefile' --target-jobs 'create_features:fasta_basename=anp32b_human_Q92688' --allowed-rules create_features --cores 'all' --attempt 21 --force-use-threads  --wait-for-files '/scratch/mmora30/workspace/AlphaPulldownSnakemake/.snakemake/tmp.l5qw7nb6' '/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/anp32b_human_Q92688.fasta' --force --target-files-omit-workdir-adjustment --max-inventory-time 0 --nocolor --no-hooks --nolock --ignore-incomplete --rerun-triggers mtime --deployment-method apptainer --conda-frontend 'conda' --apptainer-prefix '/group/rmedin6-g00/singularity_cache' --shared-fs-usage source-cache storage-local-copies software-deployment input-output sources persistence --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --printshellcmds  --latency-wait 600 --scheduler 'ilp' --local-storage-prefix base64//LnNuYWtlbWFrZS9zdG9yYWdl --scheduler-solver-path '/users/mmora30/.conda/envs/snake/bin' --runtime-source-cache-path '/users/mmora30/.cache/snakemake/snakemake/source-cache/runtime-cache/tmpnr_bj8j0' --default-resources base64//bWVtX21iPTgwMDA= base64//ZGlza19tYj1tYXgoMippbnB1dC5zaXplX21iLCAxMDAwKSBpZiBpbnB1dCBlbHNlIDUwMDAw base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= base64//cnVudGltZT02MA== --executor slurm-jobstep --jobs 1 --mode 'remote'"
For further error details see the cluster/cloud log and the log files of the involved rule(s).

It's clearly that the time to complete the job is not enough, nevertheless I have been changing the restart times multiple time in the last couple of weeks. The jobs ran from 5-24 hours depending on the restart-times argument, so I wonder what can I do different? My strategy is clearly not working.

My config.yaml file, in case you need it:

# Arguments for feature generation
create_feature_arguments:
  --save_msa_files: false  # Do not save multiple sequence alignment (MSA) files
  --use_precomputed_msas: true  # Use precomputed MSA files if available
  --max_template_date: 2050-01-01  # Set maximum template date to include all templates
  --compress_features: true  # Compress generated features
  --data_pipeline: alphafold2 # Use alphafold2 or alphafold3 data pipeline for generating features

Click to expand complete error message

Activating singularity image /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/singularity_images/alphafold2_latest.sif
INFO:    Environment variable SINGULARITY_TMPDIR is set, but APPTAINER_TMPDIR is preferred
I1225 18:05:47.749933 139845101958400 templates.py:981] Using precomputed obsolete pdbs /group/rmedin6-g00/full_db/pdb_mmcif/obsolete.dat.
W1225 18:05:48.073805 139845101958400 save_meta_data.py:148] Path does not exist: /group/rmedin6-g00/full_db/small_bfd/bfd-first_non_consensus_sequences.fasta
I1225 18:05:48.652778 139845101958400 objects.py:151] will save msa files in :/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371
I1225 18:05:48.693320 139845101958400 jackhmmer.py:136] Launching subprocess "/opt/conda/bin/jackhmmer -o /dev/null -A /group/rmedin6-g00/singularity_cache/tmpqlsjpa46/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /group/rmedin6-g00/singularity_cache/tmpl3bh61fi.fasta /group/rmedin6-g00/full_db/uniref90/uniref90.fasta"
I1225 18:05:48.693897 139845101958400 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I1225 18:11:52.555329 139845101958400 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 363.861 seconds
I1225 18:11:52.591964 139845101958400 jackhmmer.py:136] Launching subprocess "/opt/conda/bin/jackhmmer -o /dev/null -A /group/rmedin6-g00/singularity_cache/tmp847xx756/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /group/rmedin6-g00/singularity_cache/tmpl3bh61fi.fasta /group/rmedin6-g00/full_db/mgnify/mgy_clusters_2022_05.fa"
I1225 18:11:52.593082 139845101958400 utils.py:36] Started Jackhmmer (mgy_clusters_2022_05.fa) query
I1225 18:23:19.246210 139845101958400 utils.py:40] Finished Jackhmmer (mgy_clusters_2022_05.fa) query in 686.653 seconds
I1225 18:23:19.342443 139845101958400 hmmbuild.py:121] Launching subprocess ['/opt/conda/bin/hmmbuild', '--hand', '--amino', '/group/rmedin6-g00/singularity_cache/tmp81__u6wb/output.hmm', '/group/rmedin6-g00/singularity_cache/tmp81__u6wb/query.msa']
I1225 18:23:19.343913 139845101958400 utils.py:36] Started hmmbuild query
I1225 18:23:19.659510 139845101958400 hmmbuild.py:129] hmmbuild stdout:
# hmmbuild :: profile HMM construction from multiple sequence alignments
# HMMER 3.4 (Aug 2023); http://hmmer.org/
# Copyright (C) 2023 Howard Hughes Medical Institute.
# Freely distributed under the BSD open source license.
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# input alignment file:             /group/rmedin6-g00/singularity_cache/tmp81__u6wb/query.msa
# output HMM file:                  /group/rmedin6-g00/singularity_cache/tmp81__u6wb/output.hmm
# input alignment is asserted as:   protein
# model architecture construction:  hand-specified by RF annotation
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

# idx name                  nseq  alen  mlen eff_nseq re/pos description
#---- -------------------- ----- ----- ----- -------- ------ -----------
1     query                  659   944   759     5.04  0.590 

# CPU time: 1.03u 0.02s 00:00:01.05 Elapsed: 00:00:00.28


stderr:


I1225 18:23:19.659555 139845101958400 utils.py:40] Finished hmmbuild query in 0.316 seconds
I1225 18:23:19.718532 139845101958400 hmmsearch.py:114] Launching sub-process ['/opt/conda/bin/hmmsearch', '--noali', '--cpu', '8', '--F1', '0.1', '--F2', '0.1', '--F3', '0.1', '--incE', '100', '-E', '100', '--domE', '100', '--incdomE', '100', '-A', '/group/rmedin6-g00/singularity_cache/tmps8dki09h/output.sto', '/group/rmedin6-g00/singularity_cache/tmps8dki09h/query.hmm', '/group/rmedin6-g00/full_db/pdb_seqres/pdb_seqres.txt']
I1225 18:23:19.719028 139845101958400 utils.py:36] Started hmmsearch (pdb_seqres.txt) query
I1225 18:23:37.086899 139845101958400 utils.py:40] Finished hmmsearch (pdb_seqres.txt) query in 17.368 seconds
I1225 18:23:37.184619 139845101958400 hhblits.py:132] Launching subprocess "/opt/conda/bin/hhblits -i /group/rmedin6-g00/singularity_cache/tmpl3bh61fi.fasta -cpu 8 -oa3m /group/rmedin6-g00/singularity_cache/tmpa50c2nj9/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /group/rmedin6-g00/full_db/bfd/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt -d /group/rmedin6-g00/full_db/uniref30/UniRef30_2023_02"
I1225 18:23:37.185426 139845101958400 utils.py:36] Started HHblits query
I1225 18:35:57.671277 139845101958400 utils.py:40] Finished HHblits query in 740.486 seconds
I1225 18:35:57.690943 139845101958400 templates.py:1070] Searching for templates for: MERIKELRDLMSQSRTREILTKTTVDHMAIIKKYTSGRQEKNPALRMKWMMAMKYPITADKRIMEMIPERNEQGQTLWSKTNDAGSDRVMVSPLAVTWWNRNGPTTSTVHYPKVYKTYFEKVERLKHGTFGPVHFRNQVKIRRRVDINPGHADLSAKEAQDVIMEVVFPNEVGARILTSESQLTITKEKKEELQDCKIAPLMVAYMLERELVRKTRFLPVAGGTSSVYIEVLHLTQGTCWEQMYTPGGEVRNDDVDQSLIIAARNIVRRATVSADPLASLLEMCHSTQIGGIRMVDILRQNPTEEQAVDICKAAMGLRISSSFSFGGFTFKRTSGSSVKKEEEVLTGNLQTLKIRVHEGYEEFTMVGRRATAILRKATRRLIQLIVSGRDEQSIAEAIIVAMVFSQEDCMIKAVRGDLNFVNRANQRLNPMHQLLRHFQKDAKVLFQNWGIEPIDNVMGMIGIMPDMTPSTEMSLRGIRVSKMGVDEYSSTERVVVSIDRFLRVRDQRGNVLLSPEEVSETQGTEKLTITYSSSMMWEINGPESVLVNTYQWIIRNWETVKIQWSQDPTMLYNKMEFEPFQSLVPKAARGQYSGFVRTLFQQMRDVLGTFDTVQIIKLLPFAAAPPEQSRMQFSSLTVNVRGSGMRILVRGNSPVFNYNKATKRLTVLGKDAGALTEDPDEGTAGVESAVLRGFLILGKEDKRYGPALSINELSNLAKGEKANVLIGQGDVVLVMKRKRDSSILTDSQTATKRIRMAIN
I1225 18:35:58.125044 139845101958400 templates.py:287] Found an exact template match 8h69_C.
I1225 18:35:58.862535 139845101958400 templates.py:287] Found an exact template match 7nha_C.
I1225 18:35:59.567736 139845101958400 templates.py:287] Found an exact template match 7nhc_C.
I1225 18:36:00.464298 139845101958400 templates.py:287] Found an exact template match 7nhx_C.
I1225 18:36:01.292593 139845101958400 templates.py:287] Found an exact template match 7ni0_C.
I1225 18:36:02.061100 139845101958400 templates.py:287] Found an exact template match 7nik_C.
I1225 18:36:02.624050 139845101958400 templates.py:287] Found an exact template match 7nil_C.
I1225 18:36:03.510381 139845101958400 templates.py:287] Found an exact template match 7nir_C.
I1225 18:36:04.065886 139845101958400 templates.py:287] Found an exact template match 7nis_C.
I1225 18:36:04.989655 139845101958400 templates.py:287] Found an exact template match 7nj3_C.
I1225 18:36:05.971237 139845101958400 templates.py:287] Found an exact template match 7nj4_C.
I1225 18:36:06.592844 139845101958400 templates.py:287] Found an exact template match 7nj5_C.
I1225 18:36:07.632427 139845101958400 templates.py:287] Found an exact template match 7nj7_C.
I1225 18:36:08.252074 139845101958400 templates.py:287] Found an exact template match 7nk1_C.
I1225 18:36:08.871070 139845101958400 templates.py:287] Found an exact template match 7nk2_C.
I1225 18:36:09.986389 139845101958400 templates.py:287] Found an exact template match 7nk4_C.
I1225 18:36:10.612493 139845101958400 templates.py:287] Found an exact template match 7nk6_C.
I1225 18:36:11.834093 139845101958400 templates.py:287] Found an exact template match 7nk8_C.
I1225 18:36:12.458610 139845101958400 templates.py:287] Found an exact template match 7nka_C.
I1225 18:36:13.079294 139845101958400 templates.py:287] Found an exact template match 7nkc_C.
I1225 18:36:14.362965 139845101958400 templates.py:287] Found an exact template match 7nki_C.
I1225 18:36:14.984066 139845101958400 templates.py:287] Found an exact template match 7nkr_C.
I1225 18:36:16.229462 139845101958400 templates.py:287] Found an exact template match 8r60_C.
I1225 18:36:16.969929 139845101958400 templates.py:287] Found an exact template match 8r65_C.
I1225 18:36:19.041202 139845101958400 templates.py:287] Found an exact template match 8r1j_C.
I1225 18:36:19.060118 139845101958400 templates.py:287] Found an exact template match 8r1j_F.
I1225 18:36:19.558774 139845101958400 templates.py:287] Found an exact template match 8r1l_C.
I1225 18:36:19.959451 139845101958400 templates.py:287] Found an exact template match 7qtl_C.
I1225 18:36:20.370981 139845101958400 templates.py:287] Found an exact template match 7r0e_C.
I1225 18:36:21.563328 139845101958400 templates.py:287] Found an exact template match 7zpl_C.
I1225 18:36:21.568003 139845101958400 templates.py:287] Found an exact template match 7zpl_F.
I1225 18:36:22.005418 139845101958400 templates.py:287] Found an exact template match 7zpm_C.
I1225 18:36:22.010027 139845101958400 templates.py:287] Found an exact template match 7zpm_F.
I1225 18:36:22.412710 139845101958400 templates.py:287] Found an exact template match 8bek_C.
I1225 18:36:24.472578 139845101958400 templates.py:287] Found an exact template match 9fyx_c.
I1225 18:36:26.483029 139845101958400 templates.py:287] Found an exact template match 9g0a_c.
I1225 18:36:26.962343 139845101958400 templates.py:287] Found an exact template match 8poh_C.
I1225 18:36:26.967480 139845101958400 templates.py:287] Found an exact template match 8poh_G.
I1225 18:36:29.178902 139845101958400 templates.py:287] Found an exact template match 6qpf_C.
I1225 18:36:29.193067 139845101958400 templates.py:287] Found an exact template match 6qpf_F.
I1225 18:36:29.207033 139845101958400 templates.py:287] Found an exact template match 6qpf_I.
I1225 18:36:29.221650 139845101958400 templates.py:287] Found an exact template match 6qpf_L.
I1225 18:36:29.608627 139845101958400 templates.py:287] Found an exact template match 8pm0_C.
I1225 18:36:29.948517 139845101958400 templates.py:287] Found an exact template match 8pnp_C.
I1225 18:36:30.293609 139845101958400 templates.py:287] Found an exact template match 8pnq_C.
I1225 18:36:30.635583 139845101958400 templates.py:287] Found an exact template match 8r3k_C.
I1225 18:36:32.290077 139845101958400 templates.py:287] Found an exact template match 8r3l_C.
I1225 18:36:32.913357 139845101958400 templates.py:287] Found an exact template match 8rmp_C.
I1225 18:36:33.493928 139845101958400 templates.py:287] Found an exact template match 8rmq_C.
I1225 18:36:35.773272 139845101958400 templates.py:287] Found an exact template match 8rmr_C.
I1225 18:36:35.794476 139845101958400 templates.py:287] Found an exact template match 8rmr_F.
I1225 18:36:36.467160 139845101958400 templates.py:287] Found an exact template match 8rms_C.
I1225 18:36:38.003492 139845101958400 templates.py:287] Found an exact template match 8rn0_C.
I1225 18:36:38.010165 139845101958400 templates.py:287] Found an exact template match 8rn0_F.
I1225 18:36:38.864227 139845101958400 templates.py:287] Found an exact template match 6qx8_C.
I1225 18:36:38.869336 139845101958400 templates.py:287] Found an exact template match 6qx8_G.
I1225 18:36:40.581145 139845101958400 templates.py:287] Found an exact template match 6qxe_C.
I1225 18:36:40.586227 139845101958400 templates.py:287] Found an exact template match 6qxe_G.
I1225 18:36:43.091608 139845101958400 templates.py:287] Found an exact template match 6qnw_C.
I1225 18:36:43.105702 139845101958400 templates.py:287] Found an exact template match 6qnw_F.
I1225 18:36:43.119287 139845101958400 templates.py:287] Found an exact template match 6qnw_I.
I1225 18:36:43.132822 139845101958400 templates.py:287] Found an exact template match 6qnw_L.
I1225 18:36:45.683219 139845101958400 templates.py:287] Found an exact template match 6qpg_C.
I1225 18:36:45.697896 139845101958400 templates.py:287] Found an exact template match 6qpg_F.
I1225 18:36:45.712638 139845101958400 templates.py:287] Found an exact template match 6qpg_I.
I1225 18:36:45.726711 139845101958400 templates.py:287] Found an exact template match 6qpg_L.
I1225 18:36:47.513681 139845101958400 templates.py:287] Found an exact template match 6rr7_C.
I1225 18:36:47.941016 139845101958400 templates.py:287] Found an exact template match 9v44_C.
I1225 18:36:48.254251 139845101958400 templates.py:287] Found an exact template match 9v46_C.
I1225 18:36:48.856227 139845101958400 templates.py:287] Found an exact template match 8y7m_C.
I1225 18:36:48.864096 139845101958400 templates.py:287] Found an exact template match 8y7m_G.
I1225 18:36:51.966907 139845101958400 templates.py:287] Found an exact template match 8y7o_C.
I1225 18:36:51.975242 139845101958400 templates.py:287] Found an exact template match 8y7o_D.
I1225 18:36:51.982300 139845101958400 templates.py:287] Found an exact template match 8y7o_G.
I1225 18:36:51.990497 139845101958400 templates.py:287] Found an exact template match 8y7o_K.
I1225 18:36:51.998162 139845101958400 templates.py:287] Found an exact template match 8y7o_O.
I1225 18:36:52.005863 139845101958400 templates.py:287] Found an exact template match 8y7o_P.
I1225 18:36:52.498811 139845101958400 templates.py:287] Found an exact template match 6qx3_C.
I1225 18:36:52.908503 139845101958400 templates.py:287] Found an exact template match 4wsb_C.
I1225 18:36:54.357005 139845101958400 templates.py:287] Found an exact template match 5m3h_C.
I1225 18:36:55.165782 139845101958400 templates.py:287] Found an exact template match 6evj_C.
I1225 18:36:55.180314 139845101958400 templates.py:287] Found an exact template match 6evj_F.
I1225 18:36:55.610076 139845101958400 templates.py:287] Found an exact template match 6evk_C.
I1225 18:36:56.034390 139845101958400 templates.py:287] Found an exact template match 6fhh_C.
I1225 18:36:56.452652 139845101958400 templates.py:287] Found an exact template match 6fhi_C.
I1225 18:36:57.936547 139845101958400 templates.py:287] Found an exact template match 6szu_C.
I1225 18:36:58.282878 139845101958400 templates.py:287] Found an exact template match 6szv_C.
I1225 18:36:58.655856 139845101958400 templates.py:287] Found an exact template match 6t0n_C.
I1225 18:36:59.014158 139845101958400 templates.py:287] Found an exact template match 6t0r_C.
I1225 18:36:59.351716 139845101958400 templates.py:287] Found an exact template match 6t0s_C.
I1225 18:36:59.669847 139845101958400 templates.py:287] Found an exact template match 6t0u_C.
I1225 18:37:01.115503 139845101958400 templates.py:287] Found an exact template match 6t0v_C.
I1225 18:37:01.405144 139845101958400 templates.py:287] Found an exact template match 6t2c_C.
I1225 18:37:01.853370 139845101958400 templates.py:287] Found an exact template match 6tw1_C.
I1225 18:37:02.333932 139845101958400 templates.py:287] Found an exact template match 9f2r_C.
I1225 18:37:03.506443 139845101958400 templates.py:287] Found an exact template match 9f37_C.
I1225 18:37:03.786947 139845101958400 templates.py:287] Found an exact template match 6qwl_Q.
I1225 18:37:04.194416 139845101958400 templates.py:287] Found an exact template match 6f5o_C.
I1225 18:37:07.616515 139845101958400 templates.py:287] Found an exact template match 5epi_C.
I1225 18:37:07.631098 139845101958400 templates.py:287] Found an exact template match 5epi_G.
I1225 18:37:07.645147 139845101958400 templates.py:287] Found an exact template match 5epi_K.
I1225 18:37:07.659389 139845101958400 templates.py:287] Found an exact template match 5epi_O.
I1225 18:37:07.673587 139845101958400 templates.py:287] Found an exact template match 5epi_S.
I1225 18:37:07.687381 139845101958400 templates.py:287] Found an exact template match 5epi_W.
I1225 18:37:08.036948 139845101958400 templates.py:287] Found an exact template match 4wrt_C.
I1225 18:37:09.044030 139845101958400 templates.py:287] Found an exact template match 4wsa_C.
I1225 18:37:09.787295 139845101958400 templates.py:287] Found an exact template match 5fmz_C.
I1225 18:37:09.800662 139845101958400 templates.py:287] Found an exact template match 5fmz_F.
I1225 18:37:10.455061 139845101958400 templates.py:287] Found an exact template match 5m3j_C.
I1225 18:37:11.519392 139845101958400 templates.py:287] Found an exact template match 5msg_C.
I1225 18:37:11.926710 139845101958400 templates.py:287] Found an exact template match 6qcs_C.
I1225 18:37:12.337319 139845101958400 templates.py:287] Found an exact template match 6qct_C.
I1225 18:37:13.009954 139845101958400 templates.py:287] Found an exact template match 6qcv_C.
I1225 18:37:14.385669 139845101958400 templates.py:287] Found an exact template match 6qcw_C.
I1225 18:37:15.055954 139845101958400 templates.py:287] Found an exact template match 6qcx_C.
I1225 18:37:15.476846 139845101958400 templates.py:287] Found an exact template match 7r1f_C.
I1225 18:37:17.013404 139845101958400 templates.py:287] Found an exact template match 7z42_C.
I1225 18:37:17.027381 139845101958400 templates.py:287] Found an exact template match 7z42_F.
I1225 18:37:17.477560 139845101958400 templates.py:287] Found an exact template match 7z43_CCC.
I1225 18:37:18.240431 139845101958400 templates.py:287] Found an exact template match 8bdr_C.
I1225 18:37:19.793001 139845101958400 templates.py:287] Found an exact template match 8be0_C.
I1225 18:37:20.591181 139845101958400 templates.py:287] Found an exact template match 8bf5_C.
I1225 18:37:20.861063 139845101958400 templates.py:287] Found an exact template match 6t0w_C.
I1225 18:37:22.372628 139845101958400 templates.py:287] Found an exact template match 8rn1_C.
I1225 18:37:23.032662 139845101958400 templates.py:287] Found an exact template match 8rn2_C.
I1225 18:37:24.420029 139845101958400 templates.py:287] Found an exact template match 8rn3_C.
I1225 18:37:25.011660 139845101958400 templates.py:287] Found an exact template match 8rn4_C.
I1225 18:37:25.702198 139845101958400 templates.py:287] Found an exact template match 8rn5_C.
I1225 18:37:26.889430 139845101958400 templates.py:287] Found an exact template match 8rn6_C.
I1225 18:37:27.430218 139845101958400 templates.py:287] Found an exact template match 8rn7_C.
I1225 18:37:29.213018 139845101958400 templates.py:287] Found an exact template match 8rn8_C.
I1225 18:37:29.229595 139845101958400 templates.py:287] Found an exact template match 8rn8_F.
I1225 18:37:29.983731 139845101958400 templates.py:287] Found an exact template match 8rn9_C.
I1225 18:37:32.647411 139845101958400 templates.py:287] Found an exact template match 8rna_C.
I1225 18:37:32.666913 139845101958400 templates.py:287] Found an exact template match 8rna_F.
I1225 18:37:32.685008 139845101958400 templates.py:287] Found an exact template match 8rna_Z.
I1225 18:37:34.315517 139845101958400 templates.py:287] Found an exact template match 8rnb_C.
I1225 18:37:34.322411 139845101958400 templates.py:287] Found an exact template match 8rnb_F.
I1225 18:37:35.800835 139845101958400 templates.py:287] Found an exact template match 8rnc_C.
I1225 18:37:35.820764 139845101958400 templates.py:287] Found an exact template match 8rnc_F.
I1225 18:37:35.992446 139845101958400 templates.py:287] Found an exact template match 5wl0_A.
I1225 18:37:36.098179 139845101958400 templates.py:287] Found an exact template match 5fmm_A.
I1225 18:37:36.203692 139845101958400 templates.py:287] Found an exact template match 5fmq_A.
I1225 18:37:37.442212 139845101958400 templates.py:287] Found an exact template match 6kuj_C.
I1225 18:37:37.750706 139845101958400 templates.py:287] Found an exact template match 6kuk_C.
I1225 18:37:38.042897 139845101958400 templates.py:287] Found an exact template match 6kup_C.
I1225 18:37:38.337961 139845101958400 templates.py:287] Found an exact template match 6kur_C.
I1225 18:37:38.662584 139845101958400 templates.py:287] Found an exact template match 6kut_C.
I1225 18:37:38.957232 139845101958400 templates.py:287] Found an exact template match 6kuu_C.
I1225 18:37:39.269271 139845101958400 templates.py:287] Found an exact template match 6kuv_C.
I1225 18:37:40.452762 139845101958400 templates.py:287] Found an exact template match 6kv5_C.
I1225 18:37:41.228105 139845101958400 templates.py:287] Found an exact template match 5d98_C.
I1225 18:37:41.243220 139845101958400 templates.py:287] Found an exact template match 5d98_F.
I1225 18:37:43.412024 139845101958400 templates.py:287] Found an exact template match 5d9a_C.
I1225 18:37:43.427936 139845101958400 templates.py:287] Found an exact template match 5d9a_F.
I1225 18:37:43.443304 139845101958400 templates.py:287] Found an exact template match 5d9a_I.
I1225 18:37:43.457928 139845101958400 templates.py:287] Found an exact template match 5d9a_L.
I1225 18:37:45.447276 139845101958400 templates.py:287] Found an exact template match 6f5p_E.
I1225 18:37:45.462201 139845101958400 templates.py:287] Found an exact template match 6f5p_F.
I1225 18:37:46.859827 139845101958400 templates.py:287] Found an exact template match 6xzd_CP1.
I1225 18:37:46.880494 139845101958400 templates.py:287] Found an exact template match 6xzd_FP1.
I1225 18:37:49.129604 139845101958400 templates.py:287] Found an exact template match 6xzg_CP1.
I1225 18:37:49.150315 139845101958400 templates.py:287] Found an exact template match 6xzg_FP1.
I1225 18:37:51.363966 139845101958400 templates.py:287] Found an exact template match 6xzp_CP1.
I1225 18:37:51.385889 139845101958400 templates.py:287] Found an exact template match 6xzp_FP1.
I1225 18:37:53.347548 139845101958400 templates.py:287] Found an exact template match 6xzq_C.
I1225 18:37:53.367330 139845101958400 templates.py:287] Found an exact template match 6xzq_F.
I1225 18:37:55.626267 139845101958400 templates.py:287] Found an exact template match 6xzr_CP1.
I1225 18:37:55.646167 139845101958400 templates.py:287] Found an exact template match 6xzr_FP1.
I1225 18:37:56.407793 139845101958400 templates.py:287] Found an exact template match 6y0c_C.
I1225 18:37:56.724563 139845101958400 templates.py:287] Found an exact template match 6s5v_AAAA.
I1225 18:37:56.732500 139845101958400 templates.py:287] Found an exact template match 6s5v_ABAA.
I1225 18:37:57.118596 139845101958400 templates.py:287] Found an exact template match 7nfq_A.
I1225 18:37:57.126131 139845101958400 templates.py:287] Found an exact template match 7nfq_B.
I1225 18:37:57.326601 139845101958400 templates.py:287] Found an exact template match 7nfr_A.
I1225 18:37:57.599290 139845101958400 templates.py:287] Found an exact template match 7nft_A.
I1225 18:37:57.604585 139845101958400 templates.py:287] Found an exact template match 7nft_B.
I1225 18:37:57.814726 139845101958400 templates.py:287] Found an exact template match 6euv_A.
I1225 18:37:57.820430 139845101958400 templates.py:287] Found an exact template match 6euv_C.
I1225 18:37:57.826167 139845101958400 templates.py:287] Found an exact template match 6euv_F.
I1225 18:37:57.831879 139845101958400 templates.py:287] Found an exact template match 6euv_I.
I1225 18:37:58.075892 139845101958400 templates.py:287] Found an exact template match 6euy_A.
I1225 18:37:58.081546 139845101958400 templates.py:287] Found an exact template match 6euy_B.
I1225 18:37:58.087141 139845101958400 templates.py:287] Found an exact template match 6euy_C.
I1225 18:37:58.143533 139845101958400 templates.py:287] Found an exact template match 3cw4_A.
I1225 18:37:58.198527 139845101958400 pipeline.py:267] Uniref90 MSA size: 661 sequences.
I1225 18:37:58.198664 139845101958400 pipeline.py:268] BFD MSA size: 93 sequences.
I1225 18:37:58.198688 139845101958400 pipeline.py:269] MGnify MSA size: 2 sequences.
I1225 18:37:58.198709 139845101958400 pipeline.py:270] Final (deduplicated) MSA size: 752 sequences.
I1225 18:37:58.198758 139845101958400 pipeline.py:274] Total number of templates (NB: this can include bad templates and is later filtered to top 4): 20.
I1225 18:37:58.201212 139845101958400 objects.py:113] now going to run uniprot runner and save uniprot alignment in /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371
I1225 18:37:58.211355 139845101958400 jackhmmer.py:136] Launching subprocess "/opt/conda/bin/jackhmmer -o /dev/null -A /group/rmedin6-g00/singularity_cache/tmps_8jp6jx/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /group/rmedin6-g00/singularity_cache/tmpl3bh61fi.fasta /group/rmedin6-g00/full_db/uniprot/uniprot.fasta"
I1225 18:37:58.212909 139845101958400 utils.py:36] Started Jackhmmer (uniprot.fasta) query
I1225 18:45:23.066232 139845101958400 utils.py:40] Finished Jackhmmer (uniprot.fasta) query in 444.853 seconds
I1225 18:45:31.076642 139845101958400 objects.py:177] You chose not to save MSA files but still want to use precomputed MSA files.
I1225 18:45:31.076762 139845101958400 objects.py:178] MSA files are deleted because features were successfully generated.
Waiting at most 600 seconds for missing files:
/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/pb2_goose_kansas.pkl.xz (missing locally)
MissingOutputException in rule create_features in file "https://raw.githubusercontent.com/KosinskiLab/AlphaPulldownSnakemake/main/workflow/Snakefile", line 183:
Job 0 completed successfully, but some output files are missing. Missing files after 600 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/pb2_goose_kansas.pkl.xz (missing locally, parent dir contents: sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-23.json.xz, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-24.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1.pkl.xz, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-19.json.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-25.json.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-18.json.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-24.json.xz, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-22.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-22.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-18.json.xz, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-25.json.xz, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-19.json.xz, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-23.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz, PB1_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-22.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-19.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-24.json.xz, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-23.json.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-18.json.xz, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-24.json.xz, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1.pkl.xz, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-22.json.xz, NP_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371.pkl.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-19.json.xz, sp_Q92688_AN32B_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_B_OS_Homo_sapiens_OX_9606_GN_ANP32B_PE_1_SV_1_feature_metadata_2025-12-25.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371, PB2_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-24.json.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4.pkl.xz, sp_P30988_CALCR_HUMAN_Calcitonin_receptor_OS_Homo_sapiens_OX_9606_GN_CALCR_PE_1_SV_4_feature_metadata_2025-12-23.json.xz, sp_P39687_AN32A_HUMAN_Acidic_leucine-rich_nuclear_phosphoprotein_32_family_member_A_OS_Homo_sapiens_OX_9606_GN_ANP32A_PE_1_SV_1_feature_metadata_2025-12-18.json.xz, PA_A_Ross_goose_Kansas_W23-930F_2023_2.3.4.4b_2023-12-04_EPI_ISL_19064371_feature_metadata_2025-12-25.json.xz)
Shutting down, this might take some time.
Exiting because a job execution failed. Look below for error messages
[Thu Dec 25 18:55:43 2025]
Error in rule create_features:
    message: None
    jobid: 0
    input: /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/pb2_goose_kansas.fasta
    output: /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/pb2_goose_kansas.pkl.xz
    shell:
        
        create_individual_features.py \
            --fasta_paths=/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/pb2_goose_kansas.fasta \
            --data_dir=/group/rmedin6-g00/full_db \
            --output_dir=/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features \
            --save_msa_files=False --use_precomputed_msas=True --max_template_date=2050-01-01 --compress_features=True --data_pipeline=alphafold2
        
        (command exited with non-zero exit code)
Storing output in storage.
WorkflowError:
At least one job did not complete successfully.
srun: error: c64-m512-dy-r7i-16xlarge-2: task 0: Exited with exit code 1
[Thu Dec 25 18:55:43 2025]
Error in rule create_features:
    message: None
    jobid: 0
    input: /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/pb2_goose_kansas.fasta
    output: /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/pb2_goose_kansas.pkl.xz
    shell:
        
        create_individual_features.py \
            --fasta_paths=/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/pb2_goose_kansas.fasta \
            --data_dir=/group/rmedin6-g00/full_db \
            --output_dir=/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features \
            --save_msa_files=False --use_precomputed_msas=True --max_template_date=2050-01-01 --compress_features=True --data_pipeline=alphafold2
        
        (command exited with non-zero exit code)
Shutting down, this might take some time.
Exiting because a job execution failed. Look below for error messages
[Thu Dec 25 18:55:43 2025]
Error in rule create_features:
    message: None
    jobid: 0
    input: /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/pb2_goose_kansas.fasta
    output: /users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features/pb2_goose_kansas.pkl.xz
    shell:
        
        create_individual_features.py \
            --fasta_paths=/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/data/pb2_goose_kansas.fasta \
            --data_dir=/group/rmedin6-g00/full_db \
            --output_dir=/users/mmora30/mmora30/workspace/AlphaPulldownSnakemake/outputs/features \
            --save_msa_files=False --use_precomputed_msas=True --max_template_date=2050-01-01 --compress_features=True --data_pipeline=alphafold2
        
        (command exited with non-zero exit code)
Storing output in storage.
WorkflowError:
At least one job did not complete successfully.

My questions?

What can I do different?
Do you recommend a different argument to complete the job?

Any comment is greatly appreciated! Thank you!

DimaMolod · 2026-01-06T10:11:23Z

DimaMolod
Jan 6, 2026
Maintainer

Hi @MauriAndresMU1313
this is not a snakemake latency problem and not fixable by --restart-times. Increasing --restart-times just resubmits more jobs, which again fail submission. You see SLURM submission failures because of hitting walltime limits that you need to increase.

More importantly, you are massively oversubscribing the cluster (--jobs 45 with 460 GB RAM). Many of your jobs never start or get killed, so Snakemake later reports “missing output”. This is a scheduler/resource issue, not an AlphaPulldown bug.

What to do:

Save MSA files and read from files:

create_feature_arguments:
 --save_msa_files: true
 --use_precomputed_msas: true
 --compress_features: false

Reduce parallelism (start with --jobs 2–4).
Fix walltime and memory in your SLURM profile to match cluster limits.
If possible, run feature generation on node-local scratch, not shared FS.
Only use --restart-times for transient failures (1–2 max), not as a workaround.

Once a single create_features job completes cleanly, you can scale up. Right now the scheduler is the bottleneck, not Snakemake or AlphaPulldown.

3 replies

MauriAndresMU1313 Jan 7, 2026
Author

Hi! @DimaMolod Happy New Year!
My apologies, big mistake by my part, I misunderstood the use of those arguments, thank you for that clarification.
I made the corrections reducing parallelism (2) and I edited the config/profile/slurm/config.yaml to change wall time (60 to 24h) and memory (8000 to 10000), I increased both).
However, again I got the same error.
Do you need that I provide another file or more details about the run?
Is possible that you provide a example with this amount of proteins? (for example, using this arguments, adding this wall time, using the following configuration, etc).
Any information that allow me to perform the job successfully, thank you again for your help.

MauriAndresMU1313 Jan 9, 2026
Author

I forgot to ask, how much memory do you think that would be needed for that amount of proteins that I'm trying to analyze?
Well, looking forward for your feedback! Thank you @DimaMolod

DimaMolod Jan 9, 2026
Maintainer

Hi @MauriAndresMU1313
The amount of memory depends not only on the protein size, but also on the depth of MSA, the number of templates, etc.
As a rule of thumb, memory depends exponentially on the total length of your complex, and we tried to approximate it for the AP2 paper for NVIDIA RTX 3090 GPU:

However, it depends on your specific protein, type of GPU, and other factors.

Hope this helps
Dima

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MissingOutputException error: apparently latency time is not enough #568

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

MissingOutputException error: apparently latency time is not enough #568

Uh oh!

MauriAndresMU1313 Dec 26, 2025

Replies: 1 comment · 3 replies

Uh oh!

DimaMolod Jan 6, 2026 Maintainer

Uh oh!

Uh oh!

MauriAndresMU1313 Jan 7, 2026 Author

Uh oh!

MauriAndresMU1313 Jan 9, 2026 Author

Uh oh!

Uh oh!

DimaMolod Jan 9, 2026 Maintainer

MauriAndresMU1313
Dec 26, 2025

Replies: 1 comment 3 replies

DimaMolod
Jan 6, 2026
Maintainer

MauriAndresMU1313 Jan 7, 2026
Author

MauriAndresMU1313 Jan 9, 2026
Author

DimaMolod Jan 9, 2026
Maintainer