OverflowError: cannot serialize a string larger than 4GiB and struct.error: 'i' format requires -2147483648 <= number <= 2147483647

[Joyjoyjoyc]

Hi, everyone! I'm using this tool and the function -f "Variant_classification" to annotate my own mutation sites, but I got this error, mainly about multiprocessing, I wonder how to figure it out? Thank you!

use /home/cuiyiran/software/DEMINING-master/DEMINING.conf as config_file
[Thu Nov 14 15:37:25 CST 2024]Finish parse_parameters
The Parameters of DEMINING:
genome_build_version=hg38
rDNA_index_bwa_mem=/home/cuiyiran/software/DEMINING-master/Test_data/DEMINING_dep_files/rDNA.fa
genome_fasta=/home/cuiyiran/data/mtDNA_mutation/reference/GRCh38.primary_assembly.genome.fa
genome_index_hisat2=/home/cuiyiran/data/mtDNA_mutation/reference/GRCh38.primary_assembly.genome
annotation_splice_sites=/home/cuiyiran/data/mtDNA_mutation/reference/splice_sites.hisat
genome_index_bwa_mem=/home/cuiyiran/reference/FASTA_and_GTF/human/GRCh38.primary_assembly.genome.fa
genome_index_gatk=/home/cuiyiran/data/mtDNA_mutation/reference/GRCh38.primary_assembly.genome.dict
dbSNP_all=/home/cuiyiran/data/mtDNA_mutation/reference/dbsnp_138.hg38.vcf
dbSNP_all_index_gatk=/home/cuiyiran/data/mtDNA_mutation/reference/dbsnp_138.hg38.vcf.idx
-f Variant_classification -n WCA157119_rna_panel -t 30 -g hg38 -o output -c /home/cuiyiran/software/DEMINING-master/DEMINING.conf -b /home/cuiyiran/data/mtDNA_mutation/sc_mut_pipline/WCA157119_rna_panel/WCA157119_rna_panel.BQSR.bam -s output_file.txt
Thu Nov 14 15:37:25 CST 2024 dir:/home/cuiyiran/data/mtDNA_mutation/DEMINING command:/home/cuiyiran/software/DEMINING-master/src/DEMINING_main.sh STEP4_DeepDDR_predict hg38 WCA157119_rna_panel output_file.txt /home/cuiyiran/data/mtDNA_mutation/sc_mut_pipline/WCA157119_rna_panel/WCA157119_rna_panel.BQSR.bam output/WCA157119_rna_panel_DEMINING.tsv output/WCA157119_rna_panel_DEMINING_tmp 30
python3 /home/cuiyiran/software/DEMINING-master/src/CMC_construction_func.py WCA157119_rna_panel output_file.txt /home/cuiyiran/data/mtDNA_mutation/sc_mut_pipline/WCA157119_rna_panel/WCA157119_rna_panel.BQSR.bam output/WCA157119_rna_panel_DEMINING_tmp 30 False /home/cuiyiran/data/mtDNA_mutation/reference/GRCh38.primary_assembly.genome.fa /home/cuiyiran/software/DEMINING-master/DeepDDR output/WCA157119_rna_panel_DEMINING_tmp/2-1-1-CMC_ER5HPB3MR2_Length-601_Sample-WCA157119_rna_panel.txt.gz output/WCA157119_rna_panel_DEMINING.tsv
[pid:12947; Thu Nov 14 15:38:04 2024 ], Start!
WCA157119_rna_panel Thu Nov 14 15:38:04 2024 Starting
Process ForkPoolWorker-2:
Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 130, in worker
put((job, i, (False, wrapped)))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
Process ForkPoolWorker-4:
Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 130, in worker
put((job, i, (False, wrapped)))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
Process ForkPoolWorker-26:
Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 130, in worker
put((job, i, (False, wrapped)))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 341, in put
obj = _ForkingPickler.dumps(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
OverflowError: cannot serialize a string larger than 4GiB
Process ForkPoolWorker-29:
Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/pool.py", line 130, in worker
put((job, i, (False, wrapped)))
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/cuiyiran/miniconda3/envs/DEMINING_env/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647