Noise Operator Extraction Runtime

[Sats2]

When extracting noise operators for systems with 9 qubits or more, runtime and memory usage become prohibitively large. At high thresholds, this step takes a significant portion of the total runtime, limiting the practicality of timely extractions. Optimizations in both runtime and memory usage are highly needed, especially for personal system simulations.

For example, running a 10-qubit system with the full noise model currently:

• Takes ~55 minutes for extraction
• Consumes ~330 GB of memory

This excessive runtime and memory footprint make the extracted noise model difficult to use in practice, as the results risk becoming outdated due to domain shift before they can be meaningfully applied.

[Sats2]

Current Idea for reduction of:

• Runtime - Parallelize the noise model generation.
• Memory - Store the noise operators as sparse matrices instead of dense storage.

[Sats2]

This optimization will be added after the upcoming Heron chip extension for the package.

[Sats2]

Current version release (v1.1.0) reduced the total memory consumption and runtime for pre-processing the noise information, but the consumption is still very large.

Currently, the 10 qubit noise model utilizes:
approx. 250GB of memory
requires 18 mins of runtime.

Although the reduction is good, the total resource utilization is still large. A solution to this issue is currently being worked and the aim will be to resolve this in the next update to the software, for a noise filtered scenario.

[Sats2]

The issue has been identified with the high memory consumption. There two main contributing factors:

1. During parallelization, the noise models are duplicated for each core. So more cores used, the higher the usage due to duplication.
2. The noise operators are stored as fully dense matrices which require higher memory for storage as the data type for element in the matrices are of type complex128.

A fix for both is being actively worked on.

[Sats2]

Implementation changes for memory isssue:

1. Use CSR matrix representation for matrix storage --> from memory profiling tests, seems to provide lower memory consumption compared to CSC format.
2. For the duplication error, maybe use pointers (or python's version of pointers) to keep single copy and provide them to the actors (parallel instances).

[Sats2]

Implementation with sparse matrix representation negates the large storage bottleneck where 84.2GiB was utilized to approximately 78MiB on single core usage.

Memory profiling logs below:

With dense matrix storage, at 10 qubits:

Filename: ./profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    11    230.2 MiB    230.2 MiB           1   @profile(stream=memory_log)
    12                                         def get_noise_model():
    13    310.2 MiB     80.0 MiB           1       model = pickle.load(open("../noise_models/Noise_Model_Heron_QPU.pkl", "rb"))
    14    310.2 MiB      0.0 MiB           1       return model


Filename: ./profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    16    310.2 MiB    310.2 MiB           1   @profile(stream=memory_log)
    17                                         def create_circuit_object(num_qubits,
    18                                                                   noise_model,
    19                                                                   num_cores,
    20                                                                   num_trajectories,
    21                                                                   threshold,
    22                                                                   jsonize,
    23                                                                   verbose):
    24  84534.6 MiB  84224.4 MiB           2       nqc = QC(num_qubits=num_qubits, 
    25    310.2 MiB      0.0 MiB           1            noise_model=noise_model, 
    26    310.2 MiB      0.0 MiB           1            num_cores=num_cores,
    27    310.2 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    28    310.2 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    29    310.2 MiB      0.0 MiB           1            threshold=threshold, 
    30    310.2 MiB      0.0 MiB           1            jsonize=jsonize,
    31    310.2 MiB      0.0 MiB           1            verbose=verbose)
    32  84534.6 MiB      0.0 MiB           1       return nqc

With sparse matrix representation:

Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    10    229.9 MiB    229.9 MiB           1   @profile(stream=memory_log)
    11                                         def get_noise_model():
    12    310.1 MiB     80.2 MiB           1       model = pickle.load(open("../noise_models/Noise_Model_Heron_QPU.pkl", "rb"))
    13    310.1 MiB      0.0 MiB           1       return model



Profiling for 2 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    310.1 MiB    310.1 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    343.1 MiB     33.0 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    310.1 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    310.1 MiB      0.0 MiB           1            num_cores=num_cores,
    26    310.1 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    310.1 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    310.1 MiB      0.0 MiB           1            threshold=threshold, 
    29    310.1 MiB      0.0 MiB           1            jsonize=jsonize,
    30    310.1 MiB      0.0 MiB           1            verbose=verbose)
    31    343.1 MiB      0.0 MiB           1       return nqc


Completed profiling for 2 qubits.

Profiling for 3 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    343.1 MiB    343.1 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    347.6 MiB      4.5 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    343.1 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    343.1 MiB      0.0 MiB           1            num_cores=num_cores,
    26    343.1 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    343.1 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    343.1 MiB      0.0 MiB           1            threshold=threshold, 
    29    343.1 MiB      0.0 MiB           1            jsonize=jsonize,
    30    343.1 MiB      0.0 MiB           1            verbose=verbose)
    31    347.6 MiB      0.0 MiB           1       return nqc


Completed profiling for 3 qubits.

Profiling for 4 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    347.6 MiB    347.6 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    350.6 MiB      3.0 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    347.6 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    347.6 MiB      0.0 MiB           1            num_cores=num_cores,
    26    347.6 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    347.6 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    347.6 MiB      0.0 MiB           1            threshold=threshold, 
    29    347.6 MiB      0.0 MiB           1            jsonize=jsonize,
    30    347.6 MiB      0.0 MiB           1            verbose=verbose)
    31    350.6 MiB      0.0 MiB           1       return nqc


Completed profiling for 4 qubits.

Profiling for 5 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    350.6 MiB    350.6 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    352.8 MiB      2.2 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    350.6 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    350.6 MiB      0.0 MiB           1            num_cores=num_cores,
    26    350.6 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    350.6 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    350.6 MiB      0.0 MiB           1            threshold=threshold, 
    29    350.6 MiB      0.0 MiB           1            jsonize=jsonize,
    30    350.6 MiB      0.0 MiB           1            verbose=verbose)
    31    352.8 MiB      0.0 MiB           1       return nqc


Completed profiling for 5 qubits.

Profiling for 6 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    352.0 MiB    352.0 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    360.2 MiB      8.2 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    352.0 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    352.0 MiB      0.0 MiB           1            num_cores=num_cores,
    26    352.0 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    352.0 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    352.0 MiB      0.0 MiB           1            threshold=threshold, 
    29    352.0 MiB      0.0 MiB           1            jsonize=jsonize,
    30    352.0 MiB      0.0 MiB           1            verbose=verbose)
    31    360.2 MiB      0.0 MiB           1       return nqc


Completed profiling for 6 qubits.

Profiling for 7 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    358.1 MiB    358.1 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    368.6 MiB     10.5 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    358.1 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    358.1 MiB      0.0 MiB           1            num_cores=num_cores,
    26    358.1 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    358.1 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    358.1 MiB      0.0 MiB           1            threshold=threshold, 
    29    358.1 MiB      0.0 MiB           1            jsonize=jsonize,
    30    358.1 MiB      0.0 MiB           1            verbose=verbose)
    31    368.6 MiB      0.0 MiB           1       return nqc


Completed profiling for 7 qubits.

Profiling for 8 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    364.5 MiB    364.5 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    381.0 MiB     16.5 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    364.5 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    364.5 MiB      0.0 MiB           1            num_cores=num_cores,
    26    364.5 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    364.5 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    364.5 MiB      0.0 MiB           1            threshold=threshold, 
    29    364.5 MiB      0.0 MiB           1            jsonize=jsonize,
    30    364.5 MiB      0.0 MiB           1            verbose=verbose)
    31    381.0 MiB      0.0 MiB           1       return nqc


Completed profiling for 8 qubits.

Profiling for 9 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    372.3 MiB    372.3 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    420.2 MiB     47.8 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    372.3 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    372.3 MiB      0.0 MiB           1            num_cores=num_cores,
    26    372.3 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    372.3 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    372.3 MiB      0.0 MiB           1            threshold=threshold, 
    29    372.3 MiB      0.0 MiB           1            jsonize=jsonize,
    30    372.3 MiB      0.0 MiB           1            verbose=verbose)
    31    420.2 MiB      0.0 MiB           1       return nqc


Completed profiling for 9 qubits.

Profiling for 10 qubits:
Filename: /mnt/userdrive/Users/home/adam-ukj7r05xnu2fywx/NoisyCircuits/examples/profiling.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    15    400.1 MiB    400.1 MiB           1   @profile(stream=memory_log)
    16                                         def create_circuit_object(num_qubits,
    17                                                                   noise_model,
    18                                                                   num_cores,
    19                                                                   num_trajectories,
    20                                                                   threshold,
    21                                                                   jsonize,
    22                                                                   verbose):
    23    478.8 MiB     78.7 MiB           2       nqc = QC(num_qubits=num_qubits, 
    24    400.1 MiB      0.0 MiB           1            noise_model=noise_model, 
    25    400.1 MiB      0.0 MiB           1            num_cores=num_cores,
    26    400.1 MiB      0.0 MiB           1            backend_qpu_type="heron", 
    27    400.1 MiB      0.0 MiB           1            num_trajectories=num_trajectories, 
    28    400.1 MiB      0.0 MiB           1            threshold=threshold, 
    29    400.1 MiB      0.0 MiB           1            jsonize=jsonize,
    30    400.1 MiB      0.0 MiB           1            verbose=verbose)
    31    478.8 MiB      0.0 MiB           1       return nqc


Completed profiling for 10 qubits.

Adding sparse matrix conversion found to have no impact on runtime.

[Sats2]

Resolved issue where the noise instructions are duplicated under a parallel regime. Memory profiling suggests successful implementation.

See memory_profiling logs for more information.

Code base where noise instructions stored as dense matrices were replaced with sparse matrix representation - Memory_Profiling_with_sparse.txt
Updated code base where all objects are stored as data types that can be referenced for serialization and de-serialization -
Memory_Profiling_with_new_sparse_version.txt

Now, moving to the runtime issue.