Path length error when doing distributed contractions

[NathanCQC]

I Have this error ValueError: The length of the contraction path (2590) must be one less than the number of operands (2593).

I think it is due to the internal simplification contracting in excess tensors.

I want to set this enum but it is not exposed in the NetworkOptions

ContractionOptimizerConfigAttribute.SIMPLIFICATION_DISABLE_DR

I am not really sure how to set it with the low level API.

Here is my code. I am path finding and slicing over multiple nodes with MPI,

    network = cq.Network(*tensor_network.cuquantum_interleaved)

    time0 = MPI.Wtime()
    path, info = network.contract_path(optimize={'samples': 20, 'slicing': {'min_slices': max(max_n_slices, size)}})

    # Select the best path from all ranks. Note that we still use the MPI communicator here for simplicity.
    opt_cost, sender = comm_mpi.allreduce(sendobj=(info.opt_cost, rank), op=MPI.MINLOC)
    time1 = MPI.Wtime()
    if rank == root:
        print(f"Process {sender} has the path with the lowest FLOP count {opt_cost}.")

    # Broadcast info from the sender to all other ranks.
    info = comm_mpi.bcast(info, sender)

    # print(f"Process {rank} is processing slice range: {info.slices}.")

    # Set path and slices.
    print(info.slices)
    path, info = network.contract_path(optimize={'path': info.path, 'slicing': info.slices})

[NathanCQC]

I figured this out

    network.optimizer_config_ptr = cutn.create_contraction_optimizer_config(network.handle)
    enum = cutn.ContractionOptimizerConfigAttribute.SIMPLIFICATION_DISABLE_DR
    value = 1
    name = 'SIMPLIFICATION_DISABLE_DR'

    network._set_opt_config_option(name,enum, value)

[NathanCQC]

Unfortunatly with the enum turned on I still get the error.

ValueError: The length of the contraction path (2590) must be one less than the number of operands (2595).

This only happens on multiple devices. It does not occur on a single gpu.

[NathanCQC]

Sorry, yes I have been using manual parallelism.

I felt that the auto parallelism was too implicit and it was difficult to check that it was running over multiple processes.

If there were some more examples of the auto parallelism I would be happy to use it.

This was a problem with the container solution I was using. I have reverted back to module loading on Perlmutter following this solution https://github.com/NVIDIA/cuQuantum/discussions and it is working as expected now. Sorry to have wasted your time.

[leofang]

No worries at all, @NathanCQC. Not sure which Perlmutter post that you were referring to. Is it how you fixed the path length error? I'd be surprised 😅

[leofang]

In particular, I'd still like to know if the problem that you described

Hence I tried to write my own MPI slicing code, but this required careful control of all the slicing options and optimiser options etc., particularly for different processes. Even the ones that were not exposed in the python API. I have been stuck on this for over a week now and it is holding back my development. I have raised this in this discussion #59. I am getting path length errors on different processes. Hence I was trying to freeze optimisation of internal tensors to see if that would help and this enum is not exposed in the python API.

has been addressed or not.

[NathanCQC]

Apologies. I meant this post #30.

I fixed it by generating the paths on a single node process and then broadcasting them. It was an error in how I was using MPI4py I think.

[leofang]

Wonderful, glad it worked out. Thank you Nathan. I see you have opened other threads. We'll make sure your questions are addressed.

[DmitryLyakh]

@NathanCQC I am glad your code works now, which is great, just let me note that the automatic distributed parallelization feature was specifically designed and implemented to relieve users from the burden of manual work distribution and accompanying mistakes. The only thing a user would need to do to make the serial code parallel (multi-GPU and multi-node) is to add these two lines, as illustrated in the corresponding example:

cutn_comm = comm.Dup()
cutn.distributed_reset_configuration(handle, MPI._addressof(cutn_comm), MPI._sizeof(cutn_comm))

The rest of the serial code stays the same, no need to tweak anything. It makes it extremely easy for users to transition from serial codes to parallel codes with no changes, except the above two lines which activate the distributed parallelization. However, for the automatic parallelization to work properly your Python environment should be properly configured with MPI, which should be the case if you install cuQuantum-Python from scratch. Maybe, you would consider giving it a try again in the future. By the way, the distributed parallelization makes parallel both the tensor network path finder and execution.

[NathanCQC]

Great I will try this hopefully it will reduce the sampling time aswell.

In the above code snippet is MPI.COMM_WORLD = comm?

Thanks a lot for your help!

[DmitryLyakh]

Yes, comm is a duplicate of MPI.COMM_WORLD (please check lines 343-345 in the example here: https://github.com/NVIDIA/cuQuantum/blob/main/samples/cutensornet/tensornet_example_mpi_auto.cu). Indeed, as long as you request multiple hyper-samples in the contraction path finder (contraction path optimizer), you should see the path finding time reduce as well with the number of MPI processes.

[NathanCQC]

I got the same error with the path length error with some simpler code that explicitly uses MPI and does not get a distributed error

def slice_contract(tensor_network: TensorNetwork, max_n_slices: int, exp_name:str):

    root = 0
    comm_mpi = MPI.COMM_WORLD
    rank, size = comm_mpi.Get_rank(), comm_mpi.Get_size()
    time0 = MPI.Wtime()

    device_id = rank % getDeviceCount()
    cp.cuda.Device(device_id).use()

    network = cq.Network(*tensor_network.cuquantum_interleaved)

    time0 = MPI.Wtime()
    path, info = network.contract_path(optimize={'samples': 20, 'slicing': {'min_slices': size}})
    time1 = MPI.Wtime()

    # Select the best path from all ranks. Note that we still use the MPI communicator here for simplicity.
    opt_cost, sender = comm_mpi.allreduce(sendobj=(info.opt_cost, rank), op=MPI.MINLOC)
    if rank == root:
        print(f"Process {sender} has the path with the lowest FLOP count {opt_cost}.")

    # Broadcast info from the sender to all other ranks.
    info = comm_mpi.bcast(info, sender)

    path, info = network.contract_path(optimize={'path': info.path, 'slicing': {'disable_slicing': 1}})

    # Contract the group of slices the process is responsible for.
    result = network.contract(slices=[rank])
    print(f"Process {rank} is done with contraction. Result: {result}.")

    result = comm_mpi.reduce(sendobj=result, op=MPI.SUM)

This is the error

ValueError: The length of the contraction path (2592) must be one less than the number of operands (2591).

This seems to run distributed fine but It seems the contract path is different on each node and causing this error.

[NathanCQC]

I am UK based. So I will have to do this tomorrow. Thanks for all your help!

[NathanCQC]

This ended up being something wrong with the config in shifter which I haven't identified yet. It runs fine when we install it directly.

[leofang]

@NathanCQC would you mind clarifying what you meant by "installing it directly"?

[NathanCQC]

Sorry I meant following this implementation #30

[leofang]

Got it, thx Nathan!

[leofang]

(Let me reopen this discussion. I noticed that if a discussion is marked as closed, at the entry time to GitHub Discussions it's not visible to readers, unless they specifically look for "closed" or "all" discussions.)