[Issue]: Does NVSHMEM support B300+CX8

[liuhe-spec]

How is this issue impacting you?

Application crash

Share Your Debug Logs

low_latency_nvshmem3.4.5.log

Steps to Reproduce the Issue

We can run test_intranode.py successfully, but we still can't run test_lowlatency.py.

NVSHMEM Version

v3.4.5+ 12090

Your platform details

No response

Error Message & Behavior

No response

[benhg]

Yes, NVSHMEM 3.4.5 should support B300 + CX8. The error you are seeing means ibv_create_ah returned an error. Are you sure your system is working correctly? Have you been able to run IB perftests on it?

[liuhe-spec]

Thanks! The system is working correctly . NCCL tests pass, and previously Deepep couldn’t run successfully. After rebuilding NVSHMEM with NVSHMEM_USE_NCCL=1, Deepep now works as expected.

[benhg]

These:

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2234: NULL value Unable to create ah.

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2966: non-zero status: 7 create DCT share err.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/transport/transport.cpp:420: non-zero status: 7 connect EPS failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:1047: non-zero status: 7 nvshmem setup connections failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1119: nvshmem initialization failed, exiting 

WARN: Failed to initialize all selected devices. Perf may be limited.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2966: non-zero status: 7 create DCT share err.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/transport/transport.cpp:420: non-zero status: 7 connect EPS failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:1047: non-zero status: 7 nvshmem setup connections failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1119: nvshmem initialization failed, exiting 

WARN: Failed to initialize all selected devices. Perf may be limited.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/common/transport_ib_common.h 138 NVSHMEM_IB_ADDR_FAMILY set by environment to AF_INET6
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/common/transport_ib_common.h 161 NVSHMEM_IB_ADDR_RANGE set by environment to ::/0
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2234: NULL value Unable to create ah.

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/common/transport_ib_common.h 138 NVSHMEM_IB_ADDR_FAMILY set by environment to AF_INET6
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/common/transport_ib_common.h 161 NVSHMEM_IB_ADDR_RANGE set by environment to ::/0
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2234: NULL value Unable to create ah.

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2966: non-zero status: 7 create DCT share err.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/transport/transport.cpp:420: non-zero status: 7 connect EPS failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:1047: non-zero status: 7 nvshmem setup connections failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1119: nvshmem initialization failed, exiting 

WARN: Failed to initialize all selected devices. Perf may be limited.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/modules/transport/ibgda/ibgda.cpp:2966: non-zero status: 7 create DCT share err.
/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/transport/transport.cpp:420: non-zero status: 7 connect EPS failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:1047: non-zero status: 7 nvshmem setup connections failed 

/dvs/p4/build/sw/rel/gpgpu/toolkit/r12.9/main_nvshmem/src/host/init/init.cu:nvshmemi_check_state_and_init:1119: nvshmem initialization failed, exiting 

WARN: Failed to initialize all selected devices. Perf may be limited.

are all initialization problems. I can't specifically say what's happening without knowing a lot more about your system, but I can suggest a few common fixes:

1. If you're using GDRCopy, make sure GDRCopy is installed and loaded on all nodes (we only support gdrcopy enabled everywhere or disabled everywhere)
2. Make sure you're mapping PEs to HCAs properly. For DGX/HGX like systems, this should be automatic, but if not you may need to set NVSHMEM_HCA_LIST or NVSHMEM_HCA_PE_MAPPING env variables.
3. If you're using IBGDA, make sure you follow all the IBGDA specific setup (e.g. regkeys, compatible hw, etc.)