Deadlock from re-entrant cuInit

[isawang92]

Hello, I am a student from National Taiwan University of Science and Technology. I would like to do some research based on your program, but after carefully reading the nvshare code and deploying it on Kubernetes, I encountered the following deadlock/loop:

The workload’s TensorFlow triggers cuInit for the first time, which also triggers initialize_client. The client_fn thread is started during initialize_client, but before initialize_client completes, client_fn calls real_cuInit.

Inside real_cuInit, dlsym("cuInit") is invoked, which is intercepted by your interposer and returns the wrapper. This re-enters the wrapper cuInit, which again calls pthread_once(&init_done, initialize_client). Since initialize_client has not finished, pthread_once blocks, and cuInit gets stuck.

Moreover, I think that even without the client_fn issue, based on the program logic and my actual runs, this could still evolve into a situation where real_cuInit and cuInit repeatedly call each other, causing a deadlock.

I would greatly appreciate your advice and help. Thank you very much.

[grgalex]

Hi, do you have a reproducer for this issue?

[isawang92]

Hi, do you have a reproducer for this issue?

Initially, after I deployed the nvshare components, I also created the workload nvshare-tf-small-pod-1.yaml. However, when I checked its logs, the output kept getting stuck at [NVSHARE][DEBUG]: Found NVML.

So I added log_debug statements to observe the flow and confirmed the following :

Btw,"(function name)" means I added log_debug in this func.

From the second and third debug lines, I can see that during bootstrap, real_cuInit was resolved to the native cuInit in libcuda.so, while the cuInit wrapper has a different address.

From the fifth debug line, I can see the workload triggered cuInit, and I again confirmed the addresses of real_cuInit and cuInit.

In the sixth debug line, because cuInit is triggered for the first time, it calls initialize_client, and then the seventh debug line shows that client_fn is created. I added debug logs before and after cu_err = real_cuInit(0); in client_fn, and only the “before” debug log was printed.

Then the eighth debug line made me suspect that real_cuInit also calls dlsym("cuInit") to look up the native CUDA API, which would route back into the interposed version.

By inspecting the call stack, I became more certain that real_cuInit calls dlsym("cuInit").

Finally, execution returned to cuInit. I added debug logs before and after true_or_exit(pthread_once(&init_done, initialize_client) == 0);, and only the “before” log was printed, which led me to this conclusion.

The above is my hypothesis and the conclusions I reached from my implementation. Since I don’t have a fully solid background, I would really appreciate your help.

[grgalex]

So you are saying that cuInit from libcuda.so calls dlsym("cuInit"), and this causes an infinite loop?

Can you prove this by running your application without nvshare and tracing the API calls made to libcuda.so?