TensorFlow Docker image uses cuDNN 9, but only cuDNN 8.9 is installed

[keeper]

Starts from tensorflow/tensorflow-2.15.0-gpu (206b54412f00), tensorflow tries to load libcudnn.so.9, but the docker image uses 8:
https://github.com/tensorflow/build/blob/master/tensorflow_runtime_dockerfiles/gpu.packages.txt#L15-L17

For more information, please see:
tensorflow/tensorflow#62412

[json-han]

I was surprised to find that this issue still hasn't been fixed.

I tried to use the latest TensorFlow by running the tensorflow/tensorflow:latest-gpu-jupyter image, which currently ships with TensorFlow 2.19.0.

After launching a container from this image and checking cuDNN with dpkg -l | grep cudnn, I found the following result:

ii  libcudnn8  8.9.6.50-1+cuda12.2  amd64  cuDNN runtime libraries

This means that the official image includes cuDNN 8.9, even though TensorFlow 2.19.0 requires cuDNN 9.3 or higher. (https://www.tensorflow.org/install/source#gpu)

To resolve this issue inside the container, I followed these steps:

1. Remove existing cuDNN

   apt-get remove --purge libcudnn*

2. Install wget

   apt install wget

3. Download and install the latest cuDNN from the official NVIDIA repository

   wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
   dpkg -i cuda-keyring_1.1-1_all.deb
   apt-get update
   apt-get -y install cudnn

Note: Since all commands inside the Docker container are run as root, sudo is not required.

After step 3, cuDNN 9.10 was successfully installed in my container.

I later found that some others resolved this issue using the following commands:

apt update \
    && apt install -y --no-install-recommends libcudnn9-cuda-12 \
    && rm -rf /var/lib/apt/lists/*

This workaround also installs the required cuDNN 9.x version.

[mariojmdavid]

discovered this the hard way, when trying to use the official tensorflow/tensorflow:2.20.0-gpu
from some 5 months ago, seems simply adding
cudnn9-cuda-12
to gpu.packages.txt, and removing libcudnn8=8.9.6.50-1...

so I'am wondering if anyone is using/running this official image