Linux-Ubuntu22.04, [nvidia-smi] Failed to initialize NVML: Driver/library version mismatch solution

1. System: Ubuntu 22.04.3 LTS (64-bit)

2. Question: It was still working normally yesterday. You can check the graphics card version. Today it prompts that the version does not match. I suspect it was because I installed CUDA yesterday? But cuda should not affect the graphics card driver.

So, I checked the driver version dpkg -l | grep nvidia

(base) fzx@fzx-System-Product-Name:~$ dpkg -l | grep nvidia
ii libnvidia-cfg1-530:amd64 530.30.02-0ubuntu1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-530 530.30.02-0ubuntu1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-530:amd64 530.30.02-0ubuntu1 amd64 NVIDIA libcompute package
ii libnvidia-compute-530:i386 530.30.02-0ubuntu1 i386 NVIDIA libcompute package
rc libnvidia-compute-535:amd64 535.113.01-0ubuntu0.22.04.3 amd64 NVIDIA libcompute package
ii libnvidia-decode-530:amd64 530.30.02-0ubuntu1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-decode-530:i386 530.30.02-0ubuntu1 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-530:amd64 530.30.02-0ubuntu1 amd64 NVENC Video Encoding runtime library
ii libnvidia-encode-530:i386 530.30.02-0ubuntu1 i386 NVENC Video Encoding runtime library
ii libnvidia-extra-530:amd64 530.30.02-0ubuntu1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-530:amd64 530.30.02-0ubuntu1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-fbc1-530:i386 530.30.02-0ubuntu1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-530:amd64 530.30.02-0ubuntu1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-gl-530:i386 530.30.02-0ubuntu1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
rc linux-modules-nvidia-535-6.2.0-26-generic 6.2.0-26.26~22.04.1 + 2 amd64 Linux kernel nvidia modules for version 6.2.0-26
rc linux-modules-nvidia-535-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Linux kernel nvidia modules for version 6.2.0-35
ii linux-objects-nvidia-535-6.2.0-26-generic 6.2.0-26.26~22.04.1 + 2 amd64 Linux kernel nvidia modules for version 6.2.0-26 (objects)
ii linux-objects-nvidia-535-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Linux kernel nvidia modules for version 6.2.0-35 (objects)
ii linux-signatures-nvidia-6.2.0-26-generic 6.2.0-26.26~22.04.1 + 2 amd64 Linux kernel signatures for nvidia modules for version 6.2.0-26-generic
ii linux-signatures-nvidia-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Linux kernel signatures for nvidia modules for version 6.2.0-35-generic
ii nvidia-compute-utils-530 530.30.02-0ubuntu1 amd64 NVIDIA compute utilities
rc nvidia-compute-utils-535 535.113.01-0ubuntu0.22.04.3 amd64 NVIDIA compute utilities
ii nvidia-dkms-530 530.30.02-0ubuntu1 amd64 NVIDIA DKMS package
ii nvidia-driver-530 530.30.02-0ubuntu1 amd64 NVIDIA driver metapackage
ii nvidia-firmware-535-535.113.01 535.113.01-0ubuntu0.22.04.3 amd64 Firmware files used by the kernel module
ii nvidia-firmware-535-535.86.05 535.86.05-0ubuntu0.22.04.1 amd64 Firmware files used by the kernel module
ii nvidia-kernel-common-530 530.30.02-0ubuntu1 amd64 Shared files used with the kernel module
rc nvidia-kernel-common-535 535.113.01-0ubuntu0.22.04.3 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-530 530.30.02-0ubuntu1 amd64 NVIDIA kernel source package
ii nvidia-modprobe 530.30.02-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device files
ii nvidia-prime 0.8.17.1 all Tools to enable NVIDIA's Prime
ii nvidia-settings 530.30.02-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-530 530.30.02-0ubuntu1 amd64 NVIDIA driver support binaries
ii screen-resolution-extra 0.18.2 all Extension for the nvidia-settings control panel
ii xserver-xorg-video-nvidia-530 530.30.02-0ubuntu1 amd64 NVIDIA binary Xorg driver

It can be seen that the driver version is 530.30.02

Kernel version is 535.113.01

(base) fzx@fzx-System-Product-Name:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 535.113.01 Tue Sep 12 19:41:24 UTC 2023
GCC version: 

Check the system driver log, there is indeed an update

(base) fzx@fzx-System-Product-Name:~$ cat /var/log/dpkg.log | grep nvidia
2023-10-22 01:20:29 install nvidia-prime:all <none> 0.8.17.1
2023-10-22 01:20:29 status half-installed nvidia-prime:all 0.8.17.1
2023-10-22 01:20:29 status unpacked nvidia-prime:all 0.8.17.1
2023-10-22 01:20:29 configure nvidia-prime:all 0.8.17.1 <none>
2023-10-22 01:20:29 status unpacked nvidia-prime:all 0.8.17.1
2023-10-22 01:20:29 status half-configured nvidia-prime:all 0.8.17.1
2023-10-22 01:20:29 status installed nvidia-prime:all 0.8.17.1
2023-10-22 18:04:57 status installed nvidia-driver-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 remove nvidia-driver-535:amd64 535.113.01-0ubuntu0.22.04.3 <none>
2023-10-22 18:04:57 status half-configured nvidia-driver-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 status half-installed nvidia-driver-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 status config-files nvidia-driver-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 status not-installed nvidia-driver-535:amd64 <none>
2023-10-22 18:04:57 status installed xserver-xorg-video-nvidia-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 remove xserver-xorg-video-nvidia-535:amd64 535.113.01-0ubuntu0.22.04.3 <none>
2023-10-22 18:04:57 status half-configured xserver-xorg-video-nvidia-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 status half-installed xserver-xorg-video-nvidia-535:amd64 535.113.01-0ubuntu0.22.04.3
2023-10-22 18:04:57 status config-files xserver-xorg-video-nvidia-535:amd64 535.113.01-0ubuntu0.22.04.3

3. Reason: Version inconsistency caused by kernel driver update

4.Solution:

Uninstall the driver and reinstall it

(1) Uninstall the command, paste it all, execute it, and keep entering y until the execution is completed.

sudo /usr/bin/nvidia-uninstall
sudo apt-get --purge remove nvidia-*
sudo apt-get purge nvidia*
sudo apt-get purge libnvidia*

(2) Check the remaining drivers of the graphics card, sudo dpkg –list | grep nvidia-* or dpkg -l | grep nvidia, and reinstall according to the current version

(base) fzx@fzx-System-Product-Name:/$ sudo dpkg --list | grep nvidia-*
ii libnvidia-compute-530:i386 530.30.02-0ubuntu1 i386 NVIDIA libcompute package
ii libnvidia-decode-530:i386 530.30.02-0ubuntu1 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-530:i386 530.30.02-0ubuntu1 i386 NVENC Video Encoding runtime library
ii libnvidia-fbc1-530:i386 530.30.02-0ubuntu1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
rc linux-modules-nvidia-535-6.2.0-26-generic 6.2.0-26.26~22.04.1 + 2 amd64 Linux kernel nvidia modules for version 6.2.0-26
rc linux-modules-nvidia-535-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Linux kernel nvidia modules for version 6.2.0-35
ii linux-objects-nvidia-535-6.2.0-26-generic 6.2.0-26.26~22.04.1 + 2 amd64 Linux kernel nvidia modules for version 6.2.0-26 (objects)
ii linux-objects-nvidia-535-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Linux kernel nvidia modules for version 6.2.0-35 (objects)
ii linux-signatures-nvidia-6.2.0-26-generic 6.2.0-26.26~22.04.1 + 2 amd64 Linux kernel signatures for nvidia modules for version 6.2.0-26-generic
ii linux-signatures-nvidia-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Linux kernel signatures for nvidia modules for version 6.2.0-35-generic
ii screen-resolution-extra 0.18.2 all Extension for the nvidia-settings control panel

(3) Driver download link: Old version download address: https://www.nvidia.cn/Download/Find.aspx?lang=cn

Select the corresponding version to download

(4) Install new driver

bash NVIDIA-Linux-x86_64-530.30.02.run