How to Fix an NVML Driver and Library Version Mismatch

Restart Linux after an NVIDIA driver update to fix an NVML driver and library version mismatch caused by an older kernel module still running in memory. If restarting does not clear the error, finish system updates and install matching NVIDIA driver packages before repairing kernel modules or the boot image.

The Ubuntu commands below apply to Canonical’s driver packages, with separate steps for WSL and supported Windows installations. Match each repair to your installation method because Ubuntu packages, NVIDIA repository packages, and standalone installers require different handling.

Restart Linux and check the result

Save your work before restarting. An update can install newer NVIDIA libraries while leaving the old kernel module loaded, and rebooting loads the replacement module.

To check for that mismatch, open a terminal and run the following command. Look for NVRM: API mismatch in the output.

sudo dmesg

Once your work is saved, restart with the command below. Wait for the system to restart and sign in again.

sudo reboot

Run the following check after signing in. If it completes without the mismatch error, stop here; otherwise, continue with the repair that matches your system and package source.

nvidia-smi

Finish Ubuntu updates

On Ubuntu Desktop using Ubuntu packages, open Software Updater and install the available updates. Restart when the updates finish, then check the driver again with nvidia-smi.

You can also complete the updates from a terminal. Run these commands in order, allowing each one to finish before starting the next.

sudo apt update
sudo apt upgrade
sudo reboot

Check the result after signing back in with the command below. If the mismatch remains, use Ubuntu’s driver manager or automatic installer to install a consistent driver.

nvidia-smi

Install a matching Ubuntu driver

Open Additional Drivers on Ubuntu Desktop. Under NVIDIA Corporation, select the newest appropriate desktop driver entry marked proprietary, tested; the full entry name depends on your hardware and available packages.

Click Apply changes, enter your password, and let installation finish. Restart, then check the driver again with the command below.

nvidia-smi

If you cannot open Additional Drivers because its package is missing, install it using the following command. Open Additional Drivers afterward and follow the same selection process.

sudo apt install software-properties-gtk

Alternatively, let Ubuntu select and install a matching driver from its packages. Run the following commands in order, then run nvidia-smi after the restart.

sudo ubuntu-drivers list
sudo ubuntu-drivers install
sudo reboot

The available driver branch depends on your hardware. For a specific branch shown in the list, replace BRANCH below with that listed value, complete installation, and restart.

sudo ubuntu-drivers install nvidia:BRANCH

Remove conflicting driver installations

Move to removal only when you have identified an unwanted driver branch or a conflicting standalone installation. On Ubuntu using branch-numbered packages, start by listing manually installed NVIDIA packages.

apt-mark showmanual | grep nvidia

The next command removes packages matching the selected NVIDIA branch and purges their package configuration. Replace BRANCH inside the quoted pattern with the unwanted branch number, and review the proposed removal list before accepting it.

sudo apt --purge remove '*nvidia*BRANCH*'

The following command removes dependencies that APT considers no longer needed. Review its proposed removals before continuing.

sudo apt autoremove

Install your selected driver again through Additional Drivers or sudo ubuntu-drivers install, then restart and run nvidia-smi. This branch-removal pattern applies to Ubuntu’s branch-numbered packages, so do not apply it to NVIDIA’s newer unversioned repository packages.

The next command removes a driver installed through NVIDIA’s standalone installer. Run it only when that standalone installation is the conflict you need to remove; it does not replace the package manager’s removal procedure for distribution-managed drivers.

sudo nvidia-installer --uninstall

After removal, install one complete driver through your chosen supported installation method and reboot. Run nvidia-smi again before moving on to kernel or boot-image repairs.

Match the driver module to your Ubuntu kernel

If the mismatch persists, your running kernel may need matching NVIDIA modules. Canonical recommends precompiled modules for ordinary Ubuntu installations when a matching package is available.

Replace BRANCH in both commands with your selected Ubuntu driver branch. Keep the branch and variant consistent; server packages require the corresponding -server names. Restart after installation.

sudo apt install linux-modules-nvidia-BRANCH-$(uname -r)
sudo apt install nvidia-driver-BRANCH
sudo reboot

For a custom Ubuntu kernel without supported precompiled NVIDIA modules, use DKMS to build matching modules instead. Install the running kernel’s headers, then start the driver installation with DKMS included.

sudo apt install linux-headers-$(uname -r)
sudo ubuntu-drivers install --include-dkms

Alternatively, after installing the headers, install matching nvidia-dkms-BRANCH and nvidia-driver-BRANCH packages. Complete any required module-signing and key-enrollment process, then restart. After signing back in, check whether the mismatch has cleared.

nvidia-smi

Rebuild a boot image holding an old module

A Linux boot image can retain an older NVIDIA module even after the installed module and libraries match. Use this repair when you have established that the installed components are consistent and the boot image retains the stale module.

Rebuilding replaces or updates the existing generated boot image. Use only the command for your system’s actual initramfs tool. For a system using initramfs-tools, run the following command.

sudo update-initramfs -u

On a system using dracut, the following command replaces the existing generated boot image. Use it instead of the initramfs-tools command.

sudo dracut --force

For a system using mkinitcpio, the following command rebuilds the generated boot images. Use it instead of either alternative above.

sudo mkinitcpio -P

Restart after the applicable rebuild finishes. This lets the system load the rebuilt boot image.

sudo reboot

Sign in again and run the NVIDIA check below. If the mismatch remains, rebuilding the boot image has not resolved it; revisit the matching package and kernel-module repair for your installation.

nvidia-smi

Repair the Windows driver when using WSL

WSL 2 uses the NVIDIA display driver installed on its Windows host. Install a compatible Windows driver on the host instead of installing a Linux display driver inside WSL.

Open the NVIDIA driver download page and use Manual Driver Search to locate the driver for your GPU and Windows version. If you need help identifying the driver to update, follow the steps to find and update your graphics driver.

Install the compatible Windows driver, then open a Windows terminal and update WSL with the following command. The Ubuntu driver-package commands above apply to their identified Linux installations, not to installing a display driver inside WSL.

wsl.exe --update

If you also need CUDA development tools, choose the WSL-Ubuntu toolkit route that does not include a Linux GPU driver. Installing the toolkit alone is not inherently a fix for a driver and library mismatch.

Update or reinstall the Windows host driver

On a supported Windows PC, open NVIDIA App and select Drivers. For an available update, click Download, then Install, finish installation, and restart.

If you need the app, open the NVIDIA App download page, select Download Now, and install it. NVIDIA no longer supports GeForce Experience, so use NVIDIA App or the standalone driver download when older instructions point you there.

Reinstalling a previous driver replaces your current driver installation with the selected version. If the problem started after a Windows driver update and NVIDIA App offers a previous installation beneath the newest driver, open that driver’s three-dot menu and select Reinstall. Finish installation and restart.

If the app does not offer that previous driver, use a compatible standalone driver from NVIDIA’s download page. These Windows procedures can repair a failed or incomplete host driver installation, but they do not establish that every NVML mismatch has that cause.

Frequently Asked Questions

Why do NVIDIA package names differ between instructions?

NVIDIA’s own repository uses unversioned Ubuntu package names beginning with branch 590, while Canonical maintains separate driver packages. Match commands to your package source and release instead of mixing naming schemes.

What should you check if the host works but a GPU container fails?

If you use NVIDIA Container Toolkit CDI and nvidia-smi works on the host, refresh stale device specifications with sudo systemctl restart nvidia-cdi-refresh.service, then check them with nvidia-ctk cdi list. If regeneration fails, inspect sudo journalctl -u nvidia-cdi-refresh.service. You can regenerate the specifications manually with sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml, which replaces the generated specification at that path. This addresses stale container specifications rather than mismatched host packages.

What should you do if a package installation was interrupted?

Finish the interrupted transaction before reinstalling the driver. When dpkg was interrupted, APT tells you to run sudo dpkg --configure -a, which configures packages that were unpacked but not yet configured. System76’s Pop!_OS 22.04 LTS recovery procedure runs that command followed by sudo apt install -f from a chroot into the installed system, then completes package updates. Afterward, install one consistent driver and reboot.

Can version pinning prevent the problem from returning?

If you use NVIDIA’s repository, version pinning can help keep your driver package set aligned. It does not replace an older module already loaded in memory, so you still need to restart after changing installed components.

Should you install the newest open kernel module package on any NVIDIA GPU?

Choose a release that supports your GPU. NVIDIA repository procedures using nvidia-open apply only to hardware supported by the selected open-module release, so the newest package is not a universal choice for older GPUs.