51

I was running a deep learning program on my Linux server and I suddenly got this error.

UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at /opt/conda/conda-bld/pytorch_1603729096996/work/c10/cuda/CUDAFunctions.cpp:100.)

Earlier when I just created this conda environment, torch.cuda.is_available() returned true and I could use CUDA & GPU. But all of a sudden I could not use CUDA and torch.cuda.is_available()returned false. What should I do?

ps. I use GeForce RTX 3080 and cuda 11.0 + pytorch 1.7.0. It worked before but now it doesn't.

7 Answers 7

50

I just tried rebooting. Problem solved. Turned out that it was caused by NVIDIA NVML Driver/library version mismatch.

4
  • 6
    I have the same issue and it disappears consistently when I reboot. However, I would like not to have to reboot. Did you find a solution to this problem?
    – desmond13
    Commented Nov 2, 2021 at 10:35
  • maybe you can try some of the solutions in here stackoverflow.com/questions/43022843/…
    – maque J
    Commented Nov 3, 2021 at 13:58
  • for me it is not a problem of version mismatch, unfortunately.
    – desmond13
    Commented Nov 4, 2021 at 14:12
  • Apt may not automatically change the kernel modules. You may be able to just modprobe nvidia to update the drivers in use.
    – mcint
    Commented Apr 2, 2022 at 0:03
12

Try to run nvidia-smi in a different terminal and if you get an error like: NVML: Driver/library version mismatch then basically you have to follow these steps, so you won't have to reboot again:

  1. In a terminal run: lsmod | grep nvidia.
  2. Then unload the module dependent on nvidia driver:
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia_uvm
  1. Finally, unload the nvidia module: sudo rmmod nvidia.
  2. Now when you try lsmod | grep nvidia, you should get nothing in the terminal output.
  3. Now run nvidia-smi to check if you get the desired output and you are good to go!
5
  • can you explain why this recipe resolves the issue?
    – meyerson
    Commented Jun 21, 2022 at 15:23
  • 1
    NVML is an API directly linked to various parameters of your GPU hardware. And your nvidia driver has been built on your hardware. I suspect that when we direct install a pre-build version of any program to run like pytorch or cudatoolkit, it happens to not properly work for the build version install on the GPU. Cloning the version of pytorch and building it from scratch might be a solution but we don't take that much of hassle unless required. You know our code can work much faster if we build the pytorch in our local machine! Commented Jun 24, 2022 at 10:23
  • 2
    I get this error when I run sudo rmmod nvidia_drm: rmmod: ERROR: Module nvidia_drm is in use
    – Philipp
    Commented Dec 4, 2022 at 22:08
  • 1
    The answer to this question stackoverflow.com/questions/43022843/… provides a solution to my problem
    – Philipp
    Commented Dec 4, 2022 at 22:16
  • 2
    What if nvidia-smi functions well? Commented Aug 4, 2023 at 17:16
8

First check the nvidia-fabricmanager service status:

systemctl status nvidia-fabricmanager

If you see that the nvidia-fabricmanager service is in active (running) state, it is running properly, otherwise restart:

systemctl start nvidia-fabricmanager

This works for me!

2
  • thanks! this solved initial set up of a100s ec2
    – aerin
    Commented May 13 at 13:09
  • Failed to connect to bus: Host is down
    – MarStarck
    Commented Jul 2 at 6:38
3

For people who are having this issue after updaing your driver. You could try

sudo apt-get install nvidia-fabricmanager-535

to update library version. Replace the 535 with your driver version.

3

Let me reboot k8s node...

In my case problem was little complex, because it was not PC, but server with k8s and nvidia-container-toolkit. Toolkit is managing some of the nvidia/cuda libraries inside container. The key to check it was running command ls -al /usr/lib/x86_64-linux-gnu/ | grep libcuda in working and not working containers. I think that output of command and linked cuda library version should match version of host.:

Working

lrwxrwxrwx  1 root root       12 Mar  8 15:15 libcuda.so -> libcuda.so.1
lrwxrwxrwx  1 root root       21 Mar  8 15:15 libcuda.so.1 -> libcuda.so.525.147.05
-rw-r--r--  1 root root 29867944 Oct 25 20:37 libcuda.so.525.147.05
lrwxrwxrwx  1 root root       29 Mar  8 15:15 libcudadebugger.so.1 -> libcudadebugger.so.525.147.05
-rw-r--r--  1 root root 10490248 Oct 25 20:18 libcudadebugger.so.525.147.05

Not working

lrwxrwxrwx 1 root root        12 Mar  8 15:17 libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root        20 Mar  8 15:17 libcuda.so.1 -> libcuda.so.530.30.02
-rw-r--r-- 1 root root  29867944 Oct 25 20:37 libcuda.so.525.147.05
-rw-r--r-- 1 root root  29900840 Feb 22  2023 libcuda.so.530.30.02
lrwxrwxrwx 1 root root        28 Mar  8 15:17 libcudadebugger.so.1 -> libcudadebugger.so.530.30.02
-rw-r--r-- 1 root root  10490248 Oct 25 20:18 libcudadebugger.so.525.147.05
-rw-r--r-- 1 root root  10488936 Feb 16  2023 libcudadebugger.so.530.30.02

Host:

/usr/lib/x86_64-linux-gnu/libcuda.so.525.147.05

To fix the problem, one can simply change the libcuda.so.1 link target from libcuda.so.530.30.02 to libcuda.so.525.147.05 in my situation. In your situations you may have different driver versions.

1
  • This is correct, I have no clue why this is the correct answer.
    – Ahmed
    Commented Apr 13 at 1:03
0

This is my experience:

  • I had a PyTorch 1.12, an Nvidia GeForce RTX 2080, cuda/11.3.1, and cudnn/8.2.4.15-11.4 on my system, and I got CUDA initialization error.

  • The error had been solved by only changing the cudnn version, i.e., I used cudnn/8.2.0.53-11.3 and the error was gone.

0

It can append. Try reinstalling a nvidia driver. Then reboot you computer (it's important), and check if nvidia-smi works.

Not the answer you're looking for? Browse other questions tagged or ask your own question.