Let me reboot k8s node...
In my case problem was little complex, because it was not PC, but server with k8s and nvidia-container-toolkit. Toolkit is managing some of the nvidia/cuda libraries inside container. The key to check it was running command ls -al /usr/lib/x86_64-linux-gnu/ | grep libcuda in working and not working containers. I think that output of command and linked cuda library version should match version of host.:
Working
lrwxrwxrwx 1 root root 12 Mar 8 15:15 libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root 21 Mar 8 15:15 libcuda.so.1 -> libcuda.so.525.147.05
-rw-r--r-- 1 root root 29867944 Oct 25 20:37 libcuda.so.525.147.05
lrwxrwxrwx 1 root root 29 Mar 8 15:15 libcudadebugger.so.1 -> libcudadebugger.so.525.147.05
-rw-r--r-- 1 root root 10490248 Oct 25 20:18 libcudadebugger.so.525.147.05
Not working
lrwxrwxrwx 1 root root 12 Mar 8 15:17 libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root 20 Mar 8 15:17 libcuda.so.1 -> libcuda.so.530.30.02
-rw-r--r-- 1 root root 29867944 Oct 25 20:37 libcuda.so.525.147.05
-rw-r--r-- 1 root root 29900840 Feb 22 2023 libcuda.so.530.30.02
lrwxrwxrwx 1 root root 28 Mar 8 15:17 libcudadebugger.so.1 -> libcudadebugger.so.530.30.02
-rw-r--r-- 1 root root 10490248 Oct 25 20:18 libcudadebugger.so.525.147.05
-rw-r--r-- 1 root root 10488936 Feb 16 2023 libcudadebugger.so.530.30.02
Host:
/usr/lib/x86_64-linux-gnu/libcuda.so.525.147.05
To fix the problem, one can simply change the libcuda.so.1 link target from libcuda.so.530.30.02 to libcuda.so.525.147.05 in my situation. In your situations you may have different driver versions.