In a multi-GPU computer, how do I designate which GPU a CUDA job should run on?
As an example, when installing CUDA, I opted to install the NVIDIA_CUDA-<#.#>_Samples
then ran several instances of the nbody
simulation, but they all ran on one GPU 0; GPU 1 was completely idle (monitored using watch -n 1 nvidia-dmi
). Checking CUDA_VISIBLE_DEVICES
using
echo $CUDA_VISIBLE_DEVICES
I found this was not set. I tried setting it using
CUDA_VISIBLE_DEVICES=1
then running nbody
again but it also went to GPU 0.
I looked at the related question, how to choose designated GPU to run CUDA program?, but deviceQuery
command is not in the CUDA 8.0 bin directory. In addition to $CUDA_VISIBLE_DEVICES$
, I saw other posts refer to the environment variable $CUDA_DEVICES
but these were not set and I did not find information on how to use it.
While not directly related to my question, using nbody -device=1
I was able to get the application to run on GPU 1 but using nbody -numdevices=2
did not run on both GPU 0 and 1.
I am testing this on a system running using the bash shell, on CentOS 6.8, with CUDA 8.0, 2 GTX 1080 GPUs, and NVIDIA driver 367.44.
I know when writing using CUDA you can manage and control which CUDA resources to use but how would I manage this from the command line when running a compiled CUDA executable?
nbody
application has a command line option to select the GPU to run on - you might want to study that code. For the more general case,CUDA_VISIBLE_DEVICES
should work. If it does not, you're probably not using it correctly, and you should probably give a complete example of what you have tried. You should also indicate what OS you are working on and for linux, what shell (e.g. bash, csh, etc.).deviceQuery
isn't necessary to any of this, it's just an example app to demonstrate the behavior ofCUDA_VISIBLE_DEVICES
. The proper environment variable name doesn't have a$
in it.CUDA_VISIBLE_DEVICES=1
doesn't permanently set the environment variable (in fact, if that's all you put on that command line, it really does nothing useful.). This:export CUDA_VISIBLE_DEVICES=1
will permanently set it for the remainder of that session. You may want to study how environment variables work in bash, and how various commands affect them, and for how long.deviceQuery
is provided with CUDA 8, but you have to build it. If you read the CUDA 8 installation guide for linux, it will explain how to builddeviceQuery
watch -n 1 nvidia-smi
...