EDIT: I tried to enumerate the valid GPUs by using their UUIDs instead of their IDs, this causes the thing to work.
It seems that it was still seeing the GT 610 even though I thought it should not. And that was the reason it was not working.
I am having difficulty with cuda MPS on one of my machines.
The machine has 4 Tesla K80s, as well as a (EDIT:) non-MPS-supported GT610
Here is the nvidia-smi:
riveale@coiworkstation1:~/code/psweep2/src$ nvidia-smi
Tue Mar 15 23:51:59 2016
+------------------------------------------------------+
| NVIDIA-SMI 352.63 Driver Version: 352.63 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GT 610 Off | 0000:01:00.0 N/A | N/A |
| 40% 29C P8 N/A / N/A | 3MiB / 1021MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 0000:04:00.0 Off | 0 |
| N/A 29C P8 26W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 2 Tesla K80 Off | 0000:05:00.0 Off | 0 |
| N/A 24C P8 30W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 3 Tesla K80 Off | 0000:08:00.0 Off | 0 |
| N/A 34C P8 27W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 4 Tesla K80 Off | 0000:09:00.0 Off | 0 |
| N/A 28C P8 29W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 5 Tesla K80 Off | 0000:84:00.0 Off | 0 |
| N/A 31C P8 28W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 6 Tesla K80 Off | 0000:85:00.0 Off | 0 |
| N/A 26C P8 30W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 7 Tesla K80 Off | 0000:88:00.0 Off | 0 |
| N/A 31C P8 26W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 8 Tesla K80 Off | 0000:89:00.0 Off | 0 |
| N/A 25C P8 31W / 149W | 55MiB / 11519MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
As you can see, I have already set the processors to exclusive process.
I can run a sanity check using only the 1st GPU, starting the MPS server etc. as follows:
export CUDA_VISIBLE_DEVICES="0"
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
nvidia-cuda-mps-control -d
Then I run my script:
NRANKS=4
mpirun -n $NRANKS gputest.exe
This successfully runs, and I see in /tmp/nvidia-log/server.log:
riveale@coiworkstation1:~/code/psweep2/src$ cat /tmp/nvidia-log/server.log
[2016-03-15 23:57:07.883 Other 6957] Start
[2016-03-15 23:57:08.513 Other 6957] New client 6956 connected
[2016-03-15 23:57:08.513 Other 6957] New client 6954 connected
[2016-03-15 23:57:08.514 Other 6957] New client 6955 connected
However, when I try to use more than 1 GPU on the system, I encounter issues. Specifically, the following (exactly the same, but now I have 2 visible CUDA devices):
export CUDA_VISIBLE_DEVICES="0,1"
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
nvidia-cuda-mps-control -d
(ps ax | grep mps shows the daemon starting just fine, no differences from the working example above). Followed by:
NRANKS=7
mpirun -n $NRANKS gputest.exe
I see:
riveale@coiworkstation1:~/code/psweep2/src$ cat /tmp/nvidia-log/server.log
[2016-03-15 23:59:55.718 Other 7102] Start
[2016-03-15 23:59:56.301 Other 7102] MPS server failed to start
[2016-03-15 23:59:56.301 Other 7102] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:56.727 Other 7105] Start
[2016-03-15 23:59:57.302 Other 7105] MPS server failed to start
[2016-03-15 23:59:57.302 Other 7105] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:57.718 Other 7107] Start
[2016-03-15 23:59:58.291 Other 7107] MPS server failed to start
[2016-03-15 23:59:58.291 Other 7107] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:58.709 Other 7109] Start
[2016-03-15 23:59:59.236 Other 7109] MPS server failed to start
[2016-03-15 23:59:59.236 Other 7109] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:59.644 Other 7111] Start
[2016-03-16 00:00:00.215 Other 7111] MPS server failed to start
[2016-03-16 00:00:00.215 Other 7111] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-16 00:00:00.651 Other 7113] Start
[2016-03-16 00:00:01.221 Other 7113] MPS server failed to start
[2016-03-16 00:00:01.221 Other 7113] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
Weird.
Thank you for any help/ideas in advance.
Another weird-ness, is that the exact same thing works on my other workstation, which has the same setup except it has a Quadro K620 instead of the GT610. The K620 is a CUDA device, so I have a feeling that that is the issue. Right now I'm remote, so I can't switch out the cards to see if that changes the issue.
export CUDA_VISIBLE_DEVICES="0,1"
, run thedeviceQuery
sample code. If the GT610 shows up in the output, that is the problem. The Quadro K620 OTOH, despite it's naming, is a Maxwell device and has a compute capability of 5.0, so it will not trip that error in the same scenario. Are you using CUDA 7.5 in both cases?