-2

EDIT: I tried to enumerate the valid GPUs by using their UUIDs instead of their IDs, this causes the thing to work.

It seems that it was still seeing the GT 610 even though I thought it should not. And that was the reason it was not working.


I am having difficulty with cuda MPS on one of my machines.

The machine has 4 Tesla K80s, as well as a (EDIT:) non-MPS-supported GT610

Here is the nvidia-smi:

riveale@coiworkstation1:~/code/psweep2/src$ nvidia-smi
Tue Mar 15 23:51:59 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.63     Driver Version: 352.63         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 610      Off  | 0000:01:00.0     N/A |                  N/A |
| 40%   29C    P8    N/A /  N/A |      3MiB /  1021MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:04:00.0     Off |                    0 |
| N/A   29C    P8    26W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:05:00.0     Off |                    0 |
| N/A   24C    P8    30W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | 0000:08:00.0     Off |                    0 |
| N/A   34C    P8    27W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           Off  | 0000:09:00.0     Off |                    0 |
| N/A   28C    P8    29W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           Off  | 0000:84:00.0     Off |                    0 |
| N/A   31C    P8    28W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           Off  | 0000:85:00.0     Off |                    0 |
| N/A   26C    P8    30W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           Off  | 0000:88:00.0     Off |                    0 |
| N/A   31C    P8    26W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   8  Tesla K80           Off  | 0000:89:00.0     Off |                    0 |
| N/A   25C    P8    31W / 149W |     55MiB / 11519MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+

As you can see, I have already set the processors to exclusive process.

I can run a sanity check using only the 1st GPU, starting the MPS server etc. as follows:

export CUDA_VISIBLE_DEVICES="0"
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
nvidia-cuda-mps-control -d

Then I run my script:

NRANKS=4
mpirun -n $NRANKS gputest.exe

This successfully runs, and I see in /tmp/nvidia-log/server.log:

riveale@coiworkstation1:~/code/psweep2/src$ cat /tmp/nvidia-log/server.log 
[2016-03-15 23:57:07.883 Other  6957] Start
[2016-03-15 23:57:08.513 Other  6957] New client 6956 connected
[2016-03-15 23:57:08.513 Other  6957] New client 6954 connected
[2016-03-15 23:57:08.514 Other  6957] New client 6955 connected

However, when I try to use more than 1 GPU on the system, I encounter issues. Specifically, the following (exactly the same, but now I have 2 visible CUDA devices):

export CUDA_VISIBLE_DEVICES="0,1"
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log
nvidia-cuda-mps-control -d

(ps ax | grep mps shows the daemon starting just fine, no differences from the working example above). Followed by:

NRANKS=7
mpirun -n $NRANKS gputest.exe

I see:

riveale@coiworkstation1:~/code/psweep2/src$ cat /tmp/nvidia-log/server.log 
[2016-03-15 23:59:55.718 Other  7102] Start
[2016-03-15 23:59:56.301 Other  7102] MPS server failed to start
[2016-03-15 23:59:56.301 Other  7102] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:56.727 Other  7105] Start
[2016-03-15 23:59:57.302 Other  7105] MPS server failed to start
[2016-03-15 23:59:57.302 Other  7105] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:57.718 Other  7107] Start
[2016-03-15 23:59:58.291 Other  7107] MPS server failed to start
[2016-03-15 23:59:58.291 Other  7107] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:58.709 Other  7109] Start
[2016-03-15 23:59:59.236 Other  7109] MPS server failed to start
[2016-03-15 23:59:59.236 Other  7109] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-15 23:59:59.644 Other  7111] Start
[2016-03-16 00:00:00.215 Other  7111] MPS server failed to start
[2016-03-16 00:00:00.215 Other  7111] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.
[2016-03-16 00:00:00.651 Other  7113] Start
[2016-03-16 00:00:01.221 Other  7113] MPS server failed to start
[2016-03-16 00:00:01.221 Other  7113] MPS is only supported on 64-bit Linux platforms, with an SM 3.5 or higher GPU.

Weird.

Thank you for any help/ideas in advance.

Another weird-ness, is that the exact same thing works on my other workstation, which has the same setup except it has a Quadro K620 instead of the GT610. The K620 is a CUDA device, so I have a feeling that that is the issue. Right now I'm remote, so I can't switch out the cards to see if that changes the issue.

5
  • GT610 is a CUDA supported device, and it may be messing up your approach (although I wouldn't have expected CUDA to enumerate it before any K80 GPU), because it has a compute capability less than 3.5. As a diagnostic, after doing export CUDA_VISIBLE_DEVICES="0,1", run the deviceQuery sample code. If the GT610 shows up in the output, that is the problem. The Quadro K620 OTOH, despite it's naming, is a Maxwell device and has a compute capability of 5.0, so it will not trip that error in the same scenario. Are you using CUDA 7.5 in both cases? Mar 15, 2016 at 15:28
  • Hi Robert, Yes, the two are otherwise exact same machine with exact same software installed, including cuda 7.5. I didn't run deviceQuery, but I came to the same conclusion you did based on the MPS documentation which mentions the little Jewel: "The MPS server will fail to start if incompatible devices are visible after the application of CUDA_VISIBLE_DEVICES."
    – rveale
    Mar 15, 2016 at 15:59
  • Robert, as I mentioned in the edit, I got it to work by doing enumerating using the GPUs UUID instead of the normal IDs 0,1 thing. Which somehow got it to work.
    – rveale
    Mar 15, 2016 at 16:02
  • 2
    To mark a question as solved, please post your resolution as an answer and accept it.
    – Zulan
    Mar 15, 2016 at 16:13
  • 1
    Yea it won't let me do that for 2 days so I marked it in the interim.
    – rveale
    Mar 16, 2016 at 14:46

1 Answer 1

1

As marked in the edit, the solution was to use the UUIDs of the cc >3.5 GPUs and set CUDA_VISIBLE_DEVICES to that. It seems that for whatever reason, even though device 0 was correctly one of the K80s, it was for some reason listing the display device (the 610 etc.) as device #1, rather than the last device, as I expected.

E.g.:

riveale@coiworkstation0:~$ nvidia-smi -L
GPU 0: Quadro K620 (UUID: GPU-1685f2e0-0f3a-fef1-c94c-00bf21afeb24)
GPU 1: Tesla K80 (UUID: GPU-9e8b10fb-8005-24c7-b7aa-5795c39b4c15)
GPU 2: Tesla K80 (UUID: GPU-3d917409-02ae-079b-3941-bacd9570b8c6)
GPU 3: Tesla K80 (UUID: GPU-8faf997f-67a1-b729-6205-1da501a39470)
GPU 4: Tesla K80 (UUID: GPU-99da7098-9e60-d67a-c5c8-de52e4b30c30)

riveale@coiworkstation0:~$ export CUDA_VISIBLE_DEVICES="GPU-9e8b10fb-8005-24c7-b7aa-5795c39b4c15,GPU-3d917409-02ae-079b-3941-bacd9570b8c6,GPU-8faf997f-67a1-b729-6205-1da501a39470,GPU-99da7098-9e60-d67a-c5c8-de52e4b30c30"

I had to do this before starting the nvidia-cuda-mps-control -d script above, on each node/machine.

It turned out that MPS is quite slow (MPS server took much CPU), so I decided against using it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.