8

I am currently working on a gpu server which has 4 Tesla T10 gpu's. While I keep testing the kernels and have to frequently kill the processes using ctrl-C, I added a few lines to the end of a simple device query code. The code is given below :

#include <stdio.h>

 // Print device properties
 void printDevProp(cudaDeviceProp devProp)
{
    printf("Major revision number:         %d\n",  devProp.major);
    printf("Minor revision number:         %d\n",  devProp.minor);
    printf("Name:                          %s\n",  devProp.name);
    printf("Total global memory:           %u\n",  devProp.totalGlobalMem);
    printf("Total shared memory per block: %u\n",  devProp.sharedMemPerBlock);
    printf("Total registers per block:     %d\n",  devProp.regsPerBlock);
    printf("Warp size:                     %d\n",  devProp.warpSize);
    printf("Maximum memory pitch:          %u\n",  devProp.memPitch);
    printf("Maximum threads per block:     %d\n",  devProp.maxThreadsPerBlock);
    for (int i = 0; i < 3; ++i)
    printf("Maximum dimension %d of block:  %d\n", i, devProp.maxThreadsDim[i]);
    for (int i = 0; i < 3; ++i)
    printf("Maximum dimension %d of grid:   %d\n", i, devProp.maxGridSize[i]);
    printf("Clock rate:                    %d\n",  devProp.clockRate);
    printf("Total constant memory:         %u\n",  devProp.totalConstMem);
    printf("Texture alignment:             %u\n",  devProp.textureAlignment);
    printf("Concurrent copy and execution: %s\n",  (devProp.deviceOverlap ? "Yes" : "No"));
    printf("Number of multiprocessors:     %d\n",  devProp.multiProcessorCount);
    printf("Kernel execution timeout:      %s\n",  (devProp.kernelExecTimeoutEnabled ? "Yes" : "No"));
    return;
}

 int main()
{
    // Number of CUDA devices
    int devCount;
    cudaGetDeviceCount(&devCount);
    printf("CUDA Device Query...\n");
    printf("There are %d CUDA devices.\n", devCount);

    // Iterate through devices
    for (int i = 0; i < devCount; ++i)
    {
        // Get device properties
        printf("\nCUDA Device #%d\n", i);
        cudaDeviceProp devProp;
        cudaGetDeviceProperties(&devProp, i);
        printDevProp(devProp);
    }

    printf("\nPress any key to exit...");
    char c;
    scanf("%c", &c);

    **for (int i = 0; i < devCount; i++) {
        cudaSetDevice(i);
        cudaDeviceReset();
    }**

    return 0;
}

My query is related to the for loop just before the main() ends in which I set each device one by one and then use cudaResetDevice command. I get a strange feeling that this code, although doesnt produce any error but I am not able to reset all the devices. Instead, the program is resetting only the default device i.e device 0 each time. Can anyone tell me what should I do to reset each of the 4 devices.

Thanks

3 Answers 3

5

It looks like you can add a function to your GPU programs to catch the ctrl+c signal (SIGINT) and call the cudaDeviceReset() function for each device that was used by the program.

The example code to call a function when SIGINT is caught can be found here:

https://stackoverflow.com/a/482725

It seems like a good practice to include code like this for every GPU program you write, and I will do the same :-)

I don't have time to write up a full detailed answer, so read the other answer and it's comments also.

3

This is probably too late but if you write a signal-handler function you can get rid of the memory leaks and reset the device in a sure way:

// State variables for 
extern int no_sigint;
int no_sigint = 1;
extern int interrupts;
int interrupts = 0;

/* Catches signal interrupts from Ctrl+c.
   If 1 signal is detected the simulation finishes the current frame and
   exits in a clean state. If Ctrl+c is pressed again it terminates the
   application without completing writes to files or calculations but
   deallocates all memory anyway. */
void
sigint_handler (int sig)
{
  if (sig == SIGINT)
    {
      interrupts += 1;
      std::cout << std::endl
                << "Aborting loop.. finishing frame."
                << std::endl;

      no_sigint = 0;

      if (interrupts >= 2)
        {
          std::cerr << std::endl
                    << "Multiple Interrupts issued: "
                    << "Clearing memory and Forcing immediate shutdown!"
                    << std::endl;

          // write a function to free dynamycally allocated memory
          free_mem ();

          int devCount;
          cudaGetDeviceCount (&devCount);

          for (int i = 0; i < devCount; ++i)
            {
              cudaSetDevice (i);
              cudaDeviceReset ();
            }
          exit (9);
        }
    }
}

....

int main(){ 
.....
for (int simulation_step=1 ; simulation_step < SIM_STEPS && no_sigint; ++simulation_step)
{
   .... simulation code
}
free_mem();
... cuda device resets
return 0;
}

If you use this code (you can even include the first snippet in an external header, it works. You can have 2 levels of control of ctrl+c: the first press stops your simulation and exits normally but the application finishes rendering the step which is great to stop gracefully and have correct results, if you press ctrl+c again it closes the application freeing all memory.

0
1

cudaDeviceReset is intended for destroying resources associated with a given GPU context within the process in which it is run. One CUDA process can't reset or otherwise effect the context of another process. So when your modified device query calls cudaDeviceReset, it is only releases resources that it allocated, not those in use by any other process.

4
  • Thanks for the prompt reply. Then how should I go about reseting each device one by one. For getting devcount, I will have to call cudaGetDeviceCount anyway. Would that start a context in the default device 0 ? Can you suggest a way of ensuring gpu memory purging explicitly for each gpu device ? Thanks
    – Abhinav
    Commented Aug 23, 2011 at 21:53
  • In CUDA 4.0, you need to call cudaSetDevice before executing any other operation. But I repeat that cudaDeviceReset only resets actions that the program that executes has done. If the GPU is in a "bad" state because of what another process has done, it cannot reset or otherwise effect that state. There is basic protection of contexts from one another implemented in the driver which explicitly prevents this.
    – talonmies
    Commented Aug 24, 2011 at 7:00
  • 1
    So, if I have to terminate a kernel in the middle using ctrl-C, and then if I use the program mentioned above with the intention of purging the resources, then you mean to say that it will be in vain and the global-device memory that might not have been erased due to sudden termination will not be cleaned ?
    – Abhinav
    Commented Aug 24, 2011 at 21:24
  • Can you please then suggest a solution for this problem because it is for sure that I will be intentionally terminating my kernel using ctrl-C during its development stage. Presently I can only think of rebooting the machine as a certain solution, very irritating nevertheless. Also, thank you so much for you prompt replies.
    – Abhinav
    Commented Aug 25, 2011 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.