47

How to kill running processes on GPUs for a specific program (e.g. python) in terminal? For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi

For example two processes are running with python in the top picture and kill them to see the bottom picture in nvidia-smi

6 Answers 6

65

The accepted answer doesn't work for me, probably because nvidia-smi has different formats across different versions/hardware.

I'm using a much cleaner command:

nvidia-smi | grep 'python' | awk '{ print $3 }' | xargs -n1 kill -9

You can replace $3 in the awk expression to fit your nvidia-smi output. It is the n-th column in which the PIDs occur.

6
  • 2
    Same here, accepted answer didn't work. This one did. Kudos !! Nov 25, 2019 at 9:18
  • 2
    I needed to sudo kill nvidia-smi | grep 'python' | awk '{ print $3 }' | sudo xargs -n1 kill -9 Dec 31, 2019 at 10:52
  • 1
    kill -9 <pid> works. use sudo wisely. Feb 21, 2021 at 4:44
  • 5
    had to change to awk '{ print $5}' but this worked
    – misinglink
    Apr 2, 2021 at 8:11
  • 7
    for me, nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9 worked. Dec 6, 2021 at 8:30
15

Use nvidia-smi or top command to see processes running and to kill command:

sudo kill -9 PID
1
  • This answer is addressed as a comment in response to @oya163 under the accepted answer. Aug 9, 2021 at 1:45
11

I guess the question is already answered when nvidia-smi shows processes occupying GPU mem. For me, even though nvidia-smi wasnt showing any processes, GPU memory was being used and I wanted to kill them.

The way to go in this case was to use the fuser command to find out the processes using the particular GPU device. In my case I wanted to kill all the processes using the GPU device 3. This can be done using the command :

sudo fuser -k /dev/nvidia3

You can use -ki to kill the processes interactively.

11

You can grep "python" in the nvidia-smi and then pass the PID to the kill -9 command, e.g.

sudo kill -9 $( nvidia-smi | grep 'python' | sed -n 's/|\s*[0-9]*\s*\([0-9]*\)\s*.*/\1/p' |  sed '/^$/d')
5
  • Where do I mention PID??
    – oya163
    Oct 15, 2018 at 20:30
  • @oya163 it should grab it using grep from nvidia-smi. No need to manually add it. Oct 16, 2018 at 21:03
  • No I meant a user-specific PID? Because I don't want to and I can't kill other's PID.
    – oya163
    Oct 17, 2018 at 20:57
  • 9
    @oya163 in this case you may try: sudo kill -9 PID Oct 18, 2018 at 21:53
  • had to change to awk '{ print $5}' but this worked.
    – misinglink
    Apr 2, 2021 at 8:09
6

As one of other answers suggest you can use: (replace 5 with the column number where process id exists)

nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9

If you might have to use this a lot you can create an alias for the command: to do that do this you should edit ~/.bash_aliases file:

nano ~/.bash_aliases

and add the following line to it and save the file:

alias killgpuprocess="nvidia-smi | grep 'python' | awk '{ print $5 }' | xargs -n1 kill -9"

then (just needed this time):

source ~/.bashrc

Then if you run

killgpuprocess

it will kill the existing processes on GPU(s).

1
  • 1
    I had to use 5 instead of 3 for the top-voted answer to work.
    – Atif Ali
    Sep 2, 2022 at 6:53
0

Keeping here as a reference. In case one wants to kill all running processes on the gpu, following command would work:

nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs -n1 kill -9

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.