63

I want to specify the gpu to run my process. And I set it as follows:

import tensorflow as tf
with tf.device('/gpu:0'):
    a = tf.constant(3.0)
with tf.Session() as sess:
    while True:
        print sess.run(a)

However it still allocate memory in both my two gpus.

|    0      7479    C   python                         5437MiB 
|    1      7479    C   python                         5437MiB 
2
  • 3
    TensorFlow initializes all GPUs it sees, you need to set CUDA_VISIBLE_DEVICES to limit visible GPUs Oct 16, 2016 at 20:45
  • Ok, I got it. thanks.
    – lhao0301
    Oct 17, 2016 at 2:14

9 Answers 9

71

There are 3 ways to achieve this:

  1. Using CUDA_VISIBLE_DEVICES environment variable. by setting environment variable CUDA_VISIBLE_DEVICES="1" makes only device 1 visible and by setting CUDA_VISIBLE_DEVICES="0,1" makes devices 0 and 1 visible. You can do this in python by having a line os.environ["CUDA_VISIBLE_DEVICES"]="0,1" after importing os package.

  2. Using with tf.device('/gpu:2') and creating the graph. Then it will use GPU device 2 to run.

  3. Using config = tf.ConfigProto(device_count = {'GPU': 1}) and then sess = tf.Session(config=config). This will use GPU device 1.

5
  • 4
    I have tried all the three methods. The second one still doesn't work. My demo code is shown as above.
    – lhao0301
    Jul 7, 2017 at 7:22
  • In theory this should work, but sometimes you have to physically switch the graphics cards. Why? No idea! But it works for me when this method fails.
    – Jon
    Apr 4, 2018 at 18:49
  • 4
    I don't think part three is entirely correct. As the name suggests device_count only sets the number of devices being used, not which. From the tf source code: message ConfigProto { // Map from device type name (e.g., "CPU" or "GPU" ) to maximum // number of devices of that type to use. If a particular device // type is not found in the map, the system picks an appropriate // number. map<string, int32> device_count = 1;, see github.com/tensorflow/tensorflow/blob/r1.11/tensorflow/core/…
    – ftiaronsem
    Oct 18, 2018 at 17:47
  • 3
    Note that if tf.ConfigProto(device_count = {'GPU': 0}) is set all will be runned on CPU, so it's not gpu_id, but number of gpus. Use gpu_options = tf.GPUOptions(allow_growth=True, visible_device_list=str(gpu_id))
    – mrgloom
    Jun 8, 2019 at 2:06
  • Note that tf.Session is outdated as of 2020 Dec 11, 2022 at 10:47
49

TF would allocate all available memory on each visible GPU if not told otherwise. Here are 5 ways to stick to just one (or a few) GPUs.

Bash solution. Set CUDA_VISIBLE_DEVICES=0,1 in your terminal/console before starting python or jupyter notebook:

CUDA_VISIBLE_DEVICES=0,1 python script.py

Python solution. run next 2 lines of code before constructing a session

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"

Automated solution. Method below will automatically detect GPU devices that are not used by other scripts and set CUDA_VISIBLE_DEVICES for you. You have to call mask_unused_gpus before constructing a session. It will filter out GPUs by current memory usage. This way you can run multiple instances of your script at once without changing your code or setting console parameters.

The function:

import subprocess as sp
import os

def mask_unused_gpus(leave_unmasked=1):
  ACCEPTABLE_AVAILABLE_MEMORY = 1024
  COMMAND = "nvidia-smi --query-gpu=memory.free --format=csv"

  try:
    _output_to_list = lambda x: x.decode('ascii').split('\n')[:-1]
    memory_free_info = _output_to_list(sp.check_output(COMMAND.split()))[1:]
    memory_free_values = [int(x.split()[0]) for i, x in enumerate(memory_free_info)]
    available_gpus = [i for i, x in enumerate(memory_free_values) if x > ACCEPTABLE_AVAILABLE_MEMORY]

    if len(available_gpus) < leave_unmasked: raise ValueError('Found only %d usable GPUs in the system' % len(available_gpus))
    os.environ["CUDA_VISIBLE_DEVICES"] = ','.join(map(str, available_gpus[:leave_unmasked]))
  except Exception as e:
    print('"nvidia-smi" is probably not installed. GPUs are not masked', e)

mask_unused_gpus(2)

Limitations: if you start multiple scripts at once it might cause a collision, because memory is not allocated immediately when you construct a session. In case it is a problem for you, you can use a randomized version as in original source code: mask_busy_gpus()

Tensorflow 2.0 suggest yet another method:

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only use the first GPU
  try:
    tf.config.experimental.set_visible_devices(gpus[0], 'GPU')
  except RuntimeError as e:
    # Visible devices must be set at program startup
    print(e)

Tensorflow/Keras also allows to specify gpu to be used with session config. I can recommend it only if setting environment variable is not an options (i.e. an MPI run). Because it tend to be the least reliable of all methods, especially with keras.

config = tf.ConfigProto()
config.gpu_options.visible_device_list = "0,1"
with tf.Session(config) as sess:
#or K.set_session(tf.Session(config))
2
  • 3
    The selected answer is technically correct but this is the best way to handle this scenario. Oct 18, 2018 at 16:14
  • 1
    For Bash solution, shouldn't it be CUDA_VISIBLE_DEVICES=0,1 python script.py, without $ ahead of the env variable CUDA_VISIBLE_DEVICES? Apr 18, 2020 at 19:18
37

I believe that you need to set CUDA_VISIBLE_DEVICES=1. Or which ever GPU you want to use. If you make only one GPU visible, you will refer to it as /gpu:0 in tensorflow regardless of what you set the environment variable to.

More info on that environment variable: https://devblogs.nvidia.com/cuda-pro-tip-control-gpu-visibility-cuda_visible_devices/

2
  • @ScottYang, try tensorflow.org/guide/…
    – Russell
    Feb 25, 2019 at 22:20
  • Using os.environ["CUDA_VISIBLE_DEVICES"]="0" like in the other answer allows the option to take user input on which GPU to use. Oct 2, 2020 at 19:02
7

You can modify the GPU options settings by adding at the begining of your python script:

gpu_options = tf.GPUOptions(visible_device_list="0")
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

"0" is here the name of the GPU you want to use. You can have the list of the GPU available by typing the command nvidia-smi in the terminal prompt.


With Keras, these 2 functions allow the selection of CPU or GPU and in the case of GPU the fraction of memory that will be used.

import os
from keras.backend.tensorflow_backend import set_session
import tensorflow as tf



def set_cpu_option():
    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"  # see issue #152
    os.environ["CUDA_VISIBLE_DEVICES"] = ""
    os.environ["CUDA_VISIBLE_DEVICES"] = ""


def set_gpu_option(which_gpu, fraction_memory):
    config = tf.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = fraction_memory
    config.gpu_options.visible_device_list = which_gpu
    set_session(tf.Session(config=config))
    return

set_gpu_option("0", 0.9)
# or 
set_cpu_option()
1
  • 1
    It gives me undefined name set_session, undefined name logger
    – Hasani
    Aug 2, 2019 at 10:43
2
def set_specific_gpu(ID):
    gpus_all_physical_list = tf.config.list_physical_devices(device_type='GPU')    
    tf.config.set_visible_devices(gpus_all_physical_list[ID], 'GPU')

refer to https://www.tensorflow.org/api_docs/python/tf/config/list_physical_devices

2
import tensorflow as tf


gpu_number = 2 #### GPU number 
gpus = tf.config.list_physical_devices('GPU')
if gpus:
    tf.config.experimental.set_visible_devices(gpus[gpu_number], 'GPU') 
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU")
1

The most elegant and clean way I have seen this work for me on my multi-core gpu setup is:

import os
os.environ["CUDA_VISIBLE_DEVICES"]="1"
tf_device='/gpu:0'

This assigns the task to gpu device 1.

Similarly, doing something on the lines:

import os 
os.environ["CUDA_VISIBLE_DEVICES"]="2"
tf_device='/gpu:0'

The os.environ command can be seen as a way of making only that GPU device exposed on which you intend to run the code. The second command just picks the first of the available devices that you specified.

enter image description here

1
  • 3
    No, it's not elegant to set environment variables of the same process. You should set them before you run the process, for example: CUDA_VISIBLE_DEVICES=XYZ python script.py May 17, 2020 at 17:54
1
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="3"

The only thing which worked for me cleanly from within Processes to assign specific GPU to each process in a Pool.

1

TF 2.9 and above version of tensorflow has changed the APIs, so updating for the same,

gpus = tf.config.list_physical_devices('GPU')
gpu_id = 0
if gpus:
    # Restrict TensorFlow to only use only one GPU based on gpu_id
    try:
        tf.config.set_visible_devices(gpus[gpu_id], 'GPU')
        logical_gpus = tf.config.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPU")
    except RuntimeError as e:
        # Visible devices must be set before GPUs have been initialized
        print(e)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.