0

I am trying to train deep residual net work (ResNet34, with a total of 21,302,722 parameters) using tensorflow 2.0 with GPU (GeForce 940 M). A sequential model is defined as follows:

model = keras.models.Sequential()
model.add(DefaultConv2D(64, kernel_size=7, strides=2,
                        input_shape=[224, 224, 3]))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.MaxPool2D(pool_size=3, strides=2, padding="SAME"))

prev_filters = 64
for filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3:
    strides = 1 if filters == prev_filters else 2
    model.add(ResidualUnit(filters, strides=strides))
    prev_filters = filters

model.add(keras.layers.GlobalAvgPool2D())
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(2, activation="softmax"))
model.summary()

This model is trained:

history = model.fit(xtrain, ytrain, epochs=10, validation_data=[xtest, ytest])

The xtrain has the shape of (2000, 224, 224, 3) and xtest has the shape of (1000, 224, 224, 3).

Then I got the OOM error message:

ResourceExhaustedError: OOM when allocating tensor with shape[256,256,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[{{node residual_unit_28/conv2d_64/Conv2D}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[GroupCrossDeviceControlEdges_0/training/Nadam/Nadam/Const/_287]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_keras_scratch_graph_30479]

Is the error caused by my computer memory (it has 16 GB RAM), or some improper configurations?

1 Answer 1

3

GPU memory and computer memory (RAM) are different. When you are using a GPU to train, it will need to load the layers and inputs into GPU memory. You have too many parameters for your GPU memory. I looked up your GPU and it has only 2 GB of memory which is not enough to do any sort of serious image network training. If you want to train with a GPU I would recommend lowering the number of units in your network, lowering batch size, or using a smaller model overall.

3
  • Thanks for your reply! How to know know if a simplified network will be manageable for my computer? I am considering to reduce number of layers, and use fewer filters.
    – jwm
    Jan 13, 2020 at 19:46
  • 1
    Well, one way is to just try it. You could also 'roughly' calculate whether it will fit, by looking at the number of parameters. Each parameter is a float and for each parameter you need to store it as well as its gradient. The keyword here is roughly though. You could also train with CPU, it will take longer, but you won't be memory bounded anymore. I'm guessing from the GPU you are using a laptop, my real suggestion if you want to train things like resnet would be try an online service like Google Cloud, Amazon AWS, etc. If you are a student they will give you free credits to use their GPUs.
    – cmxu
    Jan 13, 2020 at 19:51
  • I don't know what kind of CPU you have, but for relatively old and underpowered GPUs the amount of speedup you get compared to CPU may not be worth the degradation in model size. Really depends on your goals/use case.
    – cmxu
    Jan 13, 2020 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.