I am trying to train deep residual net work (ResNet34
, with a total of 21,302,722 parameters) using tensorflow 2.0 with GPU (GeForce 940 M). A sequential model is defined as follows:
model = keras.models.Sequential()
model.add(DefaultConv2D(64, kernel_size=7, strides=2,
input_shape=[224, 224, 3]))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.MaxPool2D(pool_size=3, strides=2, padding="SAME"))
prev_filters = 64
for filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3:
strides = 1 if filters == prev_filters else 2
model.add(ResidualUnit(filters, strides=strides))
prev_filters = filters
model.add(keras.layers.GlobalAvgPool2D())
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(2, activation="softmax"))
model.summary()
This model is trained:
history = model.fit(xtrain, ytrain, epochs=10, validation_data=[xtest, ytest])
The xtrain
has the shape of (2000, 224, 224, 3)
and xtest
has the shape of (1000, 224, 224, 3)
.
Then I got the OOM error message:
ResourceExhaustedError: OOM when allocating tensor with shape[256,256,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node residual_unit_28/conv2d_64/Conv2D}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[GroupCrossDeviceControlEdges_0/training/Nadam/Nadam/Const/_287]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[Op:__inference_keras_scratch_graph_30479]
Is the error caused by my computer memory (it has 16 GB RAM), or some improper configurations?