I'm running what I believe is a pretty small CNN on an nVidia Jetson Nano with Jetpack 4.4. nVidia claims the Nano can run a ResNet-50 at 36fps, so I expected my much smaller network to run at 30+ fps with ease.
Actually though, each forward pass takes 160-180ms, so I score 5-6 fps at best.
My CNN:
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
lambda (Lambda) (None, 210, 848, 3) 0
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 210, 282, 3) 0
_________________________________________________________________
conv2d (Conv2D) (None, 102, 138, 16) 2368
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 51, 69, 16) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 24, 33, 32) 12832
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 12, 16, 32) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 4, 6, 64) 51264
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 2, 3, 64) 0
_________________________________________________________________
flatten (Flatten) (None, 384) 0
_________________________________________________________________
dropout (Dropout) (None, 384) 0
_________________________________________________________________
dense (Dense) (None, 64) 24640
_________________________________________________________________
dropout_1 (Dropout) (None, 64) 0
_________________________________________________________________
elu (ELU) (None, 64) 0
_________________________________________________________________
dense_1 (Dense) (None, 1) 65
=================================================================
Total params: 91,169
Trainable params: 91,169
Non-trainable params: 0
_________________________________________________________________
Code:
import numpy as np
import cv2
import time
import tensorflow as tf
from tensorflow import keras
model_name = 'v9_small_FC_epoch_3'
loaded_model = keras.models.load_model('/home/jetson/notebooks/trained_models/' + model_name + '.h5')
loaded_model.summary()
frame = cv2.imread('/home/jetson/notebooks/frame1.jpg')
test_data = np.expand_dims(frame, axis=0)
for i in range(10):
start = time.time()
predictions = loaded_model.predict(test_data)
print(predictions[0][0])
end = time.time()
print("Inference took {}s".format(end-start))
Result:
4.7763316333293915
Inference took 10.111131191253662s
4.7763316333293915
Inference took 0.1822071075439453s
4.7763316333293915
Inference took 0.17330455780029297s
4.7763316333293915
Inference took 0.18085694313049316s
4.7763316333293915
Inference took 0.16646790504455566s
4.7763316333293915
Inference took 0.1703803539276123s
4.7763316333293915
Inference took 0.1788337230682373s
4.7763316333293915
Inference took 0.17131853103637695s
4.7763316333293915
Inference took 0.1660606861114502s
4.7763316333293915
Inference took 0.18377089500427246s