I'm using Tensorflow Object Detection API to train detectors for 3 objects, which have distinct looks apart from everyday objects. I collected my own dataset and I am finetuning from the pre-trained Faster-RCNN ResNet101 model. In model config for the network, does it make sense to set following parameters for the second_stage_post_processing?
max_detections_per_class: 1
max_total_detections: 3
Since both in my train and test environments, the maximum number of objects that could appear is 3 (1 instance for each class). But I'm not sure if these parameters make sense for training. For the second stage BoxClassifier, if the max_total_detection is 3, does that mean the loss will be calculated from 3 predictions (bounding box + label) at maximum for one training image? If so, it seems that the model loses a lot of opportunities to learn "background" class. If not, what are a few good values to use?