I'm using TensorFlow 2.1 in order to train models with quantization-aware training.
The code to do that is:
import tensorflow_model_optimization as tfmot
model = tfmot.quantization.keras.quantize_annotate_model(model)
This will add fake-quantize nodes to the graph. These nodes should adjust the model's weights so they are more easier to be quantized into int8 and to work with int8 data.
When the training ends, I convert and quantize the model to TF-Lite like so:
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = [give data provider]
quantized_tflite_model = converter.convert()
At this point, I wouldn't expect to see the fake-quantize layers in the TL-Lite graph. But surprisingly, I do see them. Moreover, when I run this quantized model in TF-Lite C++ sample app, I see that it's also running the fake-quantize nodes during inference. In addition to that, it also dequantize and quantize the activations between each layer.
That's a sample of the output from the C++ code:
Node 0 Operator Builtin Code 80 FAKE_QUANT
Inputs: 1
Outputs: 237
Node 1 Operator Builtin Code 114 QUANTIZE
Inputs: 237
Outputs: 238
Node 2 Operator Builtin Code 3 CONV_2D
Inputs: 238 59 58
Outputs: 167
Temporaries: 378
Node 3 Operator Builtin Code 6 DEQUANTIZE
Inputs: 167
Outputs: 239
Node 4 Operator Builtin Code 80 FAKE_QUANT
Inputs: 239
Outputs: 166
Node 5 Operator Builtin Code 114 QUANTIZE
Inputs: 166
Outputs: 240
Node 6 Operator Builtin Code 3 CONV_2D
Inputs: 240 61 60
Outputs: 169
So I find all this very weird, taking also into account the fact that this model should run only on int8 and actually fake-quantize nodes are getting float32 as inputs.
Any help here would be appreciated.