6

I have a custom CNN model, and I have converted it to .tflite format and deployed it on my Android app. However, I can't figure out how to do batching while inference with tensorflow lite.

From this Google doc, it seems you have to set the input format of your model. However, this doc is using a code example with Firebase API, which I'm not planning on using.

To be more specific:

I want to inference multiple 100x100x3 images at once, so the input size is Nx100x100x3.

Question:

How to do this with TF lite?

2 Answers 2

11

You can just call the resizeInput API (Java) or ResizeInputTensor API (if you're using C++).

For example, in Java:

interpreter.resizeInput(tensor_index, [num_batch, 100, 100, 3]);

Let us know if you have any problem batching in TensorFlow lite.

3
  • 5
    Don't forget to call allocate_tensors() afterwards! My kernel was crashing with the python API until i realized this.
    – leonard
    Commented Aug 28, 2019 at 16:28
  • 1
    In regard to the previous command, allocate_tensors() is not a public method in the Java API. Rather, it is called automatically by the underlying framework as needed.
    – Matt
    Commented Nov 3, 2019 at 3:41
  • 4
    Python :interpreter.resize_tensor_input(tensor_index, [num_batch, 100, 100, 3]) Do run the command : interpreter.allocate_tensors() for the above things to take effect. Commented Sep 13, 2020 at 12:12
0

Here is an example, in Java that explains more about inference tflite models with different batch size:

    // First get the input shape of the interpreter, this will give you smth like this [1, 300, 300, 3]
    int[] inputs = interpreter.getInputTensor(0).shape();
    // The first element of above array represents batch size, so we change that
    inputs[0] = 4 // 4 is batch size in this case
    // update interpreter with new input size
    interpreter.resizeInput(0, inputs);

One other important thing is how you prepare the input and output of interpreter, input data will be smth like this (extra attention to the "IMPORTANT" comment):

    int numBytesPerChannel;
    if (isQuantized) {
      numBytesPerChannel = 1; // Quantized
    } else {
      numBytesPerChannel = 4; // Floating point
    }
    imgData = ByteBuffer.allocateDirect(batchSize * inputWidth * inputHeight * 3 * numBytesPerChannel);
    imgData.order(ByteOrder.nativeOrder());
    ........
    // Here you add the data from bitmap into the ByteBuffer (imgData)
    // IMPORTANT: make sure that you write all the data into the same ByteBuffer (in our case imgData)
    ........
    Object[] inputArray = {imgData};
    Map<Integer, Object> outputMap = new HashMap<>();
    // add your output here, I am adding [4][1][1][1024] as example, 4 is batch size
    outputMap.put(0, new float[4][1][1][1024]);
    // and then run the tflite interpreter
    interpreter.runForMultipleInputsOutputs(inputArray, outputMap);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.