2

This question is about finding a solution on how to run a trained model on an Android device without using the convert TF Lite and without using a external service.

I don't owned the model and cannot modify it. I just have the trained saved model files.

The device is out of network and should embed the trained model. No connection to an external server is possible.

Tensorflow Lite is not an option since TF Lite doesn't support 5D tensors: https://github.com/tensorflow/tensorflow/issues/56946

In order to do my test I will get the basic model I have provided in the above tensorflow issue to do my tests.

I have found this blog article, but didn't manage to make it work yet: https://medium.com/@vladislavsd/undocumented-tensorflow-c-api-b527c0b4ef6

Do you know any updated solution that enables to load the model inside a Java or C++ lib on Android?

No example is proposed by Tensorflow on their GitHub: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android

5
  • What is the context of the problem you are trying to solve/nature of the model? If the model is already trained, you could house it behind a web API, and then use a standard HTTP call on Android. Aug 24, 2022 at 18:42
  • As I said, the model is already trained and exported. I don't own its code. It is provided by a partnership. You mean building a web API on the Android device so it make request on localhost? The model has to be embedded on the device and cannot be externalized on a server.
    – Thibault
    Aug 25, 2022 at 7:36
  • Why does it have to be on the device? It is because of performance concerns (network latency)? Does it need to be re-trained with dynamic data from the user? My point is that I don't think this is an Android-specific problem, but a TensorFlow one. The question seems to be "how do I integrate a TensorFlow model with an application?", and there are many ways - the best choice depends on the nature of your application. Aug 25, 2022 at 15:04
  • Here is one link that might be helpful, specifically the last section: medium.com/google-cloud/…. Aug 25, 2022 at 15:06
  • It has to be on the device because it's a specification of the project. We want it to work offline. The model will not be re-trained once deployed. Your link seems to be a bit outdated. I will give it a try and return to you. I am currently trying converting the model to an ONNX model in order to deploy this new model on Android.
    – Thibault
    Aug 26, 2022 at 8:16

2 Answers 2

1

I have succeeded to deploy my trained model using 5D tensor on Android Emulator.

In order to do that, I have converted my model using the converter from Tensorflow to ONNX: https://github.com/onnx/tensorflow-onnx

python -m tf2onnx.convert --saved-model tensorflow-model-path --output model.onnx

Then I have created a C++ lib that loads the ONNX model from the converted file and calls it.

In order to copy the asset on the phone storage, I have followed this topic: https://stackoverflow.com/a/69941051/12851157

You can find ONNX samples here: https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx

And finally I have integrated the C++ lib in Android like this: https://github.com/android/ndk-samples/tree/master/hello-libs

If I have enough time, I will try to use the TF API.

0

If TFLite doesn't work for your model due to the limited support, you can use Select TensorFlow ops feature. https://www.tensorflow.org/lite/guide/ops_select

It allows you to use TF ops in TFLite so you can overcome the limited 5D supports of TFLite but it impacts your binary size.

1
  • Sorry, but I have tried it and it doesn't work. You can read the GitHub issue about this.
    – Thibault
    Sep 7, 2022 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.