Questions tagged [tensorflow-serving]
The tensorflow-serving tag has no usage guidance, but it has a tag wiki.
1,257
questions
0
votes
0
answers
28
views
tensorflow multi-thread inference gpu utilization low
I load a tensorflow model with LoadSavedModel and predict with the singleton bundle.session->Run, when predict only single thread(last prediction finished then next require post), it cost very ...
0
votes
1
answer
23
views
Tensorflow serving post api calling
HOW TO HANDLE THIS ERROR: "Failed to get input map for signature: serving_default"
This error typically occurs when there is a mismatch between the format of the input data provided to the ...
2
votes
1
answer
28
views
How to secure TensorFlow Serving API
As a serving solution for production environments, does TensorFlow Serving include any options for securing it's REST API?
I can run-up a REST API endpoint by following these instructions https://...
0
votes
0
answers
16
views
Is it possible to remove the "batch" dimension of input after training?
When I train the image classification model, the input shape is (None, Height, Width, Channels).
After train, I need to deploy it by tf-serving, and I still need to post a tensor with shape (None, ...
0
votes
0
answers
27
views
How to build TensorFlow Serving 2.11?
I am trying to build TensorFlow Serving 2.11 according to the instruction :
git clone https://github.com/tensorflow/serving
cd serving/
git checkout r2.11
tools/run_in_docker.sh bazel build -c opt ...
0
votes
1
answer
16
views
Tensorflow serving model versioning
I'm evaluation an option to host a saved tensorflow model using the tensorflow/serving docker image.
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/docker.md
If I mount a ...
0
votes
0
answers
23
views
Serving a RegNet model
I post the question here as recommended in tf-serving github repo
I trained a RegNetX008 model using tensorflow 2.12 and now I struggle with serving it. I use tensorflow/serving-2.12.1-gpu.
When ...
0
votes
0
answers
50
views
File system scheme 'hdfs' not implemented FROM docker image tensorflow/serving:2.10.0-gpu
Here is my Dockerfile, I add hadoop into the image:
FROM tensorflow/serving:2.10.0-gpu
RUN apt update && apt install -y openjdk-8-jre && apt-get clean
COPY hadoop-2.10.2 /root/hadoop
...
1
vote
0
answers
84
views
Turn protobuf-like string into json in python without scheme
I have a log of grpc messages for a TensorflowServing system looking like this:
model_spec:{name:"model_name"} inputs:{key:"args_0" value:{dtype:DT_STRING tensor_shape:{dim:{...
0
votes
1
answer
24
views
Using the Same Model with Multiple Tensorflow Serving Instances
I am currently running two Tensorflow Serving Docker images (one for production and one for testing) that use the same exact model. If my testing instance is having a lot of traffic, will it still ...
0
votes
0
answers
55
views
tensorflow-serving in kube uses more memory on larger kube workers
We run tensorflow serving in kubernetes. We increased the kubernetes node instance size from 16x64 to 32x128 and found that tensorflow serving container started to use more memory (more than the ...
1
vote
1
answer
128
views
Import cycle in tensorflow protobufs
I am trying to write client code to talk to a tensorflow server. I need the golang compiled protobufs for tensorflow and tensorflow_serving. These are not easy to come by, I managed to do so through ...
0
votes
1
answer
50
views
TF Served Model Over Docker Missing Model Information, Returns 404 If Requested
I have a trained model I am trying to deploy an image classifier through tensorflow serving in a Docker container. When I run my python code to request over REST I get a 404 response. The request ...
1
vote
0
answers
89
views
Compiling tensorflow serving protos
I've been trying to request a tensorflow serving model however I need to serialize & encode the data so I've been trying to compile tensorflow serving protos into golang following this tutorial.
...
0
votes
0
answers
78
views
The predictions of the same model and the same images in python and tfgo are different
I had to make an image classification program. I trained a tensorflow model in python and had it working alright. However, now I was asked to implement the model in golang. I was using tfgo but it ...
0
votes
1
answer
60
views
Getting <Response [400]> from the TF serving api
I have trained a model to classify the leafs using TF anf TFX. I am TFServing to deploy the model in local environment. While I am been able to deploy the model but when I make a POST request to the ...
0
votes
0
answers
28
views
Datetime handline in tensorflow
How does one handle datetime preprocessing/serving in tensorflow? The use case is you have a string, and you want to figure out what time it is. There is more information at https://github.com/...
0
votes
1
answer
154
views
While integrating sagemaker endpoint with datadog is there any way we dont use cloudwatch logs
I am trying to integrate SageMaker endpoint which uses inbuilt sagemaker container with datadog. So if i want to integrate it with datadog is there a way to not use cloudwatch and still get the logs ...
0
votes
0
answers
65
views
Tensorflow serving Deep learning model in key value pair
I have build deep learning model which has training and testing data with these columns.
DeadorAlive InscClaimAmtReimbursed Age Fraud
1 500 23 1
I am dividing this ...
0
votes
0
answers
80
views
AutoML model served with tensorflow_model_server always returns same output
I've got a tabular model trained with Google Cloud AutoML that I have exported to my local disk and want to serve with tensorflow_model_server.
I'm running tensorflow_model_server in a Docker ...
0
votes
0
answers
36
views
TfServing : How to pass parameters in addition to feature inputs to model call
I have a keras custom model that performs simple linear scoring given a set of features. I am hosting this model using TFServing REST API. If I want to pass a URL parameter or a header to the ...
0
votes
0
answers
58
views
SageMaker Serverless Inference Preprocessing Question
I've recently seen that there is a serverless version of SageMaker and I wanted to use it for a personal project (first time using SageMaker). I used the guide below to try and deploy my model, only ...
0
votes
0
answers
47
views
Problems with saving custom model for TF serving
I defined simple custom model:
import tensorflow as tf
class CustomModule(tf.keras.layers.Layer):
def __init__(self):
super(CustomModule, self).__init__()
self.v = tf.Variable(1.)
def ...
4
votes
1
answer
272
views
How to tell Tensorflow Serving to leave my named output untouched?
I want to create a model signature that returns named tensors (using Keras). This is what I mean by that. When the model is deployed to TF-Serving, I want it to return a JSON like this:
{
"...
0
votes
0
answers
41
views
How to save a tensorflow function that takes an arbitrary length input
I have built a custom 1-dimensional KMeans module in tensorflow. When executing in pure python everything works as specified. I.e. after training, I can call km.predict(tensor) where tensor is of ...
1
vote
2
answers
302
views
How to name the outputs of a Keras Functional API model?
I have an ML model developed using Keras and more accurately, it's using Functional API. Once I save the model and use the saved_model_cli tool on it:
$ saved_model_cli show --dir /...
0
votes
1
answer
99
views
There are some TFServing metrics in Prometheus, where can I find documentation that describes them?
I have connected my tfserving to Prometheus and followed by that to Grafana, There are several metrics but I couldn't find description for them! all of them have followed this structure :tensorflow:cc....
0
votes
1
answer
44
views
Maximum HTTP post size in Tensorflow Serving
what is the maximum http post body size for the TFX serving?
I can not find it anywhere in the https://www.tensorflow.org/tfx/serving/api_rest documentation nor in other parts of documentation. Most ...
0
votes
0
answers
83
views
In Java, is there any way to convert TensorProto to TensorContent?
When I use tensorflow's grpc client to request tf serving in Java, many TensorProto are built. In a batch request, a large number of TensorProto are duplicated. I found that the comments of ...
1
vote
1
answer
120
views
Deploy a Tensorflow model built using TF2 in TF1 format (no SavedModel bundles found!)
I have used Recommenders https://github.com/microsoft/recommenders library to train an NCF recommendation model. Currently I'm getting issues in deployment through Amazon TensorflowModel library
Model ...
0
votes
0
answers
168
views
Request to Tensorflow Serving Server got timeout error for models using grpc python API
Synopsis
10 models for computer vision was deployed to tensorflow serving server (TSS) running on Ubuntu 22.04.
TSS installed as system service on the dedicated server having MSI RTX3060 12 Gb on ...
1
vote
1
answer
585
views
Optimize batch transform inference on sagemaker
With current batch transform inference I see a lot of bottlenecks,
Each input file can only have close to 1000 records
Currently it is processing 2000/min records on 1 instance of ml.g4dn.12xlarge
...
0
votes
1
answer
68
views
SavedModel usage is just for Tensor serving or to retrain
Is this SavedModel just for Tensorflow front-end applications or it can be use to reload model in keras format. I created it using tf.saved_model.save and now I don't know what to make of it.
...
0
votes
1
answer
77
views
Unable to run tensorflow serving on Ubuntu using docker
I followed all the steps listed here to use tensorflow serving
https://www.tensorflow.org/tfx/serving/tensorboard
used below command to run using docker
sudo docker run -it --rm -p 8500:8500 -p 8502:...
0
votes
0
answers
101
views
Keras model custom signatures are not saved when learning is done on GCP
After reading blog "An Object Oriented Approach To Train an Image Classifier with Tensorflow" I've tried to set custom signatures for my Keras model (TF2) like this:
class ...
1
vote
0
answers
37
views
Tiny single Bidirectional layer big RAM usage?
I have several different small models 1-2MB size on disk, some are just fully connected layers, Transformer, and a couple of Bidirectional GRU networks. But they have very different memory ...
0
votes
0
answers
131
views
seldon-code tfserver grpc requests returns `StatusCode.UNIMPLEMENTED`
I have a tensorflow model .pb with the spec as below
The given SavedModel SignatureDef contains the following input(s):
inputs['Conv1_input'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 28, ...
0
votes
1
answer
182
views
Inputs to TensorFlow serving REST API
I have build a seq2seq model by referring to this tutorial.
https://keras.io/examples/nlp/lstm_seq2seq/
After training the model, instead of saving the model directly, I saved the encoder_model and ...
0
votes
0
answers
51
views
InfraValidator component not "blessing" my model
The InfraValidator TFX component is not blessing my model, and I'm having trouble finding out why it isn't getting "blessed". I'm not sure what part of my model would be causing it to not be ...
0
votes
0
answers
54
views
Estimating model memory footprint in tensorflow serving
We want to measure the memory footprint of each model live in tensorflow serving container. We are using the open source tf container (https://hub.docker.com/r/tensorflow/serving)
Tried scraping the ...
1
vote
0
answers
163
views
How to get prediction labels with Tensorflow Serving Predict API?
I have a keras classification model saved remotely, and I use RESTful API to call the model on my site and make predictions. The JSON request is done as follows
URL:
POST https://my.site/models/v1:...
0
votes
1
answer
268
views
Tensorflow v2.10 mutate output of signature function to be a map of label to results
I'm trying to save my model so that when called from tf-serving the output is:
{
"results": [
{ "label1": x.xxxxx, "label2": x.xxxxx },
{ "label1"...
0
votes
0
answers
116
views
How to log tensorflow-serving docker container request and error logs?
I have deployed the tensorflow-serving docker image with the path to tf model.
The REST api (/predict) is working as expected. But, the problem is , to use the model in production environment ...
0
votes
0
answers
45
views
When to use spark with tensorflow compared to tf serving for inference?
I am currently facing a situation where I have a vast amount of images coming in from time to time and on which I will have to run some models for inferencing.
To solve this scaling issue, I came ...
0
votes
1
answer
72
views
Migrating TensorFlow serving 1 to 2, RESSOURCE_EXHAUSTED from grpc calls
We are trying to migrate from /tensorflow/serving:1.12 to /tensorflow/serving:2.8.2, we use grpc calls to get the recommendation, we have two types of calls, one for each memeber where the grpc call ...
1
vote
1
answer
444
views
Does AWS Sagemaker supports gRPC prediction requests?
I deployed a Sagemaker's Tensorflow model from an estimator in local mode and when trying to call the Tensorflow Serving (TFS) predict endpoint using gRPC I get the error:
grpc._channel....
0
votes
3
answers
402
views
Tensorflow serving failing with std::bad_alloc
I'm trying to run tensorflow-serving using docker compose (served model + microservice) but the tensorflow serving container fails with the error below and then restarts.
microservice | To enable them ...
1
vote
0
answers
192
views
TensorFlow Serving SSD Mobile Net not responding to gRPC inference requests
new poster here.
I've downloaded the SSD MobileNet v2 320x320 from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
I have hosted it with a ...
2
votes
0
answers
130
views
Tensorflow SavedModel serving for What-If Tool Tensorboard
I have one CSV with around 10k rows and around 370 columns mostly numerical (int or float) and ID columns which is unique and I know the target column (integer type column) which needs to be used as ...
1
vote
0
answers
210
views
GRPC request gives different response with multiple instances
I am trying to do model prediction through tensorflow serving through grpc connection and my main code runs with multiple instances. I send Predict request with the following code
req = predict_pb2....