Questions tagged [tensorflow-serving]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0 votes
0 answers
28 views

tensorflow multi-thread inference gpu utilization low

I load a tensorflow model with LoadSavedModel and predict with the singleton bundle.session->Run, when predict only single thread(last prediction finished then next require post), it cost very ...
zhu3359's user avatar
0 votes
1 answer
23 views

Tensorflow serving post api calling

HOW TO HANDLE THIS ERROR: "Failed to get input map for signature: serving_default" This error typically occurs when there is a mismatch between the format of the input data provided to the ...
Mayur Katre's user avatar
2 votes
1 answer
28 views

How to secure TensorFlow Serving API

As a serving solution for production environments, does TensorFlow Serving include any options for securing it's REST API? I can run-up a REST API endpoint by following these instructions https://...
ChrisTaylorDeveloper's user avatar
0 votes
0 answers
16 views

Is it possible to remove the "batch" dimension of input after training?

When I train the image classification model, the input shape is (None, Height, Width, Channels). After train, I need to deploy it by tf-serving, and I still need to post a tensor with shape (None, ...
huang's user avatar
  • 927
0 votes
0 answers
27 views

How to build TensorFlow Serving 2.11?

I am trying to build TensorFlow Serving 2.11 according to the instruction : git clone https://github.com/tensorflow/serving cd serving/ git checkout r2.11 tools/run_in_docker.sh bazel build -c opt ...
crossbows's user avatar
0 votes
1 answer
16 views

Tensorflow serving model versioning

I'm evaluation an option to host a saved tensorflow model using the tensorflow/serving docker image. https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/docker.md If I mount a ...
senjin.hajrulahovic's user avatar
0 votes
0 answers
23 views

Serving a RegNet model

I post the question here as recommended in tf-serving github repo I trained a RegNetX008 model using tensorflow 2.12 and now I struggle with serving it. I use tensorflow/serving-2.12.1-gpu. When ...
Mayis's user avatar
  • 180
0 votes
0 answers
50 views

File system scheme 'hdfs' not implemented FROM docker image tensorflow/serving:2.10.0-gpu

Here is my Dockerfile, I add hadoop into the image: FROM tensorflow/serving:2.10.0-gpu RUN apt update && apt install -y openjdk-8-jre && apt-get clean COPY hadoop-2.10.2 /root/hadoop ...
KeithTt's user avatar
  • 71
1 vote
0 answers
84 views

Turn protobuf-like string into json in python without scheme

I have a log of grpc messages for a TensorflowServing system looking like this: model_spec:{name:"model_name"} inputs:{key:"args_0" value:{dtype:DT_STRING tensor_shape:{dim:{...
mrzo's user avatar
  • 2,057
0 votes
1 answer
24 views

Using the Same Model with Multiple Tensorflow Serving Instances

I am currently running two Tensorflow Serving Docker images (one for production and one for testing) that use the same exact model. If my testing instance is having a lot of traffic, will it still ...
Joey Holtzman's user avatar
0 votes
0 answers
55 views

tensorflow-serving in kube uses more memory on larger kube workers

We run tensorflow serving in kubernetes. We increased the kubernetes node instance size from 16x64 to 32x128 and found that tensorflow serving container started to use more memory (more than the ...
Paul Slater's user avatar
1 vote
1 answer
128 views

Import cycle in tensorflow protobufs

I am trying to write client code to talk to a tensorflow server. I need the golang compiled protobufs for tensorflow and tensorflow_serving. These are not easy to come by, I managed to do so through ...
maininformer's user avatar
0 votes
1 answer
50 views

TF Served Model Over Docker Missing Model Information, Returns 404 If Requested

I have a trained model I am trying to deploy an image classifier through tensorflow serving in a Docker container. When I run my python code to request over REST I get a 404 response. The request ...
parabolic_boomerang's user avatar
1 vote
0 answers
89 views

Compiling tensorflow serving protos

I've been trying to request a tensorflow serving model however I need to serialize & encode the data so I've been trying to compile tensorflow serving protos into golang following this tutorial. ...
Regalia's user avatar
  • 139
0 votes
0 answers
78 views

The predictions of the same model and the same images in python and tfgo are different

I had to make an image classification program. I trained a tensorflow model in python and had it working alright. However, now I was asked to implement the model in golang. I was using tfgo but it ...
ChicoKeitaro's user avatar
0 votes
1 answer
60 views

Getting <Response [400]> from the TF serving api

I have trained a model to classify the leafs using TF anf TFX. I am TFServing to deploy the model in local environment. While I am been able to deploy the model but when I make a POST request to the ...
Bhargav Patel's user avatar
0 votes
0 answers
28 views

Datetime handline in tensorflow

How does one handle datetime preprocessing/serving in tensorflow? The use case is you have a string, and you want to figure out what time it is. There is more information at https://github.com/...
Pritam Dodeja's user avatar
0 votes
1 answer
154 views

While integrating sagemaker endpoint with datadog is there any way we dont use cloudwatch logs

I am trying to integrate SageMaker endpoint which uses inbuilt sagemaker container with datadog. So if i want to integrate it with datadog is there a way to not use cloudwatch and still get the logs ...
Janhavi Khindkar's user avatar
0 votes
0 answers
65 views

Tensorflow serving Deep learning model in key value pair

I have build deep learning model which has training and testing data with these columns. DeadorAlive InscClaimAmtReimbursed Age Fraud 1 500 23 1 I am dividing this ...
Sam's user avatar
  • 165
0 votes
0 answers
80 views

AutoML model served with tensorflow_model_server always returns same output

I've got a tabular model trained with Google Cloud AutoML that I have exported to my local disk and want to serve with tensorflow_model_server. I'm running tensorflow_model_server in a Docker ...
Joko's user avatar
  • 166
0 votes
0 answers
36 views

TfServing : How to pass parameters in addition to feature inputs to model call

I have a keras custom model that performs simple linear scoring given a set of features. I am hosting this model using TFServing REST API. If I want to pass a URL parameter or a header to the ...
A. Ganesh's user avatar
0 votes
0 answers
58 views

SageMaker Serverless Inference Preprocessing Question

I've recently seen that there is a serverless version of SageMaker and I wanted to use it for a personal project (first time using SageMaker). I used the guide below to try and deploy my model, only ...
rinja9's user avatar
  • 1
0 votes
0 answers
47 views

Problems with saving custom model for TF serving

I defined simple custom model: import tensorflow as tf class CustomModule(tf.keras.layers.Layer): def __init__(self): super(CustomModule, self).__init__() self.v = tf.Variable(1.) def ...
Alex's user avatar
  • 21
4 votes
1 answer
272 views

How to tell Tensorflow Serving to leave my named output untouched?

I want to create a model signature that returns named tensors (using Keras). This is what I mean by that. When the model is deployed to TF-Serving, I want it to return a JSON like this: { "...
Mehran's user avatar
  • 15.6k
0 votes
0 answers
41 views

How to save a tensorflow function that takes an arbitrary length input

I have built a custom 1-dimensional KMeans module in tensorflow. When executing in pure python everything works as specified. I.e. after training, I can call km.predict(tensor) where tensor is of ...
Logister's user avatar
  • 1,862
1 vote
2 answers
302 views

How to name the outputs of a Keras Functional API model?

I have an ML model developed using Keras and more accurately, it's using Functional API. Once I save the model and use the saved_model_cli tool on it: $ saved_model_cli show --dir /...
Mehran's user avatar
  • 15.6k
0 votes
1 answer
99 views

There are some TFServing metrics in Prometheus, where can I find documentation that describes them?

I have connected my tfserving to Prometheus and followed by that to Grafana, There are several metrics but I couldn't find description for them! all of them have followed this structure :tensorflow:cc....
Sahel rad's user avatar
0 votes
1 answer
44 views

Maximum HTTP post size in Tensorflow Serving

what is the maximum http post body size for the TFX serving? I can not find it anywhere in the https://www.tensorflow.org/tfx/serving/api_rest documentation nor in other parts of documentation. Most ...
Matěj Račinský's user avatar
0 votes
0 answers
83 views

In Java, is there any way to convert TensorProto to TensorContent?

When I use tensorflow's grpc client to request tf serving in Java, many TensorProto are built. In a batch request, a large number of TensorProto are duplicated. I found that the comments of ...
Mr.Yin's user avatar
  • 71
1 vote
1 answer
120 views

Deploy a Tensorflow model built using TF2 in TF1 format (no SavedModel bundles found!)

I have used Recommenders https://github.com/microsoft/recommenders library to train an NCF recommendation model. Currently I'm getting issues in deployment through Amazon TensorflowModel library Model ...
Muhammad Humza's user avatar
0 votes
0 answers
168 views

Request to Tensorflow Serving Server got timeout error for models using grpc python API

Synopsis 10 models for computer vision was deployed to tensorflow serving server (TSS) running on Ubuntu 22.04. TSS installed as system service on the dedicated server having MSI RTX3060 12 Gb on ...
Hakob Petrosyan's user avatar
1 vote
1 answer
585 views

Optimize batch transform inference on sagemaker

With current batch transform inference I see a lot of bottlenecks, Each input file can only have close to 1000 records Currently it is processing 2000/min records on 1 instance of ml.g4dn.12xlarge ...
user1657939's user avatar
0 votes
1 answer
68 views

SavedModel usage is just for Tensor serving or to retrain

Is this SavedModel just for Tensorflow front-end applications or it can be use to reload model in keras format. I created it using tf.saved_model.save and now I don't know what to make of it. ...
redflag's user avatar
0 votes
1 answer
77 views

Unable to run tensorflow serving on Ubuntu using docker

I followed all the steps listed here to use tensorflow serving https://www.tensorflow.org/tfx/serving/tensorboard used below command to run using docker sudo docker run -it --rm -p 8500:8500 -p 8502:...
utkarsh katiyar's user avatar
0 votes
0 answers
101 views

Keras model custom signatures are not saved when learning is done on GCP

After reading blog "An Object Oriented Approach To Train an Image Classifier with Tensorflow" I've tried to set custom signatures for my Keras model (TF2) like this: class ...
Patrick's user avatar
  • 2,597
1 vote
0 answers
37 views

Tiny single Bidirectional layer big RAM usage?

I have several different small models 1-2MB size on disk, some are just fully connected layers, Transformer, and a couple of Bidirectional GRU networks. But they have very different memory ...
Bob Bills's user avatar
  • 519
0 votes
0 answers
131 views

seldon-code tfserver grpc requests returns `StatusCode.UNIMPLEMENTED`

I have a tensorflow model .pb with the spec as below The given SavedModel SignatureDef contains the following input(s): inputs['Conv1_input'] tensor_info: dtype: DT_FLOAT shape: (-1, 28, ...
Sam's user avatar
  • 2,545
0 votes
1 answer
182 views

Inputs to TensorFlow serving REST API

I have build a seq2seq model by referring to this tutorial. https://keras.io/examples/nlp/lstm_seq2seq/ After training the model, instead of saving the model directly, I saved the encoder_model and ...
IronMan's user avatar
  • 23
0 votes
0 answers
51 views

InfraValidator component not "blessing" my model

The InfraValidator TFX component is not blessing my model, and I'm having trouble finding out why it isn't getting "blessed". I'm not sure what part of my model would be causing it to not be ...
Jon's user avatar
  • 1
0 votes
0 answers
54 views

Estimating model memory footprint in tensorflow serving

We want to measure the memory footprint of each model live in tensorflow serving container. We are using the open source tf container (https://hub.docker.com/r/tensorflow/serving) Tried scraping the ...
YetAnotherCoder's user avatar
1 vote
0 answers
163 views

How to get prediction labels with Tensorflow Serving Predict API?

I have a keras classification model saved remotely, and I use RESTful API to call the model on my site and make predictions. The JSON request is done as follows URL: POST https://my.site/models/v1:...
sound wave's user avatar
  • 3,191
0 votes
1 answer
268 views

Tensorflow v2.10 mutate output of signature function to be a map of label to results

I'm trying to save my model so that when called from tf-serving the output is: { "results": [ { "label1": x.xxxxx, "label2": x.xxxxx }, { "label1"...
ndtreviv's user avatar
  • 3,469
0 votes
0 answers
116 views

How to log tensorflow-serving docker container request and error logs?

I have deployed the tensorflow-serving docker image with the path to tf model. The REST api (/predict) is working as expected. But, the problem is , to use the model in production environment ...
Santhosh Shanmugam's user avatar
0 votes
0 answers
45 views

When to use spark with tensorflow compared to tf serving for inference?

I am currently facing a situation where I have a vast amount of images coming in from time to time and on which I will have to run some models for inferencing. To solve this scaling issue, I came ...
Shrijit Basak's user avatar
0 votes
1 answer
72 views

Migrating TensorFlow serving 1 to 2, RESSOURCE_EXHAUSTED from grpc calls

We are trying to migrate from /tensorflow/serving:1.12 to /tensorflow/serving:2.8.2, we use grpc calls to get the recommendation, we have two types of calls, one for each memeber where the grpc call ...
boukaibat yassine's user avatar
1 vote
1 answer
444 views

Does AWS Sagemaker supports gRPC prediction requests?

I deployed a Sagemaker's Tensorflow model from an estimator in local mode and when trying to call the Tensorflow Serving (TFS) predict endpoint using gRPC I get the error: grpc._channel....
theeDude's user avatar
0 votes
3 answers
402 views

Tensorflow serving failing with std::bad_alloc

I'm trying to run tensorflow-serving using docker compose (served model + microservice) but the tensorflow serving container fails with the error below and then restarts. microservice | To enable them ...
Robert Campbell's user avatar
1 vote
0 answers
192 views

TensorFlow Serving SSD Mobile Net not responding to gRPC inference requests

new poster here. I've downloaded the SSD MobileNet v2 320x320 from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md I have hosted it with a ...
Johnno1011's user avatar
2 votes
0 answers
130 views

Tensorflow SavedModel serving for What-If Tool Tensorboard

I have one CSV with around 10k rows and around 370 columns mostly numerical (int or float) and ID columns which is unique and I know the target column (integer type column) which needs to be used as ...
yashas123's user avatar
  • 270
1 vote
0 answers
210 views

GRPC request gives different response with multiple instances

I am trying to do model prediction through tensorflow serving through grpc connection and my main code runs with multiple instances. I send Predict request with the following code req = predict_pb2....
foxel's user avatar
  • 155

1
2 3 4 5
26