Questions tagged [tensorflow-serving]
The tensorflow-serving tag has no usage guidance, but it has a tag wiki.
1,226
questions
1
vote
1
answer
21
views
How to pass google cloud application credentials file to docker container
I would like to pass my Google Cloud Platform's service account JSON credentials file to a docker container so that the container can access a cloud storage bucket. So far I tried to pass the file as ...
0
votes
1
answer
9
views
Stuck at training model with CPU
As the example points out:
docker run -it -p 8500:8500 --gpus all tensorflow/serving:latest-devel
should train the mnist mode, however I want to use intel cpu for training, not gpu. But no luck, it ...
0
votes
1
answer
22
views
How to send an image that was sent with a post request to a model for prediction
I'm running a flask app that receives a base64 encoded image through a post request and should send it for classification. Unfortunately, this is where I come across an error. This is my code for ...
0
votes
0
answers
11
views
Tensorflow op: ParseExampleV2 is super slow
When using native tensorflow for training and prediction/serving, we found that the bottle neck of training/serving speed is on the op of ParseExampleV2 , i.e. the op of transforming a vector of tf....
0
votes
1
answer
28
views
How to user a tensorflow server saved model with string tensor input to predict in the local machine?
I'm trying to run saved serving models in my local machine. However, it takes string tensor as input, and I'm having trouble converting the images to the correct string format.
To load the model I use:...
1
vote
0
answers
33
views
Saving a TF model with crossed features
I have a trained model where I cross every feature with a cross feature, and encode all the features using hash encoding. Here's a snippet of how I have implemented the encoding for the features:
...
0
votes
0
answers
16
views
How can I call a model that is running in tensorflow serving with docker from ajax?
should I first call a rest api built with fastapi or flask or is it possible to make a call directly from ajax to tensorflow serving?
thanks
1
vote
0
answers
42
views
(Tensorflow 1.14) Not able to convert saved_model.pb and variables in to frozen_inference_graph.pb
I am using tensorflow=1.14. During periodic evaluation, I save best model using tf.estimator.BestExporter. I have following questions.
1)
When I try to convert saved_model.pb stored by BestExporter ...
1
vote
0
answers
38
views
Postprocessing with Sklearn in Tensorflow Serving
I developed a classificator using Tensorflow. The class labels are One-Hot Encoded using
sklearn.preprocessing.LabelBinarizer()
Wrapping the whole application into a Flask-App worked totally fine, ...
0
votes
1
answer
32
views
How to use trained model in Tensorflow Serving?
I have trained Mask-RCNN model, and want to try Tensorflow Serving to use it in web.
So i just run tensorflow/serving docker container. And realised what i have no clue how to send image to predict. ...
0
votes
0
answers
18
views
How to serve Generative Language Models via TensorFlow Serving
Generative Language Models like GPT, seq2seq or chatbots based on transformers have their decoder layers and its inputs.
The model generates words until they meet [EOS] tokens by repeating feeding ...
0
votes
0
answers
15
views
How to create a function that returns outputs from tensorflow 1.x model given input?
It might look like a silly question but I have only worked with tf 2.x with eager execution so I have no idea about functionality of tf 1.x. When I run the code twice, (code given below works fine ...
0
votes
0
answers
40
views
Tensorflow Serving: Still getting No versions of servable <MODEL> found under base path despite versioning model
I am trying to deploy a tensorflow model using docker. My directory structure is as follows: serving->kcb_models->1->(assets folder + variables folder + saved_model.pb + keras_metadata.pb). ...
0
votes
0
answers
21
views
TensorFlow Serving Custom Metadata
I am looking to have custom metadata(Json/Dict) in the tensorflow Saved Model which will be deployed in tensorflow serving. I want to fetch the metadata using the api and validate the data being input ...
0
votes
1
answer
32
views
Tensorflow Core or js ? Discuss about pros and cons architecture
So, we want to build an application for some machine learning stuff and we are having trouble to define which of the API we should choose so I would be very happy if we could discuss the pros and cons ...
0
votes
0
answers
49
views
TensorFlow Servering returns incorrect results
I have 2 different models (Model A is Keras .h5 and Model B is Torch .pth). They need to be served with TFServing. I converted both of these models to Tensorflow (with index .pb) for serving.
I ...
1
vote
1
answer
170
views
Error Running TensorFlow Serving from Dockerfile
I am working on a container to run TensorFlow Serving.
Here is my Dockerfile:
FROM tensorflow/serving:latest
WORKDIR /
COPY models.config /models/models.config
COPY models/mnist /models/mnist
Here ...
0
votes
1
answer
58
views
Tensorflow Serving connection aborts without response
I have a basic tensorflow serving docker container exposing a model on a kubernetes pod.
FROM tensorflow/serving:2.6.0
RUN mkdir /serving_model
WORKDIR /serving_model
COPY src/serving_model /...
0
votes
1
answer
62
views
'Doc2Vec' object has no attribute 'outputs', while saving doc2vec for tensorflow serving
I have been trying to save a movie recommendation model from github to then serve using tf-serving.
The code below will first create a list of taggs from my corpus and then provide me vectors based on ...
0
votes
0
answers
17
views
Tf serving multiple requests but return single output
I'm using tf-serving version 1.0.0.5(grpc protocol) to serve a customized tf2.0 model,
my request in json looks like this:
{'task1':{"feature1":0.0,"feature2":0.0, "feature3&...
0
votes
0
answers
97
views
Building tensorflow serving version 2.4.1 from source
I am attempting to build tensorflow-serving from source using the official documentation but I am running into problems with missing dependencies or errors concerning bash.
When trying the following ...
1
vote
0
answers
34
views
How to support grpc stream request on tensorflow serving
I have a tts model. I want input stream data to tensorflow-serving.
So , I get stream audio data from stream rpc interface.
What I should to do?
0
votes
0
answers
21
views
Does TensorFlow Serving supports inference in compiled runtimes?
Is it possible to serve models optimized by inference compilers (ONNXRuntime, TVM or TensorRT in particular) in TF Serving? Or TF Serving can serve only SavedModels created by TensorFlow, in a ...
0
votes
0
answers
160
views
Skipping loop optimisation error in tensorflow_serving
I am getting the following message when testing a model with tensorflow_serving,
can u please help me debug and understand what it means
2022-02-13 23:20:52.460513: I tensorflow_serving/model_servers/...
0
votes
0
answers
30
views
error while i deploy yolov5 odel using tensoflow serving
I use yolov5 6.0's export.py to saved_model, but when i test it by restapi
def test_http():
parser = argparse.ArgumentParser()
parser.add_argument("--img_path", default=ROOT / 'data/images/...
0
votes
1
answer
49
views
Grappler optimization failed. Error: Op type not registered 'FusedBatchNormV3' in Tensorflow Serving
I am serving a model using Tensorflow Serving.
TensorFlow ModelServer: 1.13.0-rc1+dev.sha.fd92d2f
TensorFlow Library: 1.13.0-rc1
I sanity tested with load_model and predict(...) in notebook and it is ...
0
votes
0
answers
28
views
compile a specific version of tensorflow serving using Docker
I'd like to compile TF Serving v2.6.3 and create a Docker image (yes, I know there's a pre-built image on Dockerhub, but still I'd like to compile it by myself).
On the TFS documentation, they refer ...
0
votes
0
answers
132
views
HealthCheck with TensorFlow Serving
We are setting up a fleet/cluster of Tensorflow Serving (TFS) instances with ECS/FarGate (using a Docker image with TFS). The instances will be served by a load balancer (AWS ALB) which needs to ...
0
votes
0
answers
9
views
How to expand Tensorboard from localhost to our internal web application portal?
I'm working on a few models where tensorboard visualizations have been very useful to our team so far. My team requested me to find a way to show embed these visualisations on our internal web ...
0
votes
0
answers
65
views
Tensorflow Serving keeps returning the same output
So, I'm following this tutorial: https://www.youtube.com/watch?v=t6NI0u_lgNo&t=1826s
and right after the tensorflow serving part I had been testing my fastapi API code which looks like this:
from ...
0
votes
0
answers
142
views
same model served via TF and TF-serving produce different predictions
SETUP: a TF model that uses Keras as backend and being saved as 2 files: model.h5 and architechture.json.
GOAL: predict using TF-serving
PROBLEM: prediction values vary in direct call and call to TF-...
1
vote
0
answers
204
views
Using S3 bucket for SavedModel with tensorflow/serving:2.7.0-gpu Docker image
I'm trying to upgrade the Docker base image that we build on top of, for our TF-serving server. Currently, we use tensorflow/serving:2.3.0-gpu as the base image. I'm now trying out tensorflow/serving:...
0
votes
0
answers
59
views
Tensorflow Serving , GRPC Variable shape value
I have this serving working which has a model with a dynamic input for one field with the shape [1,?,32,3]. I have been trying to get that working with no luck.
I tried this
TensorShapeProto.Dim....
0
votes
1
answer
55
views
AWS Sagemaker Tensorflow Serving Error: Tensor name: <> has no shape information
When I am preparing a tensorflow model on my system and loading the saved model for prediction, I am getting prediction output
from tensorflow.contrib import predictor
loaded = predictor....
0
votes
1
answer
48
views
tf.math.top_k not accepting integers {"error": "k must be scalar, got shape [1]\n\t [[{{node TopKV2}}]]"}
Here is my code-
import tensorflow as tf
SAVE_MODEL_PATH = "test/1"
class GetTopK(tf.Module):
def __init__(self):
self.wow = "wowness!"
@tf.function(...
0
votes
1
answer
32
views
Crop the image using tensoflow Imagegenerator preprocessing fucntion by opencv
i create a function for croping eye image. because outer side of eye image is balack and unnecessary pixel .but if i using this function for loading image in opencv ,for large resolution ,allocate ...
0
votes
2
answers
96
views
Not able to run linux command in background from dockerfile?
Here's my docker file,
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt update && apt upgrade -y
RUN apt install -y -q software-properties-common
RUN apt install -y -q build-...
0
votes
0
answers
26
views
Serving caffe models
How to Serving caffe models?
I'm doing a project on image classification using pre trained caffe model, I want to serve my model like TFServing,How to Serving caffe models?
0
votes
0
answers
74
views
Input data format for SSD EfficientDet D3 Tensorflow Serving API
I trained on my custom dataset with the pretrained model of efficientdet_d3_coco17_tpu-32 from TensorFlow Object Detection API and exported the frozen graph using the exporter_main_v2.py.
I deployed ...
0
votes
0
answers
36
views
I have a keras model with conv1D and before that some label encoder, one hot encoder and tokenizer. I want to save the model using pipeline and use
Steps which I am following,
Labelencoder
one hot encoding using tf.keras.utils.to_categorical
keras pp text Tokenizer --> tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)
...
0
votes
1
answer
133
views
Tensorflow Serving on docker using tflite file
I was trying to deploy ML model with the Tensorflow Serving. Even though I added sharing file path on the docker. This error keep comes out somehow. And I am working with a tflite file to deploy it, ...
1
vote
0
answers
79
views
Saving a model that uses tensorflow.lookup.StaticVocabularyTable in .pb format in Tensorflow 2
I am building a model that accepts as input a 2d array of string tokens, then uses a lookup table to get the assigned indices of the input tokens in the vocabulary. The model then uses those indices ...
0
votes
0
answers
23
views
Batching gives random unexpected behavior on Tensorflow Serving
I have exported a Keras model to TFServing and when I use it with TFServing it firstly worked with batch size 1, i.e. with shape (1, 250, 38) and later on with batch 4, i.e. (4, 250, 38). I also tried ...
0
votes
0
answers
26
views
TF serving loads up to 7 models on Windows Server 2019 Datacenter
I'm running docker on Windows Server 2019 Datacenter. I've managed to run tf serving on it with my own models. The problem is when I'm trying to serve more than 7 models. It seems like the models can'...
0
votes
0
answers
38
views
How to predict model with a bert model trained with TFrecords
I use google bert offical code(run_classifer.py) to train a classification model and saved a ckpt model(checkpoint and 3 ckpt files). The official code use tfrecords to feed the input data rather than ...
1
vote
0
answers
49
views
Utilize TensorFlow Serving directly in a python process as a library
I would like to run tensorflow serving without the HTTP or gRPC interfaces. In fact, I don't want it to even bind a port for receiving client requests. I'd like to leverage it as a library directly ...
1
vote
0
answers
67
views
Logging predict request/response in tensorflow serving
Currently using TensorFlow serving via docker to deploy an object detection model. Is there a way to log all predict requests/responses so I can save and view them locally for debugging? Input image ...
0
votes
0
answers
39
views
tensorflow serving compile Error :absl lib not defined
when i using centos 7 compile tensorflow_serving 2.5.1,something maybe wrong.
I using this command to compile tensorflow_serving:
bazel build --color=yes --curses=yes --verbose_failures --...
1
vote
1
answer
109
views
How to predict with Bulkinferrer from a previously pushed model in Tensorflow Extended (TFX)
In the TFX pipelines, how do we make use of BulkInferrer?
It is quite standard to connect the BulkInferrer with a trained model or pushed_model. However, what if I don't want to train the model again, ...
0
votes
0
answers
72
views
Tensorflow serving: Cannot get the right input format
I want to serve an MNIST model(hand written digits) and generate predictions using REST endpoints.
The server is up and running and I can get gRPC based predictions successfully. REST based prediction ...