Questions tagged [tfx]
TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines
164
questions
0
votes
0
answers
9
views
How to pass a *serialized* tensor to a TensorFlow Model Serving server?
I implemented a simple TF model. The model received a serialized tensor of a gray image (simply a 2d ndarray), and restored it to a 2-d tensor. After then, some inference is applied on this 2-d tensor....
0
votes
0
answers
3
views
How to perform Rolling window in tensorflow-extended?
I am new to TFX. My doubts may seems trivial but it may help many people who want to implement TFX.
What are various ways to load the data for rolling window needed for time series? (My input source ...
0
votes
0
answers
11
views
TFX AutoEncoder model output in serve_tf_examples_fn
I am building a TFX pipeline that takes data input from CSV file and trains an autoencoder. The issue I am facing is that when I get output from model in serve_tf_examples_fn it is a tensor of shape [...
0
votes
0
answers
14
views
is TFX (Tensorflow Extended) being used in big companies
I've recently stumbled upon tensorflow extended, and I've finally started to understand the need and uses of it. It makes the whole machine learning pipeline a lot easier and looks like it can be used ...
0
votes
0
answers
29
views
How to customize tfx Evaluator component
We are developing a ML pipeline with TFX, the model is a Keras sequential DNN with mostly dense layers. We would like to customize the Evaluator component to compare the blessed and the candidate ...
0
votes
0
answers
39
views
How to use jupyter notebook in a local tfx docker image?
I have a windows laptop where I have WSL2 installed. In that I have installed docker and successfully run the docker installation of TFX.
I am not sure on how to customize the code and run it like we ...
2
votes
2
answers
50
views
Image processing in Tensor flow TFX pipelines
I am trying to get a Tensorflow TFX pipeline up and running using the MNIST dataset.
# Imports
import pandas as pd
import numpy as np
from keras.datasets import mnist
import tensorflow as tf
from tfx ...
0
votes
1
answer
39
views
Using String parameter for nvidia triton
I'm trying to deploy a simple model on the Triton Inference Server. It is loaded well but I'm having trouble formatting the input to do a proper inference request.
My model has a config.pbtxt set up ...
0
votes
0
answers
12
views
Filtering examples in Tensorflow Extended (TFX)
In the dataset I am working on, there are a lot of data points that I want to filter, e.g. contain nan values, out-of-bound values, etc. I also want to do the same filtering on the data points in the ...
0
votes
1
answer
28
views
TFX - What is example_gen_pb2 and where is it documented?
The ExampleGen TFX Pipeline Component has the example code which uses example_gen_pb2 but what is it and where is it documented?
# Input has a single split 'input_dir/*'.
# Output 2 splits: train:eval=...
0
votes
1
answer
15
views
TFX - How to inspect records from CsvExampleGen
Question
How to inspect the data loaded into TFX CsvExampleGen?
CSV
Top 3 rows from the california_housing_train.csv looks below.
longitude
latitude
housing_median_age
total_rooms
total_bedrooms
...
0
votes
0
answers
17
views
TFX - Bus error (core dumped) at importing CsvExampleGen
Question
How to work-around or solve the problem.
Environment
$ uname -a
Linux ubuntu 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ python3 --...
0
votes
0
answers
35
views
TFX ExampleGen from CSV file - wrong split
I'm trying to import a CSV file into a TFX pipeline but I see a strange behaviour I don't understand.
Here's the file I'm trying to import:
X,XSQUARE
1,2
2,4
3,9
4,16
5,25
1,2
2,4
3,9
4,16
5,25
Here'...
2
votes
1
answer
44
views
Do TFX pipelines support training on TPUs?
I am new to TFX and I would like to know if it's possible to create a TFX pipeline that can train Keras models on TPUs nodes using TPUStrategy from a TPUClusterResolver. Looking at TFX documentation, ...
0
votes
0
answers
36
views
How do you feed Ragged Tensors to a DNN trained by TensorFlow Extended?
We are developing a ML pipeline with TFX, with the most common components such as ExampleGen, Transform, Trainer, and so on.
The examples that have to be fed to the DNN have varying length, so we ...
1
vote
0
answers
47
views
Add reserved tokens to `tft.vocabulary`
I would like to append words to the vocabulary created by tft.vocabulary that are not a part of the training samples (i.e. <mask> and <pad> tokens).
I see in the docs that the tft....
0
votes
1
answer
75
views
What does DataAccessor do in tfx?
I'm reading the tfx tutorials, which all uses the DataAccessor to load data. The code looks something like this:
return data_accessor.tf_dataset_factory(
file_pattern,
tfxio....
0
votes
1
answer
30
views
join datasets with tfx tensorflow transform
I am trying to replicate some data preprocessing that I have done in pandas into tensorflow transform.
I have a few CSV files, which I joined and aggregated with pandas to produce a training dataset. ...
0
votes
1
answer
164
views
Is there an implemented way to use a kubeflow pipeline's output outside the pipeline?
I'm using local kubeflow pipelines for building a continuous machine learning test project. I have one pipeline that preprocess the data using TFX, and it saves the outputs automatically to minio. ...
1
vote
0
answers
26
views
what is the v1 folder about in TFX
I'm new to TFX and a bit confused about what the v1 about, and what is its relationship with the code outside v1?
Do I understand correctly that the official APIs are in v1, meaning I should always do
...
2
votes
0
answers
144
views
TensorFlow Extended installing error with some conficts
I need help installing tfx, I just ran this command:
pip install -U tfx
But I get this error:
ERROR: Cannot install tfx==0.13.0, tfx==0.14.0, tfx==0.15.0, tfx==0.21.0, tfx==0.21.1, tfx==0.21.2, tfx==...
0
votes
0
answers
70
views
Loading Custom Image Dataset with TFX as TFRecord
I'm trying to use TFX on a custom dataset [Images/labels]
So I'm using ImportExampleGen with .TFRecord file as follows
example_gen = ImportExampleGen(input_base=TFRecord_DIR_PATH)
context.run(...
1
vote
0
answers
77
views
Customize Airflow / TFX Pipeline to upload Files in S3
I am currently trying to customize a tfx pipeline, which is being orchestrated by Airflow.
Basically I want to upload all generated files by the DAG to a S3 bucket storage after the run of the ...
0
votes
1
answer
127
views
How to get vocabulary size in tensorflow_transform before apply_vocabulary?
Also posted the question at https://github.com/tensorflow/transform/issues/261
I am using tft in TFX and needs to transform string list class labels into multi-hot indicators inside preprocesing_fn. ...
1
vote
0
answers
55
views
How can I use BigQuery in a standalone tensorflow transform (TFT) pipeline?
I'm interested in interactive development of a preprocessing_fn for tft.AnalyzeAndTransformDataSet. By interactive development, I mean running a standalone beam pipeline in a Jupyter Notebook and ...
0
votes
0
answers
55
views
local TFX pipeline run create ERROR Failed to make stateful working dir ; Protocol error
I am following Building a TFX Pipeline Locally (https://www.tensorflow.org/tfx/guide/build_local_pipeline) on ubuntu 21.04. I am only running the CsvExampleGen component and I am getting the following ...
0
votes
0
answers
46
views
tensorflow-model-analysis: Using the EvalConfig protobuf without importing tensorflow
I am writing an API which configures a tfma EvalConfig protobuf object and sends it downstream for use in a TFX pipeline. Currently this API runs
from tensorflow_model_analysis import EvalConfig
...
0
votes
1
answer
136
views
tfx extension create failed while running npm script
I'm using this package to compile and build my extension
{
"id": "extensionAzureDevOps",
"name": "extension",
"publisher": "ME",
&...
0
votes
2
answers
122
views
Equivalent of TFX Standard Components in KubeFlow
I have an existing TFX pipeline here that I want to rewrite using the KubeFlow Pipelines SDK.
The existing pipeline is using many TFX Standard Components such as ExampleValidator. When checking the ...
3
votes
1
answer
367
views
Specify signature name on Vertex AI Predict
I've deployed a tensorflow model in vertex AI platform using TFX Pipelines. The model have custom serving signatures but I'm strugling to specify the signature when I'm predicting.
I've the exact same ...
0
votes
0
answers
79
views
What to pass into Tensorflow parse_example()
I'm completely lost.
I'm trying to make a data pipeline in Airflow for object detection.
My images are 640x640x3 images.
I have preprocessing_fn that results in a single tensor dictionary:
"image/...
1
vote
1
answer
110
views
How to predict with Bulkinferrer from a previously pushed model in Tensorflow Extended (TFX)
In the TFX pipelines, how do we make use of BulkInferrer?
It is quite standard to connect the BulkInferrer with a trained model or pushed_model. However, what if I don't want to train the model again, ...
-1
votes
1
answer
41
views
Displaying TensorflowExtended/TFX results of visualize_statistics
I'm trying to display the output of visualize_statistics like below but the results are pretty disappointing:
train_stats = tfdv.generate_statistics_from_tfrecord(data_location='data/tf_records/train....
1
vote
0
answers
62
views
How do I find the length of a vocabulary computed during TFX Transform?
I'm currently building a project in TFX and during the Transform step I compute the "vocabulary" for a categorical variable. For later steps (but still during preprocessing), I want to use ...
0
votes
0
answers
49
views
How to write an aggregation function inside the Transform component of TFX?
I have datetime feature, cluster number and coordinates in my tfrecord file. How to write a function inside the transform component of TFX to count the records for each cluster number based on a ...
0
votes
1
answer
571
views
TFX Tensorflow model validator component - You passed a data dictionary with keys ['image_raw_xf']. Expected the following keys: ['input_1']
I'm building a tfx pipeline based on the cifar10 example : [https://github.com/tensorflow/tfx/tree/master/tfx/examples/cifar10]
The difference is that I don't want to convert it to tf_lite model and ...
0
votes
1
answer
549
views
Kubeflow, passing Python dataframe across components?
I am writing a Kubeflow component which reads an input query and creates a dataframe, roughly as:
from kfp.v2.dsl import component
@component(...)
def read_and_write():
# read the input query
...
1
vote
1
answer
67
views
Disabling useless logs/ouputs from TFX/setuptools
I have been working with TensorFlow Extended (TFX) for about a month and a half now and there has always been one thing that really bothers me about it. The logging and mountain of stdout outputs from ...
0
votes
0
answers
57
views
Processing variable-length feature in TensorflowExtended (TFX)
I am working with TFX Transform and facing a problem with variable-length array.
My problem statement is like this: I have a few, let's say, "groups" of data. The number of example in those ...
0
votes
2
answers
252
views
Beam + Flink: No parallelism when using SDFBoundedSourceReader
Background: I am using TFX pipelines with Flink as the runner for Beam (flink session cluster using flink-on-k8s-operator). The Flink cluster has 2 taskmanagers with 16 cores each, and parallelism is ...
0
votes
1
answer
252
views
Delete a column from TFRecord Dataset (for feature selection)
I am trying to implement a Feature Selection component with the following plan in mind:
The implementation
Component takes and InputArtifact[Example] as input
Since the data is stored in the form of ...
0
votes
1
answer
196
views
Why pod on GKE cluster is OOMkilled when trying to run a very simple Kubeflow pipeline using TFX?
I'm following the TFX on Cloud AI Platform Pipelines tutorial to implement a Kubeflow orchestrated pipeline on Google Cloud. The main difference is that I'm trying to implement an Object Detection ...
0
votes
0
answers
150
views
Understand pipeline's signatures and ConcreteFunction signature_wrapper in Tensorflow Extended (TFX)
I am working with TFX. I'd like to understand output of the following cell:
ppl = pipelines['my_pipeline_name']
print(ppl)
print(ppl.signatures)
print(ppl.signatures['serving_production'])
which is
&...
0
votes
1
answer
355
views
How do you fit a tf.Dataset to a Keras Autoencoder Model when the Dataset has been generated using TFX?
Problem
As the title suggests I have been trying to create a pipeline for training an Autoencoder model using TFX. The problem I'm having is fitting the tf.Dataset returned by the DataAccessor....
2
votes
0
answers
169
views
TFX runs out of memory on 32GB of RAM with only a 232.9MB TFRecord file. Is there something wrong with my code?
I am trying to train a model that detects whether someone is using sunglasses using tfx and a subset of the celebsA dataset (~ 26k images). I have written the images and labels to a TFrecord that is ...
1
vote
0
answers
134
views
ExampleGen on production
I was wondering how is ExampleGen used in production? I understand that their outputs can be feeded into the TFDV components of TFX to validate schema, skews, and others.
But I get lost since ...
0
votes
1
answer
72
views
Tensorflow Extended (TFX): Is there an easy way to debug functions from Transorm component?
I am supposed to modify a function which is a part of Transorm component. It is a long series of tensorflow operations and I am not sure a. how particular steps affect processed variables b. what does ...
0
votes
0
answers
40
views
'MaxRetryError' while implementing Kubeflow pipeline using TFX
I am trying to implement a Kubeflow pipeline using TFX and pipeline was initially failing due to problems with the dataset. During the process of fixing that, the pipeline ran into a MaxRetryError ...
0
votes
0
answers
20
views
Post-process model prediction before passing to Tensorflow Model Analysis
Is it possible to do post processing on the model prediction before passing it into tfma.run_model_analysis? As far as I've seen on the tutorials there is no way to do this, since it only accepts a ...
0
votes
0
answers
119
views
tfx.orchestration.experimental has no attribute 'get_default_kubeflow_metadata_config'
I am trying to orchestrate a TFX pipeline using kubeflow and I am getting the following error during the creation of the pipeline.
AttributeError: module 'tfx.orchestration.experimental' has no ...