Questions tagged [tfx]
TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines
189
questions
0
votes
0
answers
20
views
TFX pipeline-root not found
im currently trying to configurate a tfx pipeline with components such as SchemaGen,StatisticGen and Transform. Providing the statistics-, and schema-gen components and running them in a tfx-workflow ...
0
votes
0
answers
17
views
TFX TypeError: Argument input_params should be a Channel of type <class 'tfx.types.standard_artifacts.ExternalArtifact'> (got test_string)
I updated the TFX from 0.22.0 to 1.8.0 and now I'm getting this error:
TypeError: Argument input_params should be a Channel of type <class 'tfx.types.standard_artifacts.ExternalArtifact'> (got ...
0
votes
1
answer
6
views
AttributeError: module 'tfx.utils.io_utils' has no attribute 'file_io'
I updated the tfx from 0.22.0 to 1.8.0 and now I'm getting this error:
AttributeError: module 'tfx.utils.io_utils' has no attribute 'file_io'
The code is:
from tfx.utils import io_utils
io_utils....
0
votes
0
answers
19
views
How to configure optional component with TFX?
I would like to run one optional custom component with TFX and Airflow. Is it possible?
This optional component is used to send updates to another system, if this optional component fails, it will not ...
0
votes
0
answers
18
views
How to Run a TFX Orchestration Pipeline Outside Jupyter?
Learning data ingestion and prediction orchestration with tensorflow from this notebook on github. It is using an InteractiveContext on the ipython environment. But obviously a production system does ...
0
votes
0
answers
17
views
TFX CSVExampleGen component: How to read data with "|" as separator?
I am trying my way at TensorFlow's TFX. I have a CSV that contains the pipe symbol (|) as a field separator instead of the default comma. How can I specify this when using the CsvExampleGen component?
...
0
votes
0
answers
25
views
Developer workflow for tensorflow/tfx
I'm trying to improve my workflow for supervised learning in the tf ecosystem, and was wondering if anyone has encountered/solved this problem.
I'm able to get a fully functional pipeline to execute ...
0
votes
0
answers
16
views
I don't understand how the Trainer component works
As far as I understand, we need to supply the Trainer component training examples directly (with the "examples" parameter) from the output of an ExampleGen or a Transform component, but at ...
0
votes
0
answers
15
views
Is there a way to save a TFX DatasetFeatureStatisticsList?
Saving a TFX DatasetFeatureStatisticsList to disk (binary)
After computing data statistics from CSV files with tfdv.generate_statistics_from_csv I can't find a good way to save it other than using ...
0
votes
0
answers
51
views
How can I make tfx use tensorflow-cpu rather than the full tensorflow package?
I am trying to build a lightweight docker image with tfx library installed, as the officially released image is pretty heavy in size (~9 GBs).
Here is the code that I am using for the image now:
FROM ...
0
votes
1
answer
79
views
TFX's Evaluator Component cannot prepare the inputs for evaluation
I am using tfx pipeline for training and evaluating an autoencoder. The data that I have is basically 5 arrays of size (15,1) that I concatenate and put together and pass to the model.
In order to ...
0
votes
1
answer
38
views
Tensorflow: How to add a property in execution object in MLMD MetadataStore?
I'm using the MLMD MetadataStore to manage the data pipelines and I need to add an execution property in MLMD to get this property later.
I'm trying add with this:
from ml_metadata.proto import ...
0
votes
1
answer
39
views
TFX docker image is heavy
I was just wondering why tensorflow/tfx image is 8Gb while gpu tensorflow/tensorflow image is less than 3gB?
At runtime, I expect tfx container to just load input artifacts, run some tensorflow code ...
0
votes
0
answers
47
views
Tensorflow Transform Fails Converting Strings to Numbers
I am trying to fit a TensorFlow model and one my features comes in as a comma-separated string of ints (possibly empty string). The feature appears in the pretransform schema as
feature {
name: &...
0
votes
1
answer
64
views
Impossible to get post transform statistics by split
I'm running a simple penguin pipeline in interactive mode with a split train/eval, the transform step run but i can't get post_transform_statistics artifacts.
Inside the dedicated artifacts folder /...
1
vote
1
answer
59
views
What does tfma.metrics.MeanLabel do?
I am interested in what tfma.metrics.MeanLabe does, and how it should be used, and what the difference is between tfma.metrics.MeanLabe and tfma.metrics.MeanPredictio and tfma.metrics.MeanAttributions....
0
votes
0
answers
18
views
need help compiling a folder named "build"
need help compiling the build folder
Inside of the build folder is a bunch of json files and .tfx.
Expecting a single file to come out that i can ./ the file
0
votes
1
answer
45
views
str() is not usable anymore to get true value of a Text tfx.data_types.RuntimeParameter during pipeline execution
how to get string as true value of tfx.orchestration.data_types.RuntimeParameter during execution pipeline?
Hi,
I'm defining a runtime parameter like data_root = tfx.orchestration.data_types....
0
votes
1
answer
69
views
tfx.components.StatisticsGen display train and eval in two different figures, is it possible to have them in a single figure as tfdv does?
a superimposed display for train/val splits using StatisticsGen
Hi,
I'm currently using tfx pipeline inside kubeflow. I struggle to have StatisticsGen showing a single graph with train and validation ...
0
votes
0
answers
78
views
Having NoneType error in multiple pipeline build using TFX and Airflow
I am working with TFX and AIrflow to build the pipeline for image classification and segmentation. I am replicating the repo by Jan Marcel Kezmann (Link). While executing the classification_dag.py in ...
0
votes
1
answer
426
views
Dealing with missing values in tensorflow
I need some guidance on the approach to imputation in tensorflow/deep learning. I am familiar with how scikit-learn handles imputation, and when I map it to the tensorflow ecosystem, I would expect ...
0
votes
1
answer
136
views
How to visualize a saved statistics artifact?
I know of two ways to run a TFX pipeline. First, using a Jupyter notebook with InteractiveContext in a browser:
from tfx import v1 as tfx
from tfx.orchestration.experimental.interactive....
0
votes
0
answers
105
views
I am struggling to write a transform python file for the tfx pipeline in transform component. I am working with object detection task
I am building a pipeline using tensorflow extended. The dataset is already conversted to the tfrecord for the ingestion. I want to do data preprocessing such as scalin, data augmentation along with ...
0
votes
1
answer
177
views
How to get the uri of the current pipeline's artifact
Consider the following pipeline:
example_gen = tfx.components.ImportExampleGen(input_base=_dataset_folder)
statistics_gen = tfx.components.StatisticsGen(examples=example_gen.outputs['examples'])
...
0
votes
0
answers
47
views
How to write a preprocessing_fn function that relies on an Artifact?
Let's consider a very straightforward pipeline like this:
example_gen = tfx.components.ImportExampleGen(input_base=_dataset_folder)
statistics_gen = tfx.components.StatisticsGen(examples=example_gen....
1
vote
0
answers
86
views
How to get the list of features along side their schema and stats using TFX
Let's say that I have the following, very straightforward pipeline:
import os
from tfx import v1 as tfx
_dataset_folder = './tfrecords/train/*'
_pipeline_data_folder = './pipeline_data'
...
0
votes
2
answers
70
views
MLOps with TFX: How to ingest data when using Sequence from Keras?
I am using a class called DataGenerator, that returns a tuple (data_array, label_array), follows the code:
from tensorflow.keras.utils import Sequence
class DataGenerator(Sequence):
""&...
0
votes
1
answer
757
views
A TFX equivalent for Pytorch [closed]
I'v recently worked with TensorFlow Extended(TFX) platform. Since my development background is on Pytorch stack, I'm looking for well-compatible alternatives of TFX for Pytorch.
While searching for ...
0
votes
1
answer
83
views
How to split the dataset into mutiple folds while keeping the ratio of an attribute fixed
Let's say that I have a dataset with multiple input features and one single output. For the sake of simplicity, let's say the output is binary. Either zero or one.
I want to split this dataset into k ...
0
votes
0
answers
54
views
InfraValidator component not "blessing" my model
The InfraValidator TFX component is not blessing my model, and I'm having trouble finding out why it isn't getting "blessed". I'm not sure what part of my model would be causing it to not be ...
1
vote
0
answers
181
views
How to get prediction labels with Tensorflow Serving Predict API?
I have a keras classification model saved remotely, and I use RESTful API to call the model on my site and make predictions. The JSON request is done as follows
URL:
POST https://my.site/models/v1:...
0
votes
1
answer
106
views
How to use the same code for machine learning "data transformations" before prediction in both reshearch & production
I wonder what is the best practice/s of taking for example a jupiyter notebook that contains the whole flow
from eda to prediction and use the same code for "data transformations from the raw ...
0
votes
2
answers
92
views
Why tfx.components.FnArgs does not have an "epochs" attribute?
tfx.components.FnArgs is the way to pass values to the run_fn function which will in turn train the model in Tensorflow Extended pipeline.
Looking at the tfx.components.FnArgs' documentation, I cannot ...
0
votes
1
answer
68
views
Why ImportExampleGen reads TFRecords as SparseTensor instead of Tensor?
I'm converting a CSV file into a TFRecords file like this:
File: ./dataset/csv/file.csv
feature_1, feture_2, output
1, 1, 1
2, 2, 2
3, 3, 3
import tensorflow as tf
import csv
import os
print(tf....
2
votes
3
answers
575
views
Why we need TFX if we have airflow for orchestration
I still don't get why we need TFX. TFX will convert your defined pipeline to Airflow DAG and run it on airflow, I could just write my pipelines in python and use Airflow's PythonOperator to build a ...
0
votes
1
answer
192
views
RuntimeError: Split pattern data/* does not match any files
I'm currently trying to convert the TFX-TFRS tutorial into an Airflow pipeline.
When I run this Pipeline with LocalDagRunner() it executes without error. When I use the AirflowDagRunner() it throws ...
1
vote
1
answer
371
views
ml_metadata.errors.AlreadyExistsError: Given node already exists
I ran into a problem using TFX, MLMD, and Apache-Airflow as the orchestrator. Local-dag-runner, provided by TFX, works fine, resulting in distinct artifacts for each pipeline component run. The ...
0
votes
1
answer
61
views
TFX tutorial "penguin_tfdv.ipynb" issue TypeCheckError
I've got this problem. How could I solve this problem?
This is part of the tutorial of TFX. link to the tutorial
I changed nothing in the example code.
TypeCheckError: Output type hint violation at ...
0
votes
2
answers
378
views
TFX results in TypeCheckError: Output type hint violation at WriteStatsOutput[train]: expected <class 'apache_beam.pvalue.PDone'>, got <class 'str'>
I tried running the tutorial provided in tfx documentation and I am facing a problem with the path to metadata.
I have described the query in detail in this issue https://github.com/tensorflow/tfx/...
0
votes
1
answer
535
views
Vertex workbench - how to run BigQueryExampleGen in Jupyter notebook
Problem
Tried to run BigQueryExampleGen in the
InvalidUserInputError: Request missing required parameter projectId [while running 'InputToRecord/QueryTable/ReadFromBigQuery/Read/SDFBoundedSourceReader/...
0
votes
1
answer
116
views
TF Serving Predict API Output Interpretation
Is the TensorFlow Serving (TFS) Predict API output the same as the tf.keras.model.predict method (i.e. the outputs of the model according to the compiled metrics)?
For example, if we have a tf.keras....
0
votes
1
answer
109
views
Issue with `TransformFeatureLayer` when using `custom_config` in `preprocessing_fn`
System information
Environment: Linux 5.13.0-41-generic #46~20.04.1-Ubuntu
TensorFlow version: 2.8.0
TFX Version: 1.8.0
Python version: 3.8.13
Python dependencies:
absl-py 1.0....
0
votes
1
answer
27
views
TFX AutoML on local System
Is it possible to perform AutoML using TFX on local environment. I know with kubeflow and GCP we can perform AutoML on the cloud, but I want to do all of these steps locally and deploy the model to a ...
1
vote
0
answers
226
views
How to pass a *serialized* tensor to a TensorFlow Model Serving server?
I implemented a simple TF model. The model received a serialized tensor of a gray image (simply a 2d ndarray), and restored it to a 2-d tensor. After then, some inference is applied on this 2-d tensor....
0
votes
1
answer
456
views
is TFX (Tensorflow Extended) being used in big companies
I've recently stumbled upon tensorflow extended, and I've finally started to understand the need and uses of it. It makes the whole machine learning pipeline a lot easier and looks like it can be used ...
0
votes
0
answers
213
views
How to customize tfx Evaluator component
We are developing a ML pipeline with TFX, the model is a Keras sequential DNN with mostly dense layers. We would like to customize the Evaluator component to compare the blessed and the candidate ...
0
votes
0
answers
176
views
How to use jupyter notebook in a local tfx docker image?
I have a windows laptop where I have WSL2 installed. In that I have installed docker and successfully run the docker installation of TFX.
I am not sure on how to customize the code and run it like we ...
2
votes
2
answers
573
views
Image processing in Tensor flow TFX pipelines
I am trying to get a Tensorflow TFX pipeline up and running using the MNIST dataset.
# Imports
import pandas as pd
import numpy as np
from keras.datasets import mnist
import tensorflow as tf
from tfx ...
2
votes
2
answers
3k
views
Using String parameter for nvidia triton
I'm trying to deploy a simple model on the Triton Inference Server. It is loaded well but I'm having trouble formatting the input to do a proper inference request.
My model has a config.pbtxt set up ...
1
vote
2
answers
438
views
TFX - What is example_gen_pb2 and where is it documented?
The ExampleGen TFX Pipeline Component has the example code which uses example_gen_pb2 but what is it and where is it documented?
# Input has a single split 'input_dir/*'.
# Output 2 splits: train:eval=...