Questions tagged [tfx]

TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

TFX pipeline-root not found

im currently trying to configurate a tfx pipeline with components such as SchemaGen,StatisticGen and Transform. Providing the statistics-, and schema-gen components and running them in a tfx-workflow ...
Bennet's user avatar
  • 1
0 votes
0 answers
17 views

TFX TypeError: Argument input_params should be a Channel of type <class 'tfx.types.standard_artifacts.ExternalArtifact'> (got test_string)

I updated the TFX from 0.22.0 to 1.8.0 and now I'm getting this error: TypeError: Argument input_params should be a Channel of type <class 'tfx.types.standard_artifacts.ExternalArtifact'> (got ...
natielle's user avatar
  • 400
0 votes
1 answer
6 views

AttributeError: module 'tfx.utils.io_utils' has no attribute 'file_io'

I updated the tfx from 0.22.0 to 1.8.0 and now I'm getting this error: AttributeError: module 'tfx.utils.io_utils' has no attribute 'file_io' The code is: from tfx.utils import io_utils io_utils....
natielle's user avatar
  • 400
0 votes
0 answers
19 views

How to configure optional component with TFX?

I would like to run one optional custom component with TFX and Airflow. Is it possible? This optional component is used to send updates to another system, if this optional component fails, it will not ...
natielle's user avatar
  • 400
0 votes
0 answers
18 views

How to Run a TFX Orchestration Pipeline Outside Jupyter?

Learning data ingestion and prediction orchestration with tensorflow from this notebook on github. It is using an InteractiveContext on the ipython environment. But obviously a production system does ...
Della's user avatar
  • 1,304
0 votes
0 answers
17 views

TFX CSVExampleGen component: How to read data with "|" as separator?

I am trying my way at TensorFlow's TFX. I have a CSV that contains the pipe symbol (|) as a field separator instead of the default comma. How can I specify this when using the CsvExampleGen component? ...
Hunting_the_Hint's user avatar
0 votes
0 answers
25 views

Developer workflow for tensorflow/tfx

I'm trying to improve my workflow for supervised learning in the tf ecosystem, and was wondering if anyone has encountered/solved this problem. I'm able to get a fully functional pipeline to execute ...
Pritam Dodeja's user avatar
0 votes
0 answers
16 views

I don't understand how the Trainer component works

As far as I understand, we need to supply the Trainer component training examples directly (with the "examples" parameter) from the output of an ExampleGen or a Transform component, but at ...
TlifeProgram's user avatar
0 votes
0 answers
15 views

Is there a way to save a TFX DatasetFeatureStatisticsList?

Saving a TFX DatasetFeatureStatisticsList to disk (binary) After computing data statistics from CSV files with tfdv.generate_statistics_from_csv I can't find a good way to save it other than using ...
nmik's user avatar
  • 1
0 votes
0 answers
51 views

How can I make tfx use tensorflow-cpu rather than the full tensorflow package?

I am trying to build a lightweight docker image with tfx library installed, as the officially released image is pretty heavy in size (~9 GBs). Here is the code that I am using for the image now: FROM ...
Shayan's user avatar
  • 125
0 votes
1 answer
79 views

TFX's Evaluator Component cannot prepare the inputs for evaluation

I am using tfx pipeline for training and evaluating an autoencoder. The data that I have is basically 5 arrays of size (15,1) that I concatenate and put together and pass to the model. In order to ...
Shayan's user avatar
  • 125
0 votes
1 answer
38 views

Tensorflow: How to add a property in execution object in MLMD MetadataStore?

I'm using the MLMD MetadataStore to manage the data pipelines and I need to add an execution property in MLMD to get this property later. I'm trying add with this: from ml_metadata.proto import ...
natielle's user avatar
  • 400
0 votes
1 answer
39 views

TFX docker image is heavy

I was just wondering why tensorflow/tfx image is 8Gb while gpu tensorflow/tensorflow image is less than 3gB? At runtime, I expect tfx container to just load input artifacts, run some tensorflow code ...
Youcef Kacer's user avatar
0 votes
0 answers
47 views

Tensorflow Transform Fails Converting Strings to Numbers

I am trying to fit a TensorFlow model and one my features comes in as a comma-separated string of ints (possibly empty string). The feature appears in the pretransform schema as feature { name: &...
user1337's user avatar
0 votes
1 answer
64 views

Impossible to get post transform statistics by split

I'm running a simple penguin pipeline in interactive mode with a split train/eval, the transform step run but i can't get post_transform_statistics artifacts. Inside the dedicated artifacts folder /...
Youcef Kacer's user avatar
1 vote
1 answer
59 views

What does tfma.metrics.MeanLabel do?

I am interested in what tfma.metrics.MeanLabe does, and how it should be used, and what the difference is between tfma.metrics.MeanLabe and tfma.metrics.MeanPredictio and tfma.metrics.MeanAttributions....
Alex's user avatar
  • 1,214
0 votes
0 answers
18 views

need help compiling a folder named "build"

need help compiling the build folder Inside of the build folder is a bunch of json files and .tfx. Expecting a single file to come out that i can ./ the file
everything 's user avatar
0 votes
1 answer
45 views

str() is not usable anymore to get true value of a Text tfx.data_types.RuntimeParameter during pipeline execution

how to get string as true value of tfx.orchestration.data_types.RuntimeParameter during execution pipeline? Hi, I'm defining a runtime parameter like data_root = tfx.orchestration.data_types....
Youcef Kacer's user avatar
0 votes
1 answer
69 views

tfx.components.StatisticsGen display train and eval in two different figures, is it possible to have them in a single figure as tfdv does?

a superimposed display for train/val splits using StatisticsGen Hi, I'm currently using tfx pipeline inside kubeflow. I struggle to have StatisticsGen showing a single graph with train and validation ...
Youcef Kacer's user avatar
0 votes
0 answers
78 views

Having NoneType error in multiple pipeline build using TFX and Airflow

I am working with TFX and AIrflow to build the pipeline for image classification and segmentation. I am replicating the repo by Jan Marcel Kezmann (Link). While executing the classification_dag.py in ...
Bhargav Patel's user avatar
0 votes
1 answer
426 views

Dealing with missing values in tensorflow

I need some guidance on the approach to imputation in tensorflow/deep learning. I am familiar with how scikit-learn handles imputation, and when I map it to the tensorflow ecosystem, I would expect ...
Pritam Dodeja's user avatar
0 votes
1 answer
136 views

How to visualize a saved statistics artifact?

I know of two ways to run a TFX pipeline. First, using a Jupyter notebook with InteractiveContext in a browser: from tfx import v1 as tfx from tfx.orchestration.experimental.interactive....
Mehran's user avatar
  • 15.8k
0 votes
0 answers
105 views

I am struggling to write a transform python file for the tfx pipeline in transform component. I am working with object detection task

I am building a pipeline using tensorflow extended. The dataset is already conversted to the tfrecord for the ingestion. I want to do data preprocessing such as scalin, data augmentation along with ...
Bhargav Patel's user avatar
0 votes
1 answer
177 views

How to get the uri of the current pipeline's artifact

Consider the following pipeline: example_gen = tfx.components.ImportExampleGen(input_base=_dataset_folder) statistics_gen = tfx.components.StatisticsGen(examples=example_gen.outputs['examples']) ...
Mehran's user avatar
  • 15.8k
0 votes
0 answers
47 views

How to write a preprocessing_fn function that relies on an Artifact?

Let's consider a very straightforward pipeline like this: example_gen = tfx.components.ImportExampleGen(input_base=_dataset_folder) statistics_gen = tfx.components.StatisticsGen(examples=example_gen....
Mehran's user avatar
  • 15.8k
1 vote
0 answers
86 views

How to get the list of features along side their schema and stats using TFX

Let's say that I have the following, very straightforward pipeline: import os from tfx import v1 as tfx _dataset_folder = './tfrecords/train/*' _pipeline_data_folder = './pipeline_data' ...
Mehran's user avatar
  • 15.8k
0 votes
2 answers
70 views

MLOps with TFX: How to ingest data when using Sequence from Keras?

I am using a class called DataGenerator, that returns a tuple (data_array, label_array), follows the code: from tensorflow.keras.utils import Sequence class DataGenerator(Sequence): ""&...
Alexander Martins's user avatar
0 votes
1 answer
757 views

A TFX equivalent for Pytorch [closed]

I'v recently worked with TensorFlow Extended(TFX) platform. Since my development background is on Pytorch stack, I'm looking for well-compatible alternatives of TFX for Pytorch. While searching for ...
Mohsen Mahmoodzadeh's user avatar
0 votes
1 answer
83 views

How to split the dataset into mutiple folds while keeping the ratio of an attribute fixed

Let's say that I have a dataset with multiple input features and one single output. For the sake of simplicity, let's say the output is binary. Either zero or one. I want to split this dataset into k ...
Mehran's user avatar
  • 15.8k
0 votes
0 answers
54 views

InfraValidator component not "blessing" my model

The InfraValidator TFX component is not blessing my model, and I'm having trouble finding out why it isn't getting "blessed". I'm not sure what part of my model would be causing it to not be ...
Jon's user avatar
  • 1
1 vote
0 answers
181 views

How to get prediction labels with Tensorflow Serving Predict API?

I have a keras classification model saved remotely, and I use RESTful API to call the model on my site and make predictions. The JSON request is done as follows URL: POST https://my.site/models/v1:...
sound wave's user avatar
  • 3,296
0 votes
1 answer
106 views

How to use the same code for machine learning "data transformations" before prediction in both reshearch & production

I wonder what is the best practice/s of taking for example a jupiyter notebook that contains the whole flow from eda to prediction and use the same code for "data transformations from the raw ...
george k's user avatar
0 votes
2 answers
92 views

Why tfx.components.FnArgs does not have an "epochs" attribute?

tfx.components.FnArgs is the way to pass values to the run_fn function which will in turn train the model in Tensorflow Extended pipeline. Looking at the tfx.components.FnArgs' documentation, I cannot ...
Mehran's user avatar
  • 15.8k
0 votes
1 answer
68 views

Why ImportExampleGen reads TFRecords as SparseTensor instead of Tensor?

I'm converting a CSV file into a TFRecords file like this: File: ./dataset/csv/file.csv feature_1, feture_2, output 1, 1, 1 2, 2, 2 3, 3, 3 import tensorflow as tf import csv import os print(tf....
Mehran's user avatar
  • 15.8k
2 votes
3 answers
575 views

Why we need TFX if we have airflow for orchestration

I still don't get why we need TFX. TFX will convert your defined pipeline to Airflow DAG and run it on airflow, I could just write my pipelines in python and use Airflow's PythonOperator to build a ...
ariverhorse's user avatar
0 votes
1 answer
192 views

RuntimeError: Split pattern data/* does not match any files

I'm currently trying to convert the TFX-TFRS tutorial into an Airflow pipeline. When I run this Pipeline with LocalDagRunner() it executes without error. When I use the AirflowDagRunner() it throws ...
Sebastian Sätzler's user avatar
1 vote
1 answer
371 views

ml_metadata.errors.AlreadyExistsError: Given node already exists

I ran into a problem using TFX, MLMD, and Apache-Airflow as the orchestrator. Local-dag-runner, provided by TFX, works fine, resulting in distinct artifacts for each pipeline component run. The ...
Parham Davari's user avatar
0 votes
1 answer
61 views

TFX tutorial "penguin_tfdv.ipynb" issue TypeCheckError

I've got this problem. How could I solve this problem? This is part of the tutorial of TFX. link to the tutorial I changed nothing in the example code. TypeCheckError: Output type hint violation at ...
Ovtchinnikov's user avatar
0 votes
2 answers
378 views

TFX results in TypeCheckError: Output type hint violation at WriteStatsOutput[train]: expected <class 'apache_beam.pvalue.PDone'>, got <class 'str'>

I tried running the tutorial provided in tfx documentation and I am facing a problem with the path to metadata. I have described the query in detail in this issue https://github.com/tensorflow/tfx/...
Pratyaksh Singh's user avatar
0 votes
1 answer
535 views

Vertex workbench - how to run BigQueryExampleGen in Jupyter notebook

Problem Tried to run BigQueryExampleGen in the InvalidUserInputError: Request missing required parameter projectId [while running 'InputToRecord/QueryTable/ReadFromBigQuery/Read/SDFBoundedSourceReader/...
mon's user avatar
  • 19.7k
0 votes
1 answer
116 views

TF Serving Predict API Output Interpretation

Is the TensorFlow Serving (TFS) Predict API output the same as the tf.keras.model.predict method (i.e. the outputs of the model according to the compiled metrics)? For example, if we have a tf.keras....
pedro malheiro's user avatar
0 votes
1 answer
109 views

Issue with `TransformFeatureLayer` when using `custom_config` in `preprocessing_fn`

System information Environment: Linux 5.13.0-41-generic #46~20.04.1-Ubuntu TensorFlow version: 2.8.0 TFX Version: 1.8.0 Python version: 3.8.13 Python dependencies: absl-py 1.0....
intrainepha's user avatar
0 votes
1 answer
27 views

TFX AutoML on local System

Is it possible to perform AutoML using TFX on local environment. I know with kubeflow and GCP we can perform AutoML on the cloud, but I want to do all of these steps locally and deploy the model to a ...
Shisui Otsutsuki's user avatar
1 vote
0 answers
226 views

How to pass a *serialized* tensor to a TensorFlow Model Serving server?

I implemented a simple TF model. The model received a serialized tensor of a gray image (simply a 2d ndarray), and restored it to a 2-d tensor. After then, some inference is applied on this 2-d tensor....
Zepp L.'s user avatar
  • 11
0 votes
1 answer
456 views

is TFX (Tensorflow Extended) being used in big companies

I've recently stumbled upon tensorflow extended, and I've finally started to understand the need and uses of it. It makes the whole machine learning pipeline a lot easier and looks like it can be used ...
Muhammad Ahmed's user avatar
0 votes
0 answers
213 views

How to customize tfx Evaluator component

We are developing a ML pipeline with TFX, the model is a Keras sequential DNN with mostly dense layers. We would like to customize the Evaluator component to compare the blessed and the candidate ...
Sara C.'s user avatar
0 votes
0 answers
176 views

How to use jupyter notebook in a local tfx docker image?

I have a windows laptop where I have WSL2 installed. In that I have installed docker and successfully run the docker installation of TFX. I am not sure on how to customize the code and run it like we ...
Yash Kanojia's user avatar
2 votes
2 answers
573 views

Image processing in Tensor flow TFX pipelines

I am trying to get a Tensorflow TFX pipeline up and running using the MNIST dataset. # Imports import pandas as pd import numpy as np from keras.datasets import mnist import tensorflow as tf from tfx ...
tim_ai's user avatar
  • 41
2 votes
2 answers
3k views

Using String parameter for nvidia triton

I'm trying to deploy a simple model on the Triton Inference Server. It is loaded well but I'm having trouble formatting the input to do a proper inference request. My model has a config.pbtxt set up ...
Regalia's user avatar
  • 149
1 vote
2 answers
438 views

TFX - What is example_gen_pb2 and where is it documented?

The ExampleGen TFX Pipeline Component has the example code which uses example_gen_pb2 but what is it and where is it documented? # Input has a single split 'input_dir/*'. # Output 2 splits: train:eval=...
mon's user avatar
  • 19.7k