Questions tagged [tensorflow-data-validation]
The tensorflow-data-validation tag has no usage guidance.
43
questions
0
votes
1
answer
33
views
Is it possible to install specific version of TFDV without upgrading TF?
I have problem with installing TensorFlow Data Validation (TFDV).
I have already TensorFlow installed (version 2.5.0) and I don't want to upgrade that version.
I used this command:
pip install ...
0
votes
0
answers
9
views
Generate data statistics with GenerateStatistics from tf.Example generated from a beam pipeline
I am trying to generate data statistics from a beam pipeline with tensorflow data validation. From what i can see online there are two ways to do this: The statsgen component, and the ...
1
vote
1
answer
69
views
apache beam rows to tfrecord in order to GenerateStatistics
I have built a pipeline that read some data, does some manipulations and create some apache beam Row objects (Steps 1 and 2 in the code below). I then would like to generate statistic and write them ...
0
votes
1
answer
68
views
tensorflow data validation tfdv fails on google cloud dataflow with "Can't get attribute 'NumExamplesStatsGenerator' "
I am following this "get started" tensorflow tutorial on how to run tfdv on apache beam on google cloud dataflow. My code is very similar to the one in the tutorial:
import ...
4
votes
1
answer
97
views
Infer multivalent features with tfdv from pandas dataframe
I want to infer a schema with tensorflow data validation (tfdv) based on a pandas dataframe of the training data. The dataframe contains a column with a multivalent feature, where multiple values (or ...
0
votes
1
answer
111
views
TensorFlow Data Validation - How to return the rows with anomalies
Tensorflow Data Validation provides a way to find anomalies in your data.
However, I am able to find only a way to provide a summarized version of the anomalies (by using tfdv.validate_statistics and ...
0
votes
0
answers
26
views
Installing TensorFow Data Validation on Jetson AGX (AArch64)
Has anyone been able to install TFDV on a Jetson AGX or similar device with AArch64 architecture? I have tried installing with PyPI, building with Docker, and building from source with no luck. I am ...
1
vote
4
answers
331
views
tensorflow-data-validation cannot be pip installed
Since I'm moving away from pandas DataFrames to TensorFlow datasets, I'd like to use tensorflow-data-validation instead of the more traditional pandas-profiling when it comes to data exploration and ...
0
votes
0
answers
31
views
Can Tensorflow '2.5.0' work with tensorflow-data-validation and big query? and is this the best method to validate schema and data entering the cloud
What is the best method to have data validated as it comes into big query? Big Query has machine learning backed into its sql is one solution. Another solution is to use tensorflow-data-validation.
...
0
votes
0
answers
29
views
Button that runs all my data validation lists
I need to create a button that runs all my data validation lists. Majority of the cells have a data validation drop down list. I need for the user to select an item from each drop down list and when ...
0
votes
0
answers
24
views
"list index out of range" when running generate_statistics_from_tfrecord
Currently when running a generate_statistics_from_tfrecord in Dataflow, I find it causing this exception:
File "/usr/local/lib/python3.6/dist-packages/tensorflow_data_validation/statistics/...
0
votes
0
answers
70
views
JavaScript Data validation design pattern - Calling library function without 'this' keyword
I'm writing a JavaScript library that would validate the data stored in flat file. It will take list of tests (reusable JavaScript code stored as text in database) and records of the datafile as array....
0
votes
0
answers
42
views
MS Excel: How to get value from Data Validation when a particular item is selected
I am new to MS Excel and learning. I have an Excel File for Budget Management (Project).
What I am trying to do is when an Item is selected from the Dropdown ( Data Validation ) another Cell should ...
0
votes
0
answers
23
views
What input should be given to Jensen shenon function to find Divergence value?
I have been trying to find the right value of JS divergence between two datasets(say 'Age'column).
Now, what should be given to the JS function from scipy as input?
Age column as it is or pdf of the ...
0
votes
1
answer
164
views
Dimension does not match when using `keras.Model.fit` in `BERT` of tensorflow
I follow the instruction of Fine-tuning BERT to build a model with my own dataset(It is kind of large, and greater than 20G), then take steps to re-cdoe my data and load them from tf_record files.
...
0
votes
0
answers
36
views
TFDV generate_stats_from_csv trigger an out of memory error
i have a problem when generating stats over a small dataset (~10MB).
it takes to much time and consume to much memory (it reaches 25 GB of my memory which makes no sense ). and at the end it stops ...
3
votes
0
answers
148
views
Is it possible to run tensorflow-data-validation on MacOS with M1 chip?
Question: Is it possible to run tensorflow-data-validation on MacOS with M1 chip?
Steps taken: I have created a conda environment (tfdv38) in which I have installed the Mac-optimized TensorFlow.
I ...
1
vote
1
answer
200
views
In TFX, is it possible to infer Schema with dates?
I'm using TFX (more precisely TensorFlow Data Validation) with the infer_schema method documented there https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/infer_schema. It generates a ...
1
vote
1
answer
246
views
TensorFlow Extended: Specifying Valency Of Features In Schema
I am presently trying to feed a dataset with a few multivalent feature columns through a TensorFlow Extended (TFX) pipeline. Here is a row from my sample data:
user_id 29601
...
0
votes
1
answer
48
views
python pandas multi-level index getting a particular value
i have a pandas dataframe of grouped values every hour
2020.05.05 17:00:00 cpu_usage_specint 4.354603
phys_iops 3075.779680
total_memory ...
2
votes
1
answer
204
views
Why isn't SchemaGen supported in tfdv.display_schema()?
Regarding TFX' tensorflow-data-validation, I'm trying to understand when I should use *Gen components vs. using TFDV provided methods.
Specifically, what's confusing me is that I have this as my ...
0
votes
0
answers
212
views
tensorflow-data-validation doesn't work on large datasets with apache-beam direct runner because of grpc timeout
I’m running into a problem of tensorflow-data-validation with direct runner to generate statistics from some large datasets over 400GB.
It seems that all workers stopped working after an error message ...
2
votes
1
answer
417
views
What is the difference between implementation of 'drift_comparator' and 'skew_comparator' available in Tensorflow Data Validation?
Need help some in understanding the difference between the underlying implementation of skew and drift comparator in TFDV.
0
votes
0
answers
77
views
Keep getting ValueError while training on tf.data.datasets and include validation_data
Running the model.fit(...) method on my dataset. I keep getting errors like this:
ValueError: in user code:
...
ValueError: Could not find matching function to call loaded from the SavedModel. Got:
...
0
votes
0
answers
381
views
Unable to run Tensorflow Data Validation on Google Cloud Platform (Dataflow)
I have been trying to run TensorFlow Data Validation following Google Documents
Followed same steps as https://www.tensorflow.org/tfx/data_validation/install:
>pip install tensorflow-data-...
1
vote
3
answers
662
views
Tensorflow Data Validation does not identify anomalies in numerical features
I've been testing Tensorflow Data Validation (version 0.22.0) to use in my current ML pipelines and I noticed it does not get any anomaly in numerical features. For instance,
> import pandas as ...
0
votes
0
answers
2k
views
This cell's contents violate its validation rule
I am getting this error in my sheet when I applied data validation.
the validation is working fine but this error is showing continuously.
I hope anyone knows this.
the condition in data validation ...
0
votes
3
answers
348
views
How do I save a TFDV stats in the correct format for them to be loaded back in?
It is puzzling to me that there is a tfdv.load_statistics() function, but no corresponding tfdv.write_statistics() function. How do I go about saving the statistics, and then loading them again?
e.g.
...
0
votes
1
answer
102
views
Tensorflow data validation for uniqueness
I'm using Tensorflow data validation and want to make sure a column has no duplicate values. But it seems Tensorflow's data validation doesn't have a feature like Deequ's isUnique function to check ...
1
vote
2
answers
669
views
Understanding L-infinity norm which is used in TFDV
I was trying to implement TensorFlow Data Validation to check drift/skew in a dataset. They are using the L-infinity norm as a metric. I didn't understand the concept. Can anyone explain how it is ...
1
vote
0
answers
126
views
How can I install tensorflow_data_validation in python 3.7?
I use anaconda and I have python 3.7, I tried to install tensorflow_data_validation from anaconda's prompt but it gives me the following error: Could not find a version that satisfies the requirement ...
1
vote
1
answer
167
views
How to see all the possible options for schema metadata in tensorflow?
I am using tensorflow data validation and I am trying to build schemas around my datasets. I've built the initial schemas and I can see/edit them in notepad, but I'm having a hard time actually ...
1
vote
1
answer
2k
views
ERROR: Could not find a version that satisfies the requirement tensorflow-data-validation (from versions: none)
I get this error with Python 3.7 and Windows 10 64-bit (which is supported).
It seems that there are only wheels for Windows for Python 3.5 and 3.6...
Thanks in advance.
pip install tensorflow-data-...
3
votes
3
answers
367
views
How to enable numerical feature drift?
I want to enable numerical feature drift without setting a domain. Ex: I am scoring customers based on age and in my training set I had a uniform distribution of the variable. Now, in my serving data, ...
4
votes
1
answer
798
views
Tensorflow model analysis, TFMA for keras model
I would like to use TFMA with keras model. The keras model was created with TF 2.0 alpha.
The model is a pretrained model with a classification layer:
...
1
vote
2
answers
371
views
Tensorflow data validation for tensorflow 2.0
Is there any version of Tensorflow data validation (tfdv) that can work with tensorflow 2.0 alpha ?
0
votes
1
answer
347
views
Reading Dataset statistics generated by StatisticsGen
I looked at the TFX guide (https://www.tensorflow.org/tfx/guide), and I was able to run the pipeline. The StatisticsGen component wrote Dataset statistics to a file on disk. How do I visualize the ...
1
vote
1
answer
146
views
TFX / Tensorflow Data Validation (TFDV) throws overflow encountered in float_scalars error
Following basic tutorial, and trying to generate stats for a TFRecord, I get a overflow error:
import tensorflow_data_validation as tfdv
stats = tfdv.generate_statistics_from_tfrecord(data_location='/...
0
votes
1
answer
280
views
Tensorflow TFDV does not work with Specific NaN values
I'm using Tensorflow Data Validation to generate stats from the data and infer an schema to input in TFX.
I dind't find any option to specify the NaN values, e. g., in pandas there is a field "...
2
votes
1
answer
475
views
Tensorflow TFDV does not work with images
I'm trying to get TFDV working with RGB images as feature inputs, reading from a TFRecords file. I can read/write the image data to TFRecord files fine. Here's the relevant code snippets for writing, ...
1
vote
1
answer
90
views
tensorflow data validation is crashing on 4 cores machine when I process more than 1.5 Gig CSV
I am trying to run tensor flow data validation feature on data sets (CSV) > 2 Gig. It is crashing after some time. It runs very well if data set is around 1 Gig. How to handle large data sets without ...
5
votes
3
answers
2k
views
TFDV Tensorflow Data Validation: how can I save/load the protobuf schema to/from a file
TFDV generates schema as a Schema protocol buffer. However it seems that there is no helper function to write/read schema to/from a file.
schema = tfdv.infer_schema(stats)
How can I save it/load it ?...
1
vote
1
answer
158
views
pip install tensorflow-data-validation fails on OS X El Capitan (10.11.6)
I am not able to install tensorflow-data-validation using pip. when I use:
pip install tensorflow-data-validation
I get this error:
Could not find a version that satisfies the requirement
...