Questions tagged [tensorflow-datasets]
TensorFlow's `tf.data` module provides a functional API for building input pipelines, using the `tf.data.Dataset` and `tf.data.Iterator` classes.
2,099
questions
0
votes
1
answer
29
views
Can't train and sim my tensorflow model with original dataset (tf.data.dataset) but it work after I split it!? Same processing on both?
I build a dataset for trainning 2 heads neural network, first with lstm and the second with simple perceptron.
My dataset process in 2 ways to get one version split into train and test set and the ...
0
votes
0
answers
13
views
Tensorflow model: Get NaN on prediction while the same datasets work well on trainning mode
I build a dataset to train a simple 2 heads model like this.
timeseries_input = tf.keras.Input(shape=(nb_timestep_in, nb_feature_input), name='input1')
attribut_input = tf.keras.Input(shape=(...
0
votes
0
answers
22
views
Build a Tensorflow dataset from generator
I have 10000 large files.
From each file, we generate large number of training samples.
def generate_sample(tokens, id):
for i in range(len(tokens):
r = tokens[i]
x =.. #list
...
0
votes
1
answer
31
views
tensorflow fashion MNIST data set
Using the Fashion MNIST data set in Tensorflow.
When building the model as below, why is the flattened layer only (28*28=724) and not 724x60000? 60000 being the number of images in the dataset?
at ...
0
votes
0
answers
12
views
How to train TensorFlow on large images? (4288x2848)
I want to train my TensorFlow model on large images (4288x2848 pixels), but it just loads all images in my vRAM and i dont have enough vRAM for this (on my GPU is only 8GB - I use TensorFlow for GPU, ...
0
votes
0
answers
16
views
Combining Sliding Windows Over Many Time-Series in Tensorflow
I have many (O(1 million)) time series each with about 30 time-steps. The number of time steps varies between samples. I would like to use a sliding window over each time series to generate training ...
0
votes
0
answers
20
views
custom loss to nullify a class contribution for the regression output in a keras multi task model
I have a model which performs two basic tasks, a binary classification and a regression.
The contribution of one class has to be nullified for the regression task, while it's of course important for ...
0
votes
0
answers
6
views
How should I configure dataset to learn xlnet, transformer_xl in multi gpu?
If the data that continues to be connected to each other enters the model,
How can data sets be configured when performing Tensorflow GPU parallel processing to ensure continuity of data sets entering ...
0
votes
0
answers
15
views
tf.data API 'flat_map' method for unpacking after window method with VariantDatasets in dictionary
When attempting to use the flat_map method to unpack the sub VariantDatasets in my feature dictionary, I struggle with understanding the function to be parsed into the flat_map method to succesfully ...
0
votes
0
answers
19
views
How can I save a customized version of CIFAR-10 from Pytorch code and use it in TensorFlow Colab code?
I have a Pytorch program that produces a customized version of CIFAR-10 on the fly (it does not save that customized CIFAR-10,but the program downloads the original dataset, customizes it, and uses it ...
0
votes
1
answer
25
views
Tensorflow. call() method to create a baseline model which averages the "inputs" with type framework.ops.Tensor for timeseries data
I have been following the "Time series forecasting" tutorial and I have some troubles making an own multi-step baseline model that can average the input data in the time axis, and repeat it ...
0
votes
0
answers
5
views
Where can I get FDMA Competition dataset?
I am working on malicious activity detection and am searching for the FDMA 2012 competition dataset. I have encountered difficulty accessing it, as SMU, which originally organized the competition, has ...
3
votes
0
answers
66
views
Image clustering with extracted feature vectors on a large dataset
I've been trying to use a number of clustering techniques on an extremely large image dataset (around 1 million images). The process is quite standard - e.g., using a pretrained CNN (VGG19 in this ...
0
votes
0
answers
23
views
Tensorflow tfds.load "generating train examples" loop
Details about the error
I used the tdfs.load to import the 'kddcup' dataset from tensorflow_dataset, but when I ran the code the execution of the code got stuck in the 'Generating train examples...' ...
0
votes
1
answer
24
views
how to use tensorflow Dataset.from_generator
I'm trying to fit a model using a dataset built using 'tf.data.Dataset.from_generator'.
But the fit fails.
here the code of dataset:
cd_gen=CordicDatasetFT(14)
cos=(tf.TensorSpec(shape=(14, 3), dtype=...
0
votes
0
answers
23
views
Why am I not getting labels of my tf.data.dataset in the correct order when using the tensor flow map function and turning it to a numpy array?
So I'm creating a tensor flow dataset with the image_dataset_from_directory method. The thing is, I want to split the tf.data.dataset returned by this function in two numpy arrays of both input data (...
-1
votes
0
answers
29
views
How to window make_csv_dataset()
I am attempting to window some data from a csv file for time series training, but as I create the dataset from make_csv_dataset() and attempt to window it:
def window_data(self, data_ds, window_size, ...
0
votes
0
answers
13
views
Accessing TensorFlow datasets
I want to access the dataset provided by TensorFlow in this link. How to get this data? I followed this notebook, however, it gives me this error:
Downloading and preparing dataset 1.59 GiB (download: ...
0
votes
1
answer
30
views
Filter Keras image_dataset_from_directory classes
I'm importing dataset from kaggle, it has 15 classess and I need only 10 classes from them, how can I filtered these classes into my dataset?
I am trying this code
image_size= 256
batch_size=8
...
1
vote
0
answers
112
views
How to use tf.repeat and another built-in highlevel funcs on Dataset?
I want to do something like this code of NER task which will align WordPieces of word with tag of this word:
import tensorflow as tf
tokens = tf.ragged.constant([[4], [2, 5, 9]], dtype=tf.int32)
tags ...
1
vote
1
answer
35
views
how to use predict_on_batch to avoid out of GPU memory error with datagenerator
I have a Keras model which consists of two parts (left and right), both working generally independently (due to a practical scenario), but these parts exchange some latent data generated by the model ...
0
votes
1
answer
20
views
How to access images from tensorflow dataset?
I have created an image dataset in tensorflow, but now I want to use an image from the dataset as a variable (to view and manipulate it, etc) and I simply do not know how to access it, since batch ...
0
votes
0
answers
22
views
Resize tensorflow image from tensorflow_datasets
I have modified the original tfds code of the CUB dataset link, as I am adding a text description to each of the images. My own version of the CUB dataset builds without problems, but my problem is ...
0
votes
1
answer
19
views
How to write a few-shot filter for tensorflow datasets?
Is there any way to write a tfds.Dataset.filter function that filters out 5 images per class?
I've tried to create an object (and also an equivalent function) with an attribute to keep track of the ...
0
votes
0
answers
12
views
Can a Tensorflow Datasets decoder have access to multiple features?
I'm using the Kitti dataset in tensorflow datasets (tfds). It has a nested structure of object features:
FeaturesDict({
'image': Image(shape=(None, None, 3), dtype=uint8),
'image/file_name': ...
0
votes
0
answers
30
views
Sampling from a custom dataset that is derived from tf.data.dataset in tensorflow
I'm new to tensorflow and trying to write a custom dataset class derived from tf.data.dataset like this example code:
import tensorflow as tf
class CustomDataset(tf.data.Dataset):
def __init__(...
1
vote
1
answer
34
views
Tensorflow dataset causes a memory leak
I'm creating a dataset for my xception model training, I'm doing it like this:
def load_dataset(height: int, width: int, path: str, kind: str, batch_size=32) -> tf.data.Dataset:
directory = os....
0
votes
0
answers
13
views
Error while defining a string variable in tf.data.experimental.CsvDataset
I build my dataset (ds) using
this fonction:
def build_dataset(path_list, index_variables_retenus, header=True, field_delim=','):
def csv_loader(path):
return tf.data.experimental....
0
votes
0
answers
19
views
Datasets for various types of noisy images
I developed a neural network to learn to classify various kinds of noise in an image - salt and pepper, Gaussian, Poisson etc.
I want to validate the model on unseen data. Are there any labelled ...
3
votes
2
answers
2k
views
"No module named 'resource'" error when using TensorFlow and TensorFlow Datasets in Windows virtual environment
I am facing a problem when trying to use TensorFlow and TensorFlow Datasets in my virtual environment on Windows. When I try to import these packages, I receive the following error:
...
0
votes
0
answers
8
views
file path error, many images files merge with csv files
I want to show my problems where arise and what is my code by this code
I want to know code what to wright in both pd.read_csv in file path
and why all files are not tracked by version control by git, ...
0
votes
0
answers
13
views
How to pass 2 tf datasets to .fit as sequence and tags?
I’ve loaded two datasets with sentences and labels from txt file with TextLineDataset, vectorized them by .map. Now I need to pass this data to model.fit. And I’ve found, that I can’t pass datasets ...
0
votes
0
answers
34
views
Tensorflow: How to make a training/validation split when using strategy.distribute_datasets_from_function?
I want to make a dataset that can be distributed for MultiWorkerMirroredStrategy.
I read the examples from TFRecords, so sharding has to be performed on the list of TFRecords files before reading them,...
0
votes
1
answer
27
views
Tensorflow Scalar Multiplication with Batches
I would like to create a tensorflow layer that takes in n inputs, n-1 are data points and the last is a weight vector of length n-1. Each of the (i in n-1) data points are then multiplied by the value ...
0
votes
1
answer
34
views
TensorFlow Probability inference error input type
I train a model using tensorflow_probability and distributions.
To train it I format my data like this (2 heads model so 2 inputs set):
input_1 = tf.data.Dataset.from_tensor_slices(Xdata)
input_2 = tf....
0
votes
0
answers
29
views
Train CNN with large dataset with data in a Google bucket
I'm trying to train a CNN on a segmentation task (classify all pixels in two classes: soil or not soil). In order to do this, I have input images with a lot of channels (approx. 47) and label masks. I ...
2
votes
1
answer
473
views
'str' object is not callable when fitting CNN model
I'm doing a deep learning project on detecting if a person has tuberculosis based on chest x-ray images. After reading and preprocess my data, I run into a problem when building my CNN model.
import ...
0
votes
0
answers
9
views
Tensorflow Data Pipeline: Read text files from sub-directories for Seq2Seq Models
I am trying to create a data pipeline using Tensorflow's text_dataset_from_directory method for training a seq-to-seq model. The folder structure as below:
BBC News Articles
|_ News Articles
|...
0
votes
0
answers
22
views
TensorFlow Conv2DTranspose's Type mismatch: actual int32 vs. expect string error on GPU
I am trying to train a variational autoencoder in TensorFlow which is causing a weird error that I cannot replicate on CPU with the same code.
Relevant parts of my model -
@tf.function
def ...
0
votes
0
answers
23
views
How can I debug the data iterator in tensorflow?
I am using a tf.data.Iterator for my training in tensorflow with where the the get_next() function fails.
Do anyone know how I can debug inside the get_next() function?
The get_next() function fails ...
0
votes
1
answer
50
views
Tensorflow memory problems with validation data
Im trying to train a tensorflow neural network. Since the training data is too big for my computer RAM I have divided it into sub-datsets and sequentially train them. I have a problem however, if I ...
0
votes
0
answers
19
views
Eumerate over two different tensorflow batchDatasets
I am trying to train a VAE to convert an images of one type into images of another type. The problem I am having is when I call to train the model, telling it that I want the after image to be ...
0
votes
0
answers
39
views
How do I look inside a column in a CacheDataset in TensorFow?
The following code was used to split the train and test set:
tf.random.set_seed(42)
shuffled = ratings.shuffle(100_000, seed=42, reshuffle_each_iteration=False)
train = shuffled.take(80_000)
test = ...
2
votes
1
answer
45
views
TypeError when using ragged tensors in TensorFlow data pipeline
I'm attempting to create a tf.data pipeline using RaggedTensors in TensorFlow, but I keep encountering a TypeError:
"Cannot convert the argument `type_value`: RaggedTensorSpec(TensorShape(\[None, ...
1
vote
0
answers
31
views
How to calculate F1 score for tensor flow dataset which is batched and prefetched (multi label image classification)
This is the code snippet for the datasets
train_ds= tf.keras.preprocessing.image_dataset_from_directory( "./tmp/Alzheimer_s_Dataset/train/", validation_split=0.2, subset="training",...
0
votes
0
answers
28
views
tf.data.Dataset.from_generator save additional data from Python generator
I would like save additional data (date, file name etc.), which I don't use for training but will need it later to identify my predictions. This data is generated together with features and target by ...
0
votes
0
answers
19
views
Batching tf dataset correctly
I am following Keras text classification tutorial for sentiment analysis.
Their data is structured in folders like so:
test
- pos
- neg
train
- pos
- neg
where pos and neg are folders ...
0
votes
0
answers
64
views
Tensorflow dataset input for encoder-decoder LSTM model
I'm trying to train an LSTM network in Tensorflow.
But this gives me an error:
ValueError: Layer "model_10" expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf....
1
vote
1
answer
226
views
Load ImageNet with TensorFlow-Datasets tfds from a PyTorch Folder Structure
Dear ImageNet and TensorFlow specialists
Originally, I have downloaded the ImageNet dataset tar files ILSVRC2012_img_train.tar, ILSVRC2012_img_val.tar and used this PyTorch script to extract the files....
0
votes
0
answers
212
views
Order of batch, prefetch, shuffle and cache in a Tensorflow dataset
When creating a dataset from a generator, what would be the correct order of the following dataset methods? Or does the order not matter here?
ds = tf.data.Dataset.from_generator(my_generator)
ds = ds....