Questions tagged [tensorflow-datasets]

TensorFlow's `tf.data` module provides a functional API for building input pipelines, using the `tf.data.Dataset` and `tf.data.Iterator` classes.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
32 views

Tensorflow data sets [closed]

I'm starting to create a classification model with Tensorflow (I use Python 3.11), for that I'm trying to bring in the tensorflow_datasets the cats_vs_dogs files. And so I can train my model, the ...
Miguel Angel's user avatar
0 votes
0 answers
7 views

Create custom Tensorflow dataset .tiff files

I am new on Tensorflow, and I would like to create a custom dataset for using ESRGAN next. The issues I meet are : I have (low_resolution, high_resolution) images as couple of data All my images are ....
salomepx's user avatar
0 votes
1 answer
29 views

Can't train and sim my tensorflow model with original dataset (tf.data.dataset) but it work after I split it!? Same processing on both?

I build a dataset for trainning 2 heads neural network, first with lstm and the second with simple perceptron. My dataset process in 2 ways to get one version split into train and test set and the ...
Jonathan Roy's user avatar
0 votes
0 answers
13 views

Tensorflow model: Get NaN on prediction while the same datasets work well on trainning mode

I build a dataset to train a simple 2 heads model like this. timeseries_input = tf.keras.Input(shape=(nb_timestep_in, nb_feature_input), name='input1') attribut_input = tf.keras.Input(shape=(...
Jonathan Roy's user avatar
0 votes
0 answers
25 views

Build a Tensorflow dataset from generator

I have 10000 large files. From each file, we generate large number of training samples. def generate_sample(tokens, id): for i in range(len(tokens): r = tokens[i] x =.. #list ...
LearnToGrow's user avatar
  • 1,678
0 votes
1 answer
32 views

tensorflow fashion MNIST data set

Using the Fashion MNIST data set in Tensorflow. When building the model as below, why is the flattened layer only (28*28=724) and not 724x60000? 60000 being the number of images in the dataset? at ...
mw00847's user avatar
0 votes
0 answers
12 views

How to train TensorFlow on large images? (4288x2848)

I want to train my TensorFlow model on large images (4288x2848 pixels), but it just loads all images in my vRAM and i dont have enough vRAM for this (on my GPU is only 8GB - I use TensorFlow for GPU, ...
Mix Mig's user avatar
0 votes
0 answers
16 views

Combining Sliding Windows Over Many Time-Series in Tensorflow

I have many (O(1 million)) time series each with about 30 time-steps. The number of time steps varies between samples. I would like to use a sliding window over each time series to generate training ...
AGecko's user avatar
  • 1
0 votes
0 answers
20 views

custom loss to nullify a class contribution for the regression output in a keras multi task model

I have a model which performs two basic tasks, a binary classification and a regression. The contribution of one class has to be nullified for the regression task, while it's of course important for ...
bio's user avatar
  • 501
0 votes
0 answers
6 views

How should I configure dataset to learn xlnet, transformer_xl in multi gpu?

If the data that continues to be connected to each other enters the model, How can data sets be configured when performing Tensorflow GPU parallel processing to ensure continuity of data sets entering ...
최원준's user avatar
0 votes
0 answers
15 views

tf.data API 'flat_map' method for unpacking after window method with VariantDatasets in dictionary

When attempting to use the flat_map method to unpack the sub VariantDatasets in my feature dictionary, I struggle with understanding the function to be parsed into the flat_map method to succesfully ...
Didlex's user avatar
  • 1
0 votes
0 answers
20 views

How can I save a customized version of CIFAR-10 from Pytorch code and use it in TensorFlow Colab code?

I have a Pytorch program that produces a customized version of CIFAR-10 on the fly (it does not save that customized CIFAR-10,but the program downloads the original dataset, customizes it, and uses it ...
Baraa's user avatar
  • 16
0 votes
1 answer
25 views

Tensorflow. call() method to create a baseline model which averages the "inputs" with type framework.ops.Tensor for timeseries data

I have been following the "Time series forecasting" tutorial and I have some troubles making an own multi-step baseline model that can average the input data in the time axis, and repeat it ...
eliasmaxil's user avatar
0 votes
0 answers
5 views

Where can I get FDMA Competition dataset?

I am working on malicious activity detection and am searching for the FDMA 2012 competition dataset. I have encountered difficulty accessing it, as SMU, which originally organized the competition, has ...
sindhuja's user avatar
  • 167
3 votes
0 answers
68 views

Image clustering with extracted feature vectors on a large dataset

I've been trying to use a number of clustering techniques on an extremely large image dataset (around 1 million images). The process is quite standard - e.g., using a pretrained CNN (VGG19 in this ...
vmchance's user avatar
0 votes
0 answers
24 views

Tensorflow tfds.load "generating train examples" loop

Details about the error I used the tdfs.load to import the 'kddcup' dataset from tensorflow_dataset, but when I ran the code the execution of the code got stuck in the 'Generating train examples...' ...
user22830098's user avatar
0 votes
1 answer
24 views

how to use tensorflow Dataset.from_generator

I'm trying to fit a model using a dataset built using 'tf.data.Dataset.from_generator'. But the fit fails. here the code of dataset: cd_gen=CordicDatasetFT(14) cos=(tf.TensorSpec(shape=(14, 3), dtype=...
G. Lippolis's user avatar
0 votes
0 answers
23 views

Why am I not getting labels of my tf.data.dataset in the correct order when using the tensor flow map function and turning it to a numpy array?

So I'm creating a tensor flow dataset with the image_dataset_from_directory method. The thing is, I want to split the tf.data.dataset returned by this function in two numpy arrays of both input data (...
Ivan González's user avatar
0 votes
0 answers
16 views

Accessing TensorFlow datasets

I want to access the dataset provided by TensorFlow in this link. How to get this data? I followed this notebook, however, it gives me this error: Downloading and preparing dataset 1.59 GiB (download: ...
Reem's user avatar
  • 107
0 votes
1 answer
30 views

Filter Keras image_dataset_from_directory classes

I'm importing dataset from kaggle, it has 15 classess and I need only 10 classes from them, how can I filtered these classes into my dataset? I am trying this code image_size= 256 batch_size=8 ...
basit khan's user avatar
1 vote
0 answers
112 views

How to use tf.repeat and another built-in highlevel funcs on Dataset?

I want to do something like this code of NER task which will align WordPieces of word with tag of this word: import tensorflow as tf tokens = tf.ragged.constant([[4], [2, 5, 9]], dtype=tf.int32) tags ...
Mihail's user avatar
  • 21
1 vote
1 answer
36 views

how to use predict_on_batch to avoid out of GPU memory error with datagenerator

I have a Keras model which consists of two parts (left and right), both working generally independently (due to a practical scenario), but these parts exchange some latent data generated by the model ...
user20780's user avatar
0 votes
1 answer
20 views

How to access images from tensorflow dataset?

I have created an image dataset in tensorflow, but now I want to use an image from the dataset as a variable (to view and manipulate it, etc) and I simply do not know how to access it, since batch ...
tcb93's user avatar
  • 101
0 votes
0 answers
22 views

Resize tensorflow image from tensorflow_datasets

I have modified the original tfds code of the CUB dataset link, as I am adding a text description to each of the images. My own version of the CUB dataset builds without problems, but my problem is ...
user1571823's user avatar
0 votes
1 answer
19 views

How to write a few-shot filter for tensorflow datasets?

Is there any way to write a tfds.Dataset.filter function that filters out 5 images per class? I've tried to create an object (and also an equivalent function) with an attribute to keep track of the ...
frebo2847's user avatar
0 votes
0 answers
12 views

Can a Tensorflow Datasets decoder have access to multiple features?

I'm using the Kitti dataset in tensorflow datasets (tfds). It has a nested structure of object features: FeaturesDict({ 'image': Image(shape=(None, None, 3), dtype=uint8), 'image/file_name': ...
frebo2847's user avatar
0 votes
0 answers
30 views

Sampling from a custom dataset that is derived from tf.data.dataset in tensorflow

I'm new to tensorflow and trying to write a custom dataset class derived from tf.data.dataset like this example code: import tensorflow as tf class CustomDataset(tf.data.Dataset): def __init__(...
TEMP123's user avatar
1 vote
1 answer
36 views

Tensorflow dataset causes a memory leak

I'm creating a dataset for my xception model training, I'm doing it like this: def load_dataset(height: int, width: int, path: str, kind: str, batch_size=32) -> tf.data.Dataset: directory = os....
Marek M.'s user avatar
  • 3,851
0 votes
0 answers
13 views

Error while defining a string variable in tf.data.experimental.CsvDataset

I build my dataset (ds) using this fonction: def build_dataset(path_list, index_variables_retenus, header=True, field_delim=','): def csv_loader(path): return tf.data.experimental....
Jonathan Roy's user avatar
0 votes
0 answers
21 views

Datasets for various types of noisy images

I developed a neural network to learn to classify various kinds of noise in an image - salt and pepper, Gaussian, Poisson etc. I want to validate the model on unseen data. Are there any labelled ...
Curiosity's user avatar
  • 105
4 votes
2 answers
3k views

"No module named 'resource'" error when using TensorFlow and TensorFlow Datasets in Windows virtual environment

I am facing a problem when trying to use TensorFlow and TensorFlow Datasets in my virtual environment on Windows. When I try to import these packages, I receive the following error: ...
Babi's user avatar
  • 41
0 votes
0 answers
8 views

file path error, many images files merge with csv files

I want to show my problems where arise and what is my code by this code I want to know code what to wright in both pd.read_csv in file path and why all files are not tracked by version control by git, ...
Shiv Bhola Bind's user avatar
0 votes
0 answers
13 views

How to pass 2 tf datasets to .fit as sequence and tags?

I’ve loaded two datasets with sentences and labels from txt file with TextLineDataset, vectorized them by .map. Now I need to pass this data to model.fit. And I’ve found, that I can’t pass datasets ...
Mihail's user avatar
  • 21
0 votes
0 answers
34 views

Tensorflow: How to make a training/validation split when using strategy.distribute_datasets_from_function?

I want to make a dataset that can be distributed for MultiWorkerMirroredStrategy. I read the examples from TFRecords, so sharding has to be performed on the list of TFRecords files before reading them,...
hawo's user avatar
  • 31
0 votes
1 answer
27 views

Tensorflow Scalar Multiplication with Batches

I would like to create a tensorflow layer that takes in n inputs, n-1 are data points and the last is a weight vector of length n-1. Each of the (i in n-1) data points are then multiplied by the value ...
Max Conway's user avatar
0 votes
1 answer
35 views

TensorFlow Probability inference error input type

I train a model using tensorflow_probability and distributions. To train it I format my data like this (2 heads model so 2 inputs set): input_1 = tf.data.Dataset.from_tensor_slices(Xdata) input_2 = tf....
Jonathan Roy's user avatar
0 votes
0 answers
29 views

Train CNN with large dataset with data in a Google bucket

I'm trying to train a CNN on a segmentation task (classify all pixels in two classes: soil or not soil). In order to do this, I have input images with a lot of channels (approx. 47) and label masks. I ...
M. Merida-Floriano's user avatar
2 votes
1 answer
514 views

'str' object is not callable when fitting CNN model

I'm doing a deep learning project on detecting if a person has tuberculosis based on chest x-ray images. After reading and preprocess my data, I run into a problem when building my CNN model. import ...
Leo Tang's user avatar
0 votes
0 answers
9 views

Tensorflow Data Pipeline: Read text files from sub-directories for Seq2Seq Models

I am trying to create a data pipeline using Tensorflow's text_dataset_from_directory method for training a seq-to-seq model. The folder structure as below: BBC News Articles |_ News Articles |...
Krishnang K Dalal's user avatar
0 votes
0 answers
22 views

TensorFlow Conv2DTranspose's Type mismatch: actual int32 vs. expect string error on GPU

I am trying to train a variational autoencoder in TensorFlow which is causing a weird error that I cannot replicate on CPU with the same code. Relevant parts of my model - @tf.function def ...
Aditya Prakash's user avatar
0 votes
0 answers
24 views

How can I debug the data iterator in tensorflow?

I am using a tf.data.Iterator for my training in tensorflow with where the the get_next() function fails. Do anyone know how I can debug inside the get_next() function? The get_next() function fails ...
Sara Larsson's user avatar
0 votes
1 answer
52 views

Tensorflow memory problems with validation data

Im trying to train a tensorflow neural network. Since the training data is too big for my computer RAM I have divided it into sub-datsets and sequentially train them. I have a problem however, if I ...
Sergio Pardina's user avatar
0 votes
0 answers
19 views

Eumerate over two different tensorflow batchDatasets

I am trying to train a VAE to convert an images of one type into images of another type. The problem I am having is when I call to train the model, telling it that I want the after image to be ...
James Delles's user avatar
0 votes
0 answers
40 views

How do I look inside a column in a CacheDataset in TensorFow?

The following code was used to split the train and test set: tf.random.set_seed(42) shuffled = ratings.shuffle(100_000, seed=42, reshuffle_each_iteration=False) train = shuffled.take(80_000) test = ...
joankusuma's user avatar
2 votes
1 answer
45 views

TypeError when using ragged tensors in TensorFlow data pipeline

I'm attempting to create a tf.data pipeline using RaggedTensors in TensorFlow, but I keep encountering a TypeError: "Cannot convert the argument `type_value`: RaggedTensorSpec(TensorShape(\[None, ...
Agirones's user avatar
1 vote
0 answers
31 views

How to calculate F1 score for tensor flow dataset which is batched and prefetched (multi label image classification)

This is the code snippet for the datasets train_ds= tf.keras.preprocessing.image_dataset_from_directory( "./tmp/Alzheimer_s_Dataset/train/", validation_split=0.2, subset="training",...
jaggu ramesh's user avatar
0 votes
0 answers
29 views

tf.data.Dataset.from_generator save additional data from Python generator

I would like save additional data (date, file name etc.), which I don't use for training but will need it later to identify my predictions. This data is generated together with features and target by ...
Mykola Zotko's user avatar
  • 16.1k
0 votes
0 answers
19 views

Batching tf dataset correctly

I am following Keras text classification tutorial for sentiment analysis. Their data is structured in folders like so: test - pos - neg train - pos - neg where pos and neg are folders ...
Lex Podgorny's user avatar
  • 2,648
0 votes
0 answers
69 views

Tensorflow dataset input for encoder-decoder LSTM model

I'm trying to train an LSTM network in Tensorflow. But this gives me an error: ValueError: Layer "model_10" expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf....
Soprano's user avatar
  • 17
1 vote
1 answer
236 views

Load ImageNet with TensorFlow-Datasets tfds from a PyTorch Folder Structure

Dear ImageNet and TensorFlow specialists Originally, I have downloaded the ImageNet dataset tar files ILSVRC2012_img_train.tar, ILSVRC2012_img_val.tar and used this PyTorch script to extract the files....
Anna Christine's user avatar

1
2 3 4 5
42