Questions tagged [dataset]

A dataset is a collection of data, generally represented in tabular form, with columns signifying different variables and rows signify different members of the set. If you are looking for a freely available dataset for any purpose, please consider asking your question on https://opendata.stackexchange.com.

dataset
Filter by
Sorted by
Tagged with
8 votes
3 answers
12k views

value toDS is not a member of org.apache.spark.rdd.RDD

I am trying to write sample Apache Spark program that converts RDD to Dataset. But in that process, I am getting compile time error. Here is my sample code and error: code: import org.apache.spark....
user6325753's user avatar
8 votes
8 answers
29k views

Interpolation over an array (or two)

I'm looking for a java library or some help to write my own interpolation function. That is I have two arrays of doubles which are potentially different sizes, but are ordered. I need to be able to ...
Robert's user avatar
  • 8,448
8 votes
5 answers
19k views

Python - Pandas, Resample dataset to have balanced classes

With the following data frame, with only 2 possible lables: name f1 f2 label 0 A 8 9 1 1 A 5 3 1 2 B 8 9 0 3 C 9 2 0 4 C 8 1 0 5 ...
Shlomi Schwartz's user avatar
8 votes
3 answers
13k views

Generate a data set consisting of N=100 2-dimensional samples

How do I generate a data set consisting of N = 100 2-dimensional samples x = (x1,x2)T ∈ R2 drawn from a 2-dimensional Gaussian distribution, with mean µ = (1,1)T and covariance matrix Σ = (0.3 0.2 ...
pythonnewbie's user avatar
8 votes
6 answers
3k views

Is Dataset an ORM?

I am a little bit confused about Dataset compared to ORM (NHibernate or Spring.Net). From my understanding the ORM sits between the application layer and the database layer. It will generate the SQL ...
Fakhrul's user avatar
  • 121
8 votes
1 answer
15k views

How can I add a column with a value to a new Dataset in Spark Java?

So, I'm creating some Datasets from the java Spark API. These datasets are populated from hive table, using the spark.sql() method. So, after performing some sql operations (like joins), I have a ...
Juan Carlos Nuño's user avatar
8 votes
1 answer
9k views

What do the TensorFlow Dataset's functions cache() and prefetch() do?

I am following TensorFlow's Image Segmentation tutorial. In there there are the following lines: train_dataset = train.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat() train_dataset = ...
robertspierre's user avatar
8 votes
5 answers
74k views

How to read a dataset from a txt file in Python?

I have a dataset in this format: I need to import the data and work with it. The main problem is that the first and the fourth columns are strings while the second and third columns are floats and ...
Ewybe's user avatar
  • 405
8 votes
7 answers
1k views

What are best practices for collecting, maintaining and ensuring accuracy of a huge data set?

I am posing this question looking for practical advice on how to design a system. Sites like amazon.com and pandora have and maintain huge data sets to run their core business. For example, amazon (...
Kyle West's user avatar
  • 9,008
8 votes
2 answers
4k views

C# : WCF Service with IXMLSerializable member turns into DataSet

.NET I have a web service, one of the data members of a message implements IXmlSerializable, when I do "Add Service Reference" that member becomes a DataSet. I am trying to pass a serialized ...
THX-1138's user avatar
  • 21.5k
8 votes
2 answers
6k views

TF.data.dataset.map(map_func) with Eager Mode

I am using TF 1.8 with eager mode enabled. I cannot print the example inside the mapfunc. It when I run tf.executing_eagerly() from within the mapfunc I get "False" import os import tensorflow as tf ...
MAltakrori's user avatar
8 votes
1 answer
13k views

Unzip a 7z file in google colab?

I am trying to write a CNN on Kaggle’s Amazon from Space dataset. I can’t spend money now. So, I want to use Google colab. I have successfully downloaded the dataset using kaggle cli tool. However, I ...
lazyV's user avatar
  • 93
8 votes
5 answers
8k views

Data sets for realistic random/test data generation [closed]

Where to get data sets for random or test data generation, such as names/surnames with distribution, address data, university/school names, company names etc.? I've found the list of English names ...
Danubian Sailor's user avatar
8 votes
3 answers
13k views

SSRS run SQL/DataSet conditionally

I have an SSRS report that contains several sub-reports. The user has the ability to select/deselect which sub-reports they want to produce using several Boolean parameters. If a sub-report is ...
MikeTWebb's user avatar
  • 9,229
8 votes
2 answers
9k views

How to use random_split with percentage split (sum of input lengths does not equal the length of the input dataset)

I tried to use torch.utils.data.random_split as follows: import torch from torch.utils.data import DataLoader, random_split list_dataset = [1,2,3,4,5,6,7,8,9,10] dataset = DataLoader(list_dataset, ...
Pro Q's user avatar
  • 4,664
8 votes
5 answers
3k views

Error in importing Cats-vs-Dogs dataset in Google Colab

While trying to download the "Cats_vs_Dogs" TensorFlow dataset using the tfds module, I get the following error 👇 DownloadError Traceback (most recent call last) ...
IHackmer19's user avatar
8 votes
1 answer
2k views

Python multiprocess share memory vs using arguments

I'm trying to get my head around what is the most efficient and less memory consuming way to share the same data source between different process. Imagine the following code, that simplify my problem....
Alessandro Mariani's user avatar
8 votes
1 answer
13k views

How to display Date in a X-axis of Line Graph using Jfreechart

I am trying to display a Line Graph with Time(HH:MM:SS) as X-axis and Number(as Y-Axis). The read data from "Time" column is of the format HH:MM:SS. The way i am populating dataset from which chart ...
Prashant's user avatar
  • 1,178
8 votes
6 answers
16k views

Delphi - TDataSet determine if it was modified when is in insert/edit state

how can I find out if an data-aware component field has been modified when the dataset is already in Insert state? I want to know if a field was 'really' modified. (I don't care if the user has input ...
RBA's user avatar
  • 12.5k
8 votes
3 answers
5k views

Performance of accessing dataSet fields using Field-names instead of indexes

Is the performance negligible? For example, myQuery.FieldbyName("MyField").AsString; myQuery.Fields[0].AsString; Cases: Table with a decent number of fields, say > 50 fields Accessing large ...
Simon's user avatar
  • 9,367
8 votes
6 answers
24k views

Visual Studio DataSet Designer Refresh Tables

In visual studio datasource designer is there any way to refresh a table and its relations/foreign key constraints while keeping the custom queries? The way I am doing it at the moment is removing ...
Michal Ciechan's user avatar
8 votes
2 answers
14k views

How do I use automapper to map a dataset with multiple tables

DISCLAIMER: this is a copy paste from an older stackoverflow post that isn't available anymore, but I have exaclty the same problem, so it seemed appropriate to repost it as it was never answered. I ...
Arne Deruwe's user avatar
  • 1,100
8 votes
2 answers
13k views

Get the DefaultView DataRowView from a DataRow

Here's the situation: I need to bind a WPF FixedPage against a DataRow. Bindings don't work against DataRows; they work against DataRowViews. I need to do this in the most generic way possible, as ...
user avatar
8 votes
1 answer
7k views

Concurrency violation: the UpdateCommand affected 0 of the expected 1 records

Here is the code, any ideas why I get this error? private SQLiteDataAdapter DA_Webfiles; // Setup connection, fill dataset etc DataTable dt = this.dataSet.Tables["WEBFILES"]; DataRow newRow = dt....
Greg's user avatar
  • 34.5k
8 votes
4 answers
8k views

Datasets to test Nonlinear SVM

I'm implementing a nonlinear SVM and I want to test my implementation on a simple not linearly separable data. Google didn't help me find what I want. Can you please advise me where I can find such ...
Morano88's user avatar
  • 2,067
8 votes
1 answer
27k views

datasets with actual tablename

i am trying to get data from my stored proc into my dataset. The problem is that in the dataset visualizer the actual table name ie customers or employees does not show up just Table1 , table2 etc. Is ...
user603007's user avatar
  • 11.6k
8 votes
2 answers
8k views

Changing XSD ConnectionString at Runtime for a Multitenant app

I'm changing our application from "one set of code & one database" to "one set of code to multiple databases (one database per customer)". The original code is VS2005 ASP.NET(VB) & lots of ...
DomBat's user avatar
  • 2,033
8 votes
3 answers
5k views

Pytorch - Can not slice torchvision MNIST dataset

In Pytorch, when using torchvision's MNIST dataset, we can get a digit as follows: from torchvision import datasets, transforms from torch.utils.data import DataLoader, Dataset, TensorDataset tsfm = ...
u2gilles's user avatar
  • 7,171
8 votes
2 answers
2k views

GPU under utilization using tensorflow dataset

During training of my data, my GPU utilization is around 40%, and I clearly see that there is a datacopy operation that's using a lot of time, based on tensorflow profiler(see attached picture). I ...
Molly Zhang's user avatar
8 votes
3 answers
8k views

Example data sets in Matlab

There are several example data sets in Matlab, for example wind and mri. If you execute the command load wind you will load the data in the data set wind. Some are included in toolboxes and some ...
Martin Stålberg's user avatar
8 votes
1 answer
6k views

Dataset lost in google colab?

I am new to colab. I recently downloaded dataset from kaggle to google colab. In my next visit the dataset isn’t there and my kaggle installation is removed too. Does anyone know why ? I have tried ...
ankus7890's user avatar
  • 143
8 votes
1 answer
5k views

Best way to access and close a postgres database using python dataset

import dataset from sqlalchemy.pool import NullPool db = dataset.connect(path_database, engine_kwargs={'poolclass': NullPool}) table_f1 = db['name_table'] # Do operations on table_f1 db.commit()...
user308827's user avatar
  • 21.8k
8 votes
3 answers
38k views

C# Dataset to Access DB

I have a dataset that is dynamically created from a csv file. What I want to do is insert the rows into my MS Access table but I cannot figure out where to start with this. The headers of the data ...
Reg's user avatar
  • 565
8 votes
4 answers
6k views

SQL Server best method to match word phrases and order relevence

What is the best way to rank a sql varchar column by the number (count)/match of words in a parameter, with four distinct unique criteria. This is likely not a trivial question but I am challenged to ...
Mark Schultheiss's user avatar
8 votes
1 answer
2k views

How to switch between training and validation dataset with tf.MonitoredTrainingSession?

I want to use feedable iterator design in tensorflow Dataset API, so I can switch to validation data after some training steps. But if I switched to validation data, it will end the whole session. ...
Xiang Zhang's user avatar
  • 2,863
8 votes
1 answer
1k views

Split queue into train/test set

I set up my pipeline starting with a filename queue as in the following pseudocode: filename_queue = tf.train.string_input_producer(["file0.pd", "file1.pd"]) pointing to TFRecords containing ...
Manuel Schmidt's user avatar
8 votes
2 answers
5k views

Where can I find a good set of benchmark clustering datasets with ground truth labels?

I am looking for a clustering dataset with "ground truth" labels for some known natural clustering, preferably with high dimensionality. I found some good candidates here (http://cs.joensuu.fi/sipu/...
user3457088's user avatar
8 votes
2 answers
5k views

Tensorflow Dataset API not using GPU

1. Problem : I have a tf.data.Dataset that I give to a Keras model (tf.python.keras) with train_on_batch. My dataset looks like this : Generate TFRecord path > tf.data.TFRecordDataset > ...
Zelgunn's user avatar
  • 85
8 votes
0 answers
1k views

Spark 2.0: Moving from RDD to Dataset

I want to adapt my Java Spark app (which actually uses RDDs for some calculations) to use Datasets instead of RDDs. I'm new to Datasets and not sure how to map which transaction to a corresponding ...
D. Müller's user avatar
  • 3,366
8 votes
1 answer
472 views

How do I import a data set (using a roxygen directive)

In a "R" package that uses roxygen2, I use a built-in data set in the example code I provide. When building and checking the package, I get Consider adding importFrom("datasets", &...
user52366's user avatar
  • 1,087
8 votes
0 answers
189 views

What are the most robust and interactive-friendly ways to structure general 2D/3D/ND datasets in Python? [closed]

I am a scientist recently converted from MATLAB to Python. I am looking for ways to structure my (mainly 2D and 3D) datasets. I have searched the net quite a bit, and it seems to me that robust and ...
cmeeren's user avatar
  • 4,030
7 votes
4 answers
4k views

What's the purpose of Datasets?

I want to understand the purpose of datasets when we can directly communicate with the database using simple SQL statements. Also, which way is better? Updating the data in dataset and then ...
Lizzy's user avatar
  • 2,103
7 votes
11 answers
10k views

Is it better to use the column name or column index on .Net DataSets?

When retrieving values from a DataRow is it better to use the column name or column index? The column name is more readable and easier to maintain: int price = (int)dr["Price"]; While column index ...
tpower's user avatar
  • 56.6k
7 votes
1 answer
5k views

Handling XSD Dataset ConstraintExceptions

Does anyone have any tips for dealing with ConstraintExceptions thrown by XSD datasets? This is the exception with the cryptic message: System.Data.ConstraintException : Failed to enable constraints....
Jonathan Webb's user avatar
7 votes
2 answers
15k views

PyTorch: "KeyError: Caught KeyError in DataLoader worker process 0."

Problem Description I tried to load image data using a PyTorch custom dataset, however, I received the error message listed below. After its occurrence, I checked the data and found that my image set ...
Kasid Khan's user avatar
7 votes
5 answers
18k views

SAS - How to get last 'n' observations from a dataset?

How can you create a SAS data set from another dataset using only the last n observations from original dataset. This is easy when you know the value of n. If I don't know 'n' how can this be done?
Riyaz Iqbal's user avatar
7 votes
5 answers
33k views

Number of elements in Python Set

I have a list of phone numbers that have been dialed (nums_dialed). I also have a set of phone numbers which are the number in a client's office (client_nums) How do I efficiently figure out how many ...
Tim's user avatar
  • 311
7 votes
4 answers
10k views

Can I add the same DataRow to a DataTable multiple times?

If I have a DataTable and want to create a new row I first call DataTable.NewRow() which returns a DataRow. This row then has the schema of the Table, and I can set the value of each field in the row. ...
Eric Anastas's user avatar
7 votes
3 answers
4k views

What's the best way to store a Delphi set in a dataset?

The title pretty much says it all. I'm using a TClientDataset to store an array of objects, and one of the objects has a member defined as a set of an enumerated type. As I understand it, Delphi sets ...
Mason Wheeler's user avatar
7 votes
2 answers
13k views

How to use very large dataset in RNN TensorFlow?

I have a very large dataset: 7.9 GB of CSV files. 80% of which shall serve as the training data, and the remaining 20% shall serve as test data. When I'm loading the training data (6.2 GB), I'm having ...
afagarap's user avatar
  • 650

1
4 5
6
7 8
229