Questions tagged [large-data]

Large data is data that is difficult to process and manage because its size is usually beyond the limits of the software being used to perform the analysis.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Best way to run a function iteratively on file pairs from large dataset stored on permanent Dropbox link

I have a very large permanent Dropbox link which has 6 folders in it. Each folder has the following file organization (I specified the organization for Folder 1 only where - represents a folder and *...
0 votes
1 answer
53 views

Conversion of large character array to integer in C

I'm looking for a way to convert a very large character array to an integer. For reference, I need to be able to convert a character array holding a number as big as: 1718944270642558716715 to an ...
0 votes
0 answers
35 views

How to take advantage of qgam and bam to deal with non-Gaussian distribution and tensor product interaction on a large data set?

I'd like to model the nonlinear relationship between y0 (response) and the interaction of two predictors xa and xb. However, I've two problems: a) y0 is not normally distributed (see below, rather ...
  • 169
0 votes
0 answers
21 views

performant gain calculation in large mongodb database

In a large mongodb database, how do you calculate, in a pipeline, a change in value over time in a subset of documents? The dataset has 100 million documents. Schema is something like: _id: long ...
  • 43
0 votes
0 answers
15 views

Missing name of the law/principle/rule/theorem

Some time ago I encountered a law/principle/rule/theorem (perhaps on Wikipedia). The content was similar to the law of large numbers, the point was like: [...] Some phenomenon may be revealed only ...
0 votes
0 answers
19 views

while loop error which only occurs with a big data frame

I am trying to randomly sample lines from a very big text file. When I run my code on a small subset it works like a charm but fails when the real corpus (44mio lines) is used instead. My code is as ...
0 votes
1 answer
43 views

split large xml data by python

I have large xml file and I need to split into smaller size when the tag is comment and has something like case description. I need to do it in python I have written this to read the large xml file ...
  • 1
2 votes
1 answer
35 views

Is there a way to run a regression with both characters and integers?

I am trying to run regression on a really large data set from IPUMS, but I am having a hard time doing anything with the data because it includes both integers and characters. Additionally, I am ...
0 votes
1 answer
19 views

How to open CSV fle with more than 2GB?

I'm trying to open a CSV file with more than 2 GB, but when I open it, all the cells are empty. Is there any way to open it . I can open it By R , but I want to see the data in excel.
0 votes
0 answers
8 views

Matrix is too large in Fclust

I am running the following code in the Fclust package: fpc_1 = FKM(fuzzy_cluster_1_allyears, m = 1.2, k = 5, RS = 10, maxit = 100) it is then coming up with the following error. Error: Mat::init(): ...
0 votes
0 answers
83 views

Compare Two Large JSON array

I have two directories "A" and "B", each directory contains textfiles containing jsonarray of jsonobjects. Need to compare jsonobjects in directory a with b and figure out which ...
  • 3
0 votes
0 answers
66 views

Is the Python garbage collection not working? Problematic processing of enormous XML files

I'm trying to process a 4.6GB XML file with the following code: context = ET.iterparse(file_name_data, events=("start", "end")) in_pandcertificaat = False pandcertificaat = {} ...
  • 715
0 votes
1 answer
42 views

How do I pad a pyspark column array like numpy?

I need to make a copy of a column on pyspark and pad it 1 at the top to shift the column down by 1. Here is an example: old df ID | count| +----------+--------+ | 11| 1666030| | ...
  • 77
-2 votes
0 answers
41 views

Is there any techniques for saving CPU memory when training model with large dataset?

I am looking for ways to train a Timesformer model (that I built with TensorFlow, Keras, and Einops) for action recognition in videos in colab. The dataset has 12 classes and consists of around 5.4k ...
0 votes
2 answers
49 views

Merge many data frames by row names

I am working on a project where I have 50 databases with the same structure. Think of it the following way df_1 <- data.frame(nationality=c("german","italian"), total=c(500,300),...
0 votes
0 answers
15 views

I need Virtual machine to download and upload 40 TB

I need to test the scalability of a web app.The data set is on google drive so I have to download it (which is up to 40 TB) and then I have to upload it to this web app. What is the best way and best ...
  • 1
0 votes
0 answers
35 views

Python SQLite3 Simultaneous Access of Large DB

I have a large dataset (~300GB) and access to a machine which can fit it into RAM (but not multiple times). I need to access sections of this dataset, perform calculations on them and then write the ...
0 votes
0 answers
30 views

pandas python how to consider number of rows with another defined number

I have 1_000_000 rows of data that only 5_000 of them are not zero and rest of the rows are 0. Som I just added 5000 rows to my Dataframe. Now let me know is there any possible command in pandas to ...
0 votes
0 answers
31 views

PostgreSQL, INSERT INTO SELECT DISTINCT generates more rows?

I have a large PostgreSQL table (NOT indexed) with duplicates so I ran the following query to eventually end up with a unique rows table: INSERT INTO newTable(Field1, Field2, Field3) SELECT DISTINCT ...
  • 1
0 votes
1 answer
36 views

get the inheritance of an element in the XML using python

Please look at the picture for reference exampleI have a large complex XML structure and I need to find the inheritance of specific elements when I type in the element name. I need Python code to do ...
0 votes
0 answers
14 views

themis::tomek() returning 0 observations

I have an imbalanced dataset of 227846 observations and 30 columns and I would like to apply smote from the smotefamily library and then tomek from the themis library on the smote data to make the ...
0 votes
1 answer
90 views

Load large json file (100GB+) in rust

I am looking to load a large json file over 100GB+. The objects in this file aren't static and are almost never the same. I found this crate called nop-json https://crates.io/crates/nop-json/2.0.5. ...
1 vote
1 answer
52 views

OutOfMemoryError when trying to find the largest kth element in a large array

I am trying to solve a task that asks me to find the largest kth element in an unsorted array of length n (n might be as large as 5,000,000; elements in the array are distinct). Due to the task limits,...
0 votes
0 answers
88 views

Pandas or any other way to run this fastest?

dfA: Var1 Var2 Var3 Var4 12.5 13.2 4.56 7.98 14.1 11.9 5.76 8.91 12.9 10.0 7.0 9.2 dataframe A above contains 10 Million rows listB: [ df1, df2, df3, df4] listB above contains 4 dataframes with ...
  • 41
2 votes
1 answer
35 views

Subsetting rows while searching for 4 specific values in multiple columns

I'm working with a large datatable (~1'000'000 x 700) and I want to subset the rows according to 4 specific values across multiple columns. As I'm a beginner R-User I need help to achieve this. The ...
0 votes
1 answer
23 views

Add text in in front of millions of lines with Linux?

I want to add important@ to 159 millions of lines. I learned it's possible with sed on Linux. Guide me if possible. I know its possible to add at the end with: sed 's/$/ | EXAMPLE/' file
0 votes
1 answer
51 views

Refactoring help for ORA-14300 partitioning key maps to a partition outside maximum permitted number of partitions

Last year i inherited a 15 year old / 250K lines of code Oracle system from a colleague who retired. It's been running flawlessly, but yesterday it completely imploded with hundreds of error mails. I ...
1 vote
1 answer
40 views

How do I make a new dataframe with data from a previous dataframe

I have the data shown in the screenshot. I want to create a new panda with the column headers of the cells in the forces column in the screenshot and I want the respective values to be listed in each ...
1 vote
0 answers
27 views

Laravel Nova How to display a resource with millions of rows?

I have a resource with 5 million rows in an InnoDB table. Nova times out when executing a COUNT(*) query. I imagine this is used in pagination. Is there a way to disable that behavior (just have a ...
0 votes
0 answers
16 views

Can i use biglm and/or ff packages for random effects estimation in R?

I have a very large panel in R, tryed to perform a plm regression and received the error "cannot allocate vector of size 11 Gb". I found out that regression in chunks could be a solution and ...
0 votes
0 answers
12 views

Object to handle storange and matching large set of genomic string sequences

I am working in R. I have a large set of 20-nucleotide DNA sequence strings (~60 million). Currently I just keep them in a matrix of strings. I need to be able to store them as efficiently as possible ...
3 votes
2 answers
100 views

Read large text files in C efficiently

I need to read large text files containing either real or complex data in C. As of right now, I am using the code shown below. These functions are simple to understand and work fine for small sizes. ...
  • 340
0 votes
0 answers
52 views

How to shorten nested for loops and if conditions coming after each other in Python?

I have a Python script, that collects matching 5x5 crosswords from a 10000 words dataset (5 letter words). It is working, but it would take a year to be finished. Is there a way to make this code more ...
  • 1
0 votes
0 answers
15 views

What file format is recommended to store large size of training data and works with PyTorch Dataset better?

I have a zipped radar observations with size of 110GB. The structure of zipped data looks like below year_1.tar/ date_1.gz/ time_1 time_2 ... time_365 date_2.gz/...
0 votes
0 answers
13 views

Pull Information by Staff Member

I'm taking over a project for a research assistant, and she had created a spreadsheet to track staff member names, their direct supervisor, FTE (Full Time Equivalence), Status, Reason for resignation, ...
0 votes
0 answers
11 views

Python structure for working with variable log messages in a large file

I am trying to get data out of debug log messages created by a certain piece of open source software. It has many lines describing what it is doing during stages. It does not have a specific structure,...
0 votes
2 answers
68 views

Update gaps in sequential table

I have a table that contains employee bank data Employee |Bank |Date |Delta --------------------------------------------------- Smith |Vacation |2023-01-01 |15....
0 votes
1 answer
52 views

Crop large GeoJson layer using a standard machine

I need to extract a portion of a large GeoJson layer. I've tried multiple times with QGis on a 16Gb Ram intel i7 11th gen processor with no result. After about one hour if the program does not crash, ...
  • 432
0 votes
0 answers
7 views

Is there an efficient way to combine dcast, melt and merge on large dataset in R?

I have 4 csv files of 63290 obs and 6 columns each. I transform them into datatable format. I am trying to use dcast and melt on each to create a count variable. But I am getting this error message: ...
  • 1
0 votes
0 answers
22 views

[MongoDb][net core]Improve performance of Count and Group By for more than 50 million record

I'm using MongoDB.Driver(.net core) with MongoDB. I'm writing aggregate data and I see that mongo count and group is very very slow. My API run around 2 minutes. Is any way to improve the performance ...
0 votes
0 answers
19 views

Pycaret data_func usage

What is the proper setup to use data_func? I have attempted to use list comprehension which is not allowed as lists are not callable. I have attempted to use pandas as a generator but generator ...
0 votes
1 answer
20 views

Table partitioning vs. splitting the table and execution with multiple connections to SQL Server?

I have to process a very large table (100MM+ rows) and perform bunch of analytical functions on it, e.g. taking average, sum, percentile, etc. I was wondering if the best approach is to partition the ...
  • 131
3 votes
1 answer
58 views

Read JSON file with nested lists in R

I have a large json dataset and I would like to convert it to a data frame in R (Sorry if it may be a duplicated question but other answers didn't help me) My Json file is as follows: [{"src"...
  • 57
0 votes
0 answers
53 views

MongoDB performance with many large documents in a collection

What performance considerations are there for storing and querying around 100 million documents consisting of around 500,000 floating point, integer and (short) string attributes each in a single ...
0 votes
0 answers
92 views

How to do zonal statistics for large size rasters in Python?

My goal is to do zonal statistics for hundreds of rasters using the same shaplefile and save the results as a csv file. I have wrote a code and it works fine for small size test shaplefile but not my ...
0 votes
0 answers
25 views

Loading huge json document

I have a large JSON file about 60gb. I am trying to view its contents using JSON package in python. But when I run it the system runs out of memory. I just want to view like a few (like 5) records. I ...
0 votes
0 answers
19 views

Batching large input file into MLlib model

Is there any way to batch a large input file (111MB) made of 22 MLN cells (222 rows for 110k columns) in MLlib (something similar to this tutorial made in keras) Keras batching tutorial. The file ...
0 votes
3 answers
74 views

Using Elasticsearch to collect information from Yahoo-Finance query server

I have been trying to query yahoo finance using the following url: https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US&region=US&query=A*&type=equity&count=...
0 votes
0 answers
48 views

tsfresh efficient feature set extraction is stuck at 0%

I am using the tsfresh library to extract features from my time-series dataset. My problem is that I can only use the setting for the minimal feature set (MinimalFCParameters) because even the ...
  • 819
0 votes
0 answers
32 views

How to use break last json while requesting for pagination through js

I have a large Json in which there are 400-500k urls. i want to create HTML using json in which 1000 URLs in each page. The problem is that we need to load whole json at that time can we somehow load ...

1
2 3 4 5
41