Questions tagged [large-data]
Large data is data that is difficult to process and manage because its size is usually beyond the limits of the software being used to perform the analysis.
2,040
questions
0
votes
0
answers
10
views
Best way to run a function iteratively on file pairs from large dataset stored on permanent Dropbox link
I have a very large permanent Dropbox link which has 6 folders in it. Each folder has the following file organization (I specified the organization for Folder 1 only where - represents a folder and *...
0
votes
1
answer
53
views
Conversion of large character array to integer in C
I'm looking for a way to convert a very large character array to an integer. For reference, I need to be able to convert a character array holding a number as big as: 1718944270642558716715 to an ...
0
votes
0
answers
35
views
How to take advantage of qgam and bam to deal with non-Gaussian distribution and tensor product interaction on a large data set?
I'd like to model the nonlinear relationship between y0 (response) and the interaction of two predictors xa and xb.
However, I've two problems:
a) y0 is not normally distributed (see below, rather ...
0
votes
0
answers
21
views
performant gain calculation in large mongodb database
In a large mongodb database, how do you calculate, in a pipeline, a change in value over time in a subset of documents?
The dataset has 100 million documents. Schema is something like:
_id: long ...
0
votes
0
answers
15
views
Missing name of the law/principle/rule/theorem
Some time ago I encountered a law/principle/rule/theorem (perhaps on Wikipedia). The content was similar to the law of large numbers, the point was like:
[...] Some phenomenon may be revealed only ...
0
votes
0
answers
19
views
while loop error which only occurs with a big data frame
I am trying to randomly sample lines from a very big text file.
When I run my code on a small subset it works like a charm but fails when the real corpus (44mio lines) is used instead.
My code is as ...
0
votes
1
answer
43
views
split large xml data by python
I have large xml file and I need to split into smaller size when the tag is comment and has something like case description. I need to do it in python
I have written this to read the large xml file ...
2
votes
1
answer
35
views
Is there a way to run a regression with both characters and integers?
I am trying to run regression on a really large data set from IPUMS, but I am having a hard time doing anything with the data because it includes both integers and characters. Additionally, I am ...
0
votes
1
answer
19
views
How to open CSV fle with more than 2GB?
I'm trying to open a CSV file with more than 2 GB, but when I open it, all the cells are empty. Is there any way to open it .
I can open it By R , but I want to see the data in excel.
0
votes
0
answers
8
views
Matrix is too large in Fclust
I am running the following code in the Fclust package:
fpc_1 = FKM(fuzzy_cluster_1_allyears, m = 1.2, k = 5, RS = 10, maxit = 100)
it is then coming up with the following error.
Error: Mat::init(): ...
0
votes
0
answers
83
views
Compare Two Large JSON array
I have two directories "A" and "B", each directory contains textfiles containing jsonarray of jsonobjects.
Need to compare jsonobjects in directory a with b and figure out which ...
0
votes
0
answers
66
views
Is the Python garbage collection not working? Problematic processing of enormous XML files
I'm trying to process a 4.6GB XML file with the following code:
context = ET.iterparse(file_name_data, events=("start", "end"))
in_pandcertificaat = False
pandcertificaat = {}
...
0
votes
1
answer
42
views
How do I pad a pyspark column array like numpy?
I need to make a copy of a column on pyspark and pad it 1 at the top to shift the column down by 1. Here is an example:
old df
ID | count|
+----------+--------+
| 11| 1666030|
| ...
-2
votes
0
answers
41
views
Is there any techniques for saving CPU memory when training model with large dataset?
I am looking for ways to train a Timesformer model (that I built with TensorFlow, Keras, and Einops) for action recognition in videos in colab. The dataset has 12 classes and consists of around 5.4k ...
0
votes
2
answers
49
views
Merge many data frames by row names
I am working on a project where I have 50 databases with the same structure.
Think of it the following way
df_1 <- data.frame(nationality=c("german","italian"), total=c(500,300),...
0
votes
0
answers
15
views
I need Virtual machine to download and upload 40 TB
I need to test the scalability of a web app.The data set is on google drive so I have to download it (which is up to 40 TB) and then I have to upload it to this web app. What is the best way and best ...
0
votes
0
answers
35
views
Python SQLite3 Simultaneous Access of Large DB
I have a large dataset (~300GB) and access to a machine which can fit it into RAM (but not multiple times). I need to access sections of this dataset, perform calculations on them and then write the ...
0
votes
0
answers
30
views
pandas python how to consider number of rows with another defined number
I have 1_000_000 rows of data that only 5_000 of them are not zero and rest of the rows are 0. Som I just added 5000 rows to my Dataframe. Now let me know is there any possible command in pandas to ...
0
votes
0
answers
31
views
PostgreSQL, INSERT INTO SELECT DISTINCT generates more rows?
I have a large PostgreSQL table (NOT indexed) with duplicates so I ran the following query to eventually end up with a unique rows table:
INSERT INTO newTable(Field1, Field2, Field3)
SELECT DISTINCT ...
0
votes
1
answer
36
views
get the inheritance of an element in the XML using python
Please look at the picture for reference exampleI have a large complex XML structure and I need to find the inheritance of specific elements when I type in the element name. I need Python code to do ...
0
votes
0
answers
14
views
themis::tomek() returning 0 observations
I have an imbalanced dataset of 227846 observations and 30 columns and I would like to apply smote from the smotefamily library and then tomek from the themis library on the smote data to make the ...
0
votes
1
answer
90
views
Load large json file (100GB+) in rust
I am looking to load a large json file over 100GB+. The objects in this file aren't static and are almost never the same. I found this crate called nop-json https://crates.io/crates/nop-json/2.0.5. ...
1
vote
1
answer
52
views
OutOfMemoryError when trying to find the largest kth element in a large array
I am trying to solve a task that asks me to find the largest kth element in an unsorted array of length n (n might be as large as 5,000,000; elements in the array are distinct). Due to the task limits,...
0
votes
0
answers
88
views
Pandas or any other way to run this fastest?
dfA:
Var1
Var2
Var3
Var4
12.5
13.2
4.56
7.98
14.1
11.9
5.76
8.91
12.9
10.0
7.0
9.2
dataframe A above contains 10 Million rows
listB:
[ df1, df2, df3, df4]
listB above contains 4 dataframes with ...
2
votes
1
answer
35
views
Subsetting rows while searching for 4 specific values in multiple columns
I'm working with a large datatable (~1'000'000 x 700) and I want to subset the rows according to 4 specific values across multiple columns.
As I'm a beginner R-User I need help to achieve this.
The ...
0
votes
1
answer
23
views
Add text in in front of millions of lines with Linux?
I want to add important@ to 159 millions of lines.
I learned it's possible with sed on Linux.
Guide me if possible.
I know its possible to add at the end with:
sed 's/$/ | EXAMPLE/' file
0
votes
1
answer
51
views
Refactoring help for ORA-14300 partitioning key maps to a partition outside maximum permitted number of partitions
Last year i inherited a 15 year old / 250K lines of code Oracle system from a colleague who retired. It's been running flawlessly, but yesterday it completely imploded with hundreds of error mails. I ...
1
vote
1
answer
40
views
How do I make a new dataframe with data from a previous dataframe
I have the data shown in the screenshot. I want to create a new panda with the column headers of the cells in the forces column in the screenshot and I want the respective values to be listed in each ...
1
vote
0
answers
27
views
Laravel Nova How to display a resource with millions of rows?
I have a resource with 5 million rows in an InnoDB table. Nova times out when executing a COUNT(*) query. I imagine this is used in pagination. Is there a way to disable that behavior (just have a ...
0
votes
0
answers
16
views
Can i use biglm and/or ff packages for random effects estimation in R?
I have a very large panel in R, tryed to perform a plm regression and received the error "cannot allocate vector of size 11 Gb". I found out that regression in chunks could be a solution and ...
0
votes
0
answers
12
views
Object to handle storange and matching large set of genomic string sequences
I am working in R.
I have a large set of 20-nucleotide DNA sequence strings (~60 million). Currently I just keep them in a matrix of strings.
I need to be able to store them as efficiently as possible ...
3
votes
2
answers
100
views
Read large text files in C efficiently
I need to read large text files containing either real or complex data in C. As of right now, I am using the code shown below. These functions are simple to understand and work fine for small sizes. ...
0
votes
0
answers
52
views
How to shorten nested for loops and if conditions coming after each other in Python?
I have a Python script, that collects matching 5x5 crosswords from a 10000 words dataset (5 letter words). It is working, but it would take a year to be finished. Is there a way to make this code more ...
0
votes
0
answers
15
views
What file format is recommended to store large size of training data and works with PyTorch Dataset better?
I have a zipped radar observations with size of 110GB.
The structure of zipped data looks like below
year_1.tar/
date_1.gz/
time_1
time_2
...
time_365
date_2.gz/...
0
votes
0
answers
13
views
Pull Information by Staff Member
I'm taking over a project for a research assistant, and she had created a spreadsheet to track staff member names, their direct supervisor, FTE (Full Time Equivalence), Status, Reason for resignation, ...
0
votes
0
answers
11
views
Python structure for working with variable log messages in a large file
I am trying to get data out of debug log messages created by a certain piece of open source software. It has many lines describing what it is doing during stages. It does not have a specific structure,...
0
votes
2
answers
68
views
Update gaps in sequential table
I have a table that contains employee bank data
Employee |Bank |Date |Delta
---------------------------------------------------
Smith |Vacation |2023-01-01 |15....
0
votes
1
answer
52
views
Crop large GeoJson layer using a standard machine
I need to extract a portion of a large GeoJson layer. I've tried multiple times with QGis on a 16Gb Ram intel i7 11th gen processor with no result. After about one hour if the program does not crash, ...
0
votes
0
answers
7
views
Is there an efficient way to combine dcast, melt and merge on large dataset in R?
I have 4 csv files of 63290 obs and 6 columns each. I transform them into datatable format. I am trying to use dcast and melt on each to create a count variable. But I am getting this error message: ...
0
votes
0
answers
22
views
[MongoDb][net core]Improve performance of Count and Group By for more than 50 million record
I'm using MongoDB.Driver(.net core) with MongoDB. I'm writing aggregate data and I see that mongo count and group is very very slow. My API run around 2 minutes.
Is any way to improve the performance ...
0
votes
0
answers
19
views
Pycaret data_func usage
What is the proper setup to use data_func?
I have attempted to use list comprehension which is not allowed as lists are not callable.
I have attempted to use pandas as a generator but generator ...
0
votes
1
answer
20
views
Table partitioning vs. splitting the table and execution with multiple connections to SQL Server?
I have to process a very large table (100MM+ rows) and perform bunch of analytical functions on it, e.g. taking average, sum, percentile, etc.
I was wondering if the best approach is to partition the ...
3
votes
1
answer
58
views
Read JSON file with nested lists in R
I have a large json dataset and I would like to convert it to a data frame in R
(Sorry if it may be a duplicated question but other answers didn't help me)
My Json file is as follows:
[{"src"...
0
votes
0
answers
53
views
MongoDB performance with many large documents in a collection
What performance considerations are there for storing and querying around 100 million documents consisting of around 500,000 floating point, integer and (short) string attributes each in a single ...
0
votes
0
answers
92
views
How to do zonal statistics for large size rasters in Python?
My goal is to do zonal statistics for hundreds of rasters using the same shaplefile and save the results as a csv file. I have wrote a code and it works fine for small size test shaplefile but not my ...
0
votes
0
answers
25
views
Loading huge json document
I have a large JSON file about 60gb. I am trying to view its contents using JSON package in python. But when I run it the system runs out of memory. I just want to view like a few (like 5) records. I ...
0
votes
0
answers
19
views
Batching large input file into MLlib model
Is there any way to batch a large input file (111MB) made of 22 MLN cells (222 rows for 110k columns) in MLlib (something similar to this tutorial made in keras) Keras batching tutorial.
The file ...
0
votes
3
answers
74
views
Using Elasticsearch to collect information from Yahoo-Finance query server
I have been trying to query yahoo finance using the following url:
https://query1.finance.yahoo.com/v1/finance/lookup?formatted=true&lang=en-US®ion=US&query=A*&type=equity&count=...
0
votes
0
answers
48
views
tsfresh efficient feature set extraction is stuck at 0%
I am using the tsfresh library to extract features from my time-series dataset.
My problem is that I can only use the setting for the minimal feature set (MinimalFCParameters) because even the ...
0
votes
0
answers
32
views
How to use break last json while requesting for pagination through js
I have a large Json in which there are 400-500k urls. i want to create HTML using json in which 1000 URLs in each page. The problem is that we need to load whole json at that time can we somehow load ...