Questions tagged [bigdata]
Big data is a concept that deals with data sets of extreme volumes. Questions may tend to be related to infrastructure, algorithms, statistics, and data structures.
7,517
questions
-1
votes
0
answers
8
views
What is the core and memory benchmark for dedupe library for bigdata
I am using python "dedupe" library for large dataset. Initially we tested with 3K dataset and it could run with "AWS ml.r5.4xlarge 16vCPU 128 GiB RAM" within few mins.
The moment ...
1
vote
0
answers
21
views
Read huge file in variable chunks based on condition using python
I need to read a huge pipe separated file from s3 with below content:
Q|A|1|X
78|WQ|
123|ABC
Q|V|5|Y
LK|HJ|
BG|78
I want to read file in such a way that my data looks like this:
1|Q|A|1|X|
1|78|WQ|
1|...
0
votes
0
answers
21
views
Django: view tracking data with tens-of-thousands of views per day
I have a Django app with a Postgres backend. Users can make posts and view them.
"Views" sit around the high-tens-of-thousands to 100K+ per day total. We want to track views so that we can ...
-1
votes
0
answers
21
views
How to search for a string in multiple files with python? [closed]
I have a directory with many different files. It contains these file types: .pdf .xls .xlsm .jpg .msg. Its a huge data set (1000 files or more).
My goal is to code a script which searches for a ...
-1
votes
0
answers
32
views
Reading huge amount of data from database C# [closed]
I have historical stockdata in a MySQL database (probably not the most optimal DB but thats what I have). The data is divided into 120 different tables, one per month for the last 10 years, with about ...
-1
votes
0
answers
17
views
How can we convert a 60 gb sql file into a csv file using python [closed]
I have a 60 gb sql file which is further divided into further smaller sql files containing 5gb each with data to be analyzed for a bigdata project. I am not able to figure out an efficient way to ...
0
votes
1
answer
36
views
Filter data in pandas by a string date
I have a DataFrame that looks like this:
Date
Parameter
2010-01-02
some value
2010-01-03
some value
2010-01-04
some value
...
...
2011-01-02
some value
2011-01-03
some value
2011-01-04
some ...
0
votes
0
answers
22
views
How to look if a value is in a group of columns in Pyspark
I've a big dataframe with multiple columns, some of those columns are called col1:col35.
I'd like to know if a specif value is contained in ANY of those columns.
If 'RP' is in (col1:col35) then val= 1;...
0
votes
2
answers
26
views
flink tumbling window is not triggered (no watermark strategy)
Problem statement: stream events from kafka source. These event payloads are of string format. Parse them into Documents and batch insert them into DB every 5 seconds based on event time.
map() ...
0
votes
0
answers
14
views
Is it possible in R to calculate all eigenvalues of a very large symmetric n by n dense matrix in blocks to conserve RAM?
To provide some context, I work with DNA methylation data that even after some filtering can still consist of 200K-300K features (with much less samples, about 500). I need to do some operations on ...
0
votes
1
answer
20
views
Read line by line and saving matching column to a new document python - Beginne
I need to say that I am a beginner at coding and maybe this can be easy if you are more experienced but I couldn't find any problem similar to mine so I am sending here.
I have a huge document (~40 GB)...
0
votes
0
answers
41
views
ETL huge data files from s3 to snowflake
I am trying to build a pipeline that reads huge files from s3 and dump it into snowflake after some transformation logic. Files in s3 are unstructured with different types and numbers of column.
e.g.
...
0
votes
0
answers
24
views
How does NoSql handle the huge data requirements?
Analyze the need of advanced database Programming techniques in the current
environment where all the individuals and businesses are collecting the huge amount of data
which is of different types (...
0
votes
0
answers
31
views
Python - unpacking nested json
I have the following problem: I can not unpack a "Nested JSON"
To obtain this dataset, I used the following code:
a = pd.json_normalize(variavel, max_level=5)
a
Nested JSON
Take a look at ...
0
votes
0
answers
39
views
Read Clickhouse using Pyspark
I want to read my clickhouse table using pyspark.read.format() but I am not able to do it,
However, I can write using Yandex driver but am unable to Read, I have also used clickhouse-driver for the ...
1
vote
0
answers
27
views
Dataflow job has high data freshness and events are dropped due to lateness
I deployed an apache beam pipeline to GCP dataflow in a DEV environment and everything worked well. Then I deployed it to production in Europe environment (to be specific - job region:europe-west1, ...
-1
votes
0
answers
9
views
hi, I'm working on sales data and I'm trying to select data with multiple salesperson, like sales for a particular day for Dave, Simon and Joshua
I've tried using the WHERE clause but there seem to be a problem with adding more conditions, I've tried separating them using a comma, underscore and plus sign, none of it works
0
votes
1
answer
70
views
How can I create an EXTERNAL table with HIVE format in databricks
I am having a external table with below format in hive.
CREATE EXTERNAL TABLE cs_mbr_prov(
key struct<inid:string,......>,
memkey string,
ob_id string,
.....
)
ROW FORMAT SERDE
'...
-3
votes
0
answers
18
views
How to buy 5 TB storage on Google Cloud? [closed]
I am going to train Machine Learning model with 5 TB dataset and want to upload it into Google Cloud. How can I buy and which storage type do you recommend ?
0
votes
1
answer
30
views
Discard 200 random healthy instances
Discard 200 random healthy instances.
How do I implement this in Rstudio?
This is the data frame:
https://www.kaggle.com/code/jamaltariqcheema/model-performance-and-comparison/data
I tried this but I ...
0
votes
1
answer
13
views
how do i extract both the week and year off a date, and have it as week1, 2022 or week40, 2021
I initially extracted the weeks from a date as 1, 2, 3 using the extract clause but now, I want to have the year along with it as week1, 2022 or week 40, 2021
0
votes
1
answer
23
views
Recalculate historical data using Apache Beam
I have an Apache Beam streaming project that calculates data and writes it to the database, what is the best way to reprocess all historical records after a bug fix or after changing the way it ...
0
votes
0
answers
25
views
Combine two keys in Apache Beam
I have an Apache Beam streaming project that uses Combine.perKey(), I need to be able to merge entities from my admin tool (to point one entity to another one), how to combine two keys with calculated ...
0
votes
0
answers
9
views
I'm trying to use either a sumif or case clause to sum up the values in a data set
I have the total amount expected to be saved, the total amount saved, the principal amount expected to be saved and the principal amount saved, now I'm trying to sum up the values of the principal ...
-1
votes
3
answers
94
views
Fastest way in numpy to get distance of product of n pairs in array
I have N number of points, for example:
A = [2, 3]
B = [3, 4]
C = [3, 3]
.
.
.
And they're in an array like so:
arr = np.array([[2, 3], [3, 4], [3, 3]])
I need as output all pairwise distances in ...
0
votes
0
answers
8
views
How to read fmt file in pyspark?
I have a file class.fmt
How can I read this file using pyspark?
I don't have experience with this file format.
-1
votes
0
answers
19
views
Reporting within a complex Event Sourced system
I am working on a complex financial system which is event sourced. Recently, a need for a centralized "data lake" has presented itself for reporting purposes which means replicating data to ...
0
votes
1
answer
25
views
Azure Synapse Exception while reading table from synapse Dwh
While reading from table I'm getting
jdbc.SQLServerException : Create External Table As Sect statement failed as the path ####### could not be used for export.
Error Code :105005
-1
votes
0
answers
38
views
How to prevent dbt from creating an incremental model for the first time?
In our continuous integration setup, we have a dbt job in dbt cloud that runs every 3 hours and updates all of our incremental models. The problem is that when a new incremental table is introduced, ...
0
votes
0
answers
26
views
What is the best way to enhance query performance for a large database?
I'm working on an RDS MariaDB database which contains more than 270 highly-connected tables with more than 10 million records. The main goal is to enhance queries performance, the most important ...
0
votes
1
answer
52
views
How to prevent dbt from rebuilding incremental tables when source view's columns change?
We have the following structure in dbt:
[events (view), with column1, column2, column3]‒‒‒>[incremental1, using column1 from events]
|
...
1
vote
1
answer
19
views
Distributed databases: What if hardware fails
Let us take some kind of distributed highly available database like Redis, Cassandra, Mongo etc. Replication factor = 5. Read quorum = 3. Write quorum = 3. Now a write occurs, but only 3 out of 5 get ...
2
votes
1
answer
24
views
How does Cassandra store variable data types like text
assumption is, Cassandra will store fixed length data in column family. like a column family: id(bigint), age(int), description(text), picture(blob). Now description and picture have no limit. How ...
-3
votes
0
answers
37
views
How to transfer 1.2TB dataset from URL in Google Cloud?
I want to transfer the following dataset (1.2TB) into Google Cloud.
Is it doable using free version which gifts 300$? Can I do it using the console or is there some already available tool on Google ...
0
votes
1
answer
32
views
Matlab can't handle data
I've been working on the processing 2 lists of 855 4000x4000 matrices. Here is a list of 855 matrices of some value, another one is a list of coordinates (another 855 4000x4000 matrices). It's ...
3
votes
2
answers
65
views
Polymorphic data transformation techniques / data lake/ big data
Background: We are working on a solution to ingest huge sets of telemetry data from various clients. The data is in xml format and contains multiple independent groups of information which have a lot ...
0
votes
1
answer
25
views
Druid generate missing records
I have a data table in druid and which has missing rows and I want to fill them by generating the missing timestamps and adding the precedent row value.
This is the table in druid :
| __time ...
0
votes
0
answers
17
views
how to make faster to foreach, search then update more than 100k data?
i have console app for background job.
the app will do like this,
get data from database for the location we can call table A(have 100k data) and place to variable
then foreach data of location will ...
0
votes
1
answer
58
views
How to match the unique ids that I created in df1 to df2 based on two column values?
I have two dataframes, and I am struggling to match the unique ids that I created in df1 to df2 based on 'name' and 'version' values. I need to add a column to df2, let's call it ['ID'], whose values ...
0
votes
1
answer
25
views
What is the benefit of using more then 1 driver core in spark yarn cluster mode?
what is the difference in using 1 vs 2 driver core in spark yarn cluster mode? If i use 2 driver cores in yarn cluster mode, then spark driver will be relaunched incase of failure? If so, how many ...
3
votes
2
answers
74
views
JQ stop after the first match
I have a large file (about 500 megabytes) and the data in it is in JSON format.
{
"0001": [
"aaaaa",
"qqqqq"
],
"0002": [
&...
0
votes
0
answers
24
views
looking up billions of data (from millions of data)
I am developing a dataflow to transform ~4 million structured data (from a csv) daily.
for each row of the data, there is a key (64 characters) and a date field of any valid non-null value.
...
8
votes
0
answers
60
views
Enabling Hive Lineage
I thought Hive lineage was not available, but after some research I have found that it can be enable. Some of the things I found while searching was enabling its lineage via either Cloudera Manager or ...
0
votes
1
answer
19
views
Apache Pig : Counting frequencies of "character"
Assume there is a text file named abalone_data with 3 attributes : name, gender and length
Kn,M,0.89
Un,F,0.77
An,M,0.89
Az,I,0.55
Au,M,0.72
with M is male, F is female and I is infant.
The question ...
0
votes
1
answer
60
views
Data caching with ClickHouse
Intro
I have ClickHouse as data warehouse (tables with billions of rows).
Users interact with the DWH using my application backend that generates SQL queries to ClickHouse.
Different users can access ...
0
votes
0
answers
12
views
default filter report subscription?
Power bi report
"Email subscriptions are sent with the report's default filter and slicer states. Any changes to the defaults that you make before or after subscribing will not show up in the ...
0
votes
1
answer
22
views
Character-level TFIDF in PySpark?
I have a dataset that consists in 10M rows:
>>> df
name job company
0 Amanda Arroyo Herbalist Norton-...
0
votes
0
answers
15
views
Cloud solution to parse and process 1M+ rows
We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB.
What AWS services are best ...
0
votes
1
answer
49
views
How do I Split Huge File into Chunks of equal size in bash
I have a huge file of 4GB of data i.e 523 rows & 2,655,566 columns. I would like to read the whole file in equally divided chunks. How to do so, suggest the best approach
Here I have 522 lines ...
1
vote
1
answer
50
views
What's the point of SeaweedFS File Store?
According to GitHub, SeaweedFS is intended to be a simple and highly scalable distributed file system which enables you to store and fetch billions of files, fast. However, I don't understand the ...