Questions tagged [bigdata]
Big data is a concept that deals with data sets of extreme volumes. Questions may tend to be related to infrastructure, algorithms, statistics, and data structures.
7,908
questions
0
votes
0
answers
7
views
Teradata Fastload utility not working in Spark Scala as more number of sessions are created
Eventhough Passing Sessions Parameter in JDBC Url. Number of fastload connections are created more than mentioned sessions. How to control the number of sessions when using fastload utility to load ...
-2
votes
0
answers
9
views
How to drools share memmory in multi node
i am new to flink and drools
I have a problem like: count the number of events with souce_ip within 10 minutes, if count > 5 => alert
i use apache flink and drools. flink to receive data from ...
-1
votes
0
answers
14
views
How to Drools share state in multi node
I am a new flink,drools.
I must handle event stream.
Example: count event same source_ip in 10 minute. If count > 5 => alert
I use apache flink combine Drools. I build a flink application, and ...
0
votes
1
answer
40
views
How to efficiently read a large property as a pandas DataFrame using SQLAlchemy?
I'm working on a project where I have a large dataset stored in a relational database, specifically a "PriceHistory" table containing historical stock prices. I also have a "Stocks"...
0
votes
0
answers
42
views
fuzzy matching millions of records
I am trying do a fuzzy matching between two columns in which col="co_zip22" will iterate through all the rows in col="co_zip23" and will find a match with a match score
So ...
0
votes
4
answers
79
views
What is the fastest way to read a csv file sort the data then write the sorted data into another csv
I have a massive data set ~600Gb of several csv files. Each csv file contains 1.3mil x 17 sets of data. It looks like this
index duration is_buy_order issued location_id ...
-4
votes
0
answers
96
views
Write a query for an sql query for a billion-record table [closed]
`Hi all
for a project I am working on I was asked to solve this 'big data' task:
Given the tables:
events
event_id int (autoincrement) --**10B distinct values**
event_ts datetime -- **10B**
...
0
votes
0
answers
51
views
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 261727: invalid start byte
I appreciate your help.
My question is method about sorting big data(csv) with 3rd column.
It operates normally up to 1 million lines input data, but for inputs longer than that, the error message ...
0
votes
0
answers
36
views
Import 1M data using laravel-excel and queue
Thanks in Advance for your help.
I am using laravel version 7 and laravel-excel package for importing big data CSV file. the problem I am facing is that the laravel telescope shows that the job has ...
-1
votes
0
answers
51
views
PyArrow Table to Spark DataFrame
import pyarrow.parquet as pq
table = pq.read_table(parquet_blob_data)
Now how to create a spark dataframe from table without using pandas library or to_pandas() in-built method?
I found only ...
0
votes
0
answers
11
views
Collecting data via windowing doesn't scale
I have flat table with a large number of rows. Some of these rows deemed to be important for some reason. I want to collect certain fields from rows around these rows in a window. I have a working ...
1
vote
1
answer
38
views
How to use filter function for broadcast variable in Pyspark
When I am trying to filter broadcasted value with below command is not working. Please help me with the correct approach
from pyspark.sql import *
spark=SparkSession.builder.appName("broadcast ...
0
votes
0
answers
14
views
Running cucumber BDD in GCP for Dataproc spark job testing
Currently my BDD is running on On-premise Hadoop cluster where it establishes JDBC connection and runs BDDs. Following are the steps done through BDD.
Insert required data in Hive and Hbase tables
...
0
votes
0
answers
22
views
Spark executors getting idle timeout
I have an EMR cluster where I am running pyspark jobs with spark dynamic allocation.
spark.dynamicAllocation.enabled=true
Below is config of cluster
1 Node
128GB Ram
10 cores
Core Nodes
Autoscaled ...
0
votes
1
answer
23
views
Hive query to find a month for each year in which maximum number of quantities were sold
I have a data like below, from which i am trying to find out which month for each year has max quantity ordered
if the data is not showing clearly i have added a snip of it
QUANTITYORDERED QTR_ID ...
0
votes
0
answers
24
views
How to do datamanagement on data chunks
structure(list(ano_date = c("0", "0", "0", "0", "0", "0", "0",
"0", "0", "0", "159", &...
0
votes
1
answer
46
views
Bulk replace nan in one column based on existing data from others [duplicate]
Let's say we have the following dataframe below is just an example with tons of rows
name
platform
year_of_release
genre
na_sales
eu_sales
jp_sales
other_sales
critic_score
user_score
rating
Madden ...
0
votes
0
answers
38
views
Updating or Fetching Big Data from Mongodb
I have around 1.9m record in Mongodb collection leads. Sample document is below:
{
"_id": {
"$oid": "60174f6cc3"
},
"customer_id": "590583",
...
0
votes
0
answers
20
views
Nifi: DeleteAzureDataLakeStorage blocked
I have a DeleteAzureDataLakeStorage processor but in some executions it crashes.
In data provenance only view:
Sometimes it runs in seconds and other times it hangs indefinitely.
In the apache logs ...
0
votes
0
answers
24
views
How to maintain online statistics for big data?
I have a DB with relativly (for our capacity) large rows ~ 45000000. I have ~ 500-1000 users who need online statistic from this DB by different criterias(client, priority, etc).
In general, ...
0
votes
1
answer
100
views
Why are spark3 dynamic partitions slow to write to hive
Question 1:
I have a table with a small amount of data, but there are a lot of dynamic partitions in the daily writes, the original spark2 writes can be solved in only 2 minutes, but after upgrading ...
0
votes
0
answers
15
views
Oozie Coordinator Jobs are getting killed automatically without much information
Oozie Coordinators in Prod environment are automatically getting killed with below logs:
Till 27th Aug Everything has run fine, on 28th Aug 09:00 the job is scheduled to run but job gets killed 20mins ...
0
votes
1
answer
50
views
Where can i find open medical datasets in HL7/FHIR format?
I need open data in HL7/FHIR format for a big data project. I try many portals, but until now i didn't find anything interesting. I'm looking for imaging, EHR and biosignals of any type (ECG, EEG, EMG ...
0
votes
1
answer
32
views
Understanding GDELT
I'm new to Gdelt world and i try to understand 3 databases, namely events, Mentions, GKG. they can be combined to each other through their ID. but my question is regarding Mentions database.
where ...
0
votes
0
answers
28
views
How to detect character encoding from stream
I want to get the character encoding of large file data acquired by stream in js (e.g. SJIS, UTF8).
I could get it by converting stream to buffer, but I can't convert it to buffer if the data is about ...
-1
votes
1
answer
46
views
Edit filter function on spark
This is my code for filtering out all the rows in the spark dataframe, the columns of which have rarely encountered values, this code is working and the number of rows really decreases (I need it for ...
0
votes
0
answers
15
views
PIG MAP_ONLY Job failing with Datanode Going OOM Error
I Have application Audit stored in HDFS for day 09 to 16. All this data have total 23 Fields. Two of them is Event_ID and Status. Event_ID can be 1 to 10. And Status can be REQUEST, SUCCESS, FAILURE.
[...
0
votes
0
answers
23
views
I am unable to connect twitter source data
`Hello Team,
I hope you are good. Actually, I am in need of an assistance regarding Apache flume.
To give you the context, I am newly learning Bigdata and Flume, while loading twitter data into flume ...
0
votes
0
answers
13
views
ParseException : cannot recognize input near 'where' 'rownum' '=' in subquery source in hive
ParseException : cannot recognize input near 'where' 'rownum' '=' in subquery source in hive
Please help
ParseException : cannot recognize input near 'where' 'rownum' '=' in subquery source in hive ...
0
votes
0
answers
27
views
Correlation Matrix of Big Data
I want to find the correlation matrix of matrix of size 503 rows × 278549 columns.
How can I find it ?
I understand that I need a larger memory , so I tried google colab and google cloud But I could ...
1
vote
1
answer
32
views
Automation of downloading NiFi flows and uploading them to another cluster
I am relatively new to NiFi and I was wondering if it is possible to automate the process of downloading some NiFi templates and then uploading them to another cluster.
More precisely, I created a ...
0
votes
1
answer
67
views
best match sql query
I have a table (companies) in my postgresql database with more than 300,000 companies (adidas, goochie, samsung, ferrari, etc.).
The table itself contains only two columns: ID, company_name.
The user ...
1
vote
4
answers
75
views
Include path of object when streaming out JSON from JQ
I've got the following (100GB+) JSON file which includes the following top level key:
{
"data": {
"<userId1>": {
"things": {
"<...
0
votes
1
answer
22
views
AWS SQS doesn't trigger expecting Lambda stepfunctions , not processing all messages in queue
I have configured sqs on s3 bucket, the sqs should invoke 21 stepfunctions for 21 files uploaded in s3 bucket, instead only 14 stepfunctions are triggered and i am missing remaining 7 events.
The ...
0
votes
1
answer
44
views
How to convert longlat coordinate into Lambert conformal conic in R
I have a big .nc file for daily climate data with 365 layers for north America. I want to use R to extract the timeseries of one climate variables for a specific location (a point with specific lat ...
0
votes
1
answer
23
views
Mutiple actions spark casuing to failures
I'm new to Spark.
I encountered some problems with part of about saving df to Hive table.
def insert_into_hive_table(df: DataFrame, table_name: str):
# for debugging - this action is working ...
0
votes
1
answer
73
views
handling nested Json structure
suppose we have following json structure :
{
"positions": {
"node": "abc"
}
"submissions" :{
"submissionOffsets":[
...
0
votes
1
answer
28
views
Extract data into standalone file with pyspark to workaround:Spark buffer holder size limit issue
Problem
I got same issue as this one: Spark buffer holder size limit issue.
My code is like :
# Calculate the statistics
stats = df.groupBy("EventType").agg(
size(collect_set("...
0
votes
1
answer
161
views
Using Apache beam Python SDK to run Flink Runner on Kubernetes but facing issue
Here is my Custom Dockerfile that include Flink and beam SDK
FROM apache/flink:1.16.2-scala_2.12-java11
ARG FLINK_VERSION=1.16.2
ARG KAFKA_VERSION=2.8.0
# Install python3.7
RUN set -ex; \
apt-get ...
0
votes
0
answers
12
views
How to use ROW_NUMBER to order date?
I want to order the date in ascending.
I've tried
select
PARSE_DATE('%A, %B %d, %Y',date) AS Scraping_Date,
ROW_NUMBER() OVER (order by date asc) as row_num
FROM `kramplab-analytics-rnd....
0
votes
0
answers
15
views
Import HDFS data using Vaex
So im looking for alternatives to access huge volume of data from HDFS beside spark and i found vaex. Is there anyway to directly access data from HDFS using vaex? can i have some example line that ...
0
votes
0
answers
9
views
How to deal with very very large matrices?
% input variables
T = 2274;
N = 58;
B = 10000;
H = 250;
K = 3;
% I simulate matrices of similar size to give the idea of the problem
rng('default') % even if replicability is not necessary in this ...
1
vote
1
answer
63
views
How to replace null value with some value using coalesce in pyspark
I have two files :- orders_renamed.csv , customers.csv
I am joining them with full outer join and then dropping same column (customer_id).
I want to replace null vaue to "-1" in "...
1
vote
1
answer
45
views
how to define Schema for semi - structured text file in pysparK
1 2013-07-25 11599,CLOSED
2 2013-07-25 256,PENDING_PAYMENT
3 2013-07-25 12111,COMPLETE
4 2013-07-25 8827,CLOSED
5 2013-07-25 11318,COMPLETE
6 2013-07-25 7130,COMPLETE
7 2013-07-25 ...
0
votes
0
answers
51
views
Python Polars efficiently enumerating Rows in new Column
I want to create a new Column in my DataFrame, that enumerates the rows in it. My DataFrame has a couple million elements in it, for this reason I need it to be processed in the Rust engine, so ...
1
vote
0
answers
41
views
Why is it that I keep getting the error Insufficient 'DISKS_TOTAL_GB' quota when trying to create cluster on google cloud console?
I keep getting the error Insufficient 'DISKS_TOTAL_GB' quota when I try create a compute engine cluster on google cloud console. It is strange because I have no disks that are active. All the disks ...
0
votes
2
answers
36
views
Selecting suitable database for analytics data
We need to create enterprise product and need to select the storage option for analytics data, initially data will be in tb's(10-40tb) but in coming years could go up to few PB's.
Requirements:
high ...
-1
votes
2
answers
66
views
Need to find a design plan to this problem related to a real time scenarios. Any ideas are appreciated. Thanks
A fleet of 1000 cabs in the warehouse are tracked by multiple fleet managers. Sensors from each cab generates Telematics data & location information. All the telematics data can be captured in to ...
2
votes
1
answer
35
views
Relational DB functionality for reconciling data ingestion against active database?
I'm trying to write an application which ingests all records in a table from an Oracle database into google cloud storage. The Oracle DB is an active database which has new records added constantly. ...
1
vote
0
answers
73
views
Connect to spark cluster using python
Good morning ;
Please I have a question regarding accessing a spark cluster with python.
This is my first time installing and using spark , attached is the docker compose I used
\version: '2'
...