Questions tagged [bigdata]

Big data is a concept that deals with data sets of extreme volumes. Questions may tend to be related to infrastructure, algorithms, statistics, and data structures.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Teradata Fastload utility not working in Spark Scala as more number of sessions are created

Eventhough Passing Sessions Parameter in JDBC Url. Number of fastload connections are created more than mentioned sessions. How to control the number of sessions when using fastload utility to load ...
PRATIKSHA PANPALIYA's user avatar
-2 votes
0 answers
9 views

How to drools share memmory in multi node

i am new to flink and drools I have a problem like: count the number of events with souce_ip within 10 minutes, if count > 5 => alert i use apache flink and drools. flink to receive data from ...
thao nguyen's user avatar
-1 votes
0 answers
14 views

How to Drools share state in multi node

I am a new flink,drools. I must handle event stream. Example: count event same source_ip in 10 minute. If count > 5 => alert I use apache flink combine Drools. I build a flink application, and ...
Thor's user avatar
  • 1
0 votes
1 answer
40 views

How to efficiently read a large property as a pandas DataFrame using SQLAlchemy?

I'm working on a project where I have a large dataset stored in a relational database, specifically a "PriceHistory" table containing historical stock prices. I also have a "Stocks"...
Roei's user avatar
  • 27
0 votes
0 answers
42 views

fuzzy matching millions of records

I am trying do a fuzzy matching between two columns in which col="co_zip22" will iterate through all the rows in col="co_zip23" and will find a match with a match score So ...
user22600865's user avatar
0 votes
4 answers
79 views

What is the fastest way to read a csv file sort the data then write the sorted data into another csv

I have a massive data set ~600Gb of several csv files. Each csv file contains 1.3mil x 17 sets of data. It looks like this index duration is_buy_order issued location_id ...
Steven's user avatar
  • 105
-4 votes
0 answers
96 views

Write a query for an sql query for a billion-record table [closed]

`Hi all for a project I am working on I was asked to solve this 'big data' task: Given the tables: events event_id int (autoincrement) --**10B distinct values** event_ts datetime -- **10B** ...
Aheavy95's user avatar
0 votes
0 answers
51 views

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 261727: invalid start byte

I appreciate your help. My question is method about sorting big data(csv) with 3rd column. It operates normally up to 1 million lines input data, but for inputs longer than that, the error message ...
Tom's user avatar
  • 1
0 votes
0 answers
36 views

Import 1M data using laravel-excel and queue

Thanks in Advance for your help. I am using laravel version 7 and laravel-excel package for importing big data CSV file. the problem I am facing is that the laravel telescope shows that the job has ...
Md. Mostafijur Rahman's user avatar
-1 votes
0 answers
51 views

PyArrow Table to Spark DataFrame

import pyarrow.parquet as pq table = pq.read_table(parquet_blob_data) Now how to create a spark dataframe from table without using pandas library or to_pandas() in-built method? I found only ...
Uday kumar's user avatar
0 votes
0 answers
11 views

Collecting data via windowing doesn't scale

I have flat table with a large number of rows. Some of these rows deemed to be important for some reason. I want to collect certain fields from rows around these rows in a window. I have a working ...
SDani's user avatar
  • 79
1 vote
1 answer
38 views

How to use filter function for broadcast variable in Pyspark

When I am trying to filter broadcasted value with below command is not working. Please help me with the correct approach from pyspark.sql import * spark=SparkSession.builder.appName("broadcast ...
Nik's user avatar
  • 23
0 votes
0 answers
14 views

Running cucumber BDD in GCP for Dataproc spark job testing

Currently my BDD is running on On-premise Hadoop cluster where it establishes JDBC connection and runs BDDs. Following are the steps done through BDD. Insert required data in Hive and Hbase tables ...
Raghu K's user avatar
  • 57
0 votes
0 answers
22 views

Spark executors getting idle timeout

I have an EMR cluster where I am running pyspark jobs with spark dynamic allocation. spark.dynamicAllocation.enabled=true Below is config of cluster 1 Node 128GB Ram 10 cores Core Nodes Autoscaled ...
Vrishank's user avatar
  • 302
0 votes
1 answer
23 views

Hive query to find a month for each year in which maximum number of quantities were sold

I have a data like below, from which i am trying to find out which month for each year has max quantity ordered if the data is not showing clearly i have added a snip of it QUANTITYORDERED QTR_ID ...
Pratap Reddy's user avatar
0 votes
0 answers
24 views

How to do datamanagement on data chunks

structure(list(ano_date = c("0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "159", &...
Seydou GORO's user avatar
  • 1,169
0 votes
1 answer
46 views

Bulk replace nan in one column based on existing data from others [duplicate]

Let's say we have the following dataframe below is just an example with tons of rows name platform year_of_release genre na_sales eu_sales jp_sales other_sales critic_score user_score rating Madden ...
Max Zaikin's user avatar
0 votes
0 answers
38 views

Updating or Fetching Big Data from Mongodb

I have around 1.9m record in Mongodb collection leads. Sample document is below: { "_id": { "$oid": "60174f6cc3" }, "customer_id": "590583", ...
Decoder's user avatar
0 votes
0 answers
20 views

Nifi: DeleteAzureDataLakeStorage blocked

I have a DeleteAzureDataLakeStorage processor but in some executions it crashes. In data provenance only view: Sometimes it runs in seconds and other times it hangs indefinitely. In the apache logs ...
Miguel's user avatar
  • 171
0 votes
0 answers
24 views

How to maintain online statistics for big data?

I have a DB with relativly (for our capacity) large rows ~ 45000000. I have ~ 500-1000 users who need online statistic from this DB by different criterias(client, priority, etc). In general, ...
KotGaf's user avatar
  • 609
0 votes
1 answer
100 views

Why are spark3 dynamic partitions slow to write to hive

Question 1: I have a table with a small amount of data, but there are a lot of dynamic partitions in the daily writes, the original spark2 writes can be solved in only 2 minutes, but after upgrading ...
langxianashen2001's user avatar
0 votes
0 answers
15 views

Oozie Coordinator Jobs are getting killed automatically without much information

Oozie Coordinators in Prod environment are automatically getting killed with below logs: Till 27th Aug Everything has run fine, on 28th Aug 09:00 the job is scheduled to run but job gets killed 20mins ...
swapnil shashank's user avatar
0 votes
1 answer
50 views

Where can i find open medical datasets in HL7/FHIR format?

I need open data in HL7/FHIR format for a big data project. I try many portals, but until now i didn't find anything interesting. I'm looking for imaging, EHR and biosignals of any type (ECG, EEG, EMG ...
Oppie's user avatar
  • 13
0 votes
1 answer
32 views

Understanding GDELT

I'm new to Gdelt world and i try to understand 3 databases, namely events, Mentions, GKG. they can be combined to each other through their ID. but my question is regarding Mentions database. where ...
Mostafa Bouzari's user avatar
0 votes
0 answers
28 views

How to detect character encoding from stream

I want to get the character encoding of large file data acquired by stream in js (e.g. SJIS, UTF8). I could get it by converting stream to buffer, but I can't convert it to buffer if the data is about ...
kaotaka's user avatar
  • 21
-1 votes
1 answer
46 views

Edit filter function on spark

This is my code for filtering out all the rows in the spark dataframe, the columns of which have rarely encountered values, this code is working and the number of rows really decreases (I need it for ...
Necrotoxxx 's user avatar
0 votes
0 answers
15 views

PIG MAP_ONLY Job failing with Datanode Going OOM Error

I Have application Audit stored in HDFS for day 09 to 16. All this data have total 23 Fields. Two of them is Event_ID and Status. Event_ID can be 1 to 10. And Status can be REQUEST, SUCCESS, FAILURE. [...
Uddhav Savani's user avatar
0 votes
0 answers
23 views

I am unable to connect twitter source data

`Hello Team, I hope you are good. Actually, I am in need of an assistance regarding Apache flume. To give you the context, I am newly learning Bigdata and Flume, while loading twitter data into flume ...
Rajesh Venkatesan's user avatar
0 votes
0 answers
13 views

ParseException : cannot recognize input near 'where' 'rownum' '=' in subquery source in hive

ParseException : cannot recognize input near 'where' 'rownum' '=' in subquery source in hive Please help ParseException : cannot recognize input near 'where' 'rownum' '=' in subquery source in hive ...
srinivasulu thuppukonda's user avatar
0 votes
0 answers
27 views

Correlation Matrix of Big Data

I want to find the correlation matrix of matrix of size 503 rows × 278549 columns. How can I find it ? I understand that I need a larger memory , so I tried google colab and google cloud But I could ...
sumaih al-hazzaa's user avatar
1 vote
1 answer
32 views

Automation of downloading NiFi flows and uploading them to another cluster

I am relatively new to NiFi and I was wondering if it is possible to automate the process of downloading some NiFi templates and then uploading them to another cluster. More precisely, I created a ...
Calin Iorga's user avatar
0 votes
1 answer
67 views

best match sql query

I have a table (companies) in my postgresql database with more than 300,000 companies (adidas, goochie, samsung, ferrari, etc.). The table itself contains only two columns: ID, company_name. The user ...
Paul's user avatar
  • 55
1 vote
4 answers
75 views

Include path of object when streaming out JSON from JQ

I've got the following (100GB+) JSON file which includes the following top level key: { "data": { "<userId1>": { "things": { "<...
Rich's user avatar
  • 8,118
0 votes
1 answer
22 views

AWS SQS doesn't trigger expecting Lambda stepfunctions , not processing all messages in queue

I have configured sqs on s3 bucket, the sqs should invoke 21 stepfunctions for 21 files uploaded in s3 bucket, instead only 14 stepfunctions are triggered and i am missing remaining 7 events. The ...
VKRV's user avatar
  • 1
0 votes
1 answer
44 views

How to convert longlat coordinate into Lambert conformal conic in R

I have a big .nc file for daily climate data with 365 layers for north America. I want to use R to extract the timeseries of one climate variables for a specific location (a point with specific lat ...
Natalia's user avatar
  • 57
0 votes
1 answer
23 views

Mutiple actions spark casuing to failures

I'm new to Spark. I encountered some problems with part of about saving df to Hive table. def insert_into_hive_table(df: DataFrame, table_name: str): # for debugging - this action is working ...
amit's user avatar
  • 71
0 votes
1 answer
73 views

handling nested Json structure

suppose we have following json structure : { "positions": { "node": "abc" } "submissions" :{ "submissionOffsets":[ ...
ravish kumar's user avatar
0 votes
1 answer
28 views

Extract data into standalone file with pyspark to workaround:Spark buffer holder size limit issue

Problem I got same issue as this one: Spark buffer holder size limit issue. My code is like : # Calculate the statistics stats = df.groupBy("EventType").agg( size(collect_set("...
Yiffany's user avatar
  • 304
0 votes
1 answer
161 views

Using Apache beam Python SDK to run Flink Runner on Kubernetes but facing issue

Here is my Custom Dockerfile that include Flink and beam SDK FROM apache/flink:1.16.2-scala_2.12-java11 ARG FLINK_VERSION=1.16.2 ARG KAFKA_VERSION=2.8.0 # Install python3.7 RUN set -ex; \ apt-get ...
kapil dev's user avatar
  • 111
0 votes
0 answers
12 views

How to use ROW_NUMBER to order date?

I want to order the date in ascending. I've tried select PARSE_DATE('%A, %B %d, %Y',date) AS Scraping_Date, ROW_NUMBER() OVER (order by date asc) as row_num FROM `kramplab-analytics-rnd....
user22132557's user avatar
0 votes
0 answers
15 views

Import HDFS data using Vaex

So im looking for alternatives to access huge volume of data from HDFS beside spark and i found vaex. Is there anyway to directly access data from HDFS using vaex? can i have some example line that ...
Beavis Luckyano's user avatar
0 votes
0 answers
9 views

How to deal with very very large matrices?

% input variables T = 2274; N = 58; B = 10000; H = 250; K = 3; % I simulate matrices of similar size to give the idea of the problem rng('default') % even if replicability is not necessary in this ...
Barbab's user avatar
  • 266
1 vote
1 answer
63 views

How to replace null value with some value using coalesce in pyspark

I have two files :- orders_renamed.csv , customers.csv I am joining them with full outer join and then dropping same column (customer_id). I want to replace null vaue to "-1" in "...
Vivek Mishra's user avatar
1 vote
1 answer
45 views

how to define Schema for semi - structured text file in pysparK

1 2013-07-25 11599,CLOSED 2 2013-07-25 256,PENDING_PAYMENT 3 2013-07-25 12111,COMPLETE 4 2013-07-25 8827,CLOSED 5 2013-07-25 11318,COMPLETE 6 2013-07-25 7130,COMPLETE 7 2013-07-25 ...
Vivek Mishra's user avatar
0 votes
0 answers
51 views

Python Polars efficiently enumerating Rows in new Column

I want to create a new Column in my DataFrame, that enumerates the rows in it. My DataFrame has a couple million elements in it, for this reason I need it to be processed in the Rust engine, so ...
CallMeMSL's user avatar
1 vote
0 answers
41 views

Why is it that I keep getting the error Insufficient 'DISKS_TOTAL_GB' quota when trying to create cluster on google cloud console?

I keep getting the error Insufficient 'DISKS_TOTAL_GB' quota when I try create a compute engine cluster on google cloud console. It is strange because I have no disks that are active. All the disks ...
Nick's user avatar
  • 11
0 votes
2 answers
36 views

Selecting suitable database for analytics data

We need to create enterprise product and need to select the storage option for analytics data, initially data will be in tb's(10-40tb) but in coming years could go up to few PB's. Requirements: high ...
Explore Mayank's user avatar
-1 votes
2 answers
66 views

Need to find a design plan to this problem related to a real time scenarios. Any ideas are appreciated. Thanks

A fleet of 1000 cabs in the warehouse are tracked by multiple fleet managers. Sensors from each cab generates Telematics data & location information. All the telematics data can be captured in to ...
subham das's user avatar
2 votes
1 answer
35 views

Relational DB functionality for reconciling data ingestion against active database?

I'm trying to write an application which ingests all records in a table from an Oracle database into google cloud storage. The Oracle DB is an active database which has new records added constantly. ...
nlbrown's user avatar
  • 23
1 vote
0 answers
73 views

Connect to spark cluster using python

Good morning ; Please I have a question regarding accessing a spark cluster with python. This is my first time installing and using spark , attached is the docker compose I used \version: '2' ...
ensberg's user avatar
  • 47

1
2 3 4 5
159