8,003
questions
0
votes
1
answer
23
views
How can I import data from several sources to a single output file with Pentaho?
I have a transformation process that gets data from several data sources and joins them into a single CSV output.
This process is a raw job that removes the previous information and loads the data ...
0
votes
0
answers
11
views
Apache Arrow Flight Server as a Data As A Service
I am planning to build a arrow flight server on top of s3 data storage, the s3 data storage has petabyte of data.
I have few concerns when the flight server loads all the 1 pb of data in-memory it ...
0
votes
0
answers
51
views
Write a large dataframe of one billion records in storage account [closed]
I have a dataframe of one billion records. It does not have a column that can be used as a partition column. I need to write the dataframe as a delta table, but the time it takes to do that write ...
-1
votes
0
answers
18
views
Dynamic Spark Configuration for Backend App Server with PySpark
I'm configuring PySpark on a backend app server with the following specifications:
Cores: 8
RAM: 16 GB
Disk Space: 100 GB (50% free)
OS: Linux Ubuntu
Django: 5.1
PySpark: 3.5.1
Python: 3.11
Node: 1
...
1
vote
0
answers
63
views
PostgreSQL Update Query Taking Too Long Using CREATE TABLE UPDATE Approch
I have a PostgreSQL query that uses a CREATE TEMP TABLE approach to update a table, but it is taking much longer than expected to execute. The query is supposed to finish in around 15 minutes, but it'...
-5
votes
0
answers
52
views
How do I efficiently handle big data in a distributed system using Hadoop? [closed]
I'm working on a project that involves processing large datasets across multiple nodes, and I'm using Hadoop for the distributed system. However, I’m facing some challenges with job execution times ...
1
vote
1
answer
72
views
GRTS for large spatial datasets in R
I am trying to apply the generalized random tessellated sampling (GRTS) algorithm implemented in spsurvey to sample a dataset of 3 million points on a map. I am running into many vector memory ...
0
votes
0
answers
31
views
Correlation and Multicolinearity Analysis for BigData
Pandas is very well optimized to handle the most common scenarios that we need when we preprocess or analyze data.
What about BigData, specifically BigData (1k+ columns, 100k rows) in BigQuery?
The ...
-3
votes
0
answers
73
views
SQL cross apply to Pyspark
I have cross apply in my query i want to convert this in pyspark-
select * from table1 a
left join table2 b on a.index = b.index
cross apply (
select top 1 * from table3 p where a.id = p.id and a....
0
votes
1
answer
39
views
Delta Lake table write without schema enforcement
I have a Delta Lake table partitioned by hour. The table schema includes:
colA (string type)
colB (int type)
colC (struct type)
When I perform a historical load, all partitions are populated ...
0
votes
1
answer
45
views
Streaming large dataset to child process (as JSON)
I'm trying to send a large amount of data that is stored in memory to a child process. Specifically I have a large dataset represented as JSON in Node.js which I want to send to a child process I am ...
0
votes
0
answers
41
views
Compare two datafile with same structure and identify detail differences PySpark
I have two DF like image below:
I want compare two DF in order to identify how detail differences between two table(add rows, delete rows, change values in columns...).
Note that my DF don't have any ...
2
votes
1
answer
43
views
How do I update collections of an XML in MarkLogic that is a very large document?
I'm working with a large XML file in MarkLogic—around 50MB and 700,000 lines long. The task involves adding and removing specific collections using an XQuery module. Previously, I used xdmp:invoke-...
1
vote
0
answers
51
views
How to Implement Distributed AutoFAISS with PySpark for Large-Scale Vector Indexing?
I’m working on a project that involves creating a vector search index for a massive dataset consisting of 1.3 trillion tokens. I want to use AutoFAISS in a distributed environment to handle the scale ...
-1
votes
0
answers
30
views
List membership syntax in Trino
I have been trying to query a database table using trino connection to check for records where column ticker has any of the values 'AAPL', 'MSFR', 'AMZN'.
The following query throws a syntax error:
...
0
votes
0
answers
64
views
PySpark python issue: Py4JJavaError: An error occurred while calling o91.showString. Python worker exited unexpectedly (crashed)
I am new to apache spark, and am currently learning to use pyspark. I am having problems when I am just learning this, in installation, I have also equated the environment path according to the guide ...
0
votes
1
answer
73
views
Efficiently fetch sequences for sliding window (large dataset)
The dataset I have stored are just coordinates of DNA sequence.
df:
chr start stop label
chr1 9000 9100 1
chr1 8803 8903 1
chr1 8903 9000 0
My goal is to ...
1
vote
1
answer
46
views
What is the nested if equivalent in PySpark?
I am a beginner to pySpark and I am working on enriching a dataframe which performs lookups from 2 other dataframes, what I want to achieve is:
This is an example:
incoming_path
outgoing_path
...
2
votes
2
answers
54
views
Efficiently Split a Large CSV File into Smaller Files by Index Range Without Loading Entire File into RAM
I have a big CSV file (60GB) that does not fit into RAM. The first column contains a sorted index that goes from 2000 to 2999 and can be repeated between rows. I want to split the 60GB file into 10 ...
0
votes
1
answer
64
views
Ruby + sidekiq - best solution for execute and handle big data
Imagine we have 10k entities-x. For each entity-x we should make async api call. Each api call returns 100 entities-y. Then in total we have 10k * 100 = 1_000_000 entities-y. For each entity-y we ...
0
votes
0
answers
18
views
How to Improve Query Performance for Multiple SQL Databases with Complex Joins and Views?
Suppose i have 4 large databases that contain records (approx. 2000-10000 records total between all dbs) about assets on an SQL server. These assets have multiple characteristics in the SQL db.
The ...
0
votes
1
answer
58
views
How to check similarities between two datasets and return a score in Snowflake (is it even possible?)
I have two data sets containing full names of my company customers. Both sets are rather large (40-70k rows). I would like to check if there are similarities between the two groups. For example: if ...
0
votes
0
answers
20
views
Is there a way to faster a Interpolation IDW done in python for a large array?
I have to do an interpolation on a data set of more than 11 thousand lines and 87 columns, which takes 1:40 minutes to complete. I've already tried using dask array and chunk list but it still takes a ...
0
votes
0
answers
64
views
fread() takes 60GB of RAM to load a 22GB CSV dataset [duplicate]
I am loading a CSV file into RStudio using fread() and despite the file being 22GB large, I can see my memory usage at 60 of my 64GB. Why is that? This becomes a problem right after as I need to join ...
1
vote
0
answers
48
views
Hash a sparse vector from CountVectorizer in pyspark
I am very new to spark so bear with me. I am currently trying to feature hash feature vectors generated by the CountVectorizer. So for the following example with a hash size of 50:
+---+---------------...
0
votes
0
answers
18
views
ES: Why Refresh delete Segment with "committed=false"?
Background:
There exists an index named "i_dm_f_da_enterprise" in ES, and I prohibited auto-refresh by setting "refresh_interval" : -1. I notice that the refresh operation of the ...
0
votes
0
answers
44
views
Finding Top Users with Common Records in a Growing Dataset
I’m working on a project where I have a large dataset containing billions of records. Each user can have one or more records, and each record can be associated with multiple users.
Given a specific ...
0
votes
0
answers
38
views
How to append time-series data with PyArrow Datasets?
Problem
I'm looking to store time-series data that's being aggregated live to Parquet Datasets via PyArrow. I receive live batched data, for example, video view count each hour for the last 24 hours. ...
0
votes
0
answers
26
views
Standalone spark 3.3.0 java application throws access denied exception when reading from files on mounted drive
I'm using spark 3.3.0 on a standalone cluster, and i have mounted drive from which i need to read some files that comes periodically. As spark application is running as spark user and mounted drive is ...
1
vote
0
answers
49
views
How to load .dat file to Hive with additional columns?
I want to load .dat(without headers) file to hive external table.
But in hive table there are extra columns like cob_date , region, file_name
which are not present in .dat file.
cob_date will be the ...
0
votes
0
answers
24
views
How to build ActorSystem in Flink 1.13.5?
This is how I build ActorSystem in Flink 1.8.5.
public static ActorSystem createNewActorSystem() throws Exception {
String ip = HostPortUtil.getLocalIp();
Configuration configuration = new ...
0
votes
1
answer
33
views
Comparing two types of data in bigQuery
We have a very big dataset. And I need to get all the values that are mapping from the source attributes to normalized attributes in my json. The relation between normalised and source is that if the ...
0
votes
0
answers
12
views
Hive 'explain' query plan / meaning of Backup Stage
My complex query returned below 'explain' command result. This query is dealing with a huge dataset. What does backup stage mean in line number 4 (Stage-63 has a backup stage: Stage-2)?
STAGE ...
-1
votes
1
answer
64
views
How to use pyspark regex to correctly break data with pipe delimited with literal pipe inside?
I have a raw databricks table with a single column named "value" which the data is from a CSV with pipe delimited.
The data inside this single column is 5 fields.
The field3 content is a ...
0
votes
2
answers
55
views
How do I deduplicate huge table in Clickhouse?
I have a huge single-column table with engine=Log:
SELECT * FROM addresses_tmp LIMIT 5
┌─address──────────────────────────────────┐
1. │ 18a0a8bdcbd1fec1785224cfc486ccf02dc3ef5d │
2. │ ...
0
votes
0
answers
35
views
Loading data from a 50 gb csv file to redshift or snowflake table
I have a question regarding processing a 50 csv file on AWS S3 using pyspark and loading to redshift stage Layer then transformed layer and so on.
How do we design such a pipeline, what ETL workflow ...
0
votes
0
answers
41
views
Deduplication , Grouping for events table at scale
I'm working with an events table where different source tables trigger writes into this table with columns: entity_id and payload. These events are then published to a Kafka topic using a message ...
0
votes
0
answers
38
views
Why Spark BUG: mode PERMISSIVE not working
Anyone know why the mode PERMISSIVE is not working with Spark? I'm trying to read an excel file, and everything is correct:
schema = StructType([StructField('Col 1', StringType(), True), \
...
0
votes
0
answers
52
views
Hive Table Issues with MSCK REPAIR and Alter Table Operations
I have a table test in Hive using ParquetHiveSerDe. The dataset is large with 10+ columns, but here’s a sample version:
CREATE EXTERNAL TABLE TEST (
name string,
age int,
address string
)
...
1
vote
0
answers
25
views
How to Configure Multiple Communication Paths Between GridDB Client and Server?
I'm currently working on setting up GridDB for an IoT application, and I'm having trouble configuring multiple communication paths between the GridDB client and server. According to the GridDB ...
0
votes
0
answers
44
views
Is the standard AWS Glue table equivalent to a Hive table?
I'm confused as to what exactly does it mean to create a standard AWS Glue table, as opposed to creating an Apache Iceberg table, which is the other option. Is AWS Glue table the equivalent of a Hive ...
0
votes
1
answer
36
views
'schematool' is not recognized as an internal or external command, operable program or batch file, in win 10 when I try to run hive commands
Certainly! Here’s a formatted version of your post suitable for Stack Overflow:
Issue with Hive Setup: Unable to Instantiate SessionHiveMetaStoreClient
I've installed the following software on my ...
2
votes
1
answer
42
views
GPS Distance calculate with noises
I have points array but there is some noises like this;
But the real direction is;
When i calculate all points with haversine formula i am getting wrong distance because noises took like %20-%30 ...
2
votes
1
answer
82
views
ADX Kusto how to merge two large tables
I have a simple case, one huge table in the database and another table which comes from a CSV file which has a mapping with the ID from the first table and another column which I want to append to the ...
0
votes
1
answer
43
views
PySpark is not creating Dataframe inside the function, when parameters are passed when calling the function
I want to create the dataframe using functions.
I have a list of Countries as Row objects.
like below:
country_list=df_correct_countries.select('NewCountry').dropDuplicates().collect()
for i in ...
0
votes
1
answer
106
views
How to run NiFi throughout nginx with secure connection
I have NiFi with OpenId authentication via Keycloak. Everything works great. But an additional task appeared: accessing NiFi via Nginx. I configured everything according to the NiFi documentation, but ...
0
votes
0
answers
40
views
How can I check if a Point, which is in one dataframe, is in the Polygon which is located in the second dataframe in pyspark?
I have two files: one contains data on taxi trips in NYC and the other is a GeoJSON file with the borough boundaries of NYC. I need to create two new columns in the taxi trip dataframe. These columns ...
0
votes
0
answers
52
views
Best algorithm to perform multi pattern search on large FASTA file
I am currently working on a bioinformatics problem where I need to lookup and count the location and count of occurences of 4000-ish 6 character long patterns in a fasta file of 700GB. As a newbie to ...
0
votes
1
answer
74
views
Loading & Analyzing Big Data from a CSV in R [closed]
Please forgive me if this is a duplicate or poorly written. I can't seem to find the resources I need.
I have a two-terabyte CSV that I need to analyze in R in various ways, but I am unsure how to ...
0
votes
1
answer
30
views
Is an intermediary persistent store needed before storing features in Feast + Cassandra?
I am currently building a big data pipeline for an MLOps project, the pipeline is intended for batch processing.
This is the current setup:
I am storing my raw structured data in Hive.
Spark jobs ...