Skip to main content
Filter by
Sorted by
Tagged with
0 votes
1 answer
23 views

How can I import data from several sources to a single output file with Pentaho?

I have a transformation process that gets data from several data sources and joins them into a single CSV output. This process is a raw job that removes the previous information and loads the data ...
Andres's user avatar
  • 322
0 votes
0 answers
11 views

Apache Arrow Flight Server as a Data As A Service

I am planning to build a arrow flight server on top of s3 data storage, the s3 data storage has petabyte of data. I have few concerns when the flight server loads all the 1 pb of data in-memory it ...
Susmit Sarkar's user avatar
0 votes
0 answers
51 views

Write a large dataframe of one billion records in storage account [closed]

I have a dataframe of one billion records. It does not have a column that can be used as a partition column. I need to write the dataframe as a delta table, but the time it takes to do that write ...
Anastasia's user avatar
-1 votes
0 answers
18 views

Dynamic Spark Configuration for Backend App Server with PySpark

I'm configuring PySpark on a backend app server with the following specifications: Cores: 8 RAM: 16 GB Disk Space: 100 GB (50% free) OS: Linux Ubuntu Django: 5.1 PySpark: 3.5.1 Python: 3.11 Node: 1 ...
Purushottam Nawale's user avatar
1 vote
0 answers
63 views

PostgreSQL Update Query Taking Too Long Using CREATE TABLE UPDATE Approch

I have a PostgreSQL query that uses a CREATE TEMP TABLE approach to update a table, but it is taking much longer than expected to execute. The query is supposed to finish in around 15 minutes, but it'...
Purushottam Nawale's user avatar
-5 votes
0 answers
52 views

How do I efficiently handle big data in a distributed system using Hadoop? [closed]

I'm working on a project that involves processing large datasets across multiple nodes, and I'm using Hadoop for the distributed system. However, I’m facing some challenges with job execution times ...
Arthur Leo's user avatar
1 vote
1 answer
72 views

GRTS for large spatial datasets in R

I am trying to apply the generalized random tessellated sampling (GRTS) algorithm implemented in spsurvey to sample a dataset of 3 million points on a map. I am running into many vector memory ...
SGE's user avatar
  • 440
0 votes
0 answers
31 views

Correlation and Multicolinearity Analysis for BigData

Pandas is very well optimized to handle the most common scenarios that we need when we preprocess or analyze data. What about BigData, specifically BigData (1k+ columns, 100k rows) in BigQuery? The ...
Hakun's user avatar
  • 483
-3 votes
0 answers
73 views

SQL cross apply to Pyspark

I have cross apply in my query i want to convert this in pyspark- select * from table1 a left join table2 b on a.index = b.index cross apply ( select top 1 * from table3 p where a.id = p.id and a....
Shivansh Mehra's user avatar
0 votes
1 answer
39 views

Delta Lake table write without schema enforcement

I have a Delta Lake table partitioned by hour. The table schema includes: colA (string type) colB (int type) colC (struct type) When I perform a historical load, all partitions are populated ...
pirate_shady's user avatar
0 votes
1 answer
45 views

Streaming large dataset to child process (as JSON)

I'm trying to send a large amount of data that is stored in memory to a child process. Specifically I have a large dataset represented as JSON in Node.js which I want to send to a child process I am ...
Jaron's user avatar
  • 5
0 votes
0 answers
41 views

Compare two datafile with same structure and identify detail differences PySpark

I have two DF like image below: I want compare two DF in order to identify how detail differences between two table(add rows, delete rows, change values in columns...). Note that my DF don't have any ...
cisco306's user avatar
2 votes
1 answer
43 views

How do I update collections of an XML in MarkLogic that is a very large document?

I'm working with a large XML file in MarkLogic—around 50MB and 700,000 lines long. The task involves adding and removing specific collections using an XQuery module. Previously, I used xdmp:invoke-...
Mehul's user avatar
  • 178
1 vote
0 answers
51 views

How to Implement Distributed AutoFAISS with PySpark for Large-Scale Vector Indexing?

I’m working on a project that involves creating a vector search index for a massive dataset consisting of 1.3 trillion tokens. I want to use AutoFAISS in a distributed environment to handle the scale ...
Cauder's user avatar
  • 2,487
-1 votes
0 answers
30 views

List membership syntax in Trino

I have been trying to query a database table using trino connection to check for records where column ticker has any of the values 'AAPL', 'MSFR', 'AMZN'. The following query throws a syntax error: ...
Raunak Singha's user avatar
0 votes
0 answers
64 views

PySpark python issue: Py4JJavaError: An error occurred while calling o91.showString. Python worker exited unexpectedly (crashed)

I am new to apache spark, and am currently learning to use pyspark. I am having problems when I am just learning this, in installation, I have also equated the environment path according to the guide ...
Alfarezza 's user avatar
0 votes
1 answer
73 views

Efficiently fetch sequences for sliding window (large dataset)

The dataset I have stored are just coordinates of DNA sequence. df: chr start stop label chr1 9000 9100 1 chr1 8803 8903 1 chr1 8903 9000 0 My goal is to ...
youtube's user avatar
  • 414
1 vote
1 answer
46 views

What is the nested if equivalent in PySpark?

I am a beginner to pySpark and I am working on enriching a dataframe which performs lookups from 2 other dataframes, what I want to achieve is: This is an example: incoming_path outgoing_path ...
Tanish57's user avatar
2 votes
2 answers
54 views

Efficiently Split a Large CSV File into Smaller Files by Index Range Without Loading Entire File into RAM

I have a big CSV file (60GB) that does not fit into RAM. The first column contains a sorted index that goes from 2000 to 2999 and can be repeated between rows. I want to split the 60GB file into 10 ...
David Davó's user avatar
0 votes
1 answer
64 views

Ruby + sidekiq - best solution for execute and handle big data

Imagine we have 10k entities-x. For each entity-x we should make async api call. Each api call returns 100 entities-y. Then in total we have 10k * 100 = 1_000_000 entities-y. For each entity-y we ...
xao0isb's user avatar
0 votes
0 answers
18 views

How to Improve Query Performance for Multiple SQL Databases with Complex Joins and Views?

Suppose i have 4 large databases that contain records (approx. 2000-10000 records total between all dbs) about assets on an SQL server. These assets have multiple characteristics in the SQL db. The ...
Paul S's user avatar
  • 1
0 votes
1 answer
58 views

How to check similarities between two datasets and return a score in Snowflake (is it even possible?)

I have two data sets containing full names of my company customers. Both sets are rather large (40-70k rows). I would like to check if there are similarities between the two groups. For example: if ...
Andrea Sordano's user avatar
0 votes
0 answers
20 views

Is there a way to faster a Interpolation IDW done in python for a large array?

I have to do an interpolation on a data set of more than 11 thousand lines and 87 columns, which takes 1:40 minutes to complete. I've already tried using dask array and chunk list but it still takes a ...
NATALIA BASTOS DE SOUSA's user avatar
0 votes
0 answers
64 views

fread() takes 60GB of RAM to load a 22GB CSV dataset [duplicate]

I am loading a CSV file into RStudio using fread() and despite the file being 22GB large, I can see my memory usage at 60 of my 64GB. Why is that? This becomes a problem right after as I need to join ...
Marti's user avatar
  • 101
1 vote
0 answers
48 views

Hash a sparse vector from CountVectorizer in pyspark

I am very new to spark so bear with me. I am currently trying to feature hash feature vectors generated by the CountVectorizer. So for the following example with a hash size of 50: +---+---------------...
EyedBread's user avatar
0 votes
0 answers
18 views

ES: Why Refresh delete Segment with "committed=false"?

Background: There exists an index named "i_dm_f_da_enterprise" in ES, and I prohibited auto-refresh by setting "refresh_interval" : -1. I notice that the refresh operation of the ...
Junbo Wang's user avatar
0 votes
0 answers
44 views

Finding Top Users with Common Records in a Growing Dataset

I’m working on a project where I have a large dataset containing billions of records. Each user can have one or more records, and each record can be associated with multiple users. Given a specific ...
Herman Streltsov's user avatar
0 votes
0 answers
38 views

How to append time-series data with PyArrow Datasets?

Problem I'm looking to store time-series data that's being aggregated live to Parquet Datasets via PyArrow. I receive live batched data, for example, video view count each hour for the last 24 hours. ...
humanlikely's user avatar
0 votes
0 answers
26 views

Standalone spark 3.3.0 java application throws access denied exception when reading from files on mounted drive

I'm using spark 3.3.0 on a standalone cluster, and i have mounted drive from which i need to read some files that comes periodically. As spark application is running as spark user and mounted drive is ...
Petar Markov's user avatar
1 vote
0 answers
49 views

How to load .dat file to Hive with additional columns?

I want to load .dat(without headers) file to hive external table. But in hive table there are extra columns like cob_date , region, file_name which are not present in .dat file. cob_date will be the ...
Big data Pyspark's user avatar
0 votes
0 answers
24 views

How to build ActorSystem in Flink 1.13.5?

This is how I build ActorSystem in Flink 1.8.5. public static ActorSystem createNewActorSystem() throws Exception { String ip = HostPortUtil.getLocalIp(); Configuration configuration = new ...
seedoilz's user avatar
0 votes
1 answer
33 views

Comparing two types of data in bigQuery

We have a very big dataset. And I need to get all the values that are mapping from the source attributes to normalized attributes in my json. The relation between normalised and source is that if the ...
Ankita Prasad's user avatar
0 votes
0 answers
12 views

Hive 'explain' query plan / meaning of Backup Stage

My complex query returned below 'explain' command result. This query is dealing with a huge dataset. What does backup stage mean in line number 4 (Stage-63 has a backup stage: Stage-2)? STAGE ...
Gyanendra Dwivedi's user avatar
-1 votes
1 answer
64 views

How to use pyspark regex to correctly break data with pipe delimited with literal pipe inside?

I have a raw databricks table with a single column named "value" which the data is from a CSV with pipe delimited. The data inside this single column is 5 fields. The field3 content is a ...
CodeForFood's user avatar
0 votes
2 answers
55 views

How do I deduplicate huge table in Clickhouse?

I have a huge single-column table with engine=Log: SELECT * FROM addresses_tmp LIMIT 5 ┌─address──────────────────────────────────┐ 1. │ 18a0a8bdcbd1fec1785224cfc486ccf02dc3ef5d │ 2. │ ...
Stepan Yakovenko's user avatar
0 votes
0 answers
35 views

Loading data from a 50 gb csv file to redshift or snowflake table

I have a question regarding processing a 50 csv file on AWS S3 using pyspark and loading to redshift stage Layer then transformed layer and so on. How do we design such a pipeline, what ETL workflow ...
shubham singh's user avatar
0 votes
0 answers
41 views

Deduplication , Grouping for events table at scale

I'm working with an events table where different source tables trigger writes into this table with columns: entity_id and payload. These events are then published to a Kafka topic using a message ...
Forece85's user avatar
  • 488
0 votes
0 answers
38 views

Why Spark BUG: mode PERMISSIVE not working

Anyone know why the mode PERMISSIVE is not working with Spark? I'm trying to read an excel file, and everything is correct: schema = StructType([StructField('Col 1', StringType(), True), \ ...
Arthur Andrade's user avatar
0 votes
0 answers
52 views

Hive Table Issues with MSCK REPAIR and Alter Table Operations

I have a table test in Hive using ParquetHiveSerDe. The dataset is large with 10+ columns, but here’s a sample version: CREATE EXTERNAL TABLE TEST ( name string, age int, address string ) ...
vnk's user avatar
  • 1,082
1 vote
0 answers
25 views

How to Configure Multiple Communication Paths Between GridDB Client and Server?

I'm currently working on setting up GridDB for an IoT application, and I'm having trouble configuring multiple communication paths between the GridDB client and server. According to the GridDB ...
Syed Zaid bin Haris's user avatar
0 votes
0 answers
44 views

Is the standard AWS Glue table equivalent to a Hive table?

I'm confused as to what exactly does it mean to create a standard AWS Glue table, as opposed to creating an Apache Iceberg table, which is the other option. Is AWS Glue table the equivalent of a Hive ...
user's user avatar
  • 234
0 votes
1 answer
36 views

'schematool' is not recognized as an internal or external command, operable program or batch file, in win 10 when I try to run hive commands

Certainly! Here’s a formatted version of your post suitable for Stack Overflow: Issue with Hive Setup: Unable to Instantiate SessionHiveMetaStoreClient I've installed the following software on my ...
Ashwanth Haridas's user avatar
2 votes
1 answer
42 views

GPS Distance calculate with noises

I have points array but there is some noises like this; But the real direction is; When i calculate all points with haversine formula i am getting wrong distance because noises took like %20-%30 ...
Ahmad Echem's user avatar
2 votes
1 answer
82 views

ADX Kusto how to merge two large tables

I have a simple case, one huge table in the database and another table which comes from a CSV file which has a mapping with the ID from the first table and another column which I want to append to the ...
nxn's user avatar
  • 119
0 votes
1 answer
43 views

PySpark is not creating Dataframe inside the function, when parameters are passed when calling the function

I want to create the dataframe using functions. I have a list of Countries as Row objects. like below: country_list=df_correct_countries.select('NewCountry').dropDuplicates().collect() for i in ...
Ajay S Pal's user avatar
0 votes
1 answer
106 views

How to run NiFi throughout nginx with secure connection

I have NiFi with OpenId authentication via Keycloak. Everything works great. But an additional task appeared: accessing NiFi via Nginx. I configured everything according to the NiFi documentation, but ...
Mikhail's user avatar
  • 103
0 votes
0 answers
40 views

How can I check if a Point, which is in one dataframe, is in the Polygon which is located in the second dataframe in pyspark?

I have two files: one contains data on taxi trips in NYC and the other is a GeoJSON file with the borough boundaries of NYC. I need to create two new columns in the taxi trip dataframe. These columns ...
John Stuart's user avatar
0 votes
0 answers
52 views

Best algorithm to perform multi pattern search on large FASTA file

I am currently working on a bioinformatics problem where I need to lookup and count the location and count of occurences of 4000-ish 6 character long patterns in a fasta file of 700GB. As a newbie to ...
Swathi Subramanyan's user avatar
0 votes
1 answer
74 views

Loading & Analyzing Big Data from a CSV in R [closed]

Please forgive me if this is a duplicate or poorly written. I can't seem to find the resources I need. I have a two-terabyte CSV that I need to analyze in R in various ways, but I am unsure how to ...
Melissa Salazar's user avatar
0 votes
1 answer
30 views

Is an intermediary persistent store needed before storing features in Feast + Cassandra?

I am currently building a big data pipeline for an MLOps project, the pipeline is intended for batch processing. This is the current setup: I am storing my raw structured data in Hive. Spark jobs ...
Eya's user avatar
  • 13

1
2 3 4 5
161