21,892 questions
0
votes
0
answers
13
views
How filtering on multilevel hive partitioning works
If I partition my data by /YEAR/MONTH/DATE and my queries always looks like Day=[DATE], will my queries benefit from partitioning. I have seen many times people use /YEAR/MONTH/DATE partition but we ...
0
votes
1
answer
25
views
pyspark integration with Hive
I am doing some work with Hadoop, Hive and Spark, in which I need to query tables that I create in Hive. I am trying to connect to Hive from a jupiter notebook with the pyspark library as follows:
...
0
votes
0
answers
24
views
Error while compiling statement: FAILED: ParseException line 9:3 missing EOF at 'WITH' near '"This is a DPCI_LIST table"'
CREATE TABLE XX.DPCI_LIST_FY17
(
DPCI varchar(100) COMMENT 'DPCI_description'
)
COMMENT "This is a DPCI_LIST table"
WITH (
external_location = 'hdfs://bigred3ns//user/XX/DPCI_Desc'
...
0
votes
1
answer
61
views
How to get rows which has data set matching
I have a table which describe place where product is selling as below:
PRODUCT
Position
A
1, 2, 3
B
2
C
7,8,9, 10
D
2,3,4
E
7,8,9
I want to get Products with match two or more selling positions ...
0
votes
1
answer
20
views
How to run a Groupby and selection conditional variables in Hive?
I'm looking to select a conditional based on the following:
SELECT income_decile, income, benefits,
COUNT(*) FROM income_dataset WHERE income > 50000 AND benefits < 2000;
GROUP BY ...
0
votes
0
answers
21
views
Hiveserver2 not starting in Kubernetes Setup
I'm experiencing issues with a HiveServer2 pod in my Kubernetes cluster. The pod is failing due to readiness probe errors, specifically being unable to connect to port 10000.
Readiness probe failed: ...
0
votes
0
answers
10
views
Group by when condition met in Hive
I'm trying to recreate an excel pivot table in hive by selecting two variables based on a condition and then sum over the grouping variable.
I'm very new to Hive/sql so I would be grateful for advice ...
0
votes
0
answers
52
views
SQL ParseException on UNION ALL with CTE: missing EOF at ')' near 'id_phase'
I'm working in a SQL environment (Hive dialect) and trying to combine data from multiple tables using a common table expression (CTE) and UNION ALL. The query involves multiple SELECT statements ...
1
vote
1
answer
77
views
Can't configure Hive Metastore client jars in Spark 3.5.1
I need to configure my Spark 3.5.1 application so it uses specific version of Hive Metastore client.
I read in the documentation that I can use:
spark.sql.hive.metastore.jars
spark.sql.hive.metastore....
0
votes
0
answers
22
views
How to connect to remote Hive server from local pyspark session
I am trying to connect to a remote Hive server from my local machine via Spark but for some reason whatever operation I am doing, it is happening in my local machine, operations like saveAsTable()
...
0
votes
0
answers
109
views
how to connect Trino + Hive metastore + MinIO
I am using Trino docker image(trinodb/trino:400) and did following changes in dockerfile(
USER root
RUN chmod -R 777 /data
RUN chmod -R 777 /etc/trino
USER trino
)
and hive docker image (apache hive: ...
0
votes
0
answers
22
views
Not having access to the phone's storage memory after updating Flutter's sdktarget to 34
Since I changed the sdk target of the app to 34. The program no longer has access to the phone's storage.
I use the hive database.
I also gave him all the manifest permissions, including
read, write, ...
0
votes
0
answers
37
views
Create a task with observable in thehive 4 triggered by kibana
Description: I have a single node ELK 7 with some beats and endpoint (Linux and windows). So I have logs of some servers in ELK. Base on SOAR, I install TheHive, MISP and Cortex to implement SOAR. In ...
0
votes
0
answers
22
views
I've tried everything in the book and I still can't fix "Exception has occurred" (HiveError: The box "tasks" is already open and of type Box<Task>.)
I'm getting an error
HiveError: The box "tasks" is already open and of type Box
in my TaskClassFunctions file, even thought I never use .openbox again and use <type> correctly.
So ...
0
votes
1
answer
25
views
Creating a new local Spark session after stopping previous one: Spark tells me that Derby is not stopped
In my unit tests I have to stop a local Spark session and create another one (using the data in metastore saved from previous session).
But when the another Spark session is created it cannot use my ...
0
votes
1
answer
36
views
com.holdenkarau.spark.testing - java.lang.NoSuchFieldError: METASTOREURIS
I have a spark scala application, it uses the below versions.
Dependencies:
org.apache.spark:spark-core_2.12:jar:3.3.2
org.apache.spark:spark-hive_2.12:jar:3.3.2
org.apache.hive:hive-common:jar:2.3.9
...
0
votes
0
answers
12
views
How to split Hive reducers output into multiple files
I inserted data into an Hive table as follow :
SET hive.execution.engine=tez;
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
INSERT INTO TABLE db.my_table
...
0
votes
1
answer
214
views
Use S3 as the source to create iceberg tables
I am currently working on a small setup where I have a basic setup of iceberg, trino, hive metastore and s3.
I am able to create iceberg tables in s3 using the trino CLI.
Now, I would like to use a ...
3
votes
0
answers
50
views
Are there any techniques to solve skew data in databricks?
I created skewed data to test a salting approach and tried three different solutions, but none achieved the desired results with a significant runtime improvement. Can you guide me on the best ...
0
votes
1
answer
76
views
Creating deciles in SQL
I'm trying to bucket my data into deciles, but not in the traditional sense where the dimension is the basis of the decile.
I have 463 unique it_scores ranging from 316-900 (my dimension) with 1,296,...
0
votes
0
answers
27
views
I cannot connect to Hive Server although it is running
I am using docker for running Apache Hive. And I sure that my hiveserver2 is running as it show HiveSessionID in the cmd(but it resfresh Hive Session ID sometimes I dont know if it affect). But when i ...
0
votes
0
answers
27
views
Got error when using Spark to read Hive external table
Im using Spark 3.5.0, Hive is 4.0.0.
I already create a table news in hive and stored as Parquet.
+----------+----------------------------------------------------+----------------------------------...
1
vote
1
answer
113
views
MERGE INTO TABLE is not supported temporarily. while trying to merge data into an iceberg table in hadoop hive environment
I need to perform a merge operation into my iceberg table. I am using a jupyter notebook on an Aws emr setup.
Spark: '3.4.1-amzn-2' Hadoop: 3.3.6 Hive: 3.1.3 EMR Version: 6.15.0 Scala: 'version 2.12....
1
vote
0
answers
21
views
Dataproc Hive Job - OutOfMemoryError: Java heap space
I have a dataproc cluster, we are running INSERT OVERWRITE QUERY through HIVE CLI which fails with OutOfMemoryError: Java heap space.
We adjusted memory configurations for reducers and Tez tasks, ...
0
votes
0
answers
11
views
How to build a Hive query to find ids that only have one attribute value?
I have the following table in Hive:
+-----+----------+
| id| type|
+-----+----------+
| 1| A|
| 1| B|
| 2| B|
| 3| A|
| 3| B|
| 4| ...
2
votes
1
answer
33
views
How to cache dataframes that will be retained after spark session terminates
I'm new to data engineering, so this might be a basic question. But I haven't been able to clarify
--context--
I have a spark job executed by an Azure Data Factory pipeline every 10 minutes.
On every ...
0
votes
1
answer
18
views
SET Variables in HIVE from function
I'm trying to insert data into a Hive table with a dynamic partition value that needs to be a computed date, but Hive's PARTITION clause doesn't seem to support expressions or functions.
I have a base ...
0
votes
0
answers
13
views
Troubles with making a Favorite with Hive DB Flutter
I want to make a Sptotify Clone application. I pull songs from Firebase and the list is swapped when the data is pulled again. I use state management with Bloc. Here my coding page image.
Here is my ...
0
votes
0
answers
31
views
Flink job writing to hive partitioned table throws "Partition already exists: Partition"
I have a flink job written in java to sink datastream to a partitioned table in hive(external table).
While the data is being sink to table, the partition committer throws the below exception
INFO ...
0
votes
0
answers
34
views
Hudi Upsert Job keeps failing at distinct step at HoodieJavaRDD.distinct(HoodieJavaRDD.java:157)
Problem Statement:
Running an ETL on a table "Aggregate X Details". When running this on a large dataset i.e. ~30M rows of data we partition our dataset into 50 partitions. But the executors ...
0
votes
0
answers
86
views
trino large query is failing due to hive metastore failure
I have a large insert into query, which is reaching 99-100% quite fast (after 5-6 min), then completion percent starting to decrease and back to high number and repeating..
This query is reading from ...
0
votes
0
answers
29
views
Can't load Hive transactional table into dataframe using Spark2
I'm trying to load a hive transactional table into a PySpark dataframe, but receive the error:
py4j.protocol.Py4JJavaError: An error occurred while calling o086.showString.
Caused by: java.util....
0
votes
0
answers
33
views
Incompatible configuration used between Spark and HBaseTestingUtility
We are using the MiniDFSCluster and MiniHbaseCluster from HBaseTestingUtility to run unit tests for our Spark jobs.
The Spark configuration that we use is :
conf.set("spark.sql....
0
votes
0
answers
18
views
Update statement in hadoop ecosystem using hive (WARNING: An illegal reflective access operation has occurred)
I am having trouble with running a simple update statement.
The query i am excecuting is the follow
NSERT INTO data_source_metadata VALUES (
'source1',
'User ...
0
votes
0
answers
44
views
Unable to Save Firebase Notification Message to Hive on iOS in Flutter (Works on Android)
I'm developing a Flutter app that uses Firebase for push notifications and Hive for local storage. I want to save incoming messages from Firebase notifications into Hive. The setup works perfectly on ...
0
votes
1
answer
72
views
How do I make Trino stop early when enough rows have been collected?
I have a Hive database and I am querying its tables through Trino. Every table has many partitions. Whenever I do a query like
SELECT * FROM my_table LIMIT 100;
it still goes through all partitions ...
0
votes
0
answers
30
views
How to make hive singleton class for saving datas like favourites cartitem?
I have created a singleton class for saving data to local storage with GetStorage, here I have a bucket named userdata and i want to save favouriteproductids, cartitems and many other list,
With this ...
1
vote
0
answers
26
views
use hive hash function inside spark sql - spark 3.5
I have an spark job wherein I need to read data from hive (hive 3) and insert it to MySQL. Before reading there needs a data refresh that needs to happen. The refresh query looks like this (Only ...
0
votes
1
answer
60
views
Deriving a column using the previous row's value of the column itself?
operation
txn_quantity
cumulative_quantity
txn_amount
cost_of_purchase
sell_ratio
NET_COST
buy
250
250
5000
5000
0
0
sell
100
150
3000
0
0.4
0
buy
150
300
1500
1500
0
0
sell
225
75
4000
0
0.75
0
...
1
vote
1
answer
37
views
Why does Spark not create a new file after inserting data into an external table?
I have a .csv file data.csv stored at location: dbfs:/raw/data/externalTables/emp_data_folder/emp_data.csv
Here is a sample of the data in the file:
Alice,25,50000,North
Bob,30,60000,South
Charlie,35,...
0
votes
0
answers
38
views
Unhandled Exception: HiveError: Cannot read, unknown typeId: 36. Did you forget to register an adapter?
I'm encountering an error in my Flutter application related to Hive. I attempted to resolve it by adding await Hive.initFlutter();, but that did not work. Additionally, I don't have any Hive model ...
-1
votes
2
answers
102
views
Multiple Join Operations on the same columns with the same criteria. Is there a way to make this more efficient?
In Presto/Hive SQL, I have multiple tables, say T1, T2, T3, T4, X1, X2, X3, X4
Assume that the table size is
T1 > T2 > T3 > T4
X1 > X3 > X4 > X4
We have
SELECT
T1.a, T2.b, T3....
1
vote
1
answer
56
views
Unable to access hiveserver2 via beeline
I am trying to run hive locally in Kind via a part of a helm chart generated from Kompose of apache/hive.
My hiveserver2-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
...
0
votes
0
answers
31
views
hive on docker with GCS fails
I am trying to setup hive with GCS storage using these containers. But hitting the error
ERROR : FAILED: Execution Error, return code 40000 from
org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(...
0
votes
0
answers
22
views
how does the CombineHiveInputFormat split the files(ORC/TextFile)
set maxsize of splits:
set mapreduce.input.fileinputformat.split.maxsize=209715200; --200MB(256MB default)
set mapreduce.input.fileinputformat.split.minsize=0;
run the hive sql on a table STORED AS ...
0
votes
0
answers
30
views
How to load all JSON files for a given day in a hive partitioned from Google Cloud Storage Bucket into a polars dataframe?
I have a Google Cloud Storage project with a bucket.
In this bucket the files are partitioned via Hive partition, e.g.:
gs://my_bucket/vendor/updates/year=2024/month=08/day=01/company_a.json
gs://...
0
votes
1
answer
28
views
Hive Flutter Error with storage_backend_js
Everything worked fine and hive saved and loaded everything but then I started getting this message. And I don't know how to fix it.
I've tried adding new type filed to hive adapter and after that ...
0
votes
1
answer
68
views
hive4 Error when running query command Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.RuntimeException: java.lang.reflect
When I start a query or insert data
I am using hive4 jdk17 and hadoop3.4 which have been added in hive's conf/hive-env.sh
export HADOOP_OPTS="$HADOOP_OPTS --add-opens=java.base/java.util=ALL-...
0
votes
0
answers
16
views
What Are the Key Differences Between Alluxio SDK Cache and File Merger Cache in Presto's Hive Connector?
I’m currently using a Hive connector with Presto, which supports two cache services: 1. Alluxio and 2. File Merger Cache. I want to cache remote data into local storage.
While there’s documentation ...
0
votes
1
answer
82
views
Hive Always Fails at Mapreduce
I just installed hadoop 3.3.6 and hive 4.0.0 with mysql as metastore. when running create table or select * from... it runs well. But when I try to do insert or select join, hive always fails. I'm ...