Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

How filtering on multilevel hive partitioning works

If I partition my data by /YEAR/MONTH/DATE and my queries always looks like Day=[DATE], will my queries benefit from partitioning. I have seen many times people use /YEAR/MONTH/DATE partition but we ...
hampi2017's user avatar
  • 721
0 votes
1 answer
25 views

pyspark integration with Hive

I am doing some work with Hadoop, Hive and Spark, in which I need to query tables that I create in Hive. I am trying to connect to Hive from a jupiter notebook with the pyspark library as follows: ...
Javier Martinez Barriga's user avatar
0 votes
0 answers
24 views

Error while compiling statement: FAILED: ParseException line 9:3 missing EOF at 'WITH' near '"This is a DPCI_LIST table"'

CREATE TABLE XX.DPCI_LIST_FY17 ( DPCI varchar(100) COMMENT 'DPCI_description' ) COMMENT "This is a DPCI_LIST table" WITH ( external_location = 'hdfs://bigred3ns//user/XX/DPCI_Desc' ...
Deepak Chandrashekar's user avatar
0 votes
1 answer
61 views

How to get rows which has data set matching

I have a table which describe place where product is selling as below: PRODUCT Position A 1, 2, 3 B 2 C 7,8,9, 10 D 2,3,4 E 7,8,9 I want to get Products with match two or more selling positions ...
Quang Minh Vu's user avatar
0 votes
1 answer
20 views

How to run a Groupby and selection conditional variables in Hive?

I'm looking to select a conditional based on the following: SELECT income_decile, income, benefits, COUNT(*) FROM income_dataset WHERE income > 50000 AND benefits < 2000; GROUP BY ...
birken_head's user avatar
0 votes
0 answers
21 views

Hiveserver2 not starting in Kubernetes Setup

I'm experiencing issues with a HiveServer2 pod in my Kubernetes cluster. The pod is failing due to readiness probe errors, specifically being unable to connect to port 10000. Readiness probe failed: ...
Dilma's user avatar
  • 645
0 votes
0 answers
10 views

Group by when condition met in Hive

I'm trying to recreate an excel pivot table in hive by selecting two variables based on a condition and then sum over the grouping variable. I'm very new to Hive/sql so I would be grateful for advice ...
Dave Diggler's user avatar
0 votes
0 answers
52 views

SQL ParseException on UNION ALL with CTE: missing EOF at ')' near 'id_phase'

I'm working in a SQL environment (Hive dialect) and trying to combine data from multiple tables using a common table expression (CTE) and UNION ALL. The query involves multiple SELECT statements ...
Yasmine Daly's user avatar
1 vote
1 answer
77 views

Can't configure Hive Metastore client jars in Spark 3.5.1

I need to configure my Spark 3.5.1 application so it uses specific version of Hive Metastore client. I read in the documentation that I can use: spark.sql.hive.metastore.jars spark.sql.hive.metastore....
mox601's user avatar
  • 432
0 votes
0 answers
22 views

How to connect to remote Hive server from local pyspark session

I am trying to connect to a remote Hive server from my local machine via Spark but for some reason whatever operation I am doing, it is happening in my local machine, operations like saveAsTable() ...
ujjawal's user avatar
0 votes
0 answers
109 views

how to connect Trino + Hive metastore + MinIO

I am using Trino docker image(trinodb/trino:400) and did following changes in dockerfile( USER root RUN chmod -R 777 /data RUN chmod -R 777 /etc/trino USER trino ) and hive docker image (apache hive: ...
Divya Gupta's user avatar
0 votes
0 answers
22 views

Not having access to the phone's storage memory after updating Flutter's sdktarget to 34

Since I changed the sdk target of the app to 34. The program no longer has access to the phone's storage. I use the hive database. I also gave him all the manifest permissions, including read, write, ...
Hacker M.t.s's user avatar
0 votes
0 answers
37 views

Create a task with observable in thehive 4 triggered by kibana

Description: I have a single node ELK 7 with some beats and endpoint (Linux and windows). So I have logs of some servers in ELK. Base on SOAR, I install TheHive, MISP and Cortex to implement SOAR. In ...
Majid Mortazavi's user avatar
0 votes
0 answers
22 views

I've tried everything in the book and I still can't fix "Exception has occurred" (HiveError: The box "tasks" is already open and of type Box<Task>.)

I'm getting an error HiveError: The box "tasks" is already open and of type Box in my TaskClassFunctions file, even thought I never use .openbox again and use <type> correctly. So ...
Andy Oviedo's user avatar
0 votes
1 answer
25 views

Creating a new local Spark session after stopping previous one: Spark tells me that Derby is not stopped

In my unit tests I have to stop a local Spark session and create another one (using the data in metastore saved from previous session). But when the another Spark session is created it cannot use my ...
Felix's user avatar
  • 3,571
0 votes
1 answer
36 views

com.holdenkarau.spark.testing - java.lang.NoSuchFieldError: METASTOREURIS

I have a spark scala application, it uses the below versions. Dependencies: org.apache.spark:spark-core_2.12:jar:3.3.2 org.apache.spark:spark-hive_2.12:jar:3.3.2 org.apache.hive:hive-common:jar:2.3.9 ...
saravana ir's user avatar
0 votes
0 answers
12 views

How to split Hive reducers output into multiple files

I inserted data into an Hive table as follow : SET hive.execution.engine=tez; SET hive.exec.dynamic.partition=true; SET hive.exec.dynamic.partition.mode=nonstrict; INSERT INTO TABLE db.my_table ...
maximus's user avatar
  • 31
0 votes
1 answer
214 views

Use S3 as the source to create iceberg tables

I am currently working on a small setup where I have a basic setup of iceberg, trino, hive metastore and s3. I am able to create iceberg tables in s3 using the trino CLI. Now, I would like to use a ...
Raj's user avatar
  • 23
3 votes
0 answers
50 views

Are there any techniques to solve skew data in databricks?

I created skewed data to test a salting approach and tried three different solutions, but none achieved the desired results with a significant runtime improvement. Can you guide me on the best ...
Learn Hadoop's user avatar
  • 3,036
0 votes
1 answer
76 views

Creating deciles in SQL

I'm trying to bucket my data into deciles, but not in the traditional sense where the dimension is the basis of the decile. I have 463 unique it_scores ranging from 316-900 (my dimension) with 1,296,...
A. Oli's user avatar
  • 43
0 votes
0 answers
27 views

I cannot connect to Hive Server although it is running

I am using docker for running Apache Hive. And I sure that my hiveserver2 is running as it show HiveSessionID in the cmd(but it resfresh Hive Session ID sometimes I dont know if it affect). But when i ...
phnnguynmnhkhoi's user avatar
0 votes
0 answers
27 views

Got error when using Spark to read Hive external table

Im using Spark 3.5.0, Hive is 4.0.0. I already create a table news in hive and stored as Parquet. +----------+----------------------------------------------------+----------------------------------...
Minh Vũ Lã's user avatar
1 vote
1 answer
113 views

MERGE INTO TABLE is not supported temporarily. while trying to merge data into an iceberg table in hadoop hive environment

I need to perform a merge operation into my iceberg table. I am using a jupyter notebook on an Aws emr setup. Spark: '3.4.1-amzn-2' Hadoop: 3.3.6 Hive: 3.1.3 EMR Version: 6.15.0 Scala: 'version 2.12....
shady xv's user avatar
1 vote
0 answers
21 views

Dataproc Hive Job - OutOfMemoryError: Java heap space

I have a dataproc cluster, we are running INSERT OVERWRITE QUERY through HIVE CLI which fails with OutOfMemoryError: Java heap space. We adjusted memory configurations for reducers and Tez tasks, ...
Parmeet Singh's user avatar
0 votes
0 answers
11 views

How to build a Hive query to find ids that only have one attribute value?

I have the following table in Hive: +-----+----------+ | id| type| +-----+----------+ | 1| A| | 1| B| | 2| B| | 3| A| | 3| B| | 4| ...
Mamaf's user avatar
  • 360
2 votes
1 answer
33 views

How to cache dataframes that will be retained after spark session terminates

I'm new to data engineering, so this might be a basic question. But I haven't been able to clarify --context-- I have a spark job executed by an Azure Data Factory pipeline every 10 minutes. On every ...
April Lee's user avatar
0 votes
1 answer
18 views

SET Variables in HIVE from function

I'm trying to insert data into a Hive table with a dynamic partition value that needs to be a computed date, but Hive's PARTITION clause doesn't seem to support expressions or functions. I have a base ...
Dulanga Heshan's user avatar
0 votes
0 answers
13 views

Troubles with making a Favorite with Hive DB Flutter

I want to make a Sptotify Clone application. I pull songs from Firebase and the list is swapped when the data is pulled again. I use state management with Bloc. Here my coding page image. Here is my ...
NurLife's user avatar
0 votes
0 answers
31 views

Flink job writing to hive partitioned table throws "Partition already exists: Partition"

I have a flink job written in java to sink datastream to a partitioned table in hive(external table). While the data is being sink to table, the partition committer throws the below exception INFO ...
AshokN's user avatar
  • 1
0 votes
0 answers
34 views

Hudi Upsert Job keeps failing at distinct step at HoodieJavaRDD.distinct(HoodieJavaRDD.java:157)

Problem Statement: Running an ETL on a table "Aggregate X Details". When running this on a large dataset i.e. ~30M rows of data we partition our dataset into 50 partitions. But the executors ...
Jaskirat Uppal's user avatar
0 votes
0 answers
86 views

trino large query is failing due to hive metastore failure

I have a large insert into query, which is reaching 99-100% quite fast (after 5-6 min), then completion percent starting to decrease and back to high number and repeating.. This query is reading from ...
roh's user avatar
  • 133
0 votes
0 answers
29 views

Can't load Hive transactional table into dataframe using Spark2

I'm trying to load a hive transactional table into a PySpark dataframe, but receive the error: py4j.protocol.Py4JJavaError: An error occurred while calling o086.showString. Caused by: java.util....
Quatervois's user avatar
0 votes
0 answers
33 views

Incompatible configuration used between Spark and HBaseTestingUtility

We are using the MiniDFSCluster and MiniHbaseCluster from HBaseTestingUtility to run unit tests for our Spark jobs. The Spark configuration that we use is : conf.set("spark.sql....
Evelina Dumitrescu's user avatar
0 votes
0 answers
18 views

Update statement in hadoop ecosystem using hive (WARNING: An illegal reflective access operation has occurred)

I am having trouble with running a simple update statement. The query i am excecuting is the follow NSERT INTO data_source_metadata VALUES ( 'source1', 'User ...
Artemis's user avatar
  • 23
0 votes
0 answers
44 views

Unable to Save Firebase Notification Message to Hive on iOS in Flutter (Works on Android)

I'm developing a Flutter app that uses Firebase for push notifications and Hive for local storage. I want to save incoming messages from Firebase notifications into Hive. The setup works perfectly on ...
Asim Nasar siddiqui's user avatar
0 votes
1 answer
72 views

How do I make Trino stop early when enough rows have been collected?

I have a Hive database and I am querying its tables through Trino. Every table has many partitions. Whenever I do a query like SELECT * FROM my_table LIMIT 100; it still goes through all partitions ...
Albert's user avatar
  • 195
0 votes
0 answers
30 views

How to make hive singleton class for saving datas like favourites cartitem?

I have created a singleton class for saving data to local storage with GetStorage, here I have a bucket named userdata and i want to save favouriteproductids, cartitems and many other list, With this ...
Irfan Ganatra's user avatar
1 vote
0 answers
26 views

use hive hash function inside spark sql - spark 3.5

I have an spark job wherein I need to read data from hive (hive 3) and insert it to MySQL. Before reading there needs a data refresh that needs to happen. The refresh query looks like this (Only ...
Aishwary Shukla's user avatar
0 votes
1 answer
60 views

Deriving a column using the previous row's value of the column itself?

operation txn_quantity cumulative_quantity txn_amount cost_of_purchase sell_ratio NET_COST buy 250 250 5000 5000 0 0 sell 100 150 3000 0 0.4 0 buy 150 300 1500 1500 0 0 sell 225 75 4000 0 0.75 0 ...
jeffreyp's user avatar
1 vote
1 answer
37 views

Why does Spark not create a new file after inserting data into an external table?

I have a .csv file data.csv stored at location: dbfs:/raw/data/externalTables/emp_data_folder/emp_data.csv Here is a sample of the data in the file: Alice,25,50000,North Bob,30,60000,South Charlie,35,...
DumbCoder's user avatar
  • 445
0 votes
0 answers
38 views

Unhandled Exception: HiveError: Cannot read, unknown typeId: 36. Did you forget to register an adapter?

I'm encountering an error in my Flutter application related to Hive. I attempted to resolve it by adding await Hive.initFlutter();, but that did not work. Additionally, I don't have any Hive model ...
Sachithya Rivisara's user avatar
-1 votes
2 answers
102 views

Multiple Join Operations on the same columns with the same criteria. Is there a way to make this more efficient?

In Presto/Hive SQL, I have multiple tables, say T1, T2, T3, T4, X1, X2, X3, X4 Assume that the table size is T1 > T2 > T3 > T4 X1 > X3 > X4 > X4 We have SELECT T1.a, T2.b, T3....
user98235's user avatar
  • 896
1 vote
1 answer
56 views

Unable to access hiveserver2 via beeline

I am trying to run hive locally in Kind via a part of a helm chart generated from Kompose of apache/hive. My hiveserver2-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ...
Liam Pieri's user avatar
0 votes
0 answers
31 views

hive on docker with GCS fails

I am trying to setup hive with GCS storage using these containers. But hitting the error ERROR : FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(...
SThota's user avatar
  • 53
0 votes
0 answers
22 views

how does the CombineHiveInputFormat split the files(ORC/TextFile)

set maxsize of splits: set mapreduce.input.fileinputformat.split.maxsize=209715200; --200MB(256MB default) set mapreduce.input.fileinputformat.split.minsize=0; run the hive sql on a table STORED AS ...
user27636532's user avatar
0 votes
0 answers
30 views

How to load all JSON files for a given day in a hive partitioned from Google Cloud Storage Bucket into a polars dataframe?

I have a Google Cloud Storage project with a bucket. In this bucket the files are partitioned via Hive partition, e.g.: gs://my_bucket/vendor/updates/year=2024/month=08/day=01/company_a.json gs://...
Vega's user avatar
  • 2,929
0 votes
1 answer
28 views

Hive Flutter Error with storage_backend_js

Everything worked fine and hive saved and loaded everything but then I started getting this message. And I don't know how to fix it. I've tried adding new type filed to hive adapter and after that ...
nadiavi's user avatar
  • 27
0 votes
1 answer
68 views

hive4 Error when running query command Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.RuntimeException: java.lang.reflect

When I start a query or insert data I am using hive4 jdk17 and hadoop3.4 which have been added in hive's conf/hive-env.sh export HADOOP_OPTS="$HADOOP_OPTS --add-opens=java.base/java.util=ALL-...
高志翔's user avatar
0 votes
0 answers
16 views

What Are the Key Differences Between Alluxio SDK Cache and File Merger Cache in Presto's Hive Connector?

I’m currently using a Hive connector with Presto, which supports two cache services: 1. Alluxio and 2. File Merger Cache. I want to cache remote data into local storage. While there’s documentation ...
Boina Sai Vinay's user avatar
0 votes
1 answer
82 views

Hive Always Fails at Mapreduce

I just installed hadoop 3.3.6 and hive 4.0.0 with mysql as metastore. when running create table or select * from... it runs well. But when I try to do insert or select join, hive always fails. I'm ...
Dzaki Wicaksono's user avatar

1
2 3 4 5
438