Questions tagged [hive]

Apache Hive is a database built on top of Hadoop and facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible distributed file system. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. Please DO NOT use this tag for flutter database which is also named Hive, use flutter-hive tag instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Hive not starting

When I try to run hive i get this error SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/apache-hive-3.1.3-bin/lib/log4j-slf4j-impl-2.17.1.jar!/org/...
Houmes's user avatar
  • 71
0 votes
0 answers
11 views

Saving noSQL dataframe Pyspark to noSQL DB

I am looking for advice on the Big Data "goto" for saving a NoSQL dataframe from PySpark. Hive is one way to saving this, e.g. from pyspark.sql import SparkSession columns = ["id", ...
Terra's user avatar
  • 1
1 vote
0 answers
11 views

Flutter Hive not work right, I'm using the Hive to store data on the device, but when I close and restart the app I lose everything

I'm working on a flutter app, I'll start by saying that I'm a beginner and I'm at the beginning of mobile development, I'm using the Hive to store data on the device, while debugging I saw that when I ...
Luca Micarelli's user avatar
0 votes
1 answer
8 views

when the window function row_number in CTE generate the result in hive

I'm using row_number window function in hive CTE(with clause) with data( select 1 a,1 b union select 1,2 union select 1,3 union select 1,4 ... union select 1,26 ), data_with_row_num ( select a,b,...
Eric's user avatar
  • 118
0 votes
1 answer
12 views

Unzip the contents for the bz2 files via HDFS/Haddop

how to unzip a bz2 file in HDFS System wanted to upload this.csv file into a table that i create in hive 2003.csv.bz2 i have this file zcat 2003.csv.bz2 | hdfs dfs -put - /user/$USER/workspace this i ...
Sweta Rawani's user avatar
0 votes
0 answers
8 views

Issue with running hive queries with spark2 using thread

I am trying to make a spark2 application which run about 40-50 queries parallel. Each query takes about 30s to 1m. And to execute query faster, I am trying to use a thread. When I run the application, ...
jeewonb's user avatar
  • 75
1 vote
0 answers
16 views

Why does Spark shuffle the data while joining two partitioned & bucketed tables

I am trying to create a view on top of two tables. Table 1: Partitioned by col1 Bucketed by col2 (no of buckets: 3600) Table 2: Partitioned by col1 Bucketed by col2 ( no of buckets:3600) View: Table1 ...
user2417458's user avatar
0 votes
0 answers
42 views

To register a specific type of data like songModel in Hive in Flutter

I have developed a music player using Flutter and I'm attempting to save a list of songs (SongModel) from the just_audio package into Hive. However, I'm encountering this error: "HiveError (...
Sahand's user avatar
  • 1
0 votes
0 answers
11 views

Error while saving large Spark Dataframe to Hive table on Databricks

I'm trying to save spark dataframe as table in Databricks. The size of df is around 7GB. df.write.format("delta").mode("overwrite").partitionBy("A","B","C&...
ekdz__'s user avatar
  • 77
0 votes
0 answers
31 views

Hive regexp_extract: Extract category number if available [duplicate]

I would like to find out whether my url contains category id or not. The category id in the url is always in the format forward slash + category id of 1 up to 3 digits + dash (for example /228- in ...
Pawels's user avatar
  • 194
0 votes
0 answers
51 views

Need to read data from Kafka and Hive in one query via Flink(FlinkSQL)

I want to run a heterogeneous query in Flink to read several tables from Kafka (the data is written in Kafka so that you can join them via FlinkSQL), and several tables from Hive, in one query. I can ...
Vladimir Frolov's user avatar
0 votes
0 answers
12 views

Hadoop Hive org.apache.hadoop.ipc.RpcException RPC response exceeds maximum data length

I am working in setting Remote PostgreSQL Database for Hive Metastore using Hive 3.1.3 Hadoop 3.3.4 Spark 3.2.4 PostgresSQL 14.9 I could successfully use hive and hiveserver2 as it is shown below ...
Ivan's user avatar
  • 119
0 votes
0 answers
18 views

How to process an array in a Hive UDF in Scala

I'm trying to write a Scala UDF for Hive that accepts an Array of integers as an input parameter and returns the length of the array. Extending the UDF class, I'm able to create simple Hive UDFs that ...
Omer Farooq Ahmed's user avatar
0 votes
0 answers
17 views

How can I configure Hive Metastore or Spark to respect owner ACL privileges in Spark SQL?

I'm currently working on Apache Spark with Hive Metastore. I've created a table in Hive Metastore using user A. However, when running a DROP query in Spark, it appears to be using user B, and ...
Midhun Pottammal's user avatar
0 votes
0 answers
13 views

multi times execute SQL 'desc dataBaseName.tableName' usage much memory

I want to query hundred thousands table's column meta data in hive;after executed ten thousands,the memory come to 7G from 3g;the simple query with multi times execute,usage so much memory,and it's ...
MY J's user avatar
  • 1
-1 votes
0 answers
11 views

can enter hive interactively, but the operation failed

In hive, the database creation is always stuck on the warning, and the final creation fails. like this enter image description here finally enter image description here How to solve the problem, ...
hello_world's user avatar
0 votes
0 answers
24 views

How to store file in csv format with header from google cloud cluster to google storage bucket?

I am currently working on a Springboot project which need me to submit a hive job with query to google cluster and retrieve data in a google bucket.However, it seems not accomplishable to contain a ...
Kailiang Chen's user avatar
0 votes
0 answers
17 views

Istio Envoy Proxy breaks Thrift Service - Hive Metastore

I am trying to run the thrift server (Hive metastore aka HMS) behind the istio Envoy proxy. The HMS works well when I run the service in a namespace with istio-injection disabled. But when i try the ...
user1960109's user avatar
0 votes
0 answers
17 views

Gcloud error - ssh key for Gcloud does not exist

error in gcloud cli I am trying to upload the review.csv file from my local file system to hdfs using gcloud compute scp command but it is showing the error as follows-- C:\Users\DELL\Downloads>...
Girish Ratnaparkhi's user avatar
0 votes
0 answers
32 views

How can i add use a SparkDFDataset to my Great Expectations' validator?

Appreciate your advice on the following issue. I am testing if Great Expectations is viable for use on my hive tables. Ideally i would want to open a html file showing my expectations in a user ...
user7579065's user avatar
1 vote
0 answers
30 views

How to save API response data into hive database using student model?

The following code is for fetching student data using get API and I am using hive package link is below: <https://pub.dev/packages/hive> static Future<void> fetchStudentData() async { var ...
Mian Samiullah's user avatar
0 votes
1 answer
20 views

Multiple hive nodes in a cluster?

What is the use of having multiple hive nodes in a cluster as hive just acts a query engine over Hadoop It can be installed in one of our client node, right ? i tried reading docs but there was no ...
Narendhar Nath's user avatar
0 votes
0 answers
27 views

hiveserver2 hangs while starting Remote PostgreSQL Database for the Hive Metastore

I am working in setting Remote PostgreSQL Database for Hive Metastore using Hive 3.1.3 Hadoop 3.3.4 Spark 3.2.4 PostgresSQL 14.9 I could use Hive Schema Tool to create metastore tables in PostgreSQL ...
Ivan's user avatar
  • 119
1 vote
1 answer
36 views

SQL/Hive - How to parse and sum all values within a row

In my table, there is a column where each row contains a list of n values in the following format: ["5","12","13"]. This is a string Is there a way for me to sum up all ...
GlowFish's user avatar
0 votes
1 answer
47 views

SQL select only max count values for each group

I have a query like this: select id, url, count(*) as count from hit where ... group by id, url There are unique ids that usually have multiple urls assigned to it but only one of these ...
Pawels's user avatar
  • 194
1 vote
2 answers
45 views

coalesce first non-null, varying null positions in hierarchical table sql

I have a table with varying depths of nulll values : dim col1 col2 col3 col4 v c b a NULL x a NULL NULL NULL y b a NULL NULL z d c b a Does anyone know how I can write a select statement to ...
user124123's user avatar
  • 1,653
0 votes
0 answers
14 views

what is the behavior of Spark when Hive enable hive.spark.dynamic.partition.pruning

I have a Hive server running with Spark as its execution engine, I am wondering how enabling hive.spark.dynamic.partition.pruning can affect the queries? Here is the description in Hive docs: hive....
huwng's user avatar
  • 61
0 votes
0 answers
31 views

As a Redshift user can I query Databricks Delta Lake via Unity Catalog Hive Metastore as an external datasource?

Summary: what I am after is: as a Redshift User... can I query Databricks Delta Lake tables via Unity Catalog as a federated query from Redshift? I already know that as a Databricks User... I can ...
Josh Peak's user avatar
  • 5,956
0 votes
0 answers
28 views

Hive set spark.sql.catalogImplementation to spark

I'm trying to set spark.sql.catalogImplementation in hive. Already set spark.sql.catalogImplementation spark-defaults.conf connecting through Jupyter notebook. When session is intilized spark.sql....
Malinda Peiris's user avatar
0 votes
0 answers
17 views

Created Hive tables STORED BY ICEBER; SELECT * returns empty

I'm trying to implement a lakehous with hive 4 and iceberg and i think i'm almost there but... When I create a table like: CREATE TABLE ice (id INT, name STRING) STORED BY ICEBERG; And then insert ...
Faneron's user avatar
0 votes
0 answers
49 views

Flutter/Hive: Changes to List in Memory Persist Across Screens

I'm encountering an issue in my Flutter app where I retrieve a record from Hive database in the first screen, make changes to a list in memory, close the first screen without saving the data to the ...
sifat ullah shah's user avatar
0 votes
1 answer
33 views

(flutter)Hive update issue

i created a basic app for creating, deleting,updating, and reading employee details using hive in Flutter and update function is not working and I am passing the object of class through the update ...
Nibin's user avatar
  • 3
0 votes
0 answers
22 views

FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/metrics/Updater (state=08S01,code=-101)

Running complex queries (like join, group by) using beeline hiveserver2 cli produces this error: CLI Output: > select id, count(\*) from test group by id; INFO : Compiling command(queryId=...
Hassaan Muzammil's user avatar
1 vote
0 answers
24 views

Iceberg copies instead of deleting when using Flink CDC to sync data from MySQL to Iceberg

I have created an iceberg table as follows: CREATE TABLE ars.results( "id" varchar, "input_md5" varchar, "output_md5" varchar, "log" varchar, &...
Martin_5125's user avatar
0 votes
1 answer
37 views

How to filter a date column that is in "INT" format

I have a table with the date column being INT in the format "yyyy/MM/dd". I need to have two separate queries. The first one needs to pull data from the start of current year up to the ...
Zaahid Munshi's user avatar
0 votes
0 answers
15 views

"NoneType" object is not iterable in Impala query

I am query to one of the tables using hive and then impala, environment is same, table is same. However, with impala when I try to query count() or count() with group by it throws below error: ...
KeenLearner's user avatar
0 votes
0 answers
40 views

Concatenate values from the same column grouping by and in a certain order [duplicate]

Using Hive, I need to concatenate the values from the column message for each different id. Starting from a table like this: id message nro juan como estas 2 juan hola 1 pedro hello 1 pedro how u ...
invalid syntax's user avatar
0 votes
2 answers
71 views

Flinksql Create Hive catalog caused "Configured default database default doesn't exist in catalog myhive."

I have followed the instruction to install relevant libs and hive depends. However it's still not working when i create catalog. It seemed like the project is not stable yet. I have also try jdbc ...
taymedee's user avatar
  • 534
0 votes
0 answers
10 views

Apache Flume giving NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter when using a Hive sink

I am learning Apache Flume and I am trying to create an agent that writes records to a Hive table using a Hive sink. When I try to start my agent, I get the following error: 2023-11-12T22:16:52,317 ...
Karthik Sankaran's user avatar
-1 votes
1 answer
61 views

How to connect image apache/hive with image apache/hadoop?

I have docker-compose.yml file. How i can connect apache hive with apache hadoop? I want create table on hive. Extract data from api and load into hadoop hdfs. Have another way like create DockerFile ...
blue's user avatar
  • 1
1 vote
2 answers
17 views

Usage of regex function in sql

Trackingid 00054123 00054214 From the above table, I want to select the trackingid's without the 000 prefix. Currently I am using substring function. Is there a way to do it using regex function? ...
Vignesh's user avatar
  • 11
1 vote
0 answers
73 views

Do I need hive-metastore when I use unity catalog

We have a databricks in azure portal and from the beginning we set up unity catalog. UC works fine we have schemas and tables there. By default there is also hive_metastore which is not working - we ...
ALdo's user avatar
  • 99
0 votes
0 answers
21 views

Calculate row level using previous month and next monthd data in Hive Query

I have below data as source source_table: Output I want is Output_Table: calculation it takes is (take next available amount - previous amount)/ (date of next available amount - date of previous ...
Saumya Shah's user avatar
2 votes
1 answer
95 views

Iceberg table snapshots not expired

I'm using Dataroc Metastore, Dataproc Batch and Pyspark. Although I'm using GCP, I beleive this is general Apache Iceberg question. I run My spark job and created Iceberg trips table with automatic ...
danny.lesnik's user avatar
  • 18.6k
2 votes
1 answer
55 views

Create dates between two dates in Hive query(HQL) and later needs further transformation

Source Desired Output My desired output is Column D and C . Formula column is for your help on how i did the calculation. I am first trying to explode the dates based on min date and max date. Which ...
Saumya Shah's user avatar
-1 votes
0 answers
8 views

Selecting some part of a string from a column in hive sql

I have hive table with column value 100+204,250+205,260+340. Have requirement to select the values before + . The expected output should be (100,250,260) using hiver query. It can have 1 to Mutiple ...
raja raja's user avatar
-1 votes
0 answers
23 views

Split string into columns in hive sql

I have table which has format as: ex value 1 [{"sub":"CA_123","number":10},{"sub":"CA_456","number":15}] 2 [{"sub":"CA_457&...
Quang Minh Vu's user avatar
0 votes
1 answer
22 views

Is DECIMAL in Apache Hive a fixed length type?

How to get the length of DECIMAL in Apache Hive, if it is a fixed length type, such as 16 byte or not? I did not found information in documentation, if it has some methods to get the length, such as ...
mkm's user avatar
  • 11
0 votes
2 answers
43 views

cannot select the case when col to count

This is the SQL query I wrote: SELECT COUNT(*) AS TOTLA_COUNT, COUNT( CASE WHEN ( market_val > 0 AND ABS( loan_amt / market_cal * 100 - ratio ) > 5 ) THEN 1 ...
TingL's user avatar
  • 51
0 votes
0 answers
13 views

Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Java heap space

I had this issue when i tried to rebuild my hive table , and i have error as below : FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Java heap space Here ...
Fauziah Mahmod's user avatar

1
2 3 4 5
438