Questions tagged [hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The core consists of a distributed file system (HDFS) and a resource manager (YARN). Various other open-source projects, such as Apache Hive use Apache Hadoop as persistence layer.

Filter by
Sorted by
Tagged with
1 vote
0 answers
27 views

What is causing my memory issues in PySpark?

I'm new to PySpark, and I am trying to code a Random Forest Regression model to predict the amount of buffering that occurs during a streaming session according to various network metrics. The model ...
MattNewman2003's user avatar
0 votes
0 answers
9 views

Hive set spark.sql.catalogImplementation to spark

I'm trying to set spark.sql.catalogImplementation in hive. Already set spark.sql.catalogImplementation spark-defaults.conf connecting through Jupyter notebook. When session is intilized spark.sql....
Malinda Peiris's user avatar
0 votes
0 answers
8 views

Created Hive tables STORED BY ICEBER; SELECT * returns empty

I'm trying to implement a lakehous with hive 4 and iceberg and i think i'm almost there but... When I create a table like: CREATE TABLE ice (id INT, name STRING) STORED BY ICEBERG; And then insert ...
Faneron's user avatar
0 votes
0 answers
13 views

PySpark regex to get value between a string and hyphen

I am trying to extract the number between the string “line_number:” and hyphen. I am struggling with generating a regex/substring for the same in PySpark. Below is my input data in a column called “...
Rohit Kadam's user avatar
0 votes
0 answers
7 views

Oozie 5.2.1 with hadoop 2.7.2

Getting the following error when I try to validate my workflow.xml using oozie validate Workflow.xml Error: E0701: XML schema error, /root/mr-wordcount/Workflow.xml, org.xml.sax....
Houda Daki's user avatar
0 votes
0 answers
9 views

I got this error There are 0 datanode(s) running and no node(s) are excluded in this operation

Do i need to install hive in order to use sqoop on my machine? while i try to import the data from mysql to hadoop it throws error like this: bhandarisamir@Lomass-MacBook-Pro ~ % sqoop import \ --...
Shamir Bhandari's user avatar
0 votes
0 answers
18 views

Apache Avro Writer throws ArrayIndexOutOfBoundsException

I am trying to generate some parquet files from my POJO classes. I am mostly successful in that but AvroWriter is giving me a failure when I try to save a list of custom POJO object. try { ...
Anil's user avatar
  • 13
0 votes
0 answers
46 views

YARN + yarn resource manager stores a ton of znodes related to running/old applications in zookeeper

We have HDP production Hadoop cluster , include 2 active / standby resource managers services Some details from our cluster HDP version - 2.6.5 OS Linux machines version - 7.9 Number of node ...
jessica's user avatar
  • 2,418
0 votes
0 answers
10 views

FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/metrics/Updater (state=08S01,code=-101)

Running complex queries (like join, group by) using beeline hiveserver2 cli produces this error: CLI Output: `select id, count(*) from test group by id; INFO : Compiling command(queryId=...
Hassaan Muzammil's user avatar
-1 votes
0 answers
35 views

Need Spark code optimization for FOR loop

I have a json file as updated data on daily bases and loads data into sales table with updates and inserts. json data file: {"full_name":"ANCHETA, GEORGE","acct_number":&...
user20998231's user avatar
0 votes
0 answers
17 views

IllegalArgumentException: java.net.UnknownHostException: NNode

I am able to connect my hive table which is in a different vm using DBeaver. When i try to connect my hive through pyspark, I was able to look at all my tables within a schema but when I try to query ...
Dishan's user avatar
  • 1
1 vote
0 answers
27 views

Error : py4j.Py4JException: Method sql([class java.lang.String, class [Ljava.lang.Object;]) does not exist

i am using Pyspark 3.4.1, java 8, hadoop 3.4.0, scala 2.12.17, python 3.11.4 and this is my code in vscode : def calculating_click(df): click_data = df.filter((df.custom_track == "click")...
Quốc Phú's user avatar
0 votes
0 answers
18 views

winutls for Windows 11 64bit

I have installed a spark cluster on my lapptop of OS windows 11 64 bit and its working smoothly. While the time of saving dataframe on my local machine i am getting the below error - df.write.mode('...
Sarvesh Pandey's user avatar
-1 votes
0 answers
18 views

How can I run this from a Mac M2, I tried everything, and still happening the same error, => ERROR [namenode internal] load metadata for docker.io/lib [closed]

l ejecutar el comando sudo docker compose up --scale dnnm=4 -d en un entorno de Hadoop_cluster, se producen una serie de advertencias y un error. Las advertencias están relacionadas con dnnm, ...
Fernando Mercado's user avatar
0 votes
0 answers
9 views

Can't find Apache Flume log files

I have just made a fresh install of Apache Flume 1.11.0 and have been able to run the sample agent. I am now trying to figure out where Flume creates log files but I can't seem to find it. It's not in ...
Karthik Sankaran's user avatar

15 30 50 per page
1
2 3 4 5
2957