Questions tagged [hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The core consists of a distributed file system (HDFS) and a resource manager (YARN). Various other open-source projects, such as Apache Hive use Apache Hadoop as persistence layer.

Filter by
Sorted by
Tagged with
0 votes
1 answer
21 views

copyFromLocal File not found - command to copy infile csv to hdfs location using pyspark script

I'm using the following code with in the Spark script. Here, I want to copy the data files from local to HDFS location. When running the HDFS command from the shell manually it's working fine. But ...
VamsiKumar_gudala's user avatar
2 votes
0 answers
25 views

Difference between `hadoop fs -du` and `hadoop fs -df` in Dataproc

I use a long-lived Google Cloud Dataproc single-node cluster, to run a Spark streaming job. I note that the reported used HDFS capacity never decrease. Currently, more than 3GB are used. However, when ...
AlexisBRENON's user avatar
  • 2,961
0 votes
0 answers
34 views

incompatible cluster id between namenode and datanode for hadoop

On Windows 11, I installed the latest available version of Docker Desktop. Following that, I visited the official Apache Hadoop GitHub repository at https://github.com/apache/hadoop/tree/docker-hadoop-...
jr134's user avatar
  • 117
-1 votes
0 answers
14 views

Service unavailable: Service unavailable: Soft memory limit exceeded

I'm facing the below issue please let me know if you need any other details. error is in Screenshot plus below in text Lost task 7.0 in stage 39.0 (TID 3872, abc, executor 44): java.lang....
Zubair Shams's user avatar
-1 votes
0 answers
26 views

Ray Dataset loading HDFS: Failed to close HdfsReadableFile: IOError: HDFS CloseFile failed. Detail: [errno 255] Unknown error 255

I am using Ray Dataset read parquets on the HDFS cluster. import ray test_dataset = ray.data.read_parquet('hdfs://[email protected]:9000/path/to/file.parquet') test_dataset.take(1) When reading ...
David Lee's user avatar
-2 votes
0 answers
49 views

Unable to run hive [closed]

Attempting to run hive but currently getting this error. Error Tried to add this line hive-site.xml to my hive-site.xml yet I'm still getting the same 'HiveConf of name hive.server2.enable....
Kobe Bryant's user avatar
-1 votes
0 answers
16 views

Apache hadoop 3.3.6 - unable to connect to port

I am trying to setup Open source Apache Hadoop 3.3.6 on rhel 8.6. But i am getting this error. 02:25:10.341 [main] ERROR com.acunu.loader.FileLoader - Fatal exception in thread Thread[main,5,main] ...
user3051956's user avatar
-1 votes
1 answer
13 views

Hadoop find all documents that contain a list of words

I realize this could be done on a single machine using a HashMap<String, SortedSet> then an intersection of all document names, however am curious how Hadoop would solve this problem. Can anyone ...
notaorb's user avatar
  • 1,986
0 votes
0 answers
10 views

Hadoop running jar files probelm

I am trying to run some jar files on my hadoop installed on Windows. I can create the Jar file perfectly from the java files but every time I try to run it gives me this error: 2023-12-04 15:02:01,966 ...
Maria R.'s user avatar
-1 votes
1 answer
8 views

MapReduce error:The main class could not be found or loaded

I use hadoop-3.2.2,A Hadoop cluster has just been configured. When using mapreduce to calculate PI, an error is reported Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last ...
JiaRu Xu's user avatar
0 votes
1 answer
13 views

Hadoop Error: Exception message: '/tmp/hadoop-user' is not recognized as an internal or external command, operable program or batch file

I'm new to Hadoop, and I've been following a guide I found on LinkedIn titled "Installation of Hadoop single node cluster and run simple applications like word count in Windows OS." ...
Umer Arif's user avatar
0 votes
0 answers
15 views

Hadoop error when trying to initiate start-dfs.sh on linux (pop_os 22.04)

I installed the latest hadoop package (3.3.6). I was also following the installation procedure listed in the official apache hadoop website, and this medium article(I didn't create a separate user ...
Harsha Vardhan's user avatar
0 votes
1 answer
19 views

Unzip the contents for the bz2 files via HDFS/Haddop

how to unzip a bz2 file in HDFS System wanted to upload this.csv file into a table that i create in hive 2003.csv.bz2 i have this file zcat 2003.csv.bz2 | hdfs dfs -put - /user/$USER/workspace this i ...
Sweta Rawani's user avatar
0 votes
1 answer
12 views

Why hadoop cmd is still showing error even after setting environment variable

i have installed hadoop and set hadoop_home and bin in path but i am still getting this error hadoop -version The system cannot find the path specified.
Ruchith Gaurav's user avatar
-3 votes
0 answers
31 views

HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: invalid variable name" and "HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: invalid variable name [closed]

I'm facing an error when trying to execute Sqoop commands. The error message is as follows: /home/ta/ap/hadoop-3.1.2/libexec/hadoop-functions.sh: line 2358: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: invalid ...
Tariku Dereje's user avatar
0 votes
0 answers
11 views

hdfs file download request url

when request download, request url is hostname, so appeared 400 bad request. my server is 5 and hadoop is 1 namenode, 1 secondary & datanode, 3datanode total 1namenode 1secondarynamenode ...
hi-inbeom's user avatar
0 votes
0 answers
9 views

Hadoop BlockMissingException on AWS EMR Cluster

I'm facing difficulties executing a job on Apache Spark within an AWS EMR (Elastic MapReduce) cluster. It's essential to note that the EMR cluster is provisioned exclusively for processing this ...
Fernando Fraga's user avatar
0 votes
0 answers
17 views

Hadoop Hive org.apache.hadoop.ipc.RpcException RPC response exceeds maximum data length

I am working in setting Remote PostgreSQL Database for Hive Metastore using Hive 3.1.3 Hadoop 3.3.4 Spark 3.2.4 PostgresSQL 14.9 I could successfully use hive and hiveserver2 as it is shown below ...
Ivan's user avatar
  • 119
0 votes
0 answers
16 views

Specified hdfs ip in pyhdfs.HdfsClient, but it show my hostname

Just like this: hdfs_client = HdfsClient(hosts='10.1.103.49:50070') root_files = hdfs_client.listdir('/') print(root_files) hdfs_client.copy_from_local('./Readme.md', '/image/Readme.md') it can ...
Perimeter Chou's user avatar
0 votes
0 answers
46 views

How to Add a Service to Ambari managed cluster (bigtop 3.2.1) , example : sqoop / sentry / kudu

I have successfully Implemented the Bigtop project, now i have a working Ambari managed cluster with the latest Bigtop Stack and i want to add a couple of services to my cluster such as Sqoop / Sentry ...
Yassir Acharki's user avatar
0 votes
1 answer
14 views

import sqoop to hdfs error Application failed 2 times due to AM Container

enter image description here I try to import data using sqoop cmd : sqoop import --connect "jdbc:sqlserver://192.168.26.8:1717;databasename=VMC" -- table contact -m 1 --username sa --...
Tien Le's user avatar
0 votes
0 answers
17 views

how many number of live datanodes required when settings dfs.namenode.replication.min=2

My HDFS cluster has 3 datanode. When I set dfs.namenode.replication.min=2, my cluster can not exit safe mode at startup and keep throwing this error: The minimum number of live datanodes is not ...
huwng's user avatar
  • 61
1 vote
0 answers
24 views

Unable to read HDFS file with Go lang

While trying to read a simple file with colinmarc go module by creating a executable file in go , it not able to read the file. The below location is a HDFS location. It is able to connect to namenode....
Dileep Dominic's user avatar
0 votes
1 answer
17 views

My hadoop reducer writes the output to the context only if I write the original value to the context

I have this code: @Override protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { Set<String> mySet = new ...
Kinyanjui Karanja's user avatar
0 votes
1 answer
16 views

When a connected namenode stops in Hadoop, doesn't the datanode automatically connect to another namenode?

We installed hadoop 2.2.0 on Ubuntu and configured HA as follows. namenode: master, master-ha datanode: slave We confirmed that master and master-ah are connected properly and the status is set to ...
john_smith's user avatar
0 votes
1 answer
20 views

Multiple hive nodes in a cluster?

What is the use of having multiple hive nodes in a cluster as hive just acts a query engine over Hadoop It can be installed in one of our client node, right ? i tried reading docs but there was no ...
Narendhar Nath's user avatar
0 votes
0 answers
10 views

Sqoop Export using multiple mappers issue

Facing issue in sqoop export to UDB database when we use more number of maps. We receiving -911 abend (no other process is using the DB) sqoop export -Dsqoop.export.records.per.statement=50000 -D ...
vicky bangre's user avatar
0 votes
0 answers
30 views

hiveserver2 hangs while starting Remote PostgreSQL Database for the Hive Metastore

I am working in setting Remote PostgreSQL Database for Hive Metastore using Hive 3.1.3 Hadoop 3.3.4 Spark 3.2.4 PostgresSQL 14.9 I could use Hive Schema Tool to create metastore tables in PostgreSQL ...
Ivan's user avatar
  • 119
0 votes
0 answers
72 views

Spark-submit yarn-client mode hangs even though spark task completed (pyspark 3.4.1)

Recently set up a dockerized env along with CDP for submitting yarn-client mode spark jobs on a kereberized hadoop cluster and seeing inconsistent behavior with application lifecycle. Scenario: ...
StrangerThinks's user avatar
0 votes
1 answer
120 views

org.apache.hadoop.fs.ParentNotDirectoryException - is not a directory

I am trying to append new records into existing csv file using Spark SQL. Getting Exception in thread "main" org.apache.hadoop.fs.ParentNotDirectoryException: /tmp/spark/input/csv/orders.csv ...
Vasanth Subramanian's user avatar
0 votes
0 answers
24 views

Oozie 5.2.1 with hadoop 2.7.2

Getting the following error when I try to validate my workflow.xml using oozie validate Workflow.xml Error: E0701: XML schema error, /root/mr-wordcount/Workflow.xml, org.xml.sax....
Houda Daki's user avatar
0 votes
1 answer
18 views

I got this error There are 0 datanode(s) running and no node(s) are excluded in this operation

Do i need to install hive in order to use sqoop on my machine? while i try to import the data from mysql to hadoop it throws error like this: bhandarisamir@Lomass-MacBook-Pro ~ % sqoop import \ --...
Shamir Bhandari's user avatar
1 vote
0 answers
35 views

Apache Parquet Writer throws ArrayIndexOutOfBoundsException when writing Avro

I am trying to generate some parquet files from my POJO classes. I am mostly successful in that but AvroWriter is giving me a failure when I try to save a list of custom POJO object. try { ...
Anil's user avatar
  • 23
0 votes
0 answers
63 views

YARN + yarn resource manager stores a ton of znodes related to running/old applications in zookeeper

We have HDP production Hadoop cluster , include 2 active / standby resource managers services Some details from our cluster HDP version - 2.6.5 OS Linux machines version - 7.9 Number of node ...
jessica's user avatar
  • 2,448
0 votes
0 answers
23 views

FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/metrics/Updater (state=08S01,code=-101)

Running complex queries (like join, group by) using beeline hiveserver2 cli produces this error: CLI Output: > select id, count(\*) from test group by id; INFO : Compiling command(queryId=...
Hassaan Muzammil's user avatar
0 votes
1 answer
34 views

IllegalArgumentException: java.net.UnknownHostException: NNode

I am able to connect my hive table which is in a different vm using DBeaver. When i try to connect my hive through pyspark, I was able to look at all my tables within a schema but when I try to query ...
Dishan's user avatar
  • 1
2 votes
1 answer
83 views

Error : py4j.Py4JException: Method sql([class java.lang.String, class [Ljava.lang.Object;]) does not exist

i am using Pyspark 3.4.1, java 8, hadoop 3.4.0, scala 2.12.17, python 3.11.4 and this is my code in vscode : def calculating_click(df): click_data = df.filter((df.custom_track == "click")...
Quốc Phú's user avatar
0 votes
0 answers
26 views

winutls for Windows 11 64bit

I have installed a spark cluster on my lapptop of OS windows 11 64 bit and its working smoothly. While the time of saving dataframe on my local machine i am getting the below error - df.write.mode('...
Sarvesh Pandey's user avatar
0 votes
0 answers
9 views

Can't find Apache Flume log files

I have just made a fresh install of Apache Flume 1.11.0 and have been able to run the sample agent. I am now trying to figure out where Flume creates log files but I can't seem to find it. It's not in ...
Karthik Sankaran's user avatar
-1 votes
0 answers
18 views

Increase Hbase timeout by rest api parameter

I have queries that are exceeding the default timeout while fetching a table in hbase rest api Currently i can't change the hbase-site.xml file to increase the timeouts on all queries, and so i've ...
maayan ben tsur's user avatar
-1 votes
1 answer
76 views

How to connect image apache/hive with image apache/hadoop?

I have docker-compose.yml file. How i can connect apache hive with apache hadoop? I want create table on hive. Extract data from api and load into hadoop hdfs. Have another way like create DockerFile ...
blue's user avatar
  • 1
0 votes
0 answers
42 views

Error when starting Yarn Nodemanager: [ERROR util.SysInfoWindows: ExitCodeException exitCode=1]

I am trying to setup Hadoop on my laptop (just to learn it and do some basics, and if I have the time a couple advanced things). I am on Windows 10, using Intellij, I also installed Hadoop 3.2.1 ...
Brahim Khalil Abid's user avatar
0 votes
0 answers
30 views

Problem with setting a environment variable in Windows

I am trying to setup a simple Hadoop Mapreduce example on my laptop. I installed everything as explained here: https://gist.github.com/vorpal56/5e2b67b6be3a827b85ac82a63a5b3b2e The thing is when I try ...
Brahim Khalil Abid's user avatar
0 votes
1 answer
40 views

Read ORC files from AWS S3 bucket in Flink app

We are using Flink version of 1.13.5 and trying to read the ORC files from AWS S3 location. And, we are deploying our application in a self-managed flink cluster. Please find the below code for ...
nirmal's user avatar
  • 107
0 votes
0 answers
39 views

issue running hadoop mapreduce wordcount

I am running Hadoop version 3.2.4 in windows and want to perform a WordCount operation on the file located in hadoop/share/hadoop/mapreduce/share/mapreduce-examples-3.2.4.jar. However, it failed, and ...
ryan's user avatar
  • 21
0 votes
1 answer
47 views

Hadoop Credentials using jceks Password File

while creating hadoop credential using : hadoop credential create mysql.password -provider jceks://file/home/mahadev.duary/credential.jceks it showing the following error : [mahadev.duary@ip-10-0-20-...
Mahadev Duary's user avatar
-1 votes
0 answers
15 views

Hadoop namenode UI is unavailable

I try to open the localhost:9780 in the browser to open hadoop-3.3.6 and the page did not open
eng.alialkazan's user avatar
0 votes
1 answer
43 views

Zero datanodes running in Hadoop 2.6.5 AWS

I am trying to input/upload the file to the hdfs hadoop but it's giving me this error When I run this command I get this : - hdfs dfs -put A.txt /input/ 23/11/08 22:19:25 WARN hdfs.DFSClient: ...
Rahul's user avatar
  • 11
0 votes
0 answers
16 views

Error when running Mapreduce in Cloud - RuntimeException: PipeMapRed.waitOutputThreads()

I am trying to find the top-n words using the hadoop map reduce where I am facing the error: Error My Mapper is: #!/usr/bin/env python import sys import re from collections import Counter # This ...
wei jong's user avatar
0 votes
1 answer
34 views

Trouble Show output using hadoop word count

I'm new to using Hadoop, and I want to execute Hadoop syntax using WordCount to count words. However, why is it that when I try to display the output, it doesn't appear? I would appreciate an ...
ryan's user avatar
  • 21

1
2 3 4 5
887