Questions tagged [hadoop2]
Hadoop 2 represents the second generation of the very popular open source distributed platform Apache Hadoop.
2,051
questions
0
votes
0
answers
16
views
while running hdfs namenode -bootstrapStandby
we have Apache hadoop cluster up and running with high availability
and Kerberized - no issues here
0
votes
0
answers
26
views
"ENOENT: No such file or directory" in hadoop while executing WordCount program
Trying on the wordcount example by using command "hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output" in linux shell, however it keep reminds me ...
0
votes
0
answers
24
views
Storing file with field names change in Hadoop using PIG STORE
I load a relation using
data = load 'path' using JsonLoader('class: chararray, marks: int');
datagrouped = group data on class;
total_marks = foreach datagrouped generate group as class, sum(data....
0
votes
0
answers
55
views
How to get application id from oozie job id
Is there a way to find out the applicationid using the oozie job id in single command?
I want to display the application id of the failed jobs for an oozie id.
oozie job -info
Can any option be used ...
1
vote
2
answers
140
views
What is the difference between FileInputStream/FileOutputStream Vs FSDataInputStream/FSDataOutputStream and where we will use them
I am trying to understand the difference between FileInputStream Vs FSDataInputStream and FileOutputStream Vs FSDataOutputStream.
I am trying to read a file from S3 bucket and apply some formatting ...
0
votes
1
answer
305
views
Spark 3.2.1 fetch HBase data not working with NewAPIHadoopRDD
Below is the sample code snippet that is used for data fetch from HBase. This worked fine with Spark 3.1.2. However after upgrading to Spark 3.2.1, it is not working i.e. returned RDD doesn't contain ...
0
votes
1
answer
219
views
Nifi service going down every day
I have install open source hadoop cluster and with Nifi. when i check status of sevice every day morning : status as below.
INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is running at PID ...
0
votes
0
answers
133
views
invalid variable name error on hadoop-functions.sh file on manjaro
when ı run this code hadoop version, ı get this error
/usr/lib/hadoop-3.3.1/bin/../libexec/hadoop-functions.sh: line 2386: HADOOP_VERSİON_USER: invalid variable name
/usr/lib/hadoop-3.3.1/bin/../...
0
votes
0
answers
59
views
Not resloved ERROR KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider
i am using open source apache Hadoop multinode cluster. with spark hive and habse.
while reading and writing data through error :ERROR KeyProviderCache: Could not find uri with key [dfs.encryption.key....
0
votes
1
answer
39
views
I successfully Copied Data From a Source Table to a Partitioned Table Using Hive but The Data Can't be found In The Partitioned Table
First of all, I load data from a local file to an unpartitioned table using hive
Creating raw data file:
stephen@stephen-VirtualBox:~/Workspace$ cat> static_source_demo.txt
11,test,2300,admin,c1
...
0
votes
0
answers
41
views
Why don't we completely fill the HDFS drives as Cache? (To Increase Locality)
Can empty HFDS space be used for cache to increase data locality? (let's call this Empty-Cache)
The flow would be:
When a spark job/yarn job runs it needs a file, it first would check if it's locally ...
0
votes
1
answer
27
views
I want to create(or atleast adopt) a new "customised" scheduler in hadoop. I saw a bunch of Scheduling Algorithms online that are stated as below
I want to create(or atleast adopt) a new "customised" scheduler in hadoop. I saw a bunch of Scheduling Algorithms online that are stated as below.
Hadoop Scheduling Algorithms
I know we can ...
0
votes
0
answers
88
views
ParquetWriter getting Cannot run program "chmod"
Hi I'm using mac with current java version 17.0.1-tem
I'm Trying to create local parquet file using ParquetWriter :
Using this code :
AvroParquetWriter.builder[GenericRecord](path)
.withSchema(...
0
votes
0
answers
62
views
In Apache hadoop 2.x. Does the current UserGroupInformation lose credentials in a child thread?
Here is the simple experiment I used to demonstrate the problem, written in scala, with Hadoop 2.7.7 as the only dependency:
import org.apache.hadoop.security.UserGroupInformation
import scala....
0
votes
0
answers
59
views
Yarn Apps API - Visibility guarantee?
I am querying Yarn's REST api to know when submitted apps have reached a terminal state (i.e., FINISHED, FAILED, or KILLED).
The query parameter that I am using is finishedTimeBegin, as in
curl host:...
0
votes
0
answers
37
views
Cannot open include file: 'org_apache_hadoop_io_nativeio_NativeIO.h' error while building "native" from Hadoop
I am trying to build the native project available in hadoop latest version (hadoop-trunk\hadoop-common-project\hadoop-common\src\main\native). However I am getting the following error. Please advise ...
1
vote
0
answers
188
views
Hadoop MapReduce job container throws java.io.FileNotFoundException for path /tmp/crunch-1324412807/p2/MAP
I am seeing the below error when I submit an Oozie job in an EMR Hadoop cluster. I could see that a certain container is not finding the temporary output file that is produced by another job (?).
I ...
1
vote
0
answers
117
views
No enum constant org.apache.orc.Compression.zlib Hive2
I have been working with Hadoop 2.7.7 and Hive 2.0 for the last 2 weeks.
I have created a table with ORC without any compression, Loaded the data from an external table pointing to the CSV.
I created ...
0
votes
0
answers
150
views
Writing dataframe to hive table fails
Pyspark:
result.write.mode("overwrite").saveAsTable('Table_name')
Derived 'result' dataframe using hive_context.sql, query is flattening json object by get_json_object, explode.
After ...
0
votes
0
answers
24
views
Running MapReduce on multicore in hadoop 2.6
I am trying to run a MapReduce application in a single node machine and utilize its multicore to act as mapper and reducer. I am setting it in the following way:
Configuration conf = new Configuration(...
0
votes
0
answers
22
views
How to solve Sequence ordering using MapReduce
if my mapper emits (key, value) pairs such as:
(A, B) implying A results in B
(B, C) implying B results in C
(C, D) implying C results in D
etc
I want my reducer to emit as follows
(A, B -> C -&...
0
votes
0
answers
44
views
Which StorageClass should use for HDFS
I am trying to deploy HDFS, Kafka, Nifi in My K8s-Cluster.
For Kafka using this helm repo and for Nifi using this helm repo.
I was using rook cephfs as a storageClass for this.
Is there any other ...
1
vote
1
answer
436
views
How to add one extra partition to external hive table?
I have hive table like below
create external table transaction(
id int,
name varchar(60))
month string
)
PARTITIONED BY (
year string,
transaction_type_code varchar(20)
)
...
1
vote
1
answer
281
views
NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V
I am using <spark.version>3.0.2</spark.version> with detla version 0.8.0 in my project.
and running with
export SPARK_HOME=/pkg/spark-3.0.2-bin-hadoop2.7-hive1.2
$SPARK_HOME/bin/spark-...
0
votes
0
answers
12
views
how to provide cluster id to two namenodes in ha-hdfs
I have configured HA-HDFS.
I am facing an issue with ClusterID Where the ClusterID of my namenodes and datanodes don't match therefore, my datanodes are not getting started.
How to provide the same ...
1
vote
0
answers
526
views
Exclude some directories while Copying Data on Hadoop Cluster using hadoop distcp
Is there any way to skip some directories while copying hadoop data from one cluster to another cluster? I.e. I am copying some data from my existing cluster to new cluster but i don't want to copy ...
1
vote
1
answer
73
views
Can we store multiple types of data in a data warehouse?
I want to ask that can we store various types of data in a Hadoop data warehouse? Data like RDBMS, JSON Doc, Cassandra Keyspace, txt, CSV, etc? Are they all stored in HDFS?
0
votes
1
answer
104
views
How to import other python modules and packages
I have the following project structure,
work_directory:
merge.py
a_package
(i.e. a python file merge.py and a directory a_package under the directory "work_directory")
I wrote a ...
0
votes
0
answers
78
views
hadoop - Compress parquet files using bzip
I have a huge databaes which I am trying to copy to other cluster using distcp. However, due to limited bandwidth I am thinking to compress this files using bzip (it's already in snappy compression. ...
1
vote
1
answer
193
views
How to config gcs-connector in local environment properly
I'm trying to config gcs-connector in my scala project but I always get java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
Here is my project ...
-1
votes
1
answer
150
views
I want the parquet file data to be into the below string format.Can some one help me?
i am trying to get the data in the below format. Can some one help me to get a UDF for spark and scala. I am new to this. Can some one help with this?
The output i am expecting is string output
Java|...
0
votes
0
answers
34
views
How to get FileSystemStats like totalNumberFiles, totalFileSize, maxFileSize etc on SparkSQL?
In hive, to get the FileSystemStats like total number of files we run the query
Show table extended like '<db>.<table>' partition(<partition>)
I tried running
show table extended ...
0
votes
0
answers
146
views
Reducing war file size of a JEE Project
I am a beginner in the big data area and I have a task of reducing the war size of a JEE project. I am getting the following error :
ERROR [stderr] (ServerService Thread Pool -- 81) log4j:ERROR A &...
1
vote
1
answer
423
views
Why Reducer required in a Hive Insert
Question is related to working of MapReduce job when we fire a insert into statement from hive command line.
While inserting records into a hive table: As there is no aggregations involved while ...
1
vote
1
answer
94
views
how can I efficiently load multiple files in the same path to hadoop?
for example:
I have more than 10 files in path /home/doc/data/ : 202012.txt, 202101.txt, 202102.txt.......
If I WANT TO load all files for 2021 I have to use :
hadoop fs -copyFromLocal /home/doc/data/...
0
votes
1
answer
134
views
Hadoop MapReduce example command not found
I've installed the Hadoop file and I'm trying to run the MapReduce example in the terminal, but am getting the command not found message, can someone help me with this issue, thanks.
Ismails-MacBook-...
0
votes
1
answer
298
views
HBase logcleaner does not delete the oldWALs files
In my lab, HBase archive Write Ahead Logs i.e. oldWALs files are not deleted and oldWALs directory is growing quickly in terabyte.
8.1 K 24.4 K /hbase/.hbase-snapshot
0 0 /hbase/.hbck
...
0
votes
0
answers
527
views
I'm getting an error java.io.FileNotFoundException: No such file or directory in S3 bucket, trying to access it from AWS EMR cluster
I'm working on a standalone java application which is using Hadoop Map Reduce framework. I'm running this application on AWS EMR cluster and the input files are present in S3 bucket. I have not facing ...
1
vote
1
answer
252
views
Problem with data locality when running Spark query with local nature on apache Hadoop
I have a Hadoop cluster that uses Apache Spark to query parquet files saved on Hadoop. For example, when I'm using the following PySpark code to find a word in parquet files:
df = spark.read.parquet(&...
0
votes
0
answers
951
views
TOK_FUNCTION not supported in insert / values
I am trying to insert data into a table and getting the error
"Unable to create temp file for insert values expression of type TOK_FUNCTION not supported in insert/values". My table ...
1
vote
0
answers
50
views
Custom health check in impala
I have a use case that if a particular service is down on a node, no job should get scheduled there anymore. For Hive or Spark, I can use the concept of external health script present in Hadoop, which ...
1
vote
1
answer
163
views
Difference between the date returned by hadoop fs -ls and hadoop fs -stat
Is there any difference between the dates returned by
hadoop fs -ls myfile
and
hadoop fs -stat myfile
0
votes
2
answers
77
views
Multiple counts in hive query not giving expected results
I have data as follows.
I am trying to get the following results with the query what I have developed.
select date,name,act_cd,type_cd, type
count(distinct CASE WHEN act_cd in (1) THEN key_id ELSE 0 ...
3
votes
0
answers
226
views
Facing an issue in submitting a mapreduce job with mapr cluster
Below are my Mapr Cluster (Non secure) configurations.
MapR version - 6.1
Os - Ubuntu 16.04
Hadoop version - 2.7.0
Nodes - Single node
core-site.xml:
<?xml version="1.0" encoding="...
1
vote
1
answer
505
views
hive script failing due to heap space issue to process too many partitions
my script failing due to a heap space issue to process too many partitions. To avoid the issue I am trying to insert all the partitions into a single partition but I am facing the below error
FAILED: ...
0
votes
1
answer
153
views
Facing issue in setting up oozie with secure MapR cluster
We are facing an issue with setting up an oozie service with secure mapr cluster.
We are using the MapR installer to setup the MapR Cluster. Below are the configuration and steps that we followed.
...
0
votes
1
answer
139
views
How to copy a file from /user/maria_dev/tutorials/test.csv (HDP) to /sandbox/tutorial-files/640/nifi/input (HDF)?
Using Hortonworks HDP 3.0.1 and HDF Sandbox on Docker Desktop
I have logged into HDP, and tried to copy a file from /user/maria_dev/tutorials/test.csv point to HDF's /sandbox/tutorial-files/640/nifi/...
2
votes
0
answers
41
views
Spark Job only processes file in a single spark container
I am reading in a csv file from gcs and I need to go through each row and call an api to get some data back and appended to a new dataframe.
the code goes something like this:
DataFrame<Row> df =...
5
votes
0
answers
2k
views
How to resolve "SecurityManager: authentication disabled; exception in spark distribution cluster"?
I have created a Spark distribution cluster in Linux (Ubuntu) environment to run Spark submit job. I have 3 machines (m1, s1 and s2; m1 is master, s1 and s2 are slaves machines).
In master Spark and ...
2
votes
2
answers
348
views
phoenix view issue on top of hbase table
I am using Hbase version 2.0.2 and phoenix V5.0.0. I have an HBase table on top of that We have created Phoenix view, using the below steps
hbase(main):007:0> create 'phownix_test','details'
...