Questions tagged [hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The core consists of a distributed file system (HDFS) and a resource manager (YARN). Various other open-source projects, such as Apache Hive use Apache Hadoop as persistence layer.

hadoop
Filter by
Sorted by
Tagged with
5 votes
1 answer
12k views

Unable to connect to HIVE2 via JAVA

Referring to Hive2 created a simple java program to connect to HIVE2 server (not local) have added all mentioned jars in the above link in the class path in eclipse as well however when I run the code ...
Vinod's user avatar
  • 386
5 votes
1 answer
2k views

How do I find where the environment variables are defined in Linux

I just started learning hadoop (CentOS7), here is a question about environment variable: In my VM, rxie is the logged on user: HADOOP_CONF_DIR = /opt/hadoop/hadoop-2.7.2/etc/rxie There is no path of ...
mdivk's user avatar
  • 3,635
5 votes
3 answers
11k views

Submit Spark job on Yarn cluster

I am struggling for more than 2 days now with the following problem. I wrote a basic "HelloWorld" script in Scala: object Hello extends App{ println("WELCOME TO A FIRST TEST WITH SCALA COMPILED ...
user2573552's user avatar
5 votes
1 answer
39k views

How to Append new data to already existing hive table

How to append the records to existing partitioned Hive table? For example I have existing external Table called "ip_country" and dataset is testdata1. If dataset grows say like my dataset in next day ...
marjun's user avatar
  • 726
5 votes
1 answer
3k views

Batching multiple files to Amazon S3 using the Java SDK

I'm trying to upload multiple files to Amazon S3 all under the same key, by appending the files. I have a list of file names and want to upload/append the files in that order. I am pretty much exactly ...
user1413793's user avatar
  • 9,217
5 votes
0 answers
3k views

Can't access hadoop web ui for job tracker [closed]

I'm trying to set up hadoop and nutch to run on EC2. To get started, I have followed the excellent NutchHadoopTutorial. Most everything works as it should, except that I am unable to access any of the ...
Jimbodean's user avatar
5 votes
2 answers
47k views

Combine columns from multiple columns into one in Hive

Is there any way to do kind of reverse thing for explode() function in Apache Hive. Let's say I have a table in this form id int, description string, url string, ... And from this table I would like ...
user1831986's user avatar
5 votes
4 answers
5k views

Does Apache Flink AWS S3 Sink require Hadoop for local testing?

I am relatively new to Apache Flink and I am trying to create a simple project that produces a file to an AWS S3 bucket. Based on the documentation it looks like I am required to have Hadoop installed ...
medium's user avatar
  • 4,186
5 votes
2 answers
10k views

Building Hadoop with Maven - "Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (create-testdirs)"

I'm trying to build the Hadoop trunk using Maven. I got the project with svn. But then, I tried mvn clean install. But I'm getting the following error: [INFO] Apache Hadoop Main .........................
Sreejith Ramakrishnan's user avatar
5 votes
2 answers
22k views

ModuleNotFoundError: No module named 'py4j'

I installed Spark and I am running into problems loading the pyspark module into ipython. I'm getting the following error: ModuleNotFoundError Traceback (most recent call ...
Jassim Elakrouch's user avatar
5 votes
5 answers
19k views

Why Hive is not supporting Stored Procedure?

Why hive is not supporting Stored procedure? If its not supporting then how we will handle Sp in Hive? have any alternate solution? (Because we have a already a data base is there in mssql) What about ...
Haris N I's user avatar
  • 6,634
5 votes
8 answers
9k views

Hadoop performance

I installed hadoop 1.0.0 and tried out word counting example (single node cluster). It took 2m 48secs to complete. Then I tried standard linux word count program, which run in 10 milliseconds on the ...
Rafal Rusin's user avatar
5 votes
1 answer
4k views

Change spark _temporary directory path

Is it possible to change the _temporary directory where spark save its temporary files before writing? In particular, since I am writing single partitions of a table I woud like the temporary folder ...
Alessandro's user avatar
5 votes
3 answers
6k views

hadoop2.2.0 append file occur AlreadyBeingCreatedException

I have meet a problem about hadoop2.2.0 append operation. I append some bytes to a hdfs file by HDFS java API.First I will create the target file if the file doesn't exist before appending operation, ...
user2994487's user avatar
5 votes
5 answers
4k views

Pig in grunt mode

I have installed cygwin, hadoop and pig in windows. The configuration seems ok, as I can run pig scripts in batch and embedded mode. When I try to run pig in grunt mode, something strange happens. ...
dimmuborgirrld's user avatar
5 votes
1 answer
3k views

Spark coalesce relationship with number of executors and cores

I'm bringing up a very silly question about Spark as I want to clear my confusion. I'm very new in Spark and still trying to understand how it works internally. Say, if I have a list of input files(...
sikara tijuhara's user avatar
5 votes
5 answers
9k views

Oozie job stuck at START action in PREP state

I have an Oozie job that I start from java client which gets stuck at START action and it says that it is RUNNING, but the START node is in PREP state. Why is that and how to fix the problem? The ...
Kobe-Wan Kenobi's user avatar
5 votes
1 answer
5k views

Apache Pig: unable to run my own pig.jar and pig-withouthadoop.jar

I have a cluster running Hadoop 0.20.2 and Pig 0.10. I'm interested to add some logs to Pig's source code and to run my own Pig version on the cluster. What I did: built the project with 'ant' ...
sunny's user avatar
  • 1,897
5 votes
1 answer
6k views

Hadoop mapreduce has "Cannot resolve the host name" error

Now I run Hadoop mapreduce job, the input data comes from HBase table, recently there is an error, the error is below: ERROR mapreduce.TableInputFormatBase: Cannot resolve the host name for /172.16.4....
sabrina's user avatar
  • 111
5 votes
3 answers
5k views

Hadoop C++ HDFS test running Exception

I'm working with Hadoop 2.2.0 and trying to run this hdfs_test.cpp application: #include "hdfs.h" int main(int argc, char **argv) { hdfsFS fs = hdfsConnect("default", 0); const char* ...
user3077628's user avatar
5 votes
2 answers
772 views

Successful task generates mapreduce.counters.LimitExceededException when trying to commit

I have a Pig script running in MapReduce mode that's been receiving a persistent error which I've been unable to fix. The script spawns multiple MapReduce applications; after running for several hours ...
Jeff Rosenberg's user avatar
5 votes
3 answers
2k views

Getting emr-ddb-hadoop.jar to connect DynamoDB with EMR Spark

I have a DynamoDB table that I need to connect to EMR Spark SQL to run queries on the table. I got the EMR Spark Cluster with release label emr-4.6.0 and Spark 1.6.1 on it. I am referring to the ...
Shweta's user avatar
  • 137
5 votes
2 answers
9k views

Parsing PDF files in Hadoop Map Reduce

I have to parse PDF files , that are in HDFS in a Map Reduce Program in Hadoop. So i get the PDF file from HDFS as Input splits and it has to be parsed and sent to the Mapper Class. For implementing ...
WR10's user avatar
  • 443
5 votes
1 answer
12k views

Hadoop ChainMapper, ChainReducer [duplicate]

I'm relatively new to Hadoop and trying to figure out how to programmatically chain jobs (multiple mappers, reducers) with ChainMapper, ChainReducer. I've found a few partial examples, but not a ...
moodywoody's user avatar
  • 2,159
5 votes
1 answer
5k views

Hive alter table change column name gives 'NULL' to the renamed column

I had tried to rename an existing column in a table to a new one. But after the name change, the new column is giving me only 'NULL' value. The storage format for the table in Parquet. For e.g, '...
Adarsh's user avatar
  • 187
5 votes
3 answers
11k views

Apache Spark: how to cancel job in code and kill running tasks?

I am running a Spark application (version 1.6.0) on a Hadoop cluster with Yarn (version 2.6.0) in client mode. I have a piece of code that runs a long computation, and I want to kill it if it takes ...
Zohar Meir's user avatar
5 votes
2 answers
4k views

Reducers stopped working at 66.68% while running HIVE Join query

Trying to join 6 tables which are having 5 million rows approximately in each table. Trying to join on account number which is sorted in ascending order on all tables. Map tasks are successfully ...
Venkat Ankam's user avatar
5 votes
2 answers
2k views

how to load a tarball to pig

i have a log files that is in a tarball (access.logs.tar.gz) loaded into my hadoop cluster. I was wondering is their way to directly load it to pig with out untaring it?
nashr rafeeg's user avatar
5 votes
2 answers
11k views

Hadoop Spill failure

I'am currently working on a project using Hadoop 0.21.0, 985326 and a cluster of 6 worker nodes and a head node. Submitting a regular mapreduce job fails, but I have no idea why. Has anybody seen this ...
Nico's user avatar
  • 411
5 votes
1 answer
12k views

properly loading datetime in pig

I'm loading a tsv file with a datetime column and long column with: A = LOAD 'tweets-clean.txt' USING PigStorage('\t') AS (date:datetime, userid:long); DUMP A; An example line of input: Tue Feb 11 ...
rcj's user avatar
  • 651
5 votes
1 answer
9k views

how to write java Log file using the logger api while using hadoop

i Wrote a map reduce code that i want to debug. in order to do so i cant use standard output, because the Hadoop platform doesn't print it to the screen unless an error occurs. instead i tried to ...
Gabriel H's user avatar
  • 1,576
5 votes
1 answer
17k views

How to submit multiple jars to workers through sparkSession?

I am using spark 2.2.0. Below is the java code snippet which I am using as a job on spark: SparkSession spark = SparkSession.builder() .appName("MySQL Connection") .master("...
ABC's user avatar
  • 354
5 votes
1 answer
4k views

getting java.lang.ClassCastException: class java.lang.String in running a simple MapReduce Program

I am trying to execute a simple MapReduce program, wherein the Map takes the input, splits it in two parts(key=> String and value=>Integer) The reducer sums up the values for a corresponding key I am ...
addicted20015's user avatar
5 votes
1 answer
6k views

Datanode failing in Hadoop on single Machine

I set up and configured sudo node hadoop environment on ubuntu 12.04 LTS using following tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#formatting-...
Jaipal R's user avatar
5 votes
1 answer
4k views

Hadoop in pseudo-distributed Mode. Connection refused

P.S: Please do not mark this as duplicate Hi, I have been trying to set up and run Hadoop in the pseudo Distributed Mode. When I run the start-all.sh script I get this output: starting namenode, ...
Raveesh Sharma's user avatar
5 votes
1 answer
1k views

How does hive handle insert into internal partition table?

I have a requirement to insert streaming of records into Hive partitioned table. The table structure is something like CREATE TABLE store_transation ( item_name string, item_count int, ...
Nageswaran's user avatar
  • 7,605
5 votes
3 answers
8k views

Connecting to remote HBase service using Java

I have a small sample code in which I try to establish a connection to a remote HBase entity. The code runs on a windows machine without HBase installed and I try to connect to a remote Ubuntu Server ...
Zeliax's user avatar
  • 5,196
5 votes
2 answers
6k views

Multiples Hadoop FileSystem instances

I have a class (I removes try/catch for readability): public class HadoopFileSystem { private FileSystem m_fileSystem = null; public HadoopFileSystem() { Configuration ...
Apaachee's user avatar
  • 900
5 votes
2 answers
19k views

Hadoop Streaming Job Failed (Not Successful) in Python

I'm trying to run a Map-Reduce job on Hadoop Streaming with Python scripts and getting the same errors as Hadoop Streaming Job failed error in python but those solutions didn't work for me. My ...
sirentian's user avatar
  • 187
5 votes
0 answers
1k views

TSocket read 0 bytes accessing hive with python

I'm trying to connect Hive to fetch some tables using pyhive in Embedded/Pseudo Mode. Below is the code i have used #!/usr/bin/env python # coding: utf-8 from pyhive import hive from TCLIService....
user7422128's user avatar
5 votes
2 answers
724 views

hadoop: tasks not local with file?

I ran a hadoop job and when I look in some map tasks I see they are not running where the file's blocks are. E.g., the map task runs on slave1, but the file blocks (all of them) are in slave2. The ...
IttayD's user avatar
  • 28.7k
5 votes
2 answers
6k views

what does 2n + 1 quorum mean?

I've come across this when describing the Zookeeper configuration for HBase, and I'm unfamiliar with the term. Does the 'N' have anything to do with the number of nodes in my HBase cluster? Or the ...
Big Bird's user avatar
  • 267
5 votes
6 answers
20k views

Get list of files from hdfs (hadoop) directory using python script

How to get a list of files from hdfs (hadoop) directory using python script? I have tried with following line: dir = sc.textFile("hdfs://127.0.0.1:1900/directory").collect() The directory have ...
sara's user avatar
  • 115
5 votes
1 answer
3k views

How do you setup multiple Spark Streaming jobs with different batch durations?

We are in the beginning phases of transforming the current data architecture of a large enterprise and I am currently building a Spark Streaming ETL framework in which we would connect all of our ...
Brendan's user avatar
  • 136
5 votes
1 answer
10k views

What's the best way to support array column types with external tables in hive?

So i have external tables of tab delimited data. A simple table looks like this: create external table if not exists categories (id string, tag string, legid string, image string, parent string, ...
Matthew Rathbone's user avatar
5 votes
1 answer
3k views

Hadoop per-file block size

In Hadoop book it is said that we can specify per-file block size at the time of creation of file. "The most natural way to increase the split size is to have larger blocks in HDFS, by setting dfs....
sunillp's user avatar
  • 993
5 votes
1 answer
843 views

Spark 1.6.1 SASL

I wonder if anyone gotten SASL to work with Spark 1.6.1 on YARN? Basically Spark documentation states that you only require 3 parameters enabled: spark.authenticate.enableSaslEncryption=true ...
Fabian Tan's user avatar
5 votes
1 answer
3k views

Select if table exists in Apache Hive

I have a hive query which is of the format, select . . . from table1 left join (select . . . from table2) on (some_condition) The table2 might not be present depending on the environment. So I would ...
Maverick's user avatar
  • 728
5 votes
2 answers
4k views

How to implement self-join/cross-product with hadoop?

It is common task to make some evaluation on pairs of items: Examples: de-duplication, collaborative filtering, similar items etc This is basically self-join or cross-product with the same source of ...
yura's user avatar
  • 14.6k
5 votes
2 answers
4k views

pig skewed join with a big table causes "Split metadata size exceeded 10000000"

We have a pig join between a small (16M rows) distinct table and a big (6B rows) skewed table. A regular join finishes in 2 hours (after some tweaking). We tried using skewed and been able to improve ...
ihadanny's user avatar
  • 4,407

1
37 38
39
40 41
87