Questions tagged [mapreduce]

MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes

mapreduce
4,256 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
0 votes
1 answer
846 views

Hadoop issue, trying to run basic WordCount MapReduce on hadoop

I'm running hadoop on my Windows 10 laptop, but I found some guides on getting it up and running and running a basic word count program. The hadoop installation seems fine as I'm able to see the ...
Redcoatwright's user avatar
0 votes
2 answers
111 views

huge difference between mappers execution time

I try to import a table with sqoop; i use 4 mappers. The problem is there's a huge difference between execution time between the mappers. Some less than 10 mints the others is more than one hour. may ...
Zied Hermi's user avatar
0 votes
1 answer
474 views

Apache Spark vs. MapReduce

I've been looking up the differences between Spark and MapReduce and all I've really found is that Spark runs in memory and on disk which makes it significantly faster. I also read that MapReduce is ...
Redcoatwright's user avatar
0 votes
2 answers
164 views

Is maven JAR runnable on hadoop?

To produce jar from hadoop mapreduce program(mapreduce wordcount example) i used maven. Here i successfully done 'clean' and 'install'. Also 'build' successfully by running as a Java Application by ...
Asif Ishtiaq's user avatar
0 votes
1 answer
3k views

Spark - most frequent word following a given word

I am learning Scala and I am trying to figure out how to create a MapReduce program in Scala to find, for each word in the file which word that follows the most. this is what I have. it works but I ...
drizzle's user avatar
0 votes
0 answers
45 views

Hadoop Task Scheduling

I am a little bit confused regarding task assignment in Hadoop, I already know how task scheduling such as FIFO, Fair and capacity schedulers for multiple jobs work. But my question is how one ...
Salah Atef's user avatar
0 votes
1 answer
126 views

MapReduce Output file is blank when trying to partition the dataset on some condition

I have a dataset which contains emoployee satisfaction level index,dept ,last evaluation, number of projects etc.From that file i have to find the average satisfation index of each dept.I have used a ...
Mandrek's user avatar
  • 1,199
0 votes
1 answer
573 views

Hadoop Mapreduce (Java) - error counting all unique words in text with Reducer as Combiner

I adapted the standard word count Hadoop example to count all the unique words from a series of input text files using a user-defined counter, with an enum defined in the driver class like so: public ...
Ian F's user avatar
  • 17
0 votes
0 answers
90 views

Hadoop - ld: symbol(s) not found for architecture x86_64

I have a problem in running hadoop pipes code. My environment is: Hadoop2.7.3 Mac10.13.3 single node. My code is following: #include <algorithm> #include <limits> #include <stdint....
Joon's user avatar
  • 63
0 votes
0 answers
944 views

Map reduce to produce n most frequent words

I have MapReduce code which prints the amount of times a word was seen in a document. I would like to change this code to produce the N most frequent words. N being an input parameter. Here is the ...
benny's user avatar
  • 53
0 votes
2 answers
74 views

How to read same input file in reduce task which I read in map task (Mapreduce)

I have been trying to read the file in the reducer which is also the input of the mapper. So is there any way I can get access to that file in reducer?
Yaseen Saleem's user avatar
0 votes
0 answers
25 views

Isn't the term shuffling in MapReduce misleading?

I think the term shuffle refers to randomly reordering elements in a sequence [1]. Therefore, the first time I saw shuffling in MapReduce, I thought it's trying to uniformly distribute workload to ...
Lingxi's user avatar
  • 14.9k
0 votes
1 answer
9 views

How to retrieve the map task output files

I am trying to retrieve the map task output so I can use it in another method. But the output is getting flushed even if I set the tmp folder in core-site.xml
Artwell's user avatar
0 votes
0 answers
170 views

Multiple input files and output the each file result in 1 line in 1 file (Hadoop:MapReduce)

I am stuck in separate each file wordcount result in 1 line. I hope to output all files result in 1 file and each file result represent in 1 line. Expected output.txt format file1 1, 2, 3, 4, …, 100 ...
Amy Chan's user avatar
0 votes
1 answer
243 views

HDP 2.4, How to collect hadoop mapreduce log using flume in one file and what is the best practice

We are using HDP 2.4 and have many map reduce jobs written in various ways ( java MR / Hive / etc. ) . The logs are collect in hadoop file system under the application ID. I want to collect all the ...
Rajiv's user avatar
  • 3
0 votes
0 answers
145 views

Design MapReduce function in Hadoop Streaming

I am using Hadoop Streaming (ver 2.9.0) with Python. The input of my MapReduce is a text file. My <key,value> pairs are as following, where key are numeric, and value are sets of numeric): k1,...
cdt's user avatar
  • 85
0 votes
0 answers
55 views

mapper and reducer code incorrect

Mapper code :- #!/usr/bin/python import sys for line in sys.stdin: data = line.strip().split("\t") if len(data) == 6: date, time, place, temp, pressure, humidity = data ...
jitender singh's user avatar
0 votes
0 answers
18 views

Can the order of values be determined in the reducer?

Suppose I want to do some research of the annual sales trending of products. In the raw input I have all products sorted by years line by line like this: product-id sales-volume year. In the mapper I ...
Marvis Lu's user avatar
  • 441
0 votes
1 answer
432 views

Scala/Spark reduce RDD of class objects

I need your help for my last step of a school project. val conf: SparkConf = new SparkConf() .setMaster("local[*]") .setAppName("AppName") .set("spark.driver.host", "localhost") val sc: ...
Skander Farhat's user avatar
0 votes
0 answers
199 views

How Impala does not use map reduce?

I am going through this link. Generally Impala is compared to Hadoop Map-Reduce/Hive but here I want it to compare it from the map reduce programming paradigm. I am having hard time understanding ...
Amit Kumar's user avatar
  • 2,715
0 votes
1 answer
294 views

Read byteswritable file from hadoop

My mapreduce job is currently writing the output(byte array) as byteswritable in a file. Now I need to read this bytearray in my java program. When I am accessing the output file in my java program ...
Y0gesh Gupta's user avatar
  • 2,194
0 votes
0 answers
309 views

While running jobs on hadoop,got the following error

I tried to run a map reduce job. Both my mapper code and reducer code (python codes)are working fine in my local system.But when i tried to run it on hadoop i am getting the following error Error: ...
Bharat Gera's user avatar
0 votes
1 answer
65 views

Map reduce | Grouping on response of map function and show count

the response of my map function is: key: [ 1525132800000, 1525152600000 ] value: { "propertyId": "DAWN", "startDate": "2018-05-01", "endDate": "2018-05-04", "verificationStatusCode": "ICMO"...
Krishan Jangid's user avatar
0 votes
1 answer
323 views

Finding Top ten most viewed movies using MapReduce program

While trying to find the top ten most viewed movies with their name (Ascending or Descending order) using MapReduce, I got the below error. "Error: java.io.IOException: Unable to initialize any ...
Arvind Pant's user avatar
0 votes
0 answers
138 views

Apache hadoop Java API - Difference between Text.readFields() vs Text.readString()

I am trying to understand the difference between Text.readFields() vs Text.readString() in Text class. From the javadoc, it is not very clear. The only slight hint is about that probably Text....
Gyanendra Dwivedi's user avatar
0 votes
1 answer
99 views

Mapreduce Mapper explaination

There is an NCDC weather data set example in Hadoop definite guide. The Mapper class code is as follows Example 2-3. Mapper for maximum temperature example import java.io.IOException; import org....
Anirudh's user avatar
  • 39
0 votes
0 answers
708 views

Sqoop - Failed to empty staging table before export

I have faced an interesting situation while trying to export data from HDFS to MySQL database. I'm getting Error during export: Failed to empty staging table before export run. First I created two ...
samba's user avatar
  • 2,961
0 votes
1 answer
57 views

execluding specific nodemanagers while running a job

I am using hadoop-2.7.3. How i can i exclude specific nodemanagers only for some jobs. I tried some configurations but it was applicable to resourcemanager level not with nodemanager.Is it possble to ...
Rahul's user avatar
  • 459
0 votes
0 answers
119 views

Always submit mapreduce job to localhost while add xxxx-site.xml to configuration object

I have problem submit MapReduce job remotely in Java. I have a CDH cluster, I copied mapred-site.xml, yarn-site.xml, core-site.xml and hdfs-site.xml to my Idea project. But I found it always generate ....
user avatar
0 votes
0 answers
257 views

Java Hadoop, how to save mapper output to HDFS in addition to Reducer output

I'm running across a situation where I have to post-process the results of the mapper output into my MySQL database for further visualization. So I need to have a single Hadoop (3.0) Map/Reduce job ...
CryptoMonkey's user avatar
0 votes
0 answers
171 views

Incorrect JAVA_HOME path when Installation of Approx-Mapreduce

I am trying to install ApproxMapreduce on Hadoop 3.0.0 on my Mac. And I followed the instructions from this link that worked for Hadoop 1.1.2. https://github.com/goiri/ApproxHadoop And I got an ...
JDDD's user avatar
  • 71
0 votes
1 answer
30 views

Execute mapreduces in Spark

My teacher said that we can execute MapReduces in Spark. But, since Spark is said to be faster that hadoop, this means that is always better to do mapreduces in spark. So, Hadoop MapReduces become ...
Qwerto's user avatar
  • 275
0 votes
0 answers
465 views

Apply function to subset of Spark Datasets (Iteratively)

I have a Dataset of geospatial data that I need to sample in a grid like fashion. I want divide the experiment area into a grid, and use a sampling function called "sample()" that takes three inputs, ...
user306603's user avatar
0 votes
0 answers
674 views

mapreduce to read hive table and write to hdfs location with context

I am looking out for the mapreduce program to read from one hive table and write to hdfs location of first column value of each record. And it should contain only map phase not reducer phase. Below ...
ae8's user avatar
  • 53
0 votes
1 answer
443 views

How to read all lines in map (mapreduce)

Map(offset,text) read only one line per time. My file includes many lines and each line represents a data in 3D: x1,x2,x3. I want to choose n of them from the block, which have most high density. (And ...
Menkot's user avatar
  • 714
0 votes
2 answers
208 views

Process huge data simultaneously

i have one table with a lot of data: id | title | server 1 | server 2 | server 3 -------------------------------------------- 1 | item1 | 110.0.0.1| 110.0.0.2| 110.0.0.3 ...
Gogo's user avatar
  • 292
0 votes
1 answer
2k views

Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:

I have created a input directory and put sample file in it.I have created an output directory also.but at the time of mapreduce program execution i got the below error.Here is my command to execute ...
Biswajit's user avatar
  • 2,466
0 votes
1 answer
1k views

How to deal with Mapreduce by identifying the keys in python Hadoop

I have two key values from map function: NY and Others. so, the output of my key is: NY 1, or Other 1. Only these two cases. my map function: #!/usr/bin/env python import sys import ...
HHKSHD_HH's user avatar
0 votes
1 answer
18 views

Is _logs/skip/ related to hadoop version?

I am doing project about MapReduce task failures. According to Hadoop Beginner's Gudie(Garry Tukington), all of the skip data is stored in _logs/skip/ folder. The author used Hadoop 1.0 version. I am ...
Miss.Saung's user avatar
0 votes
2 answers
873 views

Exception in thread "main" org.apache.Hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf

I am a new Hadoop user. My program is to skip bad record data in mapreduce. I didnot skip bad data so firstly, I am not trying to skip data and I want to find the which error occur. So, I add ...
Miss.Saung's user avatar
0 votes
0 answers
730 views

Hadoop on Windows cmd, one mapper and multiple inputs, Error: subprocess failed

I want to execute a Python file which is related to machine learning and as you know there are two files as inputs (train and test) which are important to make learning process. Also I have no reduce ...
Mahsa Hassankashi's user avatar
0 votes
1 answer
1k views

Container Allocation | Container Size in Hadoop Cluster

As we know, Default Minimum Container Size is 1024MB Consider this Scenario: I have setup a Hadoop Cluster <1-Master & 3-Datanodes> in VIRTUAL BOX. Namenode: master (3GB RAM) Datanodes: ...
YoxBox's user avatar
  • 1
0 votes
1 answer
127 views

Phoenix csv Bulk load fails with large data sets

I'm trying to load a dataset (280GB) using the Phoenix csv bulk load tool on a HDInsight Hbase cluster. The job fails with the following error: 18/02/23 06:09:10 INFO mapreduce.Job: Task Id : ...
IronMan's user avatar
  • 329
0 votes
0 answers
93 views

Java CSV Regex matching per element for validation of each cell

Ok, so I have this issue that I can't seem to wrap my head around. what I'm trying to do, is the following: read in a CSV file line by line, split it up at the comma and pass it into a hashmap and ...
Glennismade's user avatar
0 votes
0 answers
83 views

DateTime non numerical format

I am a beginner in mapreduce and I want to create a job that reads parquet files containing data of a taxi comapny involving the trip-id, pick_up datetime, amount ... the parquet file structure is as ...
rafik_bougacha's user avatar
0 votes
2 answers
131 views

mapreduce job fails on AvroKeyInputFormat only on a cluster

I have some map-reduce working fine on y local machine. When I run the job on a remote cluster, I get this error : Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org....
paris inserm's user avatar
0 votes
1 answer
133 views

Get list of executed job on Hadoop cluster after cluster reboot

I have a hadoop cluster 2.7.4 version. Due to some reason, I have to restart my cluster. I need job IDs of those jobs that were executed on cluster before cluster reboot. Command mapred -list provide ...
Hafiz Muhammad Shafiq's user avatar
0 votes
0 answers
1k views

Why mapred.min.split.size doesnt change the number of mappers for my query

I was doing some deep dive with a simple use case to see how we can control the number of mappers launched in hive. This is what I did : Step1 : Found out how much is the block size of the system. ...
Abhi's user avatar
  • 61
0 votes
0 answers
28 views

Making third collection out of two collections and want all of fields in the third collections in MONGODB

var mapDevice = function(){ var values = { Device_Id: this.Device_Id, Module_Id : this.Module_Id, Area_Id:...
Yashwin Munsadwala's user avatar
0 votes
1 answer
775 views

Error in Container launch in hadoop on Windows

I am running a map reduce job on windows and it gives me this error. It specified that there is exception in container launch. Stack trace: ExitCodeException exitCode=1: '/tmp/hadoop-user' is not ...
Vipin Dubey's user avatar

1
40 41
42
43 44
86