Questions tagged [mapreduce]

MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes

mapreduce
Filter by
Sorted by
Tagged with
0 votes
1 answer
16k views

Hive Error : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

I have got twitter data using flume on HDFS. Have 3 node cluster and MySQL Metastore for hive. When i execute below query select user_name.screen_name, user_name.followers_count from ...
0 votes
0 answers
10 views

Streaming Command Failed [closed]

2024-04-12 03:34:14,276 INFO mapreduce.Job: map 100% reduce 100% 2024-04-12 03:34:14,281 INFO mapreduce.Job: Job job_1712890757540_0011 failed w ...
0 votes
0 answers
14 views

Node Participation in Hadoop-Streaming with Python Scripts

I am working with Hadoop-Streaming version 3.4.0 for a data processing task (work count). I have two Python scripts: mapper.py and reducer.py. I want to ensure that all three nodes in the cluster are ...
0 votes
0 answers
16 views

Partition function Hadoop using java [closed]

I want to build partition class to partition data by mounth and I need to calculate the number of hits of each ip address using hadoop java cloudera I want to make my function with title ...
0 votes
1 answer
17 views

ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 16, column 46> mismatched input ',' expecting LEFT_PAREN

grunt> joined_data = JOIN filtered_features BY (store, date), sales BY (store, date); 2024-04-02 13:19:05,110 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 16, column 46> ...
0 votes
1 answer
37 views

MarkLogic Optic javaScript Geospatial Difference

I want to reduce the selected items by their distance from a point using MarkLogic Optic. I have a table with data and a lat long const geoData = op.fromView("namespace","coordinate"); geoData.where(...
0 votes
2 answers
2k views

Merge duplicates and remove the oldest

I have a collection where there are some duplicate documents. In example: First document: { "_id" : ObjectId("56f3d7cc1de31cb20c08ae6b"), "AddedDate" : ISODate("2016-05-01T00:00:00.000Z"), ...
1 vote
1 answer
14k views

vertext failled Error and Mapper initialized failed - Hive

I'm using Hortonworks data platform in our server with 2 nodes. I'm running query successfully in hive. Suddenly I'm facing mapping with source table to add column to my new table, By this below query;...
0 votes
0 answers
14 views

Hadoop No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster)

I was trying to run the Word Count mapreduce using eclipse, i created a java project and export it to jar then run it with hadoop. This is the code: import java.io.BufferedReader; import java.io....
0 votes
1 answer
29 views

Spark Driver vs MapReduce Driver on YARN

I know in spark you can run the driver program on the client machine if you specify `yarn-client` deployment mode. Or you can run it on a random machine in the cluster if you specify `yarn-cluster` ...
2 votes
1 answer
16k views

Setting the Number of Reducers in a MapReduce job which is in an Oozie Workflow

I have a five node cluster, three nodes of which contain DataNodes and TaskTrackers. I've imported around 10million rows from Oracle via Sqoop and process it via MapReduce in an Oozie workflow. The ...
0 votes
0 answers
89 views

Unable to Preview data loaded to HDFS using WebUI

I am trying to view data on HDFS via the WebUI using port 9870. I am able to view the data from the terminal but get the error code as shown below. I am trying to run Hadoop in a full cluster mode (1 ...
1 vote
1 answer
38 views

Hadoop MapReduce WordPairsCount produces inconsistent results

I have a very confusing results when I run MapReduce on Hadoop. Here is the code (see below). As you can see, it is a very simple MapReduce operation. The input is 1 directory with 100 .lineperdoc ...
0 votes
0 answers
13 views

Hadoop MiniCluster Web UI

Does the hadoop Minicluster supports web UI..?If yes how to access the web UI in Minicluster setup Facing Forbidden error while trying to load web UI.Using Hadoop Version 2.10.2 Maven
0 votes
0 answers
19 views

Java lang runtime exception or jar file does not exist error

I am trying to run simple pagerank labtask on my hadoop 3.3.6 installed on ubuntu virtual box but it is giving this error while all my commands are true and my instructor just tole me to download ...
0 votes
1 answer
92 views

job count test in mit6.824 lab1-mapreduce, ioutil.ReadDir function get duplicate files

my code can pass all the test but not include the job count test in mit6.824 lab1-mapreduce. output is as follow: *** Starting job count test. --- map jobs ran incorrect number of times (10 != 8) --- ...
0 votes
0 answers
17 views

basic python but wierd problem in hadoop-stream text value changes in MapReduce

I am processing a 121983 rows txt file over hadoop. But I met some wierd problem in the mapReduce phase. This is my mapper function: #!/usr/bin/env python import sys import re pattern = r'\b[a-zA-Z0-9]...
0 votes
1 answer
46 views

Hadoop is writing to file using context.write() but output file turns out empty

I am running a hadoop code, and having problems. Notice the the commented lines "debug exception 1" and "debug exception 2" and the line below each of them. Since I can't print ...
0 votes
0 answers
54 views

Error while executing load_summarize_chain with custom prompts

I'm currently haveing troubles with executing load_summarize_chain with custom prompt on the text which is splitted into chunks using chain_type='map_reduce'. The code I wrote: def chunk_data(docs, ...
0 votes
1 answer
13k views

Hive "select to_date(column2) from tablename"

I having the below table structure: Table name: test Table Structure:col1 and col2 are Strings here. col1 col2 "abc" "15/04/2016" "def" "31/05/2016" When I ...
0 votes
0 answers
9 views

Apache Crunch Job On AWS EMR using Oozie

Context: I want to run an apache crunch job on AWS EMR this job is part of a pipeline of oozie java actions and oozie subworkflows (this particular job is part of a subworkflow). In oozie we have a ...
2 votes
1 answer
11 views

Hadoop MapReducee WordCountLength - Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.IntWritable

I was trying to create a MapReduce application to for WordLengthCount as the below code public class WordLengthCount { public static class TokenizerMapper extends Mapper<Object, Text, ...
2 votes
2 answers
2k views

Word percentage program for Hadoop

I'm working on a slightly improved version of famous WordCount program, which should output what percentage of the book does the word take. For example: ... war 0.00002332423% peace 0.0034234324% ... ...
4 votes
4 answers
40k views

hardware requirement for hadoop installation on Laptop

What is the best Laptop hardware recommended for installing Hadoop on windows? I assume I have to install a single node installation, if not please guide me for the correct configuration.
1 vote
2 answers
305 views

Distributed chained computing with Dask on a high failure-rate cluster?

I am using Dask Bag to run some simple map-reduce computation on a special cluster: import dask.bag as bag summed_image = bag.from_sequence(my_ids).map(gen_image_from_ids).reduction(sum, sum).compute(...
2 votes
7 answers
5k views

yarn hadoop 2.4.0: info message: ipc.Client Retrying connect to server

i've searched for two days for a solution. but nothing worked. First, i'm new to the whole hadoop/yarn/hdfs topic and want to configure a small cluster. the message above doesn't show up everytime i ...
0 votes
1 answer
49 views

Error: java.io.IOException: wrong value class: class org.apache.hadoop.io.Text is not class org.apache.hadoop.io.FloatWritable

Im running a Hadoop Mapreduce program to calculate the average, maximum and minimum temperature. Temperature is stored in input1.csv file with three columns Date in YYYY-MM-DD format, temperature in ...
0 votes
0 answers
282 views

Why values become very large after torch.dist.all_reduce?

I'm using pytorch to train a net and found that the loss values become very strange after all_reduce. I've printed the related values as following: >>> print(f'{rank=}, before reduce, {loss=}...
0 votes
1 answer
49 views

I'm having trouble with a map reduce script

I'm creating my first map reduce script. I'm running an item search in the get input stage that outputs: { "recordType": "assemblyitem", "id": "XXXXX", "...
8 votes
9 answers
18k views

Hadoop Streaming - Unable to find file error

I am trying to run a hadoop-streaming python job. bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar -D stream.non.zero.exit.is.failure=true -input /ixml -output /oxml -mapper scripts/...
1 vote
2 answers
9k views

"INFO : Tez session hasn't been created yet. Opening session" hang

I'm working with HDP 2.3.0.0-2557 and I'm trying to run a hive query like the following: beeline> !connect jdbc:hive2://localhost:10000 Connecting to jdbc:hive2://localhost:10000 Enter username ...
0 votes
0 answers
33 views

No Output for MapReduce Program even after successful job completion on Cloudera VM

Programming Environment and Brief Overview: I am working on one of my Big Data Assignments that involved finding the Strike Rate of Gamers using Hadoop Mapreduce 2.6.0 version. I am supposed to work ...
0 votes
0 answers
12 views

Context.write method returns wrong result in Mapreduce java

I'm new to hadoop, and I encountered a weird issue. here is the reducer code : @Override protected void reduce(Text key, Iterable<Deck> values, Context context) throws ...
1 vote
1 answer
118 views

Map-reduce functional outline

Note: this is more a basic programming question and nothing about the Hadoop or Map/Reduce methods of "big data processing". Let's take a sequence (1 2 3 4 5): To map it to some function, ...
0 votes
1 answer
456 views

How to configure memory and vcore on hadoop YARN multinode cluster?

I want to implement hadoop YARN on multinode cluster, So I want to ask. Should I write about memory and vcore configuration in yarn-site.xml and mapred-site.xml only on master(namenode), or should I ...
2 votes
1 answer
9k views

HIVE installation on windows

I am new to Hadoop. I have installed hadoop on windows by following the link(not used cygwin) http://www.codeproject.com/Articles/757934/Apache-Hadoop-for-Windows-Platform. I want to process a csv ...
3 votes
2 answers
9k views

Hadoop mapreduce container exited with a non-zero exit code 1

I'm trying to run some hadoop program to extracting keywords of some abstracts in Ubuntu. When I run my program using Hadoop, I get the following error. WARN util.NativeCodeLoader: Unable to load ...
11 votes
6 answers
53k views

Where is the classpath set for hadoop

Where is the classpath for hadoop set? When I run the below command it gives me the classpath. Where is the classpath set? bin/hadoop classpath I'm using hadoop 2.6.0
2 votes
1 answer
7k views

Does Spark internally use Map-Reduce?

Is Spark using Map Reduce internally ? (its own map reduce) The first time I heard somebody tell me, "Spark uses map-reduce", I was so confused, I always learned that spark was an alterative ...
217 votes
3 answers
179k views

Good MapReduce examples [closed]

I couldn't think of any good examples other than the "how to count words in a long text with MapReduce" task. I found this wasn't the best example to give others an impression of how powerful this ...
0 votes
0 answers
40 views

Optimizing Code for Computing Products from Correlation Matrix

I have a Python code that calculates products based on combinations of keys from a correlation matrix. The code works well for when the dataframe have small numbers of columns (e.g., less than 95 ...
0 votes
0 answers
9 views

How to get shuffle time in Hadoop 3.2.2?

I have some map/reduce code where I believe the shuffle phase in the mapper (which is performed implicitly by Hadoop) will be shorter as compared to a previously existing approach trying to solve the ...
25 votes
8 answers
51k views

How to specify AWS Access Key ID and Secret Access Key as part of a amazon s3n URL

I am passing input and output folders as parameters to mapreduce word count program from webpage. Getting below error: HTTP Status 500 - Request processing failed; nested exception is java.lang....
0 votes
0 answers
11 views

Map reduce to update a column on all rows

I've a table with 500M records. I need to update a column with some business logic. It is not a BULK update, its an individual update. Was thinking to process/update 500M records, can MapReduce jobs ...
2 votes
5 answers
2k views

Map reduce and hash partitioning

While learning about MapReduce, I encountered this question: A given Mapreduce program has the Map phase generate 100 key-value pairs with 10 unique keys. How many Reduce tasks can this ...
0 votes
0 answers
35 views

CouchDB Flatten Data in a View

What am I trying to achieve? I am trying to create a CouchDB view that builds up aggregate data from my workouts document (specifically based on a custom date range), by returning a completed result ...
4 votes
2 answers
6k views

Apache Spark mapPartitionsWithIndex

Can someone give example of correct usage of mapPartitionsWithIndex in Java? I've found a lot of Scala examples, but there is lack of Java ones. Is my understanding correct that separate partitions ...
-1 votes
1 answer
28 views

Hadoop mapreduce code failed with state FAILED due to: NA

I'm trying to run the below Hadoop mapreduce program. public static class MovieFilterMapper extends Mapper<LongWritable, Text, Text, IntWritable> { private Text movieId = new Text(); ...
0 votes
0 answers
14 views

Hadoop Map reduce in Java gives incorrect count for filtered columns

I have the following Map-Reducer in Java which reads data from a text file and filters the records based on 3 conditions. The requirement is to return the number of records in the text file which ...
0 votes
1 answer
62 views

YARN job fails due to the connection issue

I've hadoop-3.3.6 setup in the Kubernetes cluster, all the hadoop components are exposed via ClusterIP services, I'm able to telnet to the ports that are exposed from respective pods. But when I run ...

1
2 3 4 5
244