Questions tagged [mapreduce]
MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes
12,048
questions
0
votes
0
answers
5
views
what is mapped and reduced in computeSVD function?
im new to pysaprk and im using a function from pyspark named computeSVD and i found an answer that this function uses mapreduce in this discussion .
Since it uses mapreduce my question is what is ...
2
votes
1
answer
27
views
MapReduce using DataFlow library
I am trying to implement a classic map-reduce problem using System.Threading.Tasks.Dataflow, and although I can get something (sort of) working I'm struggling to see how to generalise this ...
1
vote
1
answer
14
views
Count bottom-level elements in nested object of objects of arrays
I have an "object of objects of arrays" and I simply need to get the total count of elements in the "bottom level" arrays ...so with the example below, I need to return 19 (...
0
votes
1
answer
19
views
Counting the frequency of a list of pairs in an RDD with pyspark
I am trying to calculate the frequency of some pairs (stored in a list) in an RDD where each element is a list of strings. Teh RDD has this form:
[['protectuаsky', 'stoprussia', 'ukraineunderattaсk'],
...
-4
votes
0
answers
21
views
I am a beginner with Python but never used MRJob and i can't seem to work a solution through documentation, please help as i am badly stuck :( [closed]
You are given the comments.csv input file. It is a Comma Separated Values (CSV) file that stores comment relationships. More specifically, the file includes 1 million rows and each row stores a tuple ...
0
votes
1
answer
30
views
Map Reduce to read a text file using Python
I'm trying to write a MapReduce program that can read an input file.But I don't really know how to use it in a MapReduce program in python.
Can someone give me a code snippet of it?
I have tried the ...
0
votes
1
answer
17
views
Hadoop: java.lang.ClassNotFoundException:map-reduce
Hi I'm using eclipse to export the jar file of map reduce program.When i try to run the jar file using the command
hadoop jar WordCountdemo.jar /Demo2/WordCount.txt /mroutput7utput7
I m getting error ...
0
votes
1
answer
28
views
Why spark is 100 times faster than Hadoop Map Reduce
Why spark is faster than Hadoop MapReduce?.
As per my understanding if spark is faster due to in-memory processing then Hadoop is also load data into RAM then it process. Every program first load into ...
-1
votes
0
answers
19
views
MapReduce Job Failed on MultiNode
I'm new to Hadoop. I have to use 'MapReduce' with WordCount. I am getting some errors. I am running a 50Gb 'MapReduce' job on a single server (8GB, 8 core). It ends successfully. I'm setting up multi ...
0
votes
0
answers
24
views
Hadoop mapreduce execution stuck
I'm using HADOOP on a VM. When I try to run a jar, the execution stops because is unable to find the file resource-type.xml.
How can I solve this? Thank you.
gaia@gaia-virtual-machine:~/hadoop-3.3.2$ ...
0
votes
0
answers
8
views
Which task counters can I use to determine the proportion of time a reduce task spends on fixed startup/teardown time?
I am trying to tune the number of reducers for a job that runs on a large cluster and I'm assuming that some proportion of the time spent by each reducer task is spent on startup/teardown of the task ...
1
vote
4
answers
38
views
How do i format this object using reduce
const data = [
{
product_id: 1245,
product_name: "XYZ",
price: "10.00",
batch_number: 1,
},
{
product_id: 12456,
product_name: "RST",
...
0
votes
0
answers
20
views
Python - Mrjob inlink count / MapReduce
I would like to count the number of times that a word appears in a line of my doc with MrJob and Mapreduce.
I just succeed to count the number of time a song appear in my doc but I don't know how to ...
0
votes
0
answers
12
views
Singular value decomposition (SVD) to tfidf dataframe using mapReduce
Here is my dataframe, i used np.linalg.svd but it doesnt use MapReduce so i want a similar function to np.linalg.svd that uses mapreduce
#there is 5 documents.
print(tfidf)
a portion of the output:
...
0
votes
0
answers
32
views
MapReduce on several columns
I am new to MapReduce and have troubles with writing a MapReduce job which would work with several columns at once for a CSV file.
I would like to find for each garment group the most frequent product,...
1
vote
1
answer
32
views
how to compare hadoop result
I am writing a map reduce program to find the file that contains the most words.
Now, I am able to use map reduce to find the number of words contained in each file. However, I am unsure how I can ...
0
votes
1
answer
31
views
Spark Map-reduce slower than naive approche
def naive_word_counting(wordsList):
words = {}
for word in wordsList:
if word in words:
words[word] += 1
else:
words[word] = 1
return sorted(...
-1
votes
0
answers
68
views
design a MapReduce Algorithm (Pseudo-codes or Java Codes)
Suppose that you have a large student file which cannot be stored in a single machine.
Each record of this file contains information: (Student_ID, Student_Name, Sex, Age,
Module, Grade, Department).
...
0
votes
0
answers
44
views
How to implement the division of two relations in mapreduce?
I want to implement the division of two relations in MapReduce. I have two relations: T(A,B,C) and U(B,C). I know that
for the relations R(A,B,D) and S(A,B). This is pretty much my scenario. I am not ...
1
vote
0
answers
116
views
MapReduce job that calculates chi-square values in Python
I am writing a MapReduce job in Python using MrJob. The sample of my dataset which is in JSON:
{"reviewerID": "A2VNYWOPJ13AFP", "asin": "0981850006", "...
-1
votes
1
answer
37
views
How to use map() and reduce() functions in Javascript?
I am writting a simple function that would allow me to calculate prime numbers on one of nosql databases. After multiple approaches, i got an error that value i am looking for is not defined, thus I ...
1
vote
1
answer
44
views
MapReduce Job: How do I take in <Text, IntWritable> during Map phase and output <Text, Text> in Reduce phase?
I am trying to make my output look like the following:
Model output
But I am stuck with this:
My output
How do I convert the value (IntWritable) from the output to Text and concatenate the string &...
0
votes
0
answers
117
views
Design a MapReduce Algorithm & Pseudocode
MapReduce Programming
Suppose that you have a large student file which cannot be stored in a single machine.
Each record of this file contains information: (Student ID, Student Name, Sex, Age,
Module, ...
0
votes
0
answers
38
views
MapReduce pseudocode explanation
I saw this pseudocode will studying how to design a map-reduce algorithm. The pseudocode is meant to compute the mean of values associated with the same key. I need help in understanding how the ...
0
votes
0
answers
29
views
Python MapReduce MRJob TypeError: write() argument must be str
I am facing an issue with the MRJob library in Python and hope you can help me out in how to solve it.
I want to apply a simple MapReduce job and count the number of characters, words and lines of a ...
0
votes
0
answers
30
views
Hadoop mapreduce with dictionary
I am trying to add another function to the code provided by hadoop apache https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Partitioner. ...
0
votes
0
answers
17
views
hadoop mapreduce folder created but files are not
I run the WordCount jar file with docker on Hadoop
Everything is ok but the output folder. The output folder is empty.
When I run
$HADOOP_HOME/bin/hadoop jar $JAR_FILEPATH $CLASS_TO_RUN $INPUT $OUTPUT
...
0
votes
1
answer
69
views
calculate median of a list of values parallely using Hadoop map-reduce
I'm new to Hadoop mrjob. I have a text file which consists of data "id groupId value" in each line. I am trying to calculate a median of all values in the text file using Hadoop map-reduce. ...
0
votes
0
answers
15
views
MapReduce retain input order from mapper to reducer java
I have a weather dataset. I process it and find the average temperature value on a certain day, but when I try to go beyond the month, the data comes into the reducer in a different order, and I need ...
0
votes
0
answers
14
views
Null pointer error in Map Reduce Motif search code
I am trying to run a motif search map reduce job and I am getting this error.
I am going to post my code below, I am sure that there are logical errors to debug, but I am really just trying to get it ...
0
votes
0
answers
27
views
"Pure" Map-reduce shuffle in pyspark df
assuming i have some pyspark df, f.e:
Key | Value
0 | "a"
2 | "c"
0 | "b"
1 | "z"
I want to perform map-reduce-like shuffle method -
i.e. I want to ...
0
votes
1
answer
27
views
Map Reduce Intermediate Output
I was referring to the map-reduce blog and it emphasised on writing intermediate result to disk. Here my question is why do map-reduce write intermediate output to disk can't it just use memory(RAM)? ...
0
votes
1
answer
33
views
How to solve the "too many connection" problem in zookeeper when I want to query too many times in reduce stage?
Sorry for my stupid question and thank you in advance.
I need to replace the outputvalue in reduce stage(or map stage). However, it will case too many connection in zookeeper. I don't know how to deal ...
0
votes
0
answers
23
views
How to use a Generic Hadoop Cluster to make a Word Counter in AWS Elastic MapReduce EMR?
I am trying to use AWS EMR to make a word counter.
Currently what I have is WordCount.java code that will take my input text and do a map reduce on AWS EMR. I want to know if it is possible for the ...
0
votes
0
answers
19
views
How to identify different user_ids of the same person in hive?
I have the following attributes in one table in hive:
user_id_a,user_id_b
Each row of data identifies a different user_id of the same person.
Like :
user_id_a,user_id_b
3242,5897
5897,6752
3242,...
0
votes
0
answers
30
views
Facing issue while running MR Job on Yarn
I am trying run a MR job on Yarn. While running the MR job , the job gets stuck while cleaning up the staging dir. created by the process. below is the log snippet
2022-03-24 00:12:48 IST INFO [...
0
votes
0
answers
9
views
Difference purpose between MapReduce and Spark in data processing
I'm new in Big Data. Recently, I read about MapReduce and Apache Spark and there are several things that I'm confused about. From what I understand MapReduce has only two functions which are Map and ...
0
votes
1
answer
19
views
Hadoop performing matrix vector multiplication several times
I'm trying to implement Eigenvector centrality in Hadoop, where we have an iteration as follows (r is a vector and A is a matrix):
r_i+1 = A*r_i
r_i+1 = r_i+1/||r_i+1||
Where || denotes the norm of ...
0
votes
0
answers
45
views
Java compiler cannot find hadoop.hbase.mapreduce packages
I'm trying to run a MapReduce app which uses a HBase table. The code is provided by my university professor and we are supposed to run it. Here is the code:
package hbase;
import java.io.IOException;
...
0
votes
0
answers
15
views
MapReduce with Text File - not sure how to evaluate
I have two text files, one with text and the other as a dictionary, they are referring to the same document ID, however So what I want to do is somehow make a MapReduce script that is able to create ...
0
votes
0
answers
22
views
Question about reduceByKeyandWindow Window Length and SlideInterval
I have this simple code that from my understanding, the Window Length must consider the DSSTREAMs collected and SlideInterval perform the movement through the windows. It happens that when placing the ...
0
votes
0
answers
9
views
Link for Hbase installtion in colab
I have installed and run hadoop in colab and execute python map reduce code.
I would like to install hadoop and work in colab. Can someone provide me the steps for installation
1
vote
1
answer
75
views
Tackling the 'Small Data' Problem with Distributed Computing Cluster?
I'm learning about Hadoop + MapReduce and Big Data and from my understanding it seems that the Hadoop ecosystem was mainly designed to analyze large amounts of data that's distributed on many servers. ...
0
votes
0
answers
18
views
Hadoop Mapreduce job map 100% reduce 0% - rduction is not running and node manager is shutting down
I am trying to run the hadoop wordcount example after installation:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input /output
I tried to run with default memory ...
0
votes
1
answer
69
views
Finding Top Ten Word Syllable Count
I am trying to make a job that takes in a text file, then counts the number of syllables in each word, then ultimately returns the top 10 words with the most syllables. I'm able to get all of the word/...
1
vote
1
answer
19
views
How to map functions across objects/arrays in SQL?
Say I have some data in table, which has the names and ages of customers who came into a store each day (the id functions as a unique day identifier).
id, obj
------------------------------------------...
0
votes
0
answers
26
views
exit code 141 in Hadoop streaming
Come across exit code 141 in Hadoop streaming jobs.
After search in Google, I found the 141 error is about pipe broke...
My job is to generate protobuf instance from json.
When I look carefully at the ...
0
votes
0
answers
20
views
Efficient way of overwriting property in grouped object
I have a list of objects with certain properties, say A, B and C.
The properties could be duplicated plenty of times, with C being possibly non-existent. In the end, I would like to end up with an ...
0
votes
1
answer
163
views
Mapreduce not running. /bin/bash: /bin/java: No such file or directory
I am getting the following error.
[2022-03-01 20:03:49.581]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/...
0
votes
2
answers
114
views
map reduce count and sum of array of objects
i have an array of objects which has properties with boolean and some properties with integer values. For eg
const arr = [{
_id: "621bb15de2ecadf024da51d3",
draw: false,
pixel: ...