Questions tagged [mapreduce]
MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes
12,149
questions
0
votes
1
answer
13
views
Miss join da in left join when data quantity increase in Hive
In this hive sql, when the quantity of data in table1 is big, t2.c will lost but it should be joined, how to exlpain this in the level of mapreduce?
SELECT
t1.a,
t1.b,
t2.c
FROM
table1 t1
LEFT ...
0
votes
0
answers
15
views
Create a hive table by processing TextOutputFormat
I am using Hadoop Mapreduce to pass data from MongoDB to HDFS in TextOutputFormat format.
I want to process data received in TextOutputFormat format and create it in hive table format, but it doesn't ...
0
votes
1
answer
16
views
Hadoop Error Code 127 - not able to figure out what the actual cause of the error is
Was doing an assignment for class. Running the mapper and reducer codes in my local system went fine and got the desired output. I have a feeling that there's something wrong with Hadoop.
Here's the ...
0
votes
0
answers
12
views
How to monitor CPU, memory, and disk utilization during query execution in pig and hive?
Same as the title. I am running a hadoop standalone cluster on my laptop. When I run a query, a map reduce job is created which can be seen on the web ui at localhost:8088. However, it doesn't give ...
2
votes
1
answer
52
views
Configure hadoop.service.shutdown.timeout property
I need to configure the value of hadoop.service.shutdown.timeout due to the shutdown hooks triggering a timeout when our MR jobs stop:
2023-08-25 08:44:39,566 [WARN] [Thread-0] [org.apache.hadoop.util....
0
votes
0
answers
26
views
Hadoop Map Reduce History Server UI - unable to find job logs (but log exists at hdfs)
In the Hadoop JobHistory WEB UI, I see all applications, running, killed, finished(succeeded), etc. However, I can not access some of the MapReduce jobs tracking URL history logs.
When I run hadoop fs ...
0
votes
1
answer
44
views
SuiteScript - Map/Reduce - no context value is passed to the summarize stage from reduce stage
I am new to this and currently learning SuiteScript. I'm having trouble understanding why there is no context value being passed from the reduce function to the summarize function. The log results ...
0
votes
0
answers
239
views
Optimize a "map reduce_documents chain" in LangChain by storing intermediate results
I have a long document and want to apply different map reduce document chains from LangChain to it.
The 'map template' is always identical and can be generated in advance and cached.
The various '...
0
votes
0
answers
54
views
MapReduce is working normally and suddenly restarts the process without aborting the Job (Sqoop)
MapReduce is working normally and, suddenly, it restarts the process without aborting the Job (Sqoop), then at the end it duplicates the information in the table?
0
votes
0
answers
15
views
PIG MAP_ONLY Job failing with Datanode Going OOM Error
I Have application Audit stored in HDFS for day 09 to 16. All this data have total 23 Fields. Two of them is Event_ID and Status. Event_ID can be 1 to 10. And Status can be REQUEST, SUCCESS, FAILURE.
[...
0
votes
1
answer
51
views
python map iterate over a list and function on another list
How in Python is the map function used to iterate over a list (iterable) and do the logic on another list(iterable). For example, we have a list of indices as indices and a list of strings as str_list....
0
votes
1
answer
14
views
MapReduce algorithm for simple internet graph example
Let G be the internet graph. The input to the first MapReduce round are tuples of the
form (siteA,siteB) if there is a link on siteA to siteB. How can I print all pairs of webpages that have a common ...
1
vote
1
answer
19
views
Grouping Array by another array 'createdAt' values
i have 2 arrays from my Api
this
passArray:[
{pass_id: 645, heure_debut: '13:48:00', heure_fin: '14:48:00', createdAt:"2023-08-03T13:48:00.000Z"}
{pass_id: 646, heure_debut: '14:40:00', ...
0
votes
0
answers
16
views
Missing configuration keys in MapReduce RunningJob interface
I have a requirement where I programmatically need to access some configuration values I have have previously set on a MapReduce Job. The way that I set the configuration object
final Configuration ...
2
votes
2
answers
138
views
Why do we have std::transform_reduce in C++ standard library?
We have std::transform and std::reduce functions in C++ standard library . What advantages do we have (except shortening the code) if we use std::transform_reduce instead of a combination of std::...
0
votes
0
answers
17
views
Mapreduce job not working for fetching data
I have a map reduce job that I'd like to use to fetch some data from the scholarly package. Here's the code I've edited from the documentation:
from multiprocessing import Pool
from scholarly import ...
0
votes
1
answer
24
views
MapReduce Netsuite Error While Running SuiteScript
Running into an issue that is driving me nuts does anyone have any idea what is wrong. Trying to do a map reduce of current S/O's in the NetSuite system calling a saved search.
Error
Fail to evaluate ...
0
votes
0
answers
35
views
Why is my sucessful run mapreduce job not showing up in the resource manager web interface(0.0.0.0:8088) as an entry?
Hello I have finished my hadoop cluster installation/configuration. I have run a couple mapreduce tests which are successfully giving back results. However when I try to keep track of them on the ...
0
votes
0
answers
28
views
Hive CRUD, where clause and aggregation queries not working
I am running hive after setting up hive and hadoop on my system. I can create the tables and load the data in it via the hdfs or via local inpath query. But, I am facing an issue when running queries ...
0
votes
0
answers
19
views
iam new to hadoop mapreduce and face a trouble when submit my mapreduce job from windows to the cluster running on linux
i am setting up a psudo hadoop mapreduce cluster and hbase, and i want to submit a hbase mapreduce job to the cluster, now the cluster is working well and dependency libs is transmited to the ...
0
votes
0
answers
18
views
Unable to run hadoop jar - incorrect number of arguments
I am new to Hadoop and followed a WordCount MapReduce example that I found online to run in Ubuntu. I managed to complete all but the final step - i.e. running the job. I uploaded the input file to ...
0
votes
1
answer
61
views
Running a hadoop streaming and mapreduce job: PipeMapRed.waitOutputThreads() : subprocess failed with code 1
i am using hadoop 3.3.4 and trying to execute a mapreduce program in python that use google page rank algorithm to rank pages.
I'm trying to run this on my own Hadoop cluster. I ran the job using the ...
0
votes
0
answers
14
views
Invoice Group created without Invoices
I am trying to create a group invoice using a map reduce script, based on Subscription IDs.When I execute the script, a group invoice is created, but no invoices are linked to it. There are no errors ...
0
votes
0
answers
29
views
Message is org.mozilla.javascript.WrappedException: Wrapped java.lang.ClassCastException
I am receiving the below error when executing my script. The script has been running for about a week now with no issues. After receiving the error, I tried testing in our Sandbox environment and did ...
1
vote
0
answers
17
views
Reduce Is Not Running In MapReduce Hadoop
I try to use MapReduce in Hadoop with Windows 11 and Python. My mapper function complete successfully, but the reducer function does not seem to be running or making progress. I got Map 100% and ...
0
votes
1
answer
20
views
How to get unique and returning users count from overall user enteries with a user key in couchDB
I have a key called 'username' that stores the user logic records with a unique user id in couchDB. Now I want to get unique users count and returning users count from those records with the help of ...
0
votes
0
answers
23
views
What is the purpose of a reduce task when inserting data into a partitioned Hive table?
I have a non partitioned table named stocks_A which only has stocks that start with character A (of size 388 MB on HDFS as 20 small files)
I am inserting all these from stocks_A into a partitioned ...
0
votes
1
answer
13
views
Update Function To handle Multiple Documents In CouchDB
I am trying to write a save function which creates multiple documents in couchdb. I am using update function to do that. In my update function I want to get a specific value (highest number) from a ...
0
votes
0
answers
49
views
Hadoop Streaming Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
I am very new to Hadoop and trying to run a MapReduce job on my University's cluster. I have tested my mapper and reducer locally and they seem to work ok, but when using streaming I get this error ...
0
votes
1
answer
46
views
Hadoop unable to see file though it exists: "Error: java.io.FileNotFoundException: File does not exist: /user/hadoop/centroids.seq"
I wrote a hadoop program and when I try to execute it (I use this command: hadoop jar kmeans-1.0-SNAPSHOT.jar it.kurapika.Kmeans dataset.txt output) I get the following error:
Error: java.io....
0
votes
0
answers
24
views
Unable to Preview data loaded to HDFS using WebUI
I am trying to view data on HDFS via the WebUI using port 9870. I am able to view the data from the terminal but get the error code as shown below. I am trying to run Hadoop in a full cluster mode (1 ...
-1
votes
1
answer
123
views
MapReduce word count using C++ transform_reduce() function with a parallel execution policy
I have a string of words and I would like to count the number of occurrences of each word, and store the results in a map. I want to use std::transform_reduce() in order to take advantage of it's ...
0
votes
1
answer
26
views
Specify N in hadoop streaming when use NLineInputFormat
If I use NLineInputFormat in hadoop streaming, how to specify N?
hadoop jar /home/Software/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar \
-D stream.non.zero.exit.is.failure=false \
-D ...
0
votes
1
answer
56
views
How to move data block from datanode to other datanode during mapreduce?
I implemented 4 node cluster for running hadoop following the site(https://www.linode.com/docs/guides/how-to-install-and-set-up-hadoop-cluster/)
By the way, I want to move data block to other datanode ...
0
votes
1
answer
21
views
Is it possible to use Hadoop's MapReduce to write key-value pairs to different files based on predetermined key
I want to separate the key/value pairs resulting from this hadoop job into different files according to the predetermined key values in different files.
For example, in the Startup (X) file there are ...
1
vote
1
answer
50
views
Hive INSERT OVERWRITE creates 1 large file in every partition
I am executing a hive INSERT OVERWRITE query.
SET hive.merge.mapredfiles=false;
SET mapred.reduce.tasks=${hiveconf:NUM_REDUCERS};
SET hive.mapred.supports.subdirectories=TRUE;
SET mapred.input.dir....
2
votes
0
answers
58
views
Performing 3 Way Join MapReduce Python
I have the following tables stored as separate csv files:
customers (c_id, gender, address, dob)
meals (r_id, c_id, date) (so a customer having a meal at a restaurant)
restaurants (type, r_id)
...
0
votes
1
answer
55
views
pool.map() not working on more than 2 CPUs
I have the following piece of code:
import sentence_transformers
import multiprocessing
from tqdm import tqdm
from multiprocessing import Pool
import numpy as np
embedding_model = ...
0
votes
1
answer
66
views
How do I reduce a list of tuples into one output?
I just completed a mapper and groupby script. Running it in my terminal gives me a list of tuples:
a [('1', '0', '0'), ('1', '1', '1')]
b [('1', '0', '0')]
e [('1', '0', '0'), ('1', '0', '1')]
f [('1',...
0
votes
1
answer
28
views
How do I use groupby on the output of a mapper?
This is a continuation of my previous question:
How to print only if a character is an alphabet?
I now have a mapper that is working perfectly, and it's giving me this output when I use a text file ...
0
votes
1
answer
112
views
pool.map() doesn't like list of lists
I have the following piece of code:
import sentence_transformers
import multiprocessing
from tqdm import tqdm
from multiprocessing import Pool
embedding_model = sentence_transformers....
0
votes
1
answer
35
views
Mapreduce not running on 8 CPUs but works on 2 CPUs (Amazon Sagemaker)
I have the following piece of code that I'm trying to run on multiple machines using Amazon Sagemaker. I am printing a few things for debugging in this code. When I run this code on an instance with 2 ...
0
votes
1
answer
58
views
Hadoop MapReduce job does not produce any logs and stats in local mode
I am writing very basic map reduce job to count words in a text file in local mode. Though job is running fine and gives output but it does not produce any map reduce logs or statistics.
I have ...
0
votes
1
answer
332
views
IndexError: pop from an empty deque while using map in Python
I have the following piece of code in which I am trying to calculate sentence embeddings in parallel.
import multiprocessing
from tqdm import tqdm
# Define the function to be executed in parallel
def ...
1
vote
1
answer
49
views
Exception in thread "main" java.lang.ClassNotFoundException in Java MapReduce Application Hadoop
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.StringTokenizer;
// bro lets see this new package works or not.
import org.apache.hadoop.conf....
0
votes
0
answers
50
views
Using MapReduce for Federated Learning
I would like to build a federated learning system with PyTorch. I've currently worked out and written the code for the following:
Finetuning models on client devices
Sending back the finetuned models ...
0
votes
2
answers
90
views
How to find top K when K is too large for memory
Let's say there is a large file with N elements and I want to find the top K largest elements from that file and output to a new file.
If N is large but K is small, I could use a min-heap of size K ...
0
votes
0
answers
85
views
User event script not getting trigged which is deployed on a entity record
I have created a user event script which is deployed on the vendor record.
There is another map reduce script which is creating the vendor record.
The user event script is working fine from UI, but ...
0
votes
5
answers
99
views
How to return 1 only if the last letter of a word is a vowel? Return 0 otherwise
This is a follow up to this question I posted earlier:
How to return 1 if the first letter of a word is a vowel, return 0 otherwise. Mapper (MapReduce) problem
I am having trouble returning 1 only if ...
2
votes
2
answers
85
views
How to return 1 if the first letter of a word is a vowel, return 0 otherwise. Mapper (MapReduce) problem
This is the first part of a MapReduce problem I am working on. I need a function that yields 1 if the first letter of a word starts with a vowel, otherwise it should yield 0.
The program is run in the ...