Questions tagged [mapreduce]

MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes

Filter by
Sorted by
Tagged with
0 votes
1 answer
13 views

Miss join da in left join when data quantity increase in Hive

In this hive sql, when the quantity of data in table1 is big, t2.c will lost but it should be joined, how to exlpain this in the level of mapreduce? SELECT t1.a, t1.b, t2.c FROM table1 t1 LEFT ...
DaSH Tai's user avatar
0 votes
0 answers
15 views

Create a hive table by processing TextOutputFormat

I am using Hadoop Mapreduce to pass data from MongoDB to HDFS in TextOutputFormat format. I want to process data received in TextOutputFormat format and create it in hive table format, but it doesn't ...
우희라's user avatar
0 votes
1 answer
16 views

Hadoop Error Code 127 - not able to figure out what the actual cause of the error is

Was doing an assignment for class. Running the mapper and reducer codes in my local system went fine and got the desired output. I have a feeling that there's something wrong with Hadoop. Here's the ...
Ganu Matta's user avatar
0 votes
0 answers
12 views

How to monitor CPU, memory, and disk utilization during query execution in pig and hive?

Same as the title. I am running a hadoop standalone cluster on my laptop. When I run a query, a map reduce job is created which can be seen on the web ui at localhost:8088. However, it doesn't give ...
catGPT's user avatar
  • 427
2 votes
1 answer
52 views

Configure hadoop.service.shutdown.timeout property

I need to configure the value of hadoop.service.shutdown.timeout due to the shutdown hooks triggering a timeout when our MR jobs stop: 2023-08-25 08:44:39,566 [WARN] [Thread-0] [org.apache.hadoop.util....
Evelina Dumitrescu's user avatar
0 votes
0 answers
26 views

Hadoop Map Reduce History Server UI - unable to find job logs (but log exists at hdfs)

In the Hadoop JobHistory WEB UI, I see all applications, running, killed, finished(succeeded), etc. However, I can not access some of the MapReduce jobs tracking URL history logs. When I run hadoop fs ...
Canerc's user avatar
  • 3
0 votes
1 answer
44 views

SuiteScript - Map/Reduce - no context value is passed to the summarize stage from reduce stage

I am new to this and currently learning SuiteScript. I'm having trouble understanding why there is no context value being passed from the reduce function to the summarize function. The log results ...
Curious Explorer's user avatar
0 votes
0 answers
239 views

Optimize a "map reduce_documents chain" in LangChain by storing intermediate results

I have a long document and want to apply different map reduce document chains from LangChain to it. The 'map template' is always identical and can be generated in advance and cached. The various '...
Frank Rettig's user avatar
0 votes
0 answers
54 views

MapReduce is working normally and suddenly restarts the process without aborting the Job (Sqoop)

MapReduce is working normally and, suddenly, it restarts the process without aborting the Job (Sqoop), then at the end it duplicates the information in the table?
Edvaldo Lucena's user avatar
0 votes
0 answers
15 views

PIG MAP_ONLY Job failing with Datanode Going OOM Error

I Have application Audit stored in HDFS for day 09 to 16. All this data have total 23 Fields. Two of them is Event_ID and Status. Event_ID can be 1 to 10. And Status can be REQUEST, SUCCESS, FAILURE. [...
Uddhav Savani's user avatar
0 votes
1 answer
51 views

python map iterate over a list and function on another list

How in Python is the map function used to iterate over a list (iterable) and do the logic on another list(iterable). For example, we have a list of indices as indices and a list of strings as str_list....
Alan's user avatar
  • 431
0 votes
1 answer
14 views

MapReduce algorithm for simple internet graph example

Let G be the internet graph. The input to the first MapReduce round are tuples of the form (siteA,siteB) if there is a link on siteA to siteB. How can I print all pairs of webpages that have a common ...
bipur24's user avatar
1 vote
1 answer
19 views

Grouping Array by another array 'createdAt' values

i have 2 arrays from my Api this passArray:[ {pass_id: 645, heure_debut: '13:48:00', heure_fin: '14:48:00', createdAt:"2023-08-03T13:48:00.000Z"} {pass_id: 646, heure_debut: '14:40:00', ...
ivory uchiwa's user avatar
0 votes
0 answers
16 views

Missing configuration keys in MapReduce RunningJob interface

I have a requirement where I programmatically need to access some configuration values I have have previously set on a MapReduce Job. The way that I set the configuration object final Configuration ...
Paul Plato's user avatar
  • 1,481
2 votes
2 answers
138 views

Why do we have std::transform_reduce in C++ standard library?

We have std::transform and std::reduce functions in C++ standard library . What advantages do we have (except shortening the code) if we use std::transform_reduce instead of a combination of std::...
akulinich's user avatar
0 votes
0 answers
17 views

Mapreduce job not working for fetching data

I have a map reduce job that I'd like to use to fetch some data from the scholarly package. Here's the code I've edited from the documentation: from multiprocessing import Pool from scholarly import ...
Patthebug's user avatar
  • 4,647
0 votes
1 answer
24 views

MapReduce Netsuite Error While Running SuiteScript

Running into an issue that is driving me nuts does anyone have any idea what is wrong. Trying to do a map reduce of current S/O's in the NetSuite system calling a saved search. Error Fail to evaluate ...
Kayle Davis's user avatar
0 votes
0 answers
35 views

Why is my sucessful run mapreduce job not showing up in the resource manager web interface(0.0.0.0:8088) as an entry?

Hello I have finished my hadoop cluster installation/configuration. I have run a couple mapreduce tests which are successfully giving back results. However when I try to keep track of them on the ...
Bruno Marques Veiga's user avatar
0 votes
0 answers
28 views

Hive CRUD, where clause and aggregation queries not working

I am running hive after setting up hive and hadoop on my system. I can create the tables and load the data in it via the hdfs or via local inpath query. But, I am facing an issue when running queries ...
RahulNans's user avatar
0 votes
0 answers
19 views

iam new to hadoop mapreduce and face a trouble when submit my mapreduce job from windows to the cluster running on linux

i am setting up a psudo hadoop mapreduce cluster and hbase, and i want to submit a hbase mapreduce job to the cluster, now the cluster is working well and dependency libs is transmited to the ...
PnEndless's user avatar
0 votes
0 answers
18 views

Unable to run hadoop jar - incorrect number of arguments

I am new to Hadoop and followed a WordCount MapReduce example that I found online to run in Ubuntu. I managed to complete all but the final step - i.e. running the job. I uploaded the input file to ...
Wasim_Khan's user avatar
0 votes
1 answer
61 views

Running a hadoop streaming and mapreduce job: PipeMapRed.waitOutputThreads() : subprocess failed with code 1

i am using hadoop 3.3.4 and trying to execute a mapreduce program in python that use google page rank algorithm to rank pages. I'm trying to run this on my own Hadoop cluster. I ran the job using the ...
Tahar Jaafer's user avatar
0 votes
0 answers
14 views

Invoice Group created without Invoices

I am trying to create a group invoice using a map reduce script, based on Subscription IDs.When I execute the script, a group invoice is created, but no invoices are linked to it. There are no errors ...
Anu Ashish's user avatar
0 votes
0 answers
29 views

Message is org.mozilla.javascript.WrappedException: Wrapped java.lang.ClassCastException

I am receiving the below error when executing my script. The script has been running for about a week now with no issues. After receiving the error, I tried testing in our Sandbox environment and did ...
stl4baseball's user avatar
1 vote
0 answers
17 views

Reduce Is Not Running In MapReduce Hadoop

I try to use MapReduce in Hadoop with Windows 11 and Python. My mapper function complete successfully, but the reducer function does not seem to be running or making progress. I got Map 100% and ...
Aurell Layalia's user avatar
0 votes
1 answer
20 views

How to get unique and returning users count from overall user enteries with a user key in couchDB

I have a key called 'username' that stores the user logic records with a unique user id in couchDB. Now I want to get unique users count and returning users count from those records with the help of ...
Arul's user avatar
  • 1
0 votes
0 answers
23 views

What is the purpose of a reduce task when inserting data into a partitioned Hive table?

I have a non partitioned table named stocks_A which only has stocks that start with character A (of size 388 MB on HDFS as 20 small files) I am inserting all these from stocks_A into a partitioned ...
learner's user avatar
  • 21
0 votes
1 answer
13 views

Update Function To handle Multiple Documents In CouchDB

I am trying to write a save function which creates multiple documents in couchdb. I am using update function to do that. In my update function I want to get a specific value (highest number) from a ...
Nizal Sha's user avatar
0 votes
0 answers
49 views

Hadoop Streaming Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2

I am very new to Hadoop and trying to run a MapReduce job on my University's cluster. I have tested my mapper and reducer locally and they seem to work ok, but when using streaming I get this error ...
Alex's user avatar
  • 37
0 votes
1 answer
46 views

Hadoop unable to see file though it exists: "Error: java.io.FileNotFoundException: File does not exist: /user/hadoop/centroids.seq"

I wrote a hadoop program and when I try to execute it (I use this command: hadoop jar kmeans-1.0-SNAPSHOT.jar it.kurapika.Kmeans dataset.txt output) I get the following error: Error: java.io....
Sissi's user avatar
  • 1
0 votes
0 answers
24 views

Unable to Preview data loaded to HDFS using WebUI

I am trying to view data on HDFS via the WebUI using port 9870. I am able to view the data from the terminal but get the error code as shown below. I am trying to run Hadoop in a full cluster mode (1 ...
nkah's user avatar
  • 13
-1 votes
1 answer
123 views

MapReduce word count using C++ transform_reduce() function with a parallel execution policy

I have a string of words and I would like to count the number of occurrences of each word, and store the results in a map. I want to use std::transform_reduce() in order to take advantage of it's ...
Will's user avatar
  • 1,509
0 votes
1 answer
26 views

Specify N in hadoop streaming when use NLineInputFormat

If I use NLineInputFormat in hadoop streaming, how to specify N? hadoop jar /home/Software/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar \ -D stream.non.zero.exit.is.failure=false \ -D ...
Perimeter Chou's user avatar
0 votes
1 answer
56 views

How to move data block from datanode to other datanode during mapreduce?

I implemented 4 node cluster for running hadoop following the site(https://www.linode.com/docs/guides/how-to-install-and-set-up-hadoop-cluster/) By the way, I want to move data block to other datanode ...
arthur seokwon choi's user avatar
0 votes
1 answer
21 views

Is it possible to use Hadoop's MapReduce to write key-value pairs to different files based on predetermined key

I want to separate the key/value pairs resulting from this hadoop job into different files according to the predetermined key values in different files. For example, in the Startup (X) file there are ...
coder_98_mt's user avatar
1 vote
1 answer
50 views

Hive INSERT OVERWRITE creates 1 large file in every partition

I am executing a hive INSERT OVERWRITE query. SET hive.merge.mapredfiles=false; SET mapred.reduce.tasks=${hiveconf:NUM_REDUCERS}; SET hive.mapred.supports.subdirectories=TRUE; SET mapred.input.dir....
Ishan Tiwary's user avatar
2 votes
0 answers
58 views

Performing 3 Way Join MapReduce Python

I have the following tables stored as separate csv files: customers (c_id, gender, address, dob) meals (r_id, c_id, date) (so a customer having a meal at a restaurant) restaurants (type, r_id) ...
Dan G's user avatar
  • 95
0 votes
1 answer
55 views

pool.map() not working on more than 2 CPUs

I have the following piece of code: import sentence_transformers import multiprocessing from tqdm import tqdm from multiprocessing import Pool import numpy as np embedding_model = ...
Patthebug's user avatar
  • 4,647
0 votes
1 answer
66 views

How do I reduce a list of tuples into one output?

I just completed a mapper and groupby script. Running it in my terminal gives me a list of tuples: a [('1', '0', '0'), ('1', '1', '1')] b [('1', '0', '0')] e [('1', '0', '0'), ('1', '0', '1')] f [('1',...
Zaku's user avatar
  • 156
0 votes
1 answer
28 views

How do I use groupby on the output of a mapper?

This is a continuation of my previous question: How to print only if a character is an alphabet? I now have a mapper that is working perfectly, and it's giving me this output when I use a text file ...
Zaku's user avatar
  • 156
0 votes
1 answer
112 views

pool.map() doesn't like list of lists

I have the following piece of code: import sentence_transformers import multiprocessing from tqdm import tqdm from multiprocessing import Pool embedding_model = sentence_transformers....
Patthebug's user avatar
  • 4,647
0 votes
1 answer
35 views

Mapreduce not running on 8 CPUs but works on 2 CPUs (Amazon Sagemaker)

I have the following piece of code that I'm trying to run on multiple machines using Amazon Sagemaker. I am printing a few things for debugging in this code. When I run this code on an instance with 2 ...
Patthebug's user avatar
  • 4,647
0 votes
1 answer
58 views

Hadoop MapReduce job does not produce any logs and stats in local mode

I am writing very basic map reduce job to count words in a text file in local mode. Though job is running fine and gives output but it does not produce any map reduce logs or statistics. I have ...
Bagira's user avatar
  • 2,149
0 votes
1 answer
332 views

IndexError: pop from an empty deque while using map in Python

I have the following piece of code in which I am trying to calculate sentence embeddings in parallel. import multiprocessing from tqdm import tqdm # Define the function to be executed in parallel def ...
Patthebug's user avatar
  • 4,647
1 vote
1 answer
49 views

Exception in thread "main" java.lang.ClassNotFoundException in Java MapReduce Application Hadoop

import java.io.IOException; import java.util.HashMap; import java.util.Map; import java.util.StringTokenizer; // bro lets see this new package works or not. import org.apache.hadoop.conf....
Bharath Raj's user avatar
0 votes
0 answers
50 views

Using MapReduce for Federated Learning

I would like to build a federated learning system with PyTorch. I've currently worked out and written the code for the following: Finetuning models on client devices Sending back the finetuned models ...
Aditya Srivastava's user avatar
0 votes
2 answers
90 views

How to find top K when K is too large for memory

Let's say there is a large file with N elements and I want to find the top K largest elements from that file and output to a new file. If N is large but K is small, I could use a min-heap of size K ...
jmtt's user avatar
  • 51
0 votes
0 answers
85 views

User event script not getting trigged which is deployed on a entity record

I have created a user event script which is deployed on the vendor record. There is another map reduce script which is creating the vendor record. The user event script is working fine from UI, but ...
Sanath kumar siripuram's user avatar
0 votes
5 answers
99 views

How to return 1 only if the last letter of a word is a vowel? Return 0 otherwise

This is a follow up to this question I posted earlier: How to return 1 if the first letter of a word is a vowel, return 0 otherwise. Mapper (MapReduce) problem I am having trouble returning 1 only if ...
Zaku's user avatar
  • 156
2 votes
2 answers
85 views

How to return 1 if the first letter of a word is a vowel, return 0 otherwise. Mapper (MapReduce) problem

This is the first part of a MapReduce problem I am working on. I need a function that yields 1 if the first letter of a word starts with a vowel, otherwise it should yield 0. The program is run in the ...
Zaku's user avatar
  • 156

1
2 3 4 5
243