Questions tagged [amazon-emr]

Amazon Elastic MapReduce (Amazon EMR) is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Filter by
Sorted by
Tagged with
1 vote
0 answers
29 views

Pyspark gives different df.count() result on every other run

I am using AWS EMR (v 6.11.1), PySpark (v 3.3.2). After some transformations (mainly after groupBy, dropDuplicates) on data getting different values in output of pyspark.sql.DataFrame.count() on every ...
Atul G's user avatar
  • 11
0 votes
0 answers
18 views

How is my total executor cores more than total cores in cluster of spark application?

I am using AWS EMR for running spark with 1 instance of master node (m6g.xlarge) and 1 instance of slave node (m6g.2xlarge). m6g.2xlarge has 8 vCpu, default cpu cores. I am running spark with ...
Hilal Kaldane's user avatar
0 votes
1 answer
33 views

EMR serverless NoClassDefFoundError: software/amazon/awssdk/transfer/s3/progress/TransferListener

I have a spark application which reads parquet files from S3. My pipeline is pretty simple and it works fine on my local cluster. Here is the relevant part where I read the parquet: final Dataset<...
Vinod Mohanan's user avatar
0 votes
0 answers
19 views

Understanding AWS EMR Behavior with Spot Instances and Failed Queries Despite 'Completed' Status

I run KPI aggregation queries every morning using AWS EMR. Since I am using spot instances, there are times when the servers are forcibly terminated due to resource shortages. In such cases, the EMR ...
how_about_aws's user avatar
0 votes
0 answers
6 views

Hadoop BlockMissingException on AWS EMR Cluster

I'm facing difficulties executing a job on Apache Spark within an AWS EMR (Elastic MapReduce) cluster. It's essential to note that the EMR cluster is provisioned exclusively for processing this ...
Fernando Fraga's user avatar
0 votes
0 answers
18 views

Save pyspark dataframe as RecordIO protobuf

I want to save my pyspark dataframe in RecordIO protobuf format. I am using Amazon EMR to run my pyspark scripts, and I want to use AWS SageMaker to train a machine learning model. SageMaker pipe mode ...
Maokai's user avatar
  • 164
0 votes
0 answers
28 views

Running a dbt snapshot model for the first time works, but on the second run, it executes indefinitely

I'm encountering an issue while running a snapshot on a table with over 1,000,000 rows, following the schema detailed here: Table Schema. The initial run of the snapshot generates the table correctly. ...
barbara vitoria's user avatar
-1 votes
0 answers
10 views

AWS EMR - Can it connect to external on prem Dell S3 file storage?

I would like to know if AWS EMR is able to read from an external on prem Dell S3 file storage system
David Sánchez's user avatar
0 votes
0 answers
33 views

Exception while running pyspark sql query

I'm new to pyspark and trying to run some basic flows. This code below fails with an exception and I could not find any suggested solution, it runs on an AWS EMR from local Pycharm as a remote ...
Nufar Dory's user avatar
0 votes
1 answer
29 views

Fail to start Flink SQL client on EMR on EKS docker image

I'm playing with the Flink docker image of EMR on EKS - public.ecr.aws/emr-on-eks/flink/emr-6.15.0-flink:latest. When I try to start the SQL client, I see the following error. (I don't see this error ...
Jaehyeon Kim's user avatar
  • 1,378
0 votes
0 answers
34 views
+50

Can't train sagemaker model using recordio protobuf data saved in S3

I want to preprocess data in Amazon EMR using PySpark and train a machine learning model in SageMaker using pipe mode. The issue I am having now is saving data in S3 and feeding it to the model. ...
Maokai's user avatar
  • 164
0 votes
1 answer
28 views

EMR Cluster logs to cloudwatch

Our Apache-Flink Job runs on EMR cluster, we have the logs produced to EMR as logs are available for limited amount of time: how to produce them to Cloudwatch?
Anonymous's user avatar
0 votes
1 answer
21 views

Multiple CSV Scan in spark structured streaming for same file in s3

I have a spark 3.4.1 structured streaming job which uses custom s3-sqs connector by AWS which reads messages from sqs, reads the provided path in SQS Message and then read from S3. Now I need to split ...
Umang Desai's user avatar
0 votes
0 answers
20 views

Most efficient way to distribute inference of trained keras model using pandas_udf, two models in one go

Using python3.8, pyspark3.4, tensorflow2.11 Say I have two models I want to make predictions with. My first approach was to use two separate pandas_udf function to make two predictions. @pandas_udf() ...
haneulkim's user avatar
  • 4,598
0 votes
0 answers
39 views

Apache Spark doesn't perform dynamic Allocation as expected

I have the following settings in my Spark that runs in AWS EMR cluster. Based on these settings, Spark should allocate up to 10 executors to my job. But I see only 2 executors allocated even when no ...
mkvakin's user avatar
  • 581
1 vote
1 answer
83 views

How to make EMR cluster auto-scaling to utilize on-demand instances while staying within the max limit?

I have an EMR cluster created by Terraform. Here is the auto scaling policy: resource "aws_emr_managed_scaling_policy" "my_aws_emr_managed_scaling_policy" { cluster_id = var....
Hongbo Miao's user avatar
  • 46.4k
0 votes
1 answer
47 views

Reading delta format parquet with Pyspark on EMR on EC2 cluster

I've come across a case where I can run pyspark on a regular EC2 instance, manually pip installing pyspark and then calling the delta-core jar package dependency as a spark config as explained in ...
geominded's user avatar
  • 198
0 votes
0 answers
51 views

Thrift server with Hudi (dbt)

I'm trying to set up Spark as engine in DBT. At first I was trying with Thriftserver in EMR 6.11, as per the docs, but it's failing with the message stating it can't find the Hudi package 20:59:12 ...
Lucas Abreu's user avatar
0 votes
1 answer
28 views

Apache Hive - Programmetically same query but different result, why?

Concern 01. I have 2 tables Table1 has 41 columns and more than 100mil records and Table2 has 20 columns and 10mil records. I was using the following queries to verify the records. Query 01. SELECT ...
Thusitha Jayasundara's user avatar
0 votes
0 answers
32 views

s3-dist-cp groupby Regex Capture

I'm using EMR to combine hundreds of thousands of very small (1-5) row csv files. I want to concatenate them into around 100MB files so they are easier to work with. My EMR job uses command-runner.jar ...
MRMDP's user avatar
  • 11
0 votes
1 answer
40 views

Amazon EMR: No matching distribution found for geopandas==0.14.0

I am trying to start a Amazon EMR 6.14.0 cluster. Here is my bootstrap script set_up.sh: #!/usr/bin/env bash set -e python3 -m pip install geopandas==0.14.0 However, the Amazon EMR cluster failed to ...
Hongbo Miao's user avatar
  • 46.4k
1 vote
1 answer
44 views

IllegalArgumentException: BigQueryConnectorException$InvalidSchemaException: Destination table's schema is not compatible with dataframe's schema

We are unable to write data into big query. facing below issue. Big query connector jar: spark-3.4-bigquery-0.33.0.jar ERROR Client: Application diagnostics message: User class threw exception: java....
Raghavender's user avatar
0 votes
1 answer
32 views

Can we control how AWS Glue manages EMR cluster

I have a use case where I need to check if a Glue Job is already running and if yes, for the new glue job reuse the same EMR cluster. Is this possible?
Sallu's user avatar
  • 116
0 votes
1 answer
20 views

AWS EMR setup auto termination policy issue

Could not setup EMR auto termination idle time AWS SDK 2.21.5 newest version, I sent job to AWS EMR. In AWS console, shows below, the idle time was correct, but the termination option is still ...
Weever's user avatar
  • 13
0 votes
0 answers
32 views

Max number of executor failures (120) reached getting in AWS EMR

I am trying to run my spark job on AWS EMR for migration work (which is running fine on on-prem) but getting failed in AWS(after AWS changes) with below error "Max number of executor failures (...
Abhishek Shah's user avatar
0 votes
1 answer
39 views

AWS EMR YARN does not allocate all requested executors

The situation: 1x Primary node: 4-cores 8GiB memory 2x core nodes: 16-cores -64GiB memory I tried to request 6 executors, each running 5 cores, so that my application would utilise 30 cores total. ...
William's user avatar
0 votes
1 answer
44 views

How to config AWS EMR to wait for while after last job finished

Currently EMR cluster would automatically terminated after last job was finished. We have service would send new job(step) to existing running EMR cluster. Since cluster launching take time and cost ...
Weever's user avatar
  • 13
0 votes
0 answers
18 views

How can I run Java MapReduce application on Apache TEZ engine?

I'm using the following code to run the Java MapReduce application in an AWS EMR. Now, I need to run this in the TEZ engine, not the MapReduce engine. When I changed the mapreduce.framework.name ...
Thusitha Jayasundara's user avatar
0 votes
0 answers
96 views

Why Iceberg tables created in Spark in EMR notebook do not show schema in AWS Glue and AWS Athena?

By following the AWS document at Use an Iceberg cluster with Spark and iceberg doc at Iceberg AWS Integrations, my goal is to create the Iceberg tables in Spark in an EMR workspace/notebook, and query ...
Nagusameta's user avatar
0 votes
0 answers
25 views

i am facing an issue to get token using juno emr oauth1

am attempting to generate a token using Juno EMR OAuth1. I have added the client ID and client secret, but I encountered a status code "404" and an error message stating "no service ...
Qaiser khan's user avatar
0 votes
0 answers
119 views

Spark fails to reconfigure log4j2 when using plug-ins bundled in fat jars

After upgrading Spark code base from 3.2 to 3.3+, porting Log4j1 to Log4j2 code, we find that Spark doesn't let us reconfigure Log4j programmatically like we were able to do before. I use Maven shade ...
jeanfrancis's user avatar
0 votes
0 answers
12 views

can BigDL be run in aws EMR?

I am beginner in aws EMR and I need to run BigDL in AWS EMR, is this possible or I have to use this service https://aws.amazon.com/fr/blogs/machine-learning/running-bigdl-deep-learning-for-apache-...
fattoun's user avatar
  • 33
1 vote
1 answer
143 views

pyspark & iceberg: `update *` not working in `merge into`?

I run pyspark on AWS EMR studio MERGE INTO iceberg_catalog.staging.tbl AS t USING tempview AS s ON t.number = s.number WHEN MATCHED THEN UPDATE * WHEN NOT MATCHED THEN INSERT * but got pyspark.sql....
BAE's user avatar
  • 8,602
0 votes
1 answer
53 views

How to read .csv and .csv.gz file with all executors in EMR cluster

I am using aws emr service (emr 6.9.0) with one core node and 8 executors of instance type m5.2xlarge. My requirement is to execute read csv or csv.gz file through a spark scala job and write it into ...
Shan's user avatar
  • 31
0 votes
0 answers
31 views

AWS EMR Master node is missing avro files

I am following the AWS workshop to create AWS EMR Cluster and play with HUDI - https://catalog.us-east-1.prod.workshops.aws/workshops/c86bd131-f6bf-4e8f-b798-58fd450d3c44/en-US I created only the ...
Judi's user avatar
  • 759
0 votes
1 answer
117 views

AWS EMR Serverless Spark Job Excepton

Getting this error when trying to run a simple spark job which reads a json file from s3 and prints the schema. "com.amazonaws.emr.serverless.shaded.software.amazon.awssdk.services....
shobhit's user avatar
  • 89
0 votes
0 answers
20 views

Trying to set spark.sql.pivotMaxValues in AWS EMR Pyspark notebook

I have similarity scores in a pyspark rdd in an EMR notebook. My columns are id_a, id_b, and my third column is similarity_score. I want to cluster these IDs using kmeans. When I try to pivot this ...
annie_b2's user avatar
0 votes
0 answers
99 views

Unable to interact with OpenSearch from EMR ( using spark ES Connector ) after Domain Access Policy is added

I am loading some Data to OpenSearch from EMR , the Dataset Size is approximately 100 Million Records on a Daily basis to a new Index everyday. Earlier the Domain Access policy was not restricted , ...
Tim-Timer's user avatar
0 votes
1 answer
72 views

How to specify spot instance type for the `aws emr create-cluster` command?

I created a EMR cluster by selecting "Use Spot purchasing option" for the Task nodes: Then I copied the CLI command by clicking "View command for cloning cluster": However, this ...
Hongbo Miao's user avatar
  • 46.4k
0 votes
1 answer
69 views

Spark executors getting idle timeout

I have an EMR cluster where I am running pyspark jobs with spark dynamic allocation. spark.dynamicAllocation.enabled=true Below is config of cluster 1 Node 128GB Ram 10 cores Core Nodes Autoscaled ...
Vrishank's user avatar
  • 312
0 votes
0 answers
26 views

Unable to see spark error in logs in EMR Cluster

I have a Amazon EMR spark cluster which was recently upgraded to emr-6.9.0 and these are the applications available and their versions Hadoop 3.3.3 Pig 0.17.0 Hue 4.10.0 Hive 3.1.3 ...
naga satish's user avatar
0 votes
1 answer
91 views

Simple UDF apply function from the doc is failing with Spark 3.3

This simple code from the latest doc does not work on the EMR Studio Spark cluster (current version: 3.3.1-amzn-0) df = spark.createDataFrame( [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ...
mountrix's user avatar
  • 1,166
0 votes
0 answers
54 views

Set Java version 11 in transient EMR Cluster for Spark

I'm creating transient EMR cluster using following lambda code : List<Configuration> configurations = new ArrayList<>(); configurations.add(new Configuration() ...
Nemanja's user avatar
  • 3,325
0 votes
0 answers
30 views

AWS EC2 instance types supported by EMR

I need to retrieve a list of EC2 instance types supported by EMR. I collected the list of all ec2 instance type for a particular region using this AmazonEC2 ec2Client = AmazonEC2ClientBuilder....
Arjun C E's user avatar
  • 401
0 votes
1 answer
741 views

How to install openSSL 1.1 on AWS EMR?

I am using a boostrap file to install python 3.10 for an EMR project running Pyspark. Python 3.10 needs openSSL 1.1 or greater to work with, but EMR has OpenSSL 1.0.2k-fips (running command openssl ...
mbaggio's user avatar
1 vote
1 answer
92 views

Efficient partial string search on large pyspark dataframes

Hello fellow developers, I'm currently working on a PySpark project where I need to perform a join between two large dataframes. One dataframe contains around 10 million entries with short strings as ...
pnv's user avatar
  • 1,447
0 votes
1 answer
71 views

Redshift interpreting boolean data type as bit and hence not able to move hudi table from S3 to Redshift if there is any boolean data type column

I'm creating a data pipeline in AWS for moving data from S3 to Redshift via EMR. Data is stored in the HUDI format in S3 in parquet files. I've created the Pyspark script for full load transfer and ...
Rahul Kohli's user avatar
0 votes
0 answers
14 views

how to invoke an application packaged inside a bundled virtual environment with spark

i have a python repository that i bundled into a virtual environment with poetry bundle venv that is then pushed to AWS S3. Inside this virtual environment exists a script named test.py. I want to ...
lollerskates's user avatar
  • 1,020
0 votes
1 answer
32 views

How to pass the list elements present within map in scala?

I'm currently learning scala-spark so please bear with me. I'm trying to apply a function to scala dataframe to create a new column as below - import org.apache.spark.sql.functions._ import org.apache....
djm's user avatar
  • 45
0 votes
1 answer
70 views

ClassNotFoundException: scala.Product$class when using elasticsearch-hadoop for spark

I am running a spark-submit job on AWS EMR to read from an elasticsearch node. when the job executes this command python: es_config = { "es.nodes": url_to_my_node, "es.port"...
whenthoughsmatch's user avatar

1
2 3 4 5
68