How are we doing? Please help us improve AWS. Take our short survey

Questions tagged [aws-glue]

AWS Glue is a fully managed ETL (extract, transform, and load) service that can categorize your data, clean it, enrich it, and move it between various data stores. AWS Glue consists of a central data repository known as the AWS Glue Data Catalog, an ETL engine that automatically generates Python code, and a scheduler that handles dependency resolution, job monitoring, and retries. AWS Glue is serverless, so there's no infrastructure to manage.

aws-glue
Filter by
Sorted by
Tagged with
0 votes
0 answers
138 views

AWS Glue Job Failing With More Than One Executor?

I have an AWS glue job that reads from a glue catalog. The glue catalog is from s3 files with only 73 objects for a total size of 126.3 KB. When the glue job runs I only have 200 rows of data however ...
Kenneth L's user avatar
1 vote
2 answers
345 views

Running R in an AWS Glue job

Imagine you had a set of R scripts that form an ETL pipeline that you wanted to run as an AWS Glue job. AWS Glue supports Python and Scala. Is it possible to call an R as a Python subprocess (or a ...
Rich Pauloo's user avatar
  • 8,104
0 votes
1 answer
331 views

Attach connection to Glue job orchestrated by AWS Airflow

I am trying to trigger a glue job by Airflow. Triggering works great but it is going to timeout because of the missing Redshift connection. When I add the connection manually inside Job details tab ...
mad4red's user avatar
  • 21
0 votes
0 answers
112 views

Mongo Spark connector doesn't connect to the provided uri

Mongo Spark connector doesn't connect to the provided uri Currently Im trying to connect to mongo from AWS glue, Using mongo-spark-connector_2.12-10.1.1 with Glue 3.0 and Python 3 to connect. using ...
Nikhil Asalkar's user avatar
0 votes
0 answers
149 views

Unable to get AWS Glue ETL logs after a certain point: What's wrong?

I have created a ETL job using AWS Glue, I'm unable to get the logs after 3,4 lines of code. The job has IAM role with Administrator Access Type of ETL Job - Spark Glue Version - 3.0, Supports Spark ...
kbya2005's user avatar
0 votes
0 answers
487 views

how to read file from s3 in scala AWS glue job without using spark?

i am trying to read a csv from my s3 bucket as a BufferReader class object. I found the solution from internet which says :- Certainly! Here's an example of how you can read a file from Amazon S3 in ...
Faisal Khan's user avatar
0 votes
1 answer
414 views

How to change schema of crawled table while running the AWS glue job

Created a glue ETL job using pyspark in where output in s3 was only .csv and .csv.gz along with glue crawler later we created a test pipline with same logic but this time output in parquet format but ...
Niraj Niyanta's user avatar
0 votes
1 answer
514 views

Merge schema of multiple parquet files in Glue DynamicFrame using Python

I want to read multiple parquet files(S3 source) with different schemas into a Glue DynamicFrame. I am unable to merge the schemas of the files. The first file's header is interpreted as the schema of ...
Dave's user avatar
  • 73
0 votes
1 answer
609 views

Connect to on-premise MongoDB from AWS Glue

I am trying to connect to on-premise MongoDB from AWS Glue. I have created a Database connection with connection type as MongoDB, SSL enabled true, VPC, Subnet, Security group in AWS console. I added ...
Raja's user avatar
  • 11
0 votes
1 answer
244 views

Redshift Spectrum query fails with Parsed manifest is not a valid JSON object

I have an S3 bucket with 5 prefixes / "sub-folders", each containing a set of CSV files that were exported from a legacy database. The CSV files have been crawled and created a Glue database ...
RobD's user avatar
  • 1,704
0 votes
0 answers
124 views

How to use AWS Glue Job Bookmarking with concurrent runs for s3 source

My requirement is I have a file with say 100 records, I want to process 20 records in 5 concurrent runs. How to achieve this?
Venu gupta's user avatar
0 votes
1 answer
172 views

AWS Glue Dynamo Extract

I exported DYnamoDB table to S3, and used this guide to import into AThena. Each column is an entire dynam entry. I get: Item {key={s=value1}, key2={n=value2}} {key={s=value3}, key2={n=value4}} {key={...
MikeF's user avatar
  • 885
0 votes
0 answers
76 views

Convert very large Xml file to Json file using Python in aws glue

I am trying to convert very large xml files which are placed in my S3 bucket to Json file. For smaller files upto 300 mb i am able to convert successfully using ‘xmltodict’. Although for larger files ...
mayank agarawal's user avatar
4 votes
1 answer
1k views

Creating a delta table in S3 with Glue + delta lake creates a glue catalog table with wrong location

I am creating an ETL job with Glue using Delta lake framework. Example demo data is defined as: data = {'visitor': ['foo', 'bar', 'baz'], 'id': [1, 2, 3], 'B': [1, 0, 1], '...
huehue's user avatar
  • 80
0 votes
0 answers
372 views

AWS GLUE Workflow: How to add both success and fail state to the same job

I have a scenario where I need to run the workflow despite the state of the jobs in the workflow. There are 2 nodes. I created a on demand trigger which will trigger Job_1 and Job_2. Once Job_1 and ...
sridhar5999's user avatar
0 votes
1 answer
211 views

Delay notification threshold in Glue job

I am using Delay notification threshold in glue job. I am getting notification on mail after minutes which I pass in glue job. I am passing filename in job parameter and I want to send it in mail body....
Sharayu Dhamale's user avatar
0 votes
1 answer
2k views

AWS Glue JDBC Connection created using Cloud Formation is not setting the password

If I create a Glue JDBC connection to a RDS Aurora instance, the connection does not work with the following error: InvalidInputException: Unable to resolve any valid connection. After checking it in ...
IamAshKS's user avatar
  • 783
0 votes
1 answer
466 views

How to access spark and glueContext in Glue Custom Visual Transform script

I am trying to create a AWS Glue Custom Visual Traform script that can truncate a MySQL table before loading the data into it. But I am not able to access the spark and glueContext variable from the ...
Piyush Pranjal's user avatar
0 votes
1 answer
91 views

Glue Custom Visual Script Running indefinitely

I am trying to create a Custom Visual Transform but unfortunately facing some issues. Here my motive is to Truncate a MySQL Table before loading the data into it and I want to do it with the help of ...
Piyush Pranjal's user avatar
0 votes
0 answers
38 views

How to efficiently flatten pyspark dataframe

I have below dataframe df.show() +--------------------+--------------------+--------------------+--------------------+--------------------+ | PKG_NBR| PKG_VINS | ...
Dcook's user avatar
  • 943
0 votes
1 answer
197 views

Issue importing custom Python modules in Jupyter notebooks with SparkMagic and AWS Glue endpoint

I'm encountering an issue while attempting to run ETL scripts in Jupyter notebooks using SparkMagic, which is connected to an AWS Glue endpoint via SSH. I followed the tutorial provided in the AWS ...
RX00's user avatar
  • 1
0 votes
1 answer
186 views

Glue Transform not finding argument passed to Glue Job

Lambda function triggered by CSV dropped into S3 bucket. Lambda then calls a Glue Job. Error occurs in "Transform-Custom Code" node in line 2: def MyTransform (glueContext, dfc) -> ...
ScotterMonkey's user avatar
-1 votes
1 answer
285 views

How to import an already existing AWS Glue Crawler in Terraform?

I have a module that creates crawlers... and uses this variable to iterate over a map of objects where the crawlers are defined with the details that are needed module main.tf resource "...
Leonardo Arango's user avatar
0 votes
1 answer
786 views

MERGE INTO with datalake on AWS Glue inserting rows instead of updating

I am trying to set up a demo Glue job that demonstrates upserts using data lake framework. I have example full load data I have saved as delta table in S3 bucket defined as follows: data = {'visitor': ...
huehue's user avatar
  • 80
0 votes
0 answers
185 views

AWS Glue ETL job removing partition keys from file

I am writing an AWS Glue ETL job. It is really simple, just reading in some s3 files, creating a dynamic frame, partitioning based on a few keys, and then writing that dynamic frame to another s3 ...
Haley Kirk's user avatar
0 votes
1 answer
715 views

AWS Glue Job - Job Bookmark - Incremental daily extraction from RDS to S3

I'm building up/developing a serveless pipeline using AWS Glue. Context: I'm extracting data from RDS AWS Postgres instance. This instance is a productive database of a mobile app (OLTP). Goal: ...
fCremer's user avatar
  • 27
0 votes
0 answers
329 views

How to run a java spark project on AWS Glue?

I have a java spark project in which, i simply want to read data from S3 and also write it in S3, but i came to that AWS glue does not support java, so i decided to write a wrapper in scala the code ...
Faisal Khan's user avatar
0 votes
0 answers
136 views

Widget not working aws glue jupyter notebook

I tried below code in aws glue jupyter notebook for widget textbox but I got text context but I expected text box. import ipywidgets as widgets from IPython.display import display   ...
Antony's user avatar
  • 966
1 vote
1 answer
112 views

Is there a way to cast columns by data types without naming the columns specifically?

Can we replace all columns of a specific data-type to be replaced by a different datatype? For instance, if we have a column in DF as decimal(38,10), can there be a blanket/generic statement to cast ...
WhiteBird's user avatar
  • 117
0 votes
1 answer
144 views

Error: 'JavaPackage' object is not callable on aws glue

I have been trying to use the spark library XSDToSchema on AWS glue to read xsd file but it gives me 'JavaPackage' object is not callable error with open('example_xsd.xsd', 'r', encoding='utf-8') as f:...
The Vinh's user avatar
0 votes
1 answer
258 views

ThrottlingException - Update JOB_POLL_INTERVAL in airflow DAG AwsGlueJobOperator

I am running an Airflow DAG ad using AwsGlueJobOperator to run AWS Glue job. Below is the sample code for reference. from airflow.providers.amazon.aws.operators.glue import AwsGlueJobOperator with ...
Himanshu Gaur's user avatar
1 vote
1 answer
154 views

Cloudfront Logs - Athena - AWS Glue

I am trying to add a Grok pattern for standard cloudfront logs: %{DATE:date}\s%{TIME:time}\s%{DATA:x_edge_location}\s%{INT:sc_bytes:int}\s%{IP:client_ip}\s%{WORD:cs_method}\s%{HOSTNAME:cs_host}\s%{...
JoaMika's user avatar
  • 1,807
1 vote
2 answers
231 views

AWS - Merge Multiple .csv from S3 into Table

I have an S3 bucket that contains multiple different files by date with different schemas (they are not in sub-folders). For example: FileTypeA-032323 FileTypeA-032423 FileTypeA-032523 FileTypeB-...
Shell35's user avatar
  • 11
0 votes
0 answers
102 views

PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes. Pass allowDiskUse:true to opt in

I have data Like this (shown below). When I run the AWS Glue crawler on this data, I get the error mentioned in the topic name {“_id”:{“$binary”:{“base64”:“EyYO+A68T2WJm/p+ny+buw==”,“subType”:“04”}},“...
pscodes's user avatar
  • 11
2 votes
2 answers
338 views

Glue PySpark - Fixed Width File Processing and Validation

I have to validate Fixed Width files that I am reading from S3 to Glue. I have the lengths of each column and I have to write a glue job to validate these files. How do I efficiently check the lengths ...
Samarth Navneet's user avatar
0 votes
1 answer
241 views

Specifying json key mappings for AWS Firehose taking DMS data as an input

I want to migrate some data on AWS using Kinesis, and store it in an S3 bucket after converting it to Parquet. The set up is very similar to this one. The architecture looks as follows: My data is ...
Rita R's user avatar
  • 11
0 votes
1 answer
877 views

How can we create a AWS Glue Crawler to Crawl on S3 Bucket?

Create glue Crawler for S3 Bucket in AWS Firstly, I started with glue in aws. I make a glue crawler on aws accounts glue after that I also intoduce database and table name but I will confuse how to ...
Dishank kabra's user avatar
-1 votes
1 answer
1k views

AWS Glue Script - Get Data from S3 Excel file with multiple Sheets and convert them each to parquet using Pyspark or Python

i have Excel file names Emp_Data.xlsx contaning 3 Sheets in S3 bucket in loaction as below: **Location :**s3://Employee/Data/Emp_Data.xlsx Ask : Acces the excel file in Aws Glue Script using Pyspark ...
malcolm richard's user avatar
0 votes
1 answer
3k views

AWS Athena and Glue indexes

Consider I have very large files (~25 millions) records of the next objects: { "name": "bob", "age": 26, "address": "...", "...
Shay Zambrovski's user avatar
-1 votes
1 answer
106 views

AWS ETL extraction from RDBMS/CSV

I am trying to see what's best AWS tools anyone using for extraction like Step/Lambda/Glue/Athena etc I got about 50 tables(Oracle to Redshift) and CSV' as source data Looking to load Delta daily ...
Corona2020's user avatar
1 vote
0 answers
124 views

Continuous run of Glue Jobs

I designed a simple solution where I just want my lambda function to trigger Glue job whenever user places CSV files in S3 bucket ( User will submit CSV files only once a day). My glue job runs fines ...
Zeeshan Ahmed's user avatar
0 votes
1 answer
98 views

What is the best way to load records in DynamoDB using Pyspark(glue)?

I have an use case to load multiple records into DynamoDB. I am thinking of using Glue with sparck context. What is the function I need to look into? PutItem runs for long, is there an alternative ...
sandeep nutakki's user avatar
0 votes
1 answer
402 views

Upload/Download large file from spring boot application

I have spring boot application that is responsible for uploading/downloading data in the Excel CSV format to/from the database.For the few records it worked perfectly fine but as records are more than ...
Sunny's user avatar
  • 1
0 votes
1 answer
1k views

Missing Columns in AWS Glue

I'm trying to upskill with AWS Glue, and implement a simple test pipeline that hopefully will later become a much bigger set of pipelines to handle our ETL. But I'm seeing some behaviour that I don't ...
James Baker's user avatar
  • 1,202
0 votes
0 answers
164 views

Alter table in Aws Athena

I've data in s3 and its respective table schema is in aws glue catalog. I write data into table using table name in my spark job. Now I want to add new column in table, So before running my spark job, ...
morgan's user avatar
  • 29
0 votes
1 answer
23 views

Transforming folder values into dataframe column when reading partitioned data

I'm trying to read multiple parquet files from multiple folders, and I want the folders to be a column on my dataframe. Example: Folders anomesdia=20230426 anomesdia=20230425 anomesdia=20230424 When ...
Hernandes Matias Junior's user avatar
0 votes
0 answers
28 views

Pyspark not abl to read catalog table from EMR

I need to read AWS glue catalog table from emr using pyspark. Please find the screenshot. It gives error like amazon s3 access denied exception. I am not doing anything with s3 in script.
KAUSHIK DEY's user avatar
1 vote
1 answer
865 views

Why does AWS Athena completly outperform a AWS Glue at querying the MAX of a string column in a large dataset?

I have a large dataset that comes from an IoT input with aproximately 5000 million rows. The dataset is stored into S3 in a single directory in the format of parquet. Each subset of rows corresponding ...
Sergio's user avatar
  • 96
0 votes
1 answer
781 views

AWS Glue job create table

I have a setup currently with a crawler that connects to a PostgreSQL database with JDBC, this works and the crawler generates around 20 tables for this database. I now want to create an ETL job that ...
davyioner's user avatar
  • 131
1 vote
1 answer
268 views

Glue Jobs creating table and writing in s3 throwing __PLACEHOLDER__ error

I am trying to create a table while writing data into s3 from glue. Below error came. Any resolution or any optimized way to create table and write. orderdf.write.format('parquet').partitionBy('...
user3858193's user avatar
  • 1,360

1
11 12
13
14 15
97