Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
28 views

Python awswrangler performance under large number of partitions

I need to store/fetch data using two hierarchy levels, date and class. So when I upload data to S3 as part of the ETL pipeline, I'm using awswrangler's to_parquet function with partition_cols=["...
Miguel Silva's user avatar
0 votes
1 answer
53 views

cast list type value of dictionary column in pandas

I am trying to write a complex type to dynamodb, but getting into the error: import pandas as pd df = pd.DataFrame.from_records([{"col_w_status": {"ACTIVE":["ABC100-01"],...
NNM's user avatar
  • 390
0 votes
0 answers
17 views

In AWS Wrangler should I use append or overwite?

I am using AWS Wrangler to read some data in and then do some checks against that data to ensure my new data doesn't have the same timestamps that exist in the original data (this may occur if the ...
T.UK's user avatar
  • 79
0 votes
0 answers
15 views

Choosing the AWS technology for building a system that performs automated random and stratified sampling on AWS S3 data

Context: I'm building a system that requires pulling up either a random/stratified sample on a dataset stored in either S3/RedShift/Relational database like RDS. The data set size could be in the ...
Don Bosco's user avatar
  • 121
1 vote
0 answers
104 views

Dynamic parameters in a parameterized query AWS wrangler

I am attempting to parameterize my existing SQL code using aws data wrangler, however, I was unable to find any documentation to help in creating a variable number of dynamic parameters. For example, ...
TanDev's user avatar
  • 427
1 vote
1 answer
230 views

why does aws sagemaker data wrangler not allow me to deploy model in canvas

I'm logged in with an administrator account. I'm using aws sagemaker data wrangler in canvas. When I try and deploy the model but I get the message 'you don't have permissions to deploy models. Please ...
Jonathan Wong's user avatar
1 vote
0 answers
48 views

AWS Wrangler creating dictionary values on its own

I am using awswrangler for my project in Glue. The project should be pretty straightforward. I have dataframe with column "A". This is the content of the dataframe: data = {'A': [['1A', '5A']...
Haggerman Swaggerman's user avatar
0 votes
0 answers
91 views

LakeFS, S3 Could not connect to the endpoint URL

I am trying to write parquet files to LakeFS with S3 configuration. I configured docker for my services. Here is my docker compose file version: "3.7" services: web: build: . ...
Anon's user avatar
  • 396
0 votes
1 answer
214 views

how to use athena VPC endpoint to query data from isolated network mode in sagemaker preprocesing job

I wrote a sagemaker processing job in isolated network . it has an Athena sql that reads from athena to a dataframe . But it throws error as "botocore.exceptions.NoCredentialsError: Unable to ...
dewdrops's user avatar
1 vote
0 answers
77 views

Partitioning Parquet AWS Wrangler with LakeFs

I was trying to partition the parquet on S3 and it worked with AWS Wrangler. basename_template = 'part.' partitioning = ['cust_id', 'file_name', 'added_year', 'added_month', 'added_date'] loop = ...
Anon's user avatar
  • 396
0 votes
0 answers
124 views

Python Mocking AWS Wrangler / DB Connection

I'm new to Python and testing and trying to write some tests that interact with AWS wrangler in my Python 3 lambda function. Here is the documentation. The function I want to test looks like def ...
Blake R's user avatar
0 votes
0 answers
69 views

How to upload periodically a .txt file to S3?

I need to convert a Dataframe into a .txt file and upload it to S3 by using CloudFormation and Cloud9 periodically. I'm used to do it with excel file and no problems found in it as wrangler brings a '...
fervifervi's user avatar
0 votes
0 answers
279 views

Saving a dataframe as a parquet with geometry column

I currently have a column that contains POINT geometry values and using AWS Wrangler I can save it as a CSV, but when I try to save it as a parquet it raises this error: pyarrow.lib.ArrowTypeError: ...
T.UK's user avatar
  • 79
0 votes
0 answers
91 views

Aws Data Wrangler - Local Pipeline Session Failure

I am trying out the aws data wrangler example Hotel Booking Demand Example locally, so I can save the cost of data wrangler Here is what I have done so far. I created flow as in the above example in ...
Raj's user avatar
  • 166
2 votes
0 answers
431 views

Is there a way to setup retries / timeouts with `awswrangler.athena.read_sql_query`?

Often I am running many long running athena queries with awswrangler and from time to time I receive: ReadTimeoutError: Read timeout on endpoint URL: "None" Sometimes I receive some other ...
Samuel Hapak's user avatar
  • 7,152
0 votes
0 answers
466 views

AWS Wrangler WaiterError: Waiter BucketExists failed: Max attempts exceeded. Previously accepted state: Matched expected HTTP status code: 404

I've been trying to query information frin Athena using the next bit of code: import boto3 import awswrangler as wr sessAWS_Test = boto3.session.Session( aws_access_key_id = 'id', ...
ricardov-DB's user avatar
0 votes
1 answer
659 views

aws wrangler (pandas layer). problem with path to S3 bucket

here is my python code in my lambda layer. Shout out to John R, for some of this paginator code. from api gateway, I pass in path param (bucket) and query string params (fmt & date), such as: ...
bob's user avatar
  • 99
0 votes
1 answer
222 views

awswrangler query Athenea: AttributeError: Can only use .dt accessor with datetimelike values

I have one table in Athena which all columns with proper datatypes(date, bigint,int,decimal(28,2) string and etc.). I need to query out the data via aws wrangler API :athena.read_sql_query I write: ...
zac yang's user avatar
0 votes
0 answers
24 views

how to remove extra space in json output from lambda [duplicate]

I have a lambda reading csv's in an S3 bucket. The api gateway calls the lambda. The data in the csv is like this: Ticker Exchange Date Open High Low Close Volume 6A BATS ...
bob's user avatar
  • 99
0 votes
0 answers
92 views

Unable to query parquet data which has array datatype

When using awswrangler and writing to S3 in parquet format, the data files are not queryable using S3 select (for csv) or Athena. For e.g. events = [{"c1": "12", "c2": [1,...
Raman's user avatar
  • 717
0 votes
1 answer
3k views

How to specify the location of athena query results when using awswrangler

The python code below can fetch data from a pre-configured athena table when it is run on local computer. But it automatically creates an S3 bucket to store temporary tables and metadata. The ...
d.b's user avatar
  • 32.5k
2 votes
1 answer
510 views

Amazon Sagemaker Studio Data Wrangler athena query failing for large datasets

Trying to query a large dataset from Athena using AWS data wrangler. The query fails for large datasets. This is for setting up a datawrangler pipeline using UI in AWS studio trying to add a Athena ...
user2242666's user avatar
1 vote
1 answer
166 views

Is there a way to specify the Parquet Version using AWS data Wrangler

We are writing parquet Files which seem to default to version 1. enter image description here which teradata NOS complains with a "Native Object Store user error: Unsupported file version" ...
Marc Williams's user avatar
0 votes
1 answer
551 views

How can I apply a unique filter to partition column of a parquet file using wr.s3.read_parquet?

I have a parquet dataset stored in s3 and I want to read it to apply a filter to the partition field, specifically the unique. I was trying as follows, however the unique function cannot be applied ...
Jeanpiere Alcocer's user avatar
2 votes
1 answer
2k views

How to add odbc driver to aws glue python shell

I want to use pyodbc in aws qlue python shell but it require odbc driver. Currently I get error like "Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)" ...
yunus kula's user avatar
1 vote
0 answers
720 views

awswrangler.s3.to_parquet arguments question

If I have the following code: import awswrangler #df = some dataframe with year, date and other columns wr.s3.to_parquet( df=df, path=f's3://some/path/', index=False, dataset=True, ...
Petar Ulev's user avatar
0 votes
1 answer
311 views

AWS Wrangler S3 reading parquet, writing to DynamoDB - Unsupported type numpy.ndarray

I am trying to read parquet into dataframe with AWS wrangler, while writing this data to DynamoDB its erroring out with unsupported type error - Unsupported type numpy.ndarray for value....... wr.s3....
NNM's user avatar
  • 390
0 votes
1 answer
313 views

Use AWS wrangler without AWS cli

I'm running the following code through AWS wrangler import awswrangler as wr my_query = wr.athena.read_sql_query( sql="""select "$path" as path from table""&...
gseph's user avatar
  • 51
1 vote
1 answer
232 views

What is AWS S3 dataset?

Looking at documentation of awswrangler.s3.to_csv or awswrangler.s3.to_parquet, there is a dataset parameter. From testing, it looks like setting dataset=True allows, among other things, to append new ...
d.b's user avatar
  • 32.5k
2 votes
1 answer
1k views

awswrangler redshift to_sql upserting specific columns

Suggest we have a table with a row like below. I want to update just col_id and slug columns with a new values. this is the code line I use to update this row. df = pd.DataFrame([[datas.get("...
Beşir Kassab's user avatar
0 votes
1 answer
706 views

Having issues with my excel to csv conversion in aws lambda using the AWSDatawrangler layer

I have a function that reads an excel file into a dataframe and then I save that dataframe in an s3 bucket using the awswrangler api to_csv function. The excel file has data starting from different ...
Murtaza Mohsin's user avatar
0 votes
0 answers
650 views

Create 'granular' date partitions (year, month, date) in S3 parquet folders from a single date column in AWS Wrangler

I am using data wrangler to upload data from a dataframe into S3 bucket parquet files, and am trying to get it in a 'Hive'-like folder structure of: prefix - year=2022 -- month=08 --- day=01 --- day=...
Ben L's user avatar
  • 83
1 vote
0 answers
561 views

Awswrangler, KeyError while trying to apply partition_filter. Key actually exists when importing dataframe

I am trying to load a pd.DataFrame, reading it form a parquet file in aws. I am applying partition_filter to get only certain data from the df that corresponds to the conditions I want. In this ...
The Dan's user avatar
  • 1,650
4 votes
1 answer
2k views

querying athena using awsdatawrangler

I am trying to query my athena database using import awswrangler as wr df = wr.athena.read_sql_query(sql="""SELECT * FROM tablename limit 10;""" ...
Devarshi Goswami's user avatar
0 votes
2 answers
3k views

Update Athena Table using AWS Data Wrangler

I started using AWS Data Wrangler and Athena, to upload my data files onto S3, and being able to query them, respectively. My question is about the procedure to "safely" update the data in ...
tafteh's user avatar
  • 41
3 votes
1 answer
4k views

AWS Data Wrangler s3.to_parquet replicate current S3 path structure

When using wr.s3.to_parquet I can construct a path with a Formatted string literal and have existing folders using the pattern. def SaveInS3_test(Ticker, Granularity, Bucket, df, keyPrefix=""...
nipy's user avatar
  • 5,450
0 votes
0 answers
746 views

Why is AWS athena slower than reading parquet directly?

I have created a table on AWS Athena. It is partitioned both on S3, and Athena. I am now trying to load the table into a pandas dataframe using 2 methods from the awswrangler library: AWS Athena ...
survival_law00's user avatar
1 vote
0 answers
274 views

ValidationException Importing from Redshift into Data Wrangler

I'm trying to build a model workflow in AWS SageMaker using Data Wrangler for preprocessing. I'm loading data from various tables in a Redshift instance, before mutating and joining them as required ...
A. White's user avatar
6 votes
3 answers
14k views

How can I bulk upload JSON records to AWS OpenSearch index using a python client library?

I have a sufficiently large dataset that I would like to bulk index the JSON objects in AWS OpenSearch. I cannot see how to achieve this using any of: boto3, awswrangler, opensearch-py, elasticsearch, ...
jtlz2's user avatar
  • 8,285
2 votes
1 answer
4k views

How do I use awswrangler to read only the first few N rows of a parquet file stored in S3?

I am trying to use awswrangler to read into a pandas dataframe an arbitrarily-large parquet file stored in S3, but limiting my query to the first N rows due to the file's size (and my poor bandwidth). ...
jtlz2's user avatar
  • 8,285
0 votes
1 answer
486 views

Unable to find AWS Wrangler 2.10.0 Layer in US-WEST-2 under public-artifacts

Please consider the example shown in the below link: https://github.com/awslabs/aws-data-wrangler/issues/923 I am trying to get the public artifacts bucket wrangler to US-West-2, but I am unable to ...
Shubham Bengani's user avatar
2 votes
1 answer
2k views

Is there any way to capture the input file name of multiple parquet files read in with a wildcard in pandas/awswrangler?

This is the exact python analogue of the following Spark question: Is there any way to capture the input file name of multiple parquet files read in with a wildcard in Spark? I am reading in a ...
jtlz2's user avatar
  • 8,285
0 votes
1 answer
4k views

AWS Wrangler Error HIVE_METASTORE_ERROR: Table is missing storage descriptor

hope you can help me with a concern about an error with awswrangler. this is the case: i have 2 aws accounts, AccountA and AccountB, both with lakeformation enabled, i have a set of databases in AccA ...
Roger Jesus Rosales Perez's user avatar
0 votes
3 answers
3k views

How to read sheet names of excel sheet from S3 in AWS Wrangler?

I have an excel sheet which is placed in S3 and I want to read sheet names of excel sheet. I have read excel sheet with aws wrangler using awswrangler.s3.read_excel(path) How can I read sheetnames ...
Abhishek's user avatar
1 vote
0 answers
233 views

AWS Data Wrangler chunking: "Length mismatch: Expected axis has 28680 elements, new values have 100000 elements"

While loading the parquet file and getting the below error. My parquet file contains 5,128,680 rows and it is loading 5,100,000 only and not loading 28,680 records and code as like below: Error : ...
murali's user avatar
  • 11
0 votes
1 answer
347 views

Automate the date parameter while deplying the model on AWS Wrangler

I have built a XGBoost model on my local machine which takes a training data and validates the model on a testing dataset. However, I have hard-coded the date values as the training data is created ...
Advait Vartak's user avatar
1 vote
0 answers
527 views

Read data from AWS S3 using pyspark and python. (Read All columns: Partioned column also)

I have saved the spark dataframe to AWS S3 in Parquet format partitionby column "channel_name". Below code is how i saved to S3: df.write.option("header",True) \ ....
SSS's user avatar
  • 83
0 votes
1 answer
373 views

Split SQL Where IN Clause When List is to big into Smaller Requests in Python

I have setup an AWS Lambda function with python to ingest requests from a CSV and then query an AWS Serverless Aurora PostgreSQL database based on this request. The function works when the requests ...
Prof. Falken's user avatar
0 votes
1 answer
584 views

AWS Maximum BadRequestException retries reached for query Using Data API to Query RDS Serveless Aurora

I have created a Lambda function that uses awswrangler data api to read in data from an RDS Serverless Aurora PostgreSQL Database from a query. The query contains a conditional that is a list of IDs. ...
Prof. Falken's user avatar
0 votes
1 answer
724 views

Sagemaker AWSWrangler>2.3.0

As I am trying to use the function read_excel as part of AWS Wrangler, available as of version 2.3.0 in Sagemaker Jupyter Lab on Amazon Web Services, it does not install properly. The Python version ...
irkmnas's user avatar