72
questions
0
votes
0
answers
28
views
Python awswrangler performance under large number of partitions
I need to store/fetch data using two hierarchy levels, date and class. So when I upload data to S3 as part of the ETL pipeline, I'm using awswrangler's to_parquet function with partition_cols=["...
0
votes
1
answer
53
views
cast list type value of dictionary column in pandas
I am trying to write a complex type to dynamodb, but getting into the error:
import pandas as pd
df = pd.DataFrame.from_records([{"col_w_status": {"ACTIVE":["ABC100-01"],...
0
votes
0
answers
17
views
In AWS Wrangler should I use append or overwite?
I am using AWS Wrangler to read some data in and then do some checks against that data to ensure my new data doesn't have the same timestamps that exist in the original data (this may occur if the ...
0
votes
0
answers
15
views
Choosing the AWS technology for building a system that performs automated random and stratified sampling on AWS S3 data
Context: I'm building a system that requires pulling up either a random/stratified sample on a dataset stored in either S3/RedShift/Relational database like RDS. The data set size could be in the ...
1
vote
0
answers
104
views
Dynamic parameters in a parameterized query AWS wrangler
I am attempting to parameterize my existing SQL code using aws data wrangler, however, I was unable to find any documentation to help in creating a variable number of dynamic parameters. For example, ...
1
vote
1
answer
230
views
why does aws sagemaker data wrangler not allow me to deploy model in canvas
I'm logged in with an administrator account. I'm using aws sagemaker data wrangler in canvas. When I try and deploy the model but I get the message 'you don't have permissions to deploy models. Please ...
1
vote
0
answers
48
views
AWS Wrangler creating dictionary values on its own
I am using awswrangler for my project in Glue. The project should be pretty straightforward.
I have dataframe with column "A". This is the content of the dataframe:
data = {'A': [['1A', '5A']...
0
votes
0
answers
91
views
LakeFS, S3 Could not connect to the endpoint URL
I am trying to write parquet files to LakeFS with S3 configuration. I configured docker for my services.
Here is my docker compose file
version: "3.7"
services:
web:
build: .
...
0
votes
1
answer
214
views
how to use athena VPC endpoint to query data from isolated network mode in sagemaker preprocesing job
I wrote a sagemaker processing job in isolated network . it has an Athena sql that reads from athena to a dataframe .
But it throws error as "botocore.exceptions.NoCredentialsError: Unable to ...
1
vote
0
answers
77
views
Partitioning Parquet AWS Wrangler with LakeFs
I was trying to partition the parquet on S3 and it worked with AWS Wrangler.
basename_template = 'part.'
partitioning = ['cust_id', 'file_name', 'added_year', 'added_month', 'added_date']
loop = ...
0
votes
0
answers
124
views
Python Mocking AWS Wrangler / DB Connection
I'm new to Python and testing and trying to write some tests that interact with AWS wrangler in my Python 3 lambda function. Here is the documentation.
The function I want to test looks like
def ...
0
votes
0
answers
69
views
How to upload periodically a .txt file to S3?
I need to convert a Dataframe into a .txt file and upload it to S3 by using CloudFormation and Cloud9 periodically. I'm used to do it with excel file and no problems found in it as wrangler brings a '...
0
votes
0
answers
279
views
Saving a dataframe as a parquet with geometry column
I currently have a column that contains POINT geometry values and using AWS Wrangler I can save it as a CSV, but when I try to save it as a parquet it raises this error:
pyarrow.lib.ArrowTypeError: ...
0
votes
0
answers
91
views
Aws Data Wrangler - Local Pipeline Session Failure
I am trying out the aws data wrangler example Hotel Booking Demand Example locally, so I can save the cost of data wrangler
Here is what I have done so far. I created flow as in the above example in ...
2
votes
0
answers
431
views
Is there a way to setup retries / timeouts with `awswrangler.athena.read_sql_query`?
Often I am running many long running athena queries with awswrangler and from time to time I receive:
ReadTimeoutError: Read timeout on endpoint URL: "None"
Sometimes I receive some other ...
0
votes
0
answers
466
views
AWS Wrangler WaiterError: Waiter BucketExists failed: Max attempts exceeded. Previously accepted state: Matched expected HTTP status code: 404
I've been trying to query information frin Athena using the next bit of code:
import boto3
import awswrangler as wr
sessAWS_Test = boto3.session.Session(
aws_access_key_id = 'id',
...
0
votes
1
answer
659
views
aws wrangler (pandas layer). problem with path to S3 bucket
here is my python code in my lambda layer. Shout out to John R, for some of this paginator code.
from api gateway, I pass in path param (bucket) and query string params (fmt & date), such as:
...
0
votes
1
answer
222
views
awswrangler query Athenea: AttributeError: Can only use .dt accessor with datetimelike values
I have one table in Athena which all columns with proper datatypes(date, bigint,int,decimal(28,2) string and etc.).
I need to query out the data via aws wrangler API :athena.read_sql_query
I write: ...
0
votes
0
answers
24
views
how to remove extra space in json output from lambda [duplicate]
I have a lambda reading csv's in an S3 bucket. The api gateway calls the lambda.
The data in the csv is like this:
Ticker Exchange Date Open High Low Close Volume
6A BATS ...
0
votes
0
answers
92
views
Unable to query parquet data which has array datatype
When using awswrangler and writing to S3 in parquet format, the data files are not queryable using S3 select (for csv) or Athena.
For e.g.
events = [{"c1": "12", "c2": [1,...
0
votes
1
answer
3k
views
How to specify the location of athena query results when using awswrangler
The python code below can fetch data from a pre-configured athena table when it is run on local computer. But it automatically creates an S3 bucket to store temporary tables and metadata. The ...
2
votes
1
answer
510
views
Amazon Sagemaker Studio Data Wrangler athena query failing for large datasets
Trying to query a large dataset from Athena using AWS data wrangler. The query fails for large datasets. This is for setting up a datawrangler pipeline using UI in AWS studio trying to add a Athena ...
1
vote
1
answer
166
views
Is there a way to specify the Parquet Version using AWS data Wrangler
We are writing parquet Files which seem to default to version 1.
enter image description here
which teradata NOS complains with a
"Native Object Store user error: Unsupported file version"
...
0
votes
1
answer
551
views
How can I apply a unique filter to partition column of a parquet file using wr.s3.read_parquet?
I have a parquet dataset stored in s3 and I want to read it to apply a filter to the partition field, specifically the unique. I was trying as follows, however the unique function cannot be applied
...
2
votes
1
answer
2k
views
How to add odbc driver to aws glue python shell
I want to use pyodbc in aws qlue python shell but it require odbc driver. Currently I get error like "Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)"
...
1
vote
0
answers
720
views
awswrangler.s3.to_parquet arguments question
If I have the following code:
import awswrangler
#df = some dataframe with year, date and other columns
wr.s3.to_parquet(
df=df,
path=f's3://some/path/',
index=False,
dataset=True,
...
0
votes
1
answer
311
views
AWS Wrangler S3 reading parquet, writing to DynamoDB - Unsupported type numpy.ndarray
I am trying to read parquet into dataframe with AWS wrangler, while writing this data to DynamoDB its erroring out with unsupported type error - Unsupported type numpy.ndarray for value.......
wr.s3....
0
votes
1
answer
313
views
Use AWS wrangler without AWS cli
I'm running the following code through AWS wrangler
import awswrangler as wr
my_query = wr.athena.read_sql_query(
sql="""select "$path" as path from table""&...
1
vote
1
answer
232
views
What is AWS S3 dataset?
Looking at documentation of awswrangler.s3.to_csv or awswrangler.s3.to_parquet, there is a dataset parameter.
From testing, it looks like setting dataset=True allows, among other things, to append new ...
2
votes
1
answer
1k
views
awswrangler redshift to_sql upserting specific columns
Suggest we have a table with a row like below.
I want to update just col_id and slug columns with a new values.
this is the code line I use to update this row.
df = pd.DataFrame([[datas.get("...
0
votes
1
answer
706
views
Having issues with my excel to csv conversion in aws lambda using the AWSDatawrangler layer
I have a function that reads an excel file into a dataframe and then I save that dataframe in an s3 bucket using the awswrangler api to_csv function. The excel file has data starting from different ...
0
votes
0
answers
650
views
Create 'granular' date partitions (year, month, date) in S3 parquet folders from a single date column in AWS Wrangler
I am using data wrangler to upload data from a dataframe into S3 bucket parquet files, and am trying to get it in a 'Hive'-like folder structure of:
prefix
- year=2022
-- month=08
--- day=01
--- day=...
1
vote
0
answers
561
views
Awswrangler, KeyError while trying to apply partition_filter. Key actually exists when importing dataframe
I am trying to load a pd.DataFrame, reading it form a parquet file in aws.
I am applying partition_filter to get only certain data from the df that corresponds to the conditions I want. In this ...
4
votes
1
answer
2k
views
querying athena using awsdatawrangler
I am trying to query my athena database using
import awswrangler as wr
df = wr.athena.read_sql_query(sql="""SELECT * FROM tablename limit 10;"""
...
0
votes
2
answers
3k
views
Update Athena Table using AWS Data Wrangler
I started using AWS Data Wrangler and Athena, to upload my data files onto S3, and being able to query them, respectively. My question is about the procedure to "safely" update the data in ...
3
votes
1
answer
4k
views
AWS Data Wrangler s3.to_parquet replicate current S3 path structure
When using wr.s3.to_parquet I can construct a path with a Formatted string literal and have existing folders using the pattern.
def SaveInS3_test(Ticker, Granularity, Bucket, df, keyPrefix=""...
0
votes
0
answers
746
views
Why is AWS athena slower than reading parquet directly?
I have created a table on AWS Athena. It is partitioned both on S3, and Athena. I am now trying to load the table into a pandas dataframe using 2 methods from the awswrangler library: AWS Athena ...
1
vote
0
answers
274
views
ValidationException Importing from Redshift into Data Wrangler
I'm trying to build a model workflow in AWS SageMaker using Data Wrangler for preprocessing. I'm loading data from various tables in a Redshift instance, before mutating and joining them as required ...
6
votes
3
answers
14k
views
How can I bulk upload JSON records to AWS OpenSearch index using a python client library?
I have a sufficiently large dataset that I would like to bulk index the JSON objects in AWS OpenSearch.
I cannot see how to achieve this using any of: boto3, awswrangler, opensearch-py, elasticsearch, ...
2
votes
1
answer
4k
views
How do I use awswrangler to read only the first few N rows of a parquet file stored in S3?
I am trying to use awswrangler to read into a pandas dataframe an arbitrarily-large parquet file stored in S3, but limiting my query to the first N rows due to the file's size (and my poor bandwidth).
...
0
votes
1
answer
486
views
Unable to find AWS Wrangler 2.10.0 Layer in US-WEST-2 under public-artifacts
Please consider the example shown in the below link:
https://github.com/awslabs/aws-data-wrangler/issues/923
I am trying to get the public artifacts bucket wrangler to US-West-2, but I am unable to ...
2
votes
1
answer
2k
views
Is there any way to capture the input file name of multiple parquet files read in with a wildcard in pandas/awswrangler?
This is the exact python analogue of the following Spark question:
Is there any way to capture the input file name of multiple parquet files read in with a wildcard in Spark?
I am reading in a ...
0
votes
1
answer
4k
views
AWS Wrangler Error HIVE_METASTORE_ERROR: Table is missing storage descriptor
hope you can help me with a concern about an error with awswrangler.
this is the case: i have 2 aws accounts, AccountA and AccountB, both with lakeformation enabled, i have a set of databases in AccA ...
0
votes
3
answers
3k
views
How to read sheet names of excel sheet from S3 in AWS Wrangler?
I have an excel sheet which is placed in S3 and I want to read sheet names of excel sheet.
I have read excel sheet with aws wrangler using awswrangler.s3.read_excel(path)
How can I read sheetnames ...
1
vote
0
answers
233
views
AWS Data Wrangler chunking: "Length mismatch: Expected axis has 28680 elements, new values have 100000 elements"
While loading the parquet file and getting the below error.
My parquet file contains 5,128,680 rows and it is loading 5,100,000 only and not loading 28,680 records and code as like below:
Error :
...
0
votes
1
answer
347
views
Automate the date parameter while deplying the model on AWS Wrangler
I have built a XGBoost model on my local machine which takes a training data and validates the model on a testing dataset. However, I have hard-coded the date values as the training data is created ...
1
vote
0
answers
527
views
Read data from AWS S3 using pyspark and python. (Read All columns: Partioned column also)
I have saved the spark dataframe to AWS S3 in Parquet format partitionby column "channel_name".
Below code is how i saved to S3:
df.write.option("header",True) \
....
0
votes
1
answer
373
views
Split SQL Where IN Clause When List is to big into Smaller Requests in Python
I have setup an AWS Lambda function with python to ingest requests from a CSV and then query an AWS Serverless Aurora PostgreSQL database based on this request. The function works when the requests ...
0
votes
1
answer
584
views
AWS Maximum BadRequestException retries reached for query Using Data API to Query RDS Serveless Aurora
I have created a Lambda function that uses awswrangler data api to read in data from an RDS Serverless Aurora PostgreSQL Database from a query. The query contains a conditional that is a list of IDs. ...
0
votes
1
answer
724
views
Sagemaker AWSWrangler>2.3.0
As I am trying to use the function read_excel as part of AWS Wrangler, available as of version 2.3.0 in Sagemaker Jupyter Lab on Amazon Web Services, it does not install properly. The Python version ...