Questions tagged [boto3]
Boto 3 - The Amazon Web Services (AWS) SDK for Python
7,647
questions
0
votes
0
answers
8
views
Unable to run start_thing_registration_task method. Getting the error message: Status Code: 400; Error Code: ResourceRegistrationFailureException
I am getting the following error message. My python code runs till completion but no "Things" gets created. I extracted the error message from the list_thing_registration_task_reports method....
0
votes
1
answer
20
views
Failed to create folder in S3 bucket using boto3
I am trying to create a folder/directory in S3 bucket using boto3 library. Here is my sample code.
import boto3
access_key = "REDACTED"
secret_key = "REDACTED"
endpoint = "...
0
votes
1
answer
10
views
How to pass containerOverrides to an ECS task when triggering as a target from eventbridge rule?
I want to write a python code that uses boto3 to assign targets to an eventbridge rule. The target should be an ECS task. Also when the eventbridge rule runs/triggers the ecs task it should override ...
0
votes
1
answer
12
views
boto3: ssm client: describe_instance_information: How to filter based on computername
I have a list of managed nodes in the fleet manager.
I want to filter the managed nodes using ComputerName field, but in the documentation the filters dont have that
ssm_client = boto3.client('...
0
votes
1
answer
19
views
Boto 3 Redshift client execute_statement lagging
I'm trying to copy some data over from MySQL to RS. The table is 650,000-ish rows, so i'm doing it in batches of 10,000 or so and also using python's multiprocessing module. The first few iterations ...
0
votes
0
answers
9
views
How to setup google identity provider with fastapi using cognito, I have done required setup with aws and google developer console
It's really confusing to use google identity data to create user at the AWS cognito end. The process of using cognito boto3 client for google identity is confusing in a sense. There is no clear ...
0
votes
0
answers
20
views
configparser Not Recognizing Sections in config.ini
I'm trying to use the configparser module in Python to read from a config.ini file. Despite having a [example_section] in my config.ini file, when I run my script, it keeps printing that the section ...
0
votes
1
answer
41
views
Python patch object, how to assert on calls
I'm trying to create some pytest unit tests on a custom ansible module. I'm trying to mock the boto3 client and just test the logic of the class.
I moved the initialization of boto into a class method ...
0
votes
2
answers
31
views
Unable to retrieve names of all folders in s3 bucket using boto3
I am using list_objects_v2 to get list of folders in s3 bucket
The below function is being used to get folder names
def get_folders(source_bucket):
pages = paginator.paginate(Bucket=source_bucket)
...
0
votes
1
answer
23
views
List bucket files in S3 files permission denied. But upload works
I'm making a simple script in Python to upload and list the files in a S3 bucket, the problem is that I can upload the files, but I receive a Permission Denied error when I try to list the files.
This ...
0
votes
0
answers
11
views
boto3 - Get "Full load throughput rows target" of a DMS Database Migration task
I am building a python script that monitors all DMS Database Migration tasks in an AWS account and checks if there is any task whose Full load throughput rows target value has been zero for certain ...
0
votes
0
answers
27
views
Getting error in serverless deploy - AWS Lambda Python [duplicate]
I'm getting below error while running the command "serverless deploy"
SyntaxError: Unexpected token '.'
node_modules\@aws-sdk\client-acm\dist-cjs\ACMClient.js:28
const ...
0
votes
1
answer
18
views
DynamoDb Query and Append item to nested list inside a list, Python Boto3
I have this kind of entity in a dynamodb table:
{
"fieldAA":152,
"Big_List":[
{
"fieldA":"value",
"Nested_List":[
...
1
vote
1
answer
25
views
how to set file type in upload_fileObj() function in boto3 for blackblaze
def getB2Resource():
b2_config = Config(
region_name = 'region_name...',
signature_version = 'v4',
retries = {
'max_attempts': ...,
'mode': '......'
...
1
vote
2
answers
44
views
Query DynamoDB table API response
I am querying/scanning the DynamoDB table and getting the expected response. The response contains data type as well in dict Key :{'S':'String'} , how can I get a response just in Key : Value format i....
-2
votes
1
answer
47
views
How to start a server on a remote machine? [duplicate]
I'm trying to set up a cluster of EC2 instances that I can use for "big data" modeling. I'm trying to stay close to the metal and would prefer to not use existing frameworks (e.g. dask, ...
0
votes
0
answers
29
views
boto3 Request Entity Too Large when uploading file with size 6MB, even if multipart_thershold is much bigger than 6MB
I'm trying to save file to s3-compatible storage (to be precise, ceph)
Here is the code:
import boto3
import requests
from boto3.s3.transfer import TransferConfig
s3 = boto3.client('s3', endpoint_url=...
-1
votes
0
answers
34
views
S3 file download through S3 Transfer manager
In python there is a class called S3Transfer in boto3.s3 where configurations can be given which supports multipart upload......configurations like the one below can be mentioned
boto3.s3.transfer....
0
votes
1
answer
34
views
Connection timeout to S3 from a python application running in ECS EC2
I have a python ECS task running via EC2 (not Fargate). In this task, I am setting up a boto3 client
cls._instance.client = boto3.client(
"s3",
region_name="eu-west-1",
)
...
0
votes
0
answers
21
views
Reuse create_client in aiobotocore for better performences
I am implementing an async Python code that uses aiobotocore to put objects in S3.
The code is something like that -
class Foo:
def __init__(self):
self._aiobotocore_session = get_session()...
-1
votes
1
answer
24
views
boto3 failing to upload file in subdirectory of existing bucket
<EDIT> Closing this, as uninstalling and reinstalling moto from scratch fixed the issue. </EDIT>
I am using boto3==1.28.45 with moto==1.3.14 running locally on port 5353.
This has worked ...
0
votes
1
answer
25
views
How can I query List of Dict using filterExpressions for DynamoDB
I am currently working on a project that involves using Amazon DynamoDB as the database for storing and retrieving data. In one of my DynamoDB tables, I have a field that is a List of Dict (Dictionary)...
0
votes
1
answer
13
views
How to fetch DynamoDB records with FilterExpression=Attr.exists via boto3?
I have the following code:
import boto3
from boto3.dynamodb.conditions import Attr
resource = boto3.resource("dynamodb")
table = resource.Table(MY_TABLE_NAME)
records = []
pagination = {}
...
0
votes
0
answers
23
views
Django storages adding “;amp” to staticfiles query string parameter only for Django-jet files resulting in unauthorized access
Some staticfiles links have wrong params, what i noticed that they are only related to "django-jet" package.
Normal Django staticfiles URL:
https://daal.nyc3.digitaloceanspaces.com/static/...
0
votes
0
answers
44
views
Why am I getting an error from this boto3 syntax?
I am attempting to write a tool in python to add or remove users from an EC2 security group on AWS. The tool is meant to enable/disable access to a server-side private system for specific people via ...
0
votes
1
answer
14
views
Cognito not running custom auth challenge
I have created in Cognito the following custom challenge triggers in Python. They are identical to the AWS ones but written in Python.
Define auth challenge:
def lambda_handler(event, context):
if ...
0
votes
1
answer
21
views
Setup boto3 to connect to custom host, similar to s3cmd
With s3cmd I am able to connect to a bucket with the following config file:
[default]
access_key = <VARIABLE>
secret_key = <VARIABLE>
host_base = <VARIABLE>
host_bucket = <...
0
votes
0
answers
12
views
How to filter using start_query in CloudWatch?
I'm writing a python script to query cloudwatch:
This query works fine:
query = '''
fields @timestamp, @message
'''
But this is not working:
query = '''
fields @timestamp, @message
| ...
0
votes
0
answers
25
views
Sagemaker AutoML not finding .gzip file
I have a file saved in an s3 bucket with the extension .gzip. I am then launching an AutoML job using boto3, but it does not find the file (I am certain it is in there). I get this error in the log of ...
0
votes
0
answers
42
views
ERROR: ALTER TABLE APPEND cannot run inside a multiple commands statement
I'm trying to iteratively pull data from MySQL and build a table in RedShift. But getting this error on APPEND
db = <some_db>
table_name = 'test'
table_name_full = f'{db}.{table_name}'
cols = ['...
0
votes
1
answer
33
views
DynamoDB append list to nested list with boto3
I have a DynamoDB object which has a list with lists:
options: [
['fish', 'cat', 'dog', 'cow']
]
I want to append another list to it so it looks like this:
options: [
['fish', 'cat', 'dog', 'cow']...
1
vote
0
answers
37
views
How to properly mock functions inside FastAPI endpoint handler
I'm having trouble mocking a function that resides inside a FastAPI endpoint handler. No matter what I've tried it doesn't seem to work. A brief explanation of what I'm trying to test. I have a ...
0
votes
1
answer
14
views
Trying to connect boto3 to linode object storage
I cannot seem to connect to Linode Object Storage with boto3. It seems like this is something that is supported, but I cannot find the right syntax. I have tried adding http/https to endpoint_url and ...
-2
votes
2
answers
29
views
Bucket name validation fails in Boto3 in a Junyper notebook
Been running into an error with my S3 bucket and folder using Boto3 to download a file into a Junyper notebook. This is the code that generates the error, which I'm assuming I'm using the wrong ...
-2
votes
0
answers
18
views
Unable to start EC2 in sequential order using lambda,python
Unable to start EC2 instances in sequential order [x, y, z] also have to make sure each instance is in running state before starting the other instance also skip if the instance is already running.
...
0
votes
1
answer
26
views
Amazon SES creds working with smtplib but not with boto3
I'm trying to send emails using Amazon SES. Created the roles and API keys and policies and tested it out using Python smtplib. Pasting the snippet below:
import smtplib
fromaddr = "some@email....
-1
votes
0
answers
13
views
How to create AWS s3 bucket with specific capacity (size limit) using python boto3 client
I'm using boto3 sdk for python and I have created s3 client
session = boto3.session.Session()
s3_client = session.client('s3',
endpoint_url=self....
1
vote
0
answers
36
views
AWS Transfer How To See In-Flight File Transfers From SFTP Connector?
I have an AWS Lambda function configured to call start_file_transfer on an AWS Transfer Family SFTP Connector at X minute intervals. The problem I'm having is that sometimes the Lambda runs again ...
0
votes
0
answers
17
views
Passing metadata of trigger to ECS task -fargate- from S3 via Eventbridge
i am receiving bucket name and key as an empty string which causes script to fail.
I am trying to retrieve filename which caused the trigger by in eventbridge from S3 but unable to get it.
aws events ...
0
votes
0
answers
18
views
Error while running a DAG task from Airflow CLI
ERROR - Unable to retrieve connection from secrets backend (MetastoreBackend). Checking subsequent secrets backend.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-...
0
votes
1
answer
16
views
How to put_parameter where value is type file using boto3?
Using the aws-cli, I can do this:
aws ssm put-parameter --name my-param --value file://my-file.txt --type String
How can I do the same in boto3 where the value is of type file?
0
votes
0
answers
58
views
error as - "cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" on AWS Lambda using a layer
for python 3.9,imported libraries "boto3_version": "1.26.90", "botocore_version": "1.29.90", "urllib3": "1.26.11". Fetch data from public ...
0
votes
1
answer
47
views
DynamoDB Query with partition key and sort key giving me Query condition missed key schema element error
I'm using boto3 in Python to make queries to a DynamoDB. I'm using a Local Secondary Index to query my results sorting by an attribute (start_date). However, when connecting two Key( ) elements with &...
-1
votes
0
answers
63
views
AccessDenied with boto3 operations in Lambda and s3
I'm developing in terraform a AWS Lambda that is getting triggered by an object in s3 and then Im posting parsed data to api gateway.
Im having a problem with:
{
"errorMessage": "An ...
0
votes
1
answer
70
views
Cognito: Using admin_user_global_sign_out in pre_authentication trigger sometimes fail due to apparent async behavior
I'm trying to sing-out all old sessions from Cognito for a specific user credentials (revoke the Refresh tokens) when a new session is started using the same user credentials (in this case e-mail and ...
-1
votes
0
answers
21
views
An error occurred: An error occurred when calling the SendCommand operation: Instances [[<instance-ID>]] not in a valid state for account
An error occurred: An error occurred (InvalidInstanceId) when calling the SendCommand operation: Instances [[]] not in a valid state for account 309178984042.
def pass_command(imageId, user_command, ...
-1
votes
0
answers
21
views
Best practices to use IAM user credentials in an ECS task
I have an app (written in Python) that runs a Fargate instance on the AWS Elastic Container Service (ECS). Credentials are injected on the container through an .env file located in S3 and the code ...
-1
votes
0
answers
11
views
ERROR KeyError : Records Error while moving files from one s3 bucket folder to another
I am new to lambda functions. I am trying to create lambda function which basically moves datafiles from one s3 bucket to Archive folder in the same bucket, which get triggered by cloud watch schedule ...
0
votes
1
answer
58
views
Automation to find EC2 instance with a public IP address
Goal: automation to find an ec2 instance information (aws ec2 describe-instances) in any project/region of an organization by public IP address (should be at most one because public IPs are unique).
...
3
votes
0
answers
33
views
AWS S3 Public (with restricted regional access) bucket access with Python?
I have a public bucket in S3 with a policy that only allows regional access from us-west-1 and us-west-2. I want to access this bucket anonymously in Python using boto3/s3fs and list all files in it, ...