Questions tagged [boto]
boto is an open-source Python Interface to Amazon Web Services
2,361
questions
0
votes
1
answer
25
views
get-log-events returns only a few results of many
I am logging to CloudWatch from a python lambda. In the AWS console I can see several hundred log entries, but if I try to use get-log-events from either a Python Boto call, or through the AWS CLI I ...
1
vote
2
answers
56
views
AWS DynamoDB Delete Item
I'm working on deleting DynamoDB item through Lambda.
I have a simple table setup and Partition key is Date (Number) and I don't have sort key.
I tested below command in AWS CLI (via CMD) and it is ...
0
votes
0
answers
85
views
S3FS fails when given aiobotocore session to initialize the File System
I am trying to use a combination of aiobotocore, botocore and s3fs to build an S3 client which refreshes its credentials automatically.
This is my code so far:
Code to create a refreshable session. It ...
0
votes
1
answer
35
views
How to get verification code/change password code in boto3?
I'm writing tests in Moto and Pytest for a confirm sign up route that receives a verification code and a email and uses boto method confirm_sign_up to confirm user.
My question is: After I created a ...
0
votes
1
answer
80
views
Cannot SSH on to AWS EC2 instance created using Boto
I have been creating AWS EC2 instances using Ubuntu and Boto for months. They start-up, I issue commands from Python etc. This has been working fine.
I create the instances using this:
instances = ec2....
1
vote
0
answers
141
views
How to get all error logs from aws cloudwatch using boto3
I am trying to get all the error logs from aws cloudwatch, to do this I tried looping through all log group names and running this code:
client.start_query(
logGroupName=group, # log group name
...
1
vote
1
answer
501
views
S3:HeadObject returns 403 on MinIO instance proxied by Cloudflare
When sending HeadObject request to MinIO instance proxied by Cloudflare, 403 is returned instead of 404 (for non-existent files). GET, PUT and DELETE operations can be performed with no issue.
For ...
0
votes
0
answers
80
views
DigitalOcean Presigned Post URL not getting the ACL as public-read
I'm trying to upload images to a Digital Ocean space from my application. These images are public and I am able to upload the images but they are getting uploaded as private
I have tried adding x-amz-...
0
votes
0
answers
76
views
Which boto or boto3 version for sts.get_session_token?
Which boto3 version do you prefer for get session token?
I got error while running the python code, my opinion of this kind error usually because wrong version of boto3 function
Here's the code
import ...
0
votes
2
answers
73
views
boto equivalent of aws client command
I have this command that works as expected.
But I will like to use boto instead.
aws cloudwatch get-metric-statistics \
--region ap-south-1 \
--namespace AWS/RDS \
--metric-name DBLoad \
...
0
votes
0
answers
169
views
Boto 2.49.0, support for TLS 1.2?
I have an old application that has a very old version of boto2 (2.6.0) that doesn't have the support for TLS 1.2
So instead of the costy re-writing of all the application in python3, using boto3, I ...
-1
votes
1
answer
54
views
Grant s3 user access in bucket level policy
I have minio with bucket named "bucket" and IAM user named "user1"
I'll try to grant access to this bucket with Bucket Level Policy
client = boto3.client('s3', endpoint_url='...
0
votes
0
answers
45
views
The XML you provided was not well-formed or did not validate against our published schema
I am trying to migrate some of the legacy code from python2 to python3, I am done with most of the part, the below functions are throwing some errors.
Here is the code :
def upload_part(
mp,
...
0
votes
0
answers
11
views
email notification if file is not uploaded to S3 Bucket
I am uploading file to S3 bucket thru Boto3.
Need help on Python code for email notification if a file is not uploaded to S3 bucket
Could any one share the python code please
Thanks
xxxx
0
votes
0
answers
75
views
Error when using pandas .to_csv function to upload to S3. "Anonymous access is forbidden for this operation"
I'm getting and error "Anonymous access is forbidden for this operation" when trying to use the df.to_csv function.
df.to_csv("s3://testdatateam1/test6.csv",
...
0
votes
1
answer
74
views
Download and upload data from AWS S3 using python [duplicate]
I would like to upload and download data from AWS S3 bucket using python code.
The files are present under a folder in AWS S3 location.
Example :
Bucket - aws-xxx-use1-00-d-s3b-xxx-yyy-xxx
Folder - ...
0
votes
1
answer
697
views
AWS Boto3 sts get_caller_identity - catching exceptions if credentials are invalid
With a Python app, using Boto3 v1.26.59 (and botocore of same version) about the first thing done is to try to get the username of the user. We have Identity Center (SSO) users. With aged credentials (...
0
votes
0
answers
21
views
Error in AWS API requesting for MTurk workers with a certain qualification type
I am trying to retrieve workers with a certain qualification type by using the following code.
response = mturk_client.list_workers_with_qualification_type(
QualificationTypeId=args....
1
vote
0
answers
268
views
Multithreading in AWS boto pagination Python not working
I tried multithreading with paginator but instead of running on multiple thread, it is running on a single thread.
response: <class 'botocore.paginate.PageIterator'>
page.get('Contents') return ...
0
votes
0
answers
54
views
how to find the creator of the IAM access key in aws?
I want to create a lambda function(python) that will list the newly created access keys and who created it.
I tried to find the access key creation using cloud trail. but couldn't find the event name.
0
votes
1
answer
530
views
AttributeError: 'dict' object has no attribute 'Instance' python
i am trying to start the EC2 Instance with PrivateIP where i provide as argument "PrivateIP" and below code will describe the ec2 and get InstanceId. Once InstanceId received, it starts the ...
0
votes
1
answer
508
views
S3 tagging and access control policies not working for limiting the tags keyset on an object
Trying to restrict tags to only a given set of keys that can be attached to the objects. Using bucket level policies to define this condition. However, the logic is not working. Bucket policy (https://...
0
votes
0
answers
182
views
Boto3 (SignatureDoesNotMatch) when calling the GetObject operation: Unknown
I'm uploading and delivering my files to an Object Storage (using Contabo as provider). Uploading works.
The problem is that I can't figure out how to A: access files without setting them to "...
0
votes
0
answers
16
views
Skipped ec2 installation error:fatal: [localhost]: FAILED! => {"changed": false, "msg": "boto required for this module"} [duplicate]
I installed boto. But when I run playbook it's skipped ec2 server creation and asking boto required.
[ec2-user@ip-172-31-11-186 ~]$ cat playbook.yml
---
- hosts: localhost
remote_user: ec2-user
...
0
votes
1
answer
38
views
Reading Reponses SMS using SNS AWS
I'm using AWS SNS service and I'm able to sent sms using that service, I was wondering if there is any way that I can read responses for my SNS SMS sent?
Appreciate the help!
1
vote
1
answer
85
views
API for creating behaviour in cloudfront
I have created a CloudFront Distribution through the console , Am trying to create a origin and behaviour in that distribution using boto3
I cant find any api to create origin and behaviour in ...
0
votes
1
answer
629
views
trying to use boto copy to s3 unless file exists
in my code below,
fn2 is the local file and "my_bucket_object.key" is a list of files in my s3 bucket.
I am looking at my local files, taking the latest one by creation date and then looking ...
1
vote
0
answers
105
views
Return filename of specific pattern which was last modified from s3 bucket
I am writing a python script where I have to fetch just the filename (Eg. 'abc.csv') from a directory in a S3 Bucket which was last modified. I have managed to fetch the filename from the bucket but ...
0
votes
1
answer
510
views
Cannot upload the files into AWS S3 bucket with ACL pubic-read using python
I am trying to upload the file into s3 bucket but I am getting access denied error when I am trying to upload the files with ACL public-read
It is working fine with ACL private
class S3:
def ...
0
votes
0
answers
36
views
pip show boto3 - missing classifiers
I am comparing boto3 of two Amazonlinux instances(server1 and server 2). I can see classifiers are missing in server 2 .
Server 1:
[root@xxxxx]: pip show boto3
---
Metadata-Version: 2.0
Name: ...
1
vote
1
answer
1k
views
Find the cost of each resource in AWS
I have a python script to get the cost of each service for the last month, but I want the cost for each resource using the tag mentioned for that resource. For example, I have got the cost for RDS ...
2
votes
0
answers
230
views
S3 boto - list objects in parallel
I want to quickly (in parallel?) list directory/prefix with 100K+ objects.
The pagination works (see below) but I'm unable to get NextKeyMarker until I download the last page - it breaks the idea of ...
0
votes
1
answer
523
views
Python : XML file downloaded from S3 full of string escaping characters
I have a number of XML files that I have added to S3 (localstack sever). I can view these files through Cyberduck and they are valid xml files. However, when I download the objects, the XML data is ...
-1
votes
1
answer
3k
views
How I can upload file to folder in bucket in S3 using Python?
I have local S3 and want to upload some files into folders in mu bucket.
For example:
my_bucket
- folder1
- folder2
I have to upload file file1.json to folder1 and file1.json to folder1
I try do ...
0
votes
1
answer
273
views
Return empty columns in athena boto response object
When executing a select query against an athena table via boto3, the response object given is in the syntax:
{
"UpdateCount":0,
"ResultSet":{
"Rows":[
...
1
vote
0
answers
183
views
AWS boto3 Quicksight "AllowedDomains" parameter non-existent
my problem is that im calling the boto3 quicksight API with this function call:
response = quickSight.generate_embed_url_for_registered_user(
AllowedDomains=[
'...
1
vote
0
answers
71
views
Can I read a trained linear model from s3, without reconstructing a local copy?
In order to run a dask pipeline on a coiled cluster that uses a previously trained linear model in each task, I believe I need to read the model directly from S3. Reading the model within a task did ...
0
votes
1
answer
56
views
run 2 instructions sequentially and execute the second instruction based on a condition in Python Lambda
I have a requirement to execute 2 tasks sequentially in a Lambda Function in Python.
But as the execution is asynchronous I have to check the status of the first task regularly and run the second task ...
0
votes
1
answer
342
views
Differentiate between website endpoint from REST API endpoint for AWS S3
I have an input provided by a user, that would be used as the endpoint url for bucket operations for an S3 bucket.
Is there a way to differentiate if the url is a REST API endpoint or a website ...
0
votes
1
answer
598
views
How do you effectively move and partition files in s3 using boto3?
There are around 10k files in an s3 location which got exported from dynamodb PITR export to s3 option. These files aren't partitioned in any way and it is within a single folder which is a problem ...
0
votes
1
answer
242
views
Return all items from dynamodb ordered by date boto
I am attempting to return all items from a dynamodb table with results ordered by date. The tables structure is:
ID
Date
Category
Email
Errors
IsRead
Message
With ID as the primary key and date ...
0
votes
1
answer
287
views
import S3 from s3 library
I installed s3 library on my python 3.8 environment using
pip install s3
which installed s3 version 3.0.0
I then try to run
from s3 import S3
But I would get an error of
Traceback (most recent call ...
6
votes
2
answers
6k
views
Best way to read aws credentials file
In my python code I need to extract AWS credentials
AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID
which are stored in the plain text file as described here:
https://docs.aws.amazon.com/sdkref/latest/...
0
votes
1
answer
802
views
How to create a table in redshift cluster via boto3
We have a use case, where we have multiple cluster in our AWS account, in one of them we want to create a table with a schema
(
marketplaceId integer ,
cohort varchar(90) ,
segment ...
0
votes
0
answers
361
views
Load a .mov video in python from S3 via Lambda
I am trying to load a .mov video file into a Lambda python script for processing.
import json
import urllib.parse
import boto3
import os
print('Loading function')
s3 = boto3.client('s3')
def ...
0
votes
1
answer
441
views
Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials via shell script
When I run from CLI(terminal) it works, it finds the credentials via sso and passes on
EX:
2022-06-18 07:38:36,421 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-...
0
votes
0
answers
102
views
Boto s3 filter to get prefix smaller than a given prefix
I've an s3 storage which stores the files as such
2021/05/06/folder1/file.ext
2022/05/06/folder2/file.ext
2022/06/06/folder3/file.ext
is there a way to get all object paths which have prefix lesser ...
0
votes
1
answer
18
views
Python 2 boto: key attribute
I could have sworn I used this before and it was working fine. Not sure if my environment changed or I am forgetting something else.
import boto
b = "some_bucket"
key = "some_s3_key&...
0
votes
1
answer
333
views
How to create spot instance with boto3 with required instance attributes instead of instance type
I am able to create spot instance with below code
import boto3
import datetime, random, string, json
client = boto3.client('ec2')
response = client.request_spot_instances(
DryRun=False,
...
0
votes
1
answer
116
views
How to get ip address of spot instance created with boto3
I am able to create spot instance with boto3 with the following code
import boto3
import datetime, random, string
client = boto3.client('ec2')
response = client.request_spot_instances(
DryRun=...