Questions tagged [boto]
boto is an open-source Python Interface to Amazon Web Services
2,337
questions
-1
votes
0
answers
4
views
Return all items from dynamodb ordered by date boto
I am attempting to return all items from a dynamodb table with results ordered by date. The tables structure is:
ID
Date
Category
Email
Errors
IsRead
Message
With ID as the primary key and date ...
0
votes
1
answer
24
views
import S3 from s3 library
I installed s3 library on my python 3.8 environment using
pip install s3
which installed s3 version 3.0.0
I then try to run
from s3 import S3
But I would get an error of
Traceback (most recent call ...
0
votes
2
answers
60
views
Best way to read aws credentials file
In my python code I need to extract AWS credentials
AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID
which are stored in the plain text file as described here:
https://docs.aws.amazon.com/sdkref/latest/...
0
votes
1
answer
20
views
How to create a table in redshift cluster via boto3
We have a use case, where we have multiple cluster in our AWS account, in one of them we want to create a table with a schema
(
marketplaceId integer ,
cohort varchar(90) ,
segment ...
0
votes
0
answers
24
views
Load a .mov video in python from S3 via Lambda
I am trying to load a .mov video file into a Lambda python script for processing.
import json
import urllib.parse
import boto3
import os
print('Loading function')
s3 = boto3.client('s3')
def ...
0
votes
1
answer
50
views
Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials via shell script
When I run from CLI(terminal) it works, it finds the credentials via sso and passes on
EX:
2022-06-18 07:38:36,421 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-...
0
votes
0
answers
14
views
Boto s3 filter to get prefix smaller than a given prefix
I've an s3 storage which stores the files as such
2021/05/06/folder1/file.ext
2022/05/06/folder2/file.ext
2022/06/06/folder3/file.ext
is there a way to get all object paths which have prefix lesser ...
0
votes
1
answer
14
views
Python 2 boto: key attribute
I could have sworn I used this before and it was working fine. Not sure if my environment changed or I am forgetting something else.
import boto
b = "some_bucket"
key = "some_s3_key&...
0
votes
1
answer
30
views
How to create spot instance with boto3 with required instance attributes instead of instance type
I am able to create spot instance with below code
import boto3
import datetime, random, string, json
client = boto3.client('ec2')
response = client.request_spot_instances(
DryRun=False,
...
0
votes
1
answer
20
views
How to get ip address of spot instance created with boto3
I am able to create spot instance with boto3 with the following code
import boto3
import datetime, random, string
client = boto3.client('ec2')
response = client.request_spot_instances(
DryRun=...
1
vote
1
answer
68
views
How can I track the progress/status of an asynchronous AWS Lambda invocation?
I have an API which I use to trigger AWS Lambda jobs. Upon request, the API invokes an AWS Lambda job with InvocationType='Event'. Hereafter, I want to periodically poll if the AWS Lambda job has ...
0
votes
1
answer
19
views
AWS Python SDK - guarantee that EBS volume and EC instance are in the same availability zone
I am scripting the setup of AWS resources using the Python SDK (boto3). I can create ec2 instances with:
ec2_resource.create_instances( .... )
and a block storage volume with:
ec2_client....
0
votes
1
answer
68
views
AWS boto iterator returns bytes instead of strings: _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
We have a large .csv file on an S3 bucket. We want to read it into a dictionary for processing line by line. botocore.response.StreamingBody provides an iterator that you can get with iter_lines(). ...
0
votes
2
answers
87
views
list S3 objects till only first level
I am trying to list s3 obejcts like this:
for key in s3_client.list_objects(Bucket='bucketname')['Contents']:
logger.debug(key['Key'])
I just want to print the folder names or file names that are ...
0
votes
0
answers
42
views
generate zip folder and add to storage django
Hi guys I need to generate a zip file from a bunch of images and add it as a file to the database
This is the code to generate the file
def create_zip_folder(asin):
print('creating zip folder for ...
0
votes
2
answers
173
views
Problems whit import boto3 in a virtual env
I'm trying to import boto3 modeule in a virtual environemnt.
import os
import boto3
Although I'm having the following problem with import command:
AttributeError Traceback (...
0
votes
1
answer
85
views
ssl error with copying file in s3 server?
I tried to collect static files on the S3 Server for my Django project with the command :
python manage.py collectstatic
But It failed because of SSLError :
During handling of the above exception, ...
0
votes
1
answer
68
views
Can I use s3fs to perform "free data transfer" between AWS EC2 and S3?
I am looking to deploy a Python Flask app on an AWS EC2 (Ubuntu 20.04) instance. The app fetches data from an S3 bucket (in the same region as the EC2 instance) and performs some data processing.
I ...
0
votes
1
answer
122
views
Incrementing a counter in DynamoDB when value to be updated is in a map field
I have a lambda function that needs to retrieve an item from DynamoDB and update the counter of that item. But..
The DynamoDB table is structured as:
id: int
options: map
some_option: 0
...
1
vote
1
answer
132
views
Using python Download the latest file from s3 bucket inside folder not from inside folder --folder
I want to download the latest file from s3-bucket inside folder only. Actually inside folder there are multiple folders along with files. But i need to download only file of latest date and upload it ...
1
vote
0
answers
113
views
Conditionally create a task definition in ECS
I have a boto3 script, which updates(create new revision) of ECS task definition and update the ECS service with the new task definition.
import boto3
import json
client = boto3.client("ecs"...
1
vote
1
answer
141
views
Gsutil multiple gs_service_key_file configuration in .boto
I am using gsutil and have multiple storages and each storage has its own gs_service_key_file configuration file. I want to be able to interact with multiple storages at the same time, but in the ....
0
votes
1
answer
105
views
Session created, but not used by boto client
I have the following code, which allows me to list information for all resources in all of my accounts in ~/.aws/config:
#!/bin/bash
#Output file
OUTFILE="out.txt"
#########################...
0
votes
0
answers
60
views
subprocess.run produces fatal error: SSL validation failed
I am trying to run this in my quest to start using sagemaker/s3/aws (in windows sorry) :
import subprocess
import boto3
import os
s3_bucket_name = "dududu"
mlruns_direc = "C:\bla\...
0
votes
1
answer
234
views
Copy file from s3 subfolder in another subfolder in same bucket
I'd like to copy file from subfolder into another subfolder in same s3 bucket. I've read lots of questions in SO and I came finally with this code. It has an issue, when I run it it works, but it ...
0
votes
0
answers
84
views
Set `"fs.s3.create.allowFileNameEndsWithFolderSuffix": "true"` via `aws emr create-cluster --configuration`
Short Description
I'm attempting to figure out how to set fs.s3.create.allowFileNameEndsWithFolderSuffix on an EMR cluster that's built by an Airflow Operator. For greater this is a work-around for ...
0
votes
1
answer
61
views
Unable to upload multiple python dataframes to s3
I am trying to upload google play console reports to s3 using boto3. The code below works well when i try to print dataframes in loop, which means i am successfully getting the files i need.
from io ...
0
votes
0
answers
83
views
Use other decorators alongside moto decorator
I am trying to use the moto decorator @mock_sqs in a test that I also mark with the decorator @pytest.mark.asyncio. but when trying to do so the @mock_sqs decorators fails to create a mock SQS queue, ...
0
votes
0
answers
86
views
Dynamically create the inventory file in Ansible for AWS EC2 instances
I am using an ansible playbook to create x amount of AWS EC2 instances in two separate groups. I wish to create two groups of differing sizes so I ask the user how many instances should be in each ...
0
votes
0
answers
29
views
Botocore firewall ports
I have a docker image running on EC2 instance that tries to issue an AWS command when I get this error
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://ec2.us-...
0
votes
0
answers
69
views
convert a .gz file from S3 to a .zip file using lambda
I need to convert a .gz file from S3 to a .zip file using boto3 python in an AWS lambda function. How to do this?
1
vote
1
answer
560
views
How to upload the folder to digital ocean spaces?
From the link, https://www.digitalocean.com/community/questions/how-to-upload-an-object-to-digital-ocean-spaces-using-python-boto3-library. It only states to upload files to the spaces.
I want to ...
0
votes
1
answer
31
views
boto not picking up ~/.aws/config in ansible amazon.aws.ec2 module
I need to assume a role from ec2 (another role) (first account) and then launch instances in the assumed role account. (2nd account)
My ~/.aws/config file:
[default]
role_arn = arn:aws:iam::123:role/...
-1
votes
2
answers
806
views
catch errors if s3 file is not downloaded
I download a file from S3 like this:
s3 = boto3.client('s3')
s3.download_file('testunzipping','DataPump_10000838.zip','/tmp/DataPump_10000838.zip')
For now it always works. However, I wanted to add ...
2
votes
1
answer
698
views
find last modified date of a particular file in S3
According to this answer: https://stackoverflow.com/a/9688496/13067694
>>> import boto
>>> s3 = boto.connect_s3()
>>> bucket = s3.lookup('mybucket')
>>> for key in ...
0
votes
2
answers
234
views
GzipFile not supported by S3?
I am trying to iterate through some file paths so that I gzip each file individually. Each item in the testList contains strings (paths) like this: /tmp/File.
After gzipping them, I want to upload ...
1
vote
1
answer
194
views
how to specify ContentType for S3 files?
I am trying to convert files to gzip files and then upload them to S3. When I check in S3, the files are there but they don't have a type specified. How can I specify the content type?
for i in ...
1
vote
1
answer
1k
views
S3 object has no attribute Bucket
I am trying to run a code snippet like this:
s3_file_path = "testunzipping/sample.csv.gz"
s3 = boto3.client('s3')
lst = s3.list_objects(Bucket='testunzipping')['Contents']
firstbucket = s3....
0
votes
0
answers
141
views
botocore ClientError occurs while collecting static files in django application
I am using AWS S3 bucket for my projects to serve staticfiles and mediafiles. I have used S3 bucket for my previous django projects, it works fine. But for my current project, it raises the following ...
0
votes
0
answers
142
views
Filter Boto3 Object Collection by depth
I am trying to list all folders at given depth level in an s3 bucket.
I can easily filter by Prefix using objects.filter.
Is there a way to limit to a fixed level of depth?
For now I am counting the ...
0
votes
1
answer
45
views
Using boto3 to retrieve a SG rule Description
We are trying to use boto3 to query our AWS instances and get all security groups which have rules with no description. I have not been able to figure out how to retrieve the Rule description. I ...
1
vote
1
answer
33
views
boto3 print rules from bucket encryption
Im writing a python script to retrieve information of AWS and im trying to get only the SSEAlgorith but i get TypeError: list indices must be integers or slices, not str
Is there any way to do this? I ...
0
votes
0
answers
25
views
AWS presigned URL get_object downloads corrput png
I am using the boto3 presigned_url for downloading files from a bucket, everything inside is downloaded fine (.csv, .pdf etc) except the .png file which is corrupted when downloaded:
snippet:
def ...
0
votes
0
answers
64
views
How do I set a tag with AWS boto3 "request_spot_fleet", unique to that spot fleet, when using a launch template?
I'm using AWS boto3 "request_spot_fleet" to launch spot fleets. I have a launch template that contains the information for the spot fleet and its instances.
The launch template contains tags ...
1
vote
0
answers
255
views
EMR Cluster: AutoScaling Policy For Instance Group Could Not Attach And Failed
I am trying to automate the EMR cluster creation through boto3. Unfortunately, I'm getting the following warning:
The Auto Scaling policy for instance group ig-MI0ANZ0C3WNN in Amazon EMR cluster j-...
0
votes
0
answers
55
views
Get Count of Spot Instances
we can use get_cost_and_usage to know the cost spent of spot instance for a particular period of time.
data = self.ce_client.get_cost_and_usage(
TimePeriod={'Start': start, ...
0
votes
0
answers
63
views
Scrapy complains about missing botocore library when uploading data to S3 bucket
I have already installed botocore library
>> pip freeze
>>boto==2.49.0
boto3==1.18.42
botocore==1.21.42
But when I tried to update data to s3, I still got this error message:
2021-09-16 ...
1
vote
1
answer
260
views
Boto3 wait_until_exists for available image (object has not attribute)
I'm creating an AMI and I want to wait until the AMI is 'available'. I'm using the following code:
import json
import boto3
import os
import logging
ec2 = boto3.client('ec2')
images = ec2....
0
votes
0
answers
54
views
Passing parameters to fixture function
I have two functions (x and y) and one common fixture function (f). Fixture function creates a database. When I am calling fixture function in x , I want database name as XB. Similarly in y I want ...