Questions tagged [amazon-s3]

Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.

amazon-s3
Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

Unable to upload to s3 with presigned URL

I'm trying to upload an object to s3 bucket through HTTP PUT with the presigned-url returned from S3.getSignedUrl(). However I received the error exception while calling s3 with unknown operation: ...
jndp's user avatar
  • 95
0 votes
0 answers
8 views

Why do I encounter no such file or directory in my s3 script

I tried running the script but encountering error import os import boto3 # Create an S3 client s3 = boto3.client('s3') # Set the bucket name bucket_name = 'heavyclick' # Function to upload a ...
Divine's user avatar
  • 1
0 votes
0 answers
16 views

How to redirect a request to an existing S3 object having the `s3.amazonaws.com` urls

I had created a bucket to store customer images on S3 quite some time ago. Our customer emails contain s3 urls of the format https://s3.amazonaws.com/<bucketName>/<fileName> so that the ...
Mohammed Suhail's user avatar
0 votes
0 answers
12 views

Cloudfront appears to be blocking POST request

I am running a static website on an S3 bucket. The website has a form component, which sends a POST request to a back-end running on an EC2 server. When I submit the form everything works as it should ...
user24474536's user avatar
0 votes
1 answer
9 views

Unable to launch remote sagemaker pipeline

I am trying to launch a sagemaker pipeline and running into an issue where the container cannot detect the py script that's being launched. Basic set up: A docker container thats been registered in ...
Cyrus Mohammadian's user avatar
0 votes
0 answers
34 views

What's the best way to upload large files to S3 Buckets

I'm using Vue js to create chunks for Laravel 11 backend and from there these chunks should be transferred to AWS S3 Buckets. I used an append for each new chunk, it was working but had plodding ...
Ahmed Alaa El-Din Hassan's user avatar
0 votes
0 answers
10 views

How do I set the path to s3 for imagedataloaders.from_df?

I want to load image data using ImageDataLoader, but I need to set the path to read from S3 folders. I'm struggling to find a solution. Does anyone have any ideas on how to accomplish this? data = ...
Hal's user avatar
  • 372
0 votes
0 answers
11 views

How to Implement Cross-Referencing between the content in Private S3 Bucket Using CloudFront and Signed Cookies

I have a web application set up with a private S3 bucket and CloudFront to deliver the content efficiently. Within the S3 bucket, I've organized static web pages into different projects, each ...
Alex leong's user avatar
0 votes
0 answers
16 views

aws s3 PreSignedUrl file upload status

I upload photos using PreSignedUrl without using the aws s3 upload ID. Is there a way to check the upload status?
HamTory's user avatar
  • 49
0 votes
1 answer
18 views

Amazon s3: Get last updated date of partition in Athena/AWS Glue Catalog

I am trying to find a way to obtain the last updated date and time for each partition. For example, if the 15-04-23 partition was created on the 15th of March and a new file was added to this ...
Nabeel Khan Ghauri's user avatar
-1 votes
0 answers
10 views

User should be able to specify URI of his S3 bucket for a file upload to my application

What would I like to achieve I have a web application that can process .avro files. In addition to uploading files from his own computer, the user should insert files from his S3 bucket into the ...
Fred's user avatar
  • 15
-2 votes
0 answers
29 views

How to restore a backup file (.bak) from S3 bucket to AWS RDS SQL Server DB using Shell Script or Power shell Script?

I need a bash or power shell script for the DB restores from S3 buckets to Amazon rds mssql db instance. However, I have found stored procedures to do it, but I must use bash or PowerShell script to ...
Ujjval Darji's user avatar
1 vote
1 answer
17 views

aws s3 PreSignedUrl used once and expired

Is there a way to use aws s3 PreSignedUrl only once and then expire PreSignedUrl? You can shorten the PreSignedUrl expiration time, but I think that is wrong.
HamTory's user avatar
  • 49
-2 votes
0 answers
27 views

determine whether a string contains regex or not [closed]

i am writing a method deleteFolderOrFiles it takes a configuration={ filePath:'path to the folder', fileName:'fileName' } in the filename we could pass regex as a string the problem here i am creating ...
mohit jassal's user avatar
0 votes
0 answers
21 views

Load data from S3 to Snowflake, overwrite vs using stream

I'm trying to load files from S3 to Snowflake using copy into commands. The pipeline is scheduled using Airflow. Each day there would be new files (e.g. source_20240501.csv). In order to rerun a ...
user3735871's user avatar
0 votes
0 answers
9 views

BigQuery, create table from partitioned data in an Amazon S3 bucket that is not in the supported hive style partitioned format

My data provider provides our data in a Hive partitioned format where the files (parquet files) are separated by year,month,date by folder s3://bucket/my_data/YYYY/MM/DD/file.parquet I have created ...
Shaina Mae Rosell's user avatar
0 votes
0 answers
9 views

How to unit test ` software. amazon. awssdk. transfer. s3.model. UploadRequest`

I have code that uses AWS SDK for Java. I call S3TransferManager.upload(UploadRequest) in one of my methods. In a test for this method, I attempt to capture the UploadRequest argument in a mocked ...
Trevor's user avatar
  • 13.2k
0 votes
1 answer
23 views

Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "s3_bucket" with prefix "env:/":

I'm trying to init the backend in Terraform this is the backend.tf terraform { backend "s3" { dynamodb_table = "cpf-tflock-dev" bucket = "s3_bucket" ...
Said Lamane's user avatar
0 votes
1 answer
21 views

Cannot run my lambda function AWS (boto core api call error 403)

I'm having problems testing my lambda function in AWS which just wants to create a thumbnail everytime you upload somee image to a bucket, then put it in another bucket as a thumbnail (official ...
Luis Fernando López Corrales's user avatar
0 votes
0 answers
14 views

CREATE_FAILED: IamRoleLambdaExecution

my task use case is Upload excel files from front end to s3 (done) Create sqs queue to process excels uploads + dlq Create producer and consumer lambda functions Add s3 upload event trigger to ...
AG Mohamed's user avatar
0 votes
0 answers
15 views

How can I make sure AWS Glue Crawler reads all lines in a S3 csv file

I have multiple csv files in an S3 bucket coming in daily, and I process them with Glue crawlers, so they end up in Glue Data Catalog. But given the size of the files(>50 GB each) I wonder if all ...
blue_pigeon's user avatar
0 votes
0 answers
18 views

How to upload Excel/CSV file to s3 using rest assured

I reads many docs that suggest me to use multipart, but I still got 403 status code. Here is my code: File file = new File(System.getProperty("user.dir") + File.separator + "src/main/...
Novaldy Fajarae's user avatar
0 votes
0 answers
41 views

Python Django ASGI - memory leak - UPDATED

EDIT 18.04.2024: It looks like the problem isn't connected with django-storages and images specifically, but requests generally. Why? I made uploading images client-side with pre-signed URLs for ...
fifdee's user avatar
  • 1
0 votes
1 answer
34 views

Create multiple AWS S3 uploadId

Can I create multiple uploadIds in advance? After uploading a photo through PreSignedUrl, I want to complete the uploaded file through CompleteMultipartUploadRequest.
HamTory's user avatar
  • 49
0 votes
0 answers
16 views

SST Typescript retrieve s3 bucket object names and metadata key pairs with filtering

I am new to SST, I tried to create an upload API + Lambda to upload files and set metadata and it worked. Now I am trying to create an API + Lambda to retrieve the list of files. I was able to ...
Amjad Shubbar's user avatar
0 votes
0 answers
19 views

Uploading image to AWS using s3 bucket

import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3"; import uniqid from "uniqid" export async function POST(req){ const data = await req.formData(); if(...
Rahul Mishra's user avatar
0 votes
0 answers
11 views

"Error: 'Destination bucket does not exist' when configuring Amazon S3 Inventory for S3 video source bucket"

I'm following an AWS tutorial on batch-transcoding videos https://docs.aws.amazon.com/AmazonS3/latest/userguide/tutorial-s3-batchops-lambda-mediaconvert-video.html), and I've encountered an issue ...
george's user avatar
  • 1
-1 votes
1 answer
13 views

How to serve files from S3 with an header / cookie / param / whatever (like a bearer token) control?

I have a bucket with public access on all files on it. I would like to restrict access to a file only to people who have the link to the concerned file plus a token. It can be a header, can be a ...
MathKimRobin's user avatar
  • 1,224
0 votes
0 answers
7 views

AWS XRay unable to sending tracing data to AWS XRay console

i installed xray daemon installed in local linux machine. created AWs acceskey and secrectkey configure my local. created role permission AWSXrayFullAccess using aws sdk my nodejs project below my ...
Venkatesh's user avatar
1 vote
0 answers
23 views

Access to S3 in root and child account through roles

We have a root AWS account and a child account for another application called prod. I want to give IAM users Full S3 Access to all the buckets in both the root account and the prod account. In order ...
ShwetaJ's user avatar
  • 509
-2 votes
0 answers
28 views

How can I host a nearly static website (with only 1-2 image uploads needed per month)? [closed]

As a computer science student, I decided to create a website for my friend using AWS for experience. Initially, the site was straightforward to develop because it consisted mainly of simple ...
user24380540's user avatar
0 votes
0 answers
10 views

Does firehose with MSK source and lambda transfomer support dynamic partitioning?

As per documentation I do not see mentioned anywhere that MSK does not support dynamic partitioning. However when I am creating firehose stream from console, I do not see enable dynamic partitioning ...
SWA PSTG's user avatar
-6 votes
0 answers
37 views

How to host many (> 1000s) of static sites using S3, Cloudfront, and ACM? [closed]

I want to host many (> 1000s) of sites with each having a custom domain and SSL. My first approach was one S3 bucket, one Cloudfront distribution, and one ACM certificate per user site. However, ...
Kameron Dawson's user avatar
0 votes
0 answers
22 views

duck db errors on decimal values in parquet

I'm new to using duck db as a quick way to nose around parquet files on S3, but it's problematic. I think the issue is with negative decimal values. It might be something else, but it's only DECIMAL(...
kmh's user avatar
  • 1,566
-1 votes
0 answers
21 views

Image component will not render valid s3 url aftering fetching and setting state in useEffect

I'm trying to render an image via Image component after retrieving the url from s3 (using aws amplify). The fetch and setting state is done in useEffect, and getUrl is implemented following example in ...
Michael Kim's user avatar
0 votes
0 answers
20 views

How read xlsx file starting on a certain line as a readable payload in mule 4 using anypoint studio?

I have an xlsx file that I need to read the file starting on line A15 into a readable payload in mule 4 using anypoint studio. The file resides in a repo location inside of aws S3 so i'm using the aws ...
ilya.fastiv's user avatar
0 votes
0 answers
19 views

CORS error getting resource made available via PreSignedUrl from AWS S3 storage

In my .net8 webapi I use the AWSSDK.S3 nuget to generate a pregisnedUrl for a resource previously put in an S3 bucket to get the resource. In the c# project I retrieve the url like this (put some ...
messerke's user avatar
0 votes
1 answer
14 views

Generate a Presigned S3 URL via AWS CLI (specifically for file UPLOAD)

Is it true you need to specify somehow put_object and method PUT when creating a presign URL. The official docs say you have to, but CLI doesn't support it https://docs.aws.amazon.com/code-library/...
Maks Hrech's user avatar
0 votes
0 answers
7 views

Upload file to S3 through AWS API Gateway REST API using serverless.yml

Is there any way to create AWS Api gateway REST API to create an endpoint and add a trigger to upload the file to S3 using serverless.yml template without using AWS lambda. I have gone through the ...
chvc's user avatar
  • 85
-1 votes
0 answers
17 views

Best practices for storing big sets of images [closed]

we have hundreds of Tb of images on S3 and we want to improve our S3 performance so we want to combine images in 1 file sets (they are already categorized in sets under s3 folders with sets name - but ...
Benny Semyonov's user avatar
-2 votes
1 answer
18 views

I need to download a file from s3 bucket I am using spring boot, where file name has %20 in it

I need it download a file from s3 bucket which has a name ayush%20inline.txt , when I generate presigned link it creates problem with encoding as %20 means space. I need a fix for this without ...
Ayush Sharma's user avatar
0 votes
0 answers
15 views

AWS Using cloudfront to forward PUT requests to S3 (including V4 signature)

I am uploading a file to an S3 bucket using PUT request and AWS V4 Signature, all working well and tested in Postman. The request looks like this... PUT https://example.com/to-process/test.json?X-Amz-...
DaveB's user avatar
  • 3,123
-1 votes
0 answers
21 views

What are the costs included for using AWS Billing data export functionality?

I want to use the AWS Billing data export feature, which exports and saves the billing cost for each service, in each region as per the granularity we configure. I wanted to know, is there any pricing ...
Jignesh M. Khatri's user avatar
0 votes
1 answer
26 views

How do i get the list of objects in S3 having Key Error and exceptions

I am new to this AWS boto3 concepts .The below code is giving a Key error and some exception that i am trying to execute can someone look to it and try to solve if any errors? import boto3 def ...
Scarface Jr's user avatar
-1 votes
2 answers
39 views

AWS EC2 Server not serving some pages and static files propperly

I just Deployed a website on AWS EC2 from github clone. When i visit to Influencer Marketing & Career pages the server serving the static files from S3 Bucket correctly. But when i am visiting my ...
Rahul Verma's user avatar
-1 votes
0 answers
18 views

Is it possible to access a private s3 bucket using temporary AWS credentials from react native application

I have react native expo app. I need to access an image from privet s3 bucket without using pre-signed url. I am able to get temporary credentials for AWS with the help of AWS Cognito identity pool. ...
ARUN CV's user avatar
  • 16
0 votes
1 answer
16 views

query_export_to_s3 split csv in multiple files

I am currently running the following query in AWS RDS Postgres to export data into an s3 bucket. The query works, but I was wondering if there is a way to export and split the csv file into multiple ...
paul590's user avatar
  • 1,405
1 vote
0 answers
45 views

Awswrangler raising an "ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs" on parquet read

Unable to read parquet file from AWS S3 using awswrangler (AWS SDK for Pandas) 3.7.2 as it raises an Arrow not implemented error. Example code: import awswrangler as wr df = wr.s3.read_parquet('...
Justin Boylan-Toomey's user avatar
-3 votes
0 answers
27 views

Why do I suddenly see Amazon Virtual Private Cloud in my AWS costs? [closed]

I'm looking at my AWS costs and I see this: Starting from Februrary 2024, I starting seeing Amazon Virtual Private Cloud that costs around 7$ a month. I didn't add any new service, so I have no idea ...
AG_HIHI's user avatar
  • 1,815
0 votes
0 answers
31 views

Copying files from AWS S3 to Glacier Flexible Retrieval in another S3

I'm looking to change some Python 3 code which currently copies an SSE-C encoded file from one S3 bucket to another. On the target S3 bucket, the file will be stored in Glacier Flexible Retrieval ...
Component 10's user avatar
  • 10.3k

1
2 3 4 5
1015