31

I know how to download a file in this way: key.generate_url(3600)

But when I tried to upload: key.generate_url(3600, method='PUT'), the url didn't work. I was told: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I cannot find example code on the boto homepage for how to use the function generate_url(method='PUT'). Does anyone here know how to use it for uploading? How to set the params for the path of upload file?

2
  • Is that a new file? To create new file, you should use POST, not PUT
    – vartec
    Apr 6, 2012 at 13:56
  • @vartec: What do you mean "a new file"? In my use case, sometimes I need to upload new key to a certain bucket, sometimes I need to overwrite the old key. So I think I need the code example for 'PUT' and 'POST'. Apr 6, 2012 at 14:11

5 Answers 5

50

I found some time to experiment with this and here's what I found.

>>> import boto
>>> c =boto.connect_s3()
>>> fp = open('myfiletoupload.txt')
>>> content_length = len(fp.read())
>>> c.generate_url(300, 'PUT', 'test-1332789015', 'foobar', headers={'Content-Length': str(content_length)}, force_http=True)
'http://test-1332789015.s3.amazonaws.com/foobar?Signature=oUARG45mR95utXsiQYRJNiCI4x4%3D&Expires=1333731456&AWSAccessKeyId=AKIAJOTCCJRP4C3NSMYA&Content-Length=16'

I was then able to use curl to PUT the file to that url like this:

$ curl --request PUT --upload-file myfiletoupload.txt "http://test-1332789015.s3.amazonaws.com/foobar?Signature=oUARG45mR95utXsiQYRJNiCI4x4%3D&Expires=1333731456&AWSAccessKeyId=AKIAJOTCCJRP4C3NSMYA&Content-Length=16"

This resulted in the file being uploaded to the bucket. So, it appears that it is possible. You might want to see if you can calculate the content-md5 value and include that in the headers but then you also have to figure out how to get curl to send that header, as well. Also, you should be able to make this work over HTTPS rather than HTTP but I haven't tried that.

2
  • 2
    headers argument in this answer will not work now, it will cause 'signature mismatch` error when you upload. If you want to specify headers in the response when you GET the file later, you need to use response_headers and use the fields such as response-content-type etc.
    – MLister
    Jul 3, 2015 at 19:22
  • Tested on Google Cloud Storage but failed. Would you mind taking a look at this question? Thanks. stackoverflow.com/questions/38938203/…
    – Randy Tang
    Aug 14, 2016 at 7:38
37

Here's what it looks like in boto3(tested with version 1.2.3).

First, create a presigned url with s3.generate_presigned_url method:

>>> import boto3
>>> s3 = boto3.client('s3')
>>> s3.generate_presigned_url('put_object', Params={'Bucket':'YourBucket', 'Key':'YourKey'}, ExpiresIn=3600, HttpMethod='PUT')
u'https://s3-ap-northeast-1.amazonaws.com/YourBucket/YourKey?AWSAccessKeyId=AKIAXXXXXXXXXXXXXXXX&Expires=1451061671&Signature=%2FtyAyCd5vrp13p%2FqLdoPkox7yTM%3D'

PUT to S3 with a presigned URL

$ curl \
  --request PUT \
  --upload-file path/to/file \
  "https://s3-ap-northeast-1.amazonaws.com/YourBucket/YourKey?AWSAccessKeyId=AKIAXXXXXXXXXXXXXXXX&Expires=1451061671&Signature=%2FtyAyCd5vrp13p%2FqLdoPkox7yTM%3D"
4
  • how do you add your credentials i.e. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY?
    – CpILL
    Feb 11, 2016 at 10:25
  • 1
    @CpILL You can pass credentials via environment variables. e.g., $ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE docs.aws.amazon.com/cli/latest/userguide/…
    – quiver
    Feb 12, 2016 at 13:57
  • 2
    Thanks for showing how to do in boto3. Currently migrating code from boto to boto3 and wish their docs were clearer on all the changes.
    – maryokhin
    Aug 14, 2016 at 1:08
  • I spent 21 hours on this problem until I finally saw this code. Thanks. The docs suck in a way that's unbelievable. Glad to find someone who's been through the same hell and went through the other side victorious.
    – JasonGenX
    Feb 8, 2021 at 19:25
19

All the other answers assume the file will be uploaded with curl, which is really not convenient in most python scripts. In the following, a pre-signed url is generated with boto3 and the file is uploaded with the requests library:

session = boto3.Session(aws_access_key_id="XXX", aws_secret_access_key="XXX")
s3client = session.client('s3')
url = s3client.generate_presigned_url('put_object', Params={'Bucket': 'mybucket', 'Key': 'mykey'})

requests.put(url, data=open("/path/to/file").read())
1
  • Other solutions scattered across SO use 'rb' and post. they didn't work for me but this one did
    – Kermit
    Nov 11, 2021 at 16:52
9

This is a follow up to garnaat's answer from Apr 6 '12.

I am generating a signed URL server side, where I have credentials, and I pass it to a client such that a client can directly upload content. I trust the client far enough to allow it to upload arbitrary sized files, but not enough to give it security tokens. I wanted to avoid having the client tell the server how large its content would be as part of the request. Hence my follow up answer.

I was able to get the signed url for the PUT method working without specifying content length in the headers or specifying force_http=True.

Using Boto 2.31.1: as in garnaat's answere:

>>> import boto
>>> c =boto.connect_s3()

then instead I used:

>>> temp_url = c.generate_url(seconds_available, 'PUT', bucket_name, s3_key)

this produced a url in the following form:

https://s3_location/bucket_name/s3_key?Signature=Ew407JMktSIcFln%2FZe00VroCmTU%3D&Expires=1405647669&AWSAccessKeyId=kM__pEQo2AEVd_Juz4Qq

I was then able to use curl to post a file:

>>> os.system('curl --request PUT --upload-file true_measure/test_files/test_file_w_content.txt "'+temp_url+'"')

I did have a very difficult time figuring this out because I usually use python requests to write tests and debug; however I get an authentication failure when I try to use it to put a file to one of these boto generated signed urls using requests. I haven't fully debugged this, but I suspect it is because requests is adding a few additional headers as compared to what curl produces.

I hope this follow up answer spares someone else the debugging pain I went through.

1
  • os.system is just a very bad way to interact with custom variables Sep 10, 2018 at 7:45
1

If you are using boto (not boto3), the only way I was able to get an upload to work was using generate_url_sigv4. Using vanilla generate_url caused the same error as reported in the original question. It is possible there is an AWS account setting I don't know about that controls which function works.

In a Python interpreter with boto 2.49.0 and requests 2.22.0:

import boto
import os
import requests
os.environ['S3_USE_SIGV4'] = 'True'
c = boto.connect_s3(host='s3.amazonaws.com')
url = c.generate_url_sigv4(3600, 'PUT', 'my-bucket-name', 'bucket-path/to/file.txt')
with open('file.txt') as f:
    resp = requests.put(url, data=f.read())

>>> resp
<Response [200]>

If you don't connect with a hostname, you will receive this error when generating the URL:

boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.

Related:
What does ''HmacAuthV1Handler' object has no attribute 'presign'' mean?

If you are using boto3, a presigned POST seems to be better-documented:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html#generating-a-presigned-url-to-upload-a-file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.