21

I'm trying to get an built that allows users to upload a file directly to my Amazon S3 bucket, from a NodeJS powered website. It seems the only tutorials out there, other than the actual amazon docs for this are all very out of date.

I've been following this tutorial, for the basic info, but again it's out dated. It doesn't have the method calls to crypto correct, as it tries to pass a raw JavaScript object to the update method, which throws an error because it's not a string or buffer.

I've also been looking at the source for the knox npm package. It doesn't have POST support built in - which I totally understand, because it's the browser doing the POST once it has the right fields. Knox does appear to have the right code to sign a policy, and I've tried to get my code working based on this... but again to no avail.

Here is what I've come up with, for code. It produces a base64 encoded policy, and it creates a signature... but it's the wrong signature according to Amazon, when I try to do a file upload.


var crypto = require("crypto");
var config = require("../../amazonConfig.json");

exports.createS3Policy = function(callback) {
  var date = new Date();

  var s3Policy = {
    "expiration": "2014-12-01T12:00:00.000Z",
    "conditions": [
      {"acl": "public-read"}, 
      ["content-length-range", 0, 2147483648],
      {"bucket": "signalleaf"}, 
      ["starts-with", "$Cache-Control", ""],
      ["starts-with", "$Content-Type", ""],
      ["starts-with", "$Content-Disposition", ""],
      ["starts-with", "$Content-Encoding", ""],
      ["starts-with", "$Expires", ""],
      ["starts-with", "$key", "/myfolder/"], 
      {"success_action_redirect": "http://example.com/uploadsuccess"},
    ]
  };

  var stringPolicy = JSON.stringify(s3Policy).toString("utf-8");
  var buffer = Buffer(stringPolicy, "utf-8");

  var encoded = buffer.toString("base64");
  var signature = crypto.createHmac("sha1", config.secretKey)
    .update(new Buffer(stringPolicy, "utf-8")).digest("base64");


  var s3Credentials = {
    s3PolicyBase64: encoded,
    s3Signature: signature
  };

  GLOBAL.s3creds = s3Credentials;

  callback(s3Credentials);
};

I'm obviously doing something wrong, here. But I have no idea what. Can anyone help identify what I'm doing wrong? Where my problem is? Does anyone have a working tutorial for how to generate a proper Amazon S3 Policy, with signature, from NodeJS v0.10.x, for a POST to the s3 REST api?

4
  • 1
    Uploading a file directly to S3 is not really a trivial task, especially if you want to support chunking, auto-resume, user metadata, etc, etc. The policy stuff can be quite complex. Consider using a library I maintain: Fine Uploader. It has native support for direct uploads to S3 in all browsers, even IE7. Chunking and auto-resume, among other features, are also supported. Furthermore, I wrote a node.js server-side example myself that, when paired with Fine Uploader S3, will handle all signatures for you. Aug 27, 2013 at 22:08
  • can you post this comment as an answer? i may end up using your library. still evaluating how it works, etc. Aug 28, 2013 at 1:05
  • 1
    I'm not sure that will go over well. It may be considered a poor or link-only answer, quite frankly. My understanding is that the community is looking for details answers that include code, and mine doesn't fit that description, which is why I posted it as a comment. If you do have any questions about Fine Uploader, have a look at the fine-uploader tag on SO though, where we handle support questions for the library. Aug 28, 2013 at 1:25
  • @RayNicholus I have been researching "browser direct to s3" uploads using AWS for a little bit now and arrived to this thread. The two methods I have tried are getSignedUrl() which is dead simple, and this form POST method which is much more complicated but a lot more configurable. I glanced over your lib and I noticed it uses the s3 REST API is that correct? Does this have anything to do with Cognito? Nov 2, 2016 at 16:09

4 Answers 4

36

Ok, I finally figured it out. After playing the random guessing game for a VERY long time, I thought to myself

"maybe i need to sign the base64 encoded policy" - me

and BAM that was it.

I also re-ordered the conditions to match how the form is posting, though I'm not sure this makes a difference.

var crypto = require("crypto");
var config = require("../../amazonConfig.json");

exports.createS3Policy = function(contentType, callback) {
  var date = new Date();

  var s3Policy = {
    "expiration": "2014-12-01T12:00:00.000Z", // hard coded for testing
    "conditions": [
      ["starts-with", "$key", "somefolder/"], 
      {"bucket": "my-bucket-name"}, 
      {"acl": "public-read"}, 
      ["starts-with", "$Content-Type", contentType],
      {"success_action_redirect": "http://example.com/uploadsuccess"},
    ]
  };

  // stringify and encode the policy
  var stringPolicy = JSON.stringify(s3Policy);
  var base64Policy = Buffer(stringPolicy, "utf-8").toString("base64");

  // sign the base64 encoded policy
  var signature = crypto.createHmac("sha1", config.secretKey)
    .update(new Buffer(base64Policy, "utf-8")).digest("base64");

  // build the results object
  var s3Credentials = {
    s3Policy: base64Policy,
    s3Signature: signature
  };

  // send it back
  callback(s3Credentials);
};

Hopefully this will help others that run in to the same problem.

4
  • THANK YOU! This code helped me out. Some quick comments: To format the date I used moment.js like so : moment.utc(expirationDate).format('YYYY-MM-DD')+'T'+moment.utc(expirationDate).format('HH:mm:ss.SSS')+'Z'. Also for buffers 'utf8' (note: no hyphen) is default encoding so I think "utf-8" is incorrect and extraneous. Feb 9, 2014 at 2:39
  • 4
    @Zugwalt, you could simplify that quite a bit with moment's built in formatting. moment.utc(expirationDate).toISOString()
    – Jonathan
    Feb 26, 2014 at 15:53
  • @Jonathan even better! Thanks! Feb 26, 2014 at 16:57
  • Not sure why this one is failing but I do get "The request signature we calculated does not match the signature you provided. Check your key and signing method." errors. So I tried using the example of @scabbiaza and that one did work. So I assume your code example is no longer working.
    – spa900
    Dec 6, 2020 at 17:38
19

I modified a bit previous example, because it didn't work for me: amazon returned an error about broken signature.

Here is how the signature should be created for Browser-Based Uploads Using POST (AWS Signature Version 4)

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

Calculating a Signature

var CryptoJS = require("crypto-js");

var accessKeyID = "PUT YOUR DATA";
var secretAccessKey = "PUT YOUR DATA";

var bucket = "PUT YOUR BUCKET NAME";
var region = "eu-central-1"; // overwrite with your region
var folder = "users/"; // overwrite with your folder
var expiration = "2015-09-28T12:00:00.000Z"; // overwrite date
var date = "20150927"; // overwrite date
var serviceName = "s3";


function getSignatureKey(key, dateStamp, regionName, serviceName) {
   var kDate = CryptoJS.HmacSHA256(dateStamp, "AWS4" + key);
   var kRegion = CryptoJS.HmacSHA256(regionName, kDate);
   var kService = CryptoJS.HmacSHA256(serviceName, kRegion);
   var kSigning = CryptoJS.HmacSHA256("aws4_request", kService);

   return kSigning;
}

var s3Policy = {"expiration": expiration,
  "conditions": [
   {"bucket": bucket},
   ["starts-with", "$key", folder],
   {"acl": "public-read"},
   ["starts-with", "$Content-Type", "image/"],
   {"x-amz-meta-uuid": "14365123651274"},
   ["starts-with", "$x-amz-meta-tag", ""],
   {"x-amz-credential": accessKeyID + "/" + date + "/" + region + "/" + serviceName +"/aws4_request"},
   {"x-amz-algorithm": "AWS4-HMAC-SHA256"},
   {"x-amz-date": date + "T000000Z" }
  ]
};

var base64Policy = new Buffer(JSON.stringify(s3Policy), "utf-8").toString("base64");
console.log('base64Policy:', base64Policy);

var signatureKey = getSignatureKey(secretAccessKey, date, region, serviceName);
var s3Signature = CryptoJS.HmacSHA256(base64Policy, signatureKey).toString(CryptoJS.enc.Hex);
console.log('s3Signature:', s3Signature);

Next generated base64Policy and s3Signature i used in the form for uploading. Example is here: http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html

Very important is to check that you have the same fields and values in the html form and in your policy.

0
3

I continued to have issues, so I worked through them and posted my solution here:

https://github.com/nikkwong/ng2-s3-uploader

In short, if you go with scabbiaza's answer in building the signature, make sure to build the form like so:

let formData = new FormData;
formData.append('acl', xAmzAcl);
formData.append('Content-Type', file.type);
formData.append('X-Amz-Date', xAmzDate);
formData.append('x-amz-server-side-encryption', xAmzServerSideEncryption);
formData.append('x-amz-meta-uuid', xAmzMetaUuid);
formData.append('X-Amz-Algorithm', xAmzAlgorithm);
formData.append('X-Amz-Credential', xAmzCredential);
formData.append('X-Amz-Signature', s3Signature);
formData.append('Policy', base64Policy);
formData.append('key', folder + '/' + file.name);
// File field must come last! 
formData.append('file', file);
0
1

AWS SDK now provides an easy way to create the POST policy with createPresignedPost().

Docs: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createPresignedPost-property

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.