19

I have a set of video files that were copied from one AWS Bucket from another account to my account in my own bucket.

I'm running into a problem now with all of the files where i am receiving Access Denied errors when I try to make all of the files public.

Specifically, I login to my AWS account, go into S3, drill down through the folder structures to locate one of the videos files.

When I look at this specificfile, the permissions tab on the files does not show any permissions assigned to anyone. No users, groups, or system permissions have been assigned.

At the bottom of the Permissions tab, I see a small box that says "Error: Access Denied". I can't change anything about the file. I can't add meta-data. I can't add a user to the file. I cannot make the file Public.

Is there a way i can gain control of these files so that I can make them public? There are over 15,000 files / around 60GBs of files. I'd like to avoid downloading and reuploading all of the files.

With some assistance and suggestions from the folks here I have tried the following. I made a new folder in my bucket called "media".

I tried this command:

aws s3 cp s3://mybucket/2014/09/17/thumb.jpg s3://mybucket/media --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=emailaddress=my_aws_account_email_address

I receive a fatal error 403 when calling the HeadObject operation: Forbidden.

5
  • Use AWS CLI to do that from command line : docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html Commented May 1, 2017 at 16:46
  • i see the --grant options, but what i am not sure about is what command to use? I dont want to copy, remove or sync it. I just want to apply the permissions to the file. I think sync would be what i want to do since that doesnt sound like it would move the file or remove it.
    – Robbiegod
    Commented May 1, 2017 at 16:56
  • @ShobhitPuri why do you think the OP will have permissions to do that from the CLI? You need to configure the CLI with AWS credentials. If you use the same credentials as the ones you're using to log in to the AWS console, chances are you will run into the same permission issues.
    – Viccari
    Commented May 1, 2017 at 17:00
  • 1
    I noticed the "owner" of the file is still from the account that i got these files from. How can i override that? Is it possible?
    – Robbiegod
    Commented May 1, 2017 at 17:08
  • @Viccari You are right. I focussed more on the 60 gb of files for which he wanted to change permissions but wanted to avoid re-uploading. There are two issues then. One with permission and for other using CLI will avoid him to go though all the files manually. Commented May 1, 2017 at 17:08

8 Answers 8

38

A very interesting conundrum! Fortunately, there is a solution.

First, a recap:

  • Bucket A in Account A
  • Bucket B in Account B
  • User in Account A copies objects to Bucket B (having been granted appropriate permissions to do so)
  • Objects in Bucket B still belong to Account A and cannot be accessed by Account B

I managed to reproduce this and can confirm that users in Account B cannot access the file -- not even the root user in Account B!

Fortunately, things can be fixed. The aws s3 cp command in the AWS Command-Line Interface (CLI) can update permissions on a file when copied to the same name. However, to trigger this, you also have to update something else otherwise you get this error:

This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.

Therefore, the permissions can be updated with this command:

aws s3 cp s3://my-bucket/ s3://my-bucket/ --recursive --acl bucket-owner-full-control --metadata "One=Two"
  • Must be run by an Account A user that has access permissions to the objects (eg the user who originally copied the objects to Bucket B)
  • The metadata content is unimportant, but needed to force the update
  • --acl bucket-owner-full-control will grant permission to Account B so you'll be able to use the objects as normal

End result: A bucket you can use!

10
  • What if Account A gets an AccessDenied error when changing the permissions of a file uploaded to Bucket B? Is this a permissions error in Account A or Account B? Or Bucket B?
    – Phlucious
    Commented Jan 22, 2018 at 18:59
  • @Phlucious The permissions would be in the account that owns the resource (in this case, Bucket B). In future, please create a new Question rather than asking a different question within a comment. Commented Jan 22, 2018 at 22:37
  • Thanks! It seemed relevant because I was able to adapt this answer to grant full control to Account B on a new upload. Googling around for that brought me here.
    – Phlucious
    Commented Jan 23, 2018 at 16:39
  • 1
    I've just tried this, and --metadata is not needed for the solution to work. --acl bucket-owner-full-control is sufficient.
    – Imma
    Commented Jun 21, 2018 at 9:21
  • 2
    The --metadata portion is required if the intention is to change permissions on an object that already exists, otherwise the cp command fails because nothing is being 'changed'. If a new object is being copied (or being copied from a different location), --metadata would not be required. Commented Oct 13, 2018 at 2:15
3
aws s3 cp s3://account1/ s3://accountb/ --recursive --acl bucket-owner-full-control 
2

To correctly set the appropriate permissions for newly added files, add this bucket policy:

[...]
{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::123456789012::user/their-user"
    },
    "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
    ],
    "Resource": "arn:aws:s3:::my-bucket/*"
}

And set ACL for newly created files in code. Python example:

import boto3

client = boto3.client('s3')
local_file_path = '/home/me/data.csv'
bucket_name = 'my-bucket'
bucket_file_path = 'exports/data.csv'
client.upload_file(
    local_file_path,
    bucket_name, 
    bucket_file_path, 
    ExtraArgs={'ACL':'bucket-owner-full-control'}
)

source: https://medium.com/artificial-industry/how-to-download-files-that-others-put-in-your-aws-s3-bucket-2269e20ed041 (disclaimer: written by me)

1

In case anyone trying to do the same but using Hadoop/Spark job instead of AWS CLI.

  • Step 1: Grant user in Account A appropriate permissions to copy objects to Bucket B. (mentioned in above answer)
  • Step 2: Set the fs.s3a.acl.default configuration option using Hadoop Configuration. This can be set in conf file or in program:

    Conf File:

    <property> <name>fs.s3a.acl.default</name> <description>Set a canned ACL for newly created and copied objects. Value may be Private, PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead, or BucketOwnerFullControl.</description> <value>$chooseOneFromDescription</value> </property>

    Programmatically:

    spark.sparkContext.hadoopConfiguration.set("fs.s3a.acl.default", "BucketOwnerFullControl")

1

by putting

--acl bucket-owner-full-control made it to work.

2
  • can u offer some more detail? Where did you put that?
    – Robbiegod
    Commented May 3, 2019 at 19:17
  • Please put that after your aws s3 <parameters> then use the above Commented May 7, 2019 at 8:51
0

I'm afraid you won't be able to transfer ownership as you wish. Here's what you did:

Old account copies objects into new account.

The "right" way of doing it (assuming you wanted to assume ownership on the new account) would be:

New account copies objects from old account.

See the small but important difference? S3 docs kind of explain it.

I think you might get away with it without needing to download the whole thing by just copying all of the files within the same bucket, and then deleting the old files. Make sure you can change the permissions after doing the copy. This should save you some money too, as you won't have to pay for the data transfer costs of downloading everything.

3
  • Yes I see. Thank you. I was not able to be given "bucket" access to the other accounts bucket, so we had to try it this way. I'll try what you suggest.
    – Robbiegod
    Commented May 1, 2017 at 17:23
  • I did try to download an image using Cloudberry, but even that doesn't work. I get a 403 error, Access Denied. I tried a cp command in CLI but I keep get 403 errors doing that too. Not sure what my options are now. :(
    – Robbiegod
    Commented May 1, 2017 at 20:04
  • I was thinking you would be able to just copy/and paste your files on the AWS console itself. If you can't download files from that bucket, it looks like you don't even have GetObject permissions, only ListBucket. That would actually make sense, since GetObject is an object permission, and ListBucket is a bucket permission, and you own the bucket, but not the files. If you can confirm that, you'll need to ask the owner of the other AWS account to either re-copy it or give you access.
    – Viccari
    Commented May 1, 2017 at 21:14
0

boto3 "copy_object" solution :

Providing Grant control to the destination bucket owner

client.copy_object(CopySource=copy_source, Bucket=target_bucket, Key=key, GrantFullControl='id=<bucket owner Canonical ID>')
  • Get for console
  • Select bucket, Permission tab, "Access Control List" tab
0

Below policy configuration helped me

{
  "Statement": [
    {
      "Action": [
        "s3:GetObject",
        "s3:GetObjectTagging",
        "s3:PutObject",
        "s3:PutObjectTagging",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my-bucket/*"
      ]
    },
    {
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my-bucket"
      ]
    }
  ],
  "Version": "2012–10–17"
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.