12

First of all, I know that there is no such thing as a folder, however, I am using the term just for simplicity. Now I have a lot of objects in my bucket. For some folders, I can use getObject to find if they exist or not, but for most of them, I get an error No such key when I clearly see that they are present. I would also like to say that I have tried headObject, listObjectV2 even but with no luck.

this is my params object
{
     Bucket: bucket //bucket is defined in the program
     Key: folder // defined in the program above
                 //example key - abc-1-1/00000N/30/2018.10.7.8/
                 //inside this folder are multiple files 
}

The goal is to find if the folder exists or not and based on that I do some processing. I saw a lot of answers to this question suggesting headObject, getObjects, etc. but none of them seem to work

This is my getObjects code snippet

params = {
            Bucket: bucket,
            Key: folder
        }
        s3.getObject(params, function (err, found) {

            if (err){ 
                console.log('bucket is'+bucket);
                ..........
                .....
            }
            else{
                ....
            }
            ..
            });
2

4 Answers 4

9

I finally found a workaround for this scenario. If your application requires you to check for the existence of a particular object("folder") and proceed further if the object("folder") actually exists, you can use this solution. I believe there is a problem with the way s3.getObject or s3.headObject work, anyway here is the solution.

1) use the s3.listObjectsV2 method.

2) check the Contents field of the response. If it is empty then the prefix you supplied does not exist. Here is the code snippet

s3.listObjectsV2(params, function (err, found) {

                if (err){ 
                    console.log(err);
                    errJson = {
                        'status': -1,
                        'message': 'Error while trying to list files'
                    };
                    callback(errJson);

                } 
                else {  

                    if (found.Contents.length === 0) {

                        errJson = {
                            'status': 0,
                            'message': 'Either the files are not present at s3 or the folder is incorrect '
                        }
                        callback(errJson);
                    }
                    else{
                               .....
                    }
....
});
9

In order to check whether a "folder" exists, check whether there's at least one object in the bucket whose key starts with the folder's name, plus a trailing slash.

This can be done using the listObjectV2 method, like so:

const s3 = ... // S3 client

const Bucket = 'mybucket';
const Prefix = 'myfolder/mysubfolder/';
const MaxKeys = 1; // If a single object is found, the folder exists.
const params = {
      Bucket,
      Prefix,
      MaxKeys
};

s3.listObjectsV2(params, (err, data) => {
  const folderExists = data.Contents.length > 0;

  // (folderExists == true) if the folder exists
});

Note: The above uses the definition of a folder to mean that there's at least one key with the folder as its prefix.

To detect "empty folders", maybe check for the folder + _$folder$ suffix, as mentioned here.

1
  • Not clear on why headObject wasn't exhibiting the behavior described in other responses --but this was the solution that worked for me! Commented Jul 7, 2020 at 1:12
2

How is headObject not working exactly? If you search for a directory within your s3 bucket it should be returning one of two things:

  1. an error with the status code of 'NotFound' if the requested resource is not there (including a directory).
  2. a response with some details about the object.

Below you can find a simple implementation of now to write a "check" function:

      s3.headObject({Bucket: 'whatever-the-bucket-name-is', Key: 'directory/'}, (err, data) => {
        if (err && err.code === 'NotFound') resolve(false)
        else if (err) reject(err)
        resolve(true)
      })

Hope this helps.

7
  • 2
    Nope, it gives me err eventhough I have that directory/directory/directory/ set as Key, its present on s3 as well. This is what I am saying, these functions are giving me an err even when the directory exists
    – capedCoder
    Commented Oct 11, 2018 at 11:47
  • Huh... do you have any content within the bucket? Take into account that folders within an s3 are a "virtual" concept, meaning that they are trully just prefixes for objects keys. More info here: docs.aws.amazon.com/AmazonS3/latest/user-guide/… Commented Oct 11, 2018 at 12:12
  • Yup I do have objects in my bucket. I know the concept of Objects and keys, used the term "folder" in the question for simplicity
    – capedCoder
    Commented Oct 11, 2018 at 12:41
  • Sorry, actual question was if there was any content within the folders. If they are empty, amazon is just creating zero length objects that are ignored by a lot of processes, therefore it is entirely possible that the "check" function is ignoring them too. Commented Oct 11, 2018 at 12:50
  • Oh okay thanks for clarifying, the object does have contents in it btw.
    – capedCoder
    Commented Oct 11, 2018 at 13:05
-2

If you're checking whether bucket exists or not then here's the solution for that

var AWS = require('aws-sdk');
var s3 = {
  accessKeyId: '',
  secretAccessKey: '',
  bucketName: ''
}
AWS.config.update({
  accessKeyId: s3.accessKeyId,
  secretAccessKey: s3.secretAccessKey,
  // region: {region} - Add if needed
});

var s3bucket = new AWS.S3({ params: { Bucket: s3.bucketName } });
var options = {
  Bucket: s3.bucketName
};

function checkBucket(){
  try {
    await s3bucket.headBucket(options).promise();  
    console.log('bucket exists');
    // do the required operations here
  } catch (err) {
    if (err.statusCode === 404) {
      return error("Bucket doesn't exists.");
    }
  }
}

(async() =>{
  await checkBucket();
})()
7
  • Hi Atishay thanks but I am actually looking to check the existence of a particular directory.
    – capedCoder
    Commented Oct 11, 2018 at 11:48
  • actually if the directory doesn't exists s3 automatically creates one while pushing Commented Oct 11, 2018 at 11:49
  • The problem is that my directory exists, but these functions are giving me an error instead of data. Now I want to check for the existence because my code requires processing only after checking the existence otherwise nothing should be done
    – capedCoder
    Commented Oct 11, 2018 at 11:52
  • message: null, code: 'NotFound', region: null, time: 2018-10-11T12:43:55.069Z, requestId: 'E480454D2E1215B3', extendedRequestId: 'IKtZH0T6Govkpyvw3BGcjekDzyTMZOH1gUeEJbXzcVrFKIqDJkH0cKiEqTt32+4AMiJe6YMwIXg=', cfId: undefined, statusCode: 404, retryable: false, retryDelay: 54.259656228362594
    – capedCoder
    Commented Oct 11, 2018 at 12:48
  • There is a bunch of other stuff also in the error but its too many characters to upload
    – capedCoder
    Commented Oct 11, 2018 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.