18

I having a program that successfully uploads all of the files that I need. I have new files everyday that I need to upload. After I have uploaded the files I no longer need them and thus am not looking to sync them.

I am curious if there is a way to check if given a path and file name if that exists within S3 using the s3cmd.

4 Answers 4

30

You can use the ls command in s3cmd to know if a file is present or not in S3.

Bash code

path=$1
count=`s3cmd ls $path | wc -l`

if [[ $count -gt 0 ]]; then
        echo "exist"
else
        echo "do not exist"
fi

Usage: ./s3_exist.sh s3://foo/bar.txt

Edit:

As cocoatomo pointed out in comments, s3cmd ls $path lists all file that begins with $path. A safer approach would be to use s3cmd info $path and check the exit code.

New Bash code

path=$1
s3cmd info $path >/dev/null 2>&1

if [[ $? -eq 0 ]]; then
    echo "exist"
else
    echo "do not exist"
fi
4
  • So count is the number of files within that directory, I understand that. I however have not ran across the -gt part and cannot find anything when I tried googling it, would you mind explaining that part?
    – Tall Paul
    Jul 3, 2013 at 20:38
  • 1
    Count the number of files that match a given path and filename. Since S3 doesn't support duplicate filenames, this count will be either 0 (file not present) or 1 (file present). "-gt" stands for "greater than". You can also do [[ $count -eq 1 ]] if this is more explicit for you. For more details about conditional expression in Bash: link Jul 8, 2013 at 22:49
  • 1
    The first argument of "s3cmd ls" means a bucket name or a prefix string following a bucket name, not a file name. Since "s3cmd ls s3://foo/b" lists "s3://foo/bar.txt" if exists, "$count -gt 0" does not show whether s3://foo/b exists or not.
    – cocoatomo
    Nov 25, 2013 at 8:34
  • Unfortunately one cannot trust the exit codes of s3cmd at all; even the version shipped in Ubuntu 14.04 universe can return 0 for s3cmd put even if the server's network is not connected. Apr 30, 2015 at 12:22
2

Assuming that bar.txt and bar.txt.bak exist in a bucket s3://foo, "s3cmd ls s3://foo/bar.txt" shows a following output.

$ s3cmd ls s3://foo/bar.txt
2013-11-11 11:11    5   s3://foo/bar.txt
2013-11-11 11:11    5   s3://foo/bar.txt.bak

Since we should remove 2nd line from the command result, we use "awk" command to filter unnecessary lines.

$ filename=s3://foo/bar.txt
$ s3cmd ls ${filename} | awk "\$4 == \"${filename}\" { print \$4 }"
2013-11-11 11:11    5   s3://foo/bar.txt

Finally, we build up all commands.

filename=s3://foo/bar.txt
count=$(s3cmd ls ${filename} | awk "\$4 == \"${filename}\" { print \$4 }" | wc -l)

if [ $count -eq 0 ]; then
  echo "file does not exist"
else
  echo "file exists"
fi
1

In the newer version of AWS CLI, you can use the following code to detect the existence of a file or directory

count=$(aws s3 ls $path | wc -l)
if [ $count -gt 0 ]
then
  (>&2 echo "$path already exists!")
  return
fi
1
0

We can use s3cmd ls , Take one flag flag_exists true if file is there and false if file is not there.

FLAG_EXISTS=false
    for j in $(s3cmd ls s3://abc//abc.txt); do
      if [[ "$j" == "s3://abc//abc.txt" ]]; then
        FLAG_EXISTS=true
        break
      fi
    done
    if [ "$FLAG_EXISTS" = false ]; then
      echo 'file not exists'
    else
      echo 'file exists'
    fi

Explanation - Since ls can return many values like if u search for s3cmd ls abc.txt , then it can return values like abc.txt abcd.txt and so on , so looping and checking using if condition if file exists.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.