7

I have a bucket/folder into which a lot for files are coming in every minutes. How can I read only the new files based on file timestamp.

eg: list all files with timestamp > my_timestamp

4 Answers 4

21

You could use some bash-fu:

gsutil ls -l gs://<your-bucket-name> | sort -k2n | tail -n1 | awk 'END {$1=$2=""; sub(/^[ \t]+/, ""); print }'

breaking that down:

# grab detailed list of objects in bucket
gsutil ls -l gs://your-bucket-name 

# sort by number on the date field
sort -k2n

# grab the last row returned 
tail -n1

# delete first two cols (size and date) and ltrim to remove whitespace
awk 'END {$1=$2=""; sub(/^[ \t]+/, ""); print }'`

Tested with Google Cloud SDK v186.0.0, gsutil v4.28

4
  • this solution is very brittle, as it will break easily if google decides to change the format a little bit Jan 26, 2018 at 4:35
  • 6
    @remisharoon You never asked for super robust in your original question. I'll add the gsutil version number to my answer so that people get an idea. Jan 29, 2018 at 20:47
  • 2
    Update that works for me on newer Google CloudSDK 190.0.1 and gsutil 4.28: gsutil ls -l gs://your-bucket-name | sort -k2 | tail -n2 | head -n1 | awk 'END {$1=$2=""; sub(/^[ \t]+/, ""); print }'
    – wearego
    Aug 1, 2019 at 12:15
  • is it possible to choose asc or dsc order while ouput the sort results Jun 29, 2021 at 11:36
5

This is not a feature that gsutil or the GCS API provides, as there is no way to list objects by timestamp.

Instead, you could subscribe to new objects using the GCS Cloud Pub/Sub feature.

1
1

For finding list of files with timestamp greater than specified timestamp, use below command by replacing <bucket-name/folder> and <my_timestamp in epoch> with required value:

Mac:

gsutil ls -l gs://<bucket-name/folder> | sed \$d | grep -v '/$' | awk '{ split($0,a," "); extracttimestamp="date -jf '%Y-%m-%dT%H:%M:%SZ' " a[2] " +%s"; extracttimestamp | getline $1; close(extracttimestamp); { if ( $1 > <my_timestamp in epoch> ) { print a[3],a[2],$1 }}}'

Linux:

gsutil ls -l gs://<bucket-name/folder> | grep -v '/$' | grep -v '^TOTAL' | awk -F, '{ split($0,a," "); gsub("-","/",a[2]); gsub("T"," ",a[2]); gsub("Z","",a[2]); extracttimestamp="date -d " "\"" a[2] "\""  " +%s"; extracttimestamp | getline $1; close(extracttimestamp); { if ( $1 > <my_timestamp in epoch> ) { print a[3],a[2],$1 }}}' 

Understanding above command:

gsutil ls -l gs://bucket/test/

#If you specify the -l option, 
#gsutil outputs additional information about each matching object e.g. file timestamp
#output
   1453783  2020-03-02T19:25:16Z  gs://bucket/test/
   2276224  2020-03-02T19:25:17Z  gs://bucket/test/obj1.html
   3914624  2020-03-02T19:30:27Z  gs://bucket/test/obj2.html
       131  2020-03-02T19:37:45Z  gs://bucket/test/obj3.txt
TOTAL: 3 objects, 6190979 bytes (5.9 MiB)

sed \$d

#all line except the last line
#output
   1453783  2020-03-02T19:25:16Z  gs://bucket/test/
   2276224  2020-03-02T19:25:17Z  gs://bucket/test/obj1.html
   3914624  2020-03-02T19:30:27Z  gs://bucket/test/obj2.html
       131  2020-03-02T19:37:45Z  gs://bucket/test/obj3.txt

grep -v '/$'

#text that does not end with "/", there by removing folder
#output
   2276224  2020-03-02T19:25:17Z  gs://bucket/test/obj1.html
   3914624  2020-03-02T19:30:27Z  gs://bucket/test/obj2.html
       131  2020-03-02T19:37:45Z  gs://bucket/test/obj3.txt

awk '{ split($0,a," "); extracttimestamp="date -jf '%Y-%m-%dT%H:%M:%SZ' " a[2] " +%s"; extracttimestamp | getline $1; close(extracttimestamp); { if ( $1 > <my_timestamp in epoch> ) { print a[3],a[2],$1 }}}'

#split each line into array of token
#split($0,a," ")
    #split("2276224  2020-03-02T19:25:17Z  gs://bucket/test/obj1.html", a, " ")
        a=("2276224" "2020-03-02T19:25:17Z" "gs://bucket/test/obj1.html")

#extract epoch time from second token(file timestamp)
#extracttimestamp="date -jf '%Y-%m-%dT%H:%M:%SZ' " a[2] " +%s"
    #date -jf '%Y-%m-%dT%H:%M:%SZ' 2020-03-02T19:25:17Z +%s = 1583157317
        #extracttimestamp = 1583157317

#read extracttimestamp and bind value to $1
#extracttimestamp | getline $1;
    #$1 = 1583157317

#compare file timestamp with my_timestamp and print filename
#if ( $1 > <my_timestamp in epoch> ) { print a[3],a[2],$1 }
    #if ( 1583157317 > 1583157320 ) { print a[3],a[2],$1 }

#final output
gs://bucket/test/obj2.html 2020-03-02T19:30:27Z 1583157627
gs://bucket/test/obj3.txt 2020-03-02T19:37:45Z 1583158065
0

If you are interested in new files or we can say in another words the files which are not present in your destination bucket then alternatively you can use gsutil -n option as it copies only those files which are not present in destination bucket.

From documentation https://cloud.google.com/storage/docs/gsutil/commands/cp?hl=ru

No-clobber. When specified, existing files or objects at the destination will not be overwritten. Any items that are skipped by this option will be reported as being skipped. This option will perform an additional GET request to check if an item exists before attempting to upload the data. This will save retransmitting data, but the additional HTTP requests may make small object transfers slower and more expensive.

cons with this approach is, it makes a check request for every file present in your source bucket

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.