One way is to use some Bash magic. I've knocked this up quickly using Cloud Shell, which has jq
pre-installed. It works by:
- Listing all datasets
- Listing each table in each dataset
- For each table, call
bq show
to grab the metadata information about the table
- Using
jq
pull out the id
and numLongTermBytes
from each table
- The numerical value for
numLongTermBytes
is how much storage is in long term. If it's 0, then it's active storage.
Not sure how this would work with partitions & clustered tables (you'll need to test), but should be enough to get you started at least. You could of course write a neater Python script or something, but I like the 'simplicity' of Bash :)
Noted that this is somewhat of a fragile/brittle solution.
bq ls --format=json | jq -r '.[].id' | xargs -n 1 -P 4 -i bq ls --format=json {} | jq -r '.[].id' | xargs -n 1 -P 4 -i bq show --format=json {} | jq -r '.id + "," + .numLongTermBytes'
