1

I'm looking for a way to compute an sha-256 value for every file contained in a tar file. The problem is that the tar are 300GB with over 200,000 contained files.

It would be possible to do this in bash a couple of different ways.

Extract and then use find

tmp=`mktmp --directory extract_XXX`
cd "$tmp"
tar -xf "$tarfile"
find "$tmp" -type f -exec shasum -ba 256 {} +
cd ..
rm -rf "$tmp"

This method is bad because it requires 300GB space space to work and is slow because it has to copy the data before computing the sum

List the tar file and compute the individual sums

tar -tf "$tarfile" awk '/\/$/ {next} {print $0}' | while read file ; do
    sum=`tar -xOf "$tarfile" "$file" | shasum -ba 256`
    echo "${sum%-}${file}"
done

This requires less disk space but is much slower

How can I do this in a single pass of the tar file without extracting it to a temp directory?

I've tagged this as bash and python... The current code is bash but I'm flexable about language.

2 Answers 2

4

The tar utility knows its way:

tar xvf "$tarfile" --to-command 'shasum -ba 256'

The -v flag is important because tar sends each file at the standard input of the command. It will output the file on one line an the SHA sum on the next, but you can further process that very easily.

EDIT: here is the complete shell only code to output the SHA256s in one single tar file pass:

shopt -s extglob
tar xvf "$tarfile" --to-command 'shasum -ba 256' | \
  while read L; do
    [[ $L == *" *-" ]] && echo $SHAFILE ${L:0:64} || SHAFILE=$L
  done

For the glibc source archive, the output would look like:

glibc-2.24/.gitattributes c3f8f279e7e7b0020028d06de61274b00b6cb84cfd005a8f380c014ef89ddf48
glibc-2.24/.gitignore 35bcd2a1d99fbb76087dc077b3e754d657118f353c3d76058f6c35c8c7f7abae
glibc-2.24/BUGS 9b2d4b25c8600508e1d148feeaed5da04a13daf988d5854012aebcc37fd84ef6
glibc-2.24/CONFORMANCE 66b6e97c93a2381711f84f34134e8910ef4ee4a8dc55a049a355f3a7582807ec

Edit by OP:

As a one-liner this can be done as:

tar xf "$tarfile" --to-command 'bash -c "sum=`shasum -ba 256`; echo \"\${sum%-}$TAR_FILENAME\""'

or (on Ubuntu 20.04 and higher):

tar xf "$tarfile" --to-command 'bash -c "sum=`shasum -ba 256 | cut -d \" \" -f 1`; echo \"\${sum%-}$TAR_FILENAME\""'

Manual Page here: https://www.gnu.org/software/tar/manual/tar.html#SEC87

2

I don't know how fast will it be, but in python it can be done the following way:

import tarfile
import hashlib

def sha256(flo):
    hash_sha256 = hashlib.sha256()
    for chunk in iter(lambda: flo.read(4096), b'')
        hash_sha256.update(chunk)
    return hash_sha256.hexdigest()

with tarfile.open('/path/to/tar/file') as mytar:
    for member in mytar.getmembers():
        with mytar.extractfile(member) as _file:
            print('{} {}'.format(sha256(_file), member.name))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.