I'm looking for a way to compute an sha-256 value for every file contained in a tar file. The problem is that the tar are 300GB with over 200,000 contained files.
It would be possible to do this in bash a couple of different ways.
Extract and then use find
tmp=`mktmp --directory extract_XXX`
cd "$tmp"
tar -xf "$tarfile"
find "$tmp" -type f -exec shasum -ba 256 {} +
cd ..
rm -rf "$tmp"
This method is bad because it requires 300GB space space to work and is slow because it has to copy the data before computing the sum
List the tar file and compute the individual sums
tar -tf "$tarfile" awk '/\/$/ {next} {print $0}' | while read file ; do
sum=`tar -xOf "$tarfile" "$file" | shasum -ba 256`
echo "${sum%-}${file}"
done
This requires less disk space but is much slower
How can I do this in a single pass of the tar file without extracting it to a temp directory?
I've tagged this as bash and python... The current code is bash but I'm flexable about language.