10

I use the jar command to build jar files. While trying to cache the jar files using md5 signatures, I found that jars built from the exact same sources had different md5 signatures.
Upon closer inspection, I found that every time the jar was created the contents were exactly the same (diff -qr was empty). It turns out that the timestamp of creation is encoded in the jar file which throws off the md5 signature. Other people have discovered the same here.

There is even a blog post on how to create jar files identically each time with maven. However, I want a simple solution using the command line using readily available commands such as jar and zip (may have to do this on a server without install permissions), possibly leading to the same "functional" jar as I'm currently getting using jar command.

EDIT: For my purpose, it also suffices to quickly find the md5 so that it is the same across builds, even if the jars are not identical. The only way I found so far is to extract the files in the jar and to md5 all component files. But I'm afraid that is slow for bigger jars and is going to defeat the purpose of caching them to avoid building them in the first place. Is there a better and faster solution?

5
  • So if you're compiling your source into a jar why don't you simply do it once and then execute an xcopy command or what not to copy that jar.
    – 3kings
    Commented Sep 18, 2016 at 4:55
  • Why is it important to have bit-identical containers? Commented Sep 18, 2016 at 5:05
  • The problem is, I'm trying to cache jars - A jar is built using several sources (java files, other jars). To determine whether a particular jar needs to be built, md5's of all source java files and jars is calculated. The component jars are rebuilt without caching and when this happens, their md5s change, which throws off the caching for the bigger jar.
    – user34812
    Commented Sep 18, 2016 at 5:14
  • This sounds like you're reinventing Maven and Gradle. Commented Sep 18, 2016 at 6:08
  • You are right. This is part of a project which has its own build system. I'm only one of the people working on it and cannot do anything about the fact that it is essentially reinventing maven.
    – user34812
    Commented Sep 18, 2016 at 7:03

2 Answers 2

4

The main issue is jar command always create META-INF\MANIFEST.MF with current time. The file time is saved in zip entry header. This is why MD5 value is different even all file content in jar remain the same: the different zip entry headers produce different zip file.

For jar command, the only solutionis is option -M: not to create a manifest file for the entries.

6
  • Confirming that this indeed creates jars with identical md5s. The jars also work. Can you briefly mention if the lack of manifest causes anything to be worried about?
    – user34812
    Commented Sep 18, 2016 at 8:48
  • Doesn't the ZIP (a jar is just a twiddled zip file IIRC) also store the metadata for the .class files? Like if you did a UNIX touch on one of the .class files, wouldn't it perturb the metadata and therefore mess up your checksum? Honestly I'm not sure; just asking.
    – BJ Black
    Commented Sep 18, 2016 at 10:26
  • @user34812. This depend on how you intend to use the jar file, because manifest file is used to store information about the jar file. For example: if you need these features/creating a OSGi bundles... etc. Manifest file is required.
    – Beck Yang
    Commented Sep 18, 2016 at 15:16
  • @BJ Black. As you said, the metadata of other files are stored in jar file too. Changing file last modification time of .class file make zip entry header changed. But they can be kept unlike META-INF\MANIFEST.MF, it always get updated. Even jar -m option is used to specify exist manifest file.
    – Beck Yang
    Commented Sep 18, 2016 at 15:16
  • Oops, @BJBlack is correct. I tried the -M flag on a smaller example earlier, without updating the class files. As soon as I update the class files, md5 doesn't match even with the -M flag. Is there possibly a way to overwrite all timestamp information in a .jar file everytime it is created?
    – user34812
    Commented Sep 18, 2016 at 17:16
1

Jar command always create META-INF\MANIFEST.MF with current time. Zip stores files with timestamp and file attributes due to which sha256 or MD5 will be different for two artifacts.

We need to make sure that created, last modified, accessed timestamp and file attributes are always same of all files which are required to create jar or zip.

I have created below script which can take a jar or zip file and make it deterministic by making timestamp constant and setting the right compression level and offset.

#!/bin/bash

usage() {
    echo "Usage : ./createDeterministicArtifact.sh <zip/jar file name>"
    exit 1
}

info() {
    echo "$1"
}

strip_artifact() {
    if [ -z ${file} ]; then
        usage
    fi
    if [ -f ${file} -a -s ${file} ]; then
        mkdir -p ${file}.tmp
        unzip -oq -d ${file}.tmp ${file}
        find ${file}.tmp -follow -exec touch -a -m -t 201912010000.00 {} \+
        if [ "$UNAME" == "Linux" ] ; then
            find ${file}.tmp -follow -exec chattr -a {} \+
        elif [[ "$UNAME" == CYGWIN* || "$UNAME" == MINGW* ]] ; then
            find ${file}.tmp -follow -exec attrib -A {} \+
        fi
        cd ${file}.tmp
        zip -rq -D -X -9 -A --compression-method deflate  ../${file}.new . 
        cd -
        rm -rf ${file}.tmp
        info "Recreated deterministic artifact: ${file}.new"
    else 
        info "Input file is empty. Please validate the file and try again"
    fi
}

file=$1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.