48

What would be the proper gradle way of downloading and unzipping the file from url (http)?

If possible, I'd like to prevent re-downloading each time I run the task (in ant.get can be achieved by skipexisting: 'true').

My current solution would be:

task foo {
  ant.get(src: 'http://.../file.zip', dest: 'somedir', skipexisting: 'true')
  ant.unzip(src: 'somedir' + '/file.zip', dest: 'unpackdir')
}

still, I'd expect ant-free solution. Any chance to achieve that?

3
  • 2
    Don't forget to wrap the execution part of a task with doLast { ... } (same mistake as in your previous question). Commented Apr 11, 2014 at 22:20
  • 1
    ~~BOUNTY~~ Can anyone provide an example for the answer below: "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ?
    – CMPS
    Commented Dec 16, 2015 at 21:22
  • netflix has released some plugin: github.com/nebula-plugins/nebula-core - not sure, why this is not maintained anymore.
    – koppor
    Commented Aug 19, 2019 at 20:12

7 Answers 7

97
+200

Let's say you want to download this zip file as a dependency:

https://github.com/jmeter-gradle-plugin/jmeter-gradle-plugin/archive/1.0.3.zip

You define your ivy repo as:

repositories {
    ivy {
        url 'https://github.com/'

        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }

        // This is required in Gradle 6.0+ as metadata file (ivy.xml) 
        // is mandatory. Docs linked below this code section
        metadataSources { artifact() } 
    }
}

reference for required metadata here

The dependency can then be used as:

dependencies {
    compile 'jmeter-gradle-plugin:jmeter-gradle-plugin:1.0.3@zip'
    //This maps to the pattern: [organisation]:[module]:[revision]:[classifier]@[ext]         
}

To unzip:

task unzip(type: Copy) {

  def zipPath = project.configurations.compile.find {it.name.startsWith("jmeter") }
  println zipPath
  def zipFile = file(zipPath)
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir

}

optional:

If you have more than one repository in your project, it may also help (for build time and somewhat security) to restrict dependency search with relevant repositories.

Gradle 6.2+:

repositories {
    mavenCentral()
    def github = ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
    }
    exclusiveContent {
        forRepositories(github)
        filter { includeGroup("jmeter-gradle-plugin") }
    }
}

Earlier gradle versions:

repositories {
    mavenCentral {
        content { excludeGroup("jmeter-gradle-plugin") }
    }
    ivy {
        url 'https://github.com/'
        patternLayout {
            artifact '/[organisation]/[module]/archive/[revision].[ext]'
        }
        metadataSources { artifact() }
        content { includeGroup("jmeter-gradle-plugin") }
    }
}
16
  • 3
    Unfortunately URLs with query strings cannot be accessed this way, '?' is encoded by github.com/gradle/gradle/blob/master/subprojects/resources/src/…
    – Akom
    Commented Apr 21, 2017 at 18:11
  • 2
    When I am trying this rather than looking for .zip file it is looking for a .xml file. Any way around for this? I followed exactly the same instruction. Commented Jun 26, 2018 at 10:09
  • 3
    Doesn't work for me. Gradle is looking for an ivy.xml file in addition to the artifact.
    – Victor L
    Commented Jun 29, 2018 at 15:42
  • 1
    This is a great answer. I would only add that the dependency should probably be compileOnly to prevent the raw zip ending up in the runtime artifacts.
    – Andy Brown
    Commented Oct 19, 2020 at 9:29
  • 1
    I can confirm this works with 7.2, @rios0rios0 have you resolved?
    – elect
    Commented Oct 21, 2021 at 16:02
14
    plugins {
        id 'de.undercouch.download' version '4.0.0'
    }

    /**
     * The following two tasks download a ZIP file and extract its
     * contents to the build directory
     */
    task downloadZipFile(type: Download) {
        src 'https://github.com/gradle-download-task/archive/1.0.zip'
        dest new File(buildDir, '1.0.zip')
    }

    task downloadAndUnzipFile(dependsOn: downloadZipFile, type: Copy) {
        from zipTree(downloadZipFile.dest)
        into buildDir
    }

https://github.com/michel-kraemer/gradle-download-task

1
  • 2
    This is not really a good solution compared to Ant as it is really slow. 26 minutes download instead of 2 minutes with Chrome.
    – Michael S.
    Commented Oct 29, 2019 at 14:30
9

There isn't currently a Gradle API for downloading from a URL. You can implement this using Ant, Groovy, or, if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL. The unzipping can be done in the usual Gradle way (copy method or Copy task).

5
  • 4
    I would appreciate an example... I'm a total gradle newbie!
    – xpmatteo
    Commented Mar 5, 2015 at 14:12
  • 4
    ~~BOUNTY~~ Can anyone provide an example for this answer "if you do want to benefit from Gradle's dependency resolution/caching features, by pretending it's an Ivy repository with a custom artifact URL" ?
    – CMPS
    Commented Dec 16, 2015 at 21:23
  • @CMPS why you no groovy?
    – RaGe
    Commented Dec 17, 2015 at 5:20
  • @RaGe my current solution is using groovy and it works fine, but I wanted one using repository to benefit from the caching system.
    – CMPS
    Commented Dec 17, 2015 at 5:41
  • @CMPS Ah, caching! Posted an answer below.
    – RaGe
    Commented Dec 17, 2015 at 5:42
7

Unzipping using the copy task works like this:

task unzip(type: Copy) {
  def zipFile = file('src/dists/dist.zip')
  def outputDir = file("${buildDir}/unpacked/dist")

  from zipTree(zipFile)
  into outputDir
}

http://mrhaki.blogspot.de/2012/06/gradle-goodness-unpacking-archive.html

2
  • I tried moving the from zipTree() & into outputDir to the execution phase by moving them to doLast { }. What I did not understand was that it never worked. Even the println code in doLast did not work. However, this code works in the gradle configuration phase. Why?
    – Dexter
    Commented Dec 20, 2018 at 9:18
  • Because it configures the task. The execution still is done during execution phase. Changing the configuration during the execution phase should not be done actually and in your case the execution phase would not be executed at all, as the task thinks there is nothing to copy, so the task execution is skipped. But actually this answer is not related to the question anyway, as this does not work with an HTTP URL.
    – Vampire
    Commented Oct 29, 2019 at 15:26
5

This works with Gradle 5 (tested with 5.5.1):

task download {
    doLast {
        def f = new File('file_path')
        new URL('url').withInputStream{ i -> f.withOutputStream{ it << i }}
    }
}

Calling gradle download downloads the file from url to file_path.

You can use the other methods from other answers to unzip the file if necessary.

1
  • Also works in gradle 4.7. In case someone needs it.
    – camikiller
    Commented Jul 7, 2021 at 17:05
1

I got @RaGe's answer working, but I had to adapt it since the ivy layout method has been depreciated see https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.repositories.IvyArtifactRepository.html#org.gradle.api.artifacts.repositories.IvyArtifactRepository:layout(java.lang.String,%20groovy.lang.Closure)

So to get it working I had to adjust it to this for a tomcat keycloak adapter:

ivy {
    url 'https://downloads.jboss.org/'
    patternLayout {
        artifact '/[organization]/[revision]/adapters/keycloak-oidc/[module]-[revision].[ext]'
    }
}

dependencies {
    // https://downloads.jboss.org/keycloak/4.8.3.Final/adapters/keycloak-oidc/keycloak-tomcat8-adapter-dist-4.8.3.Final.zip
    compile "keycloak:keycloak-tomcat8-adapter-dist:$project.ext.keycloakAdapterVersion@zip"
}

task unzipKeycloak(type: Copy) {

    def zipPath = project.configurations.compile.find {it.name.startsWith("keycloak") }
    println zipPath
    def zipFile = file(zipPath)
    def outputDir = file("${buildDir}/tomcat/lib")

    from zipTree(zipFile)
    into outputDir
}
2
  • Unfortunately, this solution also has Gradle looking for an Ivy XML file.
    – barfuin
    Commented Apr 17, 2019 at 16:51
  • Doesn't work. Gradle 7.2 is looking for a normal maven dependency.
    – rios0rios0
    Commented Sep 7, 2021 at 13:16
1

"native gradle", only the download part (see other answers for unzipping)

task foo {
  def src = 'http://example.com/file.zip'
  def destdir = 'somedir'
  def destfile = "$destdir/file.zip"
  doLast {
    def url = new URL(src)
    def f = new File(destfile)
    if (f.exists()) {
      println "file $destfile already exists, skipping download"
    } else {
      mkdir "$destdir"
      println "Downloading $destfile from $url..."
      url.withInputStream { i -> f.withOutputStream { it << i } }
    }
  }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.