2

I was wondering as to how I can download an entire folder or a repository from Nexus 3.37. I tried the following command

curl -X GET -u userid:password "https://nexus.com/abc/def/ghi/" -O

I was able to download a single file in a directory using the above mentioned command, Does anyone have ideas as to how I can download the entire folder?

Thanks!

8
  • You may also want to look at wget which is well-suited to downloading files recursively from the web. Look at the --recursive and --level options in particular.
    – cherdt
    Jan 7, 2022 at 0:08
  • The only viable option I know is going through the API to list all components/assets and download them one after an other. Jan 7, 2022 at 0:14
  • @cherdt , I tried wget with --recursive alone and also with --no-parent as well, no luck, I will try using --level now
    – Neoneuron
    Jan 7, 2022 at 13:41
  • @Zeitounator, There are a lot of files to download if I'm doing that, thanks for responding though! :)
    – Neoneuron
    Jan 7, 2022 at 13:43
  • What I'm actually trying to do is take a backup of the data I have in the repository, any ideas on how I can do that?
    – Neoneuron
    Jan 7, 2022 at 13:44

3 Answers 3

2

The script above really helped me with this issue. I slightly modified it and posted it below. This script is by no means perfect and you may need to edit it. I know one issue is that the very first download this script initiates is put in the wrong directory.

sourceServer=
sourceRepo=
sourceUser=
sourcePassword=
logfile=$sourceRepo-backup.log
outputFile=$sourceRepo-artifacts.txt

# ======== GET DOWNLOAD URLs =========
url=$sourceServer"/service/rest/v1/assets?repository="$sourceRepo
contToken="initial"
while [ ! -z "$contToken" ]; do
    if [ "$contToken" != "initial" ]; then
        url=$sourceServer"/service/rest/v1/assets?continuationToken="$contToken"&repository="$sourceRepo
    fi
    echo Processing repository token: $contToken | tee -a $logfile
    response=`curl -ksSL -u "$sourceUser:$sourcePassword" -X GET --header 'Accept: application/json' "$url"`
    readarray -t artifacts < <( jq  '[.items[].downloadUrl]' <<< "$response" )
    printf "%s\n" "${artifacts[@]}" > artifacts.temp
    sed 's/\"//g' artifacts.temp > artifacts1.temp
    sed 's/,//g' artifacts1.temp > artifacts.temp
    sed 's/[][]//g' artifacts.temp > artifacts1.temp
    cat artifacts1.temp >> $outputFile
#for filter in "${filters[@]}"; do
     #   cat artifacts.temp | grep "$filter" >> $outputFile
    #done
    #cat maven-public-artifacts.txt
    contToken=( $(echo $response | sed -n 's|.*"continuationToken" : "\([^"]*\)".*|\1|p') )
done


# ======== DOWNLOAD EVERYTHING =========
    echo Downloading artifacts...
    urls=($(cat $outputFile)) > /dev/null 2>&1
    for url in "${urls[@]}"; do
        path=${url#http://*:*/*/*/}
        dir=$sourceRepo"/"${path%/*}
        mkdir -p  $dir
        cd $dir
        pwd
        curl -vks -u "$sourceUser:$sourcePassword" -D response.header -X GET "$url" -O  >> /dev/null 2>&1
        responseCode=`cat response.header | sed -n '1p' | cut -d' ' -f2`
        if [ "$responseCode" == "200" ]; then
            echo Successfully downloaded artifact: $url
        else
            echo ERROR: Failed to download artifact: $url  with error code: $responseCode
        fi
        rm response.header > /dev/null 2>&1
        cd $curFolder
    done
1

I am a little bit intrigued by the fact that there is no "download all files in a folder" in the REST API

So here are my changes:

  • Taking back a simple filter (for e g a sub-folder)
  • Preserving spaces in the URLs and converting to %20 for download
  • Setting the curFolder to the result of pwd
  • Removing the output file if it exists when you do consecutive runs
  • Changed from http to https
sourceServer=
sourceRepo=
sourceFolder=
sourceUser=
sourcePassword=
logfile=$sourceRepo-backup.log
outputFile=$sourceRepo-artifacts.txt
[ -e $outputFile ] && rm $outputFile

# ======== GET DOWNLOAD URLs =========
url=$sourceServer"/service/rest/v1/assets?repository="$sourceRepo
contToken="initial"
while [ ! -z "$contToken" ]; do
    if [ "$contToken" != "initial" ]; then
        url=$sourceServer"/service/rest/v1/assets?continuationToken="$contToken"&repository="$sourceRepo
    fi
    echo Processing repository token: $contToken | tee -a $logfile
    response=`curl -ksSL -u "$sourceUser:$sourcePassword" -X GET --header 'Accept: application/json' "$url"`
    readarray -t artifacts < <( jq  '[.items[].downloadUrl]' <<< "$response" )
    printf "%s\n" "${artifacts[@]}" > artifacts.temp
    sed 's/\"//g' artifacts.temp > artifacts1.temp
    sed 's/,//g' artifacts1.temp > artifacts2.temp
    sed 's/[][]//g' artifacts2.temp > artifacts3.temp
    cat artifacts3.temp | grep "$sourceFolder" >> $outputFile
    contToken=( $(echo $response | sed -n 's|.*"continuationToken" : "\([^"]*\)".*|\1|p') )
done


# ======== DOWNLOAD EVERYTHING =========
    echo Downloading artifacts...
    IFS=$'\n' read -d '' -r -a urls < $outputFile
    for url in "${urls[@]}"; do
        url="$(echo -e "${url}" | sed -e 's/^[[:space:]]*//')"
        path=${url#https://*/*/*/}
        dir="\""$sourceRepo"/"${path%/*}"\""
        curFolder=$(pwd)
        mkdir -p $dir
        cd $dir
        url="$(echo -e "${url}" | sed -e 's/\s/%20/g')"
        curl -vks -u "$sourceUser:$sourcePassword" -D response.header -X GET "$url" -O  >> /dev/null 2>&1
        responseCode=`cat response.header | sed -n '1p' | cut -d' ' -f2`
        if [ "$responseCode" == "200" ]; then
            echo Successfully downloaded artifact: $url
        else
            echo ERROR: Failed to download artifact: $url  with error code: $responseCode
        fi
        rm response.header > /dev/null 2>&1
        cd $curFolder
    done
0

You can use curl to retrieve a list of all your artifacts download URLs by executing:

curFolder=`pwd`
filters=("\\.jar$" "\\.pom$" "\\.zip$")
sourceServer=https://myserver.com:1234
sourceRepo=myrepo
sourceUser=username
sourcePassword=password
logfile=$sourceRepo-backup.log
outputFile=$sourceRepo-artifacts.txt
# ======== GET DOWNLOAD URLs =========
url=$sourceServer"/service/rest/v1/assets?repository="$sourceRepo
contToken="initial"
while [ ! -z "$contToken" ]; do
    if [ "$contToken" != "initial" ]; then
        url=$sourceServer"/service/rest/v1/assets?continuationToken="$contToken"&repository="$sourceRepo
    fi
    echo Processing repository token: $contToken | tee -a $logfile
    response=`curl -ksSL -u "$sourceUser:$sourcePassword" -X GET --header 'Accept: application/json' "$url"`
    artifacts=( $(echo $response | sed -n 's|.*"downloadUrl" : "\([^"]*\)".*|\1|p') )
    printf "%s\n" "${artifacts[@]}" > artifacts.temp
    for filter in "${filters[@]}"; do
        cat artifacts.temp | grep "$filter" >> $outputFile
    done
    contToken=( $(echo $response | sed -n 's|.*"continuationToken" : "\([^"]*\)".*|\1|p') )
done

At this point you have all the artifacts filtered out from sha1 and md5 files, metadata etc. so you can loop through the lines of the output file and download them using curl.

# ======== DOWNLOAD EVERYTHING =========
    echo Downloading artifacts...
    urls=($(cat $outputFile)) > /dev/null 2>&1
    for url in "${urls[@]}"; do
        path=${url#http://*:*/*/*/}
        dir=$sourceRepo"/"${path%/*}
        mkdir -p $dir
        cd $dir
        curl -vks -u "$sourceUser:$sourcePassword" -D response.header -X GET "$url" -O  >> /dev/null 2>&1
        responseCode=`cat response.header | sed -n '1p' | cut -d' ' -f2`
        if [ "$responseCode" == "200" ]; then
            echo Successfully downloaded artifact: $url  | tee -a $logfile 2>&1
        else
            echo ERROR: Failed to download artifact: $url  with error code: $responseCode  | tee -a $logfile 2>&1
        fi
        rm response.header > /dev/null 2>&1
        cd $curFolder
    done

This is downloading everything and preserving the folder structure so later you can upload everything using Maven (assuming Maven2 repo). It will also report if it was successful at downloading it or not. Try to check first the credentials, otherwise you will end up doing thousands of wrong attempts and you will end up locking your account.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.