137

I use Powershell's Invoke-WebRequest method to download a file from Amazon S3 to my Windows EC2 instance.

If I download the file using Chrome, I am able to download a 200 MB file in 5 seconds. The same download in PowerShell using Invoke-WebRequest takes up to 5 minutes.

Why is using Invoke-WebRequest slower and is there a way to download at full speed in a PowerShell script?

8 Answers 8

255

Without switching away from Invoke-WebRequest, turning off the progress bar did it for me. I found the answer from this thread: https://github.com/PowerShell/PowerShell/issues/2138 (jasongin commented on Oct 3, 2016)

$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest <params>

For my 5MB file on localhost, the download time went from 30s to 250ms.

Note that to get the progress bar back in the active shell, you need to call $ProgressPreference = 'Continue'.

11
  • 9
    For a 100MB file this reduces the time from 10 Minutes to 2 Seconds. I wish developers would think more about the architecture of their software.
    – lanoxx
    Commented Oct 22, 2018 at 14:42
  • 62
    A lesson on how not to implement a progress bar.
    – sakra
    Commented Jan 9, 2019 at 19:56
  • 36
    It appear the progress bar updates after every byte, which is utter madness.
    – OrangeDog
    Commented Aug 22, 2019 at 15:00
  • 18
    Can't believe that -ProgressPreference isn't a parameter with Invoke-WebRequest
    – CJBS
    Commented May 5, 2020 at 22:27
  • 5
    Windows hasn't changed one bit.
    – nhooyr
    Commented Dec 14, 2022 at 11:19
74

I was using

Invoke-WebRequest $video_url -OutFile $local_video_url

I changed the above to

$wc = New-Object net.webclient
$wc.Downloadfile($video_url, $local_video_url)

This restored the download speed to what I was seeing in my browsers.

4
  • 3
    So I just ran wc.downloadFile(ibm-s3-url, "./test.tar.gz"), it did something, presumably download that file, but it didnt put it in my working directory... any idea where it might have gone?
    – Groostav
    Commented Aug 28, 2019 at 23:51
  • @Groostav Mine showed up in c:\windows\system32. Not exactly the first place I'd look when downloading to a relative location.
    – BryanC
    Commented Oct 13, 2019 at 9:39
  • 1
    @Groostav the current process' working directory is not the same thing as PowerShell's current location.
    – Mark
    Commented Oct 15, 2021 at 23:38
  • Can you show progress with this approach? Commented Feb 27 at 16:13
30

$ProgressPreference = 'SilentlyContinue' I got this down from 52min down to 14sec, for a file of 450 M. Spectacular.

3
  • 1
    please include more information in your answer, eg where does the setting you describe get set/defined?
    – danimal
    Commented Aug 7, 2020 at 14:49
  • 2
    This is a built-in preference variable to PowerShell, and controls whether or not a progress bar is displayed for certain operations, such as downloading files via Invoke-WebRequest. The reason this improves performance here is because Invoke-WebRequest (and Invoke-RestMethod) count the downloaded bytes too often (every single byte I believe), so the cmdlet is actually slowed down as it tallies how many bytes have been processed.
    – codewario
    Commented Nov 6, 2020 at 20:45
  • 1
    Is this answer actually a "thank you" comment to this answer? Commented Apr 23 at 9:49
20

One-liner to download a file to the temp directory:

(New-Object Net.WebClient).DownloadFile("https://www.google.com", "$env:temp\index.html")
6
  • 1
    How does this answer the question? Commented Nov 12, 2019 at 6:33
  • 5
    Lloyd asked: "is there a way to download at full speed in a PowerShell script?" This answer is a one-liner way to do that. Downloading to the temp directory is the canonical way to demonstrate this.
    – BlueSky
    Commented Nov 13, 2019 at 23:59
  • Invoke-WebRequest, which the OP is already using, does internally the same what WebClient.DownloadFile. Commented Nov 14, 2019 at 6:20
  • 5
    This is actually a good answer, WebClient.DownloadFile is much faster than Invoke-WebRequest due to the way Invoke-WebRequest tracks its own progress
    – codeulike
    Commented Nov 14, 2020 at 14:30
  • 2
    Absolutely relevant, this fixes exactly the problem OP has (and mine :) )
    – fl0w
    Commented Feb 10, 2021 at 9:45
2

I just hit this issue today, if you change the ContentType argument to application/octet-stream it is much faster (as fast as using webclient). The reason is because the Invoke-Request command will not try and parse the response as JSON or XML.

Invoke-RestMethod -ContentType "application/octet-stream" -Uri $video_url  -OutFile $local_video_url
2
  • 3
    -ContentType is for POST requests only AFAIK so it probably won't make a difference.
    – rednoah
    Commented Nov 2, 2016 at 19:30
  • 3
    You'd be better off using the -UseBasicParsing parameter but that won't make much of a difference for static files. Setting $ProgressPreference = 'SilentlyContinue' before calling Invoke-WebRequest greatly improves the performance.
    – codewario
    Commented Jan 28, 2020 at 17:58
2

Unfortunately the progress bar of Invoke-WebRequest is slowing file download a lot on Windows Powershell 5.1 (the version included in Windows OS). It's much faster on later Powershell versions (I tested it on Powershell 7.3).

IMO, if you are forced to use Windows Powershell then the best way is to use curl since it's included on Windows by default now. Just be aware that by default Windows Powershell has alias named curl for Invoke-WebRequest so to run curl program you need to write curl.exe to tell Windows Powershell that you don't want to use curl alias.

This command takes 11 minutes on Windows Powershell 5.1 and 23 seconds on Powershell 7.3:

Invoke-WebRequest -Verbose -Uri "https://download.visualstudio.microsoft.com/download/pr/7c048383-52b1-47cb-91d1-acfaf1a3fcc9/ea510c0bfa44f33cc3ddea79090a51e1/dotnet-sdk-6.0.410-win-x64.exe" -OutFile ".\dotnet-sdk-6.0.410-win-x64.exe"

and this takes 15 seconds:

curl.exe -fSLo .\dotnet-sdk-6.0.410-win-x64.exe https://download.visualstudio.microsoft.com/download/pr/7c048383-52b1-47cb-91d1-acfaf1a3fcc9/ea510c0bfa44f33cc3ddea79090a51e1/dotnet-sdk-6.0.410-win-x64.exe
-1

I had the same issue, I replace Invoke-WebRequest by CURL.exe call using Invoke-Expression and a command line in my script

went from 10 min to 40 sec of download

1
  • 1
    This does not answer why one method is faster than the other
    – Nico Haase
    Commented Jan 30 at 15:50
-1

$ProgressPreference = 'SilentlyContinue' I got this down from 30min to ~40 seconds.

Unbelievable, Microsoft, unbelievable...

1
  • That's what two existing answers say already. Commented Jul 3 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.