2

I wrote some screen-scraping code in PowerShell and was surprised that it took around 30 seconds to parse a few HTML tables. I stripped it down to try and figure out where all the time was being spent, and it seems to be in the getElementsByTagName calls.

I've included a script below which on both my home desktop, my work desktop and my home slate, takes around 1-2 seconds for each iteration (full results pasted below). However, other people in the PowerShell community are reporting far shorter times (only several milliseconds for each iteration).

I'm struggling to find any way of narrowing down the problem, and there doesn't seem to be a pattern to the OS/PS/.NET/IE versions.

The desktop I'm currently running it on is a brand new Windows 8 install with only PS3 and .NET 4.5 installed (and all Windows Update patches). No Visual Studio. No PowerShell profile.

$url = "http://www.icy-veins.com/restoration-shaman-wow-pve-healing-gear-loot-best-in-slot"
$response = (iwr $url).ParsedHtml

# Loop through the h2 tags
$response.body.getElementsByTagName("h2") | foreach {

    # Get the table that comes after the heading
    $slotTable = $_.nextSibling

    # Grab the rows from the table, skipping the first row (column headers)
    measure-command { $rows = $slotTable.getElementsByTagName("tr") | select -Skip 1 } | select TotalMilliseconds
}

Results from my desktop (the work PC and slate give near identical results):

TotalMilliseconds
-----------------
        1575.7633
        2371.5566
        1073.7552
        2307.8844
        1779.5518
        1063.9977
        1588.5112
        1372.4927
        1248.7245
        1718.3555
         3283.843
        2931.1616
        2557.8595
        1230.5093
         995.2934

However, some people in the Google+ PowerShell community reported results like this:

 TotalMilliseconds
 -----------------
           76.9098
          112.6745
           56.6522
          140.5845
           84.9599
           48.6669
           79.9283
           73.4511
           94.0683
           81.4443
           147.809
          139.2805
          111.4078
           56.3881
           41.3386

I've tried both PowerShell ISE and a standard console, no difference. For the work being done, these times seem kinda excessive, and judging by the posts in the Google+ community, it can go quicker!

8
  • Is it possible the speed is related to the version of IE installed on the machine?
    – Pete
    Jan 7, 2013 at 18:52
  • getting the same results as you. actually worse, though it may be because I have 10-15 webpages up atm
    – Frode F.
    Jan 7, 2013 at 19:10
  • @Pete I thought that, but both of the sets of times posted are from Windows 8 (IE 10). Jan 7, 2013 at 19:50
  • @Graimer Yikes! :-/ I can't find anything online suggesting others have this problem, seems really strange! Jan 7, 2013 at 19:51
  • I did notice when running with Fiddler open (wondering if it was fetching all the external resources - my internet is slow) there was a request to paypalobjects. No other external resources were fetched, however. Couldn't explain that! Jan 7, 2013 at 19:52

3 Answers 3

5

See my comment in: https://connect.microsoft.com/PowerShell/feedback/details/778371/invoke-webrequest-getelementsbytagname-is-incredibly-slow-on-some-machines#tabs

I got the same slowness running the script in 64 bits, but when running in 32bits mode, everything is very fast !

Lee Holmes was able to reproduce the issue, and here is his writeup

"The issue is that he’s piping COM objects into another cmdlet – in this case, Select-Object. When that happens, we attempt to bind parameters by property name. Enumerating property names of a COM object is brutally slow – so we’re spending 86% of our time on two very basic CLR API calls:

(…) // Get the function description from a COM type typeinfo.GetFuncDesc(index, out pFuncDesc); (…) // Get the function name from a COM function description typeinfo.GetDocumentation(funcdesc.memid, out strName, out strDoc, out id, out strHelp); (…)

We might be able to do something smart here with caching.

A workaround is to not pipe into Select-Object, but instead use language features:

# Grab the rows from the table, skipping the first row (column headers)
$allRows = @($slotTable.getElementsByTagName("tr"))
$rows = $allRows[1..$allRows.Count]

"

1
  • 1
    Link is broken.
    – Mike Cole
    Feb 11, 2018 at 16:50
5

Did you try disabling progress?

$ProgressPreference = "SilentlyContinue"

In my case this solved serious performance problems with Invoke-WebRequest.

1
  • I didn't try that; though it still doesn't explain the massive difference between different peoples machines :( Dec 20, 2013 at 11:40
0

I have noticed the same phenomenon on a new Windows 8 system. I have a 104MB file stored on a remote web server that takes from 2 to 5 minutes to download depending on server load, bandwidth, etc. Tried through FF, IE10, Chrome, cURL, and even set up a test FTP server and tried with Windows FTP command. I consistently get the same results.

However, the same file (this is not an exaggeration) takes nearly 3 HOURS to transfer.

$file = 'C:\User\me\Desktop\file.mp4'
$site = 'http://my.site/file.mp4'
Invoke-WebRequest $site -Method Get -OutFile $file

Seems pretty cut-and-dry -- site's not https, file's not an executable or anything that Windows might see as 'unsafe', and there's no authentication needed. It just takes forever to finish.

I thought that my AV's real-time scanning might be the culprit, but disabling that made no discernible difference. Is there maybe was some memory allocation at work here? Similar to how adding the -ReadCount option to Get-Content makes getting the content of large files much faster? I can't find any such option for Invoke-WebRequest.

2
  • Very strange; though I believe in my case the file was already downloaded, and just the parsing was slow, but who knows. What times so you see when running my code above? Feb 2, 2013 at 9:08
  • My results were similar to yours, though doing it via the ISE was drastically different for some lines (e.g. twice as fast or twice as slow). Tried posting results here, but can't get the line breaks to work.. Feb 2, 2013 at 23:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.