4

I've been trying to write a script that will web scrape a certain field/label on a render website. The website is rendered with a search parameter that i have in a column list in Excel. Appox 20 items that will grow. After researching how to use vbscripts to web scrape, the issue I am running into is how to do this 20 times without it breaking. Here is my code.

Excel Column

1492565
1528417
1529041
1530688
1492038
1492319
1492972
1508824
1513351
1514724
1514750
1518526
1520627
1520706
1520979
1523367
1523563

Script: Main sub(gets user/pass from excel inputfields, sets up loop through rows on a specific column. Only spits back a msgbox until I can get the looping to work. Then I'll output it to another column

Sub WebScraper()
    'itg on mainWS start row 6, column 5
    'itg status column column 19
    'declare variables
    Dim url As String
    Dim ITGNUMBER As Long
    Dim user As String
    Dim pwd As String
    
    'set variables
    url = "https://website/itg/web/knta/crt/RequestDetail.jsp?REQUEST_ID="
    Set oMainWS = ActiveWorkbook.Worksheets("MainWS")
    Set oITGStatusWS = ActiveWorkbook.Worksheets("ITGStatus")
    user = ""
    pwd = ""
    
    user = oITGStatusWS.ITGusername.Value
    pwd = oITGStatusWS.ITGpassword.Value
    
    If user = "" Or pwd = "" Then
        MsgBox ("You must enter username/password before continuing")
        Exit Sub
    End If
    
    'log in
    Set objIE = FirstIEConnect(user, pwd)
    
    'start row is 6
    RowCounter = 58
    ColumnCounter = 5
    ITGStatusColumn = 16
    Do Until IsEmpty(oMainWS.Cells(RowCounter, 5).Value)
        'get ITG number
        currentITGNumber = oMainWS.Cells(RowCounter, 5).Value
        MsgBox (currentITGNumber)
        'get remote status
        currentITGStatus = getITGStatusFunction(objIE.Application, Str(currentITGNumber))
        MsgBox (currentITGStatus)
        
        'paste into column 19
        'oMainWS.Cells(RowCounter, 19).Value = currentITGStatus
        
        'increment counter
        RowCounter = RowCounter + 1
        currentITGStatus = ""
        currentITGNumber = ""
    Loop
    quitIE (objIE.Application)
End Sub

Sub to quitIE object cleaning, there is a javascript function to log the user out.

Sub quitIE(obj As Object)
    obj.Navigate ("javascript: closeChildWindowsAndLogout();")
    obj.Quit
End Sub

Got this sub from google, used to wait for IE object to become ready. This actually fails a lot in the loop. On Do While IE.Busy:Loop. Just hangs.

Sub Wait(obj As Object)
    Do While obj.Busy: Loop
    Do While obj.readyState <> 4:   Loop
     Application.Wait (Now + TimeValue("0:00:01"))
End Sub

The website requires logon, user/pwd is from first sub. This sub creates the IE object, navigates to the log on page and inserts user/pwd into Document.logon.UserName and Document.logon.Password. Finally submits it.

Function FirstIEConnect(user As String, pwd As String)
    loginURL = "https://website/Logon.jsp"
    Set IE = CreateObject("InternetExplorer.Application")
    IE.Visible = False
    IE.Navigate loginURL
    Wait (IE.Application)
    With IE.Document.logon
      .UserName.Value = user
      .Password.Value = pwd
      .submit
    End With

    Set FirstIEConnect = IE
End Function

this is the actual web scraping function. requires user to be logged into IE object from above. enters num into url GET request to render the specific page. Finally grabs responseText based on ElementID

Function getITGStatusFunction(obj, num)
    On Error Resume Next
    'set url and num
    url = "https://website/RequestDetail.jsp?REQUEST_ID=" & num
  
    obj.Navigate url
    Wait (obj.Application)
    responseText = obj.Document.getElementByID("DRIVEN_STATUS_ID").innerHTML
   
    getStatusFunction = responseText
    
End Function

Again, The issue is that I keep getting object errors when trying to pass the IE object from different subs and functions.

Expectation: I want the script to loop through the column information in excel which contains the unique numbers. Take those numbers one by one and append them to the search URL one by one. Once the page loads scrape the ElementID(DRIVEN_STATUS_ID). Eventually taking that value and outputting it to another column.

13
  • 1
    See if this link help? stackoverflow.com/questions/8798260/… Aug 31, 2012 at 14:55
  • 1
    I tried using the MSXML2.ServerXMLHTTP object at first, but it fails when trying to log in to the page first.
    – Jim
    Aug 31, 2012 at 14:58
  • 1
    Getting a Runtime error 424: object needed on line "Do While IE.Busy: Loop"
    – Jim
    Aug 31, 2012 at 15:00
  • 1
    Set FirstIEConnect = IE. Missing Set in that functions last line. Aug 31, 2012 at 15:07
  • 2
    Wait IE without the parentheses. You only use parens when calling a function (or if you use Call) Also: quitIE IE Aug 31, 2012 at 16:16

3 Answers 3

1

Wait (IE) is passing IE.Name instead of the object. To pass the object, use IE.Application.

Tim Williams comment is also correct, leaving out the parenthesis will also result in the object being passed. So Wait IE instead of Wait (IE) will also work. But I think it is safer to use Wait IE.Application just to be explicit. Obviously, you decide.

If you were to update the Wait Function, it still wouldn't fix the issue, but it would be more obvious where the problem is. Function Wait(IE as object) fails when you try to call the Function stating that an object is required.

This should be obvious as this point, but do not pass IE, pass IE.Application in all instances unless you actually only want the name of the object.

As an aside, the Set variable = IE lines are fine because the keyword Set makes it clear you want the object.

1
  • 1
    You cannot reference the elements of the HTML document that way. I don't have much experience with this but, try something like this: IE.Document.GetElementByID("UserName").value = user and IE.Document.GetElementByID("Password").value = pwd I'm not sure of the equivalent of the submit and am unwilling to make a fake page just to experiment. Probably someone else can help you there if you can't figure it out, but I would consider making a new question at that point.
    – Daniel
    Aug 31, 2012 at 17:59
1

Just a shot in the dark but this sounds like the HTML has not fully loaded when you try to set the value. Try using a longer wait value. Usually 3-4 seconds will do the trick.

Or you could create a new variable and set a loop until the object is set:

Dim obj_testVariable as Object

Do Until Not obj_testVariable Is Nothing
   set obj_testVariable = obj.Document.getElementByID("DRIVEN_STATUS_ID")
Loop

responseText = obj_testVariable.innerHTML

set obj_testVariable = Nothing

(untested)

1

This is my IEWait function:

Dim IE As Object

Function IEWait(t As Long)

  Do While IE.Busy
    Application.Wait DateAdd("s", t, Now)
  Loop

End Function

You can then call this function in your subroutine:

IEWait (1)
Do Until IE.readyState = READYSTATE_COMPLETE: DoEvents: Loop

Use only IEWait (1) when you just need to submit something. Use both lines above when you need to wait for a new page to be loaded. This will take some time but just play around. Good luck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.