3

I've been battling with this for hours now and read several threads on SO and other places, but have hit a dead end. I am trying to scrape a web page and one of the items I need is a text file that is accessed by clicking a button which performs a postback and responds with the text file (or at a box asking me to download it actually).

I've logged into the site by creating a requests session with login info, and I can get to the actual page where I need to download the file, but I can't seem to actually get it to respond with the text file I want and instead keep getting the HTML on the page that has the postback link.

I looked at the parameters to the postback in dev tools and copied them into a dict, and then tried to post to the same URL again with those items as the payload, but I still get the same HTML.

Here is a simplified example (the parameter strings are thousands of lines long). All the below is after successfully logging in to the main page by posting login data.

  ajaxpost={
    '__VIEWSTATE':'lotsoftext',
    '__EVENTVALIDATION':'seriously_too_much_text',
    's$maincontent$showitem':'true',
    's$maincontent$date':'Monday August 17 2015',
    's$maincontent$xyz':''
    }

    r=requests.Session()
    r.post(login_url,data=login_info)
    stuff=r.post(stuff_url,ajaxpost)
    print(stuff.text) #still html from the above site, not the text file i want

It may be worth noting that the _doPostBack function only has two explicitly visible parameters, as shown below, however when I inspect the post in dev tools i see the additional postings (the three with s$ at the beginning), so i included them as well. Also, the outrageously long text fields (__VIEWSTATE and __EVENTVALIDATION) appear to change each time I post, so I'm not sure if that is an issue or there is an easy way to pull the data directly out of the website without having to view the post parameters in dev tools.

var theForm = document.forms['form1'];
if (!theForm) {
    theForm = document.form1;
}
function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}
5
  • 1
    Did you ever solve this Solaxun? I have exactly the same __doPostBack function I need to get an excel file from.
    – Gareth
    Apr 26, 2018 at 18:37
  • 1
    @Gareth I just resort to using selenium for anything other than the most basic requests now. Anything that requires button clicking and/or javascript changing the page is better handled by selenium since it's essentially replicating a web browser's interaction w/ the page.
    – Solaxun
    Apr 26, 2018 at 19:48
  • Thanks @Solaxun. I'll do the same. Limited exposure to selenium so if you have any links that helped great otherwise I know what I'm learning tonight :)
    – Gareth
    Apr 26, 2018 at 22:30
  • Wasn't too difficult at all. Note the application/xls extension for mime type: gist.github.com/gyaresu/ef7b906e333a955370e4915a7f7646a4
    – Gareth
    Apr 27, 2018 at 0:43
  • 1
    @Gareth nice! You can also use phantomjs as the webdriver over firefox if you don’t want to see the browser performing the actions. Phantomjs will do everything behind the scenes.
    – Solaxun
    Apr 27, 2018 at 1:03

2 Answers 2

1

This looks very much like an ASP.NET website, and gosh are they irritating to parse.

The viewstate changes with every action you do on the website. Those two parameters are handled by some javascript code.

This means that you have two/three options:

  1. Extract the values with a regex
  2. Use selenium instead of requests
  3. Use a binding to V8 from your script and execute the javascript code.

If you're interested, send me a page source and I'll give you the specifics of the code.

1

Just putting this here for future cosmonauts looking for a simple Python/Selenium solution.

Best of luck.

"""
Automatically downloading Excel file from a form.

Initially because accessing __doPostBack function was too hard to script in Python.

Software installed using conda and homebrew.
 - selenium from conda
 - geckodriver from homebrew
"""
import os
from selenium import webdriver

def main():
    """ Setup Firefox profile preferences and go get em"""
    profile = webdriver.FirefoxProfile()
    profile.set_preference('browser.download.folderList', 2)
    profile.set_preference('browser.download.manager.showWhenStarting', False)
    profile.set_preference('browser.download.dir', os.getcwd())
    profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/xls')
    driver = webdriver.Firefox(profile)

    url = 'https://example.com/some_file.aspx'
    driver.get(url)
    driver.find_element_by_name("btnExportToExcel").click()
    driver.quit()


if __name__ == "__main__":
    main()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.