Here is my situation: I have to login to a Website and download a CSV from there, headless from a linux server. The page uses JS and does not work without it.
After some research I went with Selenium and PhantomJS. Logging in, setting the parameters for the CSV and finding the download button with Selenium/PhantomJS/Py3 was no problem, actually surprisingly enjoyable.
But clicking the download button did not do anything. After some research I found out that PhantomJS does not seem to support download-dialogs and downloads but that it is on the upcoming feature list.
So I thought I use a workaround with urllib
after I found out that the download button is just calling a REST API Url. Problem is, it only works if you're logged into the site.
So the first attempt failed as it returned: b'{"success":false,"session":"expired"}'
which makes sense as I expect Selenium and urllib to use different sessions.
So I thought I use the headers from Seleniums driver in urrlib trying this:
...
url = 'http://www.foo.com/api/index'
data = urllib.parse.urlencode({
'foopara': 'cadbrabar',
}).encode('utf-8')
headers = {}
for cookie in driver.get_cookies():
headers[cookie['name']] = cookie['value']
req = urllib.request.Request(url, data, headers)
with urllib.request.urlopen(req) as response:
page = response.read()
driver.close()
Unfortunately this yielded the same result of an expired session. Am I doing somthing wrong, is there a way around this, other suggestions or am I at a dead end? Thanks in advance.