I'm trying to download files from http://www.oracle.com/technetwork/server-storage/developerstudio/downloads/index.html in a headless context. I have an account (they are free), but the site really doesn't make it easy, apparently it uses a chain of javascript forms/redirection. With Firefox I can use the element inspector, copy the url of the file as cURL when the download starts, and use it in a headless machine to download the file, but so far all my attempts to get the file only in the headless machine have failed.
I've managed to get the login with:
#!/usr/bin/env python3
username="<my username>"
password="<my password>"
import requests
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.PHANTOMJS
caps["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0"
driver = webdriver.PhantomJS("/usr/local/bin/phantomjs")
driver.set_window_size(1120, 550)
driver.get("http://www.oracle.com/technetwork/server-storage/developerstudio/downloads/index.html")
print("loaded")
driver.find_element_by_name("agreement").click()
print("clicked agreement")
driver.find_element_by_partial_link_text("RPM installer").click()
print("clicked link")
driver.find_element_by_id("sso_username").send_keys(username)
driver.find_element_by_id("ssopassword").send_keys(password)
driver.find_element_by_xpath("//input[contains(@title,'Please click here to sign in')]").click()
print("submitted")
print(driver.get_cookies())
print(driver.current_url)
print(driver.page_source)
driver.quit()
I suspect the login worked, because in the cookies I see some data associated with my username, but in Firefox submitting the form results in a download starting after 3-4 redirections, while here I get nothing and the page_source
and current_url
still belong to the login page.
Maybe the site is actively blocking this kind of use, or maybe I'm doing something wrong. Any idea how to actually download the file?
Firefox element inspector
why don't you tryHeadless Firefox
browser?