12

I'm trying to use Selenium to log into a website and then use urllib2 to make RESTy requests. In order for it to work though, I need urllib2 to be able to use the same session Selenium used.

The logging in with selenium worked great and I can call

self.driver.get_cookies()

and I have a list of all the cookies selenium knows about, and it ends up looking a little something like this:

[{u'domain': u'my.awesome.web.app.local',
  u'expiry': 1319230106,
  u'name': u'ci_session',
  u'path': u'/',
  u'secure': False,
  u'value': u'9YEz6Qs9rNlONzXbZPZ5i9jm2Nn4HNrbaCJj2c%2B...'
}]

I've tried a few different ways to to use the cooky in urllib2, I think this one looks the best:

# self.driver is my selenium driver
all_cookies = self.driver.get_cookies()
cp = urllib2.HTTPCookieProcessor()
cj = cp.cookiejar
for s_cookie in all_cookies:
    cj.set_cookie(
        cookielib.Cookie(
            version=0
            , name=s_cookie['name']
            , value=s_cookie['value']
            , port='80'
            , port_specified=False
            , domain=s_cookie['domain']
            , domain_specified=True
            , domain_initial_dot=False
            , path=s_cookie['path']
            , path_specified=True
            , secure=s_cookie['secure']
            , expires=None
            , discard=False
            , comment=None
            , comment_url=None
            , rest=None
            , rfc2109=False
        )
    )
opener = urllib2.build_opener(cp)
response = opener.open(url_that_requires_a_logged_in_user)
response.geturl()

It does not work though.

That last call to response.geturl() returns the login page.

Am I missing something?

Any ideas for how should go about looking for the problem?

Thanks.

2 Answers 2

15

I was able to overcome this problem by using the requests library instead. I iterated over the cookies from selenium, and then passed them in a simple dictionary with name:value pairs.

all_cookies = self.driver.get_cookies()

cookies = {}  
for s_cookie in all_cookies:
    cookies[s_cookie["name"]]=s_cookie["value"]

r = requests.get(my_url,cookies=cookies)
0

You may try as below.

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
f_opener = opener.open(url_that_requires_a_logged_in_user)
content = f_opener.read()
2
  • what is "cj" from opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) ?
    – Tito
    Commented Nov 8, 2017 at 16:00
  • cj = cp.cookiejar. It's in the code of the original question.
    – ronnefeldt
    Commented Nov 12, 2017 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.