181

I am using the requests module. I have figured out how to submit data to a login form on a website and retrieve the session key, but I can't see an obvious way to use this session key in subsequent requests. Can someone fill in the ellipsis in the code below or suggest another approach?

>>> import requests
>>> login_data =  {'formPosted': '1', 'login_email': '[email protected]', 'password': 'pw'}
>>> r = requests.post('https://localhost/login.py', login_data)
>>> 
>>> r.text
'You are being redirected <a href="profilePage?_ck=1349394964">here</a>'
>>> r.cookies
{'session_id_myapp': '127-0-0-1-825ff22a-6ed1-453b-aebc-5d3cf2987065'}
>>> 
>>> r2 = requests.get('https://localhost/profile_data.json', ...)

8 Answers 8

314

You can easily create a persistent session using:

s = requests.Session()

After that, continue with your requests as you would:

s.post('https://localhost/login.py', login_data)
# logged in! cookies saved for future requests.
r2 = s.get('https://localhost/profile_data.json', ...)
# cookies sent automatically!
# do whatever, s will keep your cookies intact :)

For more about Sessions: https://requests.readthedocs.io/en/latest/user/advanced/#session-objects

6
  • 4
    Any ways to save Session itself between script runs?
    – Gtx
    Dec 10, 2015 at 18:37
  • 17
    Can pickle.dump session cookies to a file like pickle.dump(session.cookies._cookies, file) and pickle.load to session like follows cookies = pickle.load(file) cj = requests.cookies.RequestsCookieJar() cj._cookies = cookies and session.cookies = cj
    – Cyril
    Apr 13, 2016 at 14:42
  • 1
    what if I involve proxy?
    – brainLoop
    Nov 1, 2018 at 11:54
  • 5
    For the requests sent to localhost, there could be troubles with login and other cookies returned by web server, if they contain incorrect domain property value. For localhost, web server should return cookies with domain property set to localhost.local, otherwise the cookie will not be applied to the session. In that case use 127.0.0.1 instead of localhost Apr 30, 2020 at 19:11
  • 3
    @SergeyNudnov Many thanks for your comment I wasted a lot of time trying to figure out why session does not handle cookies correctly. Changing domain from localhost to localhost.local solved the problem. Thanks again.
    – Pulkownik
    Jun 3, 2020 at 19:32
47

The other answers help one to understand how to maintain such a session. Additionally, I want to provide a class which keeps the session maintained over different runs of a script (with a cache file). This means a proper "login" is only performed when required (after a timeout or when no session exists in cache). Also it supports proxy settings over subsequent calls to 'get' or 'post'.

It is tested with Python3.

Use it as a basis for your own code. The following snippets are released with GPL v3

import pickle
import datetime
import os
from urllib.parse import urlparse
import requests    

class MyLoginSession:
    """
    a class which handles and saves login sessions. It also keeps track of proxy settings.
    It does also maintine a cache-file for restoring session data from earlier
    script executions.
    """
    def __init__(self,
                 loginUrl,
                 loginData,
                 loginTestUrl,
                 loginTestString,
                 sessionFileAppendix = '_session.dat',
                 maxSessionTimeSeconds = 30 * 60,
                 proxies = None,
                 userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1',
                 debug = True,
                 forceLogin = False,
                 **kwargs):
        """
        save some information needed to login the session

        you'll have to provide 'loginTestString' which will be looked for in the
        responses html to make sure, you've properly been logged in

        'proxies' is of format { 'https' : 'https://user:pass@server:port', 'http' : ...
        'loginData' will be sent as post data (dictionary of id : value).
        'maxSessionTimeSeconds' will be used to determine when to re-login.
        """
        urlData = urlparse(loginUrl)

        self.proxies = proxies
        self.loginData = loginData
        self.loginUrl = loginUrl
        self.loginTestUrl = loginTestUrl
        self.maxSessionTime = maxSessionTimeSeconds
        self.sessionFile = urlData.netloc + sessionFileAppendix
        self.userAgent = userAgent
        self.loginTestString = loginTestString
        self.debug = debug

        self.login(forceLogin, **kwargs)

    def modification_date(self, filename):
        """
        return last file modification date as datetime object
        """
        t = os.path.getmtime(filename)
        return datetime.datetime.fromtimestamp(t)

    def login(self, forceLogin = False, **kwargs):
        """
        login to a session. Try to read last saved session from cache file. If this fails
        do proper login. If the last cache access was too old, also perform a proper login.
        Always updates session cache file.
        """
        wasReadFromCache = False
        if self.debug:
            print('loading or generating session...')
        if os.path.exists(self.sessionFile) and not forceLogin:
            time = self.modification_date(self.sessionFile)         

            # only load if file less than 30 minutes old
            lastModification = (datetime.datetime.now() - time).seconds
            if lastModification < self.maxSessionTime:
                with open(self.sessionFile, "rb") as f:
                    self.session = pickle.load(f)
                    wasReadFromCache = True
                    if self.debug:
                        print("loaded session from cache (last access %ds ago) "
                              % lastModification)
        if not wasReadFromCache:
            self.session = requests.Session()
            self.session.headers.update({'user-agent' : self.userAgent})
            res = self.session.post(self.loginUrl, data = self.loginData, 
                                    proxies = self.proxies, **kwargs)

            if self.debug:
                print('created new session with login' )
            self.saveSessionToCache()

        # test login
        res = self.session.get(self.loginTestUrl)
        if res.text.lower().find(self.loginTestString.lower()) < 0:
            raise Exception("could not log into provided site '%s'"
                            " (did not find successful login string)"
                            % self.loginUrl)

    def saveSessionToCache(self):
        """
        save session to a cache file
        """
        # always save (to update timeout)
        with open(self.sessionFile, "wb") as f:
            pickle.dump(self.session, f)
            if self.debug:
                print('updated session cache-file %s' % self.sessionFile)

    def retrieveContent(self, url, method = "get", postData = None, **kwargs):
        """
        return the content of the url with respect to the session.

        If 'method' is not 'get', the url will be called with 'postData'
        as a post request.
        """
        if method == 'get':
            res = self.session.get(url , proxies = self.proxies, **kwargs)
        else:
            res = self.session.post(url , data = postData, proxies = self.proxies, **kwargs)

        # the session has been updated on the server, so also update in cache
        self.saveSessionToCache()            

        return res

A code snippet for using the above class may look like this:

if __name__ == "__main__":
    # proxies = {'https' : 'https://user:pass@server:port',
    #           'http' : 'http://user:pass@server:port'}

    loginData = {'user' : 'usr',
                 'password' :  'pwd'}

    loginUrl = 'https://...'
    loginTestUrl = 'https://...'
    successStr = 'Hello Tom'
    s = MyLoginSession(loginUrl, loginData, loginTestUrl, successStr, 
                       #proxies = proxies
                       )
    
    res = s.retrieveContent('https://....')
    print(res.text)

    # if, for instance, login via JSON values required try this:
    s = MyLoginSession(loginUrl, None, loginTestUrl, successStr, 
                       #proxies = proxies,
                       json = loginData)
    
7
  • 11
    This is a great answer, it's strangely hard to search for this solution as well.
    – duality
    Jun 3, 2017 at 19:08
  • 1
    Shouldn't be implemented as part of the request module? Dec 20, 2019 at 6:27
  • It does use the requests module. How would you go about implementing it as part of the module? or how do you mean @user1602 ?
    – DomTomCat
    Mar 10, 2020 at 15:41
  • Excellent answer! If I may ask, knowing this way long time ago you posted this. Can you/someone explain the line: res.text.lower().find(self.loginTestString.lower()) < 0 ? Doesnt that mean that it looks for a string and returns 0 if it doesnt find a match?
    – Vichtor
    Sep 17, 2020 at 19:02
  • 1
    the find method of a string returns -1 if the search string was not found in it (i.e. here: looking for self.loginTestString.lower() within res.text.lower())
    – DomTomCat
    Sep 18, 2020 at 8:02
22

Check out my answer in this similar question:

python: urllib2 how to send cookie with urlopen request

import urllib2
import urllib
from cookielib import CookieJar

cj = CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
# input-type values from the html form
formdata = { "username" : username, "password": password, "form-id" : "1234" }
data_encoded = urllib.urlencode(formdata)
response = opener.open("https://page.com/login.php", data_encoded)
content = response.read()

EDIT:

I see I've gotten a few downvotes for my answer, but no explaining comments. I'm guessing it's because I'm referring to the urllib libraries instead of requests. I do that because the OP asks for help with requests or for someone to suggest another approach.

2
  • 3
    I am not one of your down-voters, but as a guess, many readers are probably glossing the OP’s last sentence as “Can someone fill in the ellipsis in the code below or suggest another approach [with the requests library that would involve more major surgery to my code than merely filling in the ellipses with something else].” — but that is just a guess on my part. Feb 21, 2015 at 19:53
  • 11
    As OP, I can say that your answer provides a useful alternative. If only to demonstrate that requests offers a simple and high-level solution to a problem that would otherwise take 3 libraries to implement.
    – ChrisGuest
    Jun 9, 2015 at 4:22
10

The documentation says that get takes in an optional cookies argument allowing you to specify cookies to use:

from the docs:

>>> url = 'http://httpbin.org/cookies'
>>> cookies = dict(cookies_are='working')

>>> r = requests.get(url, cookies=cookies)
>>> r.text
'{"cookies": {"cookies_are": "working"}}'

http://docs.python-requests.org/en/latest/user/quickstart/#cookies

10

Upon trying all the answers above, I found that using "RequestsCookieJar" instead of the regular CookieJar for subsequent requests fixed my problem.

import requests
import json

# The Login URL
authUrl = 'https://whatever.com/login'

# The subsequent URL
testUrl = 'https://whatever.com/someEndpoint'

# Logout URL
testlogoutUrl = 'https://whatever.com/logout'

# Whatever you are posting
login_data =  {'formPosted':'1', 
               'login_email':'[email protected]', 
               'password':'pw'
               }

# The Authentication token or any other data that we will receive from the Authentication Request. 
token = ''

# Post the login Request
loginRequest = requests.post(authUrl, login_data)
print("{}".format(loginRequest.text))

# Save the request content to your variable. In this case I needed a field called token. 
token = str(json.loads(loginRequest.content)['token'])  # or ['access_token']
print("{}".format(token))

# Verify Successful login
print("{}".format(loginRequest.status_code))

# Create your Requests Cookie Jar for your subsequent requests and add the cookie
jar = requests.cookies.RequestsCookieJar()
jar.set('LWSSO_COOKIE_KEY', token)

# Execute your next request(s) with the Request Cookie Jar set
r = requests.get(testUrl, cookies=jar)
print("R.TEXT: {}".format(r.text))
print("R.STCD: {}".format(r.status_code))

# Execute your logout request(s) with the Request Cookie Jar set
r = requests.delete(testlogoutUrl, cookies=jar)
print("R.TEXT: {}".format(r.text))  # should show "Request Not Authorized"
print("R.STCD: {}".format(r.status_code))  # should show 401
1
  • 1
    Thank you @jim-chertkov for this! Great answer even years later! I was able to use and understand this.
    – JayRizzo
    Jun 5, 2020 at 20:40
3

Save only required cookies and reuse them.

import os
import pickle
from urllib.parse import urljoin, urlparse

login = '[email protected]'
password = 'secret'
# Assuming two cookies are used for persistent login.
# (Find it by tracing the login process)
persistentCookieNames = ['sessionId', 'profileId']
URL = 'http://example.com'
urlData = urlparse(URL)
cookieFile = urlData.netloc + '.cookie'
signinUrl = urljoin(URL, "/signin")
with requests.Session() as session:
    try:
        with open(cookieFile, 'rb') as f:
            print("Loading cookies...")
            session.cookies.update(pickle.load(f))
    except Exception:
        # If could not load cookies from file, get the new ones by login in
        print("Login in...")
        post = session.post(
            signinUrl,
            data={
                'email': login,
                'password': password,
            }
        )
        try:
            with open(cookieFile, 'wb') as f:
                jar = requests.cookies.RequestsCookieJar()
                for cookie in session.cookies:
                    if cookie.name in persistentCookieNames:
                        jar.set_cookie(cookie)
                pickle.dump(jar, f)
        except Exception as e:
            os.remove(cookieFile)
            raise(e)
    MyPage = urljoin(URL, "/mypage")
    page = session.get(MyPage)
2
  • How do we get persistentCookieNames = ['sessionId', 'profileId']? Sep 19, 2021 at 22:01
  • To find cookie ID use browser 'Developer tools'. It is mentioned in the comments above. Sep 21, 2021 at 6:22
2

snippet to retrieve json data, password protected

import requests

username = "my_user_name"
password = "my_super_secret"
url = "https://www.my_base_url.com"
the_page_i_want = "/my_json_data_page"

session = requests.Session()
# retrieve cookie value
resp = session.get(url+'/login')
csrf_token = resp.cookies['csrftoken']
# login, add referer
resp = session.post(url+"/login",
                  data={
                      'username': username,
                      'password': password,
                      'csrfmiddlewaretoken': csrf_token,
                      'next': the_page_i_want,
                  },
                  headers=dict(Referer=url+"/login"))
print(resp.json())
2

This will work for you in Python;

# Call JIRA API with HTTPBasicAuth
import json
import requests
from requests.auth import HTTPBasicAuth

JIRA_EMAIL = "****"
JIRA_TOKEN = "****"
BASE_URL = "https://****.atlassian.net"
API_URL = "/rest/api/3/serverInfo"

API_URL = BASE_URL+API_URL

BASIC_AUTH = HTTPBasicAuth(JIRA_EMAIL, JIRA_TOKEN)
HEADERS = {'Content-Type' : 'application/json;charset=iso-8859-1'}

response = requests.get(
    API_URL,
    headers=HEADERS,
    auth=BASIC_AUTH
)

print(json.dumps(json.loads(response.text), sort_keys=True, indent=4, separators=(",", ": ")))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.