143

I am trying to log in to a page and access another link in the page.

I get a "405 Not Allowed" error from this attempt:

payload={'username'=<username>,'password'=<password>}
with session() as s:
    r = c.post(<URL>, data=payload)
    print(r)
    print(r.content)

I checked the post method details using Chrome developer tools and found a URL that appeard to be an API endpoint. I posted to that URL with the payload and it seemed to work; I got a response similar to what I could see in the developer.

Unfortunately, when trying to 'get' another URL after logging in, I am still getting the content from the login page. Why is the login not sticking? Should I use cookies? How?

5 Answers 5

175

You can use a session object. It stores the cookies so you can make requests, and it handles the cookies for you

s = requests.Session() 
# all cookies received will be stored in the session object

s.post('http://www...',data=payload)
s.get('http://www...')

Docs: https://requests.readthedocs.io/en/master/user/advanced/#session-objects

You can also save the cookie data to an external file, and then reload them to keep session persistent without having to login every time you run the script:

How to save requests (python) cookies to a file?

4
  • 3
    :O This is amazing! I wish I had known about this feature before. Is it storing the cookies on disk like a browser, or just holding them in memory? Commented Jun 30, 2018 at 10:16
  • 3
    It just keeps them in memory while the application is running. To save/load cookies from disk see this: stackoverflow.com/a/30441145/4411196
    – gtalarico
    Commented Jul 20, 2018 at 13:51
  • I instead of saving them, made a global variable assigned it a session and made a function so it logins if the cookie is not there and/or returns the session
    – modbender
    Commented Jul 13, 2020 at 15:06
  • for me print(s.cookies) shows that cookies are in the session but are not always passed in subsequent requests (in fact the same code works against localhost but against a test server does not) Commented Dec 30, 2021 at 10:55
111

From the documentation:

  1. get cookie from response

     url = 'http://example.com/some/cookie/setting/url'
     r = requests.get(url)
     r.cookies
    

    {'example_cookie_name': 'example_cookie_value'}

  2. give cookie back to server on subsequent request

     url = 'http://httpbin.org/cookies'
     cookies = {'cookies_are': 'working'}
     r = requests.get(url, cookies=cookies)`
    
4
  • Thanks. Actually, it seems there are no cookies that getting created. I checked the request headers and could not see any 'cookies'. Meanwhile there is one cookie created in response headers. How can i make my login stick if there is no cookie. Commented Jul 22, 2015 at 9:40
  • 2
    The normal flow for an authentication cookie is: (1) when you submit a login form, you receive a cookie in the response headers. (2) on subsequent page requests, you add the cookie to the request headers. Commented Jul 22, 2015 at 9:54
  • how do you add multiple cookies ?
    – Jitin
    Commented Oct 27, 2020 at 6:11
  • 3
    The kwarg “cookies” is a dict, you can add as many items as you want. Commented Oct 27, 2020 at 6:38
18

Summary (@Freek Wiekmeijer, @gtalarico) other's answer:

Logic of Login

  • Many resource(pages, api) need authentication, then can access, otherwise 401=Unauthorized
  • Common authentication=grant access method are:
    • cookie
    • auth header
      • Basic xxx
      • Authorization xxx

How use cookie in requests to auth

  1. first get/generate cookie
  2. send cookie for following request
  • manual set cookie in headers
  • auto process cookie by requests's
    • session to auto manage cookies
    • response.cookies to manually set cookies

use requests's session auto manage cookies

curSession = requests.Session() 
# all cookies received will be stored in the session object

payload={'username': "yourName",'password': "yourPassword"}
curSession.post(firstUrl, data=payload)
# internally return your expected cookies, can use for following auth

# internally use previously generated cookies, can access the resources
curSession.get(secondUrl)

curSession.get(thirdUrl)

manually control requests's response.cookies

payload={'username': "yourName",'password': "yourPassword"}
resp1 = requests.post(firstUrl, data=payload)

# manually pass previously returned cookies into following request
resp2 = requests.get(secondUrl, cookies= resp1.cookies)

resp3 = requests.get(thirdUrl, cookies= resp2.cookies)
2
  • re: "Many resource(pages, api) need authentication, then can access, otherwise 405 Not Allowed" This is not what the 405 status should be used for. The answer makes other wrong or incomplete statements and is redundant to the top voted one.
    – Michal M
    Commented Oct 1, 2024 at 5:47
  • @MichalM thanks for notification. I will update my answer
    – crifan
    Commented Oct 1, 2024 at 6:30
5

As others noted, Here is an example of how to add cookies as string variable to the headers parameter -

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
    "cookie": "_fbp=fb.1.1654447470850.2143140577; _ga=GA1.2.1...",
}
response = requests.get(url, headers=headers)
0

Unfortunately, the requests does not allow cookies to be passed in the headers, so this method does not work. Сookies should be passed as a separate argument r = requests.get(url, cookies=cookies)

1
  • That’s not true, you can pass {"Cookie": "..."} in the headers.
    – bfontaine
    Commented Oct 16, 2024 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.