2

Iam trying to login to a web page using pythons requests module. I captured the post request when submitting the Login form using chrome as well as rechecking the code of that form. (Website as well as the parameters are mentioned in the code snippet below)

Based on someones post i was able to create a python snippet that tries to login and access protected content but for some reason this is not working as intended.

Here is my python snippet:

import requests

LOGIN = 'https://www.hackthis.co.uk/index.php'
PROTECTED_PAGE = 'https://www.hackthis.co.uk/news'
payload = {
    'username': 'VALIDUSERNAME',
    'password': 'VALIDPASSWORD'
}

with requests.session() as s:
    s.post(LOGIN, data=payload)
    response = s.get(PROTECTED_PAGE)
    print(response.text)

Even though i send the post request with the required payload iam not able to access protected content. The response HTML code contains the login form as well as a note that iam not allowed to access that content without logging in.

What do i miss here?

Edit:

Thanks for the very helpful answers! Pretty stupid mistake was made. I forgot the ?login part of the URL. Here is a working code snippet for subsequent questions on that topic (tested, works as intended):

import requests

    LOGIN = 'https://www.hackthis.co.uk/index.php?login'
    PROTECTED_PAGE = 'https://www.hackthis.co.uk/news'
    payload = {
        'username': 'VALIDUSERNAME',
        'password': 'VALIDPASSWORD'
    }

    with requests.session() as s:
        s.post(LOGIN, data=payload)
        response = s.get(PROTECTED_PAGE)
        print(response.text)
1
  • hi there Peepe - wow thanks for the heads up. It is exactly what i am workin on: so i have to admit: many thanks for sharing your solution with us - i am currently ironing out a solution for login to the forums here: wordpress.org/support/forums i think that with your code i finally will be able to get there. There is a way to achive this. Again - many thanks diear Shane. - yours Zero
    – zero
    Jun 20, 2020 at 21:00

2 Answers 2

2

About Web Authentication

Most websites use authentication mechanisms in order to make the content they want accessible only to users that provide valid credentials. To make this more simple imagine that you want to enter a museum. You can enter the ticket booth without anyone asking you for anything, but you need to have a ticket in order to see the inside. With that in mind you can think of the website/server as the ticket booth, the credentials as the money to buy the ticket and the ticket itself should be used in order enter different parts of the museum. To bring this example back to web services, I will try to explain two of the techniques that are used for authentication

Cookie Authentication

Cookies are small pieces of (usually encrypted) data that the website is sending and are stored to your computer while you are browsing. The cookie authentication process is as follows:

  1. User sends his credentials to web server (e.g. username/password)
  2. Web Server validates these credentials and then creates and stores a session with a specific identifier (sessionID)
  3. Web Server places a cookie on the User's browser with that sessionID
  4. Any subsequent request the user makes upon protected parts of the website, the server checks the sessionID in the cookie and if it is valid, returns the private information
  5. Once the user logs out the session is destroyed and any requests with this sessionID will be invalid

Although this technique is great it has some security issues

Token Authentication

This technique is similar to cookies but it simulates our museum ticket example better. A token is an encrypted piece of data that contains private information. It's important to note that tokens usually have an expiry date. The authentication process is as follows:

  1. User sends his credentials to web server (e.g. username/password)
  2. Web Server validates these credentials and then creates a token that contains information about the user. In my applications I usually include the ID of the user that is stored in my database. Note that the server doesn't have to store the token.
  3. Web Server return the token to the user
  4. Any subsequent request the user makes, should include the token
  5. Once the user logs out the token should be destroyed from the browser. It's important to note that the server doesn't have to do anything when the user logs out and the token is active until it's expiration date.

Using an HTTP requests tool like Postman or Curl, you can see that this website uses the cookie technique and the login URL is index.php?login.

TL;DR

Your code should work just fine, the only problem is the URL you are using for the login.

import requests

LOGIN = 'https://www.hackthis.co.uk/index.php?login'
PROTECTED_PAGE = 'https://www.hackthis.co.uk/news'
payload = {
    'username': 'VALIDUSERNAME',
    'password': 'VALIDPASSWORD'
}

with requests.session() as s:
    s.post(LOGIN, data=payload)
    response = s.get(PROTECTED_PAGE)
    print(response.text)
2
  • Thanks for the answer and explanation. I made a pretty stupid mistake up there.
    – Peepe
    Aug 17, 2016 at 13:17
  • hi there Kodzsd: - wow thanks for the heads up and this very intersting solution:; you privide great help. This is encouraging me. So i have to say a big big thank you.. It is exactly what i am workin on: so i have to admit: many thanks for sharing your solution with us - i am currently ironing out a solution for login to the forums here: wordpress.org/support/forums i think that with your code i finally will be able to get there. There is a way to achive this. Again - many thanks diear Shane. - yours Zero
    – zero
    Jun 20, 2020 at 21:02
1

The post is to https://www.hackthis.co.uk/index.php?login, notice the ?login at the end:

LOGIN = 'https://www.hackthis.co.uk/index.php?login'
PROTECTED_PAGE = 'https://www.hackthis.co.uk/news'
payload = {
    'username': 'uname',
    'password': 'pass'
}

with requests.session() as s:
    r = s.post(LOGIN, data=payload)
    response = s.get(PROTECTED_PAGE)
    print(response.text)

You can see the post url in developer tools/firebug under the network tab:

enter image description here

Or in the html you can see action="?login" in the form:

<form id="login_form" action="?login" method="POST">
2
  • Thanks for the answer. I made a pretty stupid mistake.
    – Peepe
    Aug 17, 2016 at 13:16
  • @Peepe, no worries, when scraping chrome tools, firebug etc.. are essential tools so you can see exactly how the requests are being made. Aug 17, 2016 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.