5

Goal

Make request to http://example.com/page? using requests.get()

Problem

The question mark ("?") is automatically stripped from the request if it is the last character in the URL (eg. http://example.com/page?1 and http://example.com/page?! work, http://example.com/page? does not)

Sample code

import requests

endpoint = "http://example.com/page?"
r = requests.get(endpoint)

print(r.url) # -> "http://example.com/page"
assert r.url == endpoint # Raises AssertionError

Question

Without modifying the library, is it possible to reach the intended endpoint? Both intended solutions (if such exist) and workarounds are welcome.

Thanks!

9
  • Is there a reason to justify this need? Cause the ? flag is used in URLs to separate the 'route' from the parameters, such as https://www.google.com/search?q=stack. So if no params are passed in the request, it makes sense to ignore a trailing ?
    – MatBBastos
    Aug 13, 2021 at 16:49
  • Yes, it is needed for communicating with an old internal API. Previously a javascript library was used for making the request, but I'm exploring if the program could be ported to python 3. Aug 13, 2021 at 17:01
  • Is it being redirected? What's the status code? Aug 13, 2021 at 17:09
  • @Jim The server returns a 404 error, however the question about the limits of the requests library interest me personally Aug 13, 2021 at 17:12
  • Does the ? at the end really make a difference? As far as the endpoint being requested from the server, the ? is doing nothing. This is not a limit of the requests library. requests uses something to parse the URL with, and since the ? means nothing in this instance, it is removed. Aug 13, 2021 at 17:19

2 Answers 2

8

This is not possible with the requests library. URLs passed into requests are parsed by urllib3.util.url.parse_url() into separate parts:

scheme
auth
host
port
path
query
fragment

The logic for getting the query part of a URL assumes that the querystring starts after ?, but since there is nothing after the question mark, it gives a blank query. The URL is then reconstructed as a string when you print r.url. That is why the URL does not have the trailing question mark.

I found that the behavior you are looking for is possible with urllib.request, though. Here's an example:

import urllib.request, urllib.error

try:
    response = urllib.request.urlopen("http://example.com/page?") 
    print(response.url)  # -> http://example.com/page?
except urllib.error.HTTPError as e:
    print(e.url)  # -> http://example.com/page?
    print(e.code) # -> 404

I have surrounded the request in a try/except because if the page you are trying to get gives a 404, urllib will raise an error, where requests will simply put up with it.

0

not an answer but more on requests behaviour

I can give you http://example.com/page?= with :

import requests


endpoint = "http://example.com/page"
r = requests.get(endpoint, params = {'':''})

# r = requests.get(endpoint)

print(r.url) # -> "http://example.com/page"

assert r.url == endpoint+'?=' # Raises AssertionError


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.