19

I've been trying to find the default behavior of the requests module about the compression, but I couldn't find anything, so I'm asking here:

Does the requests module asks the server for compression by default, or not? Do we have to specify it in the header?

I found a page here talking about urllib, where it is said that the header must ask for compression to get it:

http://www.diveintopython.net/http_web_services/gzip_compression.html

Is it the same thing for requests?

2 Answers 2

18

You can test it yourself. Try the following:

import requests
req = requests.get("http://google.com")
print(req.request.headers)

This will print the headers sent to the server (the default ones, since no other headers are defined). On my system I get:

{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.7.0 CPython/2.7.8 Linux/4.1.8-100.fc21.x86_64'}

So, gzip and deflate are accepted compression types. The server will choose one of those.

14

For requests: Requests with a body (e.g. POST) will not have their body compressed by default, since very few servers would support that.

For responses: By default, it accepts (via request header) both compressed and uncompressed responses. This can be overriden by setting the Accept-Encoding header by usual means. If set Accept to only allow compressed, the server may fail to serve your request if it doesn't support compression.

7
  • Ok, I have two answers telling me the header specifies it ACCEPTS compressed data. But if the server can serve compressed and uncompressed data, what will it choose ?
    – JPFrancoia
    Oct 25, 2015 at 20:54
  • @Rififi You're thinking the wrong way around: the client (any client) can list accepted encodings. After that, the server decides which encoding to use. The only way to force compression of response is only list compressed in the Accept header. But then, if the server does not want to use compression, the request will fail.
    – Kroltan
    Oct 25, 2015 at 20:57
  • Yes, sorry if I was unclear, that was exactly my question: how will the server choose among the encodings ? Let's say you're telling it you accept both compressed and uncompressed data, what will it choose (in general) ?
    – JPFrancoia
    Oct 25, 2015 at 21:02
  • 1
    @Rififi You can't know. It depends on what server, what server configuration, and possibly extra logic (e.g. suppose there's a script on the server that only compresses the response if it's a monday night and the ice cream shop closed early). But usually the sane option is compressing, to save bandwidth.
    – Kroltan
    Oct 25, 2015 at 21:29
  • Ok. But I can investigate, right ? Like you said, I can mention only compressed encodings, and check the response. If the server failed to serve me something, I will know it doesn't use a compressed encoding (assuming the weather and the ice cream shop haven't any influence on the compression).
    – JPFrancoia
    Oct 25, 2015 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.