39

Trying to get the raw data of the HTTP response content in requests in Python. I am interested in forwarding the response through another channel, which means that ideally the content should be as pristine as possible.

What would be a good way to do this?

3 Answers 3

36

If you are using a requests.get call to obtain your HTTP response, you can use the raw attribute of the response. Here is the code from the requests docs. The stream=True parameter in the requests.get call is required for this to work.

>>> r = requests.get('https://github.com/timeline.json', stream=True)
>>> r.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>
>>> r.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
10
  • 9
    looks like it's r.raw.data
    – Brien
    Jun 4, 2013 at 18:37
  • 5
    This doesnt seem to work correctly, I tried res.raw.data and res.raw.read(100) but they both return empty. Aug 18, 2015 at 5:48
  • 11
    @DoronCohen Did you include stream=True ?
    – farthVader
    Oct 20, 2015 at 23:27
  • 9
    you could use use r.raw.decode_content = True to handle Content-Encoding http header.
    – jfs
    Oct 23, 2015 at 12:22
  • 2
    You specifically said if you have used "get"; Does that mean we cannot use this for response object generated as a result of "post"? Jan 31, 2020 at 4:44
34

After requests.get(), you can use r.content to extract the raw Byte-type content.

r = requests.get('https://yourweb.com', stream=True)
r.content
4

To add to @brien answer, as stated in the docs:

In general, however, you should use a pattern like this to save what is being streamed to a file:

r = requests.get('https://yourweb.com', stream=True)

with open(filename, 'wb') as fd:
   for chunk in r.iter_content(chunk_size=128):
      fd.write(chunk)

Using Response.iter_content will handle a lot of what you would otherwise have to handle when using Response.raw directly. When streaming a download, the above is the preferred and recommended way to retrieve the content. Note that chunk_size can be freely adjusted to a number that may better fit your use cases.

That pattern not only has the advantages described above, but is also a good to fetch data in environments with limited memory.

3
  • 1
    Doesn't seem a complete answer. What is r? Nov 19, 2021 at 17:10
  • 1
    I edited to add clarity, but had you looked at the two other answers, I was simply following their pattern of setting r to be the request object. Nov 23, 2021 at 6:24
  • This answer worked perfectly for me. Thanks. I also thought that 'r' is intuitive (esp. given the other responses).
    – Mark
    Dec 20, 2021 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.