44

Trying to get the raw data of the HTTP response content in requests in Python. I am interested in forwarding the response through another channel, which means that ideally the content should be as pristine as possible.

What would be a good way to do this?

4 Answers 4

43

After requests.get(), you can use r.content to extract the raw Byte-type content.

r = requests.get('https://yourweb.com', stream=True)
r.content
36

If you are using a requests.get call to obtain your HTTP response, you can use the raw attribute of the response. Here is the code from the requests docs. The stream=True parameter in the requests.get call is required for this to work.

>>> r = requests.get('https://github.com/timeline.json', stream=True)
>>> r.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>
>>> r.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
10
  • 9
    looks like it's r.raw.data
    – Blair
    Commented Jun 4, 2013 at 18:37
  • 5
    This doesnt seem to work correctly, I tried res.raw.data and res.raw.read(100) but they both return empty. Commented Aug 18, 2015 at 5:48
  • 11
    @DoronCohen Did you include stream=True ?
    – farthVader
    Commented Oct 20, 2015 at 23:27
  • 9
    you could use use r.raw.decode_content = True to handle Content-Encoding http header.
    – jfs
    Commented Oct 23, 2015 at 12:22
  • 2
    You specifically said if you have used "get"; Does that mean we cannot use this for response object generated as a result of "post"? Commented Jan 31, 2020 at 4:44
5

To add to @blair answer, as stated in the docs:

In general, however, you should use a pattern like this to save what is being streamed to a file:

r = requests.get('https://yourweb.com', stream=True)

with open(filename, 'wb') as fd:
   for chunk in r.iter_content(chunk_size=128):
      fd.write(chunk)

Using Response.iter_content will handle a lot of what you would otherwise have to handle when using Response.raw directly. When streaming a download, the above is the preferred and recommended way to retrieve the content. Note that chunk_size can be freely adjusted to a number that may better fit your use cases.

That pattern not only has the advantages described above, but is also a good to fetch data in environments with limited memory.

3
  • 1
    Doesn't seem a complete answer. What is r? Commented Nov 19, 2021 at 17:10
  • 1
    I edited to add clarity, but had you looked at the two other answers, I was simply following their pattern of setting r to be the request object. Commented Nov 23, 2021 at 6:24
  • This answer worked perfectly for me. Thanks. I also thought that 'r' is intuitive (esp. given the other responses).
    – Mark
    Commented Dec 20, 2021 at 21:15
3

The following is an easy way to recreate the whole HTTP response, including the HTTP header's initial Status Line:

r = requests.get('https://yourweb.com/', stream=True)
print(f"HTTP/{r.raw.version/10} {r.raw.status} {r.raw.reason}")
for k,v in r.raw.headers.items(): print(f"{k}: {v}")
print(r.text)

This may not be 100% pristine, but it should be very close. And you could use print()'s file= parameter to redirect the output to a file.

1
  • Adding header_reply = {} above the first line and changing print(f"{k}: {v}") to header_reply[k]=v makes a quick dictionary. Handy for wrangling API's and such that don't return any content. Commented Mar 6 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.