91

I am writing an application that performs REST operations using Kenneth Reitz's requests library and I'm struggling to find a nice way to unit test these applications, because requests provides its methods via module-level methods.

What I want is the ability to synthesize the conversation between the two sides; provide a series of request assertions and responses.

3
  • 2
    So you need to mock out the REST server?
    – kgr
    Mar 4, 2012 at 23:57
  • why does that make unittest unsuitable? Check how the library does its own unit tests; might offer ideas.
    – John Mee
    Mar 5, 2012 at 1:00
  • 3
    FWIW, the Requests library does its own tests using live URLs (github.com, the authors own domain, etc.).
    – Romløk
    Nov 28, 2012 at 13:29

7 Answers 7

66

It is in fact a little strange that the library has a blank page about end-user unit testing, while targeting user-friendliness and ease of use. There's however an easy-to-use library by Dropbox, unsurprisingly called responses. Here is its intro post. It says they've failed to employ httpretty, while stating no reason of the fail, and written a library with similar API.

import unittest

import requests
import responses


class TestCase(unittest.TestCase):

  @responses.activate  
  def testExample(self):
    responses.add(**{
      'method'         : responses.GET,
      'url'            : 'http://example.com/api/123',
      'body'           : '{"error": "reason"}',
      'status'         : 404,
      'content_type'   : 'application/json',
      'adding_headers' : {'X-Foo': 'Bar'}
    })

    response = requests.get('http://example.com/api/123')

    self.assertEqual({'error': 'reason'}, response.json())
    self.assertEqual(404, response.status_code)
2
  • Updated URL for the responses intro post Feb 21, 2018 at 10:19
  • Interestingly, since this was posted as an answer, David Cramer –who authored this library– moved on and founded Sentry, and moved the library with him. This is why on GitHub it’s under the getsentry org.
    – bfontaine
    Aug 18, 2019 at 17:50
50

If you use specifically requests try httmock. It's wonderfully simple and elegant:

from httmock import urlmatch, HTTMock
import requests

# define matcher:
@urlmatch(netloc=r'(.*\.)?google\.com$')
def google_mock(url, request):
    return 'Feeling lucky, punk?'

# open context to patch
with HTTMock(google_mock):
    # call requests
    r = requests.get('http://google.com/')
print r.content  # 'Feeling lucky, punk?'

If you want something more generic (e.g. to mock any library making http calls) go for httpretty.

Almost as elegant:

import requests
import httpretty

@httpretty.activate
def test_one():
    # define your patch:
    httpretty.register_uri(httpretty.GET, "http://yipit.com/",
                        body="Find the best daily deals")
    # use!
    response = requests.get('http://yipit.com')
    assert response.text == "Find the best daily deals"

HTTPretty is far more feature-rich - it offers also mocking status code, streaming responses, rotating responses, dynamic responses (with a callback).

1
  • That httpretty thing is straight from out of this world, thanks!
    – mccc
    Sep 17, 2015 at 8:53
22

You could use a mocking library such as Mocker to intercept the calls to the requests library and return specified results.

As a very simple example, consider this class which uses the requests library:

class MyReq(object):
    def doSomething(self):
        r = requests.get('https://api.github.com', auth=('user', 'pass'))
        return r.headers['content-type']

Here's a unit test that intercepts the call to requests.get and returns a specified result for testing:

import unittest
import requests
import myreq

from mocker import Mocker, MockerTestCase

class MyReqTests(MockerTestCase):
    def testSomething(self):
        # Create a mock result for the requests.get call
        result = self.mocker.mock()
        result.headers
        self.mocker.result({'content-type': 'mytest/pass'})

        # Use mocker to intercept the call to requests.get
        myget = self.mocker.replace("requests.get")
        myget('https://api.github.com', auth=('user', 'pass'))
        self.mocker.result(result)

        self.mocker.replay()

        # Now execute my code
        r = myreq.MyReq()
        v = r.doSomething()

        # and verify the results
        self.assertEqual(v, 'mytest/pass')
        self.mocker.verify()

if __name__ == '__main__':
    unittest.main()

When I run this unit test I get the following result:

.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK
3

Missing from these answers is requests-mock.

From their page:

>>> import requests
>>> import requests_mock

As a context manager:

>>> with requests_mock.mock() as m:

...     m.get('http://test.com', text='data')
...     requests.get('http://test.com').text
...
'data'

Or as a decorator:

>>> @requests_mock.mock()
... def test_func(m):
...     m.get('http://test.com', text='data')
...     return requests.get('http://test.com').text
...
>>> test_func()
'data'
3
  • Do you have any knowledge of how to make this work with pytest? I tried the exact example you're citing. Ref.: stackoverflow.com/questions/47703748/… Dec 8, 2017 at 0:59
  • If I remember correctly, I used the decorator. And I think this also worked with pytest.
    – Unapiedra
    Dec 18, 2017 at 16:08
  • I made it work with decorators, but it appears (on my system) that it's in conflict with some other argument, so I had to pass a kw argument to the Mocker, as mentioned in docs. Not sure if this has to do with pytest, but the error that came up mentioned fixtures. Thanks for getting back to the issue. Dec 18, 2017 at 17:37
2

using mocker like in srgerg's answer:

def replacer(method, endpoint, json_string):
    from mocker import Mocker, ANY, CONTAINS
    mocker = Mocker()
    result = mocker.mock()
    result.json()
    mocker.count(1, None)
    mocker.result(json_string)
    replacement = mocker.replace("requests." + method)
    replacement(CONTAINS(endpoint), params=ANY)
    self.mocker.result(result)
    self.mocker.replay()

For the requests library, this would intercept the request by method and endpoint you're hitting and replace the .json() on the response with the json_string passed in.

2

If you break out your response handler/parser into a separate function, you can work with requests.Response objects directly, without needing to mock the client-server interaction.

Code under test

from xml.dom import minidom
from requests.models import Response

def function_under_test(s3_response: Response):
    doc = minidom.parseString(s3_response.text)

    return (
        s3_response.status_code,
        doc.getElementsByTagName('Code').item(0).firstChild.data,
    )

Test code

import unittest
from io import BytesIO

class Test(unittest.TestCase):

    def test_it(self):
        s3_response = Response()
        s3_response.status_code = 404
        s3_response.raw = BytesIO(b"""<?xml version="1.0" encoding="UTF-8"?>
            <Error>
            <Code>NoSuchKey</Code>
            <Message>The resource you requested does not exist</Message>
            <Resource>/mybucket/myfoto.jpg</Resource> 
            <RequestId>4442587FB7D0A2F9</RequestId>
            </Error>
        """)

        parsed_response = function_under_test(s3_response)

        self.assertEqual(404, parsed_response[0])
        self.assertEqual("NoSuchKey", parsed_response[1])
0

There's a library for this, if you want to write your test server with Flask: requests-flask-adaptor

You just have to be careful with the order of imports when monkeypatching.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.