418

I am trying to use Pythons mock package to mock Pythons requests module. What are the basic calls to get me working in below scenario?

In my views.py, I have a function that makes variety of requests.get() calls with different response each time

def myview(request):
  res1 = requests.get('aurl')
  res2 = request.get('burl')
  res3 = request.get('curl')

In my test class I want to do something like this but cannot figure out exact method calls

Step 1:

# Mock the requests module
# when mockedRequests.get('aurl') is called then return 'a response'
# when mockedRequests.get('burl') is called then return 'b response'
# when mockedRequests.get('curl') is called then return 'c response'

Step 2:

Call my view

Step 3:

verify response contains 'a response', 'b response' , 'c response'

How can I complete Step 1 (mocking the requests module)?

1

20 Answers 20

477

This is how you can do it (you can run this file as-is):

import requests
import unittest
from unittest import mock

# This is the class we want to test
class MyGreatClass:
    def fetch_json(self, url):
        response = requests.get(url)
        return response.json()

# This method will be used by the mock to replace requests.get
def mocked_requests_get(*args, **kwargs):
    class MockResponse:
        def __init__(self, json_data, status_code):
            self.json_data = json_data
            self.status_code = status_code

        def json(self):
            return self.json_data

    if args[0] == 'http://someurl.com/test.json':
        return MockResponse({"key1": "value1"}, 200)
    elif args[0] == 'http://someotherurl.com/anothertest.json':
        return MockResponse({"key2": "value2"}, 200)

    return MockResponse(None, 404)

# Our test case class
class MyGreatClassTestCase(unittest.TestCase):

    # We patch 'requests.get' with our own method. The mock object is passed in to our test case method.
    @mock.patch('requests.get', side_effect=mocked_requests_get)
    def test_fetch(self, mock_get):
        # Assert requests.get calls
        mgc = MyGreatClass()
        json_data = mgc.fetch_json('http://someurl.com/test.json')
        self.assertEqual(json_data, {"key1": "value1"})
        json_data = mgc.fetch_json('http://someotherurl.com/anothertest.json')
        self.assertEqual(json_data, {"key2": "value2"})
        json_data = mgc.fetch_json('http://nonexistenturl.com/cantfindme.json')
        self.assertIsNone(json_data)

        # We can even assert that our mocked method was called with the right parameters
        self.assertIn(mock.call('http://someurl.com/test.json'), mock_get.call_args_list)
        self.assertIn(mock.call('http://someotherurl.com/anothertest.json'), mock_get.call_args_list)

        self.assertEqual(len(mock_get.call_args_list), 3)

if __name__ == '__main__':
    unittest.main()

Important Note: If your MyGreatClass class lives in a different package, say my.great.package, you have to mock my.great.package.requests.get instead of just 'request.get'. In that case your test case would look like this:

import unittest
from unittest import mock
from my.great.package import MyGreatClass

# This method will be used by the mock to replace requests.get
def mocked_requests_get(*args, **kwargs):
    # Same as above


class MyGreatClassTestCase(unittest.TestCase):

    # Now we must patch 'my.great.package.requests.get'
    @mock.patch('my.great.package.requests.get', side_effect=mocked_requests_get)
    def test_fetch(self, mock_get):
        # Same as above

if __name__ == '__main__':
    unittest.main()

Enjoy!

17
  • 2
    MockResponse class is a great idea! I was trying to fake a resuests.Response class object but it wasn't easy. I could use this MockResponse in place of the real thing. Thank you!
    – yoshi
    Commented Jun 11, 2015 at 8:13
  • 10
    And in Python 2.x, just replace from unittest import mock with import mock and the rest works as is. You do need to install the mock package separately.
    – haridsv
    Commented Apr 2, 2016 at 13:54
  • 5
    Fantastic. I had to make a slight change in Python 3 as mock_requests_get needed to yield instead of return because of the change to returning iterators in Python 3.
    – erip
    Commented Nov 22, 2016 at 13:29
  • 2
    this solution works great with the GET request. I am trying to generalise it for POST and PUT but cannot understand, how I can supply extra data to be used inside the mocked_requests_get. All the input arguments in mocked_requests_get will be used in the request. Is there any way to add more arguments, such that they are not used in the request itself, but only for the data manipulation before the request?
    – user1329187
    Commented Apr 25, 2018 at 7:31
  • 1
    that was what the question was originally asking about. I've figured out ways (pack the app into package and fixture a test_client() to do the call ). thanks for the post though, was still using the backbone of the code. Commented Jul 12, 2018 at 16:51
287

Try using the responses library. Here is an example from their documentation:

import responses
import requests

@responses.activate
def test_simple():
    responses.add(responses.GET, 'http://twitter.com/api/1/foobar',
                  json={'error': 'not found'}, status=404)

    resp = requests.get('http://twitter.com/api/1/foobar')

    assert resp.json() == {"error": "not found"}

    assert len(responses.calls) == 1
    assert responses.calls[0].request.url == 'http://twitter.com/api/1/foobar'
    assert responses.calls[0].response.text == '{"error": "not found"}'

It provides quite a nice convenience over setting up all the mocking yourself.

There's also HTTPretty... it's not specific to requests library, more powerful in some ways though I found it doesn't lend itself so well to inspecting the requests that it intercepted, which responses does quite easily

There's also httmock.

A new library gaining popularity recently over the venerable requests is httpx, which adds first-class support for async. A mocking library for httpx is: https://github.com/lundberg/respx

7
  • At a glance, I didn't see a way for responses to match a wildcard url - that is, implement callback logic like "take the last part of the url, look it up in a Map, and return the corresponding value". Is that possible, and I'm just missing it?
    – scubbo
    Commented Jan 6, 2019 at 18:16
  • 4
    @scubbo you can pass a pre-compiled regex as the url param and use the callback style github.com/getsentry/responses#dynamic-responses this will give you the wildcard behaviour you want I think (can access the passed url on the request arg received by the callback func)
    – Anentropic
    Commented Jan 12, 2019 at 18:05
  • 1
    don't use this library if you use pytest
    – marti_
    Commented Feb 18, 2022 at 21:40
  • 1
    I regularly use this library with pytest...
    – Anentropic
    Commented Feb 19, 2022 at 18:12
  • 5
    responses dev here. We run all our unittests on pytest without any issue. If you face any problem, please submit it on GitHub Commented Mar 2, 2022 at 8:19
87

Here is what worked for me:

from unittest import mock
@mock.patch('requests.get', mock.Mock(side_effect = lambda k:{'aurl': 'a response', 'burl' : 'b response'}.get(k, 'unhandled request %s'%k)))
2
53

I used requests-mock for writing tests for separate module:

# module.py
import requests

class A():

    def get_response(self, url):
        response = requests.get(url)
        return response.text

And the tests:

# tests.py
import requests_mock
import unittest

from module import A


class TestAPI(unittest.TestCase):

    @requests_mock.mock()
    def test_get_response(self, m):
        a = A()
        m.get('http://aurl.com', text='a response')
        self.assertEqual(a.get_response('http://aurl.com'), 'a response')
        m.get('http://burl.com', text='b response')
        self.assertEqual(a.get_response('http://burl.com'), 'b response')
        m.get('http://curl.com', text='c response')
        self.assertEqual(a.get_response('http://curl.com'), 'c response')

if __name__ == '__main__':
    unittest.main()
2
  • Where do you get m in '(self, m):' Commented Jun 16, 2020 at 13:03
  • @DenisEvseev, that's passed in through the annotation @requests_mock.mock(). It's very similar (but harder to read) than this approach: @mock.patch('requests.get', side_effect=mocked_requests_get). def test_fetch(self, mock_get): Commented Jun 21, 2021 at 15:19
38

this is how you mock requests.post, change it to your http method

@patch.object(requests, 'post')
def your_test_method(self, mockpost):
    mockresponse = Mock()
    mockpost.return_value = mockresponse
    mockresponse.text = 'mock return'

    #call your target method now
3
  • 2
    What if I want to mock a function? How to mock this for example: mockresponse.json() = {"key": "value"}
    – primoz
    Commented Aug 7, 2017 at 13:50
  • 2
    @primoz, I used an anonymous function/lambda for that: mockresponse.json = lambda: {'key': 'value'}
    – salsbury
    Commented Sep 15, 2017 at 17:52
  • 5
    Or mockresponse.json.return_value = {"key": "value"} Commented Sep 16, 2017 at 17:32
37

Here is a solution with requests Response class. It is cleaner IMHO.

import json
from unittest.mock import patch
from requests.models import Response

def mocked_requests_get(*args, **kwargs):
    response_content = None
    request_url = kwargs.get('url', None)
    if request_url == 'aurl':
        response_content = json.dumps('a response')
    elif request_url == 'burl':
        response_content = json.dumps('b response')
    elif request_url == 'curl':
        response_content = json.dumps('c response')
    response = Response()
    response.status_code = 200
    response._content = str.encode(response_content)
    return response

@mock.patch('requests.get', side_effect=mocked_requests_get)
def test_fetch(self, mock_get):
     response = requests.get(url='aurl')
     assert ...
3
  • For this to work for me, I needed to replace kwargs.get('url', None) with args[0].
    – CodeBiker
    Commented May 19, 2021 at 15:44
  • 1
    Don't change it to args[0], just pass the URL param in the requests. requests.get(url="aurl") Commented Jun 24, 2021 at 4:57
  • 3
    I really dislike using _content since it is an internal method, but it's quite trialsome trying to set the content through the raw attribute, so this is the best method that I've found to get a real Response object as a patched requests.get return value. Commented Dec 3, 2021 at 20:44
11

I started out with Johannes Farhenkrug's answer here and it worked great for me. I needed to mock the requests library because my goal is to isolate my application and not test any 3rd party resources.

Then I read up some more about python's Mock library and I realized that I can replace the MockResponse class, which you might call a 'Test Double' or a 'Fake', with a python Mock class.

The advantage of doing so is access to things like assert_called_with, call_args and so on. No extra libraries are needed. Additional benefits such as 'readability' or 'its more pythonic' are subjective, so they may or may not play a role for you.

Here is my version, updated with using python's Mock instead of a test double:

import json
import requests
from unittest import mock

# defube stubs
AUTH_TOKEN = '{"prop": "value"}'
LIST_OF_WIDGETS = '{"widgets": ["widget1", "widget2"]}'
PURCHASED_WIDGETS = '{"widgets": ["purchased_widget"]}'


# exception class when an unknown URL is mocked
class MockNotSupported(Exception):
  pass


# factory method that cranks out the Mocks
def mock_requests_factory(response_stub: str, status_code: int = 200):
    return mock.Mock(**{
        'json.return_value': json.loads(response_stub),
        'text.return_value': response_stub,
        'status_code': status_code,
        'ok': status_code == 200
    })


# side effect mock function
def mock_requests_post(*args, **kwargs):
    if args[0].endswith('/api/v1/get_auth_token'):
        return mock_requests_factory(AUTH_TOKEN)
    elif args[0].endswith('/api/v1/get_widgets'):
        return mock_requests_factory(LIST_OF_WIDGETS)
    elif args[0].endswith('/api/v1/purchased_widgets'):
        return mock_requests_factory(PURCHASED_WIDGETS)
    
    raise MockNotSupported


# patch requests.post and run tests
with mock.patch('requests.post') as requests_post_mock:
  requests_post_mock.side_effect = mock_requests_post
  response = requests.post('https://myserver/api/v1/get_widgets')
  assert response.ok is True
  assert response.status_code == 200
  assert 'widgets' in response.json()
  
  # now I can also do this
  requests_post_mock.assert_called_with('https://myserver/api/v1/get_widgets')

Repl.it links:

https://repl.it/@abkonsta/Using-unittestMock-for-requestspost#main.py

https://repl.it/@abkonsta/Using-test-double-for-requestspost#main.py

8

If you want to mock a fake response, another way to do it is to simply instantiate an instance of the base HttpResponse class, like so:

from django.http.response import HttpResponseBase

self.fake_response = HttpResponseBase()
1
  • 3
    This is the answer for what I was trying to find: get a fake django response object that can make it through the gamut of middleware for an almost e2e test. HttpResponse, rather than ...Base, did the trick for me though. Thanks!
    – low_ghost
    Commented Feb 11, 2020 at 13:54
6

Using requests_mock is easy to patch any requests

pip install requests-mock
from unittest import TestCase
import requests_mock
from <yourmodule> import <method> (auth)

class TestApi(TestCase):
  @requests_mock.Mocker()
  def test_01_authentication(self, m):
        """Successful authentication using username password"""
        token = 'token'
        m.post(f'http://localhost/auth', json= {'token': token})
        act_token =auth("user", "pass")
        self.assertEqual(act_token, token)

5

This worked for me, although I haven't done much complicated testing yet.

import json
from requests import Response

class MockResponse(Response):
    def __init__(self,
                 url='http://example.com',
                 headers={'Content-Type':'text/html; charset=UTF-8'},
                 status_code=200,
                 reason = 'Success',
                 _content = 'Some html goes here',
                 json_ = None,
                 encoding='UTF-8'
                 ):
    self.url = url
    self.headers = headers
    if json_ and headers['Content-Type'] == 'application/json':
        self._content = json.dumps(json_).encode(encoding)
    else:
        self._content = _content.encode(encoding)

    self.status_code = status_code
    self.reason = reason
    self.encoding = encoding

Then you can create responses :

mock_response = MockResponse(
    headers={'Content-Type' :'application/json'},
    status_code=401,
    json_={'success': False},
    reason='Unauthorized'
)
mock_response.raise_for_status()

gives

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://example.com
5

Can you use requests-mock instead?

Suppose your myview function instead takes a requests.Session object, makes requests with it, and does something to the output:

# mypackage.py
def myview(session):
    res1 = session.get("http://aurl")
    res2 = session.get("http://burl")
    res3 = session.get("http://curl")
    return f"{res1.text}, {res2.text}, {res3.text}"
# test_myview.py
from mypackage import myview
import requests

def test_myview(requests_mock):
    # set up requests
    a_req = requests_mock.get("http://aurl", text="a response")
    b_req = requests_mock.get("http://burl", text="b response")
    c_req = requests_mock.get("http://curl", text="c response")

    # test myview behaviour
    session = requests.Session()
    assert myview(session) == "a response, b response, c response"

    # check that requests weren't called repeatedly
    assert a_req.called_once
    assert b_req.called_once
    assert c_req.called_once
    assert requests_mock.call_count == 3

You can also use requests_mock with frameworks other than Pytest - the documentation is great.

4

One possible way to work around requests is using the library betamax, it records all requests and after that if you make a request in the same url with the same parameters the betamax will use the recorded request, I have been using it to test web crawler and it save me a lot time.

import os

import requests
from betamax import Betamax
from betamax_serializers import pretty_json


WORKERS_DIR = os.path.dirname(os.path.abspath(__file__))
CASSETTES_DIR = os.path.join(WORKERS_DIR, u'resources', u'cassettes')
MATCH_REQUESTS_ON = [u'method', u'uri', u'path', u'query']

Betamax.register_serializer(pretty_json.PrettyJSONSerializer)
with Betamax.configure() as config:
    config.cassette_library_dir = CASSETTES_DIR
    config.default_cassette_options[u'serialize_with'] = u'prettyjson'
    config.default_cassette_options[u'match_requests_on'] = MATCH_REQUESTS_ON
    config.default_cassette_options[u'preserve_exact_body_bytes'] = True


class WorkerCertidaoTRT2:
    session = requests.session()

    def make_request(self, input_json):
        with Betamax(self.session) as vcr:
            vcr.use_cassette(u'google')
            response = session.get('http://www.google.com')

https://betamax.readthedocs.io/en/latest/

1
4

For those, who don't want to install additional libs for pytest, there is an example. I will duplicate it here with some extension, based on examples above:

import datetime

import requests


class MockResponse:
    def __init__(self, json_data, status_code):
        self.json_data = json_data
        self.status_code = status_code
        self.elapsed = datetime.timedelta(seconds=1)

    # mock json() method always returns a specific testing dictionary
    def json(self):
        return self.json_data


def test_get_json(monkeypatch):
    # Any arguments may be passed and mock_get() will always return our
    # mocked object, which only has the .json() method.
    def mock_get(*args, **kwargs):
        return MockResponse({'mock_key': 'mock_value'}, 418)

    # apply the monkeypatch for requests.get to mock_get
    monkeypatch.setattr(requests, 'get', mock_get)

    # app.get_json, which contains requests.get, uses the monkeypatch
    response = requests.get('https://fakeurl')
    response_json = response.json()

    assert response_json['mock_key'] == 'mock_value'
    assert response.status_code == 418
    assert response.elapsed.total_seconds() == 1


============================= test session starts ==============================
collecting ... collected 1 item

test_so.py::test_get_json PASSED                                          [100%]

============================== 1 passed in 0.07s ===============================
3

The simplest way so far:

from unittest import TestCase
from unittest.mock import Mock, patch

from .utils import method_foo


class TestFoo(TestCase):

    @patch.object(utils_requests, "post")  # change to desired method here
    def test_foo(self, mock_requests_post):
        # EXPLANATION: mocked 'post' method above will return some built-in mock, 
        # and its method 'json' will return mock 'mock_data',
        # which got argument 'return_value' with our data to be returned
        mock_data = Mock(return_value=[{"id": 1}, {"id": 2}])
        mock_requests_post.return_value.json = mock_data

        method_foo()

        # TODO: asserts here


"""
Example of method that you can test in utils.py
"""
def method_foo():
    response = requests.post("http://example.com")
    records = response.json()
    for record in records:
        print(record.get("id"))
        # do other stuff here
2

I will add this information since I had a hard time figuring how to mock an async api call.

Here is what I did to mock an async call.

Here is the function I wanted to test

async def get_user_info(headers, payload):
    return await httpx.AsyncClient().post(URI, json=payload, headers=headers)

You still need the MockResponse class

class MockResponse:
    def __init__(self, json_data, status_code):
        self.json_data = json_data
        self.status_code = status_code

    def json(self):
        return self.json_data

You add the MockResponseAsync class

class MockResponseAsync:
    def __init__(self, json_data, status_code):
        self.response = MockResponse(json_data, status_code)

    async def getResponse(self):
        return self.response

Here is the test. The important thing here is I create the response before since init function can't be async and the call to getResponse is async so it all checked out.

@pytest.mark.asyncio
@patch('httpx.AsyncClient')
async def test_get_user_info_valid(self, mock_post):
    """test_get_user_info_valid"""
    # Given
    token_bd = "abc"
    username = "bob"
    payload = {
        'USERNAME': username,
        'DBNAME': 'TEST'
    }
    headers = {
        'Authorization': 'Bearer ' + token_bd,
        'Content-Type': 'application/json'
    }
    async_response = MockResponseAsync("", 200)
    mock_post.return_value.post.return_value = async_response.getResponse()

    # When
    await api_bd.get_user_info(headers, payload)

    # Then
    mock_post.return_value.post.assert_called_once_with(
        URI, json=payload, headers=headers)

If you have a better way of doing that tell me but I think it's pretty clean like that.

1

To avoid installing other dependencies you should create a fake response. This FakeResponse could be a child of Response (I think this is a good approach because it's more realistic) or just a simple class with the attributes you need.

Simple Fake class

class FakeResponse:
        status_code = None

        def __init__(self, *args, **kwargs):
            self.status_code = 500
            self.text = ""

Child of Response

class FakeResponse(Response):
        encoding = False
        _content = None

        def __init__(*args, **kwargs):
            super(FakeResponse).__thisclass__.status_code = 500
            # Requests requires to be not be None, if not throws an exception
            # For reference: https://github.com/psf/requests/issues/3698#issuecomment-261115119
            super(FakeResponse).__thisclass__.raw = io.BytesIO()
0

Just a helpful hint to those that are still struggling, converting from urllib or urllib2/urllib3 to requests AND trying to mock a response- I was getting a slightly confusing error when implementing my mock:

with requests.get(path, auth=HTTPBasicAuth('user', 'pass'), verify=False) as url:

AttributeError: __enter__

Well, of course, if I knew anything about how with works (I didn't), I'd know it was a vestigial, unnecessary context (from PEP 343). Unnecessary when using the requests library because it does basically the same thing for you under the hood. Just remove the with and use bare requests.get(...) and Bob's your uncle.

0

For pytest users there is a convinient fixture from https://pypi.org/project/pytest-responsemock/

For example to mock GET to http://some.domain you can:

def test_me(response_mock):

    with response_mock('GET http://some.domain -> 200 :Nice'):
        response = send_request()
        assert result.ok
        assert result.content == b'Nice'

0

I will demonstrate how to detach your programming logic from the actual external library by swapping the real request with a fake one that returns the same data. In your view if external api call then this process is best

import pytest
from unittest.mock import patch
from django.test import RequestFactory

@patch("path(projectname.appname.filename).requests.post")
def test_mock_response(self, mock_get, rf: RequestFactory):
    mock_get.return_value.ok = Mock(ok=True)
    mock_get.return_value.status_code = 400
    mock_get.return_value.json.return_value = {you can define here dummy response}
    request = rf.post("test/", data=self.payload)
    response = view_name_view(request)

    expected_response = {
        "success": False,
        "status": "unsuccessful",
    }

    assert response.data == expected_response
    assert response.status_code == 400
0

If using pytest:

>>> import pytest
>>> import requests

>>> def test_url(requests_mock):
...     requests_mock.get('http://test.com', text='data')
...     assert 'data' == requests.get('http://test.com').text

Taken from the official documentation

1
  • 2
    This approach only apply if your project is using the additional requests-mock library.
    – nbrew
    Commented Jan 6, 2023 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.