8

Context

I have started using pytest-vcr which is a pytest plugin wrapping VCR.py which I have documented in this blog post on Advanced Python Testing.

It records all HTTP traffic to cassettes/*.yml files on the first test run to save snapshots. Similar to Jest snapshot testing for web components.

On subsequent test runs, if a request is malformed, it won't find a match and throws an exception saying that recording new requests is forbidden and it did not find an existing recording.

Question

VCR.py raises a CannotOverwriteExistingCassetteException which is not particularly informative as to why it didn't match.

How do I leverage pytest pytest_exception_interact hooks to replace this exception with a more informative one leveraging fixture information?

I dove into my site-packages where VCR.py is pip installed and rewrote how I want it to handle the exception. I just need to know how to get this pytest_exception_interact hook to work correctly to access the fixtures from that test node (before it gets cleaned up) and raise a different exception.

Example

Lets get the dependencies.

$ pip install pytest pytest-vcr requests

test_example.py:

import pytest
import requests

@pytest.mark.vcr
def test_example():
    r = requests.get("https://www.stackoverflow.com")
    assert r.status_code == 200
$ pytest test_example.py --vcr-record=once
...
test_example.py::test_example PASSED 
...
$ ls cassettes/
cassettes/test_example.yml
$ head cassettes/test_example.yml
interactions:
- request:
    uri: https://wwwstackoverflow.com
    body: null
    headers:
        Accept:
        - '*/*'
$ pytest test_example.py --vcr-record=none
...
test_example.py::test_example PASSED 
...

Now change the URI in the test to "https://www.google.com":

test_example.py:

import pytest
import requests

@pytest.mark.vcr
def test_example():
    r = requests.get("https://www.google.com")
    assert r.status_code == 200

And run the test again to detect the regression:

$ pytest test_example.py --vcr-record=none
E               vcr.errors.CannotOverwriteExistingCassetteException: No match for the request (<Request (GET) https://www.google.com/>)
...

I can add a conftest.py file to the root of my test structure to create a local plugin, and I can verify that I can intercept the exception and inject my own using:

conftest.py

import pytest
from vcr.errors import CannotOverwriteExistingCassetteException
from vcr.config import VCR
from vcr.cassette import Cassette

class RequestNotFoundCassetteException(CannotOverwriteExistingCassetteException):
    ...

@pytest.fixture(autouse=True)
def _vcr_marker(request):
    marker = request.node.get_closest_marker("vcr")
    if marker:
        cassette = request.getfixturevalue("vcr_cassette")
        vcr = request.getfixturevalue("vcr")
        request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
    yield


@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
    excinfo = call.excinfo
    if report.when == "call" and isinstance(excinfo.value, CannotOverwriteExistingCassetteException):
        # Safely check for fixture pass through on this node
        cassette = None
        vcr = None
        if hasattr(node, "__vcr_fixtures"):
            for fixture_name, fx in node.__vcr_fixtures.items():
                vcr = fx if isinstance(fx, VCR)
                cassette = fx if isinstance(fx, Cassette)

        # If we have the extra fixture context available...
        if cassette and vcr:
            match_properties = [f.__name__ for f in cassette._match_on]
            cassette_reqs = cassette.requests
            #  filtered_req = cassette.filter_request(vcr._vcr_request)
            #  this_req, req_str = __format_near_match(filtered_req, cassette_reqs, match_properties)

            # Raise and catch a new excpetion FROM existing one to keep the traceback
            # https://stackoverflow.com/a/24752607/622276
            # https://docs.python.org/3/library/exceptions.html#built-in-exceptions
            try:
                raise RequestNotFoundCassetteException(
                    f"\nMatching Properties: {match_properties}\n" f"Cassette Requests: {cassette_reqs}\n"
                ) from excinfo.value
            except RequestNotFoundCassetteException as e:
                excinfo._excinfo = (type(e), e)
                report.longrepr = node.repr_failure(excinfo)


This is the part where the documentation on the internet gets pretty thin.

How do I access the vcr_cassette fixture and return a different exception?

What I want to do is get the filtered_request that was attempting to be requested and the list of cassette_requests and using the Python difflib standard library produce deltas against the information that diverged.

PyTest Code Spelunking

The internals of running a single test with pytest triggers pytest_runtest_protocol which effectively runs the following three call_and_report calls to get a collection of reports.

src/_pytest/runner.py:L77-L94

def runtestprotocol(item, log=True, nextitem=None):
    # Abbreviated
    reports = []
    reports.append(call_and_report(item, "setup", log))
    reports.append(call_and_report(item, "call", log))
    reports.append(call_and_report(item, "teardown", log))
    return reports

So I'm after modifying the report at the call stage... but still no clue how I get access to the fixture information.

src/_pytest/runner.py:L166-L174

def call_and_report(item, when, log=True, **kwds):
    call = call_runtest_hook(item, when, **kwds)
    hook = item.ihook
    report = hook.pytest_runtest_makereport(item=item, call=call)
    if log:
        hook.pytest_runtest_logreport(report=report)
    if check_interactive_exception(call, report):
        hook.pytest_exception_interact(node=item, call=call, report=report)
    return report

It looks like there are some helper methods for generating a new ExceptionRepresentation so I updated the conftest.py example.

src/_pytest/reports.py:L361

longrepr = item.repr_failure(excinfo)

UPDATE #1 2019-06-26: Thanks to some pointers from @hoefling in the comments I updated my conftest.py.

  • Correctly re-raising the exception using the raise ... from ... form.
  • Override the _vcr_marker to attach the vcr and vcr_cassette fixtures to the request.node which represent that individual test item.
  • Remaining: Get access to the intercepted request from the patched VCRConnection...

UPDATE #2 2019-06-26

It would seem impossible to get at the VCRHTTPConnections that were patched in creating the cassette context manager. I have opened up the following pull request to pass as arguments when the exception is thrown, to then catch and handle arbitrarily down stream.

https://github.com/kevin1024/vcrpy/pull/445

Related

Related questions that are informative but still don't answer this question.

8
  • 2
    I wish all questions on SO were like this.
    – hoefling
    Commented Jun 25, 2019 at 11:18
  • 1
    Regarding your actual question: you can passthrough the fixture values by attaching them to a global object, usually I use the config for this. E.g. redefine vcr fixture with the body request.config._vcr = vcr; return vcr and access it via node.session.config._vcr in the hook. I can describe it in an answer with a working example if you need it.
    – hoefling
    Commented Jun 25, 2019 at 11:19
  • 1
    Regarding the exception modification: you are on the right track with the pytest_exception_interact hook; however, I would not modify the existing ExceptionInfo object and create a new one instead. What I don't like in your approach is that the exchanged exception won't match the traceback anymore; I would probably raise a new exception of your own type from the current one in the hook (raise MyEx('text') from call.excinfo.value), catch it and inject new excinfo via e.g. call.excinfo = ExceptionInfo.from_current(), so at least you preserve the original traceback.
    – hoefling
    Commented Jun 25, 2019 at 11:28
  • Thanks @hoefling! I'll give that a try tomorrow and circle back. I've managed to intercept the exception and replace it with a new one. This might be the trick to getting the fixture context I need for the new detailed exception. Oh and good tip on raising a new Exception. I didn't like the replacement anyhow.
    – Josh Peak
    Commented Jun 25, 2019 at 11:31
  • 1
    I'm not sure whether the raising and immediate catch is a good approach either, I can imagine it will look ugly in code. I'll think about something better and report back.
    – hoefling
    Commented Jun 25, 2019 at 11:34

1 Answer 1

3

Thanks to comments and guidance in the comments from @hoefling.

I could attach the cassette fixture to the request.node in a conftest.py local plugin overriding the pytest-vcr marker...

@pytest.fixture(autouse=True)
def _vcr_marker(request):
    marker = request.node.get_closest_marker("vcr")
    if marker:
        cassette = request.getfixturevalue("vcr_cassette")
        vcr = request.getfixturevalue("vcr")
        request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
    yield

But I needed more than the cassette to get to my solution.

Ingredients

Recipe

Get latest VCRpy

These patches were released in vcrpy v2.1.0

pip install vcrpy==2.1.0

Override the pytest_exception_interact hook

In the root of your test directory create a conftest.py to create a local plugin that overrides the pytest_exception_interact hook.

@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
    """Intercept specific exceptions from tests."""
    if report.when == "call" and isinstance(call.excinfo.value, CannotOverwriteExistingCassetteException):
        __handle_cassette_exception(node, call, report)

    yield

Extract the Cassette and the Request from the exception.

# Define new exception to throw
class RequestNotFoundCassetteException(Exception):
   ...

def __handle_cassette_exception(node, call, report):
    # Safely check for attributes attached to exception
    vcr_request = None
    cassette = None
    if hasattr(call.excinfo.value, "cassette"):
        cassette = call.excinfo.value.cassette
    if hasattr(call.excinfo.value, "failed_request"):
        vcr_request = call.excinfo.value.failed_request

    # If we have the extra context available...
    if cassette and vcr_request:

        match_properties = [f.__name__ for f in cassette._match_on]
        this_req, req_str = __format_near_match(cassette.requests, vcr_request, match_properties)

        try:
            raise RequestNotFoundCassetteException(f"{this_req}\n\n{req_str}\n") from call.excinfo.value
        except RequestNotFoundCassetteException as e:
            call.excinfo._excinfo = (type(e), e)
            report.longrepr = node.repr_failure(call.excinfo)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.