Context
I have started using pytest-vcr
which is a pytest
plugin wrapping VCR.py
which I have documented in this blog post on Advanced Python Testing.
It records all HTTP traffic to cassettes/*.yml
files on the first test run to save snapshots. Similar to Jest snapshot testing for web components.
On subsequent test runs, if a request is malformed, it won't find a match and throws an exception saying that recording new requests is forbidden and it did not find an existing recording.
Question
VCR.py
raises a CannotOverwriteExistingCassetteException
which is not particularly informative as to why it didn't match.
How do I leverage pytest
pytest_exception_interact
hooks to replace this exception with a more informative one leveraging fixture information?
I dove into my site-packages
where VCR.py
is pip installed
and rewrote how I want it to handle the exception. I just need to know how to get this pytest_exception_interact
hook to work correctly to access the fixtures from that test node (before it gets cleaned up) and raise a different exception.
Example
Lets get the dependencies.
$ pip install pytest pytest-vcr requests
test_example.py:
import pytest
import requests
@pytest.mark.vcr
def test_example():
r = requests.get("https://www.stackoverflow.com")
assert r.status_code == 200
$ pytest test_example.py --vcr-record=once
...
test_example.py::test_example PASSED
...
$ ls cassettes/
cassettes/test_example.yml
$ head cassettes/test_example.yml
interactions:
- request:
uri: https://wwwstackoverflow.com
body: null
headers:
Accept:
- '*/*'
$ pytest test_example.py --vcr-record=none
...
test_example.py::test_example PASSED
...
Now change the URI in the test to "https://www.google.com":
test_example.py:
import pytest
import requests
@pytest.mark.vcr
def test_example():
r = requests.get("https://www.google.com")
assert r.status_code == 200
And run the test again to detect the regression:
$ pytest test_example.py --vcr-record=none
E vcr.errors.CannotOverwriteExistingCassetteException: No match for the request (<Request (GET) https://www.google.com/>)
...
I can add a conftest.py
file to the root of my test structure to create a local plugin, and I can verify that I can intercept the exception and inject my own using:
conftest.py
import pytest
from vcr.errors import CannotOverwriteExistingCassetteException
from vcr.config import VCR
from vcr.cassette import Cassette
class RequestNotFoundCassetteException(CannotOverwriteExistingCassetteException):
...
@pytest.fixture(autouse=True)
def _vcr_marker(request):
marker = request.node.get_closest_marker("vcr")
if marker:
cassette = request.getfixturevalue("vcr_cassette")
vcr = request.getfixturevalue("vcr")
request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
yield
@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
excinfo = call.excinfo
if report.when == "call" and isinstance(excinfo.value, CannotOverwriteExistingCassetteException):
# Safely check for fixture pass through on this node
cassette = None
vcr = None
if hasattr(node, "__vcr_fixtures"):
for fixture_name, fx in node.__vcr_fixtures.items():
vcr = fx if isinstance(fx, VCR)
cassette = fx if isinstance(fx, Cassette)
# If we have the extra fixture context available...
if cassette and vcr:
match_properties = [f.__name__ for f in cassette._match_on]
cassette_reqs = cassette.requests
# filtered_req = cassette.filter_request(vcr._vcr_request)
# this_req, req_str = __format_near_match(filtered_req, cassette_reqs, match_properties)
# Raise and catch a new excpetion FROM existing one to keep the traceback
# https://stackoverflow.com/a/24752607/622276
# https://docs.python.org/3/library/exceptions.html#built-in-exceptions
try:
raise RequestNotFoundCassetteException(
f"\nMatching Properties: {match_properties}\n" f"Cassette Requests: {cassette_reqs}\n"
) from excinfo.value
except RequestNotFoundCassetteException as e:
excinfo._excinfo = (type(e), e)
report.longrepr = node.repr_failure(excinfo)
This is the part where the documentation on the internet gets pretty thin.
How do I access the
vcr_cassette
fixture and return a different exception?
What I want to do is get the filtered_request
that was attempting to be requested and the list of cassette_requests
and using the Python difflib standard library produce deltas against the information that diverged.
PyTest Code Spelunking
The internals of running a single test with pytest triggers pytest_runtest_protocol
which effectively runs the following three call_and_report
calls to get a collection of reports.
def runtestprotocol(item, log=True, nextitem=None):
# Abbreviated
reports = []
reports.append(call_and_report(item, "setup", log))
reports.append(call_and_report(item, "call", log))
reports.append(call_and_report(item, "teardown", log))
return reports
So I'm after modifying the report at the call stage... but still no clue how I get access to the fixture information.
src/_pytest/runner.py:L166-L174
def call_and_report(item, when, log=True, **kwds):
call = call_runtest_hook(item, when, **kwds)
hook = item.ihook
report = hook.pytest_runtest_makereport(item=item, call=call)
if log:
hook.pytest_runtest_logreport(report=report)
if check_interactive_exception(call, report):
hook.pytest_exception_interact(node=item, call=call, report=report)
return report
It looks like there are some helper methods for generating a new ExceptionRepresentation so I updated the conftest.py example.
longrepr = item.repr_failure(excinfo)
UPDATE #1 2019-06-26: Thanks to some pointers from @hoefling in the comments I updated my conftest.py.
- Correctly re-raising the exception using the
raise ... from ...
form. - Override the
_vcr_marker
to attach thevcr
andvcr_cassette
fixtures to therequest.node
which represent that individual test item. - Remaining: Get access to the intercepted request from the patched VCRConnection...
UPDATE #2 2019-06-26
It would seem impossible to get at the VCRHTTPConnections that were patched in creating the cassette context manager. I have opened up the following pull request to pass as arguments when the exception is thrown, to then catch and handle arbitrarily down stream.
https://github.com/kevin1024/vcrpy/pull/445
Related
Related questions that are informative but still don't answer this question.
config
for this. E.g. redefinevcr
fixture with the bodyrequest.config._vcr = vcr; return vcr
and access it vianode.session.config._vcr
in the hook. I can describe it in an answer with a working example if you need it.pytest_exception_interact
hook; however, I would not modify the existingExceptionInfo
object and create a new one instead. What I don't like in your approach is that the exchanged exception won't match the traceback anymore; I would probably raise a new exception of your own type from the current one in the hook (raise MyEx('text') from call.excinfo.value
), catch it and inject new excinfo via e.g.call.excinfo = ExceptionInfo.from_current()
, so at least you preserve the original traceback.