0

I have recently written a Pytest plugin that exposes several mark functions, but I have noticed that when I write tests that use this plugin, the marks I created aren't recognised, and can't be checked by mypy.

import pytest

# .my_mark is not an editor suggestion, and mypy provides no
# errors if I make a mistake with the calling logic
@pytest.mark.my_mark('some', 'args')
def test_thing():
    ...

As such, I want to create a .pyi file that extends the type definitions for Pytest to add type checking for my marks, much like how it can be done in TypeScript. However, I am unsure how to create this definition.

I've attempted to create a pytest-stubs library containing the definition for my marks, but doing so appears to delete all of the type definitions for everything in Pytest, causing many issues in other parts of my code. Adding a from pytest import * in this stub file does not improve things.

# pytest-stubs/__init__.pyi
from typing import Callable

# This overwrites the `mark` class entirely
# and all other properties in `pytest` are considered to be deleted
# by Mypy as well
class mark:
    @staticmethod
    def my_mark(arg1: str, arg2: str) -> Callable[[Callable], Callable]:
        ...
0

1 Answer 1

1

I don't think what you are trying to achieve is possible. Pytest provides it's own typing information, and extending mypy stubs for a third-party library seems impossible. See perhaps Extending stub file for a third-party library/module and mypy#5880.

The options I see are:

  1. Create your own, typed, decorator functions to ship with your library.
def my_mark(foo: str, bar: str):
    return pytest.mark.my_mark(foo, bar)

@my_mark('some', 'args')
def test_thing():
    ...
  1. Open an issue in Pytest and see whether it can be extended to support custom typing annotations

  2. Completely recreate at least the top level stubs for Pytest. I imagine this would be a nightmare to maintain and would probably only work for your codebase. For the sake of completeness, I included this definition at the bottom of the answer.


Background

When using

@pytest.mark.my_mark('some', 'args')
def test_thing():
    ...

you are using a Python decorator, in this case pytest.mark.my_mark. This decorator replaces your function test_thing with whatever my_mark returns. Perhaps see Primer on Python Decorators. In essence, calling test_thing() is equivalent to calling pytest.mark.my_mark('some', 'args')(test_thing)()

But pytest.mark.my_mark is not an ordinary function! Looking at Pytest's source code, pytest.mark is an instance of a MarkGenerator class defined as at the bottom of this answer. The __getattr__ method is called whenever an attribute lookup, in your case for my_mark, has failed. Its typing defines that the result is always an instance of a MarkDecorator, which in turn implements a __call__ method with a predefined signature, the signature I assume mypy infers.

Notice the if TYPE_CHECKING part in Pytest's implementation. That's how Pytest solves the typing issue. Unfortunately, unless you use your own patched version of Pytest, this solution is unfeasible for you.

What you could do instead is write your own custom "typed" decorator as below.

def my_mark(foo: str, bar: str):
    return pytest.mark.my_mark(foo, bar)

and use it instead of pytest.mark.my_mark.

pytest/src/_pytest/mark/structures.py

@final
class MarkGenerator:
    """Factory for :class:`MarkDecorator` objects - exposed as
    a ``pytest.mark`` singleton instance.

    Example::

         import pytest

         @pytest.mark.slowtest
         def test_function():
            pass

    applies a 'slowtest' :class:`Mark` on ``test_function``.
    """

    # See TYPE_CHECKING above.
    if TYPE_CHECKING:
        skip: _SkipMarkDecorator
        skipif: _SkipifMarkDecorator
        xfail: _XfailMarkDecorator
        parametrize: _ParametrizeMarkDecorator
        usefixtures: _UsefixturesMarkDecorator
        filterwarnings: _FilterwarningsMarkDecorator

    def __init__(self, *, _ispytest: bool = False) -> None:
        check_ispytest(_ispytest)
        self._config: Optional[Config] = None
        self._markers: Set[str] = set()

    def __getattr__(self, name: str) -> MarkDecorator:
        """Generate a new :class:`MarkDecorator` with the given name."""
        if name[0] == "_":
            raise AttributeError("Marker name must NOT start with underscore")

        if self._config is not None:
            # We store a set of markers as a performance optimisation - if a mark
            # name is in the set we definitely know it, but a mark may be known and
            # not in the set.  We therefore start by updating the set!
            if name not in self._markers:
                for line in self._config.getini("markers"):
                    # example lines: "skipif(condition): skip the given test if..."
                    # or "hypothesis: tests which use Hypothesis", so to get the
                    # marker name we split on both `:` and `(`.
                    marker = line.split(":")[0].split("(")[0].strip()
                    self._markers.add(marker)

            # If the name is not in the set of known marks after updating,
            # then it really is time to issue a warning or an error.
            if name not in self._markers:
                if self._config.option.strict_markers or self._config.option.strict:
                    fail(
                        f"{name!r} not found in `markers` configuration option",
                        pytrace=False,
                    )

                # Raise a specific error for common misspellings of "parametrize".
                if name in ["parameterize", "parametrise", "parameterise"]:
                    __tracebackhide__ = True
                    fail(f"Unknown '{name}' mark, did you mean 'parametrize'?")

                warnings.warn(
                    "Unknown pytest.mark.%s - is this a typo?  You can register "
                    "custom marks to avoid this warning - for details, see "
                    "https://docs.pytest.org/en/stable/how-to/mark.html" % name,
                    PytestUnknownMarkWarning,
                    2,
                )

        return MarkDecorator(Mark(name, (), {}, _ispytest=True), _ispytest=True)

Recreating Pytest's stubs (not recommended)

Generate the default stubs via

$ stubgen -m pytest

then alter to include the custom my_mark mark as described above. For pytest==7.4.0 the result is as follows.

stubs/pytest/__init__.pyi

from _pytest import version_tuple as version_tuple
from _pytest._code import ExceptionInfo as ExceptionInfo
from _pytest.assertion import register_assert_rewrite as register_assert_rewrite
from _pytest.cacheprovider import Cache as Cache
from _pytest.capture import CaptureFixture as CaptureFixture
from _pytest.config import Config as Config, ExitCode as ExitCode, PytestPluginManager as PytestPluginManager, UsageError as UsageError, cmdline as cmdline, console_main as console_main, hookimpl as hookimpl, hookspec as hookspec, main as main
from _pytest.config.argparsing import OptionGroup as OptionGroup, Parser as Parser
from _pytest.doctest import DoctestItem as DoctestItem
from _pytest.fixtures import FixtureLookupError as FixtureLookupError, FixtureRequest as FixtureRequest, fixture as fixture, yield_fixture as yield_fixture
from _pytest.freeze_support import freeze_includes as freeze_includes
from _pytest.legacypath import TempdirFactory as TempdirFactory, Testdir as Testdir
from _pytest.logging import LogCaptureFixture as LogCaptureFixture
from _pytest.main import Session as Session
# Custom definition below
# from _pytest.mark import MARK_GEN as mark, Mark as Mark, MarkDecorator as MarkDecorator, MarkGenerator as MarkGenerator, param as param
from _pytest.monkeypatch import MonkeyPatch as MonkeyPatch
from _pytest.nodes import Collector as Collector, File as File, Item as Item
from _pytest.outcomes import exit as exit, fail as fail, importorskip as importorskip, skip as skip, xfail as xfail
from _pytest.pytester import HookRecorder as HookRecorder, LineMatcher as LineMatcher, Pytester as Pytester, RecordedHookCall as RecordedHookCall, RunResult as RunResult
from _pytest.python import Class as Class, Function as Function, Metafunc as Metafunc, Module as Module, Package as Package
from _pytest.python_api import approx as approx, raises as raises
from _pytest.recwarn import WarningsRecorder as WarningsRecorder, deprecated_call as deprecated_call, warns as warns
from _pytest.reports import CollectReport as CollectReport, TestReport as TestReport
from _pytest.runner import CallInfo as CallInfo
from _pytest.stash import Stash as Stash, StashKey as StashKey
from _pytest.terminal import TestShortLogReport as TestShortLogReport
from _pytest.tmpdir import TempPathFactory as TempPathFactory
from _pytest.warning_types import PytestAssertRewriteWarning as PytestAssertRewriteWarning, PytestCacheWarning as PytestCacheWarning, PytestCollectionWarning as PytestCollectionWarning, PytestConfigWarning as PytestConfigWarning, PytestDeprecationWarning as PytestDeprecationWarning, PytestExperimentalApiWarning as PytestExperimentalApiWarning, PytestRemovedIn8Warning as PytestRemovedIn8Warning, PytestReturnNotNoneWarning as PytestReturnNotNoneWarning, PytestUnhandledCoroutineWarning as PytestUnhandledCoroutineWarning, PytestUnhandledThreadExceptionWarning as PytestUnhandledThreadExceptionWarning, PytestUnknownMarkWarning as PytestUnknownMarkWarning, PytestUnraisableExceptionWarning as PytestUnraisableExceptionWarning, PytestWarning as PytestWarning
from _typeshed import Incomplete

set_trace: Incomplete

# Custom definition
from _pytest.mark import Mark as Mark, MarkDecorator as MarkDecorator, MarkGenerator as _MarkGenerator, param as param

class _MyMarkDecorator(MarkDecorator):
    def __call__(  # type: ignore[override]
            self,
            foo: str,
            bar: str,
    ) -> MarkDecorator:
        ...

class MarkGenerator(_MarkGenerator):  # type: ignore[misc]
    my_mark: _MyMarkDecorator


mark: MarkGenerator

4
  • Thanks for the very interesting explanation! I agree that patching the stub files is my best bet, but I'm not 100% sure on how to do that without my type definitions causing the rest of Pytest's type safety to break. Since my plugin is a package intended for installation on other people's systems, a simple hacky tweak to Pytest's library code definitely isn't a good idea. Jul 11 at 16:58
  • 1
    @MiguelGuthridge, I decided to learn more about mypy and updated my answer after some research. There might be a better alternative, but you should probably open an issue either at mypy or pytest to find out. Jul 11 at 18:28
  • 1
    I think creating my own wrapper function is the best bet. Thanks for the awesome, detailed explanation! Jul 11 at 23:59
  • I'm glad it helped. I remembered comming across How do I make pytest fixtures work with decorated functions?. Not sure if still relevant, but might be something to keep in mind when creating your wrappers. Jul 12 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.