90

I want to run each selected py.test item an arbitrary number of times, sequentially.
I don't see any standard py.test mechanism for doing this.

I attempted to do this in the pytest_collection_modifyitems() hook. I modified the list of items passed in, to specify each item more than once. The first execution of a test item works as expected, but that seems to cause some problems for my code.

Further, I would prefer to have a unique test item object for each run, as I use id (item) as a key in various reporting code. Unfortunately, I can't find any py.test code to duplicate a test item, copy.copy() doesn't work, and copy.deepcopy() gets an exception.

Can anybody suggest a strategy for executing a test multiple times?

7 Answers 7

144

One possible strategy is parameterizing the test in question, but not explicitly using the parameter.

For example:

@pytest.mark.parametrize('execution_number', range(5))
def run_multiple_times(execution_number):
    assert True

The above test should run five times.

Check out the parametrization documentation: https://pytest.org/latest/parametrize.html

7
  • 2
    This will execute the test the number of times specified in the test file. It doesn't achieve my goal of executing a test an arbitrary number of times, as specified by a command-line option. Commented Feb 22, 2014 at 2:37
  • 1
    My apologies, I completely misunderstood your question. I think I've figured out how to do what you want, which I'll add as another answer (since this one is completely wrong :).
    – Frank T
    Commented Feb 23, 2014 at 2:14
  • 40
    I'm glad this answer is here. I think it adds to the context for future users looking for possible solutions.
    – moorecm
    Commented Dec 1, 2015 at 14:36
  • 2
    Nice solution if you want to "hardcode" number of times test should be executed
    – Anthony
    Commented Feb 15, 2018 at 12:56
  • 1
    This is particularly useful for debugging the occasional "flakey" test that fails 1 out 25 times due to side effects of fixture data etc.
    – shacker
    Commented Sep 2, 2020 at 23:30
83

The pytest module pytest-repeat exists for this purpose, and I recommend using modules where possible, rather than re-implementing their functionality yourself.

To use it simply add pytest-repeat to your requirements.txt or pip install pytest-repeat, then execute your tests with --count n.

1
41

In order to run each test a number of times, we will programmatically parameterize each test as the tests are being generated.

First, let's add the parser option (include the following in one of your conftest.py's):

def pytest_addoption(parser):
    parser.addoption('--repeat', action='store',
        help='Number of times to repeat each test')

Now we add a "pytest_generate_tests" hook. Here is where the magic happens.

def pytest_generate_tests(metafunc):
    if metafunc.config.option.repeat is not None:
        count = int(metafunc.config.option.repeat)

        # We're going to duplicate these tests by parametrizing them,
        # which requires that each test has a fixture to accept the parameter.
        # We can add a new fixture like so:
        metafunc.fixturenames.append('tmp_ct')

        # Now we parametrize. This is what happens when we do e.g.,
        # @pytest.mark.parametrize('tmp_ct', range(count))
        # def test_foo(): pass
        metafunc.parametrize('tmp_ct', range(count))

Running without the repeat flag:

(env) $ py.test test.py -vv
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 2 items 

test.py:4: test_1 PASSED
test.py:8: test_2 PASSED

=========================== 2 passed in 0.01 seconds ===========================

Running with the repeat flag:

(env) $ py.test test.py -vv --repeat 3
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 6 items 

test.py:4: test_1[0] PASSED
test.py:4: test_1[1] PASSED
test.py:4: test_1[2] PASSED
test.py:8: test_2[0] PASSED
test.py:8: test_2[1] PASSED
test.py:8: test_2[2] PASSED

=========================== 6 passed in 0.01 seconds ===========================

Further reading:

5
  • While this method may work, I used your suggestion to find a simpler way, which doesn't require any parametrized arguments, etc. I will post it and accept it as the answer. Thanks for your suggestion; it led me directly to my solution. Commented Feb 26, 2014 at 20:36
  • 1
    Working on a new project, so I came here to figure out how I had done this previously. I now understand that this solution does not require me to declare a fixture parameter to each test. So I am selecting this answer as correct, and I am using your strategy on my new project. Thanks! Commented Jul 20, 2017 at 18:48
  • How does one repeat the entire test file in order? Here test 1 is run 3 times, what about running 1,2,3 (in order) 3 times?
    – mungayree
    Commented Apr 30, 2020 at 23:09
  • Can it be made to repeat tests in the order though? I mean, if you want to call two tests, you usually do pytest -k "test_foo or test_bar" and it results in foo and bar being in order. However, when I use your code and do a pytest --repeat=99 -k "test_foo or test_bar", instead of calling foo and bar repeatedly it calls 99 times foo and then goes on to call bar.
    – Hi-Angel
    Commented Aug 17, 2023 at 9:58
  • how about running at once,, I am actually trying to test how our server can hold with 1000 request in one go,,? Commented Feb 15 at 9:04
8

Based on Frank T's suggestion, I found a very simple solution in the pytest_generate_tests() callout:

parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')

def pytest_generate_tests (metafunc):
    for i in range (metafunc.config.option.count):
        metafunc.addcall()

Now executing py.test --count 5 causes each test to be executed five times in the test session.

And it requires no changes to any of our existing tests.

3
  • 2
    metafunc.addcall() is deprecated, which is why I preferred .parametrize() in my implementation. See also: pytest.org/latest/…
    – Frank T
    Commented Mar 4, 2014 at 22:03
  • Your solution required me to add a parameter/funcarg to each of my 1,000 existing tests, which I didn't want to do. There is likely a .parametrize() equivalent to my simple solution, but I couldn't figure it out. Commented Mar 5, 2014 at 14:51
  • 6
    Working on a new project, so I came here to figure out how I had done this previously. I now understand that your solution does not require me to declare a fixture parameter to each test. So I am selecting your original answer as correct, and I am using your strategy on my new project. Thanks! Commented Jul 20, 2017 at 18:48
7

While pytest-repeat (the most popular answer) doesn't work for unittest class tests, pytest-flakefinder does:

pip install pytest-flakefinder
pytest --flake-finder --flake-runs=5 tests...

Before finding test-flakefinder, I wrote a little hack of a script that does a similar thing. You can find it here. The top of the script includes instructions to how it can be run.

1
  • 2
    Thanks for mentioning pytest-repeat
    – Edmond
    Commented Feb 10, 2022 at 13:48
4

Based on what I've seen here, and given that I already do some filtering of tests in pytest_collection_modifyitems, my method of choice is the following. In conftest.py

def pytest_addoption(parser):
    parser.addoption('--count', default=1, type=int, metavar='count', help='Run each test the specified number of times')


def pytest_collection_modifyitems(session, config, items):
    count = config.option.count
    items[:] = items * count  # add each test multiple times
1
  • 2
    To avoid a warning, type='int' should be type=int
    – Joel
    Commented Feb 16, 2022 at 20:21
0

I've been looking for a simple solution for a long time. I just had to run one test n-number of times. He shouldn't have fallen once. I solved this problem as simply and stupidly as possible, but it worked for me.

def run_multiple_times(number):
    i = 0
    while i < number:
        #test
        i += 1
run_multiple_times(5)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.