pytest-rerunfailures
====================

.. START-SHORT-DESCRIPTION

pytest-rerunfailures is a plugin for `pytest <https://pytest.org>`_ that
re-runs tests to eliminate intermittent failures.

.. END-SHORT-DESCRIPTION

.. image:: https://img.shields.io/badge/license-MPL%202.0-blue.svg
   :target: https://github.com/pytest-dev/pytest-rerunfailures/blob/master/LICENSE
   :alt: License
.. image:: https://img.shields.io/pypi/v/pytest-rerunfailures.svg
   :target: https://pypi.python.org/pypi/pytest-rerunfailures/
   :alt: PyPI
.. image:: https://github.com/pytest-dev/pytest-rerunfailures/workflows/Test/badge.svg
   :target: https://github.com/pytest-dev/pytest-rerunfailures/actions
   :alt: GitHub Actions
.. image:: https://readthedocs.org/projects/pytest-rerunfailures/badge/?version=latest
    :target: https://pytest-rerunfailures.readthedocs.io/latest/?badge=latest
    :alt: Documentation Status

.. START-INSTALLATION

Requirements
------------

You will need the following prerequisites in order to use pytest-rerunfailures:

- Python 3.10+ or PyPy3
- pytest 8.2 or newer

This plugin can recover from a hard crash with the following optional
prerequisites:

- pytest-xdist 2.3.0 or newer

This package is currently tested against the last 5 minor pytest releases. In
case you work with an older version of pytest you should consider updating or
use one of the earlier versions of this package.

Installation
------------

To install pytest-rerunfailures:

.. code-block:: bash

  $ pip install pytest-rerunfailures

.. END-INSTALLATION

Recover from hard crashes
-------------------------

If one or more tests trigger a hard crash (for example: segfault), this plugin
will ordinarily be unable to rerun the test. However, if a compatible version of
pytest-xdist is installed, and the tests are run within pytest-xdist using the ``-n``
flag, this plugin will be able to rerun crashed tests, assuming the workers and
controller are on the same LAN (this assumption is valid for almost all cases
because most of the time the workers and controller are on the same computer).
If this assumption is not the case, then this functionality may not operate.

Re-run all failures
-------------------

To re-run all test failures, use the ``--reruns`` command line option with the
maximum number of times you'd like the tests to run:

.. code-block:: bash

  $ pytest --reruns 5

To add a delay time between re-runs use the ``--reruns-delay`` command line
option with the amount of seconds that you would like wait before the next
test re-run is launched:

.. code-block:: bash

   $ pytest --reruns 5 --reruns-delay 1

To grow the delay after every attempt (exponential backoff), use the
``--reruns-delay-backoff-factor`` option. The delay before the *n*-th re-run is
``reruns_delay * reruns_delay_backoff_factor ** (n - 1)``. The default factor is
``1.0``, which keeps the delay constant. For example, the following waits 1, 2
and 4 seconds before the three re-runs:

.. code-block:: bash

   $ pytest --reruns 3 --reruns-delay 1 --reruns-delay-backoff-factor 2

.. START-FIXTURES

Fixture support
---------------

Failures during fixture setup or ``setup_class`` are also rerun. This includes
class-, module-, package-, and session-scoped fixtures. A rerun repeats only
the affected test item: fixtures that have already completed setup remain
cached at their scope and are torn down once after reruns.

.. END-FIXTURES

Re-run all failures matching certain expressions
------------------------------------------------

To re-run only those failures that match a certain list of expressions, use the
``--only-rerun`` flag and pass it a regular expression. For example,
the following would only rerun those errors that match ``AssertionError``:

.. code-block:: bash

   $ pytest --reruns 5 --only-rerun AssertionError

Passing the flag multiple times accumulates the arguments, so the following
would only rerun those errors that match ``AssertionError`` or ``ValueError``:

.. code-block:: bash

   $ pytest --reruns 5 --only-rerun AssertionError --only-rerun ValueError

The same matching is applied to each exception in the ``__cause__`` /
``__context__`` chain, so a wrapped error such as
``raise RuntimeError(...) from MemoryError(...)`` is still rerun by
``--only-rerun MemoryError``.

The default list can also live in the ``pytest.ini`` (or ``pyproject.toml``)
file, one expression per line:

.. code-block:: ini

   [pytest]
   only_rerun =
       AssertionError
       ValueError

A ``--only-rerun`` flag on the command line replaces the ini list rather than
accumulating with it.

Re-run all failures other than matching certain expressions
-----------------------------------------------------------

To re-run only those failures that do not match a certain list of expressions, use the
``--rerun-except`` flag and pass it a regular expression. For example,
the following would only rerun errors other than that match ``AssertionError``:

.. code-block:: bash

   $ pytest --reruns 5 --rerun-except AssertionError

Passing the flag multiple times accumulates the arguments, so the following
would only rerun those errors that does not match with ``AssertionError`` or ``OSError``:

.. code-block:: bash

   $ pytest --reruns 5 --rerun-except AssertionError --rerun-except OSError

Matching for ``--rerun-except`` follows explicit ``__cause__`` links
(``raise ... from ...``), so ``raise RuntimeError(...) from ValueError(...)``
is excluded by ``--rerun-except ValueError``. Implicit ``__context__`` from
``except`` / ``finally`` is not walked, so a ``ConnectionError`` raised inside
``except AssertionError`` is still rerun by ``--rerun-except AssertionError``.

The exclusion list can also live in the ``pytest.ini`` (or
``pyproject.toml``) file as ``rerun_except``, one expression per line.

Exclude test paths from re-runs
--------------------------------

To exclude a test file or all tests under a directory from re-runs, use the
``--rerun-exclude-path`` option. Relative paths are interpreted relative to
pytest's root directory:

.. code-block:: bash

   $ pytest --reruns 3 --rerun-exclude-path docs

The option may be passed multiple times to exclude multiple paths.

Re-run individual failures
--------------------------

To mark individual tests as flaky, and have them automatically re-run when they
fail, add the ``flaky`` mark with the maximum number of times you'd like the
test to run:

.. code-block:: python

  @pytest.mark.flaky(reruns=5)
  def test_example():
      import random
      assert random.choice([True, False])

To disable re-runs for a single test while keeping global re-runs enabled,
use ``reruns=0``. Since the marker takes priority over the global setting,
this test will not be re-run even when ``--reruns`` is passed:

.. code-block:: python

  @pytest.mark.flaky(reruns=0)
  def test_example():
      assert foo() == bar()

Note that ``--force-reruns`` still overrides the marker, and with
``--reruns-mode=append`` the marker count is added to the global count, so
``reruns=0`` does not opt the test out in that mode.

Note that when teardown fails, two reports are generated for the case, one for
the test case and the other for the teardown error.

You can also specify the re-run delay time in the marker, as well as the
backoff factor for an exponential backoff:

.. code-block:: python

  @pytest.mark.flaky(reruns=5, reruns_delay=2)
  def test_example():
      import random
      assert random.choice([True, False])

.. code-block:: python

  @pytest.mark.flaky(reruns=5, reruns_delay=2, reruns_delay_backoff_factor=2)
  def test_example():
      import random
      assert random.choice([True, False])

You can also specify an optional ``condition`` in the re-run marker:

.. code-block:: python

   @pytest.mark.flaky(reruns=5, condition=sys.platform.startswith("win32"))
   def test_example():
      import random
      assert random.choice([True, False])

Exception filtering can be accomplished by specifying regular expressions for
``only_rerun`` and ``rerun_except``. They override the ``--only-rerun`` and
``--rerun-except`` command line arguments, respectively.

Arguments can be a single string:

.. code-block:: python

   @pytest.mark.flaky(rerun_except="AssertionError")
   def test_example():
       raise AssertionError()

Or a list of strings:

.. code-block:: python

   @pytest.mark.flaky(only_rerun=["AssertionError", "ValueError"])
   def test_example():
       raise AssertionError()

Exception classes are also accepted and match any subclass:

.. code-block:: python

   @pytest.mark.flaky(only_rerun=[AssertionError, ValueError])
   def test_example():
       raise AssertionError()


You can use ``@pytest.mark.flaky(condition)`` similarly as ``@pytest.mark.skipif(condition)``, see `pytest-mark-skipif <https://docs.pytest.org/en/6.2.x/reference.html#pytest-mark-skipif>`_

.. code-block:: python

    @pytest.mark.flaky(reruns=2,condition="sys.platform.startswith('win32')")
    def test_example():
        import random
        assert random.choice([True, False])
    # totally same as the above
    @pytest.mark.flaky(reruns=2,condition=sys.platform.startswith("win32"))
    def test_example():
      import random
      assert random.choice([True, False])

Note that the test will re-run for any ``condition`` that is truthy.

Force rerun count
-----------------

To force a specific re-run count globally, irrespective of the number
of re-runs specified in test markers, pass ``--force-reruns``:

.. code-block:: bash

   $ pytest --force-reruns 5

Rerun mode
----------

By default the marker count takes strict priority over the global ``--reruns``
setting. To make them additive instead, pass ``--reruns-mode=append``. With
``append``, a test decorated with ``@pytest.mark.flaky(reruns=2)`` run with
``--reruns 4`` will be re-run up to ``2 + 4 = 6`` times:

.. code-block:: bash

   $ pytest --reruns 4 --reruns-mode append

Limit total reruns across the suite
------------------------------------

To cap the total number of reruns across the entire test suite regardless of
how many individual tests fail, pass ``--max-suite-reruns``. Once the limit
is reached, no further reruns occur even if individual tests have remaining
reruns:

.. code-block:: bash

   $ pytest --reruns 3 --max-suite-reruns 10

This is useful in large test suites to bound resource usage when many tests
are flaky at the same time. The cap applies after rerun selection, including
tests configured with ``--force-reruns`` and ``@pytest.mark.flaky``.

Show tracebacks for retried failures
------------------------------------

By default only the *final* attempt of a flaky test produces a traceback,
so failures from earlier attempts (including those of tests that ultimately
pass after a rerun) are silently discarded. To inspect them, pass
``--rerun-show-tracebacks``:

.. code-block:: bash

   $ pytest --reruns 2 --rerun-show-tracebacks

Each retried attempt's traceback is appended to the ``rerun test summary
info`` section. The section is emitted automatically when the flag is set,
so ``-rR`` is not required.

Output
------

Here's an example of the output provided by the plugin when run with
``--reruns 2`` and ``-r aR``::

  test_report.py RRF

  ================================== FAILURES ==================================
  __________________________________ test_fail _________________________________

      def test_fail():
  >       assert False
  E       assert False

  test_report.py:9: AssertionError
  ============================ rerun test summary info =========================
  RERUN test_report.py::test_fail
  RERUN test_report.py::test_fail
  FAILED test_report.py::test_fail
  ============================ short test summary info =========================
  FAIL test_report.py::test_fail
  ======================= 1 failed, 2 rerun in 0.02 seconds ====================

Note that output will show all re-runs. Tests that fail on all the re-runs will
be marked as failed.

.. START-COMPATIBILITY

Compatibility
-------------

* This plugin is *not* compatible with pytest-xdist's --looponfail flag.
* This plugin is *not* compatible with
  `pytest-forked <https://pypi.org/project/pytest-forked/>`_: both plugins
  override ``pytest_runtest_protocol``, and whichever runs first prevents
  the other from working, so tests are not re-run.
* When the core --pdb flag is used, reruns are disabled (a warning is
  emitted and tests are run once, as if ``--reruns 0`` was passed).
* This plugin is *not* compatible with the plugin
  `flaky <https://pypi.org/project/flaky/>`_, you can only have
  ``pytest-rerunfailures`` or ``flaky`` but not both.

.. END-COMPATIBILITY

.. START-PRIORITY

Priority
-------------

You can specify arguments in three places. So if you set the number of reruns in all three,
which one takes priority?

* Top priority is the marker, such as ``@pytest.mark.flaky(reruns=1)``
* Second priority is what's specified on the command line, like ``--reruns=2``
* Last priority is the ``pyproject.toml`` (or ``pytest.ini``) file setting, like ``reruns = 3``

The same order applies to the rerun filters: a marker's ``only_rerun`` /
``rerun_except`` beats ``--only-rerun`` / ``--rerun-except`` on the command
line, which in turn beats the ``only_rerun`` / ``rerun_except`` ini settings.

Additionally, all three can be overridden by passing ``--force-reruns`` argument
on the command line. Passing ``--reruns-mode=append`` makes the marker count and
the global ``--reruns`` / ``reruns`` ini setting additive instead of strict.

.. END-PRIORITY


.. START-CONTRIBUTING

Resources
---------

- `Issue Tracker <https://github.com/pytest-dev/pytest-rerunfailures/issues>`_
- `Code <https://github.com/pytest-dev/pytest-rerunfailures/>`_

Development
-----------

* Test execution count can be retrieved from the ``execution_count`` attribute
  in test ``item``'s object. Example:

  .. code-block:: python

    @hookimpl(tryfirst=True)
    def pytest_runtest_makereport(item, call):
        print(item.execution_count)

.. END-CONTRIBUTING