Python "pytest-randomly ordering breaks a test" in CI

By Kaveh Alemi·Latchkey

pytest-randomly shuffles test order each run. A test that passes in file order but fails when shuffled has an order dependency: it relies on state another test created or left behind.

What this error means

A test fails intermittently in CI; the header prints "Using --randomly-seed=NNNN" and re-running with that seed reproduces the failure, while a different seed passes.

Using --randomly-seed=305418216
...
FAILED tests/test_orders.py::test_total - AssertionError

Common causes

Shared mutable module/global state

One test mutated a module-level or class-level value the failing test assumed was pristine.

Reliance on data another test created

The failing test only passes when a specific earlier test ran first and set up data.

How to fix it

Reproduce with the seed and isolate state

1. Re-run with -p randomly --randomly-seed=NNNN from the failing run to reproduce.
2. Make each test set up its own data; reset module/class globals in fixtures.
3. Replace shared mutable state with function-scoped fixtures.

pytest -p randomly --randomly-seed=305418216 tests/test_orders.py::test_total

How to prevent it

• Keep tests independent of execution order.
• Reset or avoid shared mutable globals.
• Run pytest-randomly in CI to surface order dependence early.

Frequently asked questions

Related guides

References

• Python documentation
• pip documentation
• GitHub Actions documentation