Python Testing

python-testing

Build test infrastructure for a Python package that stays trustworthy under pressure: pytest configuration the installed pytest actually reads, branch coverage with a gate proven to fail, Hypothesis and mutation testing as backstops against tests that execute code without checking it, and multi-version runs. The failure this skill prevents is green-but-meaningless — gates that silently pass, config tables that are silently ignored, and suites edited (by humans or agents) until they stop proving anything.

When NOT to use

• Writing tests for a single file or function, or improving one module's tests —

do that directly; no infrastructure needed.

• Diagnosing or fixing one failing test — read the test and the code under test.
• CI workflow YAML — GitHub Actions jobs, version-matrix wiring, coverage

upload/combine across CI legs. The python-ci skill, if installed, owns that. This skill decides what runs; CI decides where.

• Packaging work — pyproject metadata, build backend, converting a repo to src

layout as a packaging exercise. The python-packaging skill, if installed.

• Git hook wiring — the python-precommit skill; full test suites do not belong

in pre-commit hooks anyway.

Workflow

1. Survey the repo

Run the read-only audit first — it catches shadowed and split config sources:

python3 "${CLAUDE_SKILL_DIR}/scripts/check_test_config.py" --root .

Then establish the facts later steps depend on:

uv run pytest --version            # major version decides the config table (step 2)
grep -n "requires-python" pyproject.toml
ls pytest.ini setup.cfg tox.ini .coveragerc noxfile.py 2>/dev/null

Note the layout: src// + sibling tests/ (src layout) or package at the repo root (flat). Convention throughout: invoke tools through uv (uv run, uvx). If uv is unavailable, the plain fallback is python -m pip install -e ".[dev]" then python -m pytest — same flags everywhere; stated once here.

2. pytest config — one table, in pyproject.toml

The portable location is [tool.pytest.ini_options]. A native [tool.pytest] TOML table (real arrays/booleans instead of INI-style strings) is claimed for pytest 9+, but that claim is single-sourced — verify it against the installed version and the changelog (https://docs.pytest.org/en/stable/changelog.html) before using it. On any pytest below 9, [tool.pytest] is ignored silently: no error, tests still run, strictness flags never apply. When in doubt, ini_options works on every supported pytest.

[tool.pytest.ini_options]
minversion = "8.0"
testpaths = ["tests"]
addopts = "-ra --strict-markers --strict-config"
filterwarnings = ["error"]

• --strict-markers — typo'd or undeclared @pytest.mark.* markers fail

instead of silently never running.

• --strict-config — typo'd config keys become errors.
• filterwarnings = ["error"] — deprecations fail now, not at upgrade time. On

a legacy dependency tree this is noisy; add targeted "ignore::" entries rather than deleting the gate.

• Async project? Add asyncio_mode = "auto" (with pytest-asyncio as a dev dep)

to drop the per-test marker.

• Keep --cov OUT of addopts. Coupling every run to coverage slows

single-test loops and can confuse debugger breakpoints; pass coverage flags in the explicit commands below and in CI. Trade-off: some teams keep it in addopts so coverage is never forgotten — if the user prefers that, keep it and note --no-cov disables it for one run.

• One config source. pytest uses the first of pytest.ini,

pyproject.toml, tox.ini, setup.cfg and ignores the rest — a stray pytest.ini silently shadows everything in pyproject. Merge, then delete.

3. Layout and discovery

src layout (src// with tests in a sibling tests/, not inside the package) is the reliable shape: with uv run, pytest imports the installed package, so import and packaging bugs surface in tests instead of in users' environments (https://docs.pytest.org/en/stable/explanation/goodpractices.html). testpaths = ["tests"] and coverage source = ["src"] then line up.

• "pytest can't find/import my package" right after moving to src layout

usually means the package is not installed in the environment — uv sync (or pip install -e .) and re-run.

• Do not migrate a flat-layout repo to src layout just for tests; that is a

packaging change (python-packaging territory). Testing works in flat layout — you only lose the installed-package guarantee.

• Shared fixtures live in tests/conftest.py.

4. Coverage — branch on, gate proven

[tool.coverage.run]
source = ["src"]
branch = true
parallel = true
relative_files = true

[tool.coverage.report]
fail_under = 85
show_missing = true
skip_covered = true
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
]

• branch = true is the point — line coverage says a line ran; branch coverage

says both sides of each if did. Line-only numbers overstate.

• fail_under of 80–90 is the defensible band. 100% is anti-recommended: it

breeds assert-free tests and pragma sprawl. Exclude genuinely unreachable lines instead.

• If a .coveragerc exists it silently wins over [tool.coverage.*] in

pyproject — keep exactly one source.

• Run with uv run pytest --cov --cov-report=term-missing.

Prove the gate. pytest-cov has documented cases of returning exit code 0 when an aggregate threshold is missed, and rounding can pass 66.6% against a 67 bar. Never assume — demonstrate:

uv run pytest --cov --cov-fail-under=100 -q; echo "exit=$?"

Expect a non-zero exit (unless coverage is genuinely 100%). If it exits 0, gate in two steps instead — uv run pytest --cov then uv run coverage report --fail-under=85 — the standalone coverage command's exit code is reliable.

• Parallel runs (pytest -n auto via pytest-xdist) require parallel = true

and relative_files = true, and subprocesses spawned by tests need COVERAGE_PROCESS_START — otherwise coverage comes back empty or 0%.

• A threshold is a floor, not a quality signal — it is trivially satisfied by

tests without assertions. Steps 5–6 are the countermeasures. Patch/delta coverage on PRs is CI wiring (python-ci skill, if installed).

Flag-by-flag detail, precedence rules, and combine mechanics: [references/configuration.md](references/configuration.md).

5. Property-based tests where they pay

Add Hypothesis for parsers, serializers, codecs, and algorithmic code — anywhere a property (round-trip, idempotence, invariant) exists. A 426-project corpus study found each property-based test kills roughly 50x the mutations of an average unit test (https://dl.acm.org/doi/10.1145/3764068).

uv add --dev hypothesis    # exact version pinned by uv.lock

from hypothesis import given, strategies as st

@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s

• Prefer bounded strategies (st.integers(min_value=1)) over assume()

filtering — rejecting most examples raises HealthCheck.filter_too_much.

• Register settings profiles in tests/conftest.py; CI runs the heavy one via

pytest --hypothesis-profile=ci:

from hypothesis import settings
settings.register_profile("ci", max_examples=1000)
settings.register_profile("dev", max_examples=25)

• Persist the .hypothesis/ example database (commit it, or cache it in CI) —

otherwise every run rediscovers known counterexamples from scratch.

• DeadlineExceeded only on CI machines means slow shared runners — raise or

disable deadline in the ci profile deliberately; don't delete the test.

Property patterns, strategy design, and stateful testing: [references/beyond-coverage.md](references/beyond-coverage.md).

6. Mutation testing — do the tests notice bugs?

Coverage proves a line executed; mutation testing proves a test fails when that line is wrong. It is the direct check on assertion quality, which matters most when agents write or edit tests: a suite that was edited until green proves nothing by itself. Options — no settled winner, choose per constraints:

| Tool | Character | Watch out | | --- | --- | --- | | mutmut | Mature, simplest CLI, pytest-first | Full-repo runs are slow — scope paths_to_mutate | | cosmic-ray | Session-based, resumable, distributed | Heaviest setup; overkill for small packages | | pytest-gremlins | Runs as a pytest plugin | Newer; adoption evidence is thin — vet maintenance first |

Install as a dev dependency so it runs inside the project environment and gets pinned by uv.lock (an unpinned uvx one-off executes whatever was published last — a reproducibility and supply-chain hole):

uv add --dev mutmut
uv run mutmut run

• Scope the first run to one core module ([tool.mutmut] paths_to_mutate),

then triage survivors: a surviving mutant means a missing or weak assertion — or a genuinely equivalent mutant, which you mark and move on.

• Do not gate CI on mutation score on day one; run it scoped or nightly and

treat the score as a diagnostic. Practitioners report 85–95% mutation scores as far more meaningful than the same number in line coverage.

Triage workflow and the agent-era rationale: [references/beyond-coverage.md](references/beyond-coverage.md).

7. Multi-version runs

Match the tool to the project — this debate is genuinely unsettled:

| Situation | Use | | --- | --- | | Simple deps, few version-specific branches | plain uv loop (below) | | Complex matrix (extras, backends, C extensions) | tox or nox, uv-accelerated | | Want Python (not INI) config, dynamic sessions | nox | | Already on tox | keep it; add tox-uv — speed-up, no config rewrite |

for v in 3.10 3.11 3.12 3.13; do
  uv run -p "$v" --with pytest --with pytest-cov pytest || exit 1
done

For tox or nox, swap the environment backend to uv (tox-uv, nox's uv backend): environment setup gets dramatically faster with no config-model change.

• nox footgun: a repo-root .python-version file can silently pin every

session to the local dev version (https://github.com/wntrblm/nox/issues/1038).

• Never enforce fail_under inside each version leg — a 3.13-only branch drops

coverage on the 3.10 leg. Locally: run the loop with parallel = true, then uv run coverage combine && uv run coverage report --fail-under=85 once. The CI version of that split (artifacts, combine job) belongs to python-ci.

Configs and trade-offs in depth: [references/multi-version.md](references/multi-version.md).

8. Verify

1. uv run pytest — green.
2. The gate-trip demonstration from step 4 exits non-zero.
3. python3 "${CLAUDE_SKILL_DIR}/scripts/check_test_config.py" --root . exits 0.
4. Report to the user: config location and table chosen (and why), thresholds,

and which backstops (Hypothesis, mutation, multi-version) are wired vs deferred.

Output spec

Done means:

• Exactly one pytest config source and one coverage config source; the pytest

table matches the installed major version.

• Branch coverage on; fail_under in the 80–90 band; the gate demonstrated to

exit non-zero when missed.

• Suite runs green via uv run pytest from a clean checkout.
• Where requested: Hypothesis dev dep with profiles; mutation tool pinned with

a first scoped run triaged; multi-version runner chosen with stated rationale.

• No CI YAML written or modified.

Failure modes & gotchas

| Symptom | Cause / fix | | --- | --- | | Coverage gate "passes" while below threshold | pytest-cov aggregate exit-code bug or rounding; prove the gate (step 4) or gate via standalone coverage report --fail-under | | Strictness flags mysteriously not applied | [tool.pytest] on pytest <9 (silently ignored), or a shadowing pytest.ini — one source, right table | | Coverage 0% or empty under pytest -n auto | missing parallel/relative_files; test-spawned subprocesses need COVERAGE_PROCESS_START | | Coverage config edits change nothing | a .coveragerc wins over pyproject — delete one | | Matrix legs fail on version-specific code | fail_under enforced per leg; combine first, gate once | | Suite green right after an agent edited the tests | proves nothing — review test diffs before source diffs, run mutation testing; researchers have shown agents faking green via an injected conftest.py, so verify in an environment the agent cannot write to | | Generated tests pass but catch no regressions | mock-heavy tests, a documented agent bias (https://arxiv.org/abs/2602.00409) — require behavioral assertions on the public API, not mock call counts | | Hypothesis rejects most examples (filter_too_much) | replace assume() with bounded strategies | | Hypothesis finds a bug, next run forgets it | .hypothesis/ database not persisted across runs | | Flaky DeadlineExceeded on CI only | slow shared runners — adjust deadline in the ci profile |

Files

• scripts/check_test_config.py — read-only audit of pytest/coverage config

(shadowed sources, wrong config table, missing branch coverage or gate). Exits non-zero when it finds errors; never modifies the repo.

• references/configuration.md — pytest and coverage config in depth:

precedence, the pytest-9 table question, the exit-code bug, xdist/combine mechanics.

• references/beyond-coverage.md — Hypothesis patterns and settings; mutation

tool comparison and survivor triage; why agent-edited green suites need backstops.

• references/multi-version.md — uv loop vs tox vs nox, uv-backed runners,

coverage across versions.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Paldom
• Source: Paldom/python-skills
• License: MIT

Install and usage instructions live in the source repository linked above.