Py Test Quality

Python Test Quality Analysis

Measure test coverage and verify test suite effectiveness using coverage analysis and mutation testing.

Objectives

1. Measure code coverage comprehensively
2. Identify untested code paths
3. Verify test suite catches bugs (mutation testing)
4. Enable safe refactoring through high test coverage
5. Track coverage trends over time

Required Tools

Add to [dependency-groups] dev: "pytest", "pytest-cov", "mutmut", "coverage" Optional: "cosmic-ray" (advanced mutation testing)

• pytest-cov: Code coverage measurement
• mutmut: Mutation testing - verifies tests catch bugs
• cosmic-ray: Advanced mutation testing (slower)

Permissions: Run py-quality-setup first to configure .claude/settings.local.json with all needed tool permissions.

Coverage Analysis

Measure Coverage

# Run tests with coverage
pytest --cov=. --cov-report=term-missing
pytest --cov=. --cov-report=html  # Generate HTML report
pytest --cov=. --cov-report=term --cov-report=html  # Both

# Coverage with specific targets
pytest --cov=src --cov=lib tests/
pytest --cov=mypackage --cov-branch  # Include branch coverage

# Fail if coverage below threshold
pytest --cov=. --cov-fail-under=80

# Show only uncovered lines
pytest --cov=. --cov-report=term-missing:skip-covered

Configure Coverage

Add to pyproject.toml (see py-quality-setup for base configuration):

[tool.coverage.run]
source = ["src"]  # Adjust to your source directory
omit = [
    "*/tests/*",
    "*/test_*.py",
    "*/__pycache__/*",
    "*/venv/*",
    "*/.venv/*",
]
branch = true  # Enable branch coverage

[tool.coverage.report]
precision = 2
show_missing = true
skip_covered = false
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise AssertionError",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
    "if TYPE_CHECKING:",
    "@abstractmethod",
]

[tool.coverage.html]
directory = "htmlcov"

Interpret Coverage Reports

Name                 Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------
src/auth.py            45      5     12      2    87%   23-25, 67
src/database.py        89      0     18      0   100%
src/handlers.py       123     35     28     12    68%   45-78, 99-110
----------------------------------------------------------------
TOTAL                 257     40     58     14    82%

Key metrics:

• Stmts: Total statements
• Miss: Uncovered statements
• Branch: Total branches (if/else, etc.)
• BrPart: Partially covered branches (one path tested, not both)
• Cover: Coverage percentage
• Missing: Line numbers not covered

Coverage targets:

• ≥80%: Minimum acceptable
• ≥90%: Good coverage
• 100%: Ideal (may not be practical for all code)

Mutation Testing

Mutation testing verifies your tests actually catch bugs by introducing small changes (mutations) and checking if tests fail.

Run Mutation Testing

# Using mutmut (easier, faster)
mutmut run                    # Run all mutations
mutmut run --paths-to-mutate=src/  # Specific directory
mutmut results                # Show summary
mutmut show      # Show specific mutation
mutmut apply     # Apply mutation to see code change

# Common workflow
mutmut run
mutmut results               # Shows: survived, killed, timeout
mutmut show 1                # Examine first surviving mutation

Configure Mutmut

Add to setup.cfg or pyproject.toml:

[tool.mutmut]
paths_to_mutate = "src/"
backup = false
runner = "pytest -x --tb=short"
tests_dir = "tests/"

Interpret Mutation Results

Legend for output:
🎉 Killed mutants: Tests caught the bug (good!)
⏰ Timeout: Mutation created infinite loop (acceptable)
🤔 Suspicious: Needs investigation
🙁 Survived: Bug not caught by tests (bad!)

Mutation score: killed / (killed + survived) * 100%

Target mutation scores:

• ≥75%: Good test quality
• ≥85%: Excellent test quality
• 100%: Perfect (rarely achievable)

Address Surviving Mutations

# 1. Identify surviving mutations
mutmut results

# 2. Show specific mutation
mutmut show 5

# Example output:
# src/auth.py:23
# -    if user.age >= 18:
# +    if user.age > 18:

# 3. Write test to kill this mutation
def test_user_exactly_18_is_adult():
    user = User(age=18)
    assert is_adult(user) is True  # This would fail with > instead of >=

# 4. Re-run mutmut
mutmut run

# 5. Verify mutation now killed
mutmut results

Coverage-Guided Refactoring

Golden rule: Only refactor well-tested code. If coverage is low, write tests first.

Workflow

1. Run: pytest --cov=. --cov-report=html --cov-fail-under=80
2. If coverage = to > in age check
5. Realize: Missing boundary test for age exactly 18
6. Write: test_user_exactly_18_is_adult()
7. Run: mutmut run --paths-to-mutate=src/auth.py
8. Results: 16 killed, 2 survived (89% mutation score)
9. Repeat for remaining survivors
10. Final: 18 killed, 0 survived (100% mutation score)

Example: Coverage-guided refactoring session

1. Target: Refactor src/payment.py (complexity D, 150 lines)
2. Check coverage: pytest --cov=src/payment.py --cov-report=term-missing
3. Coverage: 92% (good! safe to refactor)
4. Check test quality: mutmut run --paths-to-mutate=src/payment.py
5. Mutation score: 78% (acceptable)
6. Proceed with refactoring:
   - Extract 4 smaller functions
   - Reduce complexity from D to A/B
7. After refactoring:
   - pytest --cov=src/payment.py --cov-fail-under=92 ✓
   - mutmut run --paths-to-mutate=src/payment.py
   - Mutation score: 80% (improved!)
8. Commit changes with confidence

Example: Set up coverage tracking in CI

1. Add to pyproject.toml:
   [tool.coverage.run]
   source = ["src"]
   branch = true

2. Create .github/workflows/test.yml:
   - pytest --cov=. --cov-fail-under=80

3. First run fails: Coverage 67%
4. Write tests to reach 80%
5. CI now passes
6. All future PRs must maintain 80% coverage
7. Consider increasing threshold as coverage improves

Related Skills

• Prerequisites: py-quality-setup (configure pytest-cov in pyproject.toml)
• Enables: py-security, py-code-health, py-complexity (safe refactoring with test coverage)
• Enforcement: py-git-hooks (add coverage checks to CI)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: l-mb
• Source: l-mb/python-refactoring-skills
• License: MIT

Install and usage instructions live in the source repository linked above.