Mutation Testing

> Project: honnibal/claude-skills. "Claude skills I'm experimenting with. Please review carefully before use." > > License: MIT License. Full license text is in [LICENSE](LICENSE) alongside this file. > > Snapshot: Frozen copy of mutation-testing.md.txt as of 2026-05-22. The maintained version lives upstream and may have evolved since this snapshot.

Mutation Testing

You are now in mutation testing mode. Your job is to assess the strength of the project's test suite by introducing deliberate bugs (mutations) into the source code, one at a time, and checking whether any test fails. A mutation that the tests don't catch reveals a gap in coverage.

Scope

$ARGUMENTS

• If the user names specific files or directories, restrict mutations to those.
• If no argument is given, look at the project's source layout and use

AskUserQuestion to agree on a starting scope. Don't try to mutate everything at once — pick a module or package and work through it.

• Test files themselves are out of scope for mutation. Only mutate production

code.

• Prioritise code that has meaningful logic (branching, arithmetic, state

changes) over boilerplate, config, or trivial accessors.

Pre-flight

Before mutating anything:

1. Clean working tree. Run git status on the files in scope. If there are

uncommitted changes to any file you plan to mutate, stop and ask the user to commit or stash first. You need a clean baseline so every mutation can be reverted cleanly with git checkout -- .

1. Find the test runner. Look for pytest.ini, `pyproject.toml

[tool.pytest], setup.cfg [tool:pytest], tox.ini, a Makefile test target, or a tests/` directory. Ask the user if you can't determine how to run tests. Confirm that the test suite passes on the unmodified code before starting — if it doesn't, stop and tell the user.

1. Map the code. Read the files in scope. Build a mental model of each

module's structure and data flow so you can choose mutations that are meaningful rather than trivially dead.

Workflow

Work through the files in scope one at a time. For each file:

1. Choose mutations. Read the file and identify 3–8 candidate mutations

using the catalogue below. Prefer mutations that test interesting behaviour — a bug a real developer might introduce — over mechanical operator swaps on dead code. For each candidate, write a one-line description of what the mutation does and what behaviour it should break.

1. Apply, test, revert. For each mutation:

a. Apply the mutation using Edit. Change as little as possible — usually a single line. b. Run the test suite (or the relevant subset if the suite is large). Use a timeout — if the tests hang, that still counts as "caught" (the mutation broke something). c. Record the result:

• Killed — a test failed. Note which test, and briefly assess how

helpful the failure message is. Would a developer reading this failure immediately understand what went wrong, or would they have to dig?

• Survived — no test failed. This is a gap. Note what behaviour is

untested. d. Revert the mutation: git checkout -- . Confirm the file is back to its original state before moving on.

1. Never leave a mutation in place. After each test run, revert immediately.

If something goes wrong and you're unsure of the file state, run git diff to check, and git checkout -- to restore.

Use TaskCreate to track progress across files when there are more than a handful.

Mutation Catalogue

Choose mutations from these categories, ordered roughly from most to least likely to reveal meaningful test gaps.

1. Delete or skip a side effect

Remove or comment out a line that modifies state — an assignment, a method call that updates an object, an append to a list, a cache write, a database call. This tests whether the suite verifies that the side effect actually happened.

# Original
self.count += 1
results.append(item)

# Mutation: delete the line entirely

2. Negate or invert a condition

Flip a boolean condition: if x → if not x, x > 0 → x = → >, == 0 → == 1, range(n) → range(n - 1).

# Original
if retry_count .py`, follow that pattern.
   Don't create new test files when an existing one covers the same module.
2. **Write focused tests.** Each test should target one survived mutation. Name
   the test to describe the behaviour it verifies, not the mutation it catches.
   For example: `test_cache_is_populated_after_first_call`, not
   `test_mutation_2`.
3. **Verify.** Run the test suite to confirm your new tests pass on the
   unmodified code. Then re-apply each corresponding mutation and confirm the
   new test catches it.
4. **Don't over-test.** If a single well-designed test would catch multiple
   survived mutations, write one test, not several. Aim for tests that verify
   real behaviour rather than just satisfying the mutation score.

## Critical Rules

- **Always revert.** After every test run, revert the mutation before doing
  anything else. Never stack mutations. Never leave mutated code in the working
  tree.
- **Verify the revert.** Run `git diff ` after reverting to confirm the
  file is clean. If it's not, run `git checkout -- ` again.
- **Don't mutate test files.** Only mutate production code.
- **Don't mutate imports, type annotations, or docstrings.** These rarely
  reveal meaningful test gaps and produce noise.
- **Keep it targeted.** 3–8 mutations per file is enough. Prefer quality over
  quantity — a well-chosen mutation that reveals a real gap is worth more than
  ten trivial operator swaps.
- **Ask when uncertain.** If you're unsure whether a mutation is meaningful or
  whether a test failure constitutes "catching" it, use `AskUserQuestion`.
- **Respect the user's time.** If the test suite is slow, ask whether they
  want to run the full suite or a relevant subset for each mutation.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [hugobowne](https://github.com/hugobowne)
- **Source:** [hugobowne/show-us-your-agent-skills](https://github.com/hugobowne/show-us-your-agent-skills)
- **License:** MIT
- **Homepage:** https://hugobowne.github.io/show-us-your-agent-skills/

Install and usage instructions live in the source repository linked above.