Audit Tautological Tests

Audit Tautological Tests

A tautological test is one where you delete the implementation and the test still passes — because the test is asserting against its own mocks, not against real behavior. These tests give false confidence and let bugs ship.

When to use

• A bug shipped to code that had "100% coverage."
• High test coverage + high bug rate is a smell.
• Reviewing a PR with many new tests that look suspicious.
• Refactor risk-assessment: which tests would actually catch a regression?

When NOT to use

• Test suite is genuinely good and bugs are escaping for other reasons (test environment doesn't match prod).
• Looking for slow tests, flaky tests, or coverage gaps — different audits.

The five test smells

1. The mock is the implementation

// Tautological
test("createUser saves user", async () => {
  const mockSave = jest.fn().mockResolvedValue({ id: 1 });
  const repo = { save: mockSave };
  const result = await createUser(repo, { email: "a@b.com" });
  expect(result.id).toBe(1);                  // asserting on the mock's value
  expect(mockSave).toHaveBeenCalled();        // asserting it was called
});

What does this test? That createUser returns whatever repo.save returns. If createUser's body is return repo.save(...), the test passes. If it's return null (bug!), the test... still passes? No — the test asserts result.id === 1, but the mock returned { id: 1 }, so result is null and the assertion fails.

OK, this one isn't quite tautological — but it's close. It tests pass-through, nothing else.

// Worse — fully tautological
test("createUser saves user", async () => {
  const repo = { save: jest.fn() };
  await createUser(repo, { email: "a@b.com" });
  expect(repo.save).toHaveBeenCalled();
});

Delete the implementation entirely (async function createUser() {}). Test fails (save not called). OK. Now make the implementation repo.save({}). Test passes. The test has no opinion on what gets saved or whether the email is validated.

Fix: assert on the content of what save was called with, and assert the result has the right shape:

expect(repo.save).toHaveBeenCalledWith(expect.objectContaining({
  email: "a@b.com",
  createdAt: expect.any(Date),
  status: "pending",
}));

2. Snapshot tests with no behavior contract

test("renders user card", () => {
  const tree = render().toJSON();
  expect(tree).toMatchSnapshot();
});

Snapshot tests pass as long as the rendered output matches what was recorded last time. They catch unintended output changes — but only if someone reviews the snapshot diff, which often nobody does ("just run jest -u to update").

Snapshot tests for entire components rarely test the thing you care about: that clicking "Save" actually saves.

Fix: keep snapshots only for stable UI primitives (icons, brand assets). For interactive components, write behavior tests:

test("clicking save submits the form", async () => {
  const onSave = jest.fn();
  render();
  await userEvent.click(screen.getByRole("button", { name: /save/i }));
  expect(onSave).toHaveBeenCalledWith(expect.objectContaining({ email: "a@b.com" }));
});

3. Mocking the system under test

test("PaymentService charges customer", () => {
  const svc = new PaymentService();
  svc.charge = jest.fn().mockResolvedValue({ id: "pay_1" });   // mocked itself!
  // ... rest of test
});

Or, more subtly:

jest.spyOn(svc, "validate").mockReturnValue(true);              // bypassing the validation
expect(await svc.charge(args)).toEqual({ id: "pay_1" });

The test confirms the non-validate path works — which is the path that doesn't exist in production. Real payments go through validate.

Fix: mock at the boundary (database, external API), not internal methods of the unit under test.

4. "Round-trip the mock"

test("API returns user", async () => {
  fetchMock.get("/api/users/1", { id: 1, email: "a@b.com" });
  const user = await client.getUser(1);
  expect(user).toEqual({ id: 1, email: "a@b.com" });
});

This tests that client.getUser makes an HTTP request and returns the JSON. It does not test that the API actually returns this shape. Production /api/users/:id could return { user_id, mail } and this test would still pass.

Fix: if you're mocking the API, you're testing the client's parsing. Use a contract test or schema validator that lives near the actual API definition.

5. Tests that import production data

test("calculator sums correctly", () => {
  expect(sum(2, 2)).toBe(4);
});
test("calculator with real data", () => {
  const inputs = require("./fixtures/yesterday-prod-export.json");
  expect(sum(inputs.a, inputs.b)).toBe(inputs.expected);
});

The second test "passes" because the fixture was generated from the same code. There's no independent ground truth. If the implementation has the same bug as when the fixture was generated, the test never catches it.

Fix: ground truth must come from outside the system — the spec, the user requirement, a hand-computed example.

The mutation test

The killer test for tautology: mutate the implementation and see if any test fails.

• Delete the function body entirely. Run tests. Any pass? Those tests are tautological.
• Replace return value with a constant. Run tests. Any still pass that shouldn't?
• Invert a conditional (if (x) → if (!x)). Run tests. Any still pass?

Tools like Stryker (JS/TS, .NET) or mutmut (Python) automate this. Run them. The "mutation score" is the real signal of test quality, not coverage.

Symptoms in your codebase

# Find tests that only assert "was called" without arguments
grep -r "toHaveBeenCalled()" tests/ | wc -l
# vs
grep -r "toHaveBeenCalledWith" tests/ | wc -l

If the first is much higher than the second, you have a tautology problem.

# Find snapshot tests
find . -name "__snapshots__" | xargs -I {} ls {} | wc -l

How many of these would survive deletion of the component's logic?

What good tests look like

1. Test the contract, not the implementation. "When I call chargeCustomer with valid input, a payments row is written and a Razorpay order is created" — not "the function calls db.save and razorpay.orders.create in that order."
2. Mock at the boundary, not internally. The DB, the HTTP client, the file system. Not the service's own private methods.
3. Prefer integration tests for business logic; reserve unit tests for pure functions and tricky algorithms.
4. Hand-write fixtures that represent real shapes, not whatever the impl currently produces.
5. Aim for mutation score > 70% on critical code (payments, auth, data).

Anti-patterns

• toHaveBeenCalled() without With — checks the function ran, not what it did.
• Mocking everything — leaves nothing real to test.
• Snapshot tests as "fast and easy" coverage — coverage without contract.
• Mocking the unit under test's own methods — bypasses the code you're testing.
• expect(actual).toEqual(actual) (with renamed variables) — yes, this happens.
• try { ... } catch { /* test passes */ } — error-swallowing tests.
• Empty assertion blocks — test("works", () => { doThing(); }); — no expect. Linter rule should catch.
• Tests with no DB / no HTTP for code that does both — too pure to be useful.
• Refusing to delete tautological tests because "coverage will drop" — coverage was lying.
• Adding tests after the bug ships, asserting the now-correct behavior, never asserting the wrong behavior wouldn't pass — write the test before the fix; watch it fail; fix; watch it pass.

Verify it worked

• [ ] Ran a mutation test (Stryker / mutmut) on critical modules; mutation score > 70%.
• [ ] Found tests where deleting the implementation didn't break them; they're either fixed or deleted.
• [ ] toHaveBeenCalled() count is much lower than toHaveBeenCalledWith() count.
• [ ] Snapshot tests cover only stable primitives, not interactive components.
• [ ] No test mocks methods of the unit under test.
• [ ] Critical path code (payments, auth) has integration tests against real DB / HTTP, not pure mocks.
• [ ] When a real bug shipped, you can identify which test should have caught it; that test is added or the gap is acknowledged.
• [ ] PR review surfaces tautology smells; toHaveBeenCalled() without args is a flag.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: MaheshAwasare
• Source: MaheshAwasare/claude-skills-pro
• License: MIT

Install and usage instructions live in the source repository linked above.