Testing

Testing Standards

When to Use

• Writing tests for any application or service
• Choosing what level of testing to apply
• Designing test data, fixtures, or factories
• Setting up CI test pipelines
• Evaluating LLM/AI agent outputs
• Building evaluation harnesses for AI systems
• Auditing an existing project's test suite for coverage gaps, weak assertions, or structural problems
• Improving test quality (naming, isolation, mocking strategy, data factories)
• Identifying untested critical paths in an existing codebase

---

Testing Pyramid

| Level | Scope | Speed | Count | |-------|-------|-------|-------| | Unit | Single function/class, no I/O | { it("applies discount when coupon is valid", () => { ... }); it("rejects order when inventory is insufficient", () => { ... }); });

---

## Test Data

### Fixtures and Factories

- Use **factory functions** that create valid test data with sensible defaults, allowing per-test overrides:

```python
def make_user(**overrides) -> User:
    defaults = {
        "id": "usr_test",
        "name": "Test User",
        "email": "test@example.com",
        "role": "viewer",
    }
    return User(**(defaults | overrides))

# Usage
admin = make_user(role="admin")

function makeUser(overrides: Partial = {}): User {
  return {
    id: "usr_test",
    name: "Test User",
    email: "test@example.com",
    role: "viewer",
    ...overrides,
  };
}

• Use pytest's tmp_path for file system tests. Never write to real paths.
• Use freezegun or time_machine in Python for time-dependent tests.

Test Isolation

• Each test must be independent. No test should depend on another test's state or execution order.
• Reset shared state (databases, caches, global variables) between tests.
• Use transactions with rollback for database tests.

---

Snapshot Testing

• Use for complex output that is correct but tedious to assert manually: HTML output, API response shapes, serialized configs.
• Review snapshot diffs carefully — don't blindly update snapshots.
• Keep snapshots small. If a snapshot is >50 lines, assert specific fields instead.

---

Coverage

• Target 80% line coverage as a floor, not a ceiling.
• Coverage measures what your tests execute, not what they verify. High coverage with weak assertions is worthless.
• Focus coverage effort on business logic, not boilerplate.
• Never add tests purely to increase coverage numbers. Every test should verify meaningful behavior.

---

CI Integration

• Tests run on every PR. Merging is blocked on test failure.
• Test suite must complete in **, "relevance": , "reasoning": ""}

"""

- Use a **stronger model** as judge than the model being evaluated.
- Include the expected answer in the judge prompt for calibration.
- Run the judge multiple times (3-5) and take the median to reduce variance.
- Track judge agreement rate — if it varies wildly, refine the rubric.

### Regression Testing

- Maintain a **golden set** of input/output pairs that represent known-correct behavior.
- Run against the golden set on every prompt change, model update, or tool change.
- Track pass rates over time. A drop signals regression.
- Pin model versions in production. Test new model versions against the eval suite before promoting.

### Cost-Aware Testing

- Log token usage (input + output) for every eval run.
- Set budget limits per eval suite run.
- Use cheaper models (e.g., GPT-4o-mini, Claude Haiku) for high-volume regression testing.
- Reserve expensive models for final validation and judge evaluations.
- Cache LLM responses during development to avoid re-running identical calls.

---

## Property-Based Testing

For functions with broad input domains, use property-based testing to generate many random inputs and verify invariants:

```python
from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_preserves_length(items):
    assert len(sorted(items)) == len(items)

@given(st.lists(st.integers(), min_size=1))
def test_sort_first_element_is_minimum(items):
    assert sorted(items)[0] == min(items)

Use when:

• Input space is large and edge cases are hard to enumerate manually.
• Function should satisfy invariants regardless of input (sort, serialize/deserialize, encode/decode).

IDE Integration

For configuring pytest and Vitest test runners in VS Code / Cursor, including Test Explorer integration, debug configurations, and local task automation, see the ide-setup skill.

iOS / Swift Testing

For iOS-specific testing patterns including Swift Testing framework (@Suite, @Test, #expect), ViewInspector for SwiftUI view testing, URLProtocol network mocking, SwiftData test containers, and protocol-based mock infrastructure, see the ios-testing skill.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: michaelsvanbeek
• Source: michaelsvanbeek/personal-agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.