Ai System Testing

AI features fail differently from deterministic software. The same input produces different outputs, correctness is subjective, and failure modes include hallucination, prompt injection, and silent quality decay. A chatbot that confidently cites a fabricated URL passes every toBeDefined() check. This skill covers how to test AI features rigorously despite nondeterminism: versioned prompts, eval suites scored against golden datasets, statistical and property-based assertions, tool-call validation, grounding checks, and red-team safety scans.

---

Quick Route

| Situation | Go to | |-----------|-------| | Prompt changed, need to catch quality regressions | Prompt Regression Testing → references/prompt-regression.md | | Run the same prompt across providers/models and compare | Cross-Provider Regression → references/tooling-evals.md | | Score open-ended output (relevance/completeness/safety) | Response Quality Evaluation → references/eval-framework.md | | Agent calls tools/functions — verify selection and args | Tool-Call Validation → references/tooling-evals.md | | Output is nondeterministic and exact-match keeps flaking | Nondeterminism Strategies | | AI states facts / cites sources / runs over RAG | Hallucination & Grounding | | Pre-launch jailbreak, injection, PII, system-prompt leak | AI Safety Testing → references/tooling-evals.md | | An AGENT (test harness, coding agent) reads tool output / RAG / scan reports / logs | Agent-as-Target Injection → references/injection-detector.md |

---

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

AI features under test:

• What AI features exist? (Chat, summarization, classification, code gen, recommendations, search) — determinism expectations differ per type.
• Which provider/model? (Anthropic, OpenAI, Google, open-source) — drives the eval harness and red-team backend.
• Are prompts hardcoded, template-based, or dynamically constructed? — only versioned prompts are regression-testable.
• Is RAG involved, and what is the knowledge source? — RAG needs grounding tests, not just output checks.

Determinism requirements:

• Which outputs must be deterministic (classification, extraction) vs. creative (chat)? — decides exact-match vs. property vs. statistical assertions.
• What temperature runs in production? — test there, not at 0 "to make tests pass."
• Can the output be constrained to a JSON schema? — schema-constrained extraction is testable with a plain validator, no LLM judge.

Quality requirements:

• How is quality defined? (Accuracy, relevance, completeness, safety, tone) — these become eval metrics with weights and thresholds.
• Is there a golden dataset of inputs and acceptable outputs? — the anchor for every regression check.
• Who evaluates quality today? (Humans, metrics, nobody) — if an LLM judges, it must be calibrated against humans.

Safety requirements:

• Does the AI process user-generated input? — prompt-injection surface.
• Are there content-policy or PII constraints, or a regulated domain? — drives the red-team probe set.

---

Core Principles

1. Nondeterminism is inherent, not a bug. LLMs are stochastic; the same prompt yields different outputs across runs. Assert on properties and boundaries, never exact strings. expect(output).toBe("The answer is 42") breaks the next run when the model says "42 is the answer."

1. Test properties, not exact outputs. A good assertion asks: does the response contain the required information, stay in the length range, exclude prohibited content, match the expected format? When you can constrain output to a JSON schema, do that and validate with a plain schema validator — it converts a statistical check into a deterministic one.

1. Evals are the test suite for AI. An eval defines inputs, runs them through the system, and scores outputs against quality criteria. Invest in evals the way you invest in test infrastructure: version them, run them in CI, gate merges on them.

1. Safety testing is non-negotiable. AI can produce harmful content, leak the system prompt, echo PII, or be steered by adversarial input. Safety tests are the security tests of AI features — run them on every prompt change, and red-team before launch.

1. A judge you have not calibrated proves nothing. LLM-as-judge scales evaluation, but a judge that disagrees with humans just launders a wrong answer. Measure judge-vs-human agreement on a labeled held-out set and set a minimum bar before trusting it (see Response Quality Evaluation).

---

Tooling

Pick the layer that fits the job. Don't reach for hand-rolled TS unless none of these match.

| Tool | Best for | Notes | |------|----------|-------| | Promptfoo | Prompt regression, A/B + cross-provider tests, redteam scans | OSS Apache-2.0 CLI; YAML-defined suites; MCP target support. Acquired by OpenAI (Mar 2026) but stays open-source under its current license + public repo; de-facto default for production LLM teams. https://github.com/promptfoo/promptfoo | | DeepEval | Pytest-style LLM + agent evals; tool-call metrics | Current 4.x ships an agent-native workflow (run eval → see metric failures inline → patch → retry) that fits Claude Code / Cursor loops. ToolCorrectnessMetric, ArgumentCorrectnessMetric, TaskCompletionMetric cover the tool-call patterns this skill teaches. https://github.com/confident-ai/deepeval | | Ragas | RAG-specific eval (faithfulness, context precision/recall, answer relevance) | Use faithfulness for the grounding/hallucination check below. https://github.com/explodinggradients/ragas | | TruLens | Production tracing + RAG triad + non-LLM feedback | Has deterministic, non-LLM feedback functions (e.g. schema/regex checks) that are cheaper than an LLM judge for structured outputs. https://github.com/truera/trulens | | Inspect AI | Government-backed agent eval harness; large pre-built eval catalog | UK AI Security Institute. Date-based releases (release/2025-11-28). https://inspect.aisi.org.uk | | Garak | Adversarial prompt scanner / red-team probes | NVIDIA, Apache-2.0. v0.15.0 current (May 2026): added multi-turn GOAT, Agent Breaker (tool-aware), system-prompt-extraction, ModernBERT refusal detector. Probe modules: encoding, dan, promptinject, latentinjection, leakreplay. Run garak --list_probes for the live set. https://github.com/NVIDIA/garak | | PyRIT | Microsoft AI Red Team's orchestration framework | Orchestrated multi-turn attacks; complements Garak. https://github.com/Azure/PyRIT | | Braintrust | Commercial evals + prompt playground | Hosted, paid; SDK works alongside any of the above. |

Public benchmarks (HELM, LMSYS Chatbot Arena, Inspect AI's catalog) are for model selection, not app regression — they don't know your domain. Use them when picking a base model; use the tools above for everything after.

For runnable entry points — DeepEval tool-call validation, the Promptfoo cross-provider YAML suite, and the Garak red-team command — see references/tooling-evals.md.

---

Prompt Regression Testing

Version prompts like code

Prompts drive your application's behavior; version, review, and test them like code. A versioned prompt object carries its version, template, typed parameters, and a changelog. See references/prompt-regression.md for the SUMMARIZE_PROMPT object.

Baseline response quality

Establish a quality baseline per prompt and detect regressions when the prompt, model, or parameters change. Each eval case pairs an input with criteria (maxLength, mustContain, mustNotContain, sentenceCount, formatCheck); the test asserts every applicable criterion. See references/prompt-regression.md for the baseline eval suite.

A/B test prompts

When changing a prompt, run both versions against the eval suite over N runs and compare aggregate scores (mean, stddev, min) to pick a winner — or declare a tie when the gap is below threshold. See references/prompt-regression.md for the A/B harness.

Cross-provider regression

The same versioned prompt can degrade silently when you switch models or run a fallback provider. Run one prompt across multiple providers in a single Promptfoo suite and assert the same criteria hold on each — this catches a provider that drops a required fact or ignores a length constraint. See references/tooling-evals.md for the cross-provider YAML (one prompt, providers: [anthropic:..., openai:...], shared assertions).

---

Response Quality Evaluation

Eval framework with weighted metrics

For open-ended output, score each response on weighted, thresholded metrics and require a passing weighted sum AND every metric over its own floor. Typical wiring:

• Relevance (weight 0.3, threshold 0.7): LLM-as-judge rates relevance 0–10, normalized to 0–1.
• Completeness (weight 0.3, threshold 0.6): compare to a reference answer.
• Safety (weight 0.4, threshold 1.0): pattern-match for prohibited content — must be perfect.

A test passes only if every metric clears its threshold and the weighted sum clears the overall bar. See references/eval-framework.md for the runnable scorer (per-metric scoring functions + weightedSum).

Calibrate the judge before trusting it

Any LLM-as-judge metric needs a calibration gate. On a held-out set of cases you have also labeled by hand, score judge-vs-human agreement (Cohen's kappa, or simple accuracy for binary pass/fail). Set a minimum bar — e.g. kappa ≥ 0.6 — and refuse to ship the judge below it. Re-run calibration whenever you change the judge model or the rubric. An uncalibrated judge can rubber-stamp wrong answers. See references/eval-framework.md for the calibration check.

Prefer schema validation over a judge when you can

If the output is structured (extraction, classification, function arguments), constrain it to a JSON schema using the provider's structured-output mode (Anthropic structured outputs, OpenAI Structured Outputs response_format: json_schema) and validate with Zod or Pydantic. This makes extraction near-deterministic and lets you drop the statistical assertion entirely — a schema validator is cheaper, faster, and more reliable than an LLM judge for anything with a fixed shape.

Golden datasets

A golden dataset is a curated set of inputs with known-good reference outputs — the most reliable anchor for regression testing. Each case includes: input, reference output, acceptance criteria (mustContainFacts, mustNotContain, formatRequirements, maxLength), and metadata (category, difficulty).

Golden dataset maintenance:
  - Add 5-10 new cases per sprint, sampled from real production traffic
  - De-PII production-sourced cases before they enter the dataset
  - Review and update existing cases quarterly
  - Include edge cases: very long inputs, multilingual, ambiguous queries
  - Minimum size: 50 cases per prompt/feature for statistical reliability

For pulling production prompts into the dataset on a schedule, see observability-driven-testing.

---

Tool-Call Validation

When AI systems use tools (function calling, API calls, DB queries), test the selection and invocation logic. DeepEval's metrics are the cleanest entry point — but mind which metric uses a reference:

• ToolCorrectnessMetric is reference-based — it compares tools_called to the expected_tools you supply. This is where the golden tool list belongs.
• ArgumentCorrectnessMetric is referenceless and LLM-based — it judges whether the arguments make sense given the input; it does NOT consume expected_tools. Don't expect it to compare against your reference arguments.
• TaskCompletionMetric scores whether the agent actually accomplished the task end to end.

See references/tooling-evals.md for the DeepEval run with the metric wiring annotated.

Verify correct tool selection

Assert the agent picks the right tool (and arguments) for a query, falls through to search for factual queries, and calls no tools for conversational turns. See references/test-patterns.md for the tool-selection suite.

Argument validation

Test that arguments are correctly typed and formatted. A "last week" query should produce valid ISO date strings spanning ~7 days. Also test sanitization: a query containing "; DROP TABLE users; -- must not reach a tool argument unsanitized.

Error handling and retry logic

Test three failure scenarios with mocked tools:

• Transient failure: tool fails twice then succeeds — assert the AI retries and returns a valid response.
• Persistent failure: tool always fails — assert a graceful fallback message, not undefined/null.
• Timeout: tool takes 30s — assert the AI times out within budget (e.g. 15s) and tells the user.

---

Nondeterminism Strategies

Statistical testing over N runs

For nondeterministic outputs, run the test over multiple iterations and assert on aggregate results — require an 8/10 or 9/10 pass rate, not a single pass. The statisticalAssert helper takes the call under test, an assertion function applied to each output, and a requiredPassRate; it runs runs iterations and asserts the observed pass rate clears the bar. See references/test-patterns.md for the helper.

Property-based assertions

Assert on properties that hold regardless of the exact output: a classification always returns a valid category and a confidence in [0,1], response language matches request language, structured extraction matches the expected JSON schema. See references/test-patterns.md for the property-based suite.

Temperature-aware testing

Different temperatures serve different purposes. Test at the temperature your application uses in production, not at 0 "just to make the test pass."

temperature=0:   Lowest variance. Use for classification, extraction, structured output.
                 NOTE: not fully deterministic — sampling/infra nondeterminism remains,
                 and some reasoning/structured-output APIs ignore or constrain temperature.
                 Even here, prefer property/schema assertions over exact match.
temperature~0.3: Slight variation. Professional content, summaries. Property assertions.
temperature~0.7: Moderate creativity. Chat, writing assistance. Statistical assertions over N runs.
temperature~1.0: High creativity. Brainstorming, creative writing. Only safety + format checks.

The scale above is illustrative; the exact knobs and whether temperature is even honored depend on the provider and model — confirm against the model's API docs.

---

Hallucination & Grounding

Fact-checking assertions

When the AI states facts, verify them against a known source:

• Feature claims: extract claimed features, verify each exists in the product database.
• URL/reference fabrication: extract URLs, HEAD-request each to confirm it resolves.
• Numerical claims: cross-reference statistics, dates, and quantities against source data.

RAG grounding verification

For RAG, every factual claim in the response must trace back to a retrieved document; a claim with no supporting context is a hallucination. Two practical paths:

1. Ragas faithfulness — the standard metric. It decomposes the answer into claims and scores the fraction grounded in the retrieved context. Gate on a faithfulness threshold (e.g. ≥ 0.9) in CI.
2. Hand-rolled claim extraction — when you can't add Ragas: prompt a model to list the answer's atomic claims, then for each claim assert it is entailed by the concatenated retrieved chunks. Fail the test on any unsupported claim.

See references/eval-framework.md for both the Ragas faithfulness snippet and the claim-extraction grounding check.

Fallback behavior testing

Test that the AI admits uncertainty for out-of-scope questions (predictions, philosophic

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: petrkindlmann
• Source: petrkindlmann/qa-skills
• License: MIT
• Homepage: https://qa-skills.com

Install and usage instructions live in the source repository linked above.