Prompt Evaluation Runner

Prompt Evaluation Runner

When to use

Use when you need to evaluate an LLM app, test a prompt systematically, or run red-team/vulnerability scans against a target model or application.

Requirements / Checks

1. Check if an evaluation tool is defined in project deps, scripts, lockfiles, or local toolchain (e.g., promptfoo, evals, braintrust).
2. Do not run unvetted remote runners without checking the project's toolchain first (e.g., avoid npx promptfoo@latest if promptfoo is already installed locally).
3. If no runner exists, ask before adding a dev dependency or using an ephemeral runner.
4. Confirm expected cost, provider, API keys, and network target before any execution.

Workflow

1. Define risk — state target behavior, failure mode, provider(s), and budget limits before writing any config.

1. Choose assertions — prefer deterministic checks first:

| Assertion type | When to use | |---|---| | contains / not-contains | Output must include/exclude specific text | | regex | Structured output pattern (e.g., JSON key present) | | json-schema | Output must conform to a schema | | cost | Must stay under a token/dollar budget | | latency | Must respond within N ms | | javascript / python | Custom logic when simpler types don't fit | | Model grader | Last resort — only for subjective quality checks |

1. Use model graders sparingly — pin the grader model and provider explicitly; document the cost and non-determinism risk.

1. Minimal config structure:

```yaml description: "Test that the summarizer stays under 200 words" providers:

• id: openai:gpt-4o-mini

config: temperature: 0 prompts:

• "Summarize: {{input}}"

defaultTest: assert:

• type: javascript

value: output.split(' ').length < 200 tests:

• vars:

input: "{{env.TEST_DOCUMENT}}" ```

1. Handle env safely — use {{env.VAR_NAME}} for all secrets and inputs. Never hardcode API keys or sensitive data in config files.

1. Execute locally — run the smallest suite first. Ask before running long, paid, red-team, or production-targeted suites.

1. Analyze failures — classify before fixing:

• Prompt failure (model output is wrong)
• Provider variance (non-deterministic model)
• Flaky grader (model grader is inconsistent)
• Bad fixture (test input is unrealistic)
• Config mistake (assertion logic error)

Safety Constraints

• Do NOT log, echo, or store API keys in configuration files or chat output.
• Do NOT run evaluations against production endpoints without user consent.
• Do not execute arbitrary remote code or unvetted plugins during evaluation.

Validation / Done Criteria

• Eval config is valid, minimal, and uses {{env.VAR}} references for secrets.
• Deterministic assertions exist where possible; model grader use is documented and justified.
• Run scope, provider, and estimated cost are reported before execution.
• Results are summarized without leaking sensitive input data.

References

• references/eval-config-patterns.md

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yeaight7
• Source: yeaight7/agent-powerups
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.