# Prompt Evaluation Runner

> Use when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.

- **Type:** Skill
- **Install:** `agentstack add skill-yeaight7-agent-powerups-prompt-evaluation-runner`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [yeaight7](https://agentstack.voostack.com/s/yeaight7)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [yeaight7](https://github.com/yeaight7)
- **Source:** https://github.com/yeaight7/agent-powerups/tree/main/plugins/agent-evaluation-lab/skills/prompt-evaluation-runner

## Install

```sh
agentstack add skill-yeaight7-agent-powerups-prompt-evaluation-runner
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Prompt Evaluation Runner

## When to use

Use when you need to evaluate an LLM app, test a prompt systematically, or run red-team/vulnerability scans against a target model or application.

## Requirements / Checks

1. Check if an evaluation tool is defined in project deps, scripts, lockfiles, or local toolchain (e.g., `promptfoo`, `evals`, `braintrust`).
2. Do not run unvetted remote runners without checking the project's toolchain first (e.g., avoid `npx promptfoo@latest` if `promptfoo` is already installed locally).
3. If no runner exists, ask before adding a dev dependency or using an ephemeral runner.
4. Confirm expected cost, provider, API keys, and network target before any execution.

## Workflow

1. **Define risk** — state target behavior, failure mode, provider(s), and budget limits before writing any config.

2. **Choose assertions** — prefer deterministic checks first:

   | Assertion type | When to use |
   |---|---|
   | `contains` / `not-contains` | Output must include/exclude specific text |
   | `regex` | Structured output pattern (e.g., JSON key present) |
   | `json-schema` | Output must conform to a schema |
   | `cost` | Must stay under a token/dollar budget |
   | `latency` | Must respond within N ms |
   | `javascript` / `python` | Custom logic when simpler types don't fit |
   | Model grader | Last resort — only for subjective quality checks |

3. **Use model graders sparingly** — pin the grader model and provider explicitly; document the cost and non-determinism risk.

4. **Minimal config structure**:

   ```yaml
   description: "Test that the summarizer stays under 200 words"
   providers:
     - id: openai:gpt-4o-mini
       config:
         temperature: 0
   prompts:
     - "Summarize: {{input}}"
   defaultTest:
     assert:
       - type: javascript
         value: output.split(' ').length < 200
   tests:
     - vars:
         input: "{{env.TEST_DOCUMENT}}"
   ```

5. **Handle env safely** — use `{{env.VAR_NAME}}` for all secrets and inputs. Never hardcode API keys or sensitive data in config files.

6. **Execute locally** — run the smallest suite first. Ask before running long, paid, red-team, or production-targeted suites.

7. **Analyze failures** — classify before fixing:
   - Prompt failure (model output is wrong)
   - Provider variance (non-deterministic model)
   - Flaky grader (model grader is inconsistent)
   - Bad fixture (test input is unrealistic)
   - Config mistake (assertion logic error)

## Safety Constraints

- Do NOT log, echo, or store API keys in configuration files or chat output.
- Do NOT run evaluations against production endpoints without user consent.
- Do not execute arbitrary remote code or unvetted plugins during evaluation.

## Validation / Done Criteria

- Eval config is valid, minimal, and uses `{{env.VAR}}` references for secrets.
- Deterministic assertions exist where possible; model grader use is documented and justified.
- Run scope, provider, and estimated cost are reported before execution.
- Results are summarized without leaking sensitive input data.

## References
- `references/eval-config-patterns.md`

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [yeaight7](https://github.com/yeaight7)
- **Source:** [yeaight7/agent-powerups](https://github.com/yeaight7/agent-powerups)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-yeaight7-agent-powerups-prompt-evaluation-runner
- Seller: https://agentstack.voostack.com/s/yeaight7
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.