# Workflow Skill Evals

> Write the evolve evaluation suite for an agent skill: its triggers (triggers.json — Tier 1 activation tests) and behavioral evals (evals.json — Tier 2 task tests), under evals/<skill>/. Use when asked to generate, write, author, scaffold, or balance an eval suite, evals, or triggers for a skill; to create or edit a triggers.json or evals.json; to add behavioral evals to a SKILL.md; to add or reba…

- **Type:** Skill
- **Install:** `agentstack add skill-bitwise-media-group-skills-workflow-skill-evals`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [bitwise-media-group](https://agentstack.voostack.com/s/bitwise-media-group)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [bitwise-media-group](https://github.com/bitwise-media-group)
- **Source:** https://github.com/bitwise-media-group/skills/tree/main/plugins/workflow/skills/workflow-skill-evals

## Install

```sh
agentstack add skill-bitwise-media-group-skills-workflow-skill-evals
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Generate an evolve evaluation suite for a skill

Turn a skill's `SKILL.md` into the two graded artifacts evolve runs against it:

- **`triggers.json`** (Tier 1) — does the skill activate on the prompts that should reach for it, and stay
  quiet on the ones that shouldn't?
- **`evals.json`** (Tier 2) — when it does fire, does it do the job? Real tasks in throwaway workspaces,
  graded by deterministic checks and, where only prose can judge, an LLM judge.

Both live **beside** the skill, never inside it: `evals//`, a sibling of `skills//`.

## 0. Read the upstream guide first — this skill is meant to evolve

evolve gains assertion types, eval fields, and CLI features over time. The authoritative, current source
is upstream, not this file. **Before authoring, fetch and skim:**

- Guide:  (sub-pages: `/triggers/`, `/evals/`,
  `/assertions/`, `/execution/`, `/results/`).
- Triggers schema: 
- Evals schema: 

If the guide or a schema lists a field, assertion `type`, or tier this file doesn't mention, **the upstream
source wins** — use it. To discover new CLI surface in a repo that vendors evolve, run `evolve --help`,
`evolve run --help`, and `evolve docs` (here: `go tool evolve …`). See [reference.md](reference.md) for the
current assertion table, the staging rule, and worked examples, but treat upstream as the tiebreaker.

## 1. Locate the skill and its eval directory

Read the target `SKILL.md` end to end — the frontmatter `name` and `description` drive the triggers; the
body drives the behavioral evals. Then create `evals//` where `` equals the skill's `name`
(and its directory). In this marketplace evals are `plugins//evals//`; elsewhere mirror
whatever layout the repo's evolve config (`.evolve.json`) declares.

## 2. Author `triggers.json` (Tier 1) — start here, it's the cheapest signal

Envelope: a `triggers` array of `{ "query", "should_trigger" }`; `skill_name` echoes the directory.
Write **10–20** entries, roughly balanced:

- **Positives** — the real phrasings a user types when they want this skill. Vary the shape: an imperative,
  a question, a review-style ask. Each should name the task and domain concretely.
- **Negatives are where the signal lives.** A positives-only suite scores 100% and tells you nothing. Weight
  negatives toward **near-misses**: (a) **sibling skills' headline positives** — the adjacent skill in the
  same plugin is where false activations actually happen; (b) **same task, wrong domain** — a real positive
  with the language or framework swapped, which catches a skill keying off the verb instead of the context.

```json
{
  "$schema": "https://raw.githubusercontent.com/bitwise-media-group/evolve/main/schemas/triggers.schema.json",
  "skill_name": "go-style",
  "triggers": [
    { "query": "Refactor this Go code to wrap errors properly", "should_trigger": true },
    { "query": "Convert these log.Printf calls to slog", "should_trigger": true },
    { "query": "Write table-driven tests for this Go function", "should_trigger": false },
    { "query": "Refactor this Rust code to use idiomatic error handling", "should_trigger": false }
  ]
}
```

Start from [templates/triggers.json](templates/triggers.json). If you changed the skill's `description`,
the trigger surface moved — re-run Tier 1.

## 3. Author `evals.json` (Tier 2) — prove it does the job

Envelope: an `evals` array. Each case needs an `id` (lowercase-kebab; it is the results key), a `prompt`
(the real task), and **at least one** `assertions` entry **or** one `expectations` entry. Write **2–5**
cases covering the skill's headline behaviors and its important refusals/guards.

**Prefer deterministic assertions over the LLM judge** — they are cheap, fast, and reproducible. Reach for
the judge only for holistic claims a rule can't express. The current types:

- `file_exists` / `file_absent` — the agent created (or correctly did not create) `path`.
- `regex` / `not_regex` — `pattern` (Go RE2, multiline) matches a workspace file (`path`) or, with no
  `path`, the agent's final response. A missing `path` file **fails** either check.
- `command` — run `run` via `/bin/sh -c`; passes when the exit code equals `expect_exit` (default `0`).
  Set `requires` to a binary so the check **skips** (not fails) where it's absent; `cwd` runs in a subdir.
  This is the strongest check — it runs the real toolchain over the output.
- `tool_call` — the agent actually invoked a tool matching `tool` (regex), optionally with args matching
  `pattern`. Inspects behavior, not just artifacts.
- `llm` (or a bare string in `assertions`) — a pinned judge verifies `text`. `expectations` is a top-level
  array of such statements, graded first.

Scope `allowed_tools` per case (e.g. `Read Write Edit Glob Grep Skill Bash(go *)`) so a pass reflects the
skill, not the model improvising. Stage inputs with `files` (see §4). Start from
[templates/evals.json](templates/evals.json); the full table and examples are in [reference.md](reference.md).

```json
{
  "id": "project-scaffold",
  "prompt": "Scaffold a new Go service called orderd with our canonical cmd / tools-module / Makefile layout.",
  "allowed_tools": "Read Write Edit Glob Grep Skill Bash(go *) Bash(gofmt *) Bash(mkdir *)",
  "assertions": [
    { "type": "file_exists", "path": "cmd/orderd/main.go" },
    { "type": "regex", "path": "Makefile", "pattern": "^pr:" },
    { "type": "command", "run": "go vet ./...", "requires": "go" }
  ]
}
```

## 4. Stage fixtures with `files`

`files` lists input paths **relative to the eval directory**, staged into the workspace before the run.
One rule decides where each lands:

- A path under **`files/`** stages at its path _relative to `files/`_, preserving the tree
  (`files/internal/cli/root.go` → `internal/cli/root.go`). Use this whenever the **path matters** —
  subdirectories, multi-file layouts, or basenames that would otherwise collide (two `go.mod`s).
- **Any other path** stages by **basename** at the workspace root
  (`fixtures/clidemo/go.mod` → `go.mod`). Use `fixtures//` for a shared scaffold many cases reference.
  A fixture dir may hold more files than a case names; only listed paths are staged.

Author fixtures as the smallest compiling/valid context the assertions need — a real `go.mod` that
`require`s the right deps lets a `command` assertion lean on the toolchain.

## 5. Validate and run

Add the `$schema` key to both files (above) for editor validation. Then:

```sh
evolve run triggers --runs 5        # Tier 1; odd runs avoid 50/50 ties
evolve run evals --jobs 4           # Tier 2; baselines run automatically
```

In this repo those are `make triggers` and `make evals` (`make all` for both tiers plus reports), and
`make fmt && make lint` gates the eval JSON, schema, and markdown before committing. The committed
`results.json` is written by the sweeps — never hand-edit it. A brand-new suite simply has none until its
first run.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [bitwise-media-group](https://github.com/bitwise-media-group)
- **Source:** [bitwise-media-group/skills](https://github.com/bitwise-media-group/skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-bitwise-media-group-skills-workflow-skill-evals
- Seller: https://agentstack.voostack.com/s/bitwise-media-group
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.