# Run Aiewf 2024 Evals

> >

- **Type:** Skill
- **Install:** `agentstack add skill-hiteshbandhu-skills-i-use-run-aiewf-2024-evals`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [hiteshbandhu](https://agentstack.voostack.com/s/hiteshbandhu)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [hiteshbandhu](https://github.com/hiteshbandhu)
- **Source:** https://github.com/hiteshbandhu/skills-i-use/tree/main/skills/ai-engineer-talks/run-aiewf-2024-evals

## Install

```sh
agentstack add skill-hiteshbandhu-skills-i-use-run-aiewf-2024-evals
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Run AIEWF 2024 evals & LLM ops

Action playbook from six AI Engineer / World's Fair talks. **Do not summarize talks** — pick a workflow and execute it.

**Supporting files** (read when needed):
- [workflows.md](workflows.md) — workflows A–F (steps, deliverables, stop conditions)
- [source-index.md](source-index.md) — src-NNN → talk learnings in ingest-into-skills

Optional: `{SKILL_OUTPUT_DIR}/run-aiewf-2024-evals/`

---

## Step 0 — Pick workflow

Use the decision tree below. Open the matching section in [workflows.md](workflows.md).

```
What is the user trying to do?
├─ Build domain eval from zero (assertions → judges)     → A
├─ Layer task evals on routers/tools (trace-native)        → B
├─ Calibrate LLM judges / courtroom rubrics              → C
├─ PM+eng eval ops (Zapier/Braintrust regression)          → D
├─ Enterprise CX: full conversation intelligence           → E
└─ Decide fine-tune vs prompt/distill (maturity curve)     → F
```

**Stop summarizing** once a workflow is identified — run its checklist.

---

## Install

```bash
cp -r skills/run-aiewf-2024-evals ~/.claude/skills/
cp -r skills/run-aiewf-2024-evals ~/.cursor/skills/
cp -r skills/run-aiewf-2024-evals ~/.codex/skills/
```

From [skills-i-use](https://github.com/hiteshbandhu/skills-i-use) or ingest-into-skills (`playlists/evals-llm-ops-aie-world-s-fair-2024/`).

---

## Cross-cutting rules

| Rule | Source |
|------|--------|
| Evals ≠ demos; log before judges | [src-001 @ 6:39] |
| Assertions before LLM-as-judge | [src-001 @ 4:41] |
| Layer scores on router/tools/answer | [src-004 @ 4:46] |
| Fine-tune only after eval + teacher data | [src-006 @ 5:55] |

Disputed steps: see [source-index.md](source-index.md). Name workflow A–F; save artifacts to `./skill-outputs/run-aiewf-2024-evals/` when requested; do not auto-commit.

---

## Invocation examples

```
@run-aiewf-2024-evals build domain eval harness for our agent
should we fine-tune or stay on GPT-4o?
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [hiteshbandhu](https://github.com/hiteshbandhu)
- **Source:** [hiteshbandhu/skills-i-use](https://github.com/hiteshbandhu/skills-i-use)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-hiteshbandhu-skills-i-use-run-aiewf-2024-evals
- Seller: https://agentstack.voostack.com/s/hiteshbandhu
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.