# Nasde Benchmark Runner

> |

- **Type:** Skill
- **Install:** `agentstack add skill-noesisvision-nasde-toolkit-nasde-benchmark-runner`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [NoesisVision](https://agentstack.voostack.com/s/noesisvision)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [NoesisVision](https://github.com/NoesisVision)
- **Source:** https://github.com/NoesisVision/nasde-toolkit/tree/main/.claude/skills/nasde-benchmark-runner
- **Website:** https://noesis.vision/nasde/

## Install

```sh
agentstack add skill-noesisvision-nasde-toolkit-nasde-benchmark-runner
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# NASDE Benchmark Runner

Run coding agent benchmarks with `nasde` and verify results. The two-stage pipeline: Harbor runs agents in Docker containers (functional test → reward 0/1), then an LLM-as-a-Judge scores architecture quality across multiple dimensions.

## Authentication setup

Before running any benchmark, set up authentication tokens for the agents you plan to run. Both OS and auth method matter — pick the right command per row.

### Step 1 — Ask the user which auth they prefer

**Always ask the user before running, never assume.** Two questions:

1. **Which agents will you run?** (Claude / Codex / Gemini, any combination)
2. **For each agent, OAuth (subscription) or API key (per-token billing)?** Default recommendation: OAuth where available — no per-token cost, no env vars to manage.

Then detect their OS and pick the matching script row from the table below. On Windows, also ask whether they're in **PowerShell** or **WSL** (cmd.exe is not directly supported — see "Windows: cmd.exe" below).

### Where the auth scripts live

The OAuth scripts ship inside this skill. After `nasde install-skills` they are at:

- **User scope** (default): `~/.claude/skills/nasde-benchmark-runner/scripts/` (macOS/Linux/WSL) or `%USERPROFILE%\.claude\skills\nasde-benchmark-runner\scripts\` (Windows PowerShell)
- **Project scope**: `/.claude/skills/nasde-benchmark-runner/scripts/` (if installed with `nasde install-skills --scope project`)
- **Editable nasde checkout** (devs only): `/scripts/` — same files, mirrored from the skill bundle

Below, `` is shorthand for whichever absolute path applies. Resolve it once, then substitute it in every command. Verify the path with `ls ` before telling the user to source anything — if the directory is missing, they need to run `nasde install-skills` first.

### Step 2 — Run the right script per agent × OS

Priority order: **Claude → Codex → Gemini.** Claude is required even for non-Claude variants when `[evaluation] backend = "claude"` (default), because the assessment evaluator spawns `claude` CLI as a subprocess.

#### Claude Code

| OS / shell | OAuth (subscription) | API key |
|---|---|---|
| macOS | `source /export_oauth_token.sh` (reads Keychain entry "Claude Code-credentials") | `export ANTHROPIC_API_KEY=sk-ant-...` |
| Linux | `source /export_oauth_token.sh` (reads `~/.claude/.credentials.json`) | `export ANTHROPIC_API_KEY=sk-ant-...` |
| Windows PowerShell | `. \export_oauth_token.ps1` (reads `%USERPROFILE%\.claude\.credentials.json`) | `$env:ANTHROPIC_API_KEY = 'sk-ant-...'` |
| Windows WSL (Ubuntu) | `source /export_oauth_token.sh` (Linux path; resolve `` from your WSL home, not the Windows host's) | `export ANTHROPIC_API_KEY=sk-ant-...` |

Prerequisite for OAuth: `claude` CLI installed and `claude` ran once to log in.

The script exports `CLAUDE_CODE_OAUTH_TOKEN`. This is required for both Claude variant runs AND assessment evaluation (when `[evaluation] backend = "claude"` — the default).

#### Codex

| OS / shell | OAuth (ChatGPT subscription) | API key |
|---|---|---|
| macOS | `codex login` once, then `source /export_codex_oauth_token.sh` | `export CODEX_API_KEY=sk-proj-...` (or `OPENAI_API_KEY`) |
| Linux | `codex login` once, then `source /export_codex_oauth_token.sh` | `export CODEX_API_KEY=sk-proj-...` |
| Windows PowerShell | `codex login` once, then `. \export_codex_oauth_token.ps1` | `$env:CODEX_API_KEY = 'sk-proj-...'` |
| Windows WSL (Ubuntu) | `codex login` once, then `source /export_codex_oauth_token.sh` | `export CODEX_API_KEY=sk-proj-...` |

The OAuth scripts only **validate** `~/.codex/auth.json` (or `%USERPROFILE%\.codex\auth.json`) — `nasde` then opts Harbor into uploading it into the sandbox (it sets `CODEX_FORCE_AUTH_JSON=true` when no API key is present but the file exists; harbor 0.13's Codex agent otherwise defaults to `OPENAI_API_KEY` and would write an empty key). API key always takes priority over OAuth when both are present.

#### Gemini CLI

| OS / shell | OAuth (Google account) | API key |
|---|---|---|
| macOS | `gemini login` once, then `source /export_gemini_oauth_token.sh` | `export GEMINI_API_KEY=...` |
| Linux | `gemini login` once, then `source /export_gemini_oauth_token.sh` | `export GEMINI_API_KEY=...` |
| Windows PowerShell | `gemini login` once, then `. \export_gemini_oauth_token.ps1` | `$env:GEMINI_API_KEY = '...'` |
| Windows WSL (Ubuntu) | `gemini login` once, then `source /export_gemini_oauth_token.sh` | `export GEMINI_API_KEY=...` |

The OAuth scripts export `GEMINI_OAUTH_CREDS` (the raw JSON) — `ConfigurableGemini` reads that env var and injects credentials into the sandbox. API key always takes priority over OAuth.

### Combined setup for cross-agent runs

Resolve `` first, then run all three.

**macOS / Linux / Windows WSL:**
```bash
SKILL_SCRIPTS=~/.claude/skills/nasde-benchmark-runner/scripts   # adjust if --scope project
source $SKILL_SCRIPTS/export_oauth_token.sh         # Claude (subscription)
source $SKILL_SCRIPTS/export_codex_oauth_token.sh   # Codex (subscription) — or: export CODEX_API_KEY=...
source $SKILL_SCRIPTS/export_gemini_oauth_token.sh  # Gemini (Google account) — or: export GEMINI_API_KEY=...
```

**Windows PowerShell:**
```powershell
$SkillScripts = "$env:USERPROFILE\.claude\skills\nasde-benchmark-runner\scripts"
. "$SkillScripts\export_oauth_token.ps1"
. "$SkillScripts\export_codex_oauth_token.ps1"
. "$SkillScripts\export_gemini_oauth_token.ps1"
```

### Windows: cmd.exe

cmd.exe is **not supported directly** — `.ps1` requires PowerShell, `.sh` requires bash. Two workarounds:

1. **Open PowerShell** (`powershell.exe`) and dot-source the `.ps1` script. This is the simplest path on a vanilla Windows install.
2. **Use WSL** (`wsl -d Ubuntu`) and source the `.sh` script. This is the recommended path if you also want Docker Desktop with the WSL2 backend, which is the most common dev setup.

If a user is in cmd.exe, point them to one of these two — don't try to extract the token manually.

### Important: chain auth into the run command

Exported env vars (`CLAUDE_CODE_OAUTH_TOKEN`, etc.) **do not persist across separate
shell invocations** — every command in an automated runner (and every separate terminal
call) is a fresh shell, so a lone `source ...` in one step is gone by the next. When
running non-interactively, **chain the `source` script(s) and the `nasde` call in a single
command** with `&&`:

```bash
# bash/zsh — both tokens chained into one invocation
source $SKILL_SCRIPTS/export_codex_oauth_token.sh && source $SKILL_SCRIPTS/export_oauth_token.sh && \
  nasde run --variant codex-vanilla --tasks my-task -C path/to/benchmark
```

Chain exactly the scripts the run needs: the agent's token (claude/codex/gemini) **and**
the evaluator's token — the evaluator uses `[evaluation] backend` from `nasde.toml`, which
defaults to `claude`, so a codex/gemini run with the default evaluator still needs
`export_oauth_token.sh` too. In an interactive terminal session a one-time `source` in the
same session is fine; the chaining rule matters for scripts, CI, and tool-driven runs.

## Running benchmarks

All commands assume `-C` points to the benchmark project directory.

### Basic run (all tasks, default variant)

```bash
nasde run -C path/to/benchmark
```

Assessment evaluation runs by default. This is the standard workflow.

### Specific variant and tasks

```bash
# Single task, specific variant
nasde run --variant guided --tasks my-task -C path/to/benchmark

# Multiple tasks
nasde run --variant baseline --tasks task-a,task-b -C path/to/benchmark
```

### With Opik tracing

```bash
nasde run --variant baseline --tasks my-task -C path/to/benchmark --with-opik
```

After this completes, ALWAYS verify Opik results (see Opik verification below).

### Harbor only (skip assessment)

```bash
nasde run --variant baseline -C path/to/benchmark --without-eval
```

### Parallel runs (multiple variants)

**Do not use `--all-variants` when you want parallelism.** `--all-variants` runs variants **sequentially** in a single process (one variant after another). To run two or more variants in parallel, launch separate `nasde run` processes with `&` and `wait` — each job directory gets a unique random suffix, so concurrent runs are collision-safe:

```bash
nasde run --variant vanilla --tasks my-task -C path/to/benchmark &
nasde run --variant guided --tasks my-task -C path/to/benchmark &
wait
```

Use `--all-variants` only when you want one variant after another (e.g. to limit total resource use, or when running Claude variants where parallel runs risk Docker OOM — see warning below).

For deterministic job names, use `--job-suffix`:

```bash
nasde run --variant vanilla --job-suffix run1 -C path/to/benchmark
```

### Running Codex variants

Codex variants use `AGENTS.md` (instead of `CLAUDE.md`) and require either `codex login` (ChatGPT subscription) or `CODEX_API_KEY`/`OPENAI_API_KEY` (API billing).

**CRITICAL: Codex model must be set explicitly.** The `nasde.toml` default model (e.g. `claude-sonnet-4-6`) is designed for Claude and will be passed to Codex if not overridden. Codex CLI will silently accept invalid model names but produce garbage results (0% pass rate). Always set the model via `--model` flag or in `variant.toml`.

```bash
# Option A: ChatGPT subscription (no env vars needed after codex login)
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark

# Option B: API key
export $(grep CODEX_API_KEY .env)
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark
```

**Codex models** (recommended first, as of 2026-03):
- `gpt-5.4` — flagship frontier model, best overall for professional work
- `gpt-5.4-mini` — fast, efficient mini model for responsive coding and subagents
- `gpt-5.3-codex` — industry-leading coding model for complex software engineering
- `gpt-5.3-codex-spark` — near-instant real-time coding iteration (ChatGPT Pro only)

Older alternatives: `gpt-5.2-codex`, `gpt-5.1-codex`, `gpt-5-codex`, `gpt-5-codex-mini`

Codex supports any model compatible with the Responses API. Chat Completions API support is deprecated.

**Setting model in variant.toml** (preferred over --model flag):
```toml
agent = "codex"
model = "gpt-5.4"
```
This avoids accidentally inheriting the Claude model from `nasde.toml`.

### Running Gemini CLI variants

Gemini variants use `GEMINI.md` (instead of `CLAUDE.md`) and require either `gemini login` (Google account) or `GEMINI_API_KEY`/`GOOGLE_API_KEY` (API billing).

**CRITICAL: Gemini model must use `google/` prefix.** Harbor requires model names in `provider/model_name` format. Always set via `--model` flag or in `variant.toml`.

```bash
# Option A: Google account OAuth (no env vars needed after gemini login)
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark

# Option B: API key
export GEMINI_API_KEY=your-key
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark
```

**Gemini models** (recommended first, as of 2026-03):
- `google/gemini-3.1-pro-preview` — advanced thinking model, deep reasoning
- `google/gemini-3-flash-preview` — best quality/speed ratio, daily coding
- `google/gemini-3.1-flash-lite-preview` — fastest, simple tasks

**Setting model in variant.toml** (preferred over --model flag):
```toml
agent = "gemini"
model = "google/gemini-3-flash-preview"
```

### Cross-agent comparison (Claude vs Codex vs Gemini)

Set up all auth tokens first (see Authentication setup above), then run:

```bash
source scripts/export_oauth_token.sh && export $(grep CODEX_API_KEY .env) && source scripts/export_gemini_oauth_token.sh

# Run Claude, Codex, and Gemini variants — non-Claude MUST have --model override
nasde run --variant claude-vanilla -C path/to/benchmark --with-opik &
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark --with-opik &
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark --with-opik &
wait
```

**WARNING**: Running Claude variants in parallel can cause OOM (exit code 137) in Docker. Claude Code containers are memory-heavy (~2-4 GB each). Run Claude variants sequentially, or increase Docker Desktop memory to 16+ GB.

### Custom model and timeout

```bash
nasde run --variant baseline --model claude-opus-4-7 --timeout 1200 -C path/to/benchmark
```

**Timeout priority**: `--timeout` flag overrides everything. Without it, Harbor uses `task.toml [agent] timeout_sec` per task. Timeouts are per-task only — there is no project-wide default in nasde.toml.

### Re-evaluate existing results

```bash
nasde eval path/to/benchmark/jobs/2026-03-16__14-05-58 --with-opik -C path/to/benchmark
```

## Token cost heuristic

**Claude Code variants:**
When using `CLAUDE_CODE_OAUTH_TOKEN` (Claude subscription — no per-token cost):
- **Run freely**: total estimated time under 30 minutes (sum of tasks × variants)
- **Ask first**: over 30 minutes, OR when using `ANTHROPIC_API_KEY` (API billing)

**Codex variants:**
- **ChatGPT OAuth** (`codex login`): uses subscription credits, no per-token cost. Same heuristic as Claude subscription — run freely under 30 minutes.
- **API key** (`CODEX_API_KEY`): billed per-token. Always ask before running. Codex uses significantly more input tokens than Claude Code (~1M vs ~250K per task).

**Gemini CLI variants:**
- **Google account OAuth** (`gemini login`): free tier via Gemini Code Assist license (1M token context). Run freely.
- **API key** (`GEMINI_API_KEY`): billed per-token through Google AI Studio. Ask before running.
- **Vertex AI** (`GOOGLE_API_KEY`): billed through Google Cloud. Ask before running.

Task estimated times are derived from `task.toml [agent] timeout_sec` (timeout_sec / 60 is a rough upper bound; agents typically finish faster). When `--tasks` filters are used, count only selected tasks.

## Viewing results

After a run, results are in `jobs//`:

```bash
# Job summary
cat jobs//result.json | python3 -m json.tool

# Per-trial results
cat jobs///result.json | python3 -m json.tool

# Verifier output (what test.sh printed)
cat jobs///verifier/test-stdout.txt

# Assessment scores
cat jobs///assessment_eval.json | python3 -m json.tool
```

### Using Harbor CLI for viewing

```bash
nasde harbor view path/to/benchmark/jobs/
```

## Comparing models — quality vs cost / tokens (PRIMARY method)

When the user asks "which model/agent is best?" or "which is most efficient?", the answer is a **two-axis scatter (quality vs cost, quality vs tokens)**, not a single ranked number. Show the data honestly and let the reader judge — the convention here follows charts like Artificial Analysis's intelligence-vs-tokens plots: **raw points with full names, a shaded "most attractive" region (high quality, low cost/tokens), and no verdict painted on individual points.** Two reasons this is the primary method:

1. **Quality and cost are two axes, and you keep both.** Collapsing them into one number throws away information. *Pareto dominance* is the concept you reason **with** (point A is dominated if some B is no-worse on both axes and strictly better on one), but it is **not** a tag stamped on the chart — at small n a hard "dominated / never pick" label on a point with no variance over-claims. State dominance in prose when it is clear from the data; let the chart stay raw.
2. **Position is invariant to where you put the score zero.** That is exactly why nasde deliberately does **not** compute a scalar "token efficiency" or "cost efficiency" (`score / denominator`). See "Why no scalar efficiency" below.

### The panels — one shared cost panel + one token panel per provider

- **Quality × cost ($)** — *price-dependent*, but **one panel for all providers**: USD is a common unit, so cross-provider comparison on cost is fair. Uses `cost_usd` (from `pricing.toml`).
- **Quality × tokens** — *price-independent*, but **one panel per provider**. Token counts are in each model'

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [NoesisVision](https://github.com/NoesisVision)
- **Source:** [NoesisVision/nasde-toolkit](https://github.com/NoesisVision/nasde-toolkit)
- **License:** MIT
- **Homepage:** https://noesis.vision/nasde/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** yes
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-noesisvision-nasde-toolkit-nasde-benchmark-runner
- Seller: https://agentstack.voostack.com/s/noesisvision
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.