Nasde Benchmark Runner

NASDE Benchmark Runner

Run coding agent benchmarks with nasde and verify results. The two-stage pipeline: Harbor runs agents in Docker containers (functional test → reward 0/1), then an LLM-as-a-Judge scores architecture quality across multiple dimensions.

Authentication setup

Before running any benchmark, set up authentication tokens for the agents you plan to run. Both OS and auth method matter — pick the right command per row.

Step 1 — Ask the user which auth they prefer

Always ask the user before running, never assume. Two questions:

1. Which agents will you run? (Claude / Codex / Gemini, any combination)
2. For each agent, OAuth (subscription) or API key (per-token billing)? Default recommendation: OAuth where available — no per-token cost, no env vars to manage.

Then detect their OS and pick the matching script row from the table below. On Windows, also ask whether they're in PowerShell or WSL (cmd.exe is not directly supported — see "Windows: cmd.exe" below).

Where the auth scripts live

The OAuth scripts ship inside this skill. After nasde install-skills they are at:

• User scope (default): ~/.claude/skills/nasde-benchmark-runner/scripts/ (macOS/Linux/WSL) or %USERPROFILE%\.claude\skills\nasde-benchmark-runner\scripts\ (Windows PowerShell)
• Project scope: /.claude/skills/nasde-benchmark-runner/scripts/ (if installed with nasde install-skills --scope project)
• Editable nasde checkout (devs only): /scripts/ — same files, mirrored from the skill bundle

Below, ` is shorthand for whichever absolute path applies. Resolve it once, then substitute it in every command. Verify the path with ls before telling the user to source anything — if the directory is missing, they need to run nasde install-skills` first.

Step 2 — Run the right script per agent × OS

Priority order: Claude → Codex → Gemini. Claude is required even for non-Claude variants when [evaluation] backend = "claude" (default), because the assessment evaluator spawns claude CLI as a subprocess.

Claude Code

| OS / shell | OAuth (subscription) | API key | |---|---|---| | macOS | source /export_oauth_token.sh (reads Keychain entry "Claude Code-credentials") | export ANTHROPIC_API_KEY=sk-ant-... | | Linux | source /export_oauth_token.sh (reads ~/.claude/.credentials.json) | export ANTHROPIC_API_KEY=sk-ant-... | | Windows PowerShell | . \export_oauth_token.ps1 (reads %USERPROFILE%\.claude\.credentials.json) | $env:ANTHROPIC_API_KEY = 'sk-ant-...' | | Windows WSL (Ubuntu) | source /export_oauth_token.sh (Linux path; resolve ` from your WSL home, not the Windows host's) | export ANTHROPICAPIKEY=sk-ant-...` |

Prerequisite for OAuth: claude CLI installed and claude ran once to log in.

The script exports CLAUDE_CODE_OAUTH_TOKEN. This is required for both Claude variant runs AND assessment evaluation (when [evaluation] backend = "claude" — the default).

Codex

| OS / shell | OAuth (ChatGPT subscription) | API key | |---|---|---| | macOS | codex login once, then source /export_codex_oauth_token.sh | export CODEX_API_KEY=sk-proj-... (or OPENAI_API_KEY) | | Linux | codex login once, then source /export_codex_oauth_token.sh | export CODEX_API_KEY=sk-proj-... | | Windows PowerShell | codex login once, then . \export_codex_oauth_token.ps1 | $env:CODEX_API_KEY = 'sk-proj-...' | | Windows WSL (Ubuntu) | codex login once, then source /export_codex_oauth_token.sh | export CODEX_API_KEY=sk-proj-... |

The OAuth scripts only validate ~/.codex/auth.json (or %USERPROFILE%\.codex\auth.json) — nasde then opts Harbor into uploading it into the sandbox (it sets CODEX_FORCE_AUTH_JSON=true when no API key is present but the file exists; harbor 0.13's Codex agent otherwise defaults to OPENAI_API_KEY and would write an empty key). API key always takes priority over OAuth when both are present.

Gemini CLI

| OS / shell | OAuth (Google account) | API key | |---|---|---| | macOS | gemini login once, then source /export_gemini_oauth_token.sh | export GEMINI_API_KEY=... | | Linux | gemini login once, then source /export_gemini_oauth_token.sh | export GEMINI_API_KEY=... | | Windows PowerShell | gemini login once, then . \export_gemini_oauth_token.ps1 | $env:GEMINI_API_KEY = '...' | | Windows WSL (Ubuntu) | gemini login once, then source /export_gemini_oauth_token.sh | export GEMINI_API_KEY=... |

The OAuth scripts export GEMINI_OAUTH_CREDS (the raw JSON) — ConfigurableGemini reads that env var and injects credentials into the sandbox. API key always takes priority over OAuth.

Combined setup for cross-agent runs

Resolve `` first, then run all three.

macOS / Linux / Windows WSL:

SKILL_SCRIPTS=~/.claude/skills/nasde-benchmark-runner/scripts   # adjust if --scope project
source $SKILL_SCRIPTS/export_oauth_token.sh         # Claude (subscription)
source $SKILL_SCRIPTS/export_codex_oauth_token.sh   # Codex (subscription) — or: export CODEX_API_KEY=...
source $SKILL_SCRIPTS/export_gemini_oauth_token.sh  # Gemini (Google account) — or: export GEMINI_API_KEY=...

Windows PowerShell:

$SkillScripts = "$env:USERPROFILE\.claude\skills\nasde-benchmark-runner\scripts"
. "$SkillScripts\export_oauth_token.ps1"
. "$SkillScripts\export_codex_oauth_token.ps1"
. "$SkillScripts\export_gemini_oauth_token.ps1"

Windows: cmd.exe

cmd.exe is not supported directly — .ps1 requires PowerShell, .sh requires bash. Two workarounds:

1. Open PowerShell (powershell.exe) and dot-source the .ps1 script. This is the simplest path on a vanilla Windows install.
2. Use WSL (wsl -d Ubuntu) and source the .sh script. This is the recommended path if you also want Docker Desktop with the WSL2 backend, which is the most common dev setup.

If a user is in cmd.exe, point them to one of these two — don't try to extract the token manually.

Important: chain auth into the run command

Exported env vars (CLAUDE_CODE_OAUTH_TOKEN, etc.) do not persist across separate shell invocations — every command in an automated runner (and every separate terminal call) is a fresh shell, so a lone source ... in one step is gone by the next. When running non-interactively, chain the source script(s) and the nasde call in a single command with &&:

# bash/zsh — both tokens chained into one invocation
source $SKILL_SCRIPTS/export_codex_oauth_token.sh && source $SKILL_SCRIPTS/export_oauth_token.sh && \
  nasde run --variant codex-vanilla --tasks my-task -C path/to/benchmark

Chain exactly the scripts the run needs: the agent's token (claude/codex/gemini) and the evaluator's token — the evaluator uses [evaluation] backend from nasde.toml, which defaults to claude, so a codex/gemini run with the default evaluator still needs export_oauth_token.sh too. In an interactive terminal session a one-time source in the same session is fine; the chaining rule matters for scripts, CI, and tool-driven runs.

Running benchmarks

All commands assume -C points to the benchmark project directory.

Basic run (all tasks, default variant)

nasde run -C path/to/benchmark

Assessment evaluation runs by default. This is the standard workflow.

Specific variant and tasks

# Single task, specific variant
nasde run --variant guided --tasks my-task -C path/to/benchmark

# Multiple tasks
nasde run --variant baseline --tasks task-a,task-b -C path/to/benchmark

With Opik tracing

nasde run --variant baseline --tasks my-task -C path/to/benchmark --with-opik

After this completes, ALWAYS verify Opik results (see Opik verification below).

Harbor only (skip assessment)

nasde run --variant baseline -C path/to/benchmark --without-eval

Parallel runs (multiple variants)

Do not use --all-variants when you want parallelism. --all-variants runs variants sequentially in a single process (one variant after another). To run two or more variants in parallel, launch separate nasde run processes with & and wait — each job directory gets a unique random suffix, so concurrent runs are collision-safe:

nasde run --variant vanilla --tasks my-task -C path/to/benchmark &
nasde run --variant guided --tasks my-task -C path/to/benchmark &
wait

Use --all-variants only when you want one variant after another (e.g. to limit total resource use, or when running Claude variants where parallel runs risk Docker OOM — see warning below).

For deterministic job names, use --job-suffix:

nasde run --variant vanilla --job-suffix run1 -C path/to/benchmark

Running Codex variants

Codex variants use AGENTS.md (instead of CLAUDE.md) and require either codex login (ChatGPT subscription) or CODEX_API_KEY/OPENAI_API_KEY (API billing).

CRITICAL: Codex model must be set explicitly. The nasde.toml default model (e.g. claude-sonnet-4-6) is designed for Claude and will be passed to Codex if not overridden. Codex CLI will silently accept invalid model names but produce garbage results (0% pass rate). Always set the model via --model flag or in variant.toml.

# Option A: ChatGPT subscription (no env vars needed after codex login)
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark

# Option B: API key
export $(grep CODEX_API_KEY .env)
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark

Codex models (recommended first, as of 2026-03):

• gpt-5.4 — flagship frontier model, best overall for professional work
• gpt-5.4-mini — fast, efficient mini model for responsive coding and subagents
• gpt-5.3-codex — industry-leading coding model for complex software engineering
• gpt-5.3-codex-spark — near-instant real-time coding iteration (ChatGPT Pro only)

Older alternatives: gpt-5.2-codex, gpt-5.1-codex, gpt-5-codex, gpt-5-codex-mini

Codex supports any model compatible with the Responses API. Chat Completions API support is deprecated.

Setting model in variant.toml (preferred over --model flag):

agent = "codex"
model = "gpt-5.4"

This avoids accidentally inheriting the Claude model from nasde.toml.

Running Gemini CLI variants

Gemini variants use GEMINI.md (instead of CLAUDE.md) and require either gemini login (Google account) or GEMINI_API_KEY/GOOGLE_API_KEY (API billing).

CRITICAL: Gemini model must use google/ prefix. Harbor requires model names in provider/model_name format. Always set via --model flag or in variant.toml.

# Option A: Google account OAuth (no env vars needed after gemini login)
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark

# Option B: API key
export GEMINI_API_KEY=your-key
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark

Gemini models (recommended first, as of 2026-03):

• google/gemini-3.1-pro-preview — advanced thinking model, deep reasoning
• google/gemini-3-flash-preview — best quality/speed ratio, daily coding
• google/gemini-3.1-flash-lite-preview — fastest, simple tasks

Setting model in variant.toml (preferred over --model flag):

agent = "gemini"
model = "google/gemini-3-flash-preview"

Cross-agent comparison (Claude vs Codex vs Gemini)

Set up all auth tokens first (see Authentication setup above), then run:

source scripts/export_oauth_token.sh && export $(grep CODEX_API_KEY .env) && source scripts/export_gemini_oauth_token.sh

# Run Claude, Codex, and Gemini variants — non-Claude MUST have --model override
nasde run --variant claude-vanilla -C path/to/benchmark --with-opik &
nasde run --variant codex-vanilla --model gpt-5.3-codex -C path/to/benchmark --with-opik &
nasde run --variant gemini-vanilla --model google/gemini-3-flash-preview -C path/to/benchmark --with-opik &
wait

WARNING: Running Claude variants in parallel can cause OOM (exit code 137) in Docker. Claude Code containers are memory-heavy (~2-4 GB each). Run Claude variants sequentially, or increase Docker Desktop memory to 16+ GB.

Custom model and timeout

nasde run --variant baseline --model claude-opus-4-7 --timeout 1200 -C path/to/benchmark

Timeout priority: --timeout flag overrides everything. Without it, Harbor uses task.toml [agent] timeout_sec per task. Timeouts are per-task only — there is no project-wide default in nasde.toml.

Re-evaluate existing results

nasde eval path/to/benchmark/jobs/2026-03-16__14-05-58 --with-opik -C path/to/benchmark

Token cost heuristic

Claude Code variants: When using CLAUDE_CODE_OAUTH_TOKEN (Claude subscription — no per-token cost):

• Run freely: total estimated time under 30 minutes (sum of tasks × variants)
• Ask first: over 30 minutes, OR when using ANTHROPIC_API_KEY (API billing)

Codex variants:

• ChatGPT OAuth (codex login): uses subscription credits, no per-token cost. Same heuristic as Claude subscription — run freely under 30 minutes.
• API key (CODEX_API_KEY): billed per-token. Always ask before running. Codex uses significantly more input tokens than Claude Code (~1M vs ~250K per task).

Gemini CLI variants:

• Google account OAuth (gemini login): free tier via Gemini Code Assist license (1M token context). Run freely.
• API key (GEMINI_API_KEY): billed per-token through Google AI Studio. Ask before running.
• Vertex AI (GOOGLE_API_KEY): billed through Google Cloud. Ask before running.

Task estimated times are derived from task.toml [agent] timeout_sec (timeout_sec / 60 is a rough upper bound; agents typically finish faster). When --tasks filters are used, count only selected tasks.

Viewing results

After a run, results are in jobs//:

# Job summary
cat jobs//result.json | python3 -m json.tool

# Per-trial results
cat jobs///result.json | python3 -m json.tool

# Verifier output (what test.sh printed)
cat jobs///verifier/test-stdout.txt

# Assessment scores
cat jobs///assessment_eval.json | python3 -m json.tool

Using Harbor CLI for viewing

nasde harbor view path/to/benchmark/jobs/

Comparing models — quality vs cost / tokens (PRIMARY method)

When the user asks "which model/agent is best?" or "which is most efficient?", the answer is a two-axis scatter (quality vs cost, quality vs tokens), not a single ranked number. Show the data honestly and let the reader judge — the convention here follows charts like Artificial Analysis's intelligence-vs-tokens plots: raw points with full names, a shaded "most attractive" region (high quality, low cost/tokens), and no verdict painted on individual points. Two reasons this is the primary method:

1. Quality and cost are two axes, and you keep both. Collapsing them into one number throws away information. Pareto dominance is the concept you reason with (point A is dominated if some B is no-worse on both axes and strictly better on one), but it is not a tag stamped on the chart — at small n a hard "dominated / never pick" label on a point with no variance over-claims. State dominance in prose when it is clear from the data; let the chart stay raw.
2. Position is invariant to where you put the score zero. That is exactly why nasde deliberately does not compute a scalar "token efficiency" or "cost efficiency" (score / denominator). See "Why no scalar efficiency" below.

The panels — one shared cost panel + one token panel per provider

• Quality × cost ($) — price-dependent, but one panel for all providers: USD is a common unit, so cross-provider comparison on cost is fair. Uses cost_usd (from pricing.toml).
• Quality × tokens — price-independent, but one panel per provider. Token counts are in each model'

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: NoesisVision
• Source: NoesisVision/nasde-toolkit
• License: MIT
• Homepage: https://noesis.vision/nasde/

Install and usage instructions live in the source repository linked above.