# Model Compare

> >

- **Type:** Skill
- **Install:** `agentstack add skill-moonlight-lupin-agent-skills-model-compare`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [moonlight-lupin](https://agentstack.voostack.com/s/moonlight-lupin)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [moonlight-lupin](https://github.com/moonlight-lupin)
- **Source:** https://github.com/moonlight-lupin/agent-skills/tree/main/mlops/model-compare
- **Website:** https://hermes-agent.nousresearch.com

## Install

```sh
agentstack add skill-moonlight-lupin-agent-skills-model-compare
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Model Compare — Blind Side-by-Side Multi-Model Testing

Send one prompt to multiple models simultaneously, present responses
anonymously, let the user pick a winner, then reveal which model is which.

Inspired by the Compare feature in PewDiePie's Odysseus project, adapted for
Hermes's multi-provider architecture (OpenRouter, NVIDIA, Ollama Cloud, any
OpenAI-compatible endpoint).

## When to use

- "Compare these models on..."
- "Which model is better for [task]?"
- "A/B test [model1] vs [model2]"
- "Blind comparison of..."
- "Test how different models handle this prompt"
- User wants to evaluate models before committing to one for a workflow
- Prompt engineering — seeing how different models interpret instructions

## When NOT to use

- **Benchmarking** (MMLU, GSM8K, etc.) → use a dedicated eval harness (e.g. lm-evaluation-harness), not blind A/B
- **Cost analysis** → just check provider pricing pages
- **Single model test** → just switch model and ask directly
- **Multi-source research synthesis** → use `deep-research` skill (iterative research loop, not model comparison)

## Architecture

Four comparison modes, all driven by `scripts/compare.py`:

| Mode | Flag | What it does | API feature |
|---|---|---|---|
| **simple** | `--mode simple` (default) | One prompt → one response | Basic chat completion |
| **tools** | `--mode tools` | Multi-turn tool calling with real web_search/web_extract. 5-turn max. Tracks full trace. | `tools` array in request, multi-turn messages |
| **coding** | `--mode coding` | Test bank coding prompts (LRU cache, concurrent fetch, debug merge sort, retry decorator) | Basic chat completion |
| **review** | `--mode review` | Code review prompts with planted bugs (SQL injection, clean code, race condition, float-for-money) | Basic chat completion |

```
User prompt + model list
  → Step 1: Resolve models to provider endpoints (free providers first)
  → Step 2: Send prompt to all models in parallel
     ├─ simple/coding/review: one-shot chat completion
     └─ tools: multi-turn loop (Think→Search→Extract→Synthesize→Stop, max 5 turns)
  → Step 3: Quality check responses (handle errors/empty)
  → Step 4: Present anonymously (shuffle + label A/B/C/D)
  → Step 5: Efficiency table (tokens in/out, turns, tool calls — auto for tools mode)
  → Step 6: User votes OR judge model evaluates
  → Step 7: Reveal identities + show mapping
  → Step 8 (optional): Save to JSON file
```

### Script: `scripts/compare.py`

The primary interface — a standalone CLI tool (no pip dependencies, pure stdlib + urllib). See `references/provider-tool-support.md` for which models support tool calling. (Running the test suite under `tests/` needs `pytest` — see `requirements-dev.txt`; the skill itself needs nothing installed.)

> **Tool-mode environment dependency:** `--mode tools` runs real `web_search`. That needs **either** `SEARXNG_URL` set to a SearXNG instance **or** the [`ddgs`](https://pypi.org/project/ddgs/) CLI available on `PATH` (the fallback). Neither is a Python import dependency, but one of them must be present for live search; without both, `web_search` returns an error result. The other three modes (`simple`, `coding`, `review`) need neither.

```bash
# Simple 2-model blind comparison
python3 scripts/compare.py --prompt "Explain X" --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5"

# Tool calling with test bank prompt A + judge + reveal
python3 scripts/compare.py --mode tools --test A --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5" --judge "ollama-cloud:glm-5.2" --reveal

# Coding test J with efficiency table
python3 scripts/compare.py --mode coding --test J --models "ollama-cloud:glm-5.2" "ollama-cloud:qwen3-coder:480b" --efficiency

# Code review test O with judge
python3 scripts/compare.py --mode review --test O --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5" --judge "ollama-cloud:glm-5.2"

# List available tests
python3 scripts/compare.py --list-tests

# List providers
python3 scripts/compare.py --list-providers
```

Key flags: `--prompt`, `--models`, `--mode`, `--test` (test bank ID), `--judge`, `--efficiency`, `--reveal`, `--output`, `--timeout`, `--list-providers`, `--list-models`.

## Available Providers

Three providers are wired in (the canonical config is `PROVIDERS` in
`scripts/compare.py`):

| Provider | Env Var | Cost | Model families | Endpoint |
|---|---|---|---|---|
| **Ollama Cloud** | `OLLAMA_API_KEY` | **Free** | GLM, Qwen, Kimi, Gemini, Gemma | `https://ollama.com/v1/chat/completions` |
| **NVIDIA** | `NVIDIA_API_KEY` | **Free** | Yi, Llama, Nemotron, … | `https://integrate.api.nvidia.com/v1/chat/completions` |
| **OpenRouter** | `OPENROUTER_API_KEY` | **Paid** (per-token) | Claude, GPT, Gemini, DeepSeek, Llama, Qwen, Mistral | `https://openrouter.ai/api/v1/chat/completions` |

> **Model inventories live elsewhere, on purpose.** Exact model counts and IDs
> drift constantly, so they are deliberately kept out of this doc. For the live
> list run `python3 scripts/compare.py --list-models ` (needs that
> provider's key); `references/providers.json` holds a curated, count-free
> snapshot with representative model IDs per provider.

### Cost-aware provider routing (mandatory)

When the user does **not** specify a provider, route to **free providers first**:

1. **Ollama Cloud** — free (GLM, Qwen, Kimi, Gemini, Gemma)
2. **NVIDIA** — free (Yi, Llama, Nemotron)
3. **OpenRouter** — paid (per-token cost). **Only use when:**
   - The user explicitly requests an OpenRouter-only model (e.g. Claude, GPT-4o)
   - The user explicitly says to use OpenRouter
   - The free providers don't have a suitable model for the task

**Before any OpenRouter call**, confirm with the user:
> "This comparison will use OpenRouter which has per-token costs. Estimated cost: ~$0.01–0.05 per model per call (varies by model). Proceed?"

Only proceed after explicit confirmation. When in doubt, default to free providers.

To call a model, POST to the provider's `/v1/chat/completions` endpoint with:
```json
{
  "model": "",
  "messages": [{"role": "user", "content": ""}],
  "max_tokens": 4096,
  "temperature": 0.7
}
```
Header: `Authorization: Bearer `

## Step 1 — Resolve Models

Determine which models to compare. The user may specify:
- **Explicit model names**: "compare claude-sonnet-4.6 vs gpt-4o vs gemini-2.5-flash"
- **Provider + model**: "compare OpenRouter claude-sonnet-4.6 vs Ollama glm-5.2"
- **Task-based**: "which model is best for coding?" → suggest 2-4 candidates
- **All from a provider**: "test 3 OpenRouter models" → pick diverse ones

### Model resolution

Map the user's model names to a `provider:model_id` spec. **Free providers
(Ollama Cloud, NVIDIA) are preferred.** OpenRouter is only used for models not
on a free provider (e.g. Claude, GPT) or when the user explicitly asks for it.

The rules, not a hardcoded catalogue (which would rot):

- **Free, leave the prefix off or name an Ollama/NVIDIA model** → resolve on a
  free provider, e.g. `ollama-cloud:glm-5.2`, `ollama-cloud:qwen3-coder:480b`,
  `nvidia:meta/llama-3.3-70b-instruct`.
- **A proprietary model (Claude, GPT, Gemini-Pro, DeepSeek-R1, …)** → only
  OpenRouter carries it, e.g. `openrouter:anthropic/claude-sonnet-4.6`,
  `openrouter:openai/gpt-4o` — **paid, confirm first** (see routing rule above).
- **Unsure of the exact ID?** Run `python3 scripts/compare.py --list-models
  ` for the live list, or see `references/providers.json` for a
  curated set of representative IDs per provider. Don't hand-maintain a model
  table here.

**When user doesn't specify models:**
1. Check if the task type maps to free models (e.g. coding → qwen3-coder, glm-5.2; general → glm-5.2, kimi-k2.5)
2. Suggest 2-4 free models from Ollama Cloud and NVIDIA
3. Only suggest OpenRouter models if the user asks for premium models (Claude, GPT-4o) or the free providers lack suitable options
4. If suggesting any paid models, flag the cost before running

If unsure which provider has a model, check with:
```bash
curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data'] if 'KEYWORD' in m['id'].lower()]"
```

**Rules:**
- 2-4 models per comparison (more = unwieldy in chat)
- If user doesn't specify, suggest a diverse set (different providers/sizes)
- Always confirm the model list with the user before running

## Step 2 — Send Prompt to All Models

Send the prompt to all models **in parallel**. The script handles this
automatically via `concurrent.futures.ThreadPoolExecutor` — no need for
delegate_task or manual parallelism.

### Simple / coding / review modes

One-shot chat completion per model. All calls fired concurrently by the script.

### Tool calling mode

Multi-turn loop per model (also concurrent across models):

```
Turn 1: Send prompt + tool definitions → Model returns tool_call(s)
Turn 2: Execute real tool → inject result → Model returns tool_call(s) or answer
Turn 3: ... until final answer or 5-turn max
```

**Real tool execution** — the script executes `web_search` and `web_extract`
for real:
- `web_search` → SearXNG (self-hosted, via `SEARXNG_URL`) with DDGS fallback
- `web_extract` → direct HTTP fetch with HTML-to-text conversion (3000 chars/page)

No mock tools, no fake results. Models must formulate good queries, pick the
right URLs, and synthesize from real content. A model that generates a bad
search query gets bad results and must recover.

**Tool definitions** passed via the OpenAI `tools` parameter:
- `web_search(query, limit)` — returns titles, URLs, snippets
- `web_extract(urls)` — returns page content as text

**No terminal tool** — we don't have a sandbox. Terminal is not exposed to
comparison models for safety reasons.

## Step 3 — Quality Check

For each response:
- **Empty/error**: if a model returns an error or empty response, note it and
  exclude from the comparison. Tell the user which model failed.
- **Truncated**: if response hit max_tokens, note it was truncated
- **Refusal**: if a model refused to answer, include it as-is (refusals are
  valid comparison data)

## Step 4 — Present Anonymously

**Shuffle the responses** and assign neutral labels. Do NOT reveal which model
is which.

**Shuffle rule:** Use a random permutation. Do not always put the same model
first. If the user is comparing 3 models, randomly assign A/B/C.

Present as:

```
🧪 Blind Model Comparison
Prompt: "[truncated to 100 chars...]"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model A:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model B:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model C:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Vote: Reply with the letter of the best response (A, B, or C), or "tie".
```

**Formatting rules:**
- Full responses, not summaries — the user needs to judge quality
- Clear visual separation between responses
- Truncate at ~3000 chars per response for chat readability (note if truncated)
- If responses are very long (>3000 chars), save full versions to files and
  present truncated versions in chat with a note

## Step 5 — User Votes

Wait for the user to vote. Accept:
- Letter: "A", "B", "C", "D"
- "tie" or "tie between A and B"
- "all bad" (valid — none won)
- Specific feedback: "A is better but B's code is cleaner"

## Step 6 — Reveal Identities

After the vote, reveal the mapping:

```
🎭 Reveal:

Model A →  ()
Model B →  ()
Model C →  ()

🏆 Your winner: Model  = 
```

## Step 7 — Synthesis (optional)

If the user wants the best possible answer, synthesize across all responses:

```
"Want me to synthesize the best parts of all responses into one?"
```

If yes, take the strongest elements from each response and produce a unified
answer. Note which model contributed each part:

```
## Synthesized Answer
[merged response]

## Contributions
- Section X: primarily from 
- Section Y: primarily from 
```

## Step 8 — Vote History (optional)

For recurring comparisons, log results to a file:

```
~/.hermes/data/model_compare_history.jsonl
```

Format:
```json
{"timestamp": "2026-06-27T12:00:00", "prompt": "...", "models": ["model_a", "model_b"], "winner": "model_a", "is_blind": true, "feedback": "..."}
```

This builds up a picture of which models win for which task types over time.

## Test Bank (12 tests)

Use `--test ` to run a pre-built test prompt. The mode is auto-set based on the test domain.

| ID | Domain | Prompt summary | Tools? |
|---|---|---|---|
| A | tool_calling | Latest Python version + top 2 features | 🔧 |
| B | tool_calling | Find Odysseus GitHub repo star count | 🔧 |
| C | tool_calling | Best reverse proxy for homelab, then find key feature | 🔧 |
| E | tool_calling | Search LRU cache implementations, then write a better one | 🔧 |
| J | coding | Implement LRU cache, O(1), type hints + docstring | |
| K | coding | Concurrent URL fetch with per-URL timeout, preserve order | |
| L | coding | Fix buggy merge sort (off-by-one in merge step) | |
| M | coding | Retry decorator, 3x, 1s delay, preserve metadata | |
| O | code_review | SQL injection + unreliable rowcount loop | |
| P | code_review | Clean code (no bugs) — test false positive rate | |
| Q | code_review | Thread-unsafe cache in production service | |
| R | code_review | Float for money + missing transfer validation | |

Each test includes evaluation criteria used by the judge. Code review tests include planted issues for objective scoring.

## Tool Calling Mode — How It Works

The `--mode tools` flag enables multi-turn tool calling with **real tools** (not mocks):

1. Script defines `web_search` and `web_extract` as OpenAI function tools
2. Sends prompt + tool defs to each model in parallel
3. When a model returns `tool_calls`, the script **executes the real call**:
   - `web_search` → SearXNG (self-hosted, via `SEARXNG_URL`) with DDGS fallback
   - `web_extract` → direct HTTP fetch, HTML stripped to text, truncated to 3000 chars/page
4. Feeds the real result back as a `tool` role message
5. Repeats until final answer or 5 turns (hard cap)
6. Tracks all turns, tool calls, tokens per turn, convergence status

**Tools exposed:** `web_search` and `web_extract` only — no `terminal` or other code-execution tool, since the harness has no sandbox to run model-requested commands in. See `references/provider-tool-support.md` for which models support tool calling.

**Judge sees the full tool call trace** — what was searched, what was extracted, how many turns, whether it converged. This lets the judge evaluate *tool selection strategy*, not just the final answer.

## Efficiency Analysis

The `--efficiency` flag (auto-enabled for tools mode) prints a token comparison table:

```
📊 Efficiency Comparison
                      Turns  Tools   Tok In  Tok Out  Ratio   Time
✅ ollama-cloud:glm-5.2      5      5     7806      777   0.10  15.3s
✅ ollama-cloud:kimi-k2.5    5      4     5434     1051   0.19  14.8s
✅ ollama-cloud:gemma4:31b   2      1      949      248   0.26   6.5s
```

Key metrics: turns, tool calls, tokens in (context consumed), tokens out (generated), efficiency ratio (out/in), time. **Efficiency without accuracy is waste** — a model that uses few tokens but gets the wrong answer is not efficient, it's just wrong fast.

## Use Cases

### Model selection for a workflow
> "I need a model for summarizing legal documents. Compare claude-sonnet-4.6, gpt-4o, and gemini-2.5-flash on this prompt: [legal text]"

### Cost vs quality tradeoff
> "Compare glm-5.2 (free) vs claude-sonnet-4.6 (paid) on this coding prompt. Is the paid one worth it?"

### Prompt robustness testing
> "Test this prompt on 3 models — I want to see which ones follow my formatting instructions"

### Provider comparison
> "Compare the same model (llama-4-maverick) on OpenRouter vs NVIDIA — is there a difference?"

## Provider Health Tracking

The `scripts/provider_health.py` module tracks provider endpoint health and
implements dead

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [moonlight-lupin](https://github.com/moonlight-lupin)
- **Source:** [moonlight-lupin/agent-skills](https://github.com/moonlight-lupin/agent-skills)
- **License:** MIT
- **Homepage:** https://hermes-agent.nousresearch.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-moonlight-lupin-agent-skills-model-compare
- Seller: https://agentstack.voostack.com/s/moonlight-lupin
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.