Model Compare

Model Compare — Blind Side-by-Side Multi-Model Testing

Send one prompt to multiple models simultaneously, present responses anonymously, let the user pick a winner, then reveal which model is which.

Inspired by the Compare feature in PewDiePie's Odysseus project, adapted for Hermes's multi-provider architecture (OpenRouter, NVIDIA, Ollama Cloud, any OpenAI-compatible endpoint).

When to use

• "Compare these models on..."
• "Which model is better for [task]?"
• "A/B test [model1] vs [model2]"
• "Blind comparison of..."
• "Test how different models handle this prompt"
• User wants to evaluate models before committing to one for a workflow
• Prompt engineering — seeing how different models interpret instructions

When NOT to use

• Benchmarking (MMLU, GSM8K, etc.) → use a dedicated eval harness (e.g. lm-evaluation-harness), not blind A/B
• Cost analysis → just check provider pricing pages
• Single model test → just switch model and ask directly
• Multi-source research synthesis → use deep-research skill (iterative research loop, not model comparison)

Architecture

Four comparison modes, all driven by scripts/compare.py:

| Mode | Flag | What it does | API feature | |---|---|---|---| | simple | --mode simple (default) | One prompt → one response | Basic chat completion | | tools | --mode tools | Multi-turn tool calling with real websearch/webextract. 5-turn max. Tracks full trace. | tools array in request, multi-turn messages | | coding | --mode coding | Test bank coding prompts (LRU cache, concurrent fetch, debug merge sort, retry decorator) | Basic chat completion | | review | --mode review | Code review prompts with planted bugs (SQL injection, clean code, race condition, float-for-money) | Basic chat completion |

User prompt + model list
  → Step 1: Resolve models to provider endpoints (free providers first)
  → Step 2: Send prompt to all models in parallel
     ├─ simple/coding/review: one-shot chat completion
     └─ tools: multi-turn loop (Think→Search→Extract→Synthesize→Stop, max 5 turns)
  → Step 3: Quality check responses (handle errors/empty)
  → Step 4: Present anonymously (shuffle + label A/B/C/D)
  → Step 5: Efficiency table (tokens in/out, turns, tool calls — auto for tools mode)
  → Step 6: User votes OR judge model evaluates
  → Step 7: Reveal identities + show mapping
  → Step 8 (optional): Save to JSON file

Script: scripts/compare.py

The primary interface — a standalone CLI tool (no pip dependencies, pure stdlib + urllib). See references/provider-tool-support.md for which models support tool calling. (Running the test suite under tests/ needs pytest — see requirements-dev.txt; the skill itself needs nothing installed.)

> Tool-mode environment dependency: --mode tools runs real web_search. That needs either SEARXNG_URL set to a SearXNG instance or the ddgs CLI available on PATH (the fallback). Neither is a Python import dependency, but one of them must be present for live search; without both, web_search returns an error result. The other three modes (simple, coding, review) need neither.

# Simple 2-model blind comparison
python3 scripts/compare.py --prompt "Explain X" --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5"

# Tool calling with test bank prompt A + judge + reveal
python3 scripts/compare.py --mode tools --test A --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5" --judge "ollama-cloud:glm-5.2" --reveal

# Coding test J with efficiency table
python3 scripts/compare.py --mode coding --test J --models "ollama-cloud:glm-5.2" "ollama-cloud:qwen3-coder:480b" --efficiency

# Code review test O with judge
python3 scripts/compare.py --mode review --test O --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5" --judge "ollama-cloud:glm-5.2"

# List available tests
python3 scripts/compare.py --list-tests

# List providers
python3 scripts/compare.py --list-providers

Key flags: --prompt, --models, --mode, --test (test bank ID), --judge, --efficiency, --reveal, --output, --timeout, --list-providers, --list-models.

Available Providers

Three providers are wired in (the canonical config is PROVIDERS in scripts/compare.py):

| Provider | Env Var | Cost | Model families | Endpoint | |---|---|---|---|---| | Ollama Cloud | OLLAMA_API_KEY | Free | GLM, Qwen, Kimi, Gemini, Gemma | https://ollama.com/v1/chat/completions | | NVIDIA | NVIDIA_API_KEY | Free | Yi, Llama, Nemotron, … | https://integrate.api.nvidia.com/v1/chat/completions | | OpenRouter | OPENROUTER_API_KEY | Paid (per-token) | Claude, GPT, Gemini, DeepSeek, Llama, Qwen, Mistral | https://openrouter.ai/api/v1/chat/completions |

> Model inventories live elsewhere, on purpose. Exact model counts and IDs > drift constantly, so they are deliberately kept out of this doc. For the live > list run python3 scripts/compare.py --list-models (needs that > provider's key); references/providers.json holds a curated, count-free > snapshot with representative model IDs per provider.

Cost-aware provider routing (mandatory)

When the user does not specify a provider, route to free providers first:

1. Ollama Cloud — free (GLM, Qwen, Kimi, Gemini, Gemma)
2. NVIDIA — free (Yi, Llama, Nemotron)
3. OpenRouter — paid (per-token cost). Only use when:

• The user explicitly requests an OpenRouter-only model (e.g. Claude, GPT-4o)
• The user explicitly says to use OpenRouter
• The free providers don't have a suitable model for the task

Before any OpenRouter call, confirm with the user: > "This comparison will use OpenRouter which has per-token costs. Estimated cost: ~$0.01–0.05 per model per call (varies by model). Proceed?"

Only proceed after explicit confirmation. When in doubt, default to free providers.

To call a model, POST to the provider's /v1/chat/completions endpoint with:

{
  "model": "",
  "messages": [{"role": "user", "content": ""}],
  "max_tokens": 4096,
  "temperature": 0.7
}

Header: Authorization: Bearer

Step 1 — Resolve Models

Determine which models to compare. The user may specify:

• Explicit model names: "compare claude-sonnet-4.6 vs gpt-4o vs gemini-2.5-flash"
• Provider + model: "compare OpenRouter claude-sonnet-4.6 vs Ollama glm-5.2"
• Task-based: "which model is best for coding?" → suggest 2-4 candidates
• All from a provider: "test 3 OpenRouter models" → pick diverse ones

Model resolution

Map the user's model names to a provider:model_id spec. Free providers (Ollama Cloud, NVIDIA) are preferred. OpenRouter is only used for models not on a free provider (e.g. Claude, GPT) or when the user explicitly asks for it.

The rules, not a hardcoded catalogue (which would rot):

• Free, leave the prefix off or name an Ollama/NVIDIA model → resolve on a

free provider, e.g. ollama-cloud:glm-5.2, ollama-cloud:qwen3-coder:480b, nvidia:meta/llama-3.3-70b-instruct.

• A proprietary model (Claude, GPT, Gemini-Pro, DeepSeek-R1, …) → only

OpenRouter carries it, e.g. openrouter:anthropic/claude-sonnet-4.6, openrouter:openai/gpt-4o — paid, confirm first (see routing rule above).

• Unsure of the exact ID? Run `python3 scripts/compare.py --list-models

for the live list, or see references/providers.json` for a curated set of representative IDs per provider. Don't hand-maintain a model table here.

When user doesn't specify models:

1. Check if the task type maps to free models (e.g. coding → qwen3-coder, glm-5.2; general → glm-5.2, kimi-k2.5)
2. Suggest 2-4 free models from Ollama Cloud and NVIDIA
3. Only suggest OpenRouter models if the user asks for premium models (Claude, GPT-4o) or the free providers lack suitable options
4. If suggesting any paid models, flag the cost before running

If unsure which provider has a model, check with:

curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data'] if 'KEYWORD' in m['id'].lower()]"

Rules:

• 2-4 models per comparison (more = unwieldy in chat)
• If user doesn't specify, suggest a diverse set (different providers/sizes)
• Always confirm the model list with the user before running

Step 2 — Send Prompt to All Models

Send the prompt to all models in parallel. The script handles this automatically via concurrent.futures.ThreadPoolExecutor — no need for delegate_task or manual parallelism.

Simple / coding / review modes

One-shot chat completion per model. All calls fired concurrently by the script.

Tool calling mode

Multi-turn loop per model (also concurrent across models):

Turn 1: Send prompt + tool definitions → Model returns tool_call(s)
Turn 2: Execute real tool → inject result → Model returns tool_call(s) or answer
Turn 3: ... until final answer or 5-turn max

Real tool execution — the script executes web_search and web_extract for real:

• web_search → SearXNG (self-hosted, via SEARXNG_URL) with DDGS fallback
• web_extract → direct HTTP fetch with HTML-to-text conversion (3000 chars/page)

No mock tools, no fake results. Models must formulate good queries, pick the right URLs, and synthesize from real content. A model that generates a bad search query gets bad results and must recover.

Tool definitions passed via the OpenAI tools parameter:

• web_search(query, limit) — returns titles, URLs, snippets
• web_extract(urls) — returns page content as text

No terminal tool — we don't have a sandbox. Terminal is not exposed to comparison models for safety reasons.

Step 3 — Quality Check

For each response:

• Empty/error: if a model returns an error or empty response, note it and

exclude from the comparison. Tell the user which model failed.

• Truncated: if response hit max_tokens, note it was truncated
• Refusal: if a model refused to answer, include it as-is (refusals are

valid comparison data)

Step 4 — Present Anonymously

Shuffle the responses and assign neutral labels. Do NOT reveal which model is which.

Shuffle rule: Use a random permutation. Do not always put the same model first. If the user is comparing 3 models, randomly assign A/B/C.

Present as:

🧪 Blind Model Comparison
Prompt: "[truncated to 100 chars...]"

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model A:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model B:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📦 Model C:
[full response]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Vote: Reply with the letter of the best response (A, B, or C), or "tie".

Formatting rules:

• Full responses, not summaries — the user needs to judge quality
• Clear visual separation between responses
• Truncate at ~3000 chars per response for chat readability (note if truncated)
• If responses are very long (>3000 chars), save full versions to files and

present truncated versions in chat with a note

Step 5 — User Votes

Wait for the user to vote. Accept:

• Letter: "A", "B", "C", "D"
• "tie" or "tie between A and B"
• "all bad" (valid — none won)
• Specific feedback: "A is better but B's code is cleaner"

Step 6 — Reveal Identities

After the vote, reveal the mapping:

🎭 Reveal:

Model A →  ()
Model B →  ()
Model C →  ()

🏆 Your winner: Model  = 

Step 7 — Synthesis (optional)

If the user wants the best possible answer, synthesize across all responses:

"Want me to synthesize the best parts of all responses into one?"

If yes, take the strongest elements from each response and produce a unified answer. Note which model contributed each part:

## Synthesized Answer
[merged response]

## Contributions
- Section X: primarily from 
- Section Y: primarily from 

Step 8 — Vote History (optional)

For recurring comparisons, log results to a file:

~/.hermes/data/model_compare_history.jsonl

Format:

{"timestamp": "2026-06-27T12:00:00", "prompt": "...", "models": ["model_a", "model_b"], "winner": "model_a", "is_blind": true, "feedback": "..."}

This builds up a picture of which models win for which task types over time.

Test Bank (12 tests)

Use --test to run a pre-built test prompt. The mode is auto-set based on the test domain.

| ID | Domain | Prompt summary | Tools? | |---|---|---|---| | A | toolcalling | Latest Python version + top 2 features | 🔧 | | B | toolcalling | Find Odysseus GitHub repo star count | 🔧 | | C | toolcalling | Best reverse proxy for homelab, then find key feature | 🔧 | | E | toolcalling | Search LRU cache implementations, then write a better one | 🔧 | | J | coding | Implement LRU cache, O(1), type hints + docstring | | | K | coding | Concurrent URL fetch with per-URL timeout, preserve order | | | L | coding | Fix buggy merge sort (off-by-one in merge step) | | | M | coding | Retry decorator, 3x, 1s delay, preserve metadata | | | O | codereview | SQL injection + unreliable rowcount loop | | | P | codereview | Clean code (no bugs) — test false positive rate | | | Q | codereview | Thread-unsafe cache in production service | | | R | codereview | Float for money + missing transfer validation | |

Each test includes evaluation criteria used by the judge. Code review tests include planted issues for objective scoring.

Tool Calling Mode — How It Works

The --mode tools flag enables multi-turn tool calling with real tools (not mocks):

1. Script defines web_search and web_extract as OpenAI function tools
2. Sends prompt + tool defs to each model in parallel
3. When a model returns tool_calls, the script executes the real call:

• web_search → SearXNG (self-hosted, via SEARXNG_URL) with DDGS fallback
• web_extract → direct HTTP fetch, HTML stripped to text, truncated to 3000 chars/page

1. Feeds the real result back as a tool role message
2. Repeats until final answer or 5 turns (hard cap)
3. Tracks all turns, tool calls, tokens per turn, convergence status

Tools exposed: web_search and web_extract only — no terminal or other code-execution tool, since the harness has no sandbox to run model-requested commands in. See references/provider-tool-support.md for which models support tool calling.

Judge sees the full tool call trace — what was searched, what was extracted, how many turns, whether it converged. This lets the judge evaluate tool selection strategy, not just the final answer.

Efficiency Analysis

The --efficiency flag (auto-enabled for tools mode) prints a token comparison table:

📊 Efficiency Comparison
                      Turns  Tools   Tok In  Tok Out  Ratio   Time
✅ ollama-cloud:glm-5.2      5      5     7806      777   0.10  15.3s
✅ ollama-cloud:kimi-k2.5    5      4     5434     1051   0.19  14.8s
✅ ollama-cloud:gemma4:31b   2      1      949      248   0.26   6.5s

Key metrics: turns, tool calls, tokens in (context consumed), tokens out (generated), efficiency ratio (out/in), time. Efficiency without accuracy is waste — a model that uses few tokens but gets the wrong answer is not efficient, it's just wrong fast.

Use Cases

Model selection for a workflow

> "I need a model for summarizing legal documents. Compare claude-sonnet-4.6, gpt-4o, and gemini-2.5-flash on this prompt: [legal text]"

Cost vs quality tradeoff

> "Compare glm-5.2 (free) vs claude-sonnet-4.6 (paid) on this coding prompt. Is the paid one worth it?"

Prompt robustness testing

> "Test this prompt on 3 models — I want to see which ones follow my formatting instructions"

Provider comparison

> "Compare the same model (llama-4-maverick) on OpenRouter vs NVIDIA — is there a difference?"

Provider Health Tracking

The scripts/provider_health.py module tracks provider endpoint health and implements dead

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: moonlight-lupin
• Source: moonlight-lupin/agent-skills
• License: MIT
• Homepage: https://hermes-agent.nousresearch.com

Install and usage instructions live in the source repository linked above.