Llm

/llm — Universal LLM Router

Route prompts to any LLM model across all providers. CLI-first for Codex/Kimi/Claude (zero cost), Google API for Gemini, OpenRouter for everything else. Auto-discovers new models.

When to Use

• You need to call a specific LLM model (not the current Claude Code session)
• You want to compare outputs across different models
• A task benefits from a specific model's strengths (e.g., GPT-5.3-Codex for code generation, Gemini for grounded search)
• You need to route through a specific provider (e.g., force OpenRouter for a model that usually goes through CLI)

When NOT to Use

• The current Claude Code session can handle the task directly (don't call yourself via CLI)
• You need Aristotle formal verification — use /prove instead
• You need multi-model debate — use /debate instead
• You need parallel Codex swarm — use /CodexCode instead

Quick Start

# Call a model
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Say hello in 3 words"

# With system prompt
python3 ~/.claude/skills/llm/scripts/llm_route.py --model chatgpt-5.4 --prompt "Write fizzbuzz" --system "You are a Python expert"

# From files
python3 ~/.claude/skills/llm/scripts/llm_route.py --model gemini-3.1-pro --prompt-file prompt.txt --system-file system.txt

# Force a different provider
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" --route openrouter

# JSON output with metadata
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" --json

# Custom parameters
python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" --temperature 0.3 --max-tokens 8192 --timeout 300

# Pipe prompt via stdin
echo "Explain quantum computing" | python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus

# List models
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models --all
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models --tier 2

# List providers
python3 ~/.claude/skills/llm/scripts/llm_route.py --list-providers

Routing Hierarchy

Models are routed to providers in this priority order:

| Model prefix | Provider | Cost | Auth | |---|---|---|---| | Anthropic (opus) | Claude CLI | Subscription (free) | claude login | | OpenAI (gpt-5.2, chatgpt-5.4) | Codex CLI | Subscription (free) | codex login | | Moonshot (kimi-2.5) | Kimi CLI | Subscription (free) | kimi login | | Google (gemini-3.1-pro, gemini-3-flash) | Google GenAI API | Free tier | GOOGLE_API_KEY | | Everything else | OpenRouter | Per-token | OPENROUTER_API_KEY |

Use --route to override the default route for any model. For example, --route openrouter forces a CLI model through OpenRouter instead.

> Important: CLI tools must be installed on the machine running Claude Code. The CLI-first routing for GPT models (Codex CLI) and Kimi models (Kimi CLI) requires those CLIs to be installed and authenticated locally. If you're running Claude Code on a remote server, CI runner, or any machine without these CLIs, those routes will silently fail. Use --route openrouter to force API-based routing instead, or install the CLIs: > ``bash > npm install -g @openai/codex && codex login # GPT models > npm install -g kimi-cli && kimi login # Kimi models > `` > Claude CLI is always available since you're already running inside Claude Code.

Model Tiers

| Tier | Description | Auto-update | Default visibility | |---|---|---|---| | 1 | Manually curated (9 models) | Never overwritten | Always shown | | 2 | Auto-discovered notable (major provider, context >= 32k) | Added automatically | Shown by default | | 3 | Auto-discovered everything else | Added automatically | Hidden (use --all) |

Tier 1 Models (Curated)

| Name | Provider | Description | |---|---|---| | opus | Claude CLI | Claude Opus 4.6 — most capable, 1M context | | chatgpt-5.4 | Codex CLI | GPT-5.3 — best for code, reasoning_effort=xhigh | | gpt-5.2 | Codex CLI | GPT-5.2 — strong general purpose | | gemini-3.1-pro | Google API | Gemini 3 Pro — thinkingLevel=HIGH | | gemini-3-flash | Google API | Gemini 3 Flash — fast + grounded | | kimi-2.5 | Kimi CLI | Kimi 2.5 — --thinking flag | | glm-5 | OpenRouter | GLM-5 — ZhipuAI, built-in thinking | | minimax-m2.5 | OpenRouter | MiniMax M2.5 — built-in thinking | | aristotle | Aristotle | Formal theorem prover (use /prove) |

Auto-Discovery

Discover new models from OpenRouter and Google APIs:

# Dry run — show what's new
python3 ~/.claude/skills/llm/scripts/discover_models.py

# Apply — update the registry
python3 ~/.claude/skills/llm/scripts/discover_models.py --apply

# Query only one source
python3 ~/.claude/skills/llm/scripts/discover_models.py --source openrouter
python3 ~/.claude/skills/llm/scripts/discover_models.py --source google

Discovery never overwrites tier 1 models. New models from major providers with context >= 32k become tier 2; everything else is tier 3.

Prompting Overrides

Per-model temperature and system prompt wrapping is applied automatically from settings/prompting-overrides.json:

• Claude (opus): XML format preferred, no temperature override
• GPT (gpt-5.2/chatgpt-5.4): CTCO framework preamble, temperature not overridden (reasoning model)
• Gemini (gemini-3.1-pro/gemini-3-flash): Forced temperature=1.0, concise preamble
• Kimi (kimi-2.5): temperature=1.0
• MiniMax (minimax-m2.5): temperature=1.0

Credential Resolution

API keys are resolved in this order:

1. Environment variable (e.g., GOOGLE_API_KEY)
2. Debate agent's shared keys at ~/.claude/skills/convolutional-debate-agent/api-keys/provider-keys.env
3. Error with setup instructions

CLI tools (claude, codex, kimi) use their own stored logins — no API keys needed.

File Structure

~/.claude/skills/llm/
├── SKILL.md                              # This file
├── scripts/
│   ├── llm_route.py                      # Core router (provider calls + CLI)
│   ├── discover_models.py                # Auto-discovery from OpenRouter/Google
│   ├── fetch_benchmarks.py              # Fetch benchmarks from public leaderboards
│   ├── scrape_vals.py                  # Headless scraper for vals.ai benchmarks
│   └── .venv/                          # Python venv for Playwright (scrape_vals.py)
├── settings/
│   ├── model-registry.json               # All known models + routes + tiers
│   ├── routing-rules.json                # Regex patterns for auto-routing new models
│   ├── prompting-overrides.json          # Per-model temperature + system preambles
│   └── benchmark-quality.json            # BetterBench quality metadata per benchmark
├── benchmarks/
│   ├── rankings.csv                      # Unified rankings — THE file other skills read
│   ├── vals.json                         # Scraped vals.ai data (all 23 benchmarks)
│   └── _meta.json                        # Fetch timestamps and source status
└── references/
    ├── provider-setup.md                 # Auth setup per provider
    └── betterbench-notes.md              # BetterBench paper findings + methodology

Benchmarks

Fetch and cache LLM benchmark rankings from public leaderboards. The unified CSV at benchmarks/rankings.csv is the canonical source other skills should read for model comparisons.

Data Sources

| Source | What it provides | Update frequency | Auth needed | |---|---|---|---| | Chatbot Arena (LMArena) | Elo rankings from human preference voting | Monthly (arena-catalog JSON, Dec 2025) | None | | Epoch AI | GPQA Diamond, MATH, SWE-bench, coding, LiveBench scores | Daily CSV updates (has 2026 data) | None | | OpenRouter | Pricing ($/1M tokens), context length | Real-time | None | | Artificial Analysis | Intelligence Index (0-100), speed (TPS/TTFT), eval scores | Continuous | ARTIFICIAL_ANALYSIS_API_KEY | | Vals.ai | Industry benchmarks: finance, legal, healthcare, coding agents (23 benchmarks) | Manual scrape via Playwright | None (headless browser) |

Evaluated but skipped: LiveBench (already in Epoch AI data), LM Council (aggregator of our sources), LLM Stats/ZeroEval (aggregator of our sources).

Quality methodology: Informed by BetterBench (NeurIPS 2024 Spotlight). Each model row gets a benchmark_quality tier (high/medium/low) based on which high-quality data sources contributed scores. See references/betterbench-notes.md for details.

Fetch Commands

# Fetch all sources, update rankings.csv
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py

# Fetch only one source
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source arena
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source epoch
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source openrouter
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source aa
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --source vals

# Scrape vals.ai (requires Playwright venv, ~90s for all 23 benchmarks)
~/.claude/skills/llm/scripts/.venv/bin/python3 ~/.claude/skills/llm/scripts/scrape_vals.py

# View local rankings (no network calls)
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --list
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --list --top 50

# Look up a specific model
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --model opus
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --model gemini-3.1-pro --json

CSV Schema (benchmarks/rankings.csv)

| Column | Description | Source | |---|---|---| | model | Display name from best available source | All | | provider | Organization (anthropic, openai, google, etc.) | All | | arena_elo | Chatbot Arena Elo score (human preference) | Arena | | gpqa | GPQA Diamond score (%) — PhD-level science reasoning | Epoch AI, AA | | mmlu | MMLU score (%) — knowledge + reasoning | Epoch AI, AA | | coding | Best coding score (Aider polyglot, %) | Epoch AI, AA | | math | MATH Level 5 score (%) | Epoch AI, AA | | swe_bench | SWE-bench Verified resolve rate (%) | Epoch AI | | aa_index | Artificial Analysis Intelligence Index (0-100 composite) | AA | | quality_index | LiveBench global average | Epoch AI | | speed_tps | Tokens per second (median) | AA | | speed_ttft | Time to first token in seconds | AA | | context | Context window (tokens) | OpenRouter, AA | | price_in | Input price ($/1M tokens) | OpenRouter | | price_out | Output price ($/1M tokens) | OpenRouter | | vals_index | Vals Index composite — finance/law/coding weighted by GDP (%) | Vals.ai | | vals_finance | CorpFin v2 — credit agreement understanding (%) | Vals.ai | | vals_fin_agent | Finance Agent — financial analyst tasks (%) | Vals.ai | | vals_legal | CaseLaw v2 — legal QA on court cases (%) | Vals.ai | | vals_medical | MedQA — medical question answering (%) | Vals.ai | | vals_lcb | LiveCodeBench — Vals implementation (%) | Vals.ai | | vals_swe | SWE-bench — Vals implementation (%) | Vals.ai | | vals_terminal | Terminal-Bench 2.0 — terminal tasks (%) | Vals.ai | | vals_aime | AIME — national math exam (%) | Vals.ai | | benchmark_quality | BetterBench-informed quality tier: high/medium/low | Computed | | registry_name | Matching name in model-registry.json (empty if none) | Computed | | sources | Comma-separated list of data sources | Computed |

Usage from Other Skills

Other skills can read the local CSV for model selection decisions:

import csv
from pathlib import Path

rankings_path = Path.home() / ".claude/skills/llm/benchmarks/rankings.csv"
with open(rankings_path) as f:
    for row in csv.DictReader(f):
        if row["registry_name"] == "opus":
            print(f"Opus GPQA: {row['gpqa']}%, Arena Elo: {row['arena_elo']}")

Or via CLI for quick lookups:

# Get opus benchmarks as JSON
python3 ~/.claude/skills/llm/scripts/fetch_benchmarks.py --model opus --json

Error Handling

| Error | Cause | Fix | |---|---|---| | "CLI not found on PATH" | CLI tool not installed | Install it or use --route openrouter | | "API key not found" | Missing credential | Set env var or add to provider-keys.env | | "Unknown model" | Model not in registry | Run discover_models.py --apply or check spelling | | "No route" | Model has no configured route | Add route in model-registry.json | | "API error 429" | Rate limited | Wait and retry, or switch provider | | "Timed out" | Slow response | Increase --timeout (default 120s, CLI default 300s) |

Usage from Claude Code (Programmatic)

When invoked as /llm from Claude Code, the skill works as a reference for how to call models. Claude Code should:

1. Read this SKILL.md for routing decisions
2. Call llm_route.py via Bash tool for external model calls
3. Use --json flag when parsing the response programmatically
4. Use --route to force a specific provider when needed

Example from Claude Code:

# Get a response from GPT-5.3 and parse it
response=$(python3 ~/.claude/skills/llm/scripts/llm_route.py --model chatgpt-5.4 --prompt "Your prompt" --json)

Setup

1. Install CLIs (for zero-cost routing):

``bash npm install -g @anthropic-ai/claude-code # claude login npm install -g @openai/codex # codex login npm install -g kimi-cli # kimi login ``

1. Set API keys (for Google/OpenRouter):

``bash export GOOGLE_API_KEY="your-key" export OPENROUTER_API_KEY="your-key" ``

1. Verify:

``bash python3 ~/.claude/skills/llm/scripts/llm_route.py --list-models python3 ~/.claude/skills/llm/scripts/llm_route.py --model opus --prompt "Hello" ``

See references/provider-setup.md for detailed per-provider instructions.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ZealousEar
• Source: ZealousEar/claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.