Which Llm

Skill: which-llm

Select the right LLM(s) for a task using real benchmark and capability data.

When to Use

• User needs to pick a model for a specific task
• User wants to compare models by capability/price/speed
• User is designing a multi-agent system and needs model recommendations

Preflight Check

Before proceeding, verify the CLI is operational:

# 1. Check CLI exists and show version
which-llm --version

# 2. Check data freshness (should be  **Note:** Thresholds calibrated for Intelligence Index v4.0 (Jan 2026), SOTA ~50. Scores within ±2 points are effectively equivalent. See [references/BENCHMARKS.md](references/BENCHMARKS.md) for dynamic threshold calculation.

| Skill Type | Examples | Min Intelligence | Min Coding | Consider Also |
|------------|----------|------------------|------------|---------------|
| **Transformational** | summarize, extract, reformat | 20 | - | `tps` for high volume |
| **Analytical** | compare, analyze, justify | 38 | - | `context_window` (models table) for long docs |
| **Tool-using** | API calls, DB queries, code execution | 35 | 35 | `tool_call` (models table) |
| **Agentic** | plan, decompose, orchestrate, self-critique | 48 | 42 | `tool_call`, `reasoning`, `context_window` (all in models table) |

## Additional Selection Factors

Beyond skill type thresholds, consider these constraints when relevant:

| Factor | Table | Column | When to Use |
|--------|-------|--------|-------------|
| Context window | `models` | `context_window` | Long documents (>32k tokens), RAG with large chunks |
| Tool calling | `models` | `tool_call` | Function calling, MCP servers, API integration |
| Structured output | `models` | `structured_output` | JSON responses, typed outputs, schema validation |
| Reasoning | `models` | `reasoning` | Complex multi-step problems, chain-of-thought |
| Latency | `benchmarks` | `latency` | Real-time chat, streaming UIs (want  100 tps) |
| Open weights | `models` | `open_weights` | Self-hosting, fine-tuning, data privacy |

> **Note:** The `benchmarks` table contains AA benchmark data (intelligence, coding, price, tps). Capability fields (tool_call, reasoning, context_window) are in the `models` table from models.dev.

## Weighted Scoring

Instead of just filtering by thresholds, use weighted scoring to rank models based on user priorities.

### Scoring Formula

Score = (intelligence × quality_weight)

• (100/price × cost_weight)
• (tps/10 × speed_weight)

### Priority Presets

| Preset | Quality | Cost | Speed | Best For |
|--------|---------|------|-------|----------|
| **Balanced** | 0.4 | 0.4 | 0.2 | General use, no strong preference |
| **Quality** | 0.7 | 0.2 | 0.1 | Critical tasks, accuracy matters most |
| **Cost** | 0.2 | 0.7 | 0.1 | High volume, budget-sensitive |
| **Speed** | 0.2 | 0.2 | 0.6 | Real-time, latency-sensitive |

### Weighted Query Example

```bash
# Balanced scoring for analytical tasks
which-llm query "SELECT name, intelligence, price, tps,
          ROUND((intelligence * 0.4) + (100/price * 0.4) + (tps/10 * 0.2), 1) as score
          FROM benchmarks 
          WHERE intelligence >= 38 AND price > 0
          ORDER BY score DESC 
          LIMIT 10"

# Cost-priority scoring
which-llm query "SELECT name, intelligence, price, tps,
          ROUND((intelligence * 0.2) + (100/price * 0.7) + (tps/10 * 0.1), 1) as score
          FROM benchmarks 
          WHERE intelligence >= 38 AND price > 0
          ORDER BY score DESC 
          LIMIT 10"

See [references/QUERIES.md](references/QUERIES.md) for more weighted scoring patterns.

Core Queries

Two-Table Architecture

The CLI provides two independent tables:

1. benchmarks table - Benchmark data from Artificial Analysis

• Contains: intelligence, coding, math, pricing (inputprice, outputprice), performance (tps, latency)
• Use for: Model selection based on benchmarks and pricing

1. models table - Capability data from models.dev

• Contains: toolcall, reasoning, structuredoutput, context_window, provider info
• Use for: Filtering by capabilities, finding providers for a model

The benchmarks Table (Benchmarks & Pricing)

The benchmarks table contains benchmark scores and pricing from Artificial Analysis.

# Find models meeting benchmark requirements, sorted by price
which-llm query "SELECT name, creator, intelligence, coding, price, tps 
          FROM benchmarks 
          WHERE intelligence >= 38 
          ORDER BY price 
          LIMIT 10"

# Find high-capability models for agentic tasks
which-llm query "SELECT name, creator, intelligence, coding, price 
          FROM benchmarks 
          WHERE intelligence >= 48 AND coding >= 42
          ORDER BY price 
          LIMIT 10"

# Speed-critical (real-time chat)
which-llm query "SELECT name, intelligence, tps, latency, price 
          FROM benchmarks 
          WHERE tps > 100 AND latency = 128000
          ORDER BY cost_input LIMIT 10"

# Find all providers offering Claude models
which-llm query "SELECT provider_name, model_id, cost_input, cost_output 
          FROM models 
          WHERE model_name LIKE '%Claude%'
          ORDER BY cost_input"

# Find cheapest provider for a specific model family
which-llm query "SELECT provider_name, model_id, cost_input, cost_output
          FROM models
          WHERE family = 'claude-3.5'
          ORDER BY cost_input LIMIT 5"

# Get provider configuration for OpenAI
which-llm query "SELECT provider_env, provider_npm, provider_api, provider_doc
          FROM models
          WHERE provider_id = 'openai'
          LIMIT 1"

# Find models with cache pricing
which-llm query "SELECT provider_name, model_id, cost_cache_read, cost_cache_write
          FROM models
          WHERE cost_cache_read IS NOT NULL
          ORDER BY cost_cache_read LIMIT 10"

Cross-Table Queries

Since the tables are independent, you may need to query both to make a complete decision:

# Step 1: Find high-capability models from benchmarks table
which-llm query "SELECT name, intelligence, coding, price FROM benchmarks 
          WHERE intelligence >= 45 ORDER BY price LIMIT 5"

# Step 2: Check capabilities for a specific model in models table
which-llm query "SELECT model_id, tool_call, reasoning, context_window FROM models 
          WHERE model_name LIKE '%GPT-4o%'"

Note: Model naming may differ between tables (e.g., claude-3.5-sonnet in benchmarks vs claude-3-5-sonnet-20241022 in models). Use LIKE with wildcards for fuzzy matching when cross-referencing.

Compare Models

Use the compare command for side-by-side model comparison with winner highlighting:

# Compare candidate models directly
which-llm compare "gpt-5 (high)" "claude 4.5 sonnet" "gemini 2.5 pro"

# Include additional metrics with --verbose
which-llm compare "gpt-5" "claude-4.5" --verbose

# Output as JSON for programmatic use
which-llm compare "gpt-5" "claude-4.5" --json

Winners for each metric are marked with *. This is useful when presenting trade-offs to users.

Calculate Token Costs

Use the cost command to estimate costs and project usage:

# Calculate cost for a single model
which-llm cost "gpt-5 (high)" --input 10k --output 5k

# Compare costs across multiple models
which-llm cost "gpt-5" "claude 4.5" --input 1M --output 500k

# Project daily/monthly costs with request volume
which-llm cost "gpt-5 (high)" --input 2k --output 1k --requests 1000 --period daily

Token units: k (thousands), M (millions), B (billions). Decimals supported (e.g., 1.5M).

Cascade Recommendations

For cost optimization, recommend a Primary + Fallback pair instead of a single model:

1. Primary model: Cheapest model meeting relaxed requirements (e.g., 5-10 points below strict threshold)
2. Fallback model: Higher-capability model meeting strict requirements for when primary fails

Relaxed vs Strict thresholds:

• Strict: Use the skill type thresholds directly (e.g., Agentic: intelligence ≥ 48)
• Relaxed: Lower by ~20% for primary selection (e.g., intelligence ≥ 40)

This approach can reduce costs by 50-70% compared to always using the best model. See [references/CASCADE-PATTERNS.md](references/CASCADE-PATTERNS.md) for detailed implementation guidance.

Output Format

## Task Classification
- **Skill Type:** [type]
- **Key Constraints:** [e.g., needs tool_call, context > 100k]
- **Priority:** [Balanced/Quality/Cost/Speed]
- **Reasoning:** [why this classification]

## Recommendations

### Primary: [Model] ($X.XX/M) - Score: Y
- Meets requirements: intelligence X, coding Y
- Capabilities: [relevant capabilities like tool_call, context_window]
- Why: Best cost/capability ratio for this task

### Fallback: [Model] ($X.XX/M) - Score: Y
- Use if: Primary fails or task is unusually complex
- Capabilities: [relevant capabilities]
- Why: Higher capability ceiling

### Other Options
| Rank | Model | Score | Intelligence | Price | Key Capability |
|------|-------|-------|--------------|-------|----------------|
| 3 | ... | ... | ... | ... | ... |
| 4 | ... | ... | ... | ... | ... |

## Cost Estimate
- **Primary only:** $X.XX/M tokens
- **Fallback only:** $Y.YY/M tokens  
- **Cascade (30% fallback):** $Z.ZZ/M tokens
- **Savings vs always fallback:** NN%

### Calculation
Cascade cost = (0.70 × $X.XX) + (0.30 × $Y.YY) = $Z.ZZ/M
Savings = (1 - Z.ZZ/Y.YY) × 100 = NN%

## Query Used
[the which-llm query command]

Cost Estimate Guidance

When presenting cost estimates:

1. Always show the cascade calculation - helps users understand the math
2. Use 30% as default escalation rate - reasonable starting assumption
3. Note the price ratio - larger ratios mean more potential savings
4. Include the caveat - actual savings depend on task complexity mix

Example:

## Cost Estimate
- **Primary only:** $0.50/M tokens
- **Fallback only:** $5.00/M tokens (10x primary)
- **Cascade (30% fallback):** $1.85/M tokens
- **Savings vs always fallback:** 63%

*Assumes 30% of requests escalate. Actual rate depends on task complexity.*

Important Disclaimer

These recommendations are indicative starting points, not definitive answers. Benchmark scores measure general capabilities but may not reflect performance on your specific task.

Always validate by testing candidate models on representative examples from your actual use case before committing to a model choice.

References

For detailed information, see:

• [references/FALLBACK.md](references/FALLBACK.md) - Quick recommendations when CLI is unavailable
• [references/INSTALL.md](references/INSTALL.md) - Installing and configuring the which-llm CLI
• [references/BENCHMARKS.md](references/BENCHMARKS.md) - What the scores mean + dynamic thresholds
• [references/BENCHMARK-LIMITATIONS.md](references/BENCHMARK-LIMITATIONS.md) - What benchmarks can't tell you
• [references/QUERIES.md](references/QUERIES.md) - Common query patterns
• [references/WORKFLOWS.md](references/WORKFLOWS.md) - End-to-end workflow examples
• [references/MULTI-MODEL.md](references/MULTI-MODEL.md) - Multi-model architecture guidance
• [references/CASCADE-PATTERNS.md](references/CASCADE-PATTERNS.md) - Cascade and fallback patterns
• [references/SPECIALIZATION.md](references/SPECIALIZATION.md) - Domain-specific model guidance
• [references/PROVIDERS.md](references/PROVIDERS.md) - Provider-specific considerations

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: richard-gyiko
• Source: richard-gyiko/which-llm
• License: MIT

Install and usage instructions live in the source repository linked above.