Artificial Analysis

Artificial Analysis: pick the right model

Live benchmark data from https://artificialanalysis.ai — intelligence index (10 evals), coding index, math index, output speed, time-to-first-token, pricing, and context-window, for ~400+ LLMs plus media-model ELO leaderboards.

Your job: when the user needs a model recommendation, run recommend.sh with the right profile — do not guess from memory. Model rankings change weekly; never answer from training data alone.

Prerequisites

1. AA_API_KEY must be exported. Free key (1,000 req/day) at https://artificialanalysis.ai/documentation. If missing, scripts prompt on TTY or fail with a clear pointer.
2. curl and jq installed.

Responses cached 24h at ~/.cache/artificial-analysis/ — repeated calls are free of rate-limit cost.

Decision flow — ALWAYS follow this for "which model" questions

1. Classify the user's need → pick ONE profile (see table below).
2. Extract hard constraints: budget, min context, required provider, required quality floor.
3. Run:  recommend.sh  [--budget $/M] [--min-context N] [--min-intel N] [--creator X] --top 5
4. Report the top pick with its trade-offs vs. #2 and #3.
5. If the user names two or more specific models, follow with: compare.sh   ...

Do NOT skip step 3. Benchmark numbers change; the API is the source of truth.

Profile selection

| User says / implies | Profile | Optimizes for | |---|---|---| | "best / smartest / most capable model" | smart | intelligenceindex (tie-break: coding) | | no strong steer, general production use | balanced | intelligence + price + speed | | "fast / low latency / streaming / realtime" | fast | output t/s + low TTFT | | "cheap / bulk / high-volume / budget" | cheap | price (with a sane quality floor) | | "coding / code-gen / codegen agent" | coding | codingindex | | "math / reasoning-heavy / proofs" | math | math_index | | "long docs / 200k+ / large context" | long-ctx | context window, then intelligence | | "agent / tool loop / long-running" | agentic | intelligence + speed + context |

When in doubt, use balanced. If the user has two conflicting hints (e.g. "fast and smart"), pick the dominant one and say so.

Hard constraints → flags

• Budget → --budget 5 means ≤ $5/M tokens blended (3:1 input:output).
• Min context → --min-context 200000.
• Quality floor → --min-intel 60 (intelligence_index is ~0–100-ish).
• Specific vendor → --creator anthropic (matches slug or name, case-insensitive substring).

Core scripts

recommend.sh — the one you'll use most

# Pick the best general model
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" balanced --top 5

# Cheapest decent model for batch processing
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" cheap --min-intel 55

# Fast Anthropic model for streaming UI
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" fast --creator anthropic

# Smart model that fits a 500k-token document
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" long-ctx --min-context 500000

# Best coding model under $10/M
"${CLAUDE_SKILL_DIR}/scripts/recommend.sh" coding --budget 10

Output: ranked table with score, slug, creator, intel/code/math, t/s, TTFT, $/M, ctx.

models.sh — leaderboard

# Top 10 by raw intelligence
"${CLAUDE_SKILL_DIR}/scripts/models.sh" --top 10 --sort-by evaluations.artificial_analysis_intelligence_index

# Just the slugs (for piping into compare.sh)
"${CLAUDE_SKILL_DIR}/scripts/models.sh" --top 20 --slugs

# Inspect available fields on the first model
"${CLAUDE_SKILL_DIR}/scripts/models.sh" --path

Sort keys containing latency, price, ttft, or time_to_first are treated as lower-is-better automatically.

compare.sh — head-to-head

"${CLAUDE_SKILL_DIR}/scripts/compare.sh" claude-4-5-sonnet gpt-5-xhigh gemini-3-1-pro-preview

Prints rows for creator, intel, coding, math, t/s, TTFT, blended / input / output price, and context window.

media.sh — image / audio / video leaderboards

"${CLAUDE_SKILL_DIR}/scripts/media.sh" image --top 15
"${CLAUDE_SKILL_DIR}/scripts/media.sh" edit  --top 10
"${CLAUDE_SKILL_DIR}/scripts/media.sh" tts   --top 10
"${CLAUDE_SKILL_DIR}/scripts/media.sh" ttv   --categories --top 10
"${CLAUDE_SKILL_DIR}/scripts/media.sh" i2v   --categories --top 10

Ranked by ELO rating.

fetch.sh — raw access (for custom jq)

"${CLAUDE_SKILL_DIR}/scripts/fetch.sh" data/llms/models --raw \
  | jq '[.data[] | select(.context_window >= 1000000)] | length'

Guidance when reporting to the user

1. Name the top pick by slug, link to its AA page (https://artificialanalysis.ai/models/).
2. Justify with the two or three metrics that drove the choice — e.g. "intelligence 72, $3.20/M blended, 190 t/s".
3. Mention the runner-up in one line with the tradeoff ("If you want lower latency, try X at $/M but intel Y").
4. Attribute: say "per Artificial Analysis benchmarks" — that's the data source and their ToS expects it.
5. If slugs are unfamiliar to the user, offer compare.sh as a follow-up.

Failure handling

• 401 → remind user about AA_API_KEY and link to https://artificialanalysis.ai/documentation.
• 429 → the script auto-falls-back to stale cache; warn the user it's stale.
• Empty filter result → loosen a constraint (usually --budget or --min-intel).
• Unknown slug in compare.sh → it warns on stderr; list candidates with models.sh --slugs | grep -i .

Notes on authentication

The Artificial Analysis API requires an API key. There is no unauthenticated public endpoint — the website renders data server-side through authenticated calls. A free key (1,000 req/day) is explicitly offered for this use case; the skill caches aggressively to stay under that limit.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mrloldev
• Source: mrloldev/artificial-analysis-skill
• License: MIT

Install and usage instructions live in the source repository linked above.