Arize Evaluator

Arize Evaluator Skill

> SPACE — All --space flags and the ARIZE_SPACE env var accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list.

This skill covers designing, creating, and running evaluators on Arize — both LLM-as-judge (template) evaluators and code evaluators (deterministic, no LLM required). An evaluator defines the judge; a task is how you run it against real data.

---

Prerequisites

Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.

If an ax command fails, troubleshoot based on the error:

• command not found or version error → see references/ax-setup.md
• 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
• Space unknown → run ax spaces list to pick by name, or ask the user
• LLM provider call fails (missing OPENAIAPIKEY / ANTHROPICAPIKEY) → run ax ai-integrations list --space SPACE to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
• Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are not available through these channels, ask the user.
• CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with ax ai-integrations list, (4) contact support at https://arize.com/support

---

Concepts

What is an Evaluator?

An evaluator is an LLM-as-judge definition. It contains:

| Field | Description | |-------|-------------| | Template | The judge prompt. Uses {variable} placeholders (e.g. {input}, {output}, {context}) that get filled in at run time via a task's column mappings. | | Classification choices | The set of allowed output labels (e.g. factual / hallucinated). Binary is the default and most common. Each choice can optionally carry a numeric score. | | AI Integration | Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. | | Model | The specific judge model (e.g. gpt-4o, claude-sonnet-4-5). | | Invocation params | Optional JSON of model settings like {"temperature": 0}. Low temperature is recommended for reproducibility. | | Optimization direction | Whether higher scores are better (maximize) or worse (minimize). Sets how the UI renders trends. | | Data granularity | Whether the evaluator runs at the span, trace, or session level. Most evaluators run at the span level. |

Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active.

What is a Task?

A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains:

| Field | Description | |-------|-------------| | Evaluators | List of evaluators to run. You can run multiple in one task. | | Column mappings | Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. "input" → "attributes.input.value"). This is what makes evaluators portable across projects and experiments. | | Query filter | SQL-style expression to select which spans/runs to evaluate (e.g. "span_kind = 'LLM'"). Optional but important for precision. | | Continuous | For project tasks: whether to automatically score new spans as they arrive. | | Sampling rate | For continuous project tasks: fraction of new spans to evaluate (0–1). |

---

Data Granularity

The --data-granularity flag controls what unit of data the evaluator scores. It defaults to span and only applies to project tasks (not dataset/experiment tasks — those evaluate experiment runs directly).

| Level | What it evaluates | Use for | Result column prefix | |-------|-------------------|---------|---------------------| | span (default) | Individual spans | Q&A correctness, hallucination, relevance | eval.{name}.label / .score / .explanation | | trace | All spans in a trace, grouped by context.trace_id | Agent trajectory, task correctness — anything that needs the full call chain | trace_eval.{name}.label / .score / .explanation | | session | All traces in a session, grouped by attributes.session.id and ordered by start time | Multi-turn coherence, overall tone, conversation quality | session_eval.{name}.label / .score / .explanation |

How trace and session aggregation works

For trace granularity, spans sharing the same context.trace_id are grouped together. Column values used by the evaluator template are comma-joined into a single string (each value truncated to 100K characters) before being passed to the judge model.

For session granularity, the same trace-level grouping happens first, then traces are ordered by start_time and grouped by attributes.session.id. Session-level values are capped at 100K characters total.

The {conversation} template variable

At session granularity, {conversation} is a special template variable that renders as a JSON array of {input, output} turns across all traces in the session, built from attributes.input.value / attributes.llm.input_messages (input side) and attributes.output.value / attributes.llm.output_messages (output side).

At span or trace granularity, {conversation} is treated as a regular template variable and resolved via column mappings like any other.

> Note: For {conversation} to work, spans must carry attributes.session.id. See the arize-instrumentation skill for how to emit session.id from application code, including the force_flush() pattern required for Jupyter notebooks and short-lived scripts.

Multi-evaluator tasks

A task can contain evaluators at different granularities. At runtime the system uses the highest granularity (session > trace > span) for data fetching and automatically splits into one child run per evaluator. Per-evaluator query_filter in the task's evaluators JSON further narrows which spans are included (e.g., only tool-call spans within a session).

---

Basic CRUD

AI Integrations

AI integrations store the LLM provider credentials the evaluator uses. For full CRUD — listing, creating for all providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Gemini, NVIDIA NIM, custom), updating, and deleting — use the arize-ai-provider-integration skill.

Quick reference for the common case (OpenAI):

# Check for an existing integration first
ax ai-integrations list --space SPACE

# Create if none exists
ax ai-integrations create \
  --name "My OpenAI Integration" \
  --provider openAI \
  --api-key $OPENAI_API_KEY

Copy the returned integration ID — it is required for ax evaluators create-template-evaluator --ai-integration-id.

Evaluators

# List / Get
ax evaluators list --space SPACE
ax evaluators list --space SPACE --name "Hallucination"   # substring filter
ax evaluators get ID                    # accepts name or ID
ax evaluators get NAME --space SPACE   # required when using name instead of ID
ax evaluators list-versions NAME_OR_ID
ax evaluators get-version VERSION_ID

# Update metadata only (name, description — not prompt/code)
ax evaluators update NAME_OR_ID \
  --name "New Name" \
  --description "Updated description"

# Delete (permanent — removes all versions)
ax evaluators delete NAME_OR_ID

Template evaluators (LLM-as-judge)

# Create a template evaluator (LLM-as-judge)
ax evaluators create-template-evaluator \
  --name "Answer Correctness" \
  --space SPACE \
  --description "Judges if the model answer is correct" \
  --template-name "correctness" \
  --commit-message "Initial version" \
  --ai-integration-id INT_ID \
  --model-name "gpt-4o" \
  --include-explanation \
  --use-function-calling \
  --classification-choices '{"correct": 1, "incorrect": 0}' \
  --template 'You are an evaluator. Given the user question and the model response, decide if the response correctly answers the question.

User question: {input}

Model response: {output}

Respond with exactly one of these labels: correct, incorrect'

# Create a new template version (for prompt or model changes — versions are immutable)
ax evaluators create-template-evaluator-version NAME_OR_ID \
  --commit-message "Added context grounding" \
  --template-name "correctness" \
  --ai-integration-id INT_ID \
  --model-name "gpt-4o" \
  --include-explanation \
  --classification-choices '{"correct": 1, "incorrect": 0}' \
  --template 'Updated prompt...

{input} / {output} / {context}'

Key flags for create-template-evaluator:

| Flag | Required | Description | |------|----------|-------------| | --name | yes | Evaluator name (unique within space) | | --space | yes | Space name or ID to create in | | --template-name | yes | Eval column name — alphanumeric, spaces, hyphens, underscores | | --commit-message | yes | Description of this version | | --ai-integration-id | yes | AI integration ID (from above) | | --model-name | yes | Judge model (e.g. gpt-4o) | | --template | yes | Prompt with {variable} placeholders (single-quoted in bash) | | --classification-choices | yes | JSON object mapping choice labels to numeric scores e.g. '{"correct": 1, "incorrect": 0}' | | --description | no | Human-readable description | | --include-explanation | no | Include reasoning alongside the label | | --use-function-calling | no | Prefer structured function-call output | | --invocation-params | no | JSON of model params e.g. '{"temperature": 0}' | | --provider-params | no | JSON object of provider-specific parameters | | --data-granularity | no | span (default), trace, or session. Only relevant for project tasks, not dataset/experiment tasks. See Data Granularity section. | | --direction | no | Optimization direction: maximize, minimize, or none. Sets how the UI renders trends. |

Code evaluators (deterministic, no LLM)

Code evaluators run without an AI integration — they use deterministic logic (regex, JSON checks, keyword matching, or custom Python). Use them for fast, low-cost checks that don't need language understanding.

Managed code evaluators use built-in patterns:

# Managed: check output matches a regex
ax evaluators create-code-evaluator \
  --name "JSON Format Check" \
  --space SPACE \
  --template-name "json_format" \
  --commit-message "Initial version" \
  --code-type managed \
  --code-name "json_check" \
  --managed-evaluator JSONParseable \
  --variables '[]'

# Managed: check output contains required keywords
ax evaluators create-code-evaluator \
  --name "Safety Keywords" \
  --space SPACE \
  --template-name "safety_check" \
  --commit-message "Initial version" \
  --code-type managed \
  --code-name "safety_keywords" \
  --managed-evaluator ContainsAnyKeyword \
  --variables '[{"name": "keywords", "value": ["unsafe", "harmful", "illegal"]}]'

Managed evaluator types:

| Value | What it checks | |-------|---------------| | MatchesRegex | Output matches a regular expression | | JSONParseable | Output is valid JSON | | ContainsAnyKeyword | Output contains at least one keyword from a list | | ContainsAllKeywords | Output contains all keywords from a list | | ExactMatch | Output exactly equals a target string |

Custom Python code evaluators:

# Custom Python: inline code
ax evaluators create-code-evaluator \
  --name "Word Count Check" \
  --space SPACE \
  --template-name "word_count" \
  --commit-message "Initial version" \
  --code-type custom \
  --code-name "word_count_eval" \
  --variables '[{"name": "max_words", "value": 100}]' \
  --code 'def evaluate(output, max_words):
    count = len(output.split())
    return {"label": "pass" if count  `PROJECT_NAME`, `DATASET_NAME`, and `evaluator_id` all accept a name or base64 ID.

```bash
# List / Get
ax tasks list --space SPACE
ax tasks list --project PROJECT_NAME
ax tasks list --dataset DATASET_NAME --space SPACE
ax tasks list --task-type template_evaluation   # filter by type: template_evaluation, code_evaluation, run_experiment
ax tasks get TASK_ID

# Create evaluation task (project — continuous)
ax tasks create-evaluation \
  --name "Correctness Monitor" \
  --task-type template_evaluation \
  --project PROJECT_NAME \
  --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \
  --is-continuous \
  --sampling-rate 0.1

# Create evaluation task (project — one-time / backfill)
ax tasks create-evaluation \
  --name "Correctness Backfill" \
  --task-type template_evaluation \
  --project PROJECT_NAME \
  --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \
  --no-continuous

# Create evaluation task (experiment / dataset)
ax tasks create-evaluation \
  --name "Experiment Scoring" \
  --task-type template_evaluation \
  --dataset DATASET_NAME --space SPACE \
  --experiment-ids "EXP_ID_1,EXP_ID_2" \   # base64 IDs from `ax experiments list --space SPACE -o json`
  --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"output": "output"}}]' \
  --no-continuous

# Create run-experiment task (runs an experiment via a task)
ax tasks create-run-experiment \
  --name "GPT-4o Baseline Run" \
  --dataset DATASET_NAME \
  --run-configuration '{"model": "gpt-4o", "temperature": 0}' \
  --space SPACE

# Update a task (mutable fields only)
ax tasks update TASK \
  --name "New Task Name" \
  --sampling-rate 0.2 \
  --is-continuous \
  --query-filter "span_kind = 'LLM'" \
  --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"output": "output"}}]'

# Delete a task (irreversible)
ax tasks delete TASK --force

# Trigger a run (project task — use data window)
ax tasks trigger-run TASK_ID \
  --data-start-time "2026-03-20T00:00:00" \
  --data-end-time "2026-03-21T23:59:59" \
  --wait

# Trigger a run (experiment task — use experiment IDs)
ax tasks trigger-run TASK_ID \
  --experiment-ids "EXP_ID_1" \   # base64 ID from `ax experiments list --space SPACE -o json`
  --wait

# Monitor
ax tasks list-runs TASK_ID
ax tasks get-run RUN_ID
ax tasks wait-for-run RUN_ID --timeout 300
ax tasks cancel-run RUN_ID --force

> Note: ax tasks create (generic) also works and dispatches by --task-type. create-evaluation and create-run-experiment are dedicated shortcuts with clearer flag validation.

Time format for trigger-run: 2026-03-21T09:00:00 — no trailing Z.

Additional trigger-run flags:

| Flag | Description | |------|-------------| | --max-spans | Cap processed spans (default 10,000) | | --override-evaluations | Re-score spans that already have labels | | --wait / -w | Block until the run finishes | | --timeout | Seconds to wait with --wait (default 600) | | --poll-interval | Poll interval in seconds when waiting (default 5) |

Run status guide:

| Status | Meaning | |--------|---------| | completed, 0 spans | The eval index l

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Arize-ai
• Source: Arize-ai/arize-skills
• License: MIT

Install and usage instructions live in the source repository linked above.