Arize Prompt Optimization

Arize Prompt Optimization Skill

> SPACE — All --space flags and the ARIZE_SPACE env var accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list.

Related skills

• arize-prompts: Create, version, and label prompts in Prompt Hub with ax prompts (JSON messages, providers, labels such as production). Use that skill when the artifact should live in Arize; use arize-prompt-optimization below to improve prompt text from traces, datasets, and experiments.

Concepts

Where Prompts Live in Trace Data

LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:

| Column | What it contains | When to use | |--------|-----------------|-------------| | attributes.llm.input_messages | Structured chat messages (system, user, assistant, tool) in role-based format | Primary source for chat-based LLM prompts | | attributes.llm.input_messages.roles | Array of roles: system, user, assistant, tool | Extract individual message roles | | attributes.llm.input_messages.contents | Array of message content strings | Extract message text | | attributes.input.value | Serialized prompt or user question (generic, all span kinds) | Fallback when structured messages are not available | | attributes.llm.prompt_template.template | Template with {variable} placeholders (e.g., "Answer {question} using {context}") | When the app uses prompt templates | | attributes.llm.prompt_template.variables | Template variable values (JSON object) | See what values were substituted into the template | | attributes.output.value | Model response text | See what the LLM produced | | attributes.llm.output_messages | Structured model output (including tool calls) | Inspect tool-calling responses |

Finding Prompts by Span Kind

• LLM span (attributes.openinference.span.kind = 'LLM'): Check attributes.llm.input_messages for structured chat messages, OR attributes.input.value for a serialized prompt. Check attributes.llm.prompt_template.template for the template.
• Chain/Agent span: attributes.input.value contains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree.
• Tool span: attributes.input.value has tool input, attributes.output.value has tool result. Not typically where prompts live.

Performance Signal Columns

These columns carry the feedback data used for optimization:

| Column pattern | Source | What it tells you | |---------------|--------|-------------------| | annotation..label | Human reviewers | Categorical grade (e.g., correct, incorrect, partial) | | annotation..score | Human reviewers | Numeric quality score (e.g., 0.0 - 1.0) | | annotation..text | Human reviewers | Freeform explanation of the grade | | eval..label | LLM-as-judge evals | Automated categorical assessment | | eval..score | LLM-as-judge evals | Automated numeric score | | eval..explanation | LLM-as-judge evals | Why the eval gave that score -- most valuable for optimization | | attributes.input.value | Trace data | What went into the LLM | | attributes.output.value | Trace data | What the LLM produced | | {experiment_name}.output | Experiment runs | Output from a specific experiment |

Prerequisites

Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.

If an ax command fails, troubleshoot based on the error:

• command not found or version error → see references/ax-setup.md
• 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
• Space unknown → run ax spaces list to pick by name, or ask the user
• Project unclear → ask the user, or run ax projects list -o json --limit 100 and present as selectable options
• LLM provider call fails (missing OPENAIAPIKEY / ANTHROPICAPIKEY) → run ax ai-integrations list --space SPACE to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
• Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are not available through these channels, ask the user.

When you must ask the user first

Still prefer ax spaces list, ax projects list, ax datasets list, ax experiments list, and exports over open-ended questions. If you still cannot proceed (e.g. multiple projects match the name the user gave, unclear trace vs experiment path, or destructive scope), do not jump straight into questions — use the same explicit framing as arize-instrumentation when it stops for scope or confirmation:

1. Acknowledge the skill, e.g.: I found the arize-prompt-optimization skill in this repo (you may add skills/arize-prompt-optimization/SKILL.md if helpful).
2. Then a clear pause line, e.g.: A few clarifying questions before I invoke it:
3. Ask minimal numbered or short bullet questions — only what blocks the next ax step in this skill.

Phase 1: Extract the Current Prompt

Find LLM spans containing prompts

# Sample LLM spans (where prompts live)
ax spans export PROJECT --filter "attributes.openinference.span.kind = 'LLM'" -l 10 --stdout

# Filter by model
ax spans export PROJECT --filter "attributes.llm.model_name = 'gpt-4o'" -l 10 --stdout

# Filter by span name (e.g., a specific LLM call)
ax spans export PROJECT --filter "name = 'ChatCompletion'" -l 10 --stdout

Export a trace to inspect prompt structure

# Export all spans in a trace
ax spans export PROJECT --trace-id TRACE_ID

# Export a single span
ax spans export PROJECT --span-id SPAN_ID

Extract prompts from exported JSON

# Extract structured chat messages (system + user + assistant)
jq '.[0] | {
  messages: .attributes.llm.input_messages,
  model: .attributes.llm.model_name
}' trace_*/spans.json

# Extract the system prompt specifically
jq '[.[] | select(.attributes.llm.input_messages.roles[]? == "system")] | .[0].attributes.llm.input_messages' trace_*/spans.json

# Extract prompt template and variables
jq '.[0].attributes.llm.prompt_template' trace_*/spans.json

# Extract from input.value (fallback for non-structured prompts)
jq '.[0].attributes.input.value' trace_*/spans.json

Reconstruct the prompt as messages

Once you have the span data, reconstruct the prompt as a messages array:

[
  {"role": "system", "content": "You are a helpful assistant that..."},
  {"role": "user", "content": "Given {input}, answer the question: {question}"}
]

If the span has attributes.llm.prompt_template.template, the prompt uses variables. Preserve these placeholders ({variable} or {{variable}}) -- they are substituted at runtime.

Phase 2: Gather Performance Data

From traces (production feedback)

# Find error spans -- these indicate prompt failures
ax spans export PROJECT \
  --filter "status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'" \
  -l 20 --stdout

# Find spans with low eval scores
ax spans export PROJECT \
  --filter "annotation.correctness.label = 'incorrect'" \
  -l 20 --stdout

# Find spans with high latency (may indicate overly complex prompts)
ax spans export PROJECT \
  --filter "attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000" \
  -l 20 --stdout

# Export error traces for detailed inspection
ax spans export PROJECT --trace-id TRACE_ID

From datasets and experiments

# Export a dataset (ground truth examples)
ax datasets export DATASET_NAME --space SPACE
# -> dataset_*/examples.json

# Export experiment results (what the LLM produced)
ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE
# -> experiment_*/runs.json

Merge dataset + experiment for analysis

Join the two files by example_id to see inputs alongside outputs and evaluations:

# Count examples and runs
jq 'length' dataset_*/examples.json
jq 'length' experiment_*/runs.json

# View a single joined record
jq -s '
  .[0] as $dataset |
  .[1][0] as $run |
  ($dataset[] | select(.id == $run.example_id)) as $example |
  {
    input: $example,
    output: $run.output,
    evaluations: $run.evaluations
  }
' dataset_*/examples.json experiment_*/runs.json

# Find failed examples (where eval score  Phase 1 (once)
2. Run experiment    -> ax experiments create ...
3. Export results    -> ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE
4. Analyze failures  -> jq to find low scores
5. Run meta-prompt   -> Phase 3 with new failure data
6. Apply revised prompt
7. Repeat from step 2

Measure improvement

# Compare scores across experiments
# Experiment A (baseline)
jq '[.[] | .evaluations.correctness.score] | add / length' experiment_a/runs.json

# Experiment B (optimized)
jq '[.[] | .evaluations.correctness.score] | add / length' experiment_b/runs.json

# Find examples that flipped from fail to pass
jq -s '
  [.[0][] | select(.evaluations.correctness.label == "incorrect")] as $fails |
  [.[1][] | select(.evaluations.correctness.label == "correct") |
    select(.example_id as $id | $fails | any(.example_id == $id))
  ] | length
' experiment_a/runs.json experiment_b/runs.json

A/B compare two prompts

1. Create two experiments against the same dataset, each using a different prompt version
2. Export both: ax experiments export EXP_A and ax experiments export EXP_B
3. Compare average scores, failure rates, and specific example flips
4. Check for regressions -- examples that passed with prompt A but fail with prompt B

Prompt Engineering Best Practices

Apply these when writing or revising prompts:

| Technique | When to apply | Example | |-----------|--------------|---------| | Clear, detailed instructions | Output is vague or off-topic | "Classify the sentiment as exactly one of: positive, negative, neutral" | | Instructions at the beginning | Model ignores later instructions | Put the task description before examples | | Step-by-step breakdowns | Complex multi-step processes | "First extract entities, then classify each, then summarize" | | Specific personas | Need consistent style/tone | "You are a senior financial analyst writing for institutional investors" | | Delimiter tokens | Sections blend together | Use ---, ###, or XML tags to separate input from instructions | | Few-shot examples | Output format needs clarification | Show 2-3 synthetic input/output pairs | | Output length specifications | Responses are too long or short | "Respond in exactly 2-3 sentences" | | Reasoning instructions | Accuracy is critical | "Think step by step before answering" | | "I don't know" guidelines | Hallucination is a risk | "If the answer is not in the provided context, say 'I don't have enough information'" |

Variable preservation

When optimizing prompts that use template variables:

• Single braces ({variable}): Python f-string / Jinja style. Most common in Arize.
• Double braces ({{variable}}): Mustache style. Used when the framework requires it.
• Never add or remove variable placeholders during optimization
• Never rename variables -- the runtime substitution depends on exact names
• If adding few-shot examples, use literal values, not variable placeholders

Workflows

Optimize a prompt from a failing trace

1. Find failing traces:

``bash ax traces list PROJECT --filter "status_code = 'ERROR'" --limit 5 ``

1. Export the trace:

``bash ax spans export PROJECT --trace-id TRACE_ID ``

1. Extract the prompt from the LLM span:

``bash jq '[.[] | select(.attributes.openinference.span.kind == "LLM")][0] | { messages: .attributes.llm.input_messages, template: .attributes.llm.prompt_template, output: .attributes.output.value, error: .attributes.exception.message }' trace_*/spans.json ``

1. Identify what failed from the error message or output
2. Fill in the optimization meta-prompt (Phase 3) with the prompt and error context
3. Apply the revised prompt

Optimize using a dataset and experiment

1. Find the dataset and experiment:

``bash ax datasets list --space SPACE ax experiments list --dataset DATASET_NAME --space SPACE ``

1. Export both:

``bash ax datasets export DATASET_NAME --space SPACE ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE ``

1. Prepare the joined data for the meta-prompt
2. Run the optimization meta-prompt
3. Create a new experiment with the revised prompt to measure improvement

Debug a prompt that produces wrong format

1. Export spans where the output format is wrong:

``bash ax spans export PROJECT \ --filter "attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'" \ -l 10 --stdout > bad_format.json ``

1. Look at what the LLM is producing vs what was expected
2. Add explicit format instructions to the prompt (JSON schema, examples, delimiters)
3. Common fix: add a few-shot example showing the exact desired output format

Reduce hallucination in a RAG prompt

1. Find traces where the model hallucinated:

``bash ax spans export PROJECT \ --filter "annotation.faithfulness.label = 'unfaithful'" \ -l 20 --stdout ``

1. Export and inspect the retriever + LLM spans together:

``bash ax spans export PROJECT --trace-id TRACE_ID jq '[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]' trace_*/spans.json ``

1. Check if the retrieved context actually contained the answer
2. Add grounding instructions to the system prompt: "Only use information from the provided context. If the answer is not in the context, say so."

Troubleshooting

| Problem | Solution | |---------|----------| | ax: command not found | See references/ax-setup.md | | No profile found | No profile is configured. See references/ax-profiles.md to create one. | | No input_messages on span | Check span kind -- Chain/Agent spans store prompts on child LLM spans, not on themselves | | Prompt template is null | Not all instrumentations emit prompt_template. Use input_messages or input.value instead | | Variables lost after optimization | Verify the revised prompt preserves all {var} placeholders from the original | | Optimization makes things worse | Check for overfitting -- the meta-prompt may have memorized test data. Ensure few-shot examples are synthetic | | No eval/annotation columns | Run evaluations first (via Arize UI or SDK), then re-export | | Experiment output column not found | The column name is {experiment_name}.output -- check exact experiment name via ax experiments get | | jq errors on span JSON | Ensure you're targeting the correct file path (e.g., trace_*/spans.json) |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Arize-ai
• Source: Arize-ai/arize-skills
• License: MIT

Install and usage instructions live in the source repository linked above.