Langsmith

LangSmith Tool

Use this tool to debug AI chains, inspect past runs, manage datasets, and analyze token costs in LangSmith.

Prerequisites

uv tool install langsmith-cli
/plugin marketplace add gigaverse-app/langsmith-cli

See [Installation Guide](references/installation.md) if install fails or for alternative methods.

---

🚨 MANDATORY Rules — Read These First

1. Always use --json

ALWAYS pass --json for machine-readable output. It can appear before or after the subcommand, but putting it first is still the clearest convention.

# ✅ CORRECT
langsmith-cli --json runs list --project my-project --limit 5

# ❌ WRONG — Rich table output, cannot be parsed
langsmith-cli runs list --project my-project --limit 5

2. Always use --output for data extraction — never shell redirection

# ✅ CORRECT — atomic write, errors visible, non-zero exit on failure
langsmith-cli --json runs list --project my-project --output runs.jsonl
langsmith-cli runs list --project my-project --format json --output runs.json
python3 -c "import json; runs = [json.loads(l) for l in open('runs.jsonl')]"

# ❌ WRONG — errors go to stderr silently, you get empty/corrupt file
langsmith-cli --json runs list --project my-project > runs.json

# ❌ WRONG — heredoc overrides pipe stdin, python3 reads empty stdin
langsmith-cli --json runs get  --fields outputs | python3  --fields inputs,outputs,error

---

Quick Command Reference

| Task | Command | |------|---------| | List recent runs | langsmith-cli --json runs list --project --limit 10 --fields id,name,status | | Get a single run | langsmith-cli --json runs get --fields inputs,outputs,error | | Get run + child outputs | langsmith-cli --json runs get --follow-children --fields id,name,inputs,outputs | | Get latest run | langsmith-cli --json runs get-latest --project --fields inputs,outputs | | Get latest error | langsmith-cli --json runs get-latest --project --failed --fields id,name,error | | Server-side search | langsmith-cli --json runs search "pattern" --fields id,name,status --limit 20 | | Scoped content search | langsmith-cli --json runs search "pattern" --in outputs --fields id,name,outputs --limit 20 | | Search cached runs | langsmith-cli --json runs cache grep "pattern" -E --grep-in outputs --project --fields id,name,outputs | | Download cache | langsmith-cli --json runs cache download --project --last 7d | | List cache | langsmith-cli --json runs cache list --fields project_name,run_count,path | | Discover cache schema | langsmith-cli --json runs cache schema --project --include outputs | | Analyze token costs | langsmith-cli --json runs usage --from-cache --breakdown model --active-only | | List projects | langsmith-cli --json projects list --name-pattern "dev/*" --fields name | | Count runs | langsmith-cli --json runs list --project --count | | Run stats | langsmith-cli --json runs stats --project | | Stratified run sample | langsmith-cli --json runs sample --project --stratify-by tag: --values a,b --fields id,name,stratum | | List datasets | langsmith-cli --json datasets list --fields id,name | | List prompts | langsmith-cli --json prompts list --fields repo_handle,description | | List feedback for a run | langsmith-cli --json feedback list --run-id --fields id,key,score | | Create feedback | langsmith-cli --json feedback create --key correctness --score 0.9 | | List annotation queues | langsmith-cli --json annotation-queues list --fields id,name | | Get annotation queue | langsmith-cli --json annotation-queues get | | View experiment results | langsmith-cli --json experiments results | | Open run in browser | Construct URL manually — see LangSmith URLs section below |

---

📚 When to Read Reference Files

When your task matches one of the sections below, you MUST load that reference file before proceeding — don't load them speculatively for unrelated tasks.

→ Read [references/runs.md](references/runs.md) when:

• You need the full flag list for runs list, runs get, runs get-latest, runs search, runs sample, runs analyze, runs tags, runs fields, runs export
• You need to use --trace-filter, --tree-filter, --sort-by, --roots, --run-type, --tag, --model, --min-latency, --max-latency
• You need to filter runs by metadata: --metadata key=value (supports wildcards key=val* and regex key=/pattern/)
• You need to paginate, export to files, or watch live runs

→ Read [references/search.md](references/search.md) when:

• You need to choose between --query (server-side, fast, first 250 chars) vs --grep (client-side, all content, regex)
• You need to write FQL filter expressions (eq, gt, has, and, search, metadata_key/metadata_value)
• You need to search by content with regex, Hebrew/Unicode characters, or multi-field patterns
• You need common search+filter patterns (find errors, filter by tag, combine filters)
• You want to avoid writing inline scripts — the CLI often handles it natively

→ Read [references/cost-analysis.md](references/cost-analysis.md) when:

• You need to analyze token usage or costs for a project or event window
• You need to break down costs by model, provider, or gateway
• You need to find missing model pricing and apply a pricing YAML
• You need hourly/daily cost distribution or cost attribution for a channel/community
• You need to use runs usage, runs pricing, runs cache download + --from-cache
• You need the Key Flags for offline analysis (--from-cache, --group-by, --breakdown, --apply-pricing)

→ Read [references/projects.md](references/projects.md) when:

• You need to create, update, or delete projects
• You need to filter projects by name pattern/regex across environments

→ Read [references/datasets.md](references/datasets.md) and [references/examples.md](references/examples.md) when:

• You need to create or manage evaluation datasets
• You need to add, update, or delete examples
• You need to push JSONL to a dataset or create examples from runs

→ Read [references/prompts.md](references/prompts.md) when:

• You need to pull, push, or version prompt templates
• You need to list prompt commits or compare versions
• Prompt commit listings support --fields, --count, --output, and --format json|csv|yaml

→ Use feedback commands when:

• You need to list, get, create, or delete feedback scores on runs
• Commands: feedback list [--run-id ] [--key ] [--limit N] [--fields a,b] [--count] [--output file.jsonl] [--format json|csv|yaml], feedback get [--fields a,b] [--output file.json], feedback create --key [--score N] [--comment ], feedback delete [--yes]

→ Use annotation-queues commands when:

• You need to manage human review queues (list, create, update, delete)
• Commands: annotation-queues list [--fields a,b] [--count] [--output file.jsonl] [--format json|csv|yaml], annotation-queues get [--fields a,b] [--output file.json], annotation-queues create [--description ], annotation-queues update [--name ] [--description ], annotation-queues delete [--yes]

→ Use experiments commands when:

• You need to view run stats and feedback scores for a named experiment (project)
• Commands: experiments results

→ Read [references/fql.md](references/fql.md) when:

• You need to write a complex --filter expression and want operator reference
• You need metadata_key/metadata_value filter syntax

→ Read [docs/examples.md](docs/examples.md) when:

• You want end-to-end workflow examples (debugging, dataset management, production monitoring)
• You want common patterns without having to piece together flags yourself
• You need to search for recognized entities in extraction chain outputs (e.g. find all runs where "Niklas" was recognized as a known entity in extracted_entities) — there's a complete recipe covering cache download, Python JSONL scanning, deduplication of sub-runs, and llm_recognition filtering

→ Read [references/cache-recipes.md](references/cache-recipes.md) when:

• You need to discover the nested structure of cached run data (inputs/outputs schema)
• You want to query cached JSONL data with Python one-liners or DuckDB SQL
• You need to extract, sort, or aggregate structured outputs from cached runs
• You need to find specific entities, values, or patterns in nested output fields

→ Read [references/troubleshooting.md](references/troubleshooting.md) when:

• CLI commands fail, return unexpected results, or produce authentication errors
• You see rate limit errors and want strategies to work around them

---

LangSmith URLs

runs open generates broken URLs. Build trace URLs manually using the project's id and tenant_id:

# Step 1: Get org ID (tenant_id) and project ID
langsmith-cli --json projects get "dev/my-project" --fields id,tenant_id

# Step 2: Build the URL
# https://smith.langchain.com/o/{tenant_id}/projects/p/{project_id}?peek={run_id}&peeked_trace={trace_id}

Example:

org_id = "b658ea18-0431-42c0-8d03-337d43fed8cf"   # tenant_id from projects get
proj_id = "730acc6c-ec97-4f08-915e-7d3f7f775300"  # id from projects get

url = f"https://smith.langchain.com/o/{org_id}/projects/p/{proj_id}?peek={run_id}&peeked_trace={trace_id}"

• peek = the specific run ID to open in the side panel
• peeked_trace = the trace (root run) ID it belongs to
• Both IDs come from your search results (run.id and run.trace_id)

---

Key Flags Cheat Sheet

# Multi-project matching
--project-pattern "prd/*"         # wildcard alias for --project-name-pattern
--project-name-pattern "prd/*"    # wildcard
--project-name-regex "^(prd|stg)" # regex

# Time windows (combinable)
--since 2026-01-15 --before 2026-01-29
--last 7d
--since 2026-01-15 --last 14d     # forward window

# Content search
--query "text"                    # server-side, fast, first ~250 chars only
--grep "pattern" --grep-regex --grep-in inputs,outputs  # client-side, all content

# Metadata filter (server-side, supports wildcards and regex)
--metadata channel_id=Gigaverse_Daily_Standup*
--metadata channel_id=/^Gigaverse/

# Reduce output size
--fields id,name,status,start_time
--roots                           # root traces only (cleaner)
--all-runs                        # include nested child runs
--limit 10 --fetch 500            # fetch 500 from API, return top 10 matches

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: gigaverse-app
• Source: gigaverse-app/langsmith-cli
• License: MIT

Install and usage instructions live in the source repository linked above.