Claude Code For Ai Engineers

Claude Code for AI Engineers

[](https://github.com/surpradhan/claude-code-for-ai-engineers-eval/actions/workflows/skill-eval.yml)

This is the open-source preview of Claude Code for AI Engineers, a methodology-first skill pack for engineers building RAG systems, multi-agent workflows, and MCP servers.

The preview includes two of six skills, one of three project templates, and one of five slash commands.

---

Verified by self-test

This pack has its own eval harness. The harness caught a real bug in the skills before this README went up: fix in 8f17007 (baseline gate enforcement). That's the result that matters; the rest is regression coverage.

Latest run (2026-06-20) on the preview skills:

• Triggering on rag-eval-harness (n=30): 15/15 should-fire prompts fired, 0/15 near-misses fired. Small N by design — this catches regressions on the description, not generalization. Prompts.
• Behavior (5 scenarios): two-variable comparison pushback, baseline required, fixed dataset required, adversarial-skip-the-gate, deterministic variables locked. All 9 assertions pass. The adversarial scenario explicitly asks Claude to skip the methodology gate. Claude held.
• Judge calibration: Haiku and Sonnet agreed on all 9 assertions in the 2026-06-14 calibration run, prior to switching to Haiku-only for production. Easy-case agreement, not robustness — the calibration value will rise as new scenarios are added. Haiku vs. human agreement: 1.000 (N=9).

Harness: claude-code-for-ai-engineers-eval. See its README for the latest scorecard.

What a skill firing looks like

You: "Scaffold a RAG benchmark comparing LangChain BM25 to LlamaIndex hybrid retrieval."

Claude (rag-eval-harness loaded): "Hold on — this varies two things at once: the framework AND the retriever. The delta you measure won't be interpretable. Two options: (1) narrow to one variable (e.g. LlamaIndex BM25 vs LlamaIndex hybrid), or (2) keep both but document the confound explicitly. Which one?"

That's the entire pitch. The skills refuse to scaffold the test that won't answer your question.

---

What's in this preview (open-source, MIT)

Two skills (skills/)

• rag-eval-harness — set up a controlled RAG evaluation harness with the variables locked. Enforces fixing the embedding model, chunker, vector store, and generator so comparisons mean something.
• agent-trace-debug — systematic debugging for multi-agent workflows from a captured trace. Refuses to propose fixes without one. Classifies the seven canonical failure modes.

One project template (claude_md_templates/)

• multi-agent-project.md — multi-agent systems (LangGraph / CrewAI / AutoGen / OpenAI Agents SDK)

One slash command (slash_commands/)

• /agent-debug — analyze a captured agent trace; identify failure mode; propose fix

What's in the full pack

Everything in this preview, plus:

Four more skills

• benchmark-scaffold — scaffold a reproducible benchmark project (configs, runners, results)
• mcp-server-bootstrap — scaffold a new MCP server in Python (FastMCP) or TypeScript
• eval-report-writer — generate a publication-quality evaluation report with mandatory methodology and "where this is honest" sections
• paper-reproduce — set up a paper reproduction scaffold with validation gates

Two more project templates

• rag-project.md — RAG projects (retrieval, eval, deployment)
• mcp-server-project.md — MCP server projects (tool schemas, transport, testing)

Four more slash commands

• /eval-run — run the configured evaluation with current changes
• /benchmark-new — scaffold a new architecture inside an existing benchmark project
• /mcp-scaffold — add a new tool to an existing MCP server
• /experiment-log — capture the current project state into the experiment log

---

How to use this preview

cp -r skills/* ~/.claude/skills/
cp -r slash_commands/* ~/.claude/commands/

Then open Claude Code in any project and try one of:

• "Compare two RAG retrieval architectures on HotpotQA" — rag-eval-harness activates and refuses if your comparison varies more than one thing at a time.
• "My LangGraph agent keeps calling the wrong tool, fix the prompt" — agent-trace-debug refuses to propose a fix without a captured trace and walks you through capturing one.

If those gates feel right, the full pack applies the same discipline to benchmarking, MCP server work, paper reproduction, and evaluation reporting.

---

Design principles

1. Fix what you can; vary one thing at a time.
2. Capture before you debug.
3. Reproducibility before novelty.
4. Honesty in reporting.

Each skill enforces at least one of these as a hard gate.

Support

If a skill misfires or you have questions, open an issue or DM me on LinkedIn.

---

Surabhi Pradhan publishes reproducible AI benchmarks — rag-benchmark, agent-workflow-comparison, rec-bench, forecasting-showdown — and maintains the open-source agent observability protocol agent-event-protocol. She writes on Medium about LLM internals and developer infrastructure.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: surpradhan
• Source: surpradhan/claude-code-for-ai-engineers
• License: MIT

Install and usage instructions live in the source repository linked above.