# Rag Evaluator

> Evaluate RAG pipeline quality across faithfulness, relevance, and hallucination metrics. Use when user asks to test, benchmark, or improve a RAG system, or when RAG outputs look wrong.

- **Type:** Skill
- **Install:** `agentstack add skill-chandrudp29-skillhub-rag-evaluator`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [chandrudp29](https://agentstack.voostack.com/s/chandrudp29)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [chandrudp29](https://github.com/chandrudp29)
- **Source:** https://github.com/chandrudp29/skillhub/tree/main/skills/rag-evaluator
- **Website:** https://pypi.org/project/skillhub-ai/

## Install

```sh
agentstack add skill-chandrudp29-skillhub-rag-evaluator
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# RAG Evaluator

A systematic evaluation framework for Retrieval-Augmented Generation pipelines. Finds where your RAG breaks before your users do.

## When to Use

- "Is my RAG system giving correct answers?"
- "My RAG is hallucinating — help me find where"
- "Evaluate my retrieval quality"
- "My RAG answers are vague / not using the documents"
- Before deploying a RAG system to production

## The Three Failure Modes

**Retrieval failure**: Right question, wrong chunks retrieved. The answer exists in the corpus but wasn't found.

**Generation failure**: Right chunks retrieved, but the LLM ignored them, hallucinated, or answered a different question.

**Context failure**: Chunks retrieved but they're too long, poorly formatted, or overlap confusingly — LLM gets lost.

Most RAG problems are retrieval failures. Check retrieval first.

## Evaluation Workflow

### Step 1 — Build a test set

Minimum 20 question-answer pairs. Include:
- Questions whose answers are clearly in the corpus (should always pass)
- Questions whose answers are NOT in the corpus (should return "I don't know")
- Multi-hop questions requiring synthesis across chunks
- Edge cases: short answers, numerical facts, recent events

Without a test set, you're evaluating by vibes. Build it first.

### Step 2 — Evaluate retrieval independently

For each test question:
1. Run retrieval only (no generation)
2. Check: is the answer in the top-k chunks?
3. Check: is the most relevant chunk ranked #1?

Compute:
- **Recall@k**: % of questions where answer chunk is in top-k
- **MRR** (Mean Reciprocal Rank): average of 1/rank of first relevant chunk

Target: Recall@5 > 0.85 before touching generation.

### Step 3 — Evaluate generation independently

Feed the oracle chunks (the correct ones) directly to the LLM. Ask it to answer using only those chunks. If it still fails, the problem is generation, not retrieval.

### Step 4 — Run the full pipeline evaluation

For each question in test set:

**Faithfulness** — Does the answer only use information from the retrieved chunks?
- LLM-as-judge: "Given these chunks and this answer, does the answer contain any claims not supported by the chunks? Yes/No and cite specific claims."
- Target: > 0.90

**Answer Relevance** — Does the answer actually address the question?
- LLM-as-judge: "On a scale of 1–5, how well does this answer address the question? Question: X, Answer: Y"
- Target: average > 4.0

**Context Precision** — Are the retrieved chunks actually useful?
- Check: what % of retrieved chunks contain information used in the answer?
- Low precision = noise in context = confused LLM

### Step 5 — Diagnose and fix

| Symptom | Root cause | Fix |
|---|---|---|
| Recall@5 0.85)
  Recall@10:   0.XX
  MRR:         0.XX  (target: >0.70)

GENERATION
  Faithfulness:      0.XX  (target: >0.90)
  Answer Relevance:  0.XX  (target: >4.0)
  Context Precision: 0.XX

FAILURE ANALYSIS
  [N] retrieval failures — top categories:
  [N] generation failures — top categories:

RECOMMENDATIONS (priority order)
  1. ...
  2. ...
```

## Verification Checklist

- [ ] Test set has ≥20 questions
- [ ] Test set includes "not in corpus" questions
- [ ] Retrieval evaluated independently before full pipeline
- [ ] Each metric has a numeric value (not "looks good")
- [ ] At least 3 failure examples analyzed for root cause
- [ ] Recommendations are specific and actionable

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [chandrudp29](https://github.com/chandrudp29)
- **Source:** [chandrudp29/skillhub](https://github.com/chandrudp29/skillhub)
- **License:** MIT
- **Homepage:** https://pypi.org/project/skillhub-ai/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-chandrudp29-skillhub-rag-evaluator
- Seller: https://agentstack.voostack.com/s/chandrudp29
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.