# Coverage Verified Rag Indexer

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-fazalrshah-claude-skills-coverage-verified-rag-indexer`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [fazalrshah](https://agentstack.voostack.com/s/fazalrshah)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [fazalrshah](https://github.com/fazalrshah)
- **Source:** https://github.com/fazalrshah/claude-skills/tree/main/coverage-verified-rag-indexer

## Install

```sh
agentstack add skill-fazalrshah-claude-skills-coverage-verified-rag-indexer
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Coverage-Verified RAG Indexer

Most RAG indexers fail silently — a crash mid-batch, a dropped insert, a duplicate run — and you only find
out when answers are bad. This pattern makes failures **loud and impossible to miss.**

## Pipeline (block-structured stages)
`resolve_file → extract (OCR) → chunk → embed (batched+retry) → upsert → verify_coverage → mark_complete`

Each stage raises `StageError(stage, msg)`. On failure, record **which stage** + the message into the job
row (`error_stage`, `error_msg`). No more "it just hangs" — you always know where it broke.

## The key idea: coverage verification
After insert + flush, query the vector DB for `count(doc, version)` and assert it **equals the number of
chunks**. Mismatch → the job **FAILS** (it never marks `complete`, so a corrupt index is never served):
```python
stored = count_in_vectordb(doc, version)
if stored != len(chunks):
    raise StageError("verify", f"coverage mismatch: stored {stored} of {len(chunks)}")
```
This single check catches dropped chunks, partial inserts, and duplication (e.g. a 2× count from two workers).

## Extraction — handle real documents
Use an OCR-capable parser (e.g. **Docling**) so scanned/image PDFs and `.docx` tables don't yield empty text.
If extraction returns near-nothing, fail at `extract` (flag `needs_ocr`) instead of embedding a blank doc.

## Chunking — measure in TOKENS, not characters
Chunk with the **embedding model's own tokenizer**, target **500–1000 tokens** (e.g. 800) with ~20% overlap,
or use a structure-aware chunker (Docling HybridChunker) that respects headings. Char-based chunking lies:
1200 chars ≈ 300 tokens, far below target.

## Concurrency — claim jobs atomically
A polling worker MUST claim jobs atomically, or two instances grab the same job and double-insert (the classic
2× coverage failure). With Postgres:
```sql
UPDATE jobs SET status='processing'
WHERE id = (SELECT id FROM jobs WHERE status='queued' ORDER BY id LIMIT 1 FOR UPDATE SKIP LOCKED)
RETURNING ...;
```

## Operational gotchas
- **Auto-ingest** by watching a folder + content hash (only re-index new/changed files); dedup against a
  **clearable** state table, NOT an append-only audit ledger (or you can't reset).
- **Run one instance** under launchd/systemd (the atomic claim is the real safety net).
- **Derive doc identity from the real filename** (slug), not a hand-typed map — spacing/casing mismatches bite.
- Vector DB query windows are capped (e.g. Milvus offset+limit ≤ 16384) — count via iteration for big corpora.

## Result
Every indexed doc is provably complete (count-verified), failures name their stage, and duplication can't slip
through. That's the difference between a demo RAG and one you can trust.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [fazalrshah](https://github.com/fazalrshah)
- **Source:** [fazalrshah/claude-skills](https://github.com/fazalrshah/claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-fazalrshah-claude-skills-coverage-verified-rag-indexer
- Seller: https://agentstack.voostack.com/s/fazalrshah
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.