Evidence Ranking

Evidence Ranking

Score and rank engineering evidence using the tier system from references/evidence-quality-tiers.md. This skill provides transparent, auditable evidence scoring — never a black-box "relevance" score.

Workflow

flowchart LR
    Input["Evidence Items"] --> Identify["Identify Sources"]
    Identify --> Score["Score on Criteria"]
    Score --> Tier["Assign Tier"]
    Tier --> Conflicts["Flag Conflicts"]
    Conflicts --> Output["Ranked Table + Rationale"]

Invocation

/evidence-ranking 

Optionally provide a list of sources to rank. If none provided, use scholarly-research to discover sources first, then rank them.

Methodology

Increment Checklist (complete before Output)

• [ ] At least 5 evidence items identified
• [ ] Each item scored on all 5 criteria (source tier, methodology, citation authority, reproducibility, recency)
• [ ] Tier assigned to each item (1-4)
• [ ] Conflicts between sources identified
• [ ] Evidence strength stated with justification

If any checkbox is unchecked, complete the missing work before writing output.

1. Identify evidence items — from user input or scholarly-research discovery. Each item needs: title, authors, year, venue, DOI/URL, and a brief summary of the claim it supports.

1. Score each source using the criteria in references/evidence-ranking-methodology.md.

1. Assign tier and confidence — see references/evidence-ranking-methodology.md.

1. Flag conflicts — when sources disagree, identify the conflict, note which is newer, which is jurisdiction-specific, and what a decision-maker should weigh.

Output

Inline Summary (chat response)

• Research question
• Top 3 sources by tier and score
• Conflicts identified (if Any)
• Overall evidence strength for the question

Full Ranking (saved to disk)

Save to outputs/evidence-ranking/.md:

# Evidence Ranking: 

## Question

## Evidence Items

| # | Source | Year | Venue | Tier | Score | Key Claim | Status |
|---|--------|------|-------|------|-------|-----------|--------|
| 1 | [Author](DOI) | 2024 | Journal | 1 | 9 |  | verified |
| 2 | [Author](DOI) | 2023 | Conference | 3 | 5 |  | verified |

## Tier Distribution
- **Tier 1:**  sources
- **Tier 2:**  sources
- **Tier 3:**  sources
- **Tier 4:**  sources (rejected as primary)

## Scoring Rationale

### Source 1: [title]
- **Tier:** 1 | **Score:** 9/10
- **Why:** Standard/code provision, cited by [standard], experimental validation
- **Limitations:** 

## Conflicts

### Conflict: 
- **Source A** says:  [Tier 1]
- **Source B** says:  [Tier 2]
- **Resolution:** A is newer and cited by standards; B is jurisdiction-limited

## Evidence Strength
- **Overall:** Strong / Moderate / Weak
- **Recommendation:** 

Quality Gate (mandatory before returning)

Self-check the output. If any check fails, retry once with feedback:

1. Source count — at least 5 sources ranked. If < 5, expand search.
2. Tier distribution — at least 2 tiers represented. If all Tier 4, re-search with better terms.
3. Scoring rationale — each source has explicit scoring rationale. If missing, add reasoning.
4. Conflict identification — conflicts between sources are flagged. If none found but sources disagree, re-analyze.
5. Evidence strength — overall strength is stated with justification. If missing, add reasoning.

Retry logic: If quality gate fails, re-run the ranking once with the specific failure as feedback. If it fails again, return the best output with a Weak evidence strength and list the unresolved issues.

Scope and Boundaries

• This skill scores evidence — it does NOT select the "best" source. Human engineering judgment decides.
• Research-only, not for final engineering sign-off.
• Never fabricate scores. If a source cannot be assessed, mark it unverified.
• Tier 4 sources are flagged as rejected for primary use but may support context.
• Evidence quality: see references/evidence-quality-tiers.md.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: adeerkhan
• Source: adeerkhan/vitruvius
• License: MIT

Install and usage instructions live in the source repository linked above.