Self Refine Reflection

Self-Refine Reflection Skill

> An AI agent skill for systematic self-reflection and iterative output refinement. > Based on Madaan et al. (2023), Shinn et al. (2023), and Andrew Ng's Reflection design pattern.

Platform Auto-Detection

At skill load time, detect your runtime environment and adjust capabilities:

| Capability | How to Check | Fallback |-----------|-------------|----------| | File system | Can you read references/reflection-templates.md? | Use the inline templates below instead | Persistent memory | Can you write to memory/? | Store reflection notes in conversation context only | Long context | Is your context window > 32K tokens? | Cap at Level 2 (skip adversarial review) | Tool access | Can you call external tools? | Use mental verification only

Detection rules:

• If you can read this file's references/ directory → full mode (all levels + memory)
• If you can read files but not write → full levels, in-conversation memory only
• If you cannot read files at all → use inline templates (copied below), cap at Level 2
• If context is limited (< 8K usable) → default to Level 1, max Level 2

This means every platform gets the best possible experience automatically — no manual configuration needed.

Inline Reflection Templates (for environments without file access)

If you cannot read references/reflection-templates.md, use these directly:

Level 1 Internal Prompt

Review drafted response: (1) Did I answer everything? (2) Logic gaps? (3) Can I cut 20%? (4) User can act on this? Fix → Deliver.

Level 2 Internal Prompt

For each dimension (Logic / Facts / Completeness / Conciseness / Actionability / Consistency):
  - Quote exact sentences with issues
  - Rate: Critical / Minor / Pass
Fix ALL Critical. Fix Minor if straightforward. Re-read once. Deliver.

Level 3 Internal Prompt

Round 1: Level 2 review.
Round 2: "If a domain expert attacked this, what would they target?" Fix valid attacks.
Round 3: Did fixes introduce new issues? If stable → deliver. If not → fix and deliver.

When to Activate

Activate when you are about to deliver a final response to the user, after you've gathered all information and formed your answer. Reflection happens before delivery, not instead of work.

Core Loop: GENERATE → CRITIQUE → REFINE → CHECK

1. GENERATE  — Produce your initial response as normal
2. CRITIQUE  — Apply the reflection framework (depth-dependent)
3. REFINE    — Fix every issue found in CRITIQUE
4. CHECK     — If issues remain AND within budget, loop to step 2
              — If clear OR budget exhausted, deliver

Source: Madaan et al., "Self-Refine: Iterative Refinement with Self-Feedback" (2023) — the same LLM acts as generator, critic, and refiner.

---

Reflection Dimensions

Every critique examines the response across these dimensions. Not all apply to every response — skip irrelevant ones silently.

| # | Dimension | What to Check | Signal Words | |---|-----------|---------------|--------------| | 1 | Logical Completeness | Does the reasoning chain have gaps? Are conclusions supported by premises? | "therefore" without prior evidence, jumps in logic | | 2 | Factual Accuracy | Are there unverified assertions? Claims that could be wrong? | Specific numbers, dates, names, "everyone knows..." | | 3 | Response Completeness | Did I address every part of the user's question? Are there sub-questions I ignored? | Multiple questions in user message, implicit needs | | 4 | Conciseness | Is there redundancy? Can paragraphs be merged? Are there filler phrases? | "It's important to note that", "In conclusion", repeated points | | 5 | Actionability | Can the user act on this immediately? Or do they need to ask follow-ups? | Vague advice without steps, missing specifics | | 6 | Internal Consistency | Do any parts of my response contradict each other? | Conflicting recommendations, contradictory statements |

---

Reflection Depth Levels

The depth is determined by task complexity, not user preference. The agent auto-selects.

Level 0: Quick Scan (Skip formal reflection)

Trigger: Simple factual lookups, greetings, trivial questions, single-sentence answers.

Action: Do nothing extra. Just respond. Cost: 0 additional tokens.

Examples: "What time is it?", "Thanks", simple formatting requests.

Level 1: Standard Review

Trigger: Medium-complexity tasks — explanations, how-to guides, code snippets, multi-paragraph responses.

Action: One pass through all 6 dimensions mentally. Fix issues. Deliver.

Budget: 1 refinement round. ~15-20% overhead on response tokens.

Internal process:

After drafting your response, ask yourself:
- Did I answer everything they asked?
- Any logical gaps or contradictions?
- Can I cut 20% of the words without losing meaning?
- Would the user know exactly what to do next?
Fix → Deliver.

Level 2: Deep Audit

Trigger: Complex tasks — technical architectures, multi-step plans, research summaries, anything with 5+ distinct claims.

Action: Explicit dimension-by-dimension review. One full refinement round with documented issues.

Budget: Up to 2 refinement rounds. ~30% overhead.

Internal process:

Draft response.
For each dimension (1-6):
  - Identify specific issues (quote the exact sentence)
  - Rate severity: Critical / Minor / Pass
If any Critical: refine entire response, then re-check
If only Minor: fix inline, deliver

Level 3: Adversarial Review

Trigger: High-stakes tasks — production code, security decisions, medical/legal adjacent, public-facing content, or user explicitly requests thorough review.

Action: Full audit PLUS an adversarial pass where you actively try to find the weakest point in your response.

Budget: Up to 3 refinement rounds. ~50% overhead.

Internal process:

Draft response.
Round 1: Standard dimension-by-dimension review.
Round 2: Adversarial pass — "If I wanted to prove this response wrong, 
         what would I attack?" Check those attack vectors.
Round 3 (if needed): Verify fixes from Round 2 didn't introduce new issues.

---

Convergence Rules

Based on the empirical finding from Madaan et al. that "most gains are in the initial iterations":

1. Maximum 3 refinement rounds regardless of depth level.
2. Stop early if no Critical or Minor issues found in a round.
3. Stop early if the changes between rounds are purely stylistic (no substantive improvement).
4. Diminishing returns rule: If Round N fixes fewer issues than Round N-1, stop after N.

Anti-pattern — DO NOT:

• Refine forever trying to reach perfection
• Make changes just to justify another round
• Re-introduce previously fixed issues

---

Cost Control Strategy

| Depth | Max Rounds | Approx. Token Overhead | When to Use | |-------|-----------|----------------------|-------------| | Level 0 | 0 | 0% | Simple Q&A | | Level 1 | 1 | ~15% | Most conversations | | Level 2 | 2 | ~30% | Complex technical | | Level 3 | 3 | ~50% | High-stakes only |

Principle: Reflection should cost less than the cost of delivering a wrong answer. For low-stakes responses, skip reflection entirely.

---

Trigger Conditions Summary

Auto-Trigger (always on)

• Response exceeds 3 paragraphs → at least Level 1
• Response makes 3+ factual claims → at least Level 1
• Response includes code → at least Level 1 (check logic + actionability)

Skip Reflection

• User is in a hurry (explicit: "quick", "brief", "just tell me")
• Response is under 2 sentences
• Pure social/chat exchange

User Manual Trigger

• User says "think carefully", "double-check", "make sure this is right" → Level 2+
• User says "this is important", "critical", "production" → Level 3
• User says "reflect" or "self-refine" → Level 2+

---

Output Format

Reflection is internal — the user should not see the raw critique. However:

After refinement, you MAY append a subtle note:

Level 1: No note (keep it invisible). Level 2: Optionally: _(response refined via self-review)_ Level 3: Optionally: _(refined through [N]-round adversarial self-review)_

NEVER:

• Show the actual critique/feedback text to the user
• Make the note prominent or distracting
• Add notes for Level 0 or Level 1 responses

---

Reflexion Memory (Cross-Session Learning)

Inspired by Shinn et al. (2023) — Reflexion stores verbal self-reflections in persistent memory for future tasks.

When to use: After completing a Level 2 or Level 3 reflection where you discovered a significant pattern in your own errors (e.g., "I tend to forget edge cases in X", "I consistently over-explain Y").

How to use: Write a brief note to memory/ capturing the pattern:

Self-reflection: When writing about [topic], I tend to [pattern]. 
Fix: [specific behavior change]. 
Date: [today].

This creates a growing corpus of self-corrections that improve future responses.

---

Relationship to Other Reflection Techniques

This skill synthesizes multiple academic approaches:

| Technique | Source | What We Borrow | |-----------|--------|---------------| | Self-Refine | Madaan et al. 2023 | Core GENERATE→CRITIQUE→REFINE loop | | Reflexion | Shinn et al. 2023 | Persistent memory of self-reflections | | Chain-of-Verification | Dhuliawala et al. 2023 | Verification questions for factual claims (Level 2-3) | | Self-Calibration | Kadavath et al. 2022 | Confidence assessment of own outputs | | Andrew Ng's Reflection | Ng 2024 | Design pattern: agent critiques and improves its own output iteratively | | CRITIC | Gou et al. 2023 | Tool-interactive critiquing (use tools to verify when possible) |

---

Quick Reference Card

REFLECT? ──→ Simple/trivial? ──→ NO → Just respond
    │
    YES
    │
    ├─ Medium complexity → LEVEL 1 (1 round, mental check)
    ├─ Complex / multi-claim → LEVEL 2 (2 rounds, explicit review)
    └─ High-stakes / user requested → LEVEL 3 (3 rounds, adversarial)
    
For each round:
  1. Check 6 dimensions
  2. Fix all issues
  3. Converged? → Deliver
  4. Budget left? → Next round
  5. Budget exhausted → Deliver current version

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Thomaszhou22
• Source: Thomaszhou22/self-refine-skill
• License: MIT

Install and usage instructions live in the source repository linked above.