Hallucination Guard

hallucination-guard

Purpose

Post-generation validation that verifies AI-generated code references actually exist. Catches the 42% of AI code that contains hallucinated imports, non-existent packages, phantom functions, and incorrect API signatures. Also defends against "slopsquatting" — where attackers register package names that AI commonly hallucinates.

Triggers

• Called by cook after code generation, before commit
• Called by fix after applying fixes
• Called by preflight as import verification sub-check
• Called by review during code review
• Auto-trigger: when new import statements are added to codebase

Calls (outbound)

Exception: L3→L3 coordination

• research (L3): verify package existence on npm/pypi

Called By (inbound)

• cook (L1): after code generation, before commit
• fix (L2): after applying fixes
• preflight (L2): import verification sub-check
• review (L2): during code review
• db (L2): verify SQL syntax and ORM method calls are real
• review-intake (L2): verify imports in code submitted for review
• skill-forge (L2): verify imports in newly generated skill code
• adversary (L2): verify APIs/packages in plan actually exist

Execution

Step 1 — Extract imports

Use Grep to find all import/require/use statements in changed files:

Grep pattern: ^(import|require|use|from)\s
Files: changed files passed as input
Output mode: content

Collect every imported module name and file path. Separate into:

• Internal imports (start with ./, ../, @/, ~/)
• External packages (bare module names)

Step 2 — Verify internal imports

For each internal import path, use Glob to confirm the file exists in the codebase.

Glob pattern: .*   (try .ts, .tsx, .js, .jsx, .py, .rs etc.)

If Glob returns no results → mark as BLOCK (file does not exist).

Also use Grep to verify that the specific exported name (function/class/const) exists in the resolved file:

Grep pattern: export (function|class|const|default) 
File: resolved file path

If export not found → mark as WARN (symbol may not be exported).

Step 3 — Verify external packages (Dependency Check Before Import)

> From taste-skill (Leonxlnx/taste-skill, 3.4k★): "Before importing ANY 3rd party lib, check package.json."

Use Read on the project's dependency manifest to confirm each external package is listed:

• JavaScript/TypeScript: package.json → check dependencies and devDependencies
• Python: requirements.txt or pyproject.toml → [project.dependencies] and [project.optional-dependencies]
• Rust: Cargo.toml → [dependencies] and [dev-dependencies]

Pre-import gate (BEFORE writing import statements, not just after):

1. If the agent is ABOUT to import a package → check manifest FIRST
2. If package is NOT in manifest → output install command before writing the import:

`` ⚠ Package '' not in dependencies. Install first: npm install # JS/TS pip install # Python cargo add # Rust ``

1. If package IS in manifest → proceed with import

Post-import verification (after code is written):

• If package is not listed in the manifest → mark as BLOCK (phantom dependency)
• If package is listed but not installed (no lockfile entry) → mark as WARN (not yet installed)

Also check for typosquatting: if package name has edit distance ≤ 2 from a known popular package (axios/axois, lodash/lodahs, react/recat), mark as SUSPICIOUS.

Step 3.5 — Slopsquatting Registry Verification

Any NEW package added to the manifest (not previously in the lockfile) MUST be verified against the actual registry. AI agents hallucinate package names at high rates. A package that doesn't exist on npm/PyPI/crates.io = supply chain risk.

For each NEW external package (present in manifest but absent from lockfile):

3.5a. Registry existence check:

JavaScript: Bash: npm view  version 2>/dev/null
Python:     Bash: pip index versions  2>/dev/null
Rust:       Bash: cargo search  --limit 1 2>/dev/null

If command returns empty/error → BLOCK (package does not exist on registry — likely hallucinated name).

3.5b. Popularity check (slopsquatting defense):

JavaScript: Bash: npm view  'dist-tags.latest' 'time.modified' 2>/dev/null
→ If last modified > 2 years ago AND weekly downloads < 100: SUSPICIOUS
Python:     Use rune:research to check PyPI page for download stats

Low-popularity packages with names similar to popular ones = SUSPICIOUS (potential slopsquatting attack).

3.5c. Known slopsquatting patterns:

Popular Package → Common AI Hallucination
axios           → axois, axio, axioss
lodash          → lodahs, loadash, lo-dash
express         → expresss, express-js
react-router    → react-routes, react-routing
python-dotenv   → dotenv (wrong package in Python context)

Flag any match with edit distance ≤ 2 from these known pairs.

Step 4 — Verify API calls

For any API endpoint or SDK method call found in the diff, use rune:docs-seeker (Context7) to confirm:

• The method/function exists in the library's documented API
• The parameter signature matches usage in code

Mark unverifiable API calls as WARN (cannot confirm without docs).

Step 5 — Report

Emit the report in the Output Format below. If any BLOCK items exist, return status BLOCK to the calling skill to halt commit/deploy.

Check Types

INTERNAL    — file exists, function/class exists, signature matches
EXTERNAL    — package exists on registry, version is valid
API         — endpoint pattern valid, method correct
TYPE        — assertion matches actual type
SUSPICIOUS  — package name similar to popular package (slopsquatting)

Output Format

## Hallucination Guard Report
- **Status**: PASS | WARN | BLOCK
- **References Checked**: [count]
- **Verified**: [count] | **Unverified**: [count] | **Suspicious**: [count]

### BLOCK (hallucination detected)
- `import { formatDate } from 'date-utils'` — Package 'date-utils' not found on npm. Did you mean 'date-fns'?
- `import { useAuth } from '@/hooks/useAuth'` — File '@/hooks/useAuth' does not exist

### WARN (verify manually)
- `import { newFunction } from 'popular-lib'` — Function 'newFunction' not found in popular-lib@3.2.0 exports

### SUSPICIOUS (potential slopsquatting)
- `import axios from 'axois'` — Typo? Similar to popular package 'axios'

### Verified
- 12/15 references verified successfully

Constraints

1. MUST verify every import against actual installed packages — not just check if name looks reasonable
2. MUST verify API signatures against docs — not assume from function name
3. MUST report BLOCK verdict with specific evidence — never "looks suspicious"
4. MUST NOT say "no hallucinations found" without listing what was checked

Sharp Edges

Known failure modes for this skill. Check these before declaring done.

| Failure Mode | Severity | Mitigation | |---|---|---| | Declaring "no hallucinations found" without listing what was checked | CRITICAL | Constraint 4 blocks this — always list verified count vs total | | Marking phantom package (not in manifest) as WARN instead of BLOCK | HIGH | Unlisted package in manifest = BLOCK — not installed = won't run | | Missing typosquatting check on external packages | MEDIUM | Edit distance ≤2 check is mandatory — check every external package name | | Only checking package name, not the specific exported symbol | MEDIUM | Step 2: verify the specific function/class is exported, not just the file exists | | Skipping registry verification for new packages | CRITICAL | Step 3.5 HARD-GATE: new packages MUST be verified against actual registry | | AI-hallucinated package name passes because it "sounds right" | HIGH | Slopsquatting defense: check registry existence, not name plausibility | | Low-popularity package with similar name to popular one not flagged | HIGH | Popularity check catches slopsquatting attacks on newly registered packages |

Done When

• All imports extracted from changed files (internal + external separated)
• Internal imports: file existence AND symbol export verified
• External packages: manifest presence checked for every package
• Suspicious package names flagged (edit distance ≤2 from popular packages)
• API signatures checked via docs-seeker for new SDK/library calls
• Hallucination Guard Report emitted with PASS/WARN/BLOCK and verified count

Cost Profile

~500-1500 tokens input, ~200-500 tokens output. Haiku for speed — this runs frequently as a sub-check.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Rune-kit
• Source: Rune-kit/rune
• License: MIT

Install and usage instructions live in the source repository linked above.