Token Doctor

Token-Doctor

Measure and cut the token cost of the files that load into every prompt.

When to run

• A CLAUDE.md / AGENTS.md / memory file has grown large
• Prompts feel expensive or context fills up fast
• Before committing context files (keep them lean)
• As a CI gate so no context file silently bloats

Input

A single file, or a directory (scans *.md, *.txt, CLAUDE.md, AGENTS.md, llms.txt, etc.; skips code, node_modules, .git).

How to run

The engine lives in scripts/ (Python 3.8+ stdlib, zero dependencies).

# Audit (read-only) — see token cost + savings
python3 -m scripts /path/to/project

# Audit a single file
python3 -m scripts CLAUDE.md

# Apply the reductions (keeps a .bak backup per file)
python3 -m scripts CLAUDE.md --apply

# CI gate: fail if any context file exceeds a token budget
python3 -m scripts /path/to/project --fail-over 1500

# Machine-readable
python3 -m scripts /path/to/project --format json --out audit.json

What it does

1. Measures estimated tokens per file (offline heuristic; % saved is reliable).
2. Splits each file into prose vs protected regions (code, inline code, URLs,

paths, tables, frontmatter).

1. Reduces ONLY prose: drops filler, shortens verbose phrases, collapses blank

bloat, removes duplicate adjacent lines.

1. Reports before/after tokens, % saved, biggest-savings files, and waste findings.
2. Applies safely on --apply (writes .bak first).

Guarantees

• Never edits fenced code, inline code, URLs, markdown links, file paths,

env vars, tables, or YAML frontmatter — these survive byte-for-byte.

• Read-only by default. --apply always writes a .bak backup.
• Meaning is preserved; only token-wasteful wording is trimmed.

Waste codes

| Code | Severity | Meaning | |------|----------|---------| | LARGE_FILE | 🔴 | File exceeds the token budget (taxes every prompt) | | FILLER_HEAVY | 🟡 | Many removable filler words | | VERBOSE_PHRASES | 🟡 | Wordy phrases that have short equivalents | | DUPLICATE_LINES | 🟡 | Repeated adjacent lines | | HUGE_CODE_BLOCK | 🔵 | A code/log block dominates the file | | BLANK_BLOAT | 🔵 | Runs of blank lines |

Workflow for the agent

1. Run the audit on the user's project or file → read the report.
2. Summarize total % savings and the biggest-savings files.
3. Offer --apply (explain the .bak backups) before changing anything.
4. For LARGEFILE / HUGECODE_BLOCK, suggest moving detail into references/

loaded on demand rather than the always-loaded file.

1. Re-run to confirm the new token total.

See references/methodology.md for the estimator and references/preservation.md for exactly what is never touched.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: enmr10
• Source: enmr10/token-doctor
• License: MIT

Install and usage instructions live in the source repository linked above.