Audit First

Audit-First Protocol

For tasks where you must first find all the problems, and only then fix. Use when:

• An area is suspicious but the specific gap is unknown
• Need to make sure we're fixing the right problem, not a symptom
• Task sounds like "look at what's wrong in X" rather than "fix specific Y"
• Before an important deploy — a preventive audit

Difference from arch-first:

| | arch-first | audit-first | |---|---|---| | Purpose | Execute a complex task | Discover problems | | Phase 0 | Understand the task | Find all gaps | | Result | Decomposition into blocks | Gap table with priorities | | When to fix | Immediately per the plan | Only after table approval | | Launch | "Do X" | "Find what's wrong in X" |

Input

A fix task where the specific problems are unknown; the module or integration area to audit.

Output

audit-doc.md with a holes table across 7 planes (Security / Data / Logic / Integrations / Observability / UX / Performance), prioritized, plus per-hole reports as fixes close.

Hands off to

Per-hole flow-first cycle for each prioritized hole, then ship-first at task-level closure.

---

Principle 1 — Plane: Security

The most critical plane. HIGH by default. Checked first.

1.1 Auth / Authz

Endpoint accessible without token → HIGH. Internal endpoint protected only by hardcoded token → HIGH. Authorization doesn't check resource-to-tenant ownership (IDOR) → HIGH.

1.2 Injections (SQL, CMD, XSS)

String concatenation in queries, shell=True, innerHTML with user data → grep and flag.

1.3 Encryption

In transit: SMTP without TLS/SSL → HIGH. HTTP for external calls → MED. At rest: passwords in plaintext → HIGH. OAuth tokens in DB without encryption → LOW (single-tenant) / MED (multi-tenant).

1.4 OAuth / Tokens

• CSRF: state = user_code instead of secrets.token_urlsafe() → HIGH
• Token TTL: access_token used without checking expires_at → MED
• Storage: tokens in logs, git, unignored env → HIGH

1.5 Webhook validation

Two mandatory levels: signature (HMAC-SHA256 verified) + timestamp (abs(now - ts) > 300 rejected). Only level 1 → replay possible → MED. Neither → HIGH.

1.6 Secrets in code and configs

Hardcoded secret in code = HIGH always, even for a private repo.

---

Principle 2 — Plane: Integrations architecture

Doesn't break immediately — breaks under load / when external service fails. MED default, HIGH on critical path.

2.1 Rate Limit (429)

Generic except Exception swallows FloodWaitError / RateLimitError → message lost silently. Correct: catch specific error before generic Exception, sleep + one retry.

2.2 Service-specific errors

Every external service has errors that must not fall into generic Exception. Telegram FloodWaitError → sleep + retry. Whisper 413 → check size BEFORE sending. Zoom webhook stale timestamp → reject with 400. SMTP wrong encryption → ValueError, not silent fail.

2.3 In-Memory state

Module-level dict resets on deploy. _oauth_state_store → in-flight OAuth breaks (MED). _token_cache → extra requests after restart (LOW). _active_sessions → everyone logged out on deploy (HIGH).

2.4 Validation BEFORE an expensive network request

File >25MB sent to Whisper without a check → learn about the problem only on 413 → retry loop. Correct: if getsize > limit: raise before opening the file.

2.5 Circuit Breaker

Every external HTTP call has timeout=. ConnectionError handled separately from business errors. Service unavailable → task marked failed, not hanging.

---

Principle 3 — Plane: Data and DB

Rare but irreversible. Especially dangerous in multi-tenant systems.

3.1 Missing transactions (Race Condition)

Trigger for check: read → decide → write based on what was read. Two execute calls in a row without with conn: → concurrent requests can insert a duplicate.

3.2 Missing constraints

The DB must protect integrity independently of the code. Email without UNIQUE → duplicate accounts. FK without ON DELETE → orphans. Status without CHECK → invalid values. Numeric without NOT NULL DEFAULT 0 → NULL surprises.

3.3 N+1 queries

Query inside a loop. 100 records = 101 queries instead of 1. In SQLite less critical than in PostgreSQL, still noticeable on large datasets.

3.4 Plaintext for sensitive fields

User passwords in plaintext → HIGH always. OAuth tokens in DB → LOW / MED. API keys in DB → MED (better in env).

3.5 No TTL and no cleanup of temporary data

In-memory stores without cleanup, tokens/nonce tables without periodic `DELETE WHERE expires_at 25MB without check. | | LOW | Technical improvement or potential vector at growing load. Doesn't break now, creates risk. Examples: in-memory cache without TTL, tokens in plaintext single-tenant, missing metrics. |

Rules

• One row = one gap (don't group even if in same file)
• File:line = specific code line or nearest related
• Gap = one sentence: what specifically is missing or wrong
• HIGH does not exceed 3 in a normal task. 5+ HIGH → re-check criteria

Final line:

Total: {N} HIGH / {M} MED / {K} LOW — {X} gaps, of which {P0} P0 blocks and {P1} P1 blocks.
Waiting for approval → then I pin the block order and ask the work mode.

---

Workflow — 5 phases

Sequential. Cannot start fixing until all 5 phases are completed.

Step 0 — Check / create the task

{log_dir} set → proceed silently. Not set → form slug {date}-audit-{area-slug}, show "Creating task: {slug}. Rename?", mkdir, INSERT into artifacts. Cannot skip.

Child task branch: triggers "create child task" / "next audit stage" / "continuation of {parent}". Form {parent}-part{N}, register with links (from, to, 'part_of'). Direct mkdir + Write + INSERT bypassing this protocol is prohibited — the part_of link won't register and ship-first won't see unclosed chain areas.

Step 0.5 — Calibrate incoming artifacts

audit-first expects idea-first-*.md with type "fix" or "problem". Type differs → propose switching to arch-first. Artifact missing → allowed (manual invocation), skip calibration, proceed with Phase 0.

Phase 0 — Define the audit area

Do not scan until the area is specified. Never scan the whole project — analog of flow-first "going into code without anchors" → 50 LOW gaps, none important. Area given → use as starting point. Not given → ask one specific question.

Phase 1 — Scan across the 7 planes

Sequential walk. Order matters:

| Order | Plane | Why here | |---|---|---| | 1 | Security | Only plane HIGH by default | | 2 | Integrations architecture | Gaps manifest under load | | 3 | Data and DB | Data loss irreversible | | 4 | Error handling | Masks gaps in 1-3 | | 5 | UI/UX | After backend planes | | 6 | Observability | Last — doesn't affect correctness | | 7 | Config in code | Silent gaps, UI+DB "work" but settings not applied |

For each plane: read area files → grep → pin what was found. "Not affected" rule: irrelevant plane → explicit "not applicable" row, don't leave blank.

Phase 2 — Assemble the gap table

All found gaps → one table. HIGH at top, then MED, then LOW. Assign a block to each HIGH and MED. LOW at user's discretion.

Phase 3 — Table approval

Show table + summary line → wait for explicit approval.

Before approval — no code. Even if the gap is obvious and fix is 2 lines. This is the main rule.

User can: approve → Phase 4; remove row (LOW deferred); change criticality; add missed gap. Each edit → updated table → wait for approval again.

After approval — register blocks in task_blocks for each HIGH/MED gap. Sub-blocks use block_num = N*10 + M, parent_block_num = N.

Append ## Blocks section to task.md. Output explicit "Registered N blocks: {list}. Launching Skill('flow-first')."

Phase 3.5 — Batch call to cadence-first

After the INSERT loop and before asking work mode → invoke cadence-first in batch mode so each gap block gets recommended_cadence pre-baked. HIGH gaps usually Tier 1 full cycle; LOW often Tier 3 plan-only — cadence-first formalizes this differentiation.

Why here: cadence is decided at GENERATION time (rich context — audit area, gap table, criticality), not at EXECUTION time.

On failure → log warning, continue. Blocks execute without pre-baked cadence.

Final step of Phase 3 — Ask work mode

Blocks pinned. How do we work?

1. 🔁 Manual — I run each block, wait for your "ok"
2. 🚀 Autopilot — dev-auto-first takes over

Choice 1 → Skill('flow-first') for first HIGH block. Choice 2 → Skill('dev-auto-first').

Save audit-doc.md to log_dir with the approved gap table. Register in task_artifacts. Cannot skip — on session drop, chat table isn't recoverable.

Phase 4 — Fix per-block

Per-block cycle for each gap in order HIGH → MED → LOW: flow-first (understand landscape) → library-first (ready pattern) → plan-first → execute → ship-first (per-block).

Block transition rule: after "go" on ship-first per-block → first action is Skill(flow-first) for the next block. Not Read, not grep, not plan-first directly. flow-first cannot be skipped even if "task is obvious".

Each fix = separate commit with clear description of what it closes.

After all fixes → ship-first task-level updates gap table (✅ Fixed / 🔁 Deferred), does final report, sessions, STATUS, push + sync.

---

Anti-patterns

❌ Shovel the whole project without an area

Grep everywhere → 80 lines of noise. Half false positives, other half LOW. Rule: area set before first grep.

❌ Fix along with scanning

Found a gap → immediately fix without waiting for remaining planes. Fixing first breaks second you haven't found yet. Rule: scan → table → approval → only then code.

❌ Silently skip a plane

"UI isn't relevant" → plane simply not mentioned. User doesn't know: didn't look, or looked and clean? Rule: explicit "not applicable" row for irrelevant planes.

❌ Mix audit and fix in one step

"Found and fixed straight away". User didn't see the table, didn't approve priorities. Rule: table shown before any code change. Approval mandatory.

❌ Everything is HIGH

Lowering the bar — user stops trusting criticality. Rule: HIGH only for exploitable-without-privileges OR data loss/corruption. 3+ HIGH → re-check criteria.

❌ Don't pin LOW in the table

LOW seems insignificant → skip. Six months later LOW is MED, then HIGH, nobody remembers it was known. Rule: all found gaps in the table. Decision what to do with LOW belongs to user.

❌ Don't check settings forms

Form saves → agent considers "Config in code" plane clean. But doesn't check whether field is READ in business logic. Rule: for each settings field grep-check reads (not save/update). Written but not read → gap.

❌ "go" on next block → plan-first instead of flow-first

After block finished, user answered "go" → agent goes directly to plan-first. Rule: "go" on next block = first action Skill(flow-first). flow-first mandatory for each block without exception.

❌ Create child task directly via bash bypassing Step 0

Direct mkdir + Write + INSERT INTO artifacts without the Step 0 child-task branch → task appears without part_of link → ship-first doesn't see unclosed siblings on closure. Rule: task creation only via Step 0 branch. Direct tools prohibited.

❌ One big fix for the whole table

All gaps in a single commit → impossible to roll back specific fix, impossible to review. Rule: one gap = one commit.

❌ Inlining the skill instead of using the Skill() tool

Executed the protocol from memory — outdated version, no accuracy guarantee. Always via Skill('audit-first').

---

Related skills

• idea-first — upstream artifact read in Step 0.5; type "fix"/"problem" → audit-first; other types → arch-first
• flow-first / library-first / plan-first / ship-first — per-block execution chain
• cadence-first — batch-invoked at Phase 3.5 to pre-bake per-block cadence
• arch-first — sibling for feature/product tasks (decomposition instead of gap discovery)

---

Step 99 — Log invocation

sqlite3 {routing_db} \
  "INSERT INTO skill_invocations (task_id, block_num, skill_name, invoked_at)
   VALUES ('{slug}', '{N}', 'audit-first', datetime('now'))" 2>/dev/null || true

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: bearded-illirian
• Source: bearded-illirian/trailmark
• License: MIT

Install and usage instructions live in the source repository linked above.