Auditsmith

Auditsmith

Audit a codebase and produce findings with evidence — not a ticked checklist.

The ten reference files in references/ hold the checklists. This file is how to run them: scope the audit, gather evidence, judge severity, report, and where possible convert findings into CI rules so they can't regress.

The core discipline

A checklist item is a question to investigate, not a box to tick. The difference between a useful audit and a worthless one:

• Every finding cites evidence — file:line, the offending code, and what specifically is wrong. No finding without a location.
• Never report an item you didn't verify. If you couldn't check something (no access, needs runtime, needs the app deployed), say so explicitly in a "not verified" list. Silence implies you checked and it passed — that's how audits mislead.
• Never invent findings to fill a section. A section with nothing wrong gets "no issues found," and that's a good result. Padding an audit destroys its signal.
• Report what's there, not what the checklist expects. The checklists are React/TS/edge-flavored; adapt to whatever stack is actually in front of you and drop items that don't apply rather than forcing them.

When not to run this

Skip Auditsmith and answer directly if any of these are true:

• Pure infrastructure / DevOps with no application code — Terraform-only, Helm-only, pipeline YAML. There's nothing here for the ten reference files to bite on.
• Writing new features from scratch. This audits existing code. If the code doesn't exist yet, help build it — audit later.
• A single snippet or homework-sized question. Scoping, evidence-gathering, and a severity-ranked report on 30 lines of code is theatre.
• Visual or design critique with no code involved. Screenshots and mockups aren't in scope; use a design-review workflow.
• Debugging one specific known error. That's troubleshooting, not an audit. Investigate the bug directly.

When in doubt, ask which area the user cares about — one honest scoping question beats a misapplied audit.

Step 1 — Scope

Don't run all ten areas by default. That produces a report nobody reads.

If the user named an area ("audit security", "review UX"), run that one and mention which adjacent ones look relevant. If the user asked broadly ("audit my app"), inspect the codebase first, then propose a scope and confirm before the deep pass.

Pick areas by what the code actually is:

| Signal in the codebase | Run these | |---|---| | Any app at all | code-quality, ux | | Frontend / components / routes | ui, ux, performance | | Auth, user data, payments, PII | security, api-contract | | Calls to an LLM API, prompts, agents | llm-ai, security | | Public API, backend, service boundary | api-contract, security | | Handles money or irreversible actions | security, api-contract, testing | | Ships to real users / in production | observability, performance, testing | | Slow, or user complains about speed | performance | | Any package.json / lockfile | dependencies | | Pre-launch, handover, or client delivery | security, observability, testing, dependencies |

Bias toward fewer areas audited deeply over all ten audited shallowly. Three areas with real evidence beats ten with generic observations.

Step 2 — Orient before auditing

Understand the codebase before judging it. Skipping this produces findings that are technically true and contextually wrong.

• Read package.json (deps, scripts), the config files, and the directory tree
• Identify: framework, language, styling system, state management, test setup, deploy target
• Find the entry points and the highest-risk paths — auth, payments, data mutations, LLM calls
• Note the project's own conventions; a deviation from their pattern is a finding, an unfamiliar-but-consistent pattern is usually not

Step 3 — Load the relevant references

Read only the reference files for the areas in scope:

| Area | File | Covers | |---|---|---| | UX behavior | references/ux.md | Async states, feedback, error handling, forms, edge cases, navigation | | Code quality | references/code-quality.md | Reuse, styling discipline, duplication, types, state architecture, naming, dead code | | UI | references/ui.md | Structural (lint/grep) + rendered (Playwright/axe) + vision-model layers | | Security | references/security.md | Authn/authz, IDOR, injection, secrets, SSRF, edge/Workers, leakage | | LLM / AI | references/llm-ai.md | Prompt injection, output trust, cost control, privacy, reliability, evals | | Performance | references/performance.md | Bundle, render cost, data layer, caching, memory, edge runtime | | Testing | references/testing.md | Behavior vs implementation, pyramid shape, flakiness, risk coverage | | Dependencies | references/dependencies.md | CVEs, maintenance health, supply chain, licenses, footprint | | Observability | references/observability.md | Logging, error tracking, metrics, alerting, graceful degradation | | API contract | references/api-contract.md | Consistency, error shape, versioning, pagination, idempotency, validation |

Step 4 — Gather evidence

Work in two passes.

Pass 1 — mechanical sweep. Run the greps. They're fast, they cover a lot of the checklists, and they give you a map of where to look closely. Useful starting set (adapt to the stack):

# security
grep -rn "dangerouslySetInnerHTML\|innerHTML\s*=" src/
grep -rniE "(api[_-]?key|secret|token|password)\s*[:=]\s*['\"]" src/
grep -rn "localStorage.setItem" src/          # tokens in localStorage
grep -rn "eval(\|new Function(" src/

# quality / types
grep -rn ": any\|as any\|@ts-ignore\|@ts-expect-error" src/ | wc -l
grep -rn "style={{" src/                       # inline styles
grep -rnE "#[0-9a-fA-F]{6}" src/               # hardcoded colors
grep -rn "console\." src/
grep -rn "TODO\|FIXME\|HACK" src/

# ux / correctness
grep -rn "key={i}\|key={index}" src/
grep -rn "catch {}\|catch (e) {}\|\.catch(() => {})" src/
grep -rn "onClick" src/ | grep -E "/dev/null     # are rules actually on?
cat .github/workflows/*.yml 2>/dev/null        # does CI actually run them?

Counts alone aren't findings — 47 uses of any is a metric. Open a representative sample, judge whether they're pragmatic or dangerous, and cite specific ones.

Pass 2 — read the risky paths. Greps can't find missing things: an absent authorization check, an unhandled empty state, a race condition, a missing rollback. Read the auth flow, the payment/mutation flow, the LLM call path, and the two or three most complex components end to end.

Run the real tooling where available: npm audit, npx depcheck, npx tsc --noEmit, the project's lint command, a bundle analyzer. Actual tool output beats inference.

For anything requiring a rendered app (contrast, overflow, focus order, CLS), you generally can't verify it statically — either write the Playwright check described in references/ui.md or list it as not verified. Don't guess.

Step 5 — Judge severity

Severity is about consequence, not effort:

• Critical — exploitable now, or loses money/data. Exposed secret, missing authz on a sensitive endpoint, injection, a mutation that can double-charge on retry.
• High — users hit it and it breaks or silently misleads. Swallowed errors, missing error states, race conditions rendering stale data, unbounded LLM spend.
• Medium — degrades UX or maintainability without breaking. Duplication, inline styles, missing empty states, weak typing at boundaries.
• Low — consistency and polish. Naming, dead code, minor drift.

Two calibration rules: don't inflate everything to Critical (it destroys the ranking's usefulness), and weight by blast radius — the same pattern in an auth path outranks it in a settings page.

Step 6 — Report

Write findings to a markdown file and present it. Use this structure:

# Audit — [app name]
[Date] · Scope: [areas audited] · [what was and wasn't accessible]

## Summary
[3–5 sentences: overall state, the single most important thing to fix, general pattern observed]

| Severity | Count |
|---|---|
| Critical | N |
| High | N |
| Medium | N |
| Low | N |

## Critical
### C1 — [Short title]
**Where:** `src/path/file.ts:42`
**What:** [the specific problem, with the offending code]
**Why it matters:** [concrete consequence, not "it's bad practice"]
**Fix:** [specific change, code where it helps]

[...High, Medium, Low in the same format...]

## Not verified
[Items requiring runtime, deploy access, or tooling unavailable — so gaps aren't mistaken for passes]

## Enforce in CI
[Which findings convert to lint rules, greps, or CI gates — see Step 7]

Order sections by severity, never by checklist order — the reader should hit the worst thing first. Keep each finding tight; specificity matters more than volume. If a pattern repeats across 30 files, report it once with a representative citation and a count, not 30 times.

Step 7 — Convert findings into CI

An audit fixed once regresses within weeks. The most valuable output is the subset that becomes automated. Always close by identifying what's enforceable and offering to wire it up.

Commonly automatable:

| Finding type | Enforcement | |---|---| | Inline styles, key={index}, onClick on div, missing labels/alt | ESLint (jsx-a11y, core rules) | | any, @ts-ignore, loose types | tsconfig strict + ESLint | | Hardcoded colors/spacing | Stylelint | | Secrets, hardcoded URLs, console., TODOs | grep script in CI | | Vulnerable / unused / duplicate deps | npm audit, depcheck, Dependabot | | Overflow, contrast, focus order | Playwright + axe in CI | | Bundle growth | bundlesize / Lighthouse CI | | Duplication | jscpd threshold |

Offer a concrete deliverable: a flat ESLint config plus an audit.sh covering the grep-only checks. That's the difference between a report and a permanent improvement.

If this skill was installed from the Auditsmith repo, starter versions of both ship in ci/ — ci/audit.sh and ci/eslint.config.audit.mjs. Adapt them to the project rather than writing from scratch.

What audits can't catch

State this in the report rather than letting it be assumed. Static and automated analysis catch broken and inconsistent. They never catch wrong — whether the architecture suits the problem, whether the flow makes sense, whether the product should exist in this shape. Flag it as needing human judgment instead of quietly leaving it out of scope.

Adapting the checklists

The references are React/TS/Cloudflare-flavored because that's where the specific traps were written from. The categories are universal; the tells are not. On a Python backend, Go service, or mobile app: keep the section structure, translate the specifics, and drop what doesn't apply. Never report a finding just because the reference file mentions it — the code in front of you is the source of truth.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: opefyre
• Source: opefyre/auditsmith
• License: MIT

Install and usage instructions live in the source repository linked above.