Rescan

dbt-wiki — Rescan Workflow (v2.0)

Rescan is the daily / per-feature update path. It assumes init has already run (so .dbt-wiki/ + SCHEMA.md + _internal/ exist) and only processes diffs against the last manifest_sha.

If .dbt-wiki/ doesn't exist, rescan refuses and points to /dbt-wiki:init.

Step 0: Pre-condition Check

# WIKI_DIR = git repo root (where .dbt-wiki/ lives); fallback to current $PWD.
# Same logic as init Step 0pre — rescan must look at the SAME location
# init wrote to, regardless of where the user invoked rescan from.
WIKI_DIR=$(git rev-parse --show-toplevel 2>/dev/null) || WIKI_DIR="$PWD"
cd "$WIKI_DIR" || { echo "Cannot cd to $WIKI_DIR"; exit 1; }

test -d .dbt-wiki || { echo "Knowledge base not initialized at $WIKI_DIR/.dbt-wiki/. Run /dbt-wiki:init first."; exit 1; }
test -f .dbt-wiki/log.md || { echo ".dbt-wiki/log.md missing. Re-run /dbt-wiki:init."; exit 1; }
# `_internal/` is a rebuildable cache (init may gitignore it), so a fresh clone
# can legitimately lack it. Self-heal from the plugin's init assets instead of
# erroring. `` = the init skill's assets dir, a sibling of this
# skill: resolve it as `/../init/assets` (this SKILL.md lives in
# `.../skills/rescan/`; init's assets are in `.../skills/init/assets/`).
if [ ! -f .dbt-wiki/_internal/extract_column_lineage.py ]; then
  mkdir -p .dbt-wiki/_internal
  # copy every production script + template the cache needs (NOT the *_test.py)
  for f in extract_column_lineage extract_sql_comments extract_recursive_column_lineage \
           format_lineage_diagram detect_source_language lint_schema_divergence \
           lint_identifier_fidelity build_evidence_pages build_index_knowledge reconcile; do
    cp "/$f.py" .dbt-wiki/_internal/
  done
  cp "/synthesis_template.md" .dbt-wiki/_internal/
  echo "Restored .dbt-wiki/_internal/ from the plugin (rebuildable cache)."
fi

# Rescan-OWNED materiality helpers (Step 2.6). These live in THIS skill's
# assets (not init's), so they get a SEPARATE copy step. Both land in the
# same `_internal/` dir so the sibling import `from logic_sha import ...`
# inside classify_materiality.py resolves. `` = the directory
# containing THIS SKILL.md (`.../skills/rescan/`). We MUST use that
# placeholder, not a bare `../`: Step 0 already `cd`-ed to the git root, so
# a relative path here would resolve against the repo root, not this skill.
if [ ! -f .dbt-wiki/_internal/classify_materiality.py ]; then
  mkdir -p .dbt-wiki/_internal
  cp "/assets/logic_sha.py"            .dbt-wiki/_internal/
  cp "/assets/classify_materiality.py" .dbt-wiki/_internal/
  echo "Restored materiality helpers (logic_sha.py + classify_materiality.py)."
fi
test -f .dbt-wiki/_internal/classify_materiality.py || {
  echo "Could not restore materiality helpers from /assets/. Re-run /dbt-wiki:init."
  exit 1
}
test -f .dbt-wiki/_internal/extract_column_lineage.py || {
  echo "Could not restore _internal/ from . Re-run /dbt-wiki:init."
  exit 1
}

# Resolve dbt project root using the 5-tier detection from init Step 0a:
#   1. /dbt-wiki:rescan  arg
#   2. $DBT_PROJECT_DIR env var (dbt-mcp / dbt CLI convention)
#   3. ancestor walk from cwd (up to 5 levels)
#   4. descendant scan from cwd (max-depth 3, excludes node_modules / .git /
#      target / .venv / __pycache__ / dbt_packages / .repo-wiki / .dbt-wiki)
#   5. legacy whitelist (./ or dbt/)
DBT_DIR=""
DBT_DIR_SOURCE=""
if [ -n "$SKILL_ARG" ] && [ -f "$SKILL_ARG/dbt_project.yml" ]; then
  DBT_DIR="$SKILL_ARG"; DBT_DIR_SOURCE="explicit arg"
fi
if [ -z "$DBT_DIR" ] && [ -n "$DBT_PROJECT_DIR" ] && [ -f "$DBT_PROJECT_DIR/dbt_project.yml" ]; then
  DBT_DIR="$DBT_PROJECT_DIR"; DBT_DIR_SOURCE="\$DBT_PROJECT_DIR"
fi
if [ -z "$DBT_DIR" ]; then
  candidate="$PWD"
  for _ in 1 2 3 4 5 6; do
    if [ -f "$candidate/dbt_project.yml" ]; then
      DBT_DIR="$candidate"; DBT_DIR_SOURCE="ancestor walk"; break
    fi
    parent=$(dirname "$candidate"); [ "$parent" = "$candidate" ] && break
    candidate="$parent"
  done
fi
if [ -z "$DBT_DIR" ]; then
  match=$(find . -maxdepth 3 -name dbt_project.yml -type f \
    -not -path '*/node_modules/*' -not -path '*/.git/*' \
    -not -path '*/target/*' -not -path '*/.venv/*' \
    -not -path '*/__pycache__/*' -not -path '*/dbt_packages/*' \
    -not -path '*/.repo-wiki/*' -not -path '*/.dbt-wiki/*' \
    2>/dev/null | head -1)
  [ -n "$match" ] && { DBT_DIR=$(dirname "$match"); DBT_DIR_SOURCE="downward scan"; }
fi
if [ -z "$DBT_DIR" ]; then
  for candidate in "dbt" "."; do
    [ -f "$candidate/dbt_project.yml" ] && { DBT_DIR="$candidate"; DBT_DIR_SOURCE="legacy whitelist"; break; }
  done
fi
test -n "$DBT_DIR" || {
  echo "Cannot find dbt_project.yml. Pass path as arg, set \$DBT_PROJECT_DIR, or cd to project root."
  exit 1
}
DBT_DIR=$(cd "$DBT_DIR" && pwd)
echo "✓ dbt project root: $DBT_DIR  (via: $DBT_DIR_SOURCE)"

test -f "$DBT_DIR/target/manifest.json" || {
  echo "Missing $DBT_DIR/target/manifest.json — run: cd $DBT_DIR && dbt parse"
  exit 1
}
test -d "$DBT_DIR/target/compiled" || {
  echo "Missing $DBT_DIR/target/compiled — run: cd $DBT_DIR && dbt compile"
  exit 1
}

# Detect Python execution mode (same as init Step 0 — uv preferred)
PY_RUNNER=""
if command -v uv >/dev/null 2>&1; then
  PY_RUNNER="uv run"
elif python3 -c "import sqlglot" 2>/dev/null; then
  PY_RUNNER="python3"
else
  echo "Need either uv (recommended) or pip-installed sqlglot."
  echo "  brew install uv     # or: curl -LsSf https://astral.sh/uv/install.sh | sh"
  echo "  OR"
  echo "  pip install 'sqlglot>=25.0'"
  exit 1
fi

Step 1: Detect drift

NEW_SHA=$(md5 -q "$DBT_DIR/target/manifest.json" 2>/dev/null || md5sum "$DBT_DIR/target/manifest.json" | cut -d' ' -f1)
LAST_SHA=$(grep -m 1 'manifest_sha:' .dbt-wiki/log.md | sed 's/.*manifest_sha: //' | tr -d ' ')

if [ "$NEW_SHA" = "$LAST_SHA" ]; then
  echo "No manifest changes since last init/rescan (sha: $NEW_SHA)."
  echo "If you only changed SQL inside a model and re-ran dbt compile, manifest"
  echo "may be unchanged. To force rescan, delete the manifest_sha line from log.md."
  exit 0
fi

Step 2: Diff models

Read both old (cached) and new manifest. Build three lists:

import json

new_manifest = json.load(open(f"{DBT_DIR}/target/manifest.json"))
new_models = {nid: n for nid, n in new_manifest['nodes'].items() if n.get('resource_type') == 'model'}

# Read existing model pages from .dbt-wiki/_evidence/models/*.md
import glob, os
existing_pages = {}
for path in glob.glob('.dbt-wiki/_evidence/models/*.md'):
    with open(path) as f:
        content = f.read()
        # Parse frontmatter to get unique_id
        fm = parse_frontmatter(content)  # standard YAML frontmatter parser
        if fm.get('removed'):
            continue  # skip already-archived
        existing_pages[fm['unique_id']] = path

added = set(new_models) - set(existing_pages)
removed = set(existing_pages) - set(new_models)
common = set(new_models) & set(existing_pages)

# For common, check if model node hash changed (compare a small subset)
modified = []
for uid in common:
    new = new_models[uid]
    existing_fm = parse_frontmatter(open(existing_pages[uid]).read())
    # Compare: materialization, tags, depends_on, columns count, raw_code hash
    if (new['config']['materialized'] != existing_fm.get('materialization')
        or set(new['depends_on']['nodes']) != set(existing_fm.get('depends_on', {}).get('refs', []) +
                                                   [f"source.{s}" for s in existing_fm.get('depends_on', {}).get('sources', [])])
        or len(new['columns']) != len(existing_fm.get('columns', []))
        or md5(new['raw_code']) != existing_fm.get('raw_code_md5')):
        modified.append(uid)

(Same logic for sources, macros, seeds, snapshots — compare frontmatter to detect material change.)

Capture the old struct now, before any overwrite. Step 2.6's materiality classification needs each modified/removed model's PRIOR {columns, depends_on, materialization} — and that lives in the existing evidence page frontmatter, which Step 4 overwrites. So while you still hold existing_fm here in the Step 2 diff (before the Step 3/4 rewrite), stash the old struct per uid:

# Stash the prior evidence-page frontmatter snapshot per uid, keyed for
# Step 2.6. MUST happen here (Step 2), before Step 4 overwrites the page.
old_struct = {}   # uid -> {"columns", "depends_on", "materialization"}
for uid in modified | set(removed):
    fm = parse_frontmatter(open(existing_pages[uid]).read())
    old_struct[uid] = {
        "columns": {c['name'] for c in fm.get('columns', [])},
        "depends_on": set(fm.get('depends_on', {}).get('refs', []) +
                          [f"source.{s}" for s in fm.get('depends_on', {}).get('sources', [])]),
        "materialization": fm.get('materialization'),
    }

Print summary before writing:

Rescan diff summary (vs last manifest sha: ):
  Added:     models,  sources,  macros
  Modified:  models,  sources,  macros
  Removed:   models,  sources,  macros (will be archived)

Continue? (yes/no)

If yes, proceed. Otherwise abort.

Step 2.6: Per-model materiality classification (0 LLM)

Sync needs to know which changed models changed their logic/structure (material — the knowledge page must be re-distilled) versus which only had a comment/whitespace edit (cosmetic — leave the page alone). Compute that here, deterministically in Python, and emit a map sync consumes. This is 0-LLM — pure sqlglot fingerprinting + set comparison, same cheap-path discipline as the rest of rescan.

The helper classify_changed_models(changed, cache, dialect="redshift") -> (materiality_map, updated_cache) is already implemented + tested (.dbt-wiki/_internal/classify_materiality.py, restored in Step 0). materiality_map = {uid: "material"|"cosmetic"}; the logic-fingerprint cache has shape {uid: {"sha", "method"}}.

# Pseudocode — per-model materiality classification (0 LLM). Run via $PY_RUNNER
# so classify_materiality.py's sibling `from logic_sha import ...` resolves
# (both helpers live in .dbt-wiki/_internal/ — see Step 0 self-heal).
import json
from pathlib import Path

wiki = Path(".dbt-wiki")
internal = wiki / "_internal"

# Build the `changed` list classify_changed_models expects: one dict per uid
# across added | modified | removed.
#   old: the PRIOR {columns(name set), depends_on(set), materialization} from
#        old_struct stashed in Step 2 BEFORE Step 4 overwrote the page; None
#        for `added` (no prior page).
#   new: {columns(name set), depends_on(set), materialization, compiled_sql}
#        from the NEW manifest + the compiled SQL file; None for `removed`
#        (gone from manifest).
def _compiled_sql(node):
    # dbt's manifest `compiled_path` is already project-root-relative and
    # ALREADY includes the `target/compiled//...` prefix — do NOT
    # re-prepend `target/compiled` (that double-prefixes → file never found →
    # empty SQL → every model force-classed material, neutering the triage).
    p = Path(DBT_DIR) / node["compiled_path"]
    return p.read_text() if p.exists() else ""

changed = []
for uid in added:
    n = new_models[uid]
    changed.append({"uid": uid, "status": "added", "old": None,
                    "new": {"columns": set(n["columns"]),
                            "depends_on": set(n["depends_on"]["nodes"]),
                            "materialization": n["config"]["materialized"],
                            "compiled_sql": _compiled_sql(n)}})
for uid in modified:
    n = new_models[uid]
    changed.append({"uid": uid, "status": "modified",
                    "old": old_struct[uid],   # captured in Step 2, pre-overwrite
                    "new": {"columns": set(n["columns"]),
                            "depends_on": set(n["depends_on"]["nodes"]),
                            "materialization": n["config"]["materialized"],
                            "compiled_sql": _compiled_sql(n)}})
for uid in removed:
    changed.append({"uid": uid, "status": "removed",
                    "old": old_struct[uid], "new": None})

# Load the prior logic-fingerprint cache ({} on first run — file absent).
cache_path = internal / "logic_sha_cache.json"
cache = json.loads(cache_path.read_text()) if cache_path.exists() else {}

# Classify. dialect comes from the same source as init (log.md / dbt_project.yml).
from classify_materiality import classify_changed_models
materiality_map, updated_cache = classify_changed_models(changed, cache, dialect=DIALECT)

# Persist the refreshed cache back, and emit the map for sync to consume.
cache_path.write_text(json.dumps(updated_cache, indent=2, sort_keys=True))
(internal / "last_rescan_materiality.json").write_text(
    json.dumps(materiality_map, indent=2, sort_keys=True))

last_rescan_materiality.json is the hand-off artifact: /dbt-wiki:sync reads it to decide which stale knowledge pages actually warrant a (LLM-costed) re-distill vs which can keep their existing content. rescan itself still makes 0 LLM calls — it only writes these two JSON files.

Step 3: Process additions

For each model in added:

1. Run column lineage extraction (same as init Step 4):

``bash $PY_RUNNER .dbt-wiki/_internal/extract_column_lineage.py \ "$DBT_DIR/target/compiled/$PROJECT/${original_file_path}" redshift > /tmp/cl.json ``

1. Reconcile sqlglot output with schema.yml columns
2. Write .dbt-wiki/_evidence/models/.md per SCHEMA's model page type
3. (Same for sources → .dbt-wiki/_evidence/sources/, macros → .dbt-wiki/_evidence/macros/, etc.)

Step 4: Process modifications

For each model in modified:

1. Read existing page from .dbt-wiki/_evidence/models/.md; preserve

custom body sections (anything outside the standard sections defined in SCHEMA.md):

• Standard: ## Description, ## Materialization Notes, ## SQL Preview,

## Column Sources (from sqlglot), ## Tests, ## Cross-references

• Custom: any other ## heading the user added → preserve verbatim at end

1. Re-run column lineage extraction (Step 3a above)
2. Build new frontmatter from current manifest + sqlglot output
3. Write merged file back to .dbt-wiki/_evidence/models/.md:

new frontmatter + regenerated standard sections + preserved custom

Step 5: Process removals (archive, don't delete)

For each model in removed:

mkdir -p .dbt-wiki/_archive//
mv .dbt-wiki/_evidence/models/.md .dbt-wiki/_archive//

Add a comment line in the moved file's frontmatter:

---
unique_id: model.example_dbt_project.deprecated_model
removed: true
removed_at: 2026-05-02
removed_reason: "no longer in manifest after dbt parse"
# ... (rest of original frontmatter preserved)
---

Never hard-delete. User can restore from _archive/ if needed.

Step 6: Always re-generate index.md and lineage.md

These two files are derived; regenerate from scratch every rescan:

• index.md: knowledge-first — the evidence sections come from the

evidence re-scan; the knowledge sections (## Entities / ## Metrics / ## Concepts) are deterministically regenerated from the current knowledge pages' frontmatter by the shipped generator:

``bash $PY_RUNNER .dbt-wiki/_internal/build_index_knowledge.py .dbt-wiki ``

Run it after the evidence-section rebuild so both halves reflect the current state. Section order: Entities → Metrics → Concepts → Evidence: Models (grouped by tier / materialization / tag / group) → Evidence: Sources → Evidence: Macros (used) → Evidence: Seeds / Snapshots / Tests / Exposures.

• Identifier-fidelity gate — if any knowledge page was re-distilled or

edited this rescan (or a column was renamed/dropped upstream), re-run the phantom-column gate so no page is left citing a column the manifest no longer has: $PY_RUNNER .dbt-wiki/_internal/lint_identifier_fidelity.py .dbt-wiki (see init Step 6.8). Ex

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kouko
• Source: kouko/monkey-skills
• License: MIT

Install and usage instructions live in the source repository linked above.