Context Engineering

Context Engineering

Multi-corpus context engineering — indexes your codebase (lat.md-compatible), Granola transcripts, Notion DBs, Gmail labels, HubSpot notes. Depth-aware packing. Concept-quality scoring.

The engine for building and querying a token-efficient, provenance-tracked company brain across code, human-curated knowledge, and connector streams.

Five tightly-coupled capabilities ship as one skill:

1. Multi-source indexer — AST (14 languages via tree-sitter) + markdown heading trees, schema-versioned cache, incremental re-indexing.
2. Source ABC — the contract connectors implement to feed events into the brain. This skill ships WorkspaceSource + GithubRepoSource only; Notion / HubSpot / Gmail / Granola adapters live elsewhere (Anabasis spec calls this SignalSource).
3. EntityStore — three-tier brain layer: raw/ (verbatim sources) + events/ (append-only JSONL of extracted claims) + wiki/.md (consolidated entity pages with full provenance — every cited claim resolves to file:line + content_hash + ts). Reference impl of Anabasis spec EntityStore ABC.
4. Synthesizer + Auditor — GAM-grade semantic-shift detector (consolidate only on cosine drift, never on every event); wiki_init.py one-shot seeder; Auditor proposes splits / merges / contradictions / dead links.
5. Retrieval surface — depth-aware packer (2 levels — full / pointer, 95% budget utilization) + multi-hop reasoning paths through [[wiki-links]] + query-as-lens reranking + RRF fusion (semantic mode) + anti-hallucination filters.

Plus an MCP server exposing the whole stack as composable tools (pack, index_workspace, index_github_repo, build_embeddings, resolve, stats, wiki.{ask,add,audit,impact_of}, and lat.{locate,section,refs,search,expand} — 15 tools total).

Comparison vs adjacent tools

| Tool | Layer | Differentiator | |---|---|---| | lat.md (Yury Selivanov) | Codebase docs only | Standardized link grammar + // @lat: source backlinks + CI validation | | Context Signals MCP (Dinesh Raghupatruni) | Code symbol map only | Published 79–95% context reduction numbers | | Context Engineering (this skill) | All of the above + multi-corpus | Depth-aware packing + concept-quality scoring + Granola/Notion/Gmail/HubSpot Source ABC |

Honest comparisons in [docs/vs-lat-md.md](./docs/vs-lat-md.md), [docs/vs-context-signals.md](./docs/vs-context-signals.md), [docs/benchmarks.md](./docs/benchmarks.md).

The depth packer is one of these five capabilities, not the headline. The full brain — index + EntityStore + synthesis + retrieval — is what makes this skill the reference implementation of Anabasis's find-links (spec v0.2).

At-a-glance: what CE replaces

• Sourcegraph for an LLM consumer: Sourcegraph excels at human-precise xref over millions of files; CE excels at packing the right slice into the LLM's token budget. Different jobs, complementary tools.
• A bare RAG over a repo: CE adds AST-grade symbol extraction, depth-variable packing (instead of fixed-size chunks), provenance-tracked entity pages, and multi-hop reasoning. RAG is one resolution mode (--semantic); CE is four (keyword, semantic, graph, wiki).
• A static markdown wiki (Obsidian-style): CE generates and maintains the wiki from events/ instead of requiring humans to author every entity page. Humans curate; the synthesizer compounds.
• A scheduled "scrape Notion → vector DB" pipeline: CE provides the engine the pipeline plugs into via Source ABC. The schedule itself is the Anabasis runtime's job, not CE's.

Quick start (one verb)

pack_context.py "users getting 401 on refresh tokens"

Pack 40+ files at 2 depth levels (full / pointer) into a token budget instead of loading 2-3 fully. The skill auto-decides:

• Mode — proper-noun / CamelCase / snake_case query → graph; how/why/what → semantic (if OPENAI_API_KEY set, else keyword); else keyword.
• Task preset — matches fix / bug / 401 / traceback → --task fix; review / pr → review; explain / how does → explain; etc.
• Index — auto-builds one for cwd if no index exists at the configured cache path.

Every call is logged to cache/usage.jsonl (metadata only — query length, mode, task, files packed, budget %, time_ms — no query content). Pass --why to see the trace inline.

Override anything with explicit flags (--mode, --task, --graph, --semantic, etc. — all still work).

Learn the engine

If you want a guided walk from one query to a multi-corpus brain, [docs/learn/](./docs/learn/README.md) is a six-step stepladder — one runnable command and one concept per step. Under an hour on a real repo, no theory homework.

| # | Step | Take-away | |---|------|-----------| | 0 | [Pack one query](./docs/learn/00-pack-one-query.md) | One file, one budget, one packed answer | | 1 | [Budget and depth](./docs/learn/01-budget-and-depth.md) | Full or pointer, not one chunk size | | 2 | [Index a workspace](./docs/learn/02-index-a-workspace.md) | The brain's short-term memory | | 3 | [Graph and multi-hop](./docs/learn/03-graph-and-multi-hop.md) | Linked reasoning, not flat keyword fan-out | | 4 | [Add a source](./docs/learn/04-add-a-source.md) | Code is one Source; transcripts/notes/CRM are others | | 5 | [EntityStore and wiki](./docs/learn/05-entitystore-and-wiki.md) | Compounding memory with full provenance |

How it works

Query → Resolution → Scoring → Depth packing → Packed markdown

1. Index a workspace (local or GitHub). The indexer parses markdown heading trees and extracts AST symbols from 14 code languages via tree-sitter.
2. Resolve which files matter using keyword matching, embedding similarity, graph traversal, or any combination.
3. Filter off-topic results (topic overlap, section extraction, confidence scoring).
4. Pack files at variable depth — most relevant get full content, the rest get progressively compressed down to a one-line mention — until the token budget is ~95% used.

User guide

Scenario 1: Full code repository

Index an entire codebase — the indexer picks up both docs and source files automatically.

# Index
python3 scripts/index_workspace.py /path/to/my-app/

# Pack a query
python3 scripts/pack_context.py "authentication middleware" --budget 8000

# With graph traversal (follows imports/dependencies)
python3 scripts/pack_context.py "PaymentService" --graph --budget 8000

# Task-aware: "fix" mode follows tests, skips docs
python3 scripts/pack_context.py "fix login bug" --task fix --graph --budget 8000

The indexer walks the directory, skipping common build artifacts (.next, dist, node_modules, target, etc.). Code files get AST symbol extraction — every function, class, and interface becomes a searchable heading. Markdown files get heading-tree parsing.

Scenario 2: Knowledge repository (mostly .md files)

Same workflow — the indexer auto-detects file types. For a pure-markdown repo (docs, RFCs, meeting notes, guidelines), heading-tree parsing gives you section-level granularity.

python3 scripts/index_workspace.py /path/to/company-knowledge/

# Conceptual query — semantic mode shines here
python3 scripts/pack_context.py "onboarding process new hires" --semantic --budget 12000

# With anti-hallucination filter for noisy knowledge bases
python3 scripts/pack_context.py "compliance requirements" --semantic --topic-filter --budget 8000

Knowledge-type classification kicks in: architecture docs rank higher than meeting notes, source-of-truth files beat changelogs. This matters when many files match and the packer has to choose who gets Full vs Mention depth.

Scenario 3: Explore a specific feature in an existing graph

If you've already indexed and optionally run Graphify, you can drill into a single feature without re-indexing.

# Graphify builds a rich knowledge graph (call graphs, inheritance, cross-language)
graphify /path/to/my-app/ --output graphify-out/

# Now query a specific feature — graph mode auto-detects graphify-out/graph.json
python3 scripts/pack_context.py "WebSocket reconnection logic" --graph --budget 8000

# Combine with semantic for maximum discovery
python3 scripts/pack_context.py "how does real-time sync work" --semantic --graph --budget 12000

Graph mode finds the entry points via keyword/semantic matching, then traverses the dependency graph outward (imports, callers, tests, docs). The result is a focused slice of the codebase around that feature — not a flat keyword search.

What gets persisted

Indexing produces two JSON files in cache/:

| File | Size | Contents | |------|------|----------| | workspace-index.json | ~3-5 MB for a 500-file repo | Full index with heading trees, AST symbols, content previews, knowledge types | | workspace-index-light.json | ~500 KB | Headings + metadata only (no tree content) — for quick lookups |

Both are human-readable JSON. A file entry in the light index looks like:

{
  "path": "src/hooks/use-tab-history.ts",
  "tokens": 406,
  "nodeCount": 3,
  "headings": [
    { "depth": 0, "title": "src/hooks/use-tab-history.ts", "tokens": 406 },
    { "depth": 1, "title": "const popDirectionHints", "tokens": 5 },
    { "depth": 1, "title": "useTabHistory", "tokens": 36 }
  ]
}

The full index adds the complete heading tree with text content, first sentences, and first paragraphs — everything the packer needs to render at each depth level.

Re-indexing: Run index_workspace.py again whenever the codebase changes. It overwrites the cache. There's no incremental mode yet.

Graph visualization

# 3D force-directed graph — opens in any browser
python3 scripts/visualize_graph.py --top 50

# With query overlay — shows how the packer "sees" the codebase
python3 scripts/visualize_graph.py --top 80 --query "authentication"

# Dual graph — two repos with cross-repo DTO linking
python3 scripts/visualize_graph.py --multi-index cache/fleet-index.json cache/backend-index.json --top 100

# File-level only (no symbols), custom output path
python3 scripts/visualize_graph.py --no-symbols -o my-graph.html

# With graphify edges for richer connections
python3 scripts/visualize_graph.py --graphify graphify-out/graph.json

Outputs a standalone graph.html — zero dependencies, interactive 3D visualization (Three.js). Functions are blue, classes purple, concepts teal, clusters amber, files gray. Click nodes to highlight connections and dim unrelated nodes. Search bar for live keyword filtering. Drag to orbit, scroll to zoom.

Query overlay (--query): Runs the pack_context scoring pipeline and color-codes nodes by relevance — ocean blue (high), teal (medium), sky blue (low), gray (unmatched). The in-browser search bar re-scores client-side for instant feedback.

Dual graph (--multi-index): Merges multiple workspace indexes with repo-prefixed paths. Automatically detects shared types across repos (e.g., matching DTOs) and draws amber cross-repo links. Concept clustering groups symbols by CamelCase prefix (Voyage, Fleet) to break up blob nodes.

If Graphify has been run in the workspace, --graph auto-detects graphify-out/graph.json and uses its richer call graphs, inheritance, and cross-language edges.

Note on TypeScript path aliases with graphify: graphify v0.4 does not resolve tsconfig.json compilerOptions.paths aliases (verified against upstream source — no tsconfig/baseUrl/compilerOptions references). For TS-heavy corpora that rely on @/foo-style aliases, the graphify-preferred path will silently miss those imports as edges. CE's native graph (the fallback when graphify-out/graph.json is absent) DOES resolve them via scripts/tsconfig_resolver.py. Workaround for now: skip graphify when indexing TS monorepos with heavy alias usage; revisit when graphify ships tsconfig support upstream.

Feature map (bird's-eye)

# Single repo — shows feature clusters and how they depend on each other
python3 scripts/feature_map.py --index cache/workspace-index.json

# Multi-repo
python3 scripts/feature_map.py --multi-index cache/fleet-index.json cache/backend-index.json

# Custom output
python3 scripts/feature_map.py --index cache/workspace-index.json -o my-features.html

# Tune cluster granularity on large repos (default 1, keeps singleton clusters)
python3 scripts/feature_map.py --index cache/backend-index.json --min-cluster 5

# v2: LLM-named concepts + sub-features per cluster, domain-tier coloring
python3 scripts/feature_map.py --index cache/fleet-index.json --concept-llm

Uses label propagation community detection on the import graph to discover natural feature clusters. Each cluster is labeled by its dominant directory or top symbol names. A second label-propagation pass folds clusters into domains (only edges with weight ≥ 2 count as structural coupling); the renderer colors each cluster by domain palette slot with lightness offsets for distinct members. Renders as interactive 2D SVG (D3 force-directed). Click a domain row in the legend to filter; click a cluster to see its concept, description, sub-features, and files.

On large repos (2000+ files) the default --min-cluster 1 keeps every disconnected file visible; raise to 5 or 10 to focus on the backbone modules.

Concept labeling (v2): Add --concept-llm to use Claude Haiku to assign product-level concept names ("Navigation" instead of "SideNavbar, TopNavBar") plus a sub-feature list shown in the click-detail panel. Labels are cached per cluster content hash under cache/concept-labels/ — the same cluster shape reuses the same label until files move between clusters. Defaults: model claude-haiku-4-5-20251001, 4 parallel workers, cache dir cache/concept-labels/. Budget: roughly $0.05 and ~1 min on a 100-cluster repo. Requires ANTHROPIC_API_KEY; fails fast if missing.

Features

Multi-language code indexing

The workspace indexer handles markdown (heading-tree) and code files:

.ts .tsx .js .jsx .py .go .rs .rb .java .c .cpp .cs .kt .scala .php

Code files get AST symbol extraction — functions, classes, interfaces, methods, and types become searchable headings, rendered either in full or as a pointer. Falls back to regex if tree-sitter is not installed.

Two depth levels

| Level | What the LLM sees | Relative cost | |-------|-------------------|---------------| | Full | Complete file content | 100% | | Pointer (Mention) | File path + token count | ~3% |

The packer assigns Full to relevant files, demotes to pointer to fit budget, then promotes pointers back to Full if budget remains. Target: 95% utilization. The intermediate Detail/Summary/Headlines bands were removed — they never moved a downstream answer vs. full-or-pointer (see references/eval-results.md).

Four resolution modes (composable)

| Mode | Flag | Use case | |------|------|----------| | Keyword | (default) | Name-based queries, free | | Semantic | --semantic | Conceptual queries ("how does auth work?") | | Graph | --graph | Structural queries ("what depends on X?") | | Semantic + Graph | --semantic --graph | Full discovery |

Graph mode traverses imports/dependencies via BFS. If Graphify has been run in the workspace, --graph auto-detects graphify-out/graph.json and uses its richer call graphs, inheritance, and cross-language edges instead.

Task-type presets

Adjust graph traversal behavior per intent:

| Task | Flag | Traversal strategy | |------|------|--------------------| | fix | --task fix | Deep imports, follow tests, skip docs | | review | --task review | Wide

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: VictorGjn
• Source: VictorGjn/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.