Shadow Frog Dream

ShadowFrog Dream

Autonomous experimentation while the user is away. Every task is an experiment — implement real code in a worktree, run it, persist as a named git branch pushed to the fork. Dream's unique value is implementation experience that compounds across sessions.

Critical Invariants

These rules are stated ONCE here and enforced by helper scripts. Violating any of them is a completion criteria failure.

Prerequisite: .shadow/ must be git-tracked

Dream moves .shadow/ content through git — artifacts are committed onto the dream branch, pushed to the remote, then read back by the reconciler via git show origin/ .shadow/.... If .shadow/ is gitignored (the "local only" option in shadow-frog-init), git add -A silently skips those files, nothing reaches the remote, and the reconciler finds no manifest — every discovery is lost without warning. dream-setup.sh runs git check-ignore .shadow up front and refuses to start if it's ignored. Use shadow-frog-update instead for local-only shadows.

Path Isolation

WORKTREE_BASE = /tmp/shadowfrog-dreams//
WORKTREE_DIR  = $WORKTREE_BASE/dream-

• Worktrees are ALWAYS in /tmp/shadowfrog-dreams//, NEVER in

the project directory. This prevents conflicts between parallel agents and keeps the main repo clean.

• DREAM_NS (namespace) isolates branches per task/instance. Resolved

from: DREAM_NAMESPACE env → TASK_INFO.json → .env → repo basename.

• Override only with DREAM_WORKTREE_BASE env var if /tmp is too small.
• dream-setup.sh computes and enforces all paths. Use it.

Branch Naming

BRANCH_NAME = dream//
DREAM_ID    = YYYYMMDD-HHMMSSZ-

Artifact Format

.shadow/_dreams//report.md
.shadow/_dreams//manifest.json
.shadow/_dreams//patch.diff

NEVER flat files (_dreams/.md). Flat files break the pipeline.

RUN_PREFIX

All python/pytest commands MUST use the RUN_PREFIX resolved during preflight. When RUN_PREFIX="uv run", use $RUN_PREFIX python3 .... Bare python3 or pytest without prefix is a violation when non-empty.

Reconciliation is Mandatory

Every dream branch must reconcile to main before the session ends. The most common failure mode is agents pushing dream branches but never reconciling — losing all discoveries.

Two modes:

• Parallel mode (default): Launch a batch of 3-4 sub-agents → wait

for all to push → run dream-reconcile.py "$REPO_ROOT" ONCE at the end of the batch. The reconciler auto-discovers every un-reconciled dream branch in the namespace — you do NOT pass branch names. One call merges every pushed branch.

• Sequential mode (fallback when sub-agents unavailable):

Complete dream → push → reconcile → verify → next dream. Adds ~30s per dream but guarantees zero data loss if the session crashes mid-batch.

Never queue multiple un-reconciled batches; reconcile at the end of each batch or each individual dream.

Script Failure Recovery

All helper scripts (dream-setup.sh, dream-reconcile.py, dream-validate.py) are self-documenting. If a script fails or is unavailable: read the script source, understand what it does, and adapt its logic manually for your situation. Never skip steps just because a script errored — the steps still need to happen.

Fork repo (user/target-repo)
  main --- .shadow/ (accumulated ALL discoveries)
  |
  +-- dream//  (cycle 1, agent A, from main)
  +-- dream//  (cycle 1, agent B, from main)
  +-- dream//  (cycle 2, from dream//, compounding)

• Branches are live — git checkout dream/ runs the code
• Shadow follows lineage — ancestor chain, not sibling branches
• Main is the accumulator — reconciliation merges ALL discoveries

Prerequisites

1. Forked repo cloned locally with pushable remote
2. ShadowFrog skills installed (install.sh --project /path/to/fork)
3. .shadow/ initialized (/shadow-frog-init)

Helper Scripts

This skill bundles 6 helper scripts. Find them in the skill directory:

SKILL_DIR=""
for DIR in .github/skills/shadow-frog-dream .claude/skills/shadow-frog-dream; do
    [ -d "$DIR" ] && SKILL_DIR="$DIR" && break
done

| Script | Purpose | When to use | |--------|---------|-------------| | dream-setup.sh | Creates worktree + branch with namespace isolation | Phase 3 — start of every experiment | | dream-validate.py | Validates artifacts before push (hard gate) | Phase 5 — before git push | | dream-reconcile.py | Merges dream branches into main's .shadow/ | Phase 6 — after all experiments done | | dream-coverage.py | Computes exploration coverage map | Phase 2 — task planning for diversity | | dream-cleanup.sh | Safely removes ONE dream worktree (with safety gate) | After push — replaces the old inline cleanup snippet | | dream-gc.sh | Sweeps orphan dream worktrees from $DREAM_WORKTREE_BASE | Auto — triggered by dream-setup.sh (per-namespace throttle, default 1× / hour) in orphan-only mode; also --task-complete --namespace "$DREAM_NS" --min-age-min 0 for end-of-session sweep of registered-but-stale dirs |

Usage patterns:

# Setup: creates worktree, prints export vars.
# IMPORTANT: capture the output FIRST, then eval it. Writing
# `eval "$(dream-setup.sh ...)" || exit 1` does NOT catch failures: if the
# command substitution exits non-zero and prints nothing, `eval ""` still
# succeeds (exit 0) and the agent silently proceeds with empty env vars.
# Assigning to a variable makes `|| exit 1` fire on the script's real exit code.
SETUP_OUT="$("$SKILL_DIR/dream-setup.sh" --slug t01-my-experiment)" || exit 1
eval "$SETUP_OUT"
# → exports (keep in sync with dream-setup.sh emit_export block):
#   REPO_ROOT, DEFAULT_BRANCH, DREAM_NS, DREAM_ID, BRANCH_NAME, PARENT_BRANCH,
#   WORKTREE_DIR, WORKTREE_BASE, BASE_COMMIT, RUN_PREFIX, SLUG

# Validate: hard gate before push
python3 "$SKILL_DIR/dream-validate.py" "$DREAM_ID" "$WORKTREE_DIR"

# Reconcile: merge all dream branches into main
python3 "$SKILL_DIR/dream-reconcile.py" "$REPO_ROOT"
# After `git push` succeeds, optionally clean up reconciled branches.
# Cleanup REFUSES to run if `.shadow/` has uncommitted changes, or unless
# HEAD is already on origin/ — so the canonical flow is:
#   reconcile → git add .shadow/ && git commit && git push → re-run --cleanup-branches
python3 "$SKILL_DIR/dream-reconcile.py" "$REPO_ROOT" --cleanup-branches

# Coverage: show which files still need exploration
python3 "$SKILL_DIR/dream-coverage.py" "$REPO_ROOT"

# All scripts support --help.

If a script is not found or fails, read its source — they are self-documenting. Adapt the steps manually if needed (see each phase for inline fallback instructions).

Phase 1: Preflight and Assess

Preflight Validation

REPO_ROOT=$(git rev-parse --show-toplevel)
cd "$REPO_ROOT"

# 0. Auto-detect DREAM_NAMESPACE
if [ -z "${DREAM_NAMESPACE:-}" ]; then
    if [ -f TASK_INFO.json ]; then
        DREAM_NAMESPACE=$(python3 -c "import json; print(json.load(open('TASK_INFO.json')).get('dream_namespace',''))" 2>/dev/null)
        export DREAM_NAMESPACE
    elif [ -f .env ]; then
        DREAM_NAMESPACE=$(grep '^DREAM_NAMESPACE=' .env | head -1 | cut -d'=' -f2-)
        export DREAM_NAMESPACE
    fi
fi
[ -n "${DREAM_NAMESPACE:-}" ] && echo "Dream namespace: $DREAM_NAMESPACE"

# 1. Detect default branch
DEFAULT_BRANCH=$(git symbolic-ref refs/remotes/origin/HEAD 2>/dev/null | sed 's|refs/remotes/origin/||')
if [ -z "$DEFAULT_BRANCH" ]; then
    if git show-ref --verify refs/remotes/origin/main >/dev/null 2>&1; then
        DEFAULT_BRANCH="main"
    elif git show-ref --verify refs/remotes/origin/master >/dev/null 2>&1; then
        DEFAULT_BRANCH="master"
    else
        echo "ERROR: Cannot detect default branch. Fix: git remote set-head origin "
    fi
fi
echo "Default branch: $DEFAULT_BRANCH"

# 2-5. Validate environment
echo "Current branch: $(git branch --show-current)"
echo "Remote: $(git remote get-url origin)"
git ls-remote origin HEAD >/dev/null 2>&1 || echo "ERROR: Cannot reach remote."
[ -d .shadow ] || echo "ERROR: .shadow/ not found. Run /shadow-frog-init first."
mkdir -p .shadow/_dreams
git diff --quiet && git diff --cached --quiet || echo "ERROR: Uncommitted changes."

# 6. Fetch all remote branches (ONE fetch for all agents)
git fetch origin --prune

# 7. List dream branches (namespace-filtered)
DREAM_NS="${DREAM_NAMESPACE:-}"
BRANCH_PATTERN="${DREAM_NS:+origin/dream/${DREAM_NS}/}"
BRANCH_PATTERN="${BRANCH_PATTERN:-origin/dream/}"
echo "Available dream branches:"
git branch -r | grep "$BRANCH_PATTERN" | sed 's|origin/||' || echo "  (none)"

# 8. Detect RUN_PREFIX from lockfiles
if [ -f uv.lock ]; then
    uv sync --all-groups 2>&1 | tail -3
    RUN_PREFIX="uv run"
elif [ -f package-lock.json ]; then
    npm install --quiet 2>&1 | tail -3
    RUN_PREFIX="npx"
elif [ -f yarn.lock ]; then
    yarn install --silent 2>&1 | tail -3
    RUN_PREFIX="npx"
else
    RUN_PREFIX=""
fi
echo "RUN_PREFIX='$RUN_PREFIX'"

# 9. List compoundable experiments
echo ""
echo "=== COMPOUNDABLE EXPERIMENTS ==="
if [ -f .shadow/_dreams/_index.md ]; then
    awk -F'|' 'NR>2 && /useful/ {
        gsub(/ /,"",$2); gsub(/ /,"",$4); gsub(/ /,"",$6);
        gsub(/^ +| +$/,"",$5);
        if ($2 != "" && $6 != "") print $6 " | " $3 " | " $5
    }' .shadow/_dreams/_index.md
    COMPOUNDABLE=$(awk -F'|' 'NR>2 && /useful/ {gsub(/ /,"",$2); if ($2 != "") c++} END {print c+0}' .shadow/_dreams/_index.md)
    echo "Total compoundable: $COMPOUNDABLE"
else
    echo "(none — first dream session)"
fi

If any check prints ERROR, STOP. Do not use exit 1 — check output and stop at the agent level.

(RUN_PREFIX MUST be threaded into every subagent prompt — see Critical Invariants above. Bare python3/pytest without prefix = completion criteria violation.)

Snapshot Branch State

After the single git fetch, capture dream branches and pass to all sub-agents — they do NOT fetch independently.

DREAM_NS="${DREAM_NAMESPACE:-}"
BRANCH_FILTER="${DREAM_NS:+origin/dream/${DREAM_NS}/}"
BRANCH_FILTER="${BRANCH_FILTER:-origin/dream/}"
git branch -r --format='%(refname:short) %(objectname:short)' \
  | grep -F "$BRANCH_FILTER" \
  | sed 's|origin/||' > .shadow/_dreams/.branch-map.txt
cat .shadow/_dreams/.branch-map.txt

# Initialize session tracking (orchestrator-only; agents do NOT write here)
: > .shadow/_dreams/.session-branches.txt

Assess Codebase

Read _meta/state.json, _index.md, existing discoveries, and past dream reports in _dreams/.

Build Exploration Coverage Map

File-level coverage breadth is the strongest predictor of dream success (r²=0.63 vs bugs found), NOT dream count (r²=0.04).

# Find and run the coverage script
COVERAGE_SCRIPT=""
for DIR in .github/skills/shadow-frog-dream .claude/skills/shadow-frog-dream; do
    [ -f "$DIR/dream-coverage.py" ] && COVERAGE_SCRIPT="$DIR/dream-coverage.py" && break
done
[ -n "$COVERAGE_SCRIPT" ] && python3 "$COVERAGE_SCRIPT" "$REPO_ROOT" || echo "WARNING: dream-coverage.py not found"

Coverage definition: A file is "covered" only when its shadow has ≥1 behavioral discovery (line starting with - ). Placeholder-only = NOT covered.

Scoped exploration (--scope) — pass --scope (repeatable) to restrict the coverage map to a specific subtree. Use this when the broader repo is well-explored but a particular area (e.g., a known frontier of bugs, a newly-added module, a subsystem the user just flagged) deserves a focused dream session. All counts (totals, %, saturated, fan-in, per-dir) are computed over the scoped subset only.

# Scope to one subtree
python3 "$COVERAGE_SCRIPT" "$REPO_ROOT" --scope src/auth/

# Scope to multiple subtrees in one pass
python3 "$COVERAGE_SCRIPT" "$REPO_ROOT" --scope src/auth/ --scope src/db/

When using --scope, the per-category task quotas (Phase 2) still apply but are interpreted against the scoped subset. Don't use scoped exploration as the default — pick it only when there's a concrete reason to concentrate effort. Unscoped diversity remains the strongest predictor of useful discoveries.

Review Past Dreams (Required)

When compoundable experiments exist (preflight step 9):

• Read each report: cat .shadow/_dreams//report.md
• Choose which to continue (extending, fixing, integrating)
• Note dead_end experiments to avoid repeating
• Trace lineage via the parent column in _dreams/_index.md

Compounding quality gate — before choosing to compound from a parent:

1. Read the parent's report.md AND manifest.json
2. Verify the parent has a non-empty patch.diff (prose-only parents

are low-value — prefer parents with working code)

1. Identify at least one specific file or function you plan to modify/extend
2. Check the parent's area isn't saturated (8+ discoveries) — if it is,

start fresh from main unless you have a concrete new angle

1. Log your compounding intent: "I will extend parent's retry logic in

src/http.py to handle connection timeouts" — vague "continue exploring" is NOT compounding

First dream session: if preflight step 9 shows (none), all tasks branch from main.

Phase 2: Plan

Generate a concrete plan. Target 12 tasks (2 per category). On small codebases (/ (compounds prior) Target: src/parsers/csv.py (extending parent's failing tests) Why: Parent found 2 crashes, need to verify fixes ...

### Diversity Rules

Prevent fixation (exploring the same files while leaving most untouched):

1. **Max 2 tasks per source file** (unless prior dream left concrete follow-up)
2. **≥30% of tasks on uncovered files** (from coverage map)
3. **≥2 tasks on "deep" files** (utilities, internals, converters)
4. **Vary directories** — no 3+ consecutive tasks in same dir

**Self-check before finalizing:** unique target files ≥ 60% of task count,
uncovered file tasks ≥ 30%, no file in > 2 tasks. Swap if failing.

**Escape hatches** (document justification): prior dream's failing test,
concrete untested hypothesis, file is 500+ lines with unexplored sections,
codebase has /)" || exit 1
eval "$SETUP_OUT"

Capture into SETUP_OUT first, then eval it — see Helper Scripts § usage patterns (above) for why bare eval "$(…)" || exit 1 silently swallows the script's exit code.

This exports (keep in sync with dream-setup.sh): REPO_ROOT, DEFAULT_BRANCH, DREAM_NS, DREAM_ID, BRANCH_NAME, PARENT_BRANCH, WORKTREE_DIR, WORKTREE_BASE, BASE_COMMIT, RUN_PREFIX, SLUG.

If dream-setup.sh fails or is not found: Apply the Script Failure Recovery rule (read the script source, adapt its logic). Common causes: missing git remote, branch already exists, /tmp permissions.

Note: Shell variables don't persist across tool calls. Either run multi-step setup in a single shell, or re-derive values. From inside a worktree, get main repo with: git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel

If worktree creation fails: mark task blocked, replace with another.

What Meaningful Compounding Looks Like

Compounding means actively engaging with the parent's code, not just sitting on its branch. Valid compounding approaches:

• Extend: import or call the parent's modules and build on them
• Modify: edit the parent's code to fix limitations noted in its report
• Refactor: restructure the parent's implementation for better design
• Integrate: wire the parent's standalone module into the real codebase
• Test deeper: add edge-case tests for the parent's implementation

Don't assume the parent dream's code is complete or frozen — iterative improvement is the whole point. If you can't find anything meaningful to build on, start fresh from main instead.

Compounding that only adds a new standalone module beside the parent's code (with no imports, edits, or integration) is NOT compounding — it's a fresh experim

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: microsoft
• Source: microsoft/ShadowFrog
• License: MIT

Install and usage instructions live in the source repository linked above.