Optimize

Performance Optimization Pipeline

Overview

Orchestrates iterative, measurement-driven performance optimization of any optimizable system — source code, container images and services, LLM hyperparameters, build pipelines, or a composite of these. Each iteration measures, profiles the dominant bottleneck, forms falsifiable speedup hypotheses, generates candidate changes, and verifies each one by measurement — banking only a change a benchmark proves faster and a correctness check proves safe.

Two principles are structural, not advisory:

• Measure first, always. No optimization before a trustworthy baseline exists. Profiling precedes hypothesizing; hypothesizing precedes changing. Every speedup number originates from the measurement harness (pipeline-measure-runner) — never from model narration.
• The model generates candidates; measurement decides. LLMs reliably pick the wrong bottleneck and over-claim speedups. This pipeline uses the model as a candidate generator and a deterministic harness as the evaluator. A change that was not measured is not a speedup.

The skill is domain-generic. The methodology — the scientific loop — is identical across domains; only the measurement command, the profiling technique, and the change space differ. Those three live in a per-run Optimization Contract (contract.md); the pipeline itself carries zero domain-specific commands. Swap the contract and the same pipeline optimizes a container image instead of a function.

When to Use

• User wants to make something faster, smaller, or cheaper against a measurable goal.
• Triggers: "optimize X", "make it faster", "profile the hotspot", "reduce the image size", "tune LLM hyperparameters", "speed up the build".
• The target can be measured by a repeatable command — or the user can supply one.
• Domains: source code, container images / services / configs, LLM hyperparameters, build pipelines, and composite system targets spanning several of these.

When NOT to use

• Removing dead code, unused imports, or reducing cyclomatic complexity — use cleanup. Cleanup makes code tidier; it does not measure speed.
• A correctness bug or a failing test — use debug.
• Making existing code meet quality criteria or pass tests — use polish.
• There is no measurable metric and none can be supplied — the skill hard-gates on a measurable goal and cannot run without one.
• Understanding how code works, with no change intent — use explain.

Input

$ARGUMENTS — the optimization target, the metric, and the goal. Examples:

• optimize make src/parser/tokenizer.py 2x faster, p95 under 120ms
• optimize type:container reduce the api image below 200MB
• optimize type:llm_hyperparams tune sampling params for lowest eval loss
• optimize type:system scope:branch cut orchestrator end-to-end latency below 400ms

Modifiers:

• scope: — branch | global | working; bounds which files / artifacts a candidate change may touch.
• type: — code | container | llm_hyperparams | build_pipeline | system | generic; selects the contract auto-detect path. If omitted, it is inferred from the target in Phase 0.5.

If $ARGUMENTS is empty, or names no measurable metric, the Phase 0 Goal Declaration gate collects it before the run proceeds.

Scope Parameter

See [skills/shared/scope-parameter.md](../shared/scope-parameter.md) for the canonical scope modes (branch, global, working) and their git commands.

• Default (no scope:): all project files / artifacts eligible for candidate changes.
• scope: controls which files / artifacts a candidate may edit. Profiling and measurement are never scope-bound — the profiler reads whatever it needs and the harness runs the whole measure_command. The contract's scope field records the resolved value; a candidate change that would cross it triggers the Phase 4.5 boundary pause.

Constants

| Constant | Value | Bounds | | ------------------------------ | ----------- | --------------------------------------------------------------- | | MAX_OPTIMIZE_ITERATIONS | 6 | Phase 2→6 loop hard cap | | CANDIDATE_COUNT | 4 | parallel pipeline-perf-candidate agents per hypothesis | | SIGNIFICANCE_THRESHOLD | 0.10 | minimum measured improvement to keep a candidate | | CV_CEILING | 0.10 | measurement-noise hard-block ceiling (coefficient of variation) | | MEASURE_REPS | 10 | default benchmark repetitions | | MEASURE_WARMUP | 3 | default warmup repetitions (discarded) | | DIMINISHING_RETURNS_WINDOW | 2 | iterations of sub-threshold gain before stopping | | MAX_CANDIDATE_REVIEW_RETRIES | 2 | retries when a candidate agent returns malformed output | | TASK_DIR | .mz/task/ | task-state root |

MEASURE_REPS, MEASURE_WARMUP, CV_CEILING, and MAX_OPTIMIZE_ITERATIONS seed the matching contract.md fields (measure_reps, measure_warmup, noise_ceiling_cv, iteration_cap). Once the contract is approved, the contract's values are authoritative for the run — the contract is the per-run source of truth.

Core Process

Phase Overview

| Phase | Goal | Reference | Gate | | ----- | -------------------------------------- | ----------------------------------- | ------------------------ | | 0 | Setup + Goal Declaration | inline below | Hard | | 0.5 | Contract Approval | inline below | Approval | | 1 | Baseline | phases/baseline.md | Hard | | 2 | Profile & Classify Bottleneck | phases/profile_and_hypothesize.md | — | | 3 | Hypothesis Generation & Prioritization | phases/profile_and_hypothesize.md | — | | 3.5 | Backlog / Strategy Approval | inline below | Approval | | 4 | Candidate Generation & Verification | phases/candidate_loop.md | autonomous (risk-tiered) | | 4.5 | Mid-loop Boundary Pause | inline below | Conditional | | 4.7 | Manual Correctness Checkpoint | inline below | Conditional | | 5 | Validation & Results | phases/validate_and_results.md | Approval | | 6 | Termination Decision | phases/terminate.md | loop-back / stop |

The loop is Phase 2 → 3 → 3.5 → 4 → (4.5 / 4.7 as triggered) → 5 → 6; Phase 6 either loops back to Phase 2 (re-profile — the bottleneck has moved) or stops. Phases 0, 0.5, and 1 run exactly once. Read each phase file when you reach it — do not load all phase files upfront.

Phase 0: Setup + Goal Declaration

Hard gate — the pipeline cannot proceed without a measurable goal.

1. Parse input — extract scope:, type:, and the free-text target + metric + goal from $ARGUMENTS.
2. Resolve scope — per [skills/shared/scope-parameter.md](../shared/scope-parameter.md); save the resolved file list to .mz/task//scope_files.txt. No scope: → all project files / artifacts eligible.
3. Task name — _optimize_, where ` is a snake_case summary of the target (max 20 chars); on same-day collision append v2, v3`.
4. Resume check — apply the resume contract in [skills/shared/resume-protocol.md](../shared/resume-protocol.md): if .mz/task//state.md exists with Status: running | failed, present the Resume gate. A pre-existing state.md without a skill_variant: methodology key belongs to the renamed cleanup skill's task history, not this skill — do not resume into it; pick a fresh _vN name.
5. Task dir & state — create .mz/task//; write state.md per [skills/shared/state-schema.md](../shared/state-schema.md): first line MUST be schema_version: 2, then Status, Phase, PhaseName, Started, Iteration (0), FilesWritten, and the progress-ledger keys phase_complete: false and what_remains: [], plus skill-specific keys skill_variant: methodology, Target, Metric, Goal, TargetType, and an empty ## History section (the persistent record of every hypothesis tried and its measured outcome — it survives context compaction and stops a failed transform from being retried).
6. Task tracking — TaskCreate per pipeline phase.

Goal Declaration. The pipeline cannot proceed without a measurable goal. A measurable goal has three parts: a metric (what is measured), a unit, and either a target value or an explicit "best effort" (optimize as far as the iteration cap allows).

• If any part is missing — e.g. "make it faster" with no metric and no unit — emit BUDGET MISSING and ask the user, via AskUserQuestion, for the missing parts. Never guess a metric or a target. Do not proceed.
• Once all three parts are present → emit BUDGET DECLARED, record Metric and Goal in state.md, and proceed to Phase 0.5.

The "budget" is the performance budget — the measurable goal the whole loop is held to.

Phase 0.5: Contract Approval

Approval gate. No measurement runs before the contract is approved. See [skills/shared/approval-gate.md](../shared/approval-gate.md) for the two-surface pattern, the MZ_DEV_PIPE_AUTO_APPROVE unattended-mode bypass, and the cost-preview format.

Draft the contract. Determine target_type (from type: or inferred from the target), then assemble .mz/task//contract.md:

• target_type: code → dispatch pipeline-tooling-detector to auto-fill measure_command, correctness_command, and change_space from project manifests.
• target_type: container | llm_hyperparams | build_pipeline | system | generic → propose defaults from domain knowledge. For any domain outside confident coverage, dispatch pipeline-researcher (with WebFetch / WebSearch) for current best practice before drafting. Mark any field you genuinely cannot propose ``.
• target_type: system → keep one end-to-end metric and make change_space a list of objects, each tagging {class, domain, paths}, so the run can attribute the metric across components and follow the bottleneck as it moves between domains.

Contract template:

target_system:       ""
target_type:         code        # code | container | llm_hyperparams | build_pipeline | system | generic
metric_name:         
metric_unit:         
metric_direction:    minimize    # minimize | maximize
target_value:        
baseline_value:      null        # filled by Phase 1
measure_command:     ""
measure_reps:        10
measure_warmup:      3
noise_ceiling_cv:    0.10        # hard-block if the measured CV exceeds this
correctness_command: ""   # or the literal: manual
change_space:        []   # objects {class, domain, paths} for target_type: system
scope:               working     # working | branch | global
iteration_cap:       6

This orchestrator (not a subagent) presents this gate. This step is interactive and must not be delegated.

Pre-read: Read .mz/task//contract.md with the Read tool and capture its full contents into context.

Surface 1 — emit the plan message. Output the contract verbatim as a normal markdown chat message. Emit the full verbatim contents of .mz/task//contract.md — do not substitute a path, summary, or placeholder. Structure:

## Contract ready for review — optimize

The contract below fully specifies this run: target, metric, the measurement and correctness commands, the allowed change space, and the iteration cap. Every later phase reads it; no measurement runs until you approve it.

---
**Approve** → proceed to Phase 1 (Baseline)  ·  **Reject** → mark task aborted, no measurement runs  ·  reply with feedback to revise

Reminder: quiesce the measurement host before approving — background load inflates measurement noise and can hard-block the baseline.
Approve cost (estimated): baseline + profile ≈ 3 agents (test-runner, measure-runner, researcher) × ~18k tokens ≈ ~$ on mixed Sonnet + Haiku

Use the shared/approval-gate.md formula for the dollar estimate.

Surface 2 — call AskUserQuestion. A short selector — do not re-embed the contract in the question body, it lives in the plan message above:

• question: The contract above is ready for review.
• options: Approve — proceed to Phase 1 (Baseline) · Reject — mark task aborted, no measurement runs

Response handling:

• Approve → permitted only if every field is concrete (no ` remains). Update state, read phases/baseline.md`, proceed to Phase 1.
• Reject → set Status: aborted_by_user and stop.
• Any other reply (feedback) → apply the changes / supplied values, overwrite contract.md, return to this gate, re-read contract.md, and re-emit the entire plan message from scratch with the full new contents — never diff-only, since context compaction may have destroyed the user's memory of earlier iterations. Then re-present the selector. This is a loop — repeat until the user explicitly approves. Never proceed to Phase 1 without explicit approval.
• MZ_DEV_PIPE_AUTO_APPROVE=1 → per shared/approval-gate.md: skip the AskUserQuestion call, log auto-approved (unattended mode), and proceed — but only if no `` field remains; an incomplete contract can never be auto-approved.

Phase 3.5: Backlog / Strategy Approval

Approval gate, once per loop iteration. The user approves the strategy — which hypotheses to pursue and in what order — before any candidate code is written. This is where a cross-module or architectural strategy gets explicit sign-off.

This orchestrator (not a subagent) presents this gate. This step is interactive and must not be delegated.

Pre-read: Read .mz/task//backlog.md with the Read tool and capture its full contents into context.

Surface 1 — emit the plan message. Output the backlog verbatim as a normal markdown chat message. Emit the full verbatim contents of .mz/task//backlog.md — do not substitute a path, summary, or placeholder. Structure:

## Hypothesis backlog ready for review — optimize

Profiling found the dominant bottleneck. The backlog below ranks the falsifiable speedup hypotheses by impact × confidence ÷ effort; approving authorizes the candidate loop to pursue them top-down.

---
**Approve** → proceed to Phase 4 (candidate generation & verification)  ·  **Reject** → mark task aborted  ·  reply with feedback to revise

Approve cost (estimated): per hypothesis ≈ 4 candidate agents (Opus) + 4 serial measurements (Haiku) + 1 promotion (Sonnet) × ~22k tokens ≈ ~$

Surface 2 — call AskUserQuestion. A short selector — do not re-embed the backlog in the question body, it lives in the plan message above:

• question: The hypothesis backlog above is ready for review.
• options: Approve — proceed to Phase 4 (candidate generation & verification) · Reject — mark task aborted

Response handling: Approve → read phases/candidate_loop.md, proceed to Phase 4. Reject → Status: aborted_by_user, stop. Any other reply (feedback) → re-rank in Phase 3, overwrite backlog.md, return to this gate, re-read backlog.md, and re-emit the entire plan message from scratch with the full new contents. This is a loop — repeat until the user explicitly approves. Never proceed to Phase 4 without explicit approval.

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: DoctorMozg
• Source: DoctorMozg/claude-pipelines
• License: MIT

Install and usage instructions live in the source repository linked above.