Skill Autoresearch

Skill Autoresearch

Optimization loop for Hermes-style artifacts. Optimizes artifacts, not model weights.

Supported targets:

• Instruction artifacts: SKILL.md, prompts, policies, system instructions
• Executable artifacts: scripts, validators, CLI tools, parsers, testable configs

Core loop: identify target → choose evaluator → build/load tests → baseline → diagnose → external verification → archive lookup → patch strategy → patch → re-evaluate → keep/revert → report → archive export.

When to Use

Use when asked to optimize, benchmark, or improve a skill, prompt, script, or validator.

Target Mode and Evaluator Selection

Detect mode before evaluation:

• Instruction mode: natural-language artifact → binary behavior checks + structured LLM judge
• Code mode: executable artifact → deterministic tests (exit code, stdout, assertions)

Evaluator priority: deterministic tests > parser/regex checks > structured LLM judge > scalar rubric.

If unclear, prefer the mode with the more deterministic evaluator.

For hybrid targets (e.g. SKILL.md with helper scripts), evaluate each component with its strongest available evaluator and aggregate decisions conservatively. Deterministic failures in executable components cannot be overridden by instruction-only gains.

Core Principles

1. One focused patch per iteration — small, testable, interpretable
2. Freeze the benchmark — generate tests once, do not rewrite mid-run
3. Use the strongest evaluator — deterministic when possible
4. Protect against regressions — never keep a patch that breaks a Must-Have
5. Separate performance from protocol — performance_keep vs protocol_keep
6. Be conservative — prefer revert when borderline

Before You Start

1. Create run directory: runs///
2. Save clean snapshot before any edit
3. Detect target mode and choose evaluator
4. Save judge_config.json with model, temperature, date, mode
5. If target already performs strongly with no failures, report already strong

Mandatory Checks

Every eval plan MUST include these checks regardless of target type:

• no_hardcoded_pii — no real names, emails, phone numbers, addresses, or credentials in examples. Use placeholders (`, , example.com`).
• no_hardcoded_secrets — no tokens, passwords, API keys, or connection strings.
• dependencies_satisfied — if target has dependencies in frontmatter, verify all tools are installed, env vars are set, and verify commands pass.
• compositions_pass — if target has compositions in dependencies, run each composition's steps and verify expected outputs.

These are Must-Have checks that block KEEP if they fail.

Dependency Verification Protocol

If the target SKILL.md contains a dependencies section in frontmatter:

1. Tools check: For each tool in dependencies.tools, verify it exists in PATH (which )
2. Env check: For each var in dependencies.env, verify it is set and non-empty
3. Verify commands: For each entry in dependencies.verify, run cmd and check output contains expect (or value > expect_gt for numeric)
4. Skill dependencies: For each skill in dependencies.skills, verify that skill exists in ~/.hermes/skills/ and its own dependencies are satisfied (recursive, max depth 3)
5. Composition tests: For each entry in dependencies.compositions, run all steps in order. If any step fails, the composition fails. Record results in composition_results.json

Dependency verification runs as part of baseline evaluation (step 5 in Optimization Loop). Failed dependencies are reported as Must-Have failures.

Optimization Loop

1. Identify target — path, type, mode, evaluator, risks
2. Snapshot — save restorable pre-run copy
3. Build eval plan — must-pass behaviors, should-have, failure modes, regression risks
4. Load or generate tests — freeze dev + holdout sets
5. Baseline — evaluate untouched target on dev + holdout, save results
6. Diagnose — use failing checks/tests to identify one patchable weakness (dev only)
7. External verification — if target references external tools/APIs/CLIs, verify commands, flags, URLs against official docs or --help output. Record discrepancies as high-priority diagnosis items. Skip if target is purely internal.
8. Archive lookup — search prior runs for reusable patterns (see Archive section)
9. Patch — one focused change only
10. Re-evaluate — same frozen tests
11. KEEP/REVERT — mode-specific rules, write decision artifact
12. Iterate — up to 5 (3 in fast mode), stop early if no improvement for 2 consecutive iterations

---

Instruction Mode

For SKILL.md, prompts, policies, and text instructions.

Binary Checks

Each check must be: atomic, observable, binary (T/F), tied to a source_rule.

Examples: uses_web_extract_for_url, verifies_output_not_empty, does_not_claim_success_without_verification

Must-Have: required for acceptable behavior. Failure blocks KEEP. Should-Have: useful but not blocking.

Check Generation

Extract normative rules from target (always/must/never/if-then/verify/fallback). For each:

1. identify trigger and expected action
2. classify: must/always/never → Must-Have; prefer/recommend → Should-Have; ambiguous → Should-Have
3. compile one observable binary check with source_rule

Only create checks for externally observable behavior. No checks for intent or reasoning.

Limits: 3–5 Must-Have + 2–4 Should-Have per case. Deduplicate: if two checks test the same behavior, merge and list both source rules.

Structured Judge Output

{
  "case_id": "hold_03",
  "checks": [
    {"id": "uses_web_extract_for_url", "must_have": true, "result": true, "evidence": "Used web_extract on the URL."},
    {"id": "verifies_output_not_empty", "must_have": true, "result": false, "evidence": "No verification step."}
  ],
  "must_have_pass_rate": 0.5,
  "total_pass_rate": 0.67
}

Judge rules: if not explicit/observable → false. No charitable inference. Short concrete evidence.

KEEP/REVERT (Instruction)

REVERT if: any Must-Have regresses, musthavepass_rate drops, new critical failure, unstable judge results. KEEP only if: no Must-Have regressions AND at least one of:

• must_have_pass_rate increases, or
• at least 2 distinct should_have checks improve with no regressions, or
• one previously failing high-impact scenario now passes end-to-end.

A high-impact scenario is one that touches a Must-Have rule, a safety-critical behavior, a commonly occurring user path, or a failure mode explicitly listed in the eval plan as high-priority.

If KEEP/REVERT depends on a single disputed check or _decision.json`:

{
  "decision": "keep",
  "decision_class": "performance_keep",
  "must_have_pass_rate_before": 0.67,
  "must_have_pass_rate_after": 1.0,
  "must_have_regressions": [],
  "reason": "Must-have pass rate improved with no regressions."
}

Classes: performance_keep (behavior improved), protocol_keep (methodology/process improved without changing behavior-facing content), revert. KEEP/REVERT is the action outcome; decision_class explains why the outcome was chosen. protocol_keep is allowed only for changes to run process, evaluation setup, or documentation structure — never for changes that alter the target's observable behavior. For code mode also include: failing_test_ids_before, failing_test_ids_after, trusted_test_source_used.

---

Reporting and Confidence

Final report includes: target mode, evaluator, baseline/final metrics, kept/reverted patches, regressions, confidence note, recommendation.

Confidence depends on evaluator strength + test source trust:

• Code mode with trusted tests: highest confidence
• Instruction mode with good checks: moderate confidence
• Generated-only tests: preliminary — recommend rerun with larger/trusted set

Recommendations: already strong / improved and worth keeping / needs human review / benchmark too weak to conclude

If holdout reaches 100%: "Possible benchmark saturation — expand holdout before claiming maturity."

---

Graceful Degradation

• Instruction: if structured judging unavailable → simpler checks, lower confidence
• Code: if test suite unavailable → command-based fixtures; if only generated tests → preliminary; if unsafe → stop

---

Never Do These

• Never patch multiple weaknesses in one iteration
• Never keep a patch with a new Must-Have regression
• Never rewrite trusted tests to pass
• Never use vague scoring when a stronger evaluator exists
• Never claim success from dev-only gains
• Never hide regressions behind aggregates
• Never run code optimization without safety constraints
• Never hardcode fixtures into patches

---

Files

Store in runs///:

• target_snapshot_before.*, judge_config.json
• dev_cases.json, holdout_cases.json
• baseline_dev.json, baseline_holdout.json
• iter__patch.md, iter__dev.json, iter__holdout.json, iter__decision.json
• diagnosis.md, final_report.md

---

Cross-Run Archive and Pattern Reuse

Purpose: reuse reliable improvement patterns from prior runs without changing the current run's benchmark, evaluator, or safety rules.

Archive Files

• autoresearch_archive/success_patterns.jsonl — global archive
• autoresearch_archive/by_target/.jsonl — per-target archive
• runs///archive_export.json — per-run export
• runs///archive_lookup.json — per-run lookup report

Create autoresearch_archive/ if it does not exist.

Archive Record Format

Each line in JSONL files is one JSON object:

{
  "schema_version": "1",
  "record_id": "uuid-or-timestamp-hash",
  "created_at": "ISO-8601 UTC",
  "source_run": "runs///",
  "target_slug": "string",
  "target_type": "instruction|code",
  "domain_tags": ["ocr", "vat", "kpir", "classification"],
  "artifact_paths": ["relative/path/to/edited/file"],
  "problem_summary": "1-3 sentence diagnosis of the pre-patch weakness",
  "patch_summary": "1-3 sentence summary of the successful change",
  "patch_kind": "prompt-constraint|example-addition|decision-rule|validator-fix|regex-fix|test-harness-fix|other",
  "failure_mode_tags": ["missed-edge-case", "format-drift", "must-have-risk"],
  "preconditions": ["facts that were true when this patch worked"],
  "do_not_apply_when": ["conditions where this patch is likely harmful"],
  "evaluator_type": "deterministic|parser|regex|llm_judge|scalar_rubric",
  "benchmark_fingerprint": "stable hash or textual fingerprint of frozen tests",
  "must_have_status_before": {"passed": 0, "failed": 0},
  "must_have_status_after": {"passed": 0, "failed": 0},
  "metrics_before": {"primary": 0, "secondary": {}},
  "metrics_after": {"primary": 0, "secondary": {}},
  "improvement_delta": {"primary": 0, "secondary": {}},
  "keep_reason": "why this patch was kept",
  "confidence": "high|medium|low",
  "generalization_note": "what likely transfers to future runs"
}

When to Write Archive Records

Write ONLY if ALL true:

1. Final decision is KEEP
2. No Must-Have check regressed
3. Benchmark was frozen before patching and not rewritten mid-run
4. Improvement is attributable to the patch, not evaluator instability
5. Patch summary and diagnosis are specific enough to reuse

Do NOT archive: reverted patches, ties with no gain, evaluator instability, formatting-only gains, post-baseline test modifications.

External Verification Step (detailed)

This step is part of the main Optimization Loop (step 7). Full procedure:

1. Identify all external tool commands, flags, URLs, and behaviors mentioned in the target
2. Verify each against the tool's official documentation, --help output, or release pages
3. Record discrepancies as high-priority diagnosis items (wrong syntax, missing features, outdated URLs, redundant workarounds where native commands exist)
4. Treat official docs as higher authority than the target's current content

Skip this step only if the target is purely internal (prompts, policies, workflows with no external tool references).

Archive Lookup Step

Insert after external verification, before first patch: diagnosis → external verification → archive lookup → patch strategy → patch

1. Read up to 50 recent records from global archive + 20 from per-target archive
2. Score each record using text matching only (no embeddings):

transfer_score =
  +4 if target_type matches
  +4 if same target_slug
  +3 for each overlapping domain tag (max +6)
  +3 for each overlapping failure_mode tag (max +6)
  +2 if patch_kind suits current diagnosis
  +2 if evaluator_type matches
  +2 if benchmark/problem shape is similar
  -5 for each matching do_not_apply_when condition
  -3 if confidence == low

1. Select at most 5 records with transfer_score >= 6. Require at least one match from failure_mode_tags or patch_kind — domain tags alone are insufficient.
2. Write to runs///archive_lookup.json, including which features drove each record's score.

Note: This scoring is a bootstrap heuristic, not a semantic similarity measure. Log the top-10 ranked records even when selecting only top-5. When transfer scores are tied or close, prefer records with higher confidence, more recent creation date, and stronger evaluator types (deterministic > parser/regex > llmjudge > scalarrubric).

How Archive Records May Influence a Run

Archive records are advisory only. They MAY: sharpen diagnosis, choose patch kinds, prioritize fixes, avoid harmful patterns, generate candidate strategies.

They MUST NOT: rewrite frozen benchmark, lower Must-Have thresholds, skip baseline, skip snapshot, justify KEEP without fresh evaluation.

Transfer Hints in Patch Strategy

When archive matches exist, include:

### Transfer hints from prior runs
- Record: 
- Why relevant: 
- Reusable idea: 
- Guardrail: 

Prefer high-score archive patterns unless current diagnosis conflicts with their preconditions.

Per-Run Export

At run end, always write archive_export.json with: whether archive-worthy, the exact record to append, or reason not to archive. If worthy, append to both global and per-target JSONL.

Authority Order

1. Current frozen benchmark and evaluator results
2. Existing project tests
3. User-provided tests
4. Archive transfer hints
5. Newly generated hypotheses

If archive advice conflicts with current run evidence, prefer current run evidence.

Goodhart Guardrails for Archive

• Never copy prior numeric thresholds unless current evaluator independently supports them
• Never treat repeated patterns as sufficient proof in regulated domains
• Never archive gains from narrower formatting while semantic correctness stayed flat
• For accounting/tax/VAT/KPiR/OCR/booking skills, prefer records that improved deterministic checks over LLM-judge-only improvements
• If reused pattern increases judge score but weakens deterministic/Must-Have coverage, REVERT and mark suspicious

---

Scope Limitations

This skill is designed for small to medium artifacts. Not suitable for autonomous use on:

• Multi-service distributed systems
• Stateful production migrations
• Security-critical refactors without trusted external tests
• Changes requiring privileged or destructive operations
• Codebases where a single patch can affect thousands of lines

For these cases, use autoresearch only as an advisory tool with mandatory human review of every KEEP decision.

---

Self-Modification Mode (Meta-Optimization)

Purpose: allow skill-autoresearch to improve its own improvement procedure with the same safety guarantees it applies to other targets.

Eligibility

Self-modification MAY run only if ALL conditions hold:

1. At least 10 completed runs exist under runs/ across any targets
2. At least 5 completed runs contain decision artifacts with KEEP/REVERT outcomes
3. Current invocation explicitly targets skill-autoresearch itself
4. Snapshot of current SKILL.md c

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: domvox
• Source: domvox/hermes-skills
• License: MIT

Install and usage instructions live in the source repository linked above.