# Hermes Autoresearch Loops > hermes-autoresearch-loops — Karpathy-style autoresearch loops for Hermes: autonomous propose→test→keep/revert cycles for config tuning, infra hardening, skill improvement, and post-update safety checks. - **Type:** Skill - **Install:** `agentstack add skill-atlasomnia-donna-starter-hermes-autoresearch-loops` - **Verified:** Pending review - **Seller:** [AtlasOmnia](https://agentstack.voostack.com/s/atlasomnia) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [AtlasOmnia](https://github.com/AtlasOmnia) - **Source:** https://github.com/AtlasOmnia/donna-starter/tree/main/skills/autonomous-ai-agents/hermes-autoresearch-loops ## Install ```sh agentstack add skill-atlasomnia-donna-starter-hermes-autoresearch-loops ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Hermes Autoresearch Loops Karpathy-style autoresearch pattern adapted for our setup: give an agent a concrete environment, a frozen evaluator, and a scalar metric; let it iterate autonomously via propose→test→keep/revert loops. Used for config tuning, infra hardening, skill improvement, and post-update safety checks. ## When to use Trigger on: - “autoresearch loop” / “Karpathy-style loop” - “self-improving agent” / “agent experiments overnight” - “continuous validation after updates” - “iterate on config/skills/scripts automatically” - Adapting Codex, Claude, or other agent-loop patterns to Hermes (e.g., “how can I do this Codex setup on Hermes”) - Any request to set up autonomous improvement on a bounded target with measurable outcomes. ## Core principles (from Karpathy’s 2024–2026 talks) 1) Context engineering: - What you put in the context window is your real design surface. - Tight, structured AGENTS.md/skills > clever prompt phrasing. - Use this skill when optimizing system prompts, skills, or delegation rules via experiments. 2) Autoresearch loop: - Agent edits code/config → runs short experiments → checks metric → keeps or reverts → repeats. - Human sets direction and constraints only; agent does the grind. 3) Decade of agents (jagged intelligence): - Models are strong but brittle; design for partial reliability, not perfection. - Enforce verification after external actions; break big tasks into small scopes with clear success criteria. 4) World models: - Maintain concise canonical reference docs so agents behave like they understand the environment. - Use this skill to iteratively improve those references via feedback loops. 5) Files over apps: - Small focused tools/scripts orchestrated by Hermes > monolithic “do everything” prompts. - Prefer scripted loops and cron jobs for recurring checks. ## How to set up an autoresearch loop (template) Use this as the blueprint whenever we implement a new loop. 1) Define target: - A concrete, bounded scope only (repo, directory, config subset). - Examples in our environment: - Dashboard-backend tests and linting. - apply-patches.sh robustness after Hermes updates. - Model/provider performance comparisons for non-critical tasks. 2) Define metric(s): - Must be scalar and unambiguous. Examples: - Test pass/fail count. - Lint error/warning count. - Startup time or latency (e.g., hermes status response). - Config alignment check (0 mismatches = good). 3) Define constraints: - What the agent is allowed to change (e.g., “only scripts under ~/.hermes/scripts”, “no changes to safety procedures”). - What it must never touch without explicit human approval. 4) Implement loop: - Use dev profile via delegation for reasoning-heavy loops. - Karpathy's actual pattern is NOT cron-based — the agent runs a continuous LOOP FOREVER until manually stopped, not discrete scheduled runs. The human decides start time only; then it iterates autonomously without pausing to ask "should I continue?" Cron jobs are only appropriate when you want bounded windows (e.g., "run 2am-6am") and even then the loop itself is continuous within that window. - Each iteration: - Propose a small patch (config change, refactor, guardrail). - Run evaluator (tests, hermes status, alignment checks). - If metric improves/holds and no regressions → keep; else revert. 5) Logging: - Log only concise summaries of accepted changes and metrics. Avoid chatty justifications. ## Launching repository improvement loops Use this pattern when the user names a worker profile and asks for repeated test-and-improve cycles on a code repository: 1. Honor the explicitly named profile; do not silently substitute the generic Dev profile. 2. Inspect the live Git state first. Start only from a clean tree unless the prompt explicitly identifies pre-existing changes that must be preserved. 3. Create a dedicated local branch for the experiment series. Keep accepted iterations as small local commits; never push, tag, publish, or release unless separately authorized. 4. Store controller prompts and loop logs under a local artifact directory such as `.hermes/`, then exclude that directory through `.git/info/exclude`. Do not modify the shared `.gitignore` merely to hide one operator's orchestration artifacts. - When using the reusable `hermes-autoresearch` harness, remember its concrete Git semantics: accepted changes are committed locally in the target named by `repo_path`, on whatever branch is currently checked out, with a message such as `autoresearch: accept trial N score=S`. The harness does not create a branch or push. - The harness records `runs/experiments.tsv` and `runs/experiments.jsonl`, then stages accepted trials with `git add -A`. Before the first trial, exclude `runs/` in the target repository—prefer `.git/info/exclude` for operator-local runs—so experiment output is not accidentally captured in accepted commits. 5. Treat an iteration duration such as “about five minutes” as a timebox for one hypothesis—not as a fixed sleep. Each iteration should select one problem, define an acceptance condition, test, patch, evaluate, and KEEP or REVERT. 6. Freeze the evaluator before the first edit. Record baseline commands and metrics, then rerun the same release gate for every accepted code change. Focused tests may guide an iteration, but they do not replace the frozen gate. 7. Run the worker as a tracked background Hermes process with completion notification. Explicitly prohibit the child from self-backgrounding or spawning a competing editing agent. 8. Process liveness and exit code are not completion evidence. Verify actual tool activity, a substantive loop log, Git commits/diffs, and evaluator output. If a bounded Hermes session reaches its turn budget, resume the same session against the same branch rather than launching a fresh worker that repeats discovery. - Diagnose timeout layers separately: the harness's `command_timeout_seconds` is the hard per-proposal/per-evaluator subprocess deadline; `max_seconds`, Hermes profile gateway/turn settings, terminal-tool timeouts, and an outer controller deadline are independent clocks. A healthy direct model smoke does not prove a repository trial can finish inside the harness deadline. - The reusable harness times out the configured proposal wrapper but does not itself guarantee that grandchildren spawned by that wrapper are terminated. A timed-out wrapper can therefore leave an orphaned `hermes --profile … chat` process that continues targeting the campaign checkout and contaminates later trials. Proposal wrappers must launch each agent in its own process group/session, enforce an internal deadline shorter than `command_timeout_seconds`, terminate then kill the whole group on timeout, and the controller must scan for exact campaign descendants before advancing or retrying. - Do not turn the profile name into an unnecessary global lock. A Hermes profile may host concurrent independent sessions; serialize writers by exact worktree and shared mutable resources, not merely by profile name. An unrelated writer in another isolated repository is not a checkout collision, while separate profiles still do not make overlapping writers safe in one checkout. - For the timeout recovery sequence, process-group wrapper pattern, fresh-ledger relaunch, and profile-versus-worktree concurrency boundary, - For review-only workers that enter startup/context-compression churn, first preserve the failed attempt and verify the source tree, then retry with process-scoped isolation (`--safe-mode` or `--ignore-user-config --ignore-rules`), explicit direct provider/model flags, restricted toolsets, and bounded line-range reads. Do not change the profile globally merely to rescue one campaign. - Safe-mode file tools may resolve repository-relative artifact paths from the user home rather than the subprocess `cwd`. Pass absolute paths for controller reports, salvage drafts, completion markers, and handoff artifacts. - If a worker reaches its turn ceiling after substantive inspection but before contract formatting, preserve its output as a slot-specific salvage source and use a fresh closeout-only session to verify cited ranges and emit the required marker/report. This is a retry of the same logical iteration, not a new hypothesis. - Apply the same standard at launch: a running Hermes PID proves only that the controller process started. Do not say the harness is configured, the baseline is complete, or iterations are underway until the expected config/log artifacts exist and show real harness or worker activity. Report launch state precisely: `controller started`, `harness initialized`, `baseline complete`, and `trial N active` are separate milestones. - A harness process may exit `0` even though the logical slot failed. Treat `best_score=None`, fewer completed trials than requested, evaluator rejection, a missing exact completion marker, or a task-lost-after-compaction response as an incomplete slot. Preserve the attempt and retry that same logical iteration; do not count it or advance campaign numbering. - When the user pastes a finalized campaign prompt as the next message, treat that as authorization to execute it—not as another request to rewrite or summarize it. Perform prerequisite checkout/Git/exclusion checks, launch the named profile through a tracked process, then verify the first substantive milestone before giving a progress report. 9. Keep controller artifacts out of commits, and finish or revert the active experiment before closeout so the branch remains coherent. 10. Do not rely on one agent invocation to remain continuous merely because the prompt says “loop forever.” If it completes one valid experiment and exits, put repetition in an external tracked controller while constraining each invocation to exactly one hypothesis and one clean closeout. 11. When switching the named worker profile, stop and verify the old controller first, then inspect the tree and loop log. The replacement profile must finish or revert any `pending` experiment before proposing another; never let two profiles edit the same working tree concurrently. 12. Cancellation must terminate the **entire campaign process tree**, not merely the parent shell/controller PID. A stopped controller can leave the harness, proposal wrapper, and active `hermes --profile … chat` child orphaned and still editing. Inventory children by target config/repository and exact worker command, stop only that campaign's PIDs, and verify no matching harness/proposal/worker remains. Never kill unrelated work merely because it uses the same profile. 13. After cancellation or profile replacement, reconcile from live evidence before relaunch: inspect accepted commits, TSV/JSONL rows, ignored iteration artifacts, current HEAD, and dirty changes. Preserve accepted gated commits unless the user explicitly requests rollback; finish or revert the one pending transaction. Archive the prior campaign's ignored logs/artifacts, clear the active run logs, and freeze a **new baseline at the retained current HEAD** so the replacement evaluator cannot accept a regression merely because it still compares against the original lower score. 14. Editing a config file does not change a harness process that already loaded it. Stop and verify the old harness before changing `max_trials`, profile commands, budgets, or evaluator inputs; otherwise the in-memory campaign can continue under the old settings while disk appears updated. 15. Treat Git state plus the durable loop log as the cross-profile handoff surface. Start a new session for the replacement profile rather than trying to resume a session owned by another profile. 16. A persistent parent controller does not emit completion notifications after successful child cycles because the parent remains alive. When reporting status, inspect the live controller, recent commits, clean/dirty Git state, and the durable loop log; explain this notification behavior so silence is not mistaken for inactivity. If per-cycle reports are required, add a separate read-only reporting mechanism rather than weakening the continuous controller. Process-supervisor completion alerts can also be stale or misleading when output is redirected; verify the OS PID/process tree and controller log before declaring a controller dead. Prefer keeping stdout attached (for example, pipe through `tee` with `pipefail`) so tracked completion remains observable. For Office add-ins, include the repository's full tests, type check, production build, manifest validation, proxy/script compilation, and safe host smoke test when available. Separate automated verification from deferred live-host validation rather than claiming host integration from mocks alone. ### Overnight live-host add-in campaigns When an autoresearch loop drives a live Office add-in while a separate model operates the host application: 1. Use a dedicated disposable workbook/document and an isolated campaign branch. Never point exploratory loops at personal, financial, payroll, claims, or other consequential data. 2. Keep four roles distinct: - persistent external controller; - application executor model connected to the live add-in bridge; - deterministic evaluator that reads host state and frozen command output; - bounded repair worker that handles one reproducible defect per invocation. 3. Require the host application, task pane, proxy/companion, machine-awake state, and live bridge/session identifier before the first trial. Treat reconnection as controller-owned lifecycle work: after a rebuild or add-in reload, obtain and verify the new bridge session before resuming. 4. Build a scenario corpus with explicit expected cells, formulas, number formats, tables, charts, comments, errors, and cleanup behavior. Record latency and recovery separately from correctness; do not collapse all evidence into an LLM-written score. 5. The executor and orchestrator must not grade themselves. Re-read the workbook through the bridge, inspect exact tool transcripts, and run the frozen repository gate. Screenshots are supporting evidence, not a substitute for workbook state. 6. Classify failures before editing: bad test expectation, model/executor mistake, bridge/session failure, unsupported host capability, or reproducible product defect. Only the last category opens a repair transaction. 7. A repair iteration may patch only the isolated branch, run focused RED→GREEN evidence plus the complete add-in gate, rebuild, reload the host integration, reconnect, and rerun the failing black-box scenario. KEEP only when both repository gates and live-host reproduction pass; otherwise REVERT. 8. Use bounded fresh worker invocations under the persistent controller. Cloud coding models such as an available Codex Spark route may outperform local models for repair/orchestration; local Qwen-class workers are useful unlimited-runtime fallbacks. Define quota/provider fallback before launch rather than changing models ad hoc mid-transaction. 9. Store concise JSONL/TSV plus a morning Markdown report: scenarios attempted, pass/fail counts, latencies, bridge interruptions, reproduced defects, accepted/reverted commits, deferred host checks, and exact external gates. Never push, publish, release, or deploy from the overnight loop without separate authorization. For the complete cheap-model campaign controller—including cross-repository profile-slot queuing, the harness first-trial baseline pitfall, ignored completion markers, three-cycle dry-run gating, locked-host downgrade, and frozen test/build/audit evaluation—follow the controller steps in this skill. ### Finite multi-profile, multi-repository campaigns When the user specifies a fixed iteration count across several profiles and repositories: - make the total iteration/time arithmetic explicit before launch; - run one repository/profile block at a time—never concurrent editors on one tree; - inventory act … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [AtlasOmnia](https://github.com/AtlasOmnia) - **Source:** [AtlasOmnia/donna-starter](https://github.com/AtlasOmnia/donna-starter) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** yes - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: flagged — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-atlasomnia-donna-starter-hermes-autoresearch-loops - Seller: https://agentstack.voostack.com/s/atlasomnia - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.