Long Running Agent Sessions

Long-Running Agent Sessions

Overview

A long task is reliable only when progress and authority survive outside a model's transient context. Keep the smallest mechanism that addresses the actual failure: an in-session task needs disciplined checkpoints; an interruptible task also needs durable recovery; genuinely parallel work may need a workflow engine. These are teaching levels defined by this Skill, not an industry standard.

This Skill is runtime-neutral. It describes contracts an implementation must satisfy and uses only a fictional local fixture. It does not promise that a prompt, scheduler, lease file, or idempotency key alone creates exactly-once execution. Public grounding and the limits of each source are documented in references/public-foundations.md.

When to Use

Use this Skill when work:

• spans many independently verifiable steps;
• should proceed without asking for routine “continue” messages;
• may lose its process, session, or scheduler invocation;
• can produce externally visible or non-repeatable effects;
• can be started by more than one runner; or
• must honor fresh cancellation or human approval.

Do not use it to bypass permission, safety, policy, budget, or human-decision boundaries. Short read-only tasks usually need neither durable state nor a recovery scheduler.

Choose a Teaching Level

| Level | Problem solved | Required mechanism | |---|---|---| | Level A — checkpointed session | Context drift and premature stopping while one session remains alive | bounded steps, explicit done definition, progress checkpoint, continuation rule | | Level B — resumable run | Session/process interruption | Level A plus durable state, effect reconciliation, cancellation refresh, and a recovery trigger | | Level C — coordinated workflow | Proven parallelism, dependencies, or distributed ownership | Level B plus an appropriate orchestration system and its verified semantics |

Task duration alone does not justify Level C. A scheduler can start a recovery attempt, but it is not the workflow state of record.

Build the Run Contract

Before execution, fill templates/run-contract.md.

1. Pin purpose and completion. State task identity, goal, non-goals, allowed actions, forbidden actions, and evidence required for completion.
2. Slice by verifiable artifact. Each step produces one checkable artifact or decision and names exactly one next action.
3. Grant bounded continuation. Routine verified steps continue automatically. Cancellation, missing approval, unsafe scope, exhausted limits, unrecoverable state, or a real external blocker stops the run.
4. Externalize the pointer. Persist the plan digest, current step, completed steps, evidence, unresolved effects, and next action after every meaningful transition.

Completion criterion: a fresh runner can determine one safe next action without relying on chat history.

Durable Progress State

Use a versioned envelope containing at least:

• task ID, schema version, plan digest, monotonic revision, and timestamps;
• status, current step, completed steps, and exact next action;
• authorization, cancellation, and approval observations with their source and time;
• per-effect idempotency key, payload digest, lifecycle state, attempts, and evidence;
• retry/resource budgets that are not reset by restart;
• lease owner, expiry, and fencing epoch/token when concurrent starts are possible.

Write a complete candidate state beside the destination, flush when required by the durability target, then replace atomically using storage semantics you have verified. Keep a validated backup or journal. Quarantine malformed state; recover only from validated state plus observed artifacts. Never silently create a new run over damaged state.

Completion criterion: truncating the primary state either recovers from validated evidence or stops state-unrecoverable without repeating an unknown effect.

Idempotent Resume and Effect Reconciliation

Assign each logical effect a stable semantic key and payload digest. Persist transitions around the effect:

planned -> started -> acknowledged -> verified
                 \-> uncertain -----> verified

• Persist started before invoking the adapter.
• Treat a response as acknowledgement, not proof.
• After timeout or interruption, mark or infer uncertainty.
• On resume, inspect receipts, target state, output digest, or another independent observation before retrying.
• Retry only when no prior effect is observed or the adapter provides deduplication for the same key and payload.
• Declare completion only from verified effects.

An idempotency key is a correlation handle, not a guarantee. The target adapter must define key scope, retention, conflicting-payload behavior, and verification.

Completion criterion: interruption after an effect but before checkpointing does not create a second logical effect on resume.

Lease, Concurrency, and Fencing

When duplicate starts are possible:

1. acquire one task-scoped lease atomically;
2. record owner, expiry, and a monotonically increasing fencing epoch or equivalent token;
3. reject a second live owner as already-running;
4. renew only while ownership still matches;
5. after lease loss, stop before any further state or effect write;
6. permit takeover only after expiry and reconciliation of state and artifacts.

A local lock can demonstrate exclusion but is not proof for distributed storage. If stale writers can reach the target, the state/effect adapter must reject writes carrying older fencing epochs; otherwise state that the implementation provides best-effort exclusion only.

Completion criterion: a live duplicate performs no effect, and an expired-owner takeover uses a newer token before resuming.

Fresh Cancellation and Manual Approval

Check cancellation and current authority before every new effect and again after recovery. Reconcile an in-flight effect even when cancellation is present, but do not begin another one.

For a manual gate, persist a request describing task ID, plan digest, step ID, payload digest, requested action, and expiry. Accept approval only when all scope fields match the pending action and the approval is still current. Missing, stale, ambiguous, or broader-than-requested approval yields awaiting-approval or unsafe-to-continue.

Completion criterion: cancellation prevents the next effect, and approval for one plan/step cannot authorize another.

Bounded Retry and Stuck Work

Classify errors before retry. Retry only transient classes with:

• a persistent attempt counter and maximum;
• capped exponential backoff with bounded jitter;
• per-operation deadlines;
• task-wide elapsed, cost, and step limits; and
• a no-progress threshold based on durable revision/effect age.

Do not reset counters after restart. Reconcile a stale started effect before classifying it as retryable. Limit exhaustion is a recorded terminal or blocked outcome, never an infinite recovery loop.

Completion criterion: repeated injected transient failures stop at the declared maximum across separate invocations.

Evidence Boundaries

Separate:

• observations — state bytes, receipts, output hashes, timestamps, and control records actually read;
• inferences — conclusions supported by those observations;
• unknowns — unverified external outcomes, stale authority, ambiguous ownership, or missing evidence.

Do not infer “still running” from a missing stop marker, “completed” from a success response, or “safe to retry” from silence. Record the evidence source and digest where practical. Unknowns that could duplicate or exceed authority stop the run.

Recovery Procedure

1. Acquire the task lease or return already-running.
2. Validate schema, task identity, plan digest, revision, and backup/journal.
3. Reconcile every started, acknowledged, or uncertain effect from observable evidence.
4. Refresh cancellation, authorization, and scoped approvals.
5. Enforce attempt, elapsed, cost, step, and no-progress limits.
6. Select one authorized action; persist intent before execution.
7. Execute through an adapter, then persist acknowledgement/uncertainty and independent verification.
8. Continue routine steps until done or a legal stop; release only after durable state.

Public Foundations

The protocol is grounded in public material on idempotent retry, leases, durable workflow state, explicit cancellation lifecycles, and scoped human approval. See [references/public-foundations.md](references/public-foundations.md) for exact sources and scope/licensing cautions.

The cited products are implementation evidence, not required dependencies or universal guarantees. This Skill's Level A/B/C model and synthetic runner are newly authored teaching material; they do not claim formal conformance or distributed exactly-once execution.

Synthetic Verification

Run the local-only standard-library fixture:

PYTHONDONTWRITEBYTECODE=1 python -B -m unittest discover -s tests -v

The suite uses temporary directories. It verifies interruption/resume, repeat-run idempotency, uncertain-effect reconciliation, live-lease exclusion, expired-lease takeover, cancellation, scoped approval, bounded retries across restarts, and truncated-state recovery.

Common Pitfalls

1. Prompt equals runtime. A prompt can prescribe checkpoints; it cannot survive a dead process by itself.
2. Checkpoint equals truth. Compare durable state with artifacts and receipts after interruption.
3. Key equals exactly once. Verify the target's deduplication and retention semantics.
4. Lease equals fencing. Exclusion without stale-write rejection may still allow late writes.
5. Old approval equals current authority. Re-check scope, digest, expiry, and cancellation.
6. Retry equals recovery. Reconciliation comes before retry; all retries remain bounded.
7. Acknowledged equals completed. Completion requires declared verification evidence.

Verification Checklist

• [ ] A fresh runner can identify exactly one safe next action.
• [ ] State is versioned, monotonic, atomically replaced, and recoverable or fail-closed.
• [ ] Every effect has a semantic key, payload digest, lifecycle, attempts, and evidence.
• [ ] Resume reconciles uncertain effects before retry.
• [ ] Lease loss prevents further writes; takeover uses a newer fencing value.
• [ ] Cancellation and scoped approvals are refreshed before new effects.
• [ ] Retry and task budgets survive restart and terminate predictably.
• [ ] Completion claims cite observed verification evidence and preserve unknowns.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: SorryDancer
• Source: SorryDancer/agent-skill
• License: MIT

Install and usage instructions live in the source repository linked above.