Resilient Server Ops

Resilient Server Operations

Why this skill exists

Coding agents that execute work on remote servers face a fundamental asymmetry: the agent's session is fragile, but server-side processes are not. When the connection between agent and human drops, or the agent's own runtime crashes, any in-flight work on the server keeps running — but the agent loses track of what state things are in.

Without protective patterns, this asymmetry leads to:

• Half-written files — cat > target.conf interrupted mid-stream leaves a corrupt config
• Lost output — long-running command's stdout vanishes when the SSH pipe breaks
• Duplicate work — agent restarts from scratch because it can't tell what already finished
• Half-applied changes — multi-step operation completed steps 1–3 but step 4 never ran
• Untracked spawns — background process is running but the agent forgot the PID

The patterns in this skill prevent all of these. They cost a small amount of extra setup per operation in exchange for predictable, recoverable state regardless of what happens to the agent's session.

When to apply

There are two modes the agent should support:

Mode A — Per-task triggering (default)

Apply the patterns when the current task matches any of these:

• The user has indicated unstable connection ("weak signal", "weak connection",

"flaky wifi", "might disconnect", "traveling", "I'm on the train", etc. — these are examples, not an exhaustive list).

• The operation will take more than ~30 seconds (long enough for a connection blip to matter).
• Multiple agents may be touching the same server (their work shouldn't trample each other).
• The operation involves a critical path (production, deploys, anything where half-state hurts).
• Any time the operation involves multiple sequential steps on a remote server.

When in doubt, apply. The patterns add minimal overhead and large robustness.

Mode B — Session-wide activation (stateful flag): "weak connection protocol"

Some users want resilience on for every server op in a session, not just per-task. Watch for explicit activation phrases like:

• "activate weak connection protocol", "weak connection mode on"
• "I'm on a flaky network, apply resilience to every operation"
• "until I deactivate, treat every ssh as fragile"

The conventional name for this stateful mode is "weak connection protocol" — chosen because "signal" is ambiguous (Unix signals, messaging apps, audio signals). Stick to "weak connection" terminology in user-facing messages so the user sees consistent vocabulary.

When the user activates this mode, apply the patterns to all server operations regardless of per-task heuristics. Apply uniformly.

Do NOT second-guess the user's activation. Once active, the agent has zero discretion to skip the patterns. Specifically forbidden agent reasoning:

• ❌ "The connection seems stable right now, I'll skip the atomic write" — wrong. The user

activated the mode because they know their connection is variable. Current stability is irrelevant; the next packet may drop.

• ❌ "This is a short one-line edit, atomic write is overkill" — wrong. Apply uniformly.
• ❌ "I just tested the connection and it's fast, I'll run foreground" — wrong. The user

decided the protection is worth it. Respect that.

• ❌ "The patterns add overhead, the user probably wants speed" — wrong. They activated the

mode explicitly. Speed is not the priority while the flag is active.

Only the user can deactivate. The agent's only valid responses to "but the connection seems fine" are: (a) ignore it (continue applying patterns), or (b) ask "do you want to deactivate weak connection protocol?". Never silently drop protection on the assumption that the current network state changes the rule.

The reason this matters: users activate the flag when they know their context is risky (traveling, mobile network, flaky hotel wifi, hot-spotting from a phone in motion). The current network state at any single moment is not representative of the session as a whole. Agents that "optimize" by skipping protections during stable moments leave users with half-applied changes when the next drop hits.

Deactivation requires an explicit phrase:

• "deactivate weak connection protocol", "weak connection mode off"
• "connection is stable, you can stop the resilience overhead"
• "I'm back at home wifi, full reliable connection"

A casual remark like "connection is OK now" does not deactivate the flag — the user may just be reporting current network state, not lifting the rule. When in doubt, ask before deactivating.

Track activation status in your working memory for the session. If the agent runtime supports persistent memory (e.g., Claude Code memory files), recommend the user records the active flag there so it survives session restarts. A typical record:

WEAK-CONNECTION-PROTOCOL: ACTIVE since 2026-05-07.
Reason: user traveling, network unstable.
Deactivate phrases: "deactivate weak connection protocol", "weak connection off".

The five core patterns

1. Atomic writes — never write to the destination directly

Problem: A direct cat > /etc/conf or scp file user@host:/etc/conf can leave the destination half-written if the SSH pipe breaks mid-transfer.

Pattern: Write to a temp file in the same filesystem, then mv to the final location. The mv (rename) is atomic on POSIX filesystems — the file is either fully old or fully new, never a mixture.

# Bad — partial write possible
ssh host "cat > /etc/myapp.conf"  /tmp/myapp.conf.tmp && mv /tmp/myapp.conf.tmp /etc/myapp.conf" >$LOG 2>&1 &
          echo "PID=$!  LOG=$LOG"'

Hand the user (or yourself, on next session) the PID + log path. Recovery becomes:

# Is it still alive?
ssh host "ps -p "
# What's the latest output?
ssh host "tail -50 /tmp/myjob.log"

For very long jobs (hours), consider also writing a status file the job updates as it progresses (echo "phase 3/5" > /tmp/myjob.status) — gives observability without parsing log text.

3. Idempotency — every step must be safely repeatable

Problem: If a multi-step operation crashes halfway, the agent has to retry. Without idempotency, retry might break things (re-installing a service, double-applying a migration).

Pattern: Every step should be safe to run when its work is already done. Patterns that work:

| Operation | Idempotent form | |-----------|-----------------| | Create directory | mkdir -p (no error if exists) | | Write file (skip if exists) | [ -f ] \|\| cat > .tmp && mv ... | | Install package | command -v >/dev/null \|\| pip install | | Restart service | pkill -f 2>/dev/null; sleep 1; nohup & (kill is no-op if not running) | | Add config line | grep -qxF '' \|\| echo '' >> | | Build artifact | [ -nt ] && build (only build if source newer than artifact) | | Apply migration | Versioned migrations with state table (run only if version not yet applied) |

Avoid: commands that fail loudly on repeat (mkdir foo, useradd foo without id foo guard, echo X >> file without dedup check).

4. Incremental phases with checkpoints on disk

Problem: A single monolithic operation that runs for an hour leaves you stranded if it fails 50 minutes in. You don't know which sub-step succeeded.

Pattern: Break work into phases where each phase's output is persistent on disk at a known path. Each phase begins by checking whether its output already exists; if so, skip.

# Phase 1: download
ssh host '[ -f /data/raw/2024-01-01.csv ] || curl -o /data/raw/2024-01-01.csv https://...'

# Phase 2: transform
ssh host '[ -f /data/transformed/2024-01-01.parquet ] || \
          python transform.py /data/raw/2024-01-01.csv /data/transformed/2024-01-01.parquet'

# Phase 3: load
ssh host '[ -f /data/.loaded/2024-01-01 ] || \
          (psql -f load.sql && touch /data/.loaded/2024-01-01)'

After interruption, just rerun the whole sequence. Already-completed phases skip themselves.

For rollback: keep the previous version's output around. If phase N produces wrong data, deleting its output and rerunning regenerates it.

5. Recovery — leave breadcrumbs the next session can follow

Problem: After interruption, the new agent (or human) needs to know what state the server is in. If the previous session left no trace, recovery requires re-deriving everything from scratch.

Pattern: When kicking off any non-trivial server work, record the contract somewhere durable:

• PID file for background processes: /tmp/.pid
• Log file at a known path: /tmp/.log or /var/log/.log
• Status file for phase tracking: /tmp/.status updated as phases complete
• Lock file to prevent concurrent runs: flock on /tmp/.lock

Then output to the user (or chat) the contract:

> Started myjob (PID=12345, log=/tmp/myjob.log). To check: ps -p 12345, > tail -f /tmp/myjob.log, cat /tmp/myjob.status.

Future agent sessions reading this can re-attach to the work without guessing.

Task-type-specific guidance

Different operation types need slightly different applications of the patterns:

File writes / config edits

• Always use atomic writes (pattern 1)
• For non-trivial edits, write the new file via temp + mv. Don't use sed -i directly on

production files — sed -i writes to a temp and mv internally, but only on Linux GNU sed. For portability, do it yourself.

• For multi-line edits, prefer "render full new file then mv" over "apply N small in-place

edits". The first is atomic at the file level; the second can leave you in a half-edited state if interrupted.

Code deploys / patches

• Phase 1: write the new code to disk (atomic write to target)
• Phase 2: validate (syntax check, lint, tests) before activating
• Phase 3: restart/reload the service (kill via supervisor, NOT manual nohup if a watchdog

exists)

• After deploy, verify the new code is actually running (timestamp, version endpoint, log

marker). Don't trust that "the deploy command succeeded".

Package installs (pip, npm, apt)

• Idempotent by default if you use the right form: pip install -U pkg, `apt-get install -y

pkg` (apt is idempotent for already-installed packages)

• For specific versions: pin them. pip install 'pkg==1.2.3' rather than pip install pkg.
• For long installs (compiling from source), wrap in nohup + log even if the install seems

short — system packages can take minutes when servers are loaded.

Data fetches / downloads

• Always download to a .tmp path, then mv when complete (atomic write — pattern 1)
• For very large files, use tools that can resume (wget -c, rsync --partial, aws s3 cp)
• Consider checksum verification after download to detect silent truncation

Batch processing pipelines

• The incremental phases pattern (pattern 4) is critical here
• For each input record / file / partition, write a marker file when processing is complete
• On rerun, skip records whose marker already exists
• Periodically (every N records) flush state to disk so a crash doesn't lose accumulated work

Service restarts

• If a watchdog/supervisor (systemd, supervisord, custom cron) manages the service, **never

manually re-launch** — the watchdog will spawn a duplicate. Only kill and let the watchdog do the restart.

• Verify the watchdog actually restarted the service: ps aux | grep , check listen port,

check log for fresh startup line.

Git operations

• git push is mostly idempotent (re-pushing the same ref is a no-op)
• git commit is NOT idempotent — re-running creates a duplicate commit. Use `git commit

--allow-empty` only when explicitly desired.

• For long commits with many files, consider staging in batches and committing once. If

interrupted between staging and commit, the index is preserved on disk and can be resumed.

• git pull can fail mid-merge and leave the working tree in conflict state. Recovery: check

git status, decide rebase/merge/abort.

Anti-patterns (avoid)

| Don't | Why | |-------|-----| | ssh host "long-command" foreground | Dies on SSH disconnect, lose all output | | cat > /etc/critical-file directly | Half-write on interruption corrupts config | | mkdir foo && cd foo && do-stuff | If mkdir fails (already exists), && short-circuits | | Manual nohup service.py & when watchdog exists | Causes duplicate processes | | echo "$LINE" >> file for config | Re-running adds duplicates; always grep-check first | | Long batch with no phase markers | Crash at 90% means restart from 0% | | Spawning background job without recording PID | Can't kill or check status later |

Recovery checklist when resuming after interruption

When picking up a session after disconnect / agent crash:

1. Check known PIDs — if previous session announced PID=12345 LOG=..., run

ps -p 12345 and tail -50 to assess state.

1. Check status / lock files — cat /tmp/.status, ls /tmp/*.pid.
2. Verify last-known-good state — if the operation was "deploy code", check the running

service version. If it was "fetch data", check /data/raw/ for completed files.

1. Plan resume — for incremental-phase pipelines, just rerun. Phase guards skip

completed work. For monolithic ops, decide: was it idempotent? If yes, rerun. If no, inspect carefully.

1. Don't blindly retry destructive ops — rm -rf, DROP TABLE, git push --force. If

you're unsure whether the previous run completed, ask first.

When NOT to use these patterns

These patterns add overhead. For one-off, short, read-only operations on a stable connection, they're noise:

• ssh host "cat /etc/hostname" — read-only, instant, no need for atomic writes
• ssh host "ls /var/log" — pure observation
• Small interactive debugging sessions where you'll see errors immediately

Apply judgment — but err toward applying the patterns. The cost is small.

Combining with other skills

This skill composes well with:

• subagent-driven-development: when delegating server work to a subagent, the subagent

should also follow these patterns. The subagent's session is also fragile.

• systematic-debugging: when debugging a partially-applied operation, the recovery

checklist (above) guides what to check first.

• executing-plans: when executing a multi-phase plan that touches a server, structure each

phase to be re-runnable per pattern 4.

Summary — the 5-second mental model

When about to issue a remote command, ask:

1. If the connection drops mid-command, what state is the server in? If the answer involves

"depends" or "half-written", apply atomic writes or background-with-log.

1. How will I know what happened? If the answer is "I'll see the output on stdout" — that

only works if the SSH pipe stays open. Capture to a log file instead.

1. If I rerun this in 10 seconds, will it succeed cleanly? If no, refactor for idempotency

before issuing.

Three checks. Two seconds each. Massively reduces the cost of any interruption.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mirosing
• Source: mirosing/resilient-server-ops
• License: MIT

Install and usage instructions live in the source repository linked above.