Health Check

Health Check

Report-first resource triage. Answers two questions: can we start `` right now, and what is the safest way to recoup headroom if not. Destructive reclaim is never automatic — only the regenerable build cache is pruned without asking.

This skill spans two scopes, and step 0 decides which one you are in. Docker inventory and every reclaim lever live on the machine running the daemon; memory, swap, disk and CPU can be read on either side but mean different things depending on where you read them. The sandbox has had no host Docker socket since #756 removed it as a host-root escape path, so a sandboxed invocation reports container-scope metrics and hands the Docker half to the orchestrator. Never present container figures as host figures — that is the failure mode this structure exists to prevent, and it is worse than reporting nothing, because a green container disk row reads like permission to start a multi-GB image build.

target is free text naming what you intend to start (a compose path, make dev, a service name). It sizes the verdict — a full docker compose build and a make dev against already-running services have very different footprints, so always pin down which before judging "sufficient."

Performance rules

Default to a fast path: one scope classification, one host snapshot, one Docker summary, one running-container list, and one exited-container list. Do not run verbose or per-container probes until the summary shows a likely binding constraint or the user asks for reclaim candidates.

• Gate expensive commands: docker system df -v, docker stats, docker exec ps ..., du, and nested-Docker inspection are second-pass diagnostics, not baseline checks.
• Run expensive diagnostics once and reuse the captured output. Never call docker system df -v inside a loop over volumes/images/containers.
• Prefer scoped reads over broad scans: explicit cache paths (~/.npm, project .pnpm-store) are acceptable; du -sh /home/* or filesystem-wide find is not.
• Keep reruns delta-oriented: if a prior health-check in the same thread already found the shape of the problem, recheck only the changed metrics unless the target changed.
• Never retry a Docker call that failed for want of a daemon. Step 0 settles reachability once. Nine copies of the same connection error is not diagnosis.

Instructions

0. Classify the scope — always first, before any other command

SNAP=${SNAP:-$(mktemp -d)}
bash .agro/skills/health-check/scripts/scope-preflight.sh | tee "$SNAP/scope.txt"

The script resolves the Docker endpoint (HEALTH_CHECK_DOCKER_SOCK, then DOCKER_HOST, then docker context, then /var/run/docker.sock, then the rootless path) and settles it with at most one round-trip. It always exits 0; a classification step that exits non-zero is the misleading failure this replaces.

It emits SCOPE, DOCKER_CLI, DOCKER_ENDPOINT, DOCKER_TRIAGE and METRICS_SCOPE.

> The KEY=VALUE lines select the branch. They are not the report. Surface the single HEALTH-CHECK SCOPE-NOTICE: line to the user and nothing else from this file. Pasting the preflight output as the answer is not a health check.

Branch on DOCKER_TRIAGE:

| Value | Meaning | What to do | |---|---|---| | available | The daemon answered | Run every step below as written. | | host-only | No endpoint here, or no CLI to reach one | Skip steps 2, 5, 7 and the reclaim ladder without issuing a Docker call. Report the notice, the scoped metrics, and the [host-side procedure](#host-side-docker-triage). | | unreachable | The endpoint exists and the daemon did not answer | Same skip. The one round-trip already spent is the whole diagnosis — do not retry per command. Report the notice; the endpoint may be a dead daemon or a socket this user cannot open. |

Do not test for the docker binary to decide this. /usr/bin/docker is installed in the sandbox and proves nothing about reachability — treating its presence as availability is precisely the defect #762 records.

1. Snapshot resources — label the scope you measured

Use one grouped command so the baseline is cheap and internally consistent:

SNAP=${SNAP:-$(mktemp -d)}
{
  echo "=== MEMORY ===" && free -h
  echo "=== SWAP ===" && (swapon --show || echo "no swap")
  echo "=== DISK ===" && df -h /
  echo "=== CPU ===" && nproc && uptime
} | tee "$SNAP/host.txt"

Read for: available memory (not "free" — buff/cache counts), presence of swap (no swap = no cushion), disk % used on /, and load vs core count.

Carry METRICS_SCOPE into every number you report. Under METRICS_SCOPE=container these are the container's cgroup and overlay figures. They answer container-local questions honestly — will uv sync OOM this container, is the workspace filesystem filling — and they do not answer what the Docker host has spare. Where a container's limits are unset, free -h reports the host's physical memory while the container cannot necessarily use it; say so rather than passing the number through.

Under METRICS_SCOPE=host, / is the binding number when Docker's data root is on the root filesystem. Confirm that rather than assuming it — docker system df in step 2 and df -h on the data root settle it. (An earlier version of this skill asserted the root overlay was always the binding number "on this host". That was written for one environment and is false wherever the daemon lives elsewhere or is not visible at all.)

2. Docker resource breakdown — host-only

> Requires DOCKER_TRIAGE=available. Under host-only or unreachable, skip this step entirely and point at the [host-side procedure](#host-side-docker-triage). Do not issue these commands to see what happens.

Keep the baseline to three Docker calls:

SNAP=${SNAP:-$(mktemp -d)}
docker system df | tee "$SNAP/docker-df.txt"
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}' | tee "$SNAP/docker-running.txt"
docker ps -a --filter status=exited --format 'table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Size}}' | tee "$SNAP/docker-exited.txt"

docker system df splits usage into Images / Containers / Local Volumes / Build Cache with a RECLAIMABLE column — that column is the entire reclaim opportunity at a glance. Only if the summary shows material reclaim or the user asks for a ranked cleanup list, capture verbose detail once:

docker system df -v > "$SNAP/docker-df-v.txt"

Read candidates from $SNAP/docker-df-v.txt; do not rerun docker system df -v per object.

If $SNAP/docker-running.txt shows a Docker-in-Docker container (common names: *dind*, ci-runner-dind), inspect only that nested daemon before concluding the reclaim plan:

docker exec  docker system df
docker exec  docker ps -a --size --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}\t{{.Image}}'

Nested CI sidecars often hide the best disk win: unused inner CI images and build artifacts inside the sidecar volume. Skip this branch when there is no DIND container. If ongoing cleanup is requested, prefer an idle-aware spindown or watchdog over repeated manual pruning — a lever that runs on a schedule beats one that needs an operator to remember it.

3. Size the target

Match the verdict to what target actually does:

| Target shape | Dominant cost | Watch | Answerable under host-only? | |---|---|---|---| | docker compose build / fresh stack | Disk — new images + build cache (multi-GB) | Disk %; build cache balloons | No. Image and cache growth land on the Docker host, which is not visible. | | make dev / run against live services | Memory + small disk (.venv, node_modules) | Available mem, no-swap spikes during uv sync / npm install | Partly — the container-local memory and workspace disk cost is measurable here. | | Pull-only (compose up, no build) | Disk — pulled image layers | Disk % | No. Layers land on the Docker host. |

Verify assumptions instead of trusting them, but scope the verification. If the user says "services are already running," confirm against the already-captured container list first (or, under host-only, against the ports themselves — a reachable port is evidence a service is up even when the container list is not available), then check only named/known ports and dependency dirs:

PORTS=""
for p in $PORTS; do (echo > /dev/tcp/127.0.0.1/$p) 2>/dev/null \
  && echo "port $p OPEN" || echo "port $p closed"; done
TARGET_DIR=""
[ -d "$TARGET_DIR" ] && du -sh "$TARGET_DIR/.venv" "$TARGET_DIR/node_modules" 2>/dev/null || echo "deps not yet installed"

A closed port or missing .venv/node_modules is a finding worth surfacing — it changes "ready to start" into "ready to provision, then start." If no ports or target directory are named, skip this probe rather than guessing.

4. Reclaim ladder (safest → most destructive) — host-only

> Every lever here acts on the Docker daemon's storage and requires DOCKER_TRIAGE=available. Under host-only or unreachable, propose nothing from this table as something you can do — hand over the [host-side procedure](#host-side-docker-triage) instead. There is no in-container substitute; a sandboxed agent cannot free a byte of the host's image store.

Walk the ladder top-down. Run tier 1 freely. Stop and confirm before tiers 2–4.

| Tier | Lever | Yield | Risk | Gate | |---|---|---|---|---| | 1 | docker builder prune -f | Often the biggest single win (regenerable cache) | None — cache rebuilds, only costs future build time | Run freely (skip if --dry-run) | | 2 | Remove exited/abandoned containers (docker rm ) + their images | High (stale sandboxes hoard GBs) | Destructive — data in the container is gone | Confirm; never auto-remove a container you didn't create | | 3 | Dangling images (docker image prune -f) | Medium | Low — only untagged layers | Confirm | | 4 | Orphaned volumes (docker volume prune / named docker volume rm) | Usually small, occasionally large | Destructive — may hold DB state | Confirm each named volume individually |

Rank candidates by safety × yield, not yield alone. The default action of this skill is tier 1 only; everything below is a proposal.

Identifying tier-2 candidates — use the baseline file for exited containers from torn-down sandboxes, not in the active set:

cat "$SNAP/docker-exited.txt"

Cross-check names against $SNAP/docker-running.txt before proposing removal. An "Exited (255) N days ago" sandbox that isn't one of the live ones is the prototypical safe-to-reclaim target — but still confirm, per the don't-delete-what-you-didn't-create rule. If size/yield is unclear, parse the single verbose snapshot ($SNAP/docker-df-v.txt) rather than issuing new Docker queries in a loop.

5. Per-container RAM reclaim — host-only

> Requires DOCKER_TRIAGE=available. This step reads resident memory across containers from the daemon's side; it is not something a container can do for itself. Under host-only or unreachable, skip it. For the RAM inside this container, step 1's figures plus a plain ps -eo rss,args --sort=-rss are the honest local equivalent, and killing a stale process here is still worth proposing.

When memory — not disk — is the binding constraint, the reclaimable RAM may be in-container process accumulation that is invisible to the Docker-object ladder above. docker system df and docker ps --size report layer/diff sizes on disk; they do not predict per-container resident-set memory.

Run docker stats only for memory-bound checks or explicit "system health" requests that ask for RAM/process reclamation:

docker stats --no-stream | tee "$SNAP/docker-stats.txt"

This emits one row per running container showing live MEM USAGE / LIMIT and %MEM. Inspect only the heaviest one or two containers, not every running container:

docker exec  ps -eo rss,args --sort=-rss | head -20 | tee "$SNAP/-top-rss.txt"

Typical findings: stale node / python dev servers from a previous session, a hung test runner, or an orphaned build worker that was never cleaned up. Killing the process (not the container) reclaims its RSS immediately and is safer than any tier-2+ ladder action.

Propose the kill to the user with the process name, RSS, and estimated RAM freed — do not auto-kill. This step is memory-only; skip it when disk is the sole binding constraint.

6. Run tier 1 where you can, then report the verdict

Under DOCKER_TRIAGE=available, and unless --dry-run, run the build-cache prune and show before/after with the smallest useful recheck:

df -h / | tail -1
docker builder prune -f
df -h / | tail -1
docker system df

Under host-only or unreachable, there is nothing to prune here. Say so once and move to the verdict.

If the request is a rerun/check-again in the same thread, make the report delta-oriented: call out what changed since the prior health check (disk %, available memory, CPU load, Docker restart/status changes) before repeating the verdict. Do not re-explain the full ladder or rerun verbose diagnostics unless the finding changed; keep it focused on current state plus material deltas.

Then emit a verdict table. Name the scope in the header, and withhold any row you cannot support:

Scope: container (METRICS_SCOPE=container) · Docker triage: host-only

| Resource | Status | Detail |
|----------|--------|--------|
| Disk     | 🟢/🟡/🔴 |  |
| Memory   | 🟢/🟡/🔴 |  |
| CPU      | 🟢/🟡/🔴 |  |
| Docker   | ⚪ N/A | host-only — see the host-side procedure |

Rating guide: 🔴 if starting the target would cross a hard limit (disk → ~90%+, OOM with no swap); 🟡 if it fits but with thin margin (no cushion, transient spikes possible); 🟢 if comfortable. State the binding constraint explicitly — it's usually one resource, not all three.

Withholding rule — the verdict must refuse what the scope cannot support. Under DOCKER_TRIAGE=host-only or unreachable with a build-shaped target (any row in step 3 marked No), the Disk row is not a RAG rating. Render it:

| Disk | ⚪ N/A | host-only — image and build-cache growth land on the Docker host, which is not visible from here |

and state in prose: no build-sizing verdict is possible from inside this container. A labelled green disk row beside a docker compose build target still reads as permission to start the build; a label explains a number, it does not withdraw a conclusion. For container-local targets, keep the real RAG rating — there the container view is the correct view, and refusing to answer would be its own failure.

7. Propose-then-confirm for tiers 2–4 — host-only

> Requires DOCKER_TRIAGE=available. Under host-only or unreachable there is nothing here for you to propose: the levers are in the host-side procedure, and the person running it owns the confirm gate.

If tier-1 reclaim already clears the target, say so and present the rest as optional headroom — don't push destructive removal that isn't needed. When you do propose it, quote the concrete artifact (container name, age, size) so the user can judge:

Optional further reclaim:
-  (exited d ago, ) — removing frees ~G → /.

Build this list from the baseline files and, when needed, one cached verbose snapshot. For volume candidates, read the Local Volumes section from $SNAP/docker-df-v.txt; never run docker system df -v once per volume.

Then ask (use AskUserQuestion). Run the removal only on explicit approval. With --dry-run, propose everything and write nothing — including skipping the tier-1 prune.

Host-side Docker triage

The relocated half of this skill. Docker inventory and every reclaim lever run where the daemon runs — they are not something a sandboxed agent can do, and #756

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mifunedev
• Source: mifunedev/agro
• License: Apache-2.0
• Homepage: https://agro.mifune.dev

Install and usage instructions live in the source repository linked above.