# Task Scheduling > This skill should be used when the user designs a "task scheduler", "job scheduler", "job queue", "cron at scale", "distributed cron", "delayed / scheduled / recurring tasks", a "worker pool", reaches for "Celery / Sidekiq / Airflow", or wrestles with "task leasing", visibility timeouts, job priorities, fairness, or duplicate task execution. Use it whenever work must run later, on a schedule, or… - **Type:** Skill - **Install:** `agentstack add skill-proyecto26-system-design-skills-task-scheduling` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [proyecto26](https://agentstack.voostack.com/s/proyecto26) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [proyecto26](https://github.com/proyecto26) - **Source:** https://github.com/proyecto26/system-design-skills/tree/main/skills/task-scheduling - **Website:** https://proyecto26.com/system-design-skills/ ## Install ```sh agentstack add skill-proyecto26-system-design-skills-task-scheduling ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Task Scheduling Decide *when* work runs and *which worker* runs it: fire jobs on a schedule (cron/delayed/recurring), hand each job to exactly one worker via a lease, and make sure it completes once despite crashes and retries. This sits *on top of* `messaging-streaming` queues — the queue is the transport; this skill adds the scheduling, leasing, priorities, and task-level idempotency. Getting it wrong shows up as jobs that never run, run twice (double charge, double email), or pile up until a worker fleet falls permanently behind. ## When to reach for this Work must run **later** (send a reminder in 24h), **on a schedule** (nightly rollups, hourly cron), or **repeatedly** (poll every 5 min); a slow operation is already off the request path (→ `messaging-streaming`) and now needs reliable allocation to a pool of workers; jobs need **priorities** (paid before free) or **fairness** (no single tenant starves others); or a job must complete **exactly once** even though the worker holding it can crash mid-flight. ## When NOT to The caller needs the result inline — that's a synchronous call, not a scheduled job. A single fire-and-forget async step with no schedule, priority, or exactly-once need — a plain queue + idempotent consumer (`messaging-streaming`) is simpler; don't add a scheduler on top. One periodic job on one box — OS `cron` is fine until you have multiple schedulers or need history and retries. A long-running multi-step saga with rollback — reach for a durable workflow engine instead of hand-rolling state across jobs. Don't stand up Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a stateful control plane to operate and monitor. ## Clarify first - **Trigger type** — scheduled (cron/at a time), delayed (run after N seconds), recurring (every N), or event-driven (a queue message arrives)? This decides whether a scheduler is even in scope. - **Exactly-once vs at-least-once** — is a duplicate run harmful (money, email) or harmless (idempotent recompute)? Drives the leasing + dedup design. - **Latency budget vs throughput** — must a delayed job fire within seconds of its time, or is "within a few minutes" fine? Tight timing is more expensive. - **Priority / fairness** — do some jobs jump the line, and must one tenant or job class be prevented from starving the rest? (→ `back-of-the-envelope` for arrival vs. service rate.) - **Job duration & variance** — seconds or hours? Sets the visibility-timeout / lease length and whether long jobs need heartbeats. - **Idempotency key** — what identifies a task as the same task on retry? (The key contract is owned by `api-design`.) ## The options **Scheduling trigger** - **OS cron / single scheduler** — one process fires jobs on a crontab. *Use when* one node, a handful of jobs, no HA requirement. - **Distributed scheduler (HA cron)** — a leader-elected scheduler enqueues due jobs into a queue; followers stand by. *Use when* the schedule must survive a node loss and must not double-fire. - **Delay queue / timer** — jobs carry a "not before" time; the queue holds them until due (delivery delay, sorted-set scoring, or a timer wheel). *Use when* per-job delays vary and you don't want a cron tick. - **Workflow/orchestration DAG** — declared task dependencies with backfill and history (Airflow-style). *Use when* batch pipelines have dependencies and you need a run history and reruns. **Worker allocation** - **Pull (worker leasing)** — workers poll the queue, lease a job for a **visibility timeout**, and ack/delete on success. *Use when* you want back-pressure for free and elastic, self-balancing workers. The default. - **Push (dispatcher assigns)** — a coordinator routes jobs to specific workers. *Use when* affinity/locality matters (a job must run where its data is). **Priority & fairness** - **Priority queues** — separate high/low queues drained in order. *Use when* some classes must run first. - **Weighted / fair scheduling** — round-robin or weighted draw across per-tenant queues. *Use when* one tenant's burst must not starve others. ## Trade-offs | Option | What it solves | What it worsens | Change it when | |---|---|---|---| | OS cron / single scheduler | Trivial; zero infra | SPOF — node dies, schedule stops; no retry/history | You need HA or missed-run recovery → distributed scheduler | | Distributed scheduler (HA cron) | Survives node loss; no double-fire (leader-elected) | Needs leader election (→ `consistency-coordination`); more moving parts | One box and one job is enough → OS cron | | Delay queue / timer | Per-job delays without a cron tick; precise-ish timing | Far-future jobs sit in the queue; timer accuracy bounded by poll interval | Delays are uniform/periodic → cron; dependencies exist → DAG | | Workflow DAG (Airflow-style) | Dependencies, backfill, run history, reruns | Heavy control plane; scheduler latency; overkill for single jobs | Jobs are independent one-shots → plain queue + scheduler | | Pull (worker leasing) | Self-balancing, elastic, natural back-pressure | At-least-once: lease expiry on a slow job re-runs it (need idempotency) | A job must run on a specific node (data locality) → push | | Push (dispatcher) | Affinity/locality; central control | Dispatcher is a bottleneck/SPOF; must track worker health | No locality need → pull is simpler | | Priority queues | Important work runs first | Low-priority starvation under sustained load | Fairness across tenants matters → weighted/fair | | Weighted / fair scheduling | No tenant starves another | More complex; per-tenant accounting | Only one workload class exists → single queue | ## Behavior under stress A scheduler can quietly fall behind, or it can *amplify* an outage by re-dispatching work a struggling fleet can't finish. - **Backlog growth:** arrival rate exceeds worker throughput; due jobs queue up and "scheduled for 09:00" runs at 09:40. End-to-end delay climbs while CPU looks fine. *Mitigate:* alarm on **oldest-due-job age** and queue depth, not just rate; scale workers; shed or defer low-priority jobs. - **Lease expiry / re-run storm (the classic):** a job runs longer than its visibility timeout, the lease expires, the queue redelivers it to a second worker, and now two workers run it — wasting capacity and, without idempotency, double-applying side effects. *Mitigate:* set the timeout above p99 job duration, **heartbeat to extend** the lease on long jobs, and make tasks idempotent/deduped. - **Poison task:** a job that always fails is retried forever, burning workers and re-loading a sick downstream. *Mitigate:* cap retries with backoff+jitter, then route to a dead-letter queue (retries/backoff/DLQ are owned by `resilience-failure`). - **Thundering herd at the tick:** thousands of cron jobs all scheduled at the top of the hour fire at once and stampede a downstream. *Mitigate:* jitter the schedule, spread triggers, or rate-limit dispatch. - **Scheduler split-brain:** two schedulers both think they're leader and double-enqueue every recurring job. *Mitigate:* a single leader via leader election + fencing (→ `consistency-coordination`); idempotent enqueue keyed by (job, scheduled_time). - **Starvation:** a flood of high-priority or one noisy tenant's jobs starves everyone else. *Mitigate:* fair/weighted scheduling and per-tenant concurrency caps. **Monitor:** oldest-due-job age (the best lateness signal), queue depth per priority, lease-expiry / redelivery rate, retry and DLQ rate, worker utilization, and per-tenant share. ## How to apply 1. **Clarify the inputs.** Settle trigger type, exactly-once vs at-least-once, latency budget, priority/fairness, and job duration (see *Clarify first*). If the work is a single async step with no schedule or priority, stop — a plain queue + idempotent consumer (`messaging-streaming`) is enough. 2. **Pick from the trade-off table.** Choose a trigger (cron → distributed scheduler → delay queue → DAG, cheapest that fits), an allocation model (pull leasing is the default; push only for locality), and a priority/fairness model only if more than one class exists. 3. **Set the key knobs.** Visibility timeout above p99 job duration, retry cap + backoff before the DLQ, the dedup/idempotency key per task, the lease heartbeat interval for long jobs, and per-tenant concurrency limits. 4. **Stress-test the choice.** Walk backlog growth, lease-expiry re-runs, poison tasks, tick stampede, scheduler split-brain, and starvation. Confirm a mitigation exists for each one the workload can trigger. 5. **Size it with numbers.** Workers needed = arrival rate × avg job seconds / target concurrency (Little's law); confirm sustained throughput drains peak arrival, and that far-future delayed jobs fit storage (→ *Numbers that matter*). 6. **Pick a provider.** Default to the generic recipe; only open a provider file if the user named a cloud (see *Choosing a provider*). ## Dos and don'ts **Do** - Default to pull-based worker leasing with a visibility timeout; it self-balances. - Set the visibility timeout above p99 job duration and heartbeat-extend long jobs. - Make every task idempotent (dedup on a task key) so a re-run is harmless. - Elect a single leader for the scheduler and key recurring enqueues by (job, time). - Cap retries with backoff+jitter, then dead-letter; alarm on oldest-due-job age. **Don't** - Don't assume a leased job ran once — at-least-once means design for duplicates. - Don't schedule every cron job at :00 — jitter so the tick doesn't stampede. - Don't run two schedulers without leader election (split-brain double-fires). - Don't retry a poison task forever; cap it and dead-letter it. - Don't reach for Airflow/Celery before a schedule, priority, or HA need is real (YAGNI). ## Numbers that matter Size the worker pool with Little's law: in-flight jobs = arrival rate × average job duration, so workers ≈ peak arrival × avg seconds-per-job / per-worker concurrency. Sustained drain must exceed peak arrival or the backlog never clears. Set the visibility timeout above the p99 job duration (a too-short timeout is the #1 cause of duplicate runs); set far-future delay storage = delayed-job rate × max delay × job size. Don't restate the latency/QPS tables — pull the rates and durations from `back-of-the-envelope`. ## Interface sketch A scheduled task is a contract. Define it, not "a job": - **Task envelope:** stable `task_id` / idempotency key (dedup on retry — key contract owned by `api-design`), `task_type`/version, `payload`, `priority`, `scheduled_for` (run-not-before), `attempt`, and a `trace_id`. - **Schedule spec:** for recurring jobs, the cron/interval expression *plus* a `dedup_key = (job_name, scheduled_time)` so a double-enqueue is a no-op. - **Lease contract:** on dequeue a worker gets the task invisible for `visibility_timeout`; it must ack/delete on success or heartbeat to extend. Lease expiry → automatic redelivery. Failure after the retry cap → DLQ with attempt count and last error. ## Choosing a provider Default to the generic recipe above. If the user names a cloud, read `references/providers/.md` for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer. ## Diagram To visualize the scheduler → queue → leased workers path, or the lease-expiry redelivery loop, use the in-plugin `architecture-diagram` skill. Quick inline sketch: `[scheduler] → [delay/priority queue] → workers (lease) ─expire→ requeue ─fail×N→ [DLQ]`; main path solid, the expiry/DLQ branches dashed. ## Related building blocks - `messaging-streaming` — *depends on* it: queues are the transport this skill schedules onto and leases from; it owns delivery guarantees, ordering, and DLQs and this skill does not reimplement them. - `resilience-failure` — *pairs with* it for the retry policy: backoff, jitter, and DLQ-as-containment are tuned there; this skill names them. - `api-design` — *depends on* its idempotency-key contract, the mechanism that makes a re-run after lease expiry safe. - `consistency-coordination` — *depends on* it for leader election (and fencing) so only one scheduler is active and recurring jobs don't double-fire. - `back-of-the-envelope` — *feeds into* sizing: arrival rate and job duration set the worker count and backlog drain. - `system-design` — *feeds into* the orchestrator's reasoning loop; it routes here when work must run later, on a schedule, or be reliably allocated to workers. ## References - **`references/deep-dive.md`** — leasing/visibility-timeout mechanics and heartbeats, distributed-cron + leader election, delay-queue implementations (sorted set, timer wheel), priority/fairness algorithms, exactly-once-effect via dedup, and Celery/Sidekiq/Airflow internals. Read when designing the scheduler in detail. - **`references/providers/{generic,aws,azure,gcp,temporal}.md`** — service mappings, decision-changing limits, and pitfalls per environment. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [proyecto26](https://github.com/proyecto26) - **Source:** [proyecto26/system-design-skills](https://github.com/proyecto26/system-design-skills) - **License:** MIT - **Homepage:** https://proyecto26.com/system-design-skills/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-proyecto26-system-design-skills-task-scheduling - Seller: https://agentstack.voostack.com/s/proyecto26 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.