Task Scheduling

Task Scheduling

Decide when work runs and which worker runs it: fire jobs on a schedule (cron/delayed/recurring), hand each job to exactly one worker via a lease, and make sure it completes once despite crashes and retries. This sits on top of messaging-streaming queues — the queue is the transport; this skill adds the scheduling, leasing, priorities, and task-level idempotency. Getting it wrong shows up as jobs that never run, run twice (double charge, double email), or pile up until a worker fleet falls permanently behind.

When to reach for this

Work must run later (send a reminder in 24h), on a schedule (nightly rollups, hourly cron), or repeatedly (poll every 5 min); a slow operation is already off the request path (→ messaging-streaming) and now needs reliable allocation to a pool of workers; jobs need priorities (paid before free) or fairness (no single tenant starves others); or a job must complete exactly once even though the worker holding it can crash mid-flight.

When NOT to

The caller needs the result inline — that's a synchronous call, not a scheduled job. A single fire-and-forget async step with no schedule, priority, or exactly-once need — a plain queue + idempotent consumer (messaging-streaming) is simpler; don't add a scheduler on top. One periodic job on one box — OS cron is fine until you have multiple schedulers or need history and retries. A long-running multi-step saga with rollback — reach for a durable workflow engine instead of hand-rolling state across jobs. Don't stand up Airflow/Celery "because we'll have batch jobs eventually" (YAGNI): it's a stateful control plane to operate and monitor.

Clarify first

• Trigger type — scheduled (cron/at a time), delayed (run after N seconds),

recurring (every N), or event-driven (a queue message arrives)? This decides whether a scheduler is even in scope.

• Exactly-once vs at-least-once — is a duplicate run harmful (money, email)

or harmless (idempotent recompute)? Drives the leasing + dedup design.

• Latency budget vs throughput — must a delayed job fire within seconds of

its time, or is "within a few minutes" fine? Tight timing is more expensive.

• Priority / fairness — do some jobs jump the line, and must one tenant or

job class be prevented from starving the rest? (→ back-of-the-envelope for arrival vs. service rate.)

• Job duration & variance — seconds or hours? Sets the visibility-timeout /

lease length and whether long jobs need heartbeats.

• Idempotency key — what identifies a task as the same task on retry?

(The key contract is owned by api-design.)

The options

Scheduling trigger

• OS cron / single scheduler — one process fires jobs on a crontab. *Use

when* one node, a handful of jobs, no HA requirement.

• Distributed scheduler (HA cron) — a leader-elected scheduler enqueues due

jobs into a queue; followers stand by. Use when the schedule must survive a node loss and must not double-fire.

• Delay queue / timer — jobs carry a "not before" time; the queue holds them

until due (delivery delay, sorted-set scoring, or a timer wheel). Use when per-job delays vary and you don't want a cron tick.

• Workflow/orchestration DAG — declared task dependencies with backfill and

history (Airflow-style). Use when batch pipelines have dependencies and you need a run history and reruns.

Worker allocation

• Pull (worker leasing) — workers poll the queue, lease a job for a

visibility timeout, and ack/delete on success. Use when you want back-pressure for free and elastic, self-balancing workers. The default.

• Push (dispatcher assigns) — a coordinator routes jobs to specific workers.

Use when affinity/locality matters (a job must run where its data is).

Priority & fairness

• Priority queues — separate high/low queues drained in order. Use when

some classes must run first.

• Weighted / fair scheduling — round-robin or weighted draw across per-tenant

queues. Use when one tenant's burst must not starve others.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | OS cron / single scheduler | Trivial; zero infra | SPOF — node dies, schedule stops; no retry/history | You need HA or missed-run recovery → distributed scheduler | | Distributed scheduler (HA cron) | Survives node loss; no double-fire (leader-elected) | Needs leader election (→ consistency-coordination); more moving parts | One box and one job is enough → OS cron | | Delay queue / timer | Per-job delays without a cron tick; precise-ish timing | Far-future jobs sit in the queue; timer accuracy bounded by poll interval | Delays are uniform/periodic → cron; dependencies exist → DAG | | Workflow DAG (Airflow-style) | Dependencies, backfill, run history, reruns | Heavy control plane; scheduler latency; overkill for single jobs | Jobs are independent one-shots → plain queue + scheduler | | Pull (worker leasing) | Self-balancing, elastic, natural back-pressure | At-least-once: lease expiry on a slow job re-runs it (need idempotency) | A job must run on a specific node (data locality) → push | | Push (dispatcher) | Affinity/locality; central control | Dispatcher is a bottleneck/SPOF; must track worker health | No locality need → pull is simpler | | Priority queues | Important work runs first | Low-priority starvation under sustained load | Fairness across tenants matters → weighted/fair | | Weighted / fair scheduling | No tenant starves another | More complex; per-tenant accounting | Only one workload class exists → single queue |

Behavior under stress

A scheduler can quietly fall behind, or it can amplify an outage by re-dispatching work a struggling fleet can't finish.

• Backlog growth: arrival rate exceeds worker throughput; due jobs queue up

and "scheduled for 09:00" runs at 09:40. End-to-end delay climbs while CPU looks fine. Mitigate: alarm on oldest-due-job age and queue depth, not just rate; scale workers; shed or defer low-priority jobs.

• Lease expiry / re-run storm (the classic): a job runs longer than its

visibility timeout, the lease expires, the queue redelivers it to a second worker, and now two workers run it — wasting capacity and, without idempotency, double-applying side effects. Mitigate: set the timeout above p99 job duration, heartbeat to extend the lease on long jobs, and make tasks idempotent/deduped.

• Poison task: a job that always fails is retried forever, burning workers

and re-loading a sick downstream. Mitigate: cap retries with backoff+jitter, then route to a dead-letter queue (retries/backoff/DLQ are owned by resilience-failure).

• Thundering herd at the tick: thousands of cron jobs all scheduled at the

top of the hour fire at once and stampede a downstream. Mitigate: jitter the schedule, spread triggers, or rate-limit dispatch.

• Scheduler split-brain: two schedulers both think they're leader and

double-enqueue every recurring job. Mitigate: a single leader via leader election + fencing (→ consistency-coordination); idempotent enqueue keyed by (job, scheduled_time).

• Starvation: a flood of high-priority or one noisy tenant's jobs starves

everyone else. Mitigate: fair/weighted scheduling and per-tenant concurrency caps.

Monitor: oldest-due-job age (the best lateness signal), queue depth per priority, lease-expiry / redelivery rate, retry and DLQ rate, worker utilization, and per-tenant share.

How to apply

1. Clarify the inputs. Settle trigger type, exactly-once vs at-least-once,

latency budget, priority/fairness, and job duration (see Clarify first). If the work is a single async step with no schedule or priority, stop — a plain queue + idempotent consumer (messaging-streaming) is enough.

1. Pick from the trade-off table. Choose a trigger (cron → distributed

scheduler → delay queue → DAG, cheapest that fits), an allocation model (pull leasing is the default; push only for locality), and a priority/fairness model only if more than one class exists.

1. Set the key knobs. Visibility timeout above p99 job duration, retry cap +

backoff before the DLQ, the dedup/idempotency key per task, the lease heartbeat interval for long jobs, and per-tenant concurrency limits.

1. Stress-test the choice. Walk backlog growth, lease-expiry re-runs, poison

tasks, tick stampede, scheduler split-brain, and starvation. Confirm a mitigation exists for each one the workload can trigger.

1. Size it with numbers. Workers needed = arrival rate × avg job seconds /

target concurrency (Little's law); confirm sustained throughput drains peak arrival, and that far-future delayed jobs fit storage (→ Numbers that matter).

1. Pick a provider. Default to the generic recipe; only open a provider file

if the user named a cloud (see Choosing a provider).

Dos and don'ts

Do

• Default to pull-based worker leasing with a visibility timeout; it self-balances.
• Set the visibility timeout above p99 job duration and heartbeat-extend long jobs.
• Make every task idempotent (dedup on a task key) so a re-run is harmless.
• Elect a single leader for the scheduler and key recurring enqueues by (job, time).
• Cap retries with backoff+jitter, then dead-letter; alarm on oldest-due-job age.

Don't

• Don't assume a leased job ran once — at-least-once means design for duplicates.
• Don't schedule every cron job at :00 — jitter so the tick doesn't stampede.
• Don't run two schedulers without leader election (split-brain double-fires).
• Don't retry a poison task forever; cap it and dead-letter it.
• Don't reach for Airflow/Celery before a schedule, priority, or HA need is real (YAGNI).

Numbers that matter

Size the worker pool with Little's law: in-flight jobs = arrival rate × average job duration, so workers ≈ peak arrival × avg seconds-per-job / per-worker concurrency. Sustained drain must exceed peak arrival or the backlog never clears. Set the visibility timeout above the p99 job duration (a too-short timeout is the #1 cause of duplicate runs); set far-future delay storage = delayed-job rate × max delay × job size. Don't restate the latency/QPS tables — pull the rates and durations from back-of-the-envelope.

Interface sketch

A scheduled task is a contract. Define it, not "a job":

• Task envelope: stable task_id / idempotency key (dedup on retry — key

contract owned by api-design), task_type/version, payload, priority, scheduled_for (run-not-before), attempt, and a trace_id.

• Schedule spec: for recurring jobs, the cron/interval expression plus a

dedup_key = (job_name, scheduled_time) so a double-enqueue is a no-op.

• Lease contract: on dequeue a worker gets the task invisible for

visibility_timeout; it must ack/delete on success or heartbeat to extend. Lease expiry → automatic redelivery. Failure after the retry cap → DLQ with attempt count and last error.

Choosing a provider

Default to the generic recipe above. If the user names a cloud, read references/providers/.md for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

Diagram

To visualize the scheduler → queue → leased workers path, or the lease-expiry redelivery loop, use the in-plugin architecture-diagram skill. Quick inline sketch: [scheduler] → [delay/priority queue] → workers (lease) ─expire→ requeue ─fail×N→ [DLQ]; main path solid, the expiry/DLQ branches dashed.

Related building blocks

• messaging-streaming — depends on it: queues are the transport this skill

schedules onto and leases from; it owns delivery guarantees, ordering, and DLQs and this skill does not reimplement them.

• resilience-failure — pairs with it for the retry policy: backoff, jitter,

and DLQ-as-containment are tuned there; this skill names them.

• api-design — depends on its idempotency-key contract, the mechanism that

makes a re-run after lease expiry safe.

• consistency-coordination — depends on it for leader election (and fencing)

so only one scheduler is active and recurring jobs don't double-fire.

• back-of-the-envelope — feeds into sizing: arrival rate and job duration set

the worker count and backlog drain.

• system-design — feeds into the orchestrator's reasoning loop; it routes here

when work must run later, on a schedule, or be reliably allocated to workers.

References

• references/deep-dive.md — leasing/visibility-timeout mechanics and

heartbeats, distributed-cron + leader election, delay-queue implementations (sorted set, timer wheel), priority/fairness algorithms, exactly-once-effect via dedup, and Celery/Sidekiq/Airflow internals. Read when designing the scheduler in detail.

• references/providers/{generic,aws,azure,gcp,temporal}.md — service

mappings, decision-changing limits, and pitfalls per environment.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: proyecto26
• Source: proyecto26/system-design-skills
• License: MIT
• Homepage: https://proyecto26.com/system-design-skills/

Install and usage instructions live in the source repository linked above.