Async Patterns

Async Patterns

Synchronous = caller waits for result. Asynchronous = caller sends work and moves on. Use async when: the work is slow, the caller doesn't need the result immediately, or you need to decouple systems.

When to Go Async

| Situation | Sync or Async? | |-----------|---------------| | User requests data, needs it now | Sync | | Send confirmation email after signup | Async | | Process payment (user waiting for result) | Sync (with timeout) | | Generate PDF report | Async (return job ID, poll for result) | | Notify 5 services about an event | Async (pub/sub) | | Resize uploaded image | Async | | Real-time API response | Sync | | Analytics event recording | Async (fire-and-forget) |

Core Patterns

1. Work Queue (Task Queue)

Producer → [Queue] → Consumer(s)

One message, one consumer. Load distributed across workers.

Use for: Background jobs, email sending, image processing, PDF generation.

Producer enqueues: { type: "send_email", to: "user@x.com", template: "welcome" }
Consumer picks up job, processes, acknowledges.
If consumer crashes → message redelivered to another consumer.

Key decisions:

• At-least-once delivery (default, safe) vs at-most-once (lossy, fast)
• Retry policy: how many times? Exponential backoff?
• Dead letter queue: where do permanently failed messages go?
• Concurrency: how many consumers? Rate limiting?

2. Pub/Sub (Fan-Out)

Publisher → [Topic] → Subscriber A
                    → Subscriber B
                    → Subscriber C

One message, multiple consumers. Each gets a copy.

Use for: Event notification, audit logging, cross-service sync.

Order service publishes: { event: "order.placed", order_id: "123", amount: 5000 }
  → Inventory service: reserves items
  → Email service: sends confirmation
  → Analytics service: records event

Key decisions:

• Subscribers are independent - one failing doesn't block others
• Message ordering: guaranteed per partition/key? Or unordered?
• Subscriber lag: what happens when a consumer falls behind?

3. Request-Reply (Async RPC)

Requester → [Request Queue] → Worker
         ← [Reply Queue]   ←

Use for: Long-running operations where the caller eventually needs the result.

1. Client POSTs /reports → returns { job_id: "abc", status: "pending" }
2. Worker processes the report
3. Client polls GET /reports/abc → { status: "completed", url: "..." }
   (or receives a webhook callback)

4. Event Sourcing

Commands → Event Store → [event1, event2, event3, ...] → Projections (read models)

Store events (facts that happened), not current state. Rebuild state by replaying events.

Use for: Audit-critical systems, financial transactions, collaborative editing.

Caution: High complexity. Only use when you genuinely need full event history and the ability to rebuild state at any point in time.

Message Design

Message Structure

{
  "id": "msg_abc123",
  "type": "order.placed",
  "version": 1,
  "timestamp": "2024-03-15T10:23:45Z",
  "source": "order-service",
  "correlation_id": "req_xyz789",
  "data": {
    "order_id": "ord_123",
    "user_id": "usr_456",
    "total_cents": 5000
  }
}

Rules:

• Every message has a unique id (for deduplication)
• Every message has a type and version (for evolution)
• Include correlation_id for tracing through async flows
• Include timestamp for ordering and debugging
• data contains the payload - keep it self-contained

Idempotency

Messages WILL be delivered more than once. Your consumer MUST handle duplicates.

Consumer receives message with id: "msg_abc123"
  → Check: have I already processed "msg_abc123"?
    YES → skip (already done)
    NO  → process, then record "msg_abc123" as processed

Store processed message IDs in a database with a TTL. Or design operations to be naturally idempotent (SET is idempotent, INCREMENT is not).

Failure Handling

Dead Letter Queue (DLQ)

Messages that fail after all retries go to a DLQ instead of being lost.

Queue → Consumer → fails → retry 1 → retry 2 → retry 3 → DLQ

DLQ rules:

• Monitor DLQ size (alert if > 0)
• Include original error in DLQ message metadata
• Build tooling to replay DLQ messages after fixing the bug
• Set retention (don't keep dead letters forever)

Poison Messages

A message that crashes the consumer every time it's processed.

Detection: same message redelivered N times
Action: move to DLQ, alert, continue processing other messages
Prevention: validate message schema before processing

Consumer Lag

When consumers can't keep up with producers.

Detection: queue depth growing over time
Actions:
  1. Scale consumers horizontally
  2. Batch process (consume N messages at once)
  3. If acceptable: shed load (drop low-priority messages)
  4. Investigate: is the consumer doing unnecessary work?

Eventual Consistency

In async systems, data across services is NOT immediately consistent. It's eventually consistent.

User places order (Order service: order = PLACED)
  → 50ms later → Inventory service: items reserved
  → 200ms later → Email service: confirmation sent
  → 2s later → Analytics: event recorded

Handling inconsistency:

• Show the user their local state immediately (optimistic UI)
• Design reads to tolerate stale data ("last updated 5s ago")
• Use compensating actions for failures (if payment fails after order created → cancel order)

> Reference: reference/queue-patterns.md - Queue comparison tables, delivery guarantees, backpressure strategies, idempotency patterns.

Technology Choices

| Tool | Type | Best For | |------|------|----------| | Redis (Streams/Lists) | Queue | Simple queues, low latency, volatile | | RabbitMQ | Queue + Pub/Sub | Flexible routing, reliable delivery | | Apache Kafka | Event log | High throughput, event sourcing, replay | | AWS SQS | Queue | Managed, simple, no ops | | AWS SNS + SQS | Pub/Sub | Managed fan-out | | NATS | Pub/Sub | Lightweight, fast, cloud-native | | BullMQ / Celery / Sidekiq | Job queue | Application-level background jobs | | Database (polling) | Queue | Already have a DB, low volume, don't want new infra |

Decision guide:

• < 1000 msgs/sec, simple jobs? → Database polling or Redis
• Need reliable delivery, routing? → RabbitMQ
• High throughput, event replay, multiple consumers? → Kafka
• Don't want to manage infrastructure? → SQS/SNS
• Already have Redis? → BullMQ/Redis Streams

Anti-Patterns

| Anti-Pattern | Problem | Fix | |-------------|---------|-----| | Async for everything | Unnecessary complexity, hard to debug | Start sync, go async only when you have a reason | | No idempotency | Duplicate processing, double charges | Store processed message IDs, design idempotent operations | | No DLQ | Failed messages disappear forever | Always configure a dead letter queue | | Synchronous mindset | Blocking on async result, negating the benefit | Return job ID, let client poll or receive callback | | Giant messages | Queue becomes a bottleneck, consumers slow | Put data in storage, put reference in message | | No monitoring | Queue backs up silently | Alert on queue depth, consumer lag, DLQ size | | Ordering assumptions | Messages arrive out of order, logic breaks | Design for out-of-order, or use partitioned ordering |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pvnarp
• Source: pvnarp/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.