# Messaging Streaming > This skill should be used when the user designs a "message queue", reaches for "Kafka", "RabbitMQ", "SQS", "Kinesis", "pub/sub", or "event-driven" architecture, asks about "async processing", "background jobs", "stream processing", or wrestles with "exactly-once vs at-least-once", "delivery guarantees", "message ordering", "duplicate handling / dedup", "dead letter queue", "backpressure", "saga o… - **Type:** Skill - **Install:** `agentstack add skill-proyecto26-system-design-skills-messaging-streaming` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [proyecto26](https://agentstack.voostack.com/s/proyecto26) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [proyecto26](https://github.com/proyecto26) - **Source:** https://github.com/proyecto26/system-design-skills/tree/main/skills/messaging-streaming - **Website:** https://proyecto26.com/system-design-skills/ ## Install ```sh agentstack add skill-proyecto26-system-design-skills-messaging-streaming ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Messaging & Streaming Move work off the synchronous request path and decouple producers from consumers, so a slow, spiky, or failure-prone operation doesn't block the caller. Getting this wrong is subtle: a queue silently changes the delivery and ordering guarantees, and under load it can absorb a spike gracefully *or* become the thing that hides a meltdown until the backlog is unrecoverable. ## When to reach for this A step is too slow to do inline (image transcode, fan-out, third-party call); the write path is spiky and needs a buffer to smooth bursts (→ `back-of-the-envelope` for the spike factor); two services must be decoupled so one can fail or deploy independently; or many consumers need the same event stream. The async hand-off buys responsiveness, isolation, and elasticity (scale producers and consumers separately). ## When NOT to The caller needs the result *now* to proceed (a synchronous read, a balance check before confirming) — a queue only adds latency and a place for work to get lost. Strong read-after-write within one request. Trivial in-process work that a function call handles. Don't add a broker before a number or a coupling problem justifies it (YAGNI): it's a new stateful system to operate, monitor, and reason about under failure. "We'll need Kafka eventually" is name-dropping, not a requirement. ## Clarify first - **Sync or async?** Does the caller need the result inline, or is fire-and-react acceptable? This decides whether a queue belongs here at all. - **Delivery guarantee needed** — is a dropped message acceptable (at-most-once), or must every message be processed (at-least-once + idempotent consumers)? - **Ordering** — must messages be processed in order, globally or per-key (per user, per account)? Global ordering is expensive; per-key usually suffices. - **Throughput and retention** — messages/sec at peak, and how long must they be replayable? (→ `back-of-the-envelope`.) One-shot work vs. a replayable log. - **Consumer count and pattern** — one worker pool draining a job, or many independent subscribers each reading every event? - **Failure handling** — what happens to a message that keeps failing? Where does it go, and who looks at it? ## The options **Sync vs. async — settle this before picking a tool.** Stay synchronous when the caller needs the result to continue and the call is fast and reliable; a direct request is simpler to build, trace, and reason about. Go async when the work is slow, spiky, fan-out-heavy, or the caller can react to the result later — this trades immediate consistency and an easy stack trace for responsiveness and isolation. Only after choosing async do the options below apply. Building request/reply *over* a queue to fake a synchronous answer is a smell — a direct call is the better design. **Queue (work/task queue)** — one logical consumer group competes to drain messages; a message is delivered to one worker and removed when acked. *Use when* there are background jobs or commands to process exactly once-ish, and workers should scale to drain a backlog. **Pub/sub (fan-out)** — each subscriber gets its own copy of every message; producers don't know subscribers. *Use when* multiple independent consumers react to the same event (notify, index, audit) and loose coupling matters. **Stream (durable, replayable log)** — an append-only, partitioned, retained log; consumers track their own offset and can replay history. *Use when* the design needs ordering per partition, multiple consumers at different positions, event sourcing, or reprocessing (→ `data-storage` for event sourcing/outbox). **Durable workflow (orchestration engine)** — code that survives process crashes; the engine persists each step and resumes where it left off, with built-in retries, timers, and compensation. *Use when* a multi-step process with retries, human delays, and rollback (a saga) would otherwise become a fragile hand-rolled mesh of queues, state flags, and cron jobs. Delivery semantics cut across all of these: **at-most-once** (fire and forget, may drop), **at-least-once** (retries until acked, may duplicate — the practical default), **exactly-once** (no loss, no dup). True end-to-end exactly-once is impractical: a broker's "EOS" (e.g. Kafka) is **intra-cluster only**, so across systems you always implement it as **at-least-once + idempotent/deduped consumers** (→ `api-design` idempotency keys). See `references/deep-dive.md` for the mechanics. ## Trade-offs | Option | What it solves | What it worsens | Change it when | |---|---|---|---| | Queue (work queue) | Decouples + buffers; scale workers to drain backlog | At-least-once means duplicates; ordering not guaranteed across workers | Replay or many independent consumers are needed → stream/pub-sub | | Pub/sub (fan-out) | One event, N decoupled reactions; add consumers freely | No replay (transient); slow subscriber can lag or drop; fan-out amplifies load | History/replay or per-key ordering is needed → stream | | Stream (log) | Ordering per partition, replay, multi-consumer, event sourcing | Operationally heavier; partition key is a hot-shard risk; consumers must manage offsets | Simple one-shot jobs don't need a log → queue | | Durable workflow | Crash-safe long-running sagas; retries/compensation built in | New runtime + programming model; latency overhead; lock-in to engine semantics | A single async step with no orchestration → plain queue | | At-least-once delivery | No message loss under retry/crash | Duplicates — consumers must be idempotent (→ `api-design` idempotency keys) | Loss is acceptable and dedup cost isn't worth it → at-most-once | | Exactly-once (effective) | No loss and no duplicate side effects | Cost/complexity; often narrow (within one broker, not across systems) | Idempotent at-least-once is good enough (it usually is) | ## Behavior under stress A broker's whole job is to absorb a spike — but it can also *hide* a meltdown. - **Backlog growth / unbounded queues:** producers outpace consumers; the queue grows past memory, spilling to disk and slowing further. End-to-end latency climbs invisibly while throughput looks fine. *Mitigate:* **backpressure** — bound the queue and shed or 503 producers (with backoff) once full, rather than buffering forever. Alarm on queue depth and message age, not just rate. - **Retry storms / poison messages:** a message that always fails is redelivered forever (at-least-once), burning consumer capacity and re-amplifying load on a downstream that's already struggling. *Mitigate:* capped retries with backoff+jitter, then route to a **dead-letter queue (DLQ)** so the poison message stops blocking the line and a human can inspect it. (Retries, backoff, DLQ-as-containment, and backpressure are owned by `resilience-failure`.) - **Duplicate amplification:** under retry, the same side effect (charge, email) fires twice unless consumers dedup. *Mitigate:* idempotency keys / dedup table. - **Hot partition:** in a stream, a skewed partition key (one celebrity, one tenant) overloads a single partition while others idle — the same hot-key shape as `data-storage` sharding. *Mitigate:* better key, sub-partitioning, or batching. - **Slow consumer in fan-out:** one lagging subscriber backs up or drops; isolate consumers so one can't stall the others. **Monitor:** queue depth, **oldest-message age / consumer lag** (the single best signal), redelivery/DLQ rate, consumer throughput vs. producer rate, and end-to-end latency. ## How to apply 1. **Clarify the inputs.** Settle sync vs. async first; if the caller needs the result inline, stop — no broker. Then answer delivery guarantee, ordering scope, throughput/retention, consumer pattern, and failure handling (see *Clarify first*). 2. **Pick the shape from the trade-off table.** One worker pool draining jobs → **queue**; many independent reactions to one event → **pub/sub**; ordering, replay, or multiple offsets → **stream**; multi-step retry/compensation saga → **durable workflow**. Pick the cheapest shape that meets the constraint. 3. **Set the key knobs.** Choose the delivery semantic (at-least-once + idempotent consumers is the default), the ack point (after-process vs. before), the partition/ordering key, retention window, and retry cap before the DLQ. 4. **Stress-test the choice.** Walk backlog growth, retry storms / poison messages, duplicate amplification, hot partition, and slow fan-out consumer. Add backpressure, a DLQ, and dedup where each applies. 5. **Size it with numbers.** Compute peak produce vs. sustained consume rate, `consume_rate − produce_rate` drain, partition count vs. consumer parallelism, and storage = rate × message size × retention (→ `back-of-the-envelope`). 6. **Pick a provider.** Default to the generic recipe; only read a provider file if the user names a cloud and a managed limit changes the choice. ## Dos and don'ts **Do** - Settle sync vs. async before naming any broker; keep synchronous work synchronous. - Default to at-least-once and make consumers idempotent (→ `api-design` keys). - Bound queues and apply backpressure; alarm on oldest-message age / consumer lag. - Cap retries with backoff+jitter, then route poison messages to a DLQ. - Define the message envelope (`message_id`, schema version, key, `trace_id`) up front. **Don't** - Don't add a broker "for scale later" before a number or coupling problem justifies it. - Don't build request/reply over a queue to fake a synchronous answer. - Don't reach for exactly-once when idempotent at-least-once already suffices. - Don't buffer an unbounded backlog — it hides a meltdown until it's unrecoverable. - Don't pick a global ordering guarantee when per-key ordering is enough. ## Numbers that matter Quantify before choosing: peak produce rate vs. sustained consume rate (if producers can outrun consumers for long, backpressure and a depth alarm are required), retention window (drives storage = rate × message size × retention), and partition count (caps consumer parallelism — one consumer per partition per group). A backlog drains at `(consume_rate − produce_rate)`; if that's negative, it never drains. Use `back-of-the-envelope` for the spike factor, message sizes, and storage; don't restate its tables here. ## Interface sketch A message is a contract. Define it explicitly, not as "an event": - **Envelope:** stable `message_id` (for dedup), `type`/`schema_version`, a `partition_key`/ordering key, `timestamp`, and a `trace_id` for correlation. - **Payload:** a versioned schema. Prefer event facts (`OrderPlaced{order_id, total}`) over commands when fanning out; keep it small and forward-compatible. - **Ack contract:** when does the consumer ack — before or after the side effect? Ack-after-process gives at-least-once; ack-before gives at-most-once. - **DLQ shape:** failed messages keep the original envelope plus failure reason and attempt count, so they can be inspected and replayed. ## Choosing a provider Default to the generic recipe above. If the user names a cloud, read `references/providers/.md` for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer. ## Diagram To visualize the producer → broker → consumer path, fan-out to multiple subscribers, or the retry → DLQ flow, use the in-plugin `architecture-diagram` skill. Quick inline sketch: `producer → [queue] → workers ─fail→ [DLQ]`; the main path solid, the DLQ branch dashed. ## Related building blocks - `resilience-failure` — *owned-concept lives in*: retries/backoff/jitter, dead-letter queues, and backpressure as outage containment. This skill names them; that one tunes them. - `api-design` — *depends on* its idempotency keys, the mechanism that makes at-least-once delivery safe. - `data-storage` — *pairs with* this for event sourcing and the transactional outbox; *owned-concept lives in*: the hot-shard/partition-key problem streams inherit. - `consistency-coordination` — *owned-concept lives in*: ordering guarantees, exactly-once vs. idempotency, and saga/distributed-transaction theory behind durable workflows. - `back-of-the-envelope` — *feeds into* sizing: the produce/consume rates, retention, and storage that size the broker. - `system-design` — *the orchestrator* that routes here when work goes async. ## References - **`references/deep-dive.md`** — delivery-guarantee mechanics (acks, visibility timeouts, offsets, idempotent/dedup consumers, transactional outbox), ordering internals, partitioning, DLQ design, and when durable-workflow engines beat hand-rolled queue+retry+saga. Read when designing the messaging layer in detail. - **`references/providers/{generic,aws,azure,gcp,temporal}.md`** — service mappings, decision-changing limits, and pitfalls per environment. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [proyecto26](https://github.com/proyecto26) - **Source:** [proyecto26/system-design-skills](https://github.com/proyecto26/system-design-skills) - **License:** MIT - **Homepage:** https://proyecto26.com/system-design-skills/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-proyecto26-system-design-skills-messaging-streaming - Seller: https://agentstack.voostack.com/s/proyecto26 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.