Azure Messaging

Azure Messaging

Choose the right Azure messaging service and delivery semantics for reliable, decoupled, event-driven systems. Handle retries, failures, and exactly-once processing.

Use This Skill When

• The user needs asynchronous processing design (commands, events, streams)
• The user asks about retries, dead-lettering, idempotency, or ordering
• The user needs to decouple systems via events or messaging
• The user needs high-throughput streaming or reactive patterns

Context: Messaging Maturity

Immature: Synchronous API calls, no retries, tightly coupled Developing: Simple queues created, basic retry logic Managed: Service Bus/Event Hub with DLQ, idempotency, monitoring → Target Optimized: Saga pattern for distributed transactions, stream processing, real-time analytics

Required Inputs

• Message pattern: Command (request-response), Event (pub-sub), Stream (high-volume telemetry)
• Throughput: 100 msg/sec? 1M msg/sec?
• Latency: 10,000 msg/sec → Event Hubs

└─ Needs low latency ( { try { var message = args.Message; var orderId = message.ApplicationProperties["OrderId"];

// Idempotency check if (await database.OrderProcessed(orderId)) { await args.CompleteMessageAsync(message); return; }

// Process order await ProcessOrder(orderId); await args.CompleteMessageAsync(message); } catch (Exception ex) { // Retry automatically (up to max-delivery-count) // After max retries, goes to DLQ throw; // Nack, retry } };

processor.ProcessErrorAsync += args => { Console.WriteLine($"Error: {args.Exception}"); return Task.CompletedTask; };

await processor.StartProcessingAsync(); ```

1. Monitor DLQ (failed messages):

``bash # Peek at DLQ az servicebus queue peek \ --namespace-name "sb-${app}-${env}" \ --resource-group $RG \ --name "order-processing/$DeadLetterQueue" ``

Phase 5: Ensure Exactly-Once Processing (Idempotency)

1. Deduplication ID (Service Bus):

`` Message A sent with MessageId="order-123" ├─ First delivery: Processed ✓ ├─ Retry (within 5-min window): Not reprocessed (dedup) └─ After 5 min: Can send again (different transaction) ``

1. Idempotency key in handler (Event Hub, Storage Queue):

```python # Python def processorder(event): orderid = event['OrderId']

# Check if already processed if db.query(f"SELECT * FROM processedorders WHERE orderid = {order_id}"): return # Already done, idempotent

# Process db.insert("orders", {"id": orderid, "status": "processing"}) # ... fulfill order ... db.insert("processedorders", {"orderid": orderid, "timestamp": now()}) ```

Phase 6: Scaling & Partitioning

1. Service Bus (sharding via topics/queues):

``` Instead of: Single queue "orders" (bottleneck at scale) Use: 10 queues: "orders-0", "orders-1", ..., "orders-9"

Producer hashes order_id % 10 → routes to specific queue Consumers: One per queue for parallelism ```

1. Event Hubs (partitions):

```bash # 4 partitions = 4 independent streams # Each consumer group member gets 1 partition # Max throughput: ~1M events/sec with 32 partitions

# Each partition maintains ordering FIFO per session_id ```

Phase 7: Monitoring & Alerting

1. Setup monitoring:

``bash az monitor metrics list \ --resource-group $RG \ --resource-type "Microsoft.ServiceBus/namespaces" \ --resource-id "sb-${app}-${env}" \ --metric "ActiveMessageCount,DeadletteredMessageCount" ``

1. Alerts:

• Active message count > 10,000 (backlog building)
• Dead-lettered message count > 0 (failures happening)
• Consumer lag > 1 hour (consumer slow/stopped)
• Message age > TTL (about to expire)

Output Contract

1. Messaging Architecture

• Service chosen (Queue, Topic, Event Hub)
• Number of queues/topics, partitions
• Consumer count and groups

1. Delivery & Ordering Design

• Delivery guarantee: At most once, at least once, exactly once
• Ordering requirements (FIFO per partition? Global?)
• Idempotency strategy (dedup ID, business key check)

1. Failure Handling

• Max retries before DLQ
• DLQ monitoring and remediation process
• Message TTL (time to live)

1. Configuration & Access

• Connection strings or managed identity scopes
• Queue/topic names and subscription details
• Retention policy (1-30 days)

1. Operations Checklist

• Alerts configured for backlog, lag, DLQ
• DLQ monitoring and replay procedure
• Capacity planning (throughput targets)

Guardrails

• Always define a DLQ: Dead-lettered messages need handling (manual replay or archive).
• Implement idempotency: Assume messages arrive >1 time; handle gracefully.
• Use TTL (time-to-live): Prevent stale messages; auto-expire to DLQ.
• Monitor backlog: If active message count grows, consumer is slow.
• Partition for scale: Don't use single queue for >2K msg/sec.
• Log message content (redacted): Include order ID, not full message body (size/privacy).
• Test failure scenarios: What if consumer crashes mid-processing? Does DLQ work?
• Avoid coupling: Producers and consumers should not know each other.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: manu14357
• Source: manu14357/zskills
• License: MIT
• Homepage: https://zskills.vercel.app

Install and usage instructions live in the source repository linked above.