Kafka Resilience And Schema Evolution

Kafka Resilience And Schema Evolution

Overview

Generic streaming guidance is not enough for production Kafka. Agents routinely introduce breaking schema changes, under-provisioned durability settings, and missing poison-message isolation. This skill mandates enforceable broker, producer, consumer, and registry guardrails before any production change ships.

When to Use

• creating or modifying Kafka topics, producers, or consumers
• setting or changing schema registry compatibility policies
• designing dead-letter queue (DLQ) routing for poison pill messages
• hardening producer durability (acks, retries, idempotence)
• reviewing consumer lag, replay, or failover behavior on Kafka-backed pipelines

Pair with streaming-and-messaging-systems for broader event design. Pair with avro-protobuf-json-schema-registry when registry subjects and compatibility CI are in scope.

Workflow

1. Define the production contract before broker changes.

Document:

• topic key strategy and partition count rationale
• retention, compaction, and replay policy
• schema format and registry subject naming
• consumer groups and downstream sinks
• delivery semantics target (at-least-once with idempotent sinks, or stricter)

1. Enforce producer durability defaults.

Require unless explicitly waived with owner approval:

• acks=all (or acks=-1)
• enable.idempotence=true when ordering and deduplication matter
• bounded retries with delivery.timeout.ms aligned to SLA
• max.in.flight.requests.per.connection=1 when strict ordering is required
• TLS/SASL configuration documented for non-development clusters

1. Block breaking schema evolution.

Before any schema change:

• set compatibility policy per subject (BACKWARD, FORWARD, or FULL — not NONE in production)
• run compatibility checks in CI against registered schemas
• document producer-then-consumer or consumer-then-producer rollout order
• reject field removals, renames, or type changes without migration plan
• load references/kafka-production-guardrails.md for DLQ and evolution patterns

1. Mandate dead-letter and poison pill isolation.

Every production consumer that parses external payloads must define:

• DLQ topic or sink with retention and access controls
• classification rules (deserialization failure, schema mismatch, business rule violation)
• alert routing when DLQ rate exceeds threshold
• replay procedure with deduplication keys
• no silent drop of unparseable records

1. Make lag and recovery observable.

Plan for:

• consumer group lag alerts with owner routing
• offset reset policy documented and restricted
• replay runbook that does not bypass DLQ classification
• broker disk and retention monitoring for high-throughput topics

1. Load MCP observability when diagnosing live lag.

Use mcp-data-observability-integration with mcp/kafka.mcp.json to inspect consumer group lag and topic metadata before changing consumer code or partition counts.

Common Rationalizations

| Rationalization | Reality | | --- | --- | | "acks=1 is fine because Kafka is durable." | Leader acknowledgment without full ISR acknowledgment loses events under failure scenarios. | | "We can fix schema breaks by redeploying consumers quickly." | Breaking changes propagate to many consumers and batch sinks before redeploy completes. | | "DLQs add too much operational overhead." | Poison pills without DLQs stall partitions, inflate lag, and hide data loss as consumer retries. | | "Schema compatibility NONE is okay for internal topics." | Internal topics still feed warehouses, stream processors, and audit systems. |

Red Flags

• production subjects use NONE compatibility
• producers use acks=0 or acks=1 without documented waiver
• consumers have no DLQ path for deserialization failures
• schema changes ship without CI compatibility validation
• consumer group lag has no alert owner
• replay procedures reset offsets without reconciliation or publish pause

Verification

• [ ] Producer durability settings meet acks=all and idempotence requirements
• [ ] Schema compatibility policy is set and CI-validated for production subjects
• [ ] DLQ routing, alerts, and replay procedure are documented
• [ ] Consumer lag and retention monitoring exist with named owners
• [ ] Rollout order for schema changes is explicit and tested in non-production

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: vaquarkhan
• Source: vaquarkhan/data-engineering-agent-skills
• License: MIT
• Homepage: https://vaquarkhan.github.io/data-engineering-agent-skills/

Install and usage instructions live in the source repository linked above.