# Kafka Resilience And Schema Evolution

> Enforces production Kafka guardrails including non-breaking schema evolution, dead-letter queues for poison messages, and acks=all producer durability. Use when designing or changing Kafka topics, producers, consumers, schema registry policies, or streaming recovery paths.

- **Type:** Skill
- **Install:** `agentstack add skill-vaquarkhan-data-engineering-agent-skills-kafka-resilience-and-schema-evolution`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vaquarkhan](https://agentstack.voostack.com/s/vaquarkhan)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [vaquarkhan](https://github.com/vaquarkhan)
- **Source:** https://github.com/vaquarkhan/data-engineering-agent-skills/tree/main/skills/kafka-resilience-and-schema-evolution
- **Website:** https://vaquarkhan.github.io/data-engineering-agent-skills/

## Install

```sh
agentstack add skill-vaquarkhan-data-engineering-agent-skills-kafka-resilience-and-schema-evolution
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Kafka Resilience And Schema Evolution

## Overview

Generic streaming guidance is not enough for production Kafka. Agents routinely introduce breaking schema changes, under-provisioned durability settings, and missing poison-message isolation. This skill mandates enforceable broker, producer, consumer, and registry guardrails before any production change ships.

## When to Use

- creating or modifying `Kafka` topics, producers, or consumers
- setting or changing schema registry compatibility policies
- designing dead-letter queue (DLQ) routing for poison pill messages
- hardening producer durability (`acks`, retries, idempotence)
- reviewing consumer lag, replay, or failover behavior on Kafka-backed pipelines

Pair with `streaming-and-messaging-systems` for broader event design. Pair with `avro-protobuf-json-schema-registry` when registry subjects and compatibility CI are in scope.

## Workflow

1. Define the production contract before broker changes.
   Document:
   - topic key strategy and partition count rationale
   - retention, compaction, and replay policy
   - schema format and registry subject naming
   - consumer groups and downstream sinks
   - delivery semantics target (at-least-once with idempotent sinks, or stricter)

2. Enforce producer durability defaults.
   Require unless explicitly waived with owner approval:
   - `acks=all` (or `acks=-1`)
   - `enable.idempotence=true` when ordering and deduplication matter
   - bounded `retries` with `delivery.timeout.ms` aligned to SLA
   - `max.in.flight.requests.per.connection=1` when strict ordering is required
   - TLS/SASL configuration documented for non-development clusters

3. Block breaking schema evolution.
   Before any schema change:
   - set compatibility policy per subject (`BACKWARD`, `FORWARD`, or `FULL` — not `NONE` in production)
   - run compatibility checks in CI against registered schemas
   - document producer-then-consumer or consumer-then-producer rollout order
   - reject field removals, renames, or type changes without migration plan
   - load `references/kafka-production-guardrails.md` for DLQ and evolution patterns

4. Mandate dead-letter and poison pill isolation.
   Every production consumer that parses external payloads must define:
   - DLQ topic or sink with retention and access controls
   - classification rules (deserialization failure, schema mismatch, business rule violation)
   - alert routing when DLQ rate exceeds threshold
   - replay procedure with deduplication keys
   - no silent drop of unparseable records

5. Make lag and recovery observable.
   Plan for:
   - consumer group lag alerts with owner routing
   - offset reset policy documented and restricted
   - replay runbook that does not bypass DLQ classification
   - broker disk and retention monitoring for high-throughput topics

6. Load MCP observability when diagnosing live lag.
   Use `mcp-data-observability-integration` with `mcp/kafka.mcp.json` to inspect consumer group lag and topic metadata before changing consumer code or partition counts.

## Common Rationalizations

| Rationalization | Reality |
| --- | --- |
| "acks=1 is fine because Kafka is durable." | Leader acknowledgment without full ISR acknowledgment loses events under failure scenarios. |
| "We can fix schema breaks by redeploying consumers quickly." | Breaking changes propagate to many consumers and batch sinks before redeploy completes. |
| "DLQs add too much operational overhead." | Poison pills without DLQs stall partitions, inflate lag, and hide data loss as consumer retries. |
| "Schema compatibility NONE is okay for internal topics." | Internal topics still feed warehouses, stream processors, and audit systems. |

## Red Flags

- production subjects use `NONE` compatibility
- producers use `acks=0` or `acks=1` without documented waiver
- consumers have no DLQ path for deserialization failures
- schema changes ship without CI compatibility validation
- consumer group lag has no alert owner
- replay procedures reset offsets without reconciliation or publish pause

## Verification

- [ ] Producer durability settings meet `acks=all` and idempotence requirements
- [ ] Schema compatibility policy is set and CI-validated for production subjects
- [ ] DLQ routing, alerts, and replay procedure are documented
- [ ] Consumer lag and retention monitoring exist with named owners
- [ ] Rollout order for schema changes is explicit and tested in non-production

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vaquarkhan](https://github.com/vaquarkhan)
- **Source:** [vaquarkhan/data-engineering-agent-skills](https://github.com/vaquarkhan/data-engineering-agent-skills)
- **License:** MIT
- **Homepage:** https://vaquarkhan.github.io/data-engineering-agent-skills/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-vaquarkhan-data-engineering-agent-skills-kafka-resilience-and-schema-evolution
- Seller: https://agentstack.voostack.com/s/vaquarkhan
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.