Connect Cdc Spanner

Redpanda Connect CDC: Google Cloud Spanner

The gcp_spanner_cdc input in Redpanda Connect streams change data capture (CDC) from a Google Cloud Spanner database into Redpanda or any Kafka-compatible topic. It uses Spanner's native change stream API, tracks multiple concurrent partitions, persists watermark state in a Spanner metadata table, and delivers each row mutation as a JSON message with rich metadata.

Introduced in version 4.56.0. This is an Enterprise feature — a Redpanda Enterprise license is required.

The connector supports both GoogleSQL and PostgreSQL Spanner dialects. It automatically detects the dialect and creates the metadata table in the same database if it does not already exist. Partitions split and merge over time as Spanner scales; the connector handles all partition lifecycle events transparently.

Quickstart

1. Create the Spanner change stream (one DDL statement)

-- Track all tables in the database:
CREATE CHANGE STREAM AllChanges FOR ALL;

-- Or track specific tables:
CREATE CHANGE STREAM OrderChanges FOR orders, customers;

-- Spanner default is OLD_AND_NEW_VALUES; opt into NEW_VALUES to drop old values:
CREATE CHANGE STREAM OrderChanges FOR orders, customers
  OPTIONS (value_capture_type = 'NEW_VALUES');

2. Create a GCP service account and grant it the required IAM roles

The connector issues a CREATE TABLE IF NOT EXISTS DDL statement on every startup to create or validate the partition metadata table. This means the service account must retain DDL permission (spanner.databases.updateDdl) permanently — not just on first run.

# Create a service account
gcloud iam service-accounts create redpanda-spanner-cdc \
  --display-name="Redpanda Spanner CDC"

PROJECT=MY_PROJECT
SA=redpanda-spanner-cdc@${PROJECT}.iam.gserviceaccount.com

# Grant Spanner Database Reader (read data and change streams)
gcloud spanner databases add-iam-policy-binding my-database \
  --instance=my-spanner-instance \
  --project=${PROJECT} \
  --member="serviceAccount:${SA}" \
  --role="roles/spanner.databaseReader"

# Grant DDL permission — required on every startup for metadata table setup
gcloud spanner databases add-iam-policy-binding my-database \
  --instance=my-spanner-instance \
  --project=${PROJECT} \
  --member="serviceAccount:${SA}" \
  --role="roles/spanner.databaseAdmin"

# Download the JSON key file and base64-encode it for credentials_json
gcloud iam service-accounts keys create spanner-cdc-key.json \
  --iam-account=${SA}

export SPANNER_CDC_CREDENTIALS=$(base64 `) to track partition state (CREATED → SCHEDULED → RUNNING → FINISHED) and per-partition watermarks.
2. Discovers root partitions at startup (or on resume).
3. Queries each partition concurrently for change records.
4. When a partition splits or merges, the connector automatically detects and schedules child partitions.
5. Watermarks are updated after each message is acknowledged downstream.
6. On restart, interrupted (SCHEDULED or RUNNING) partitions are resumed from their last watermark.

## Reference Directory

- [Config Reference](references/config-reference.md): Every `gcp_spanner_cdc` field with type, default, and description — grounded in the source.
- [Setup Spanner](references/setup-spanner.md): Creating the change stream, IAM permissions, metadata table details, dialects, and retention notes.
- [Pipeline and Output](references/pipeline-and-output.md): Full pipeline YAML examples, the message payload and metadata shape, per-table routing with Bloblang, landing CDC into an Iceberg topic, and restart/resume behavior.
- [Enterprise Features](references/enterprise-features.md): Redpanda enterprise differentiators for Spanner CDC — the connector's Enterprise license gating; Iceberg Topics (`redpanda.iceberg.mode/delete/invalid.record.action/partition.spec/target.lag.ms`); Tiered Storage (`redpanda.remote.write/read`, `cloud_storage_enabled`); Cloud Topics (`redpanda.cloud_topic.enabled` / `redpanda.storage.mode=cloud`, `cloud_topics_enabled`); Remote Read Replicas (`redpanda.remote.readreplica`); Shadowing DR (`rpk shadow`); the Connect `redpanda` config-service block, secrets management, allow/deny lists, FIPS; and destination-cluster security (RBAC, OIDC/OAUTHBEARER, Kerberos, Audit Logging, Schema ID Validation). All require an Enterprise license.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [redpanda-data](https://github.com/redpanda-data)
- **Source:** [redpanda-data/skills](https://github.com/redpanda-data/skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.