Databricks

Databricks - Playbook

Companion rule: [481-databricks.mdc](../../rules/481-databricks.mdc). This skill turns those patterns into end-to-end workflows.

> [!NOTE] > Current Databricks names: Lakeflow Spark Declarative Pipelines is the current name for Delta Live Tables (DLT). Declarative Automation Bundles is the current name for Databricks Asset Bundles (DABs). Existing DLT/DAB code and docs still work, but new guidance should use the current names and mention the former names for searchability.

---

When to invoke

Use when the user is:

• Bootstrapping a new Databricks workspace or account
• Migrating to Unity Catalog (from Hive metastore or no-governance state)
• Designing or reviewing a Lakeflow Spark Declarative Pipeline (formerly DLT)
• Deploying jobs, notebooks, and pipelines with Declarative Automation Bundles
• Setting cluster policies or rolling out Serverless / SQL Warehouses
• Tuning cost (cluster sizing, Photon, spot, job compute vs all-purpose)
• Governing data (Unity Catalog, tags, access control, lineage)
• Recovering from a bad change (Delta time travel / RESTORE)
• Hardening access (SCIM, SSO, service principals, OAuth tokens)

---

Golden Rules

1. Unity Catalog is the default. Hive metastore is for legacy migration only.
2. Service principals and OAuth, not personal access tokens, for automation.
3. Standard access mode by default. Dedicated access mode only when the workload requires it; no-isolation shared is not acceptable for governed production.
4. Job compute for jobs. All-purpose clusters are for exploration.
5. Delta is the default table format; Delta Lake for Spark, Delta tables under Unity Catalog.
6. Lineage and tags are not optional - govern at ingest, not post hoc.
7. Every notebook / repo change runs in CI before it runs against prod data.
8. Declarative Automation Bundles deploy projects; Terraform provisions account/workspace infrastructure.
9. System tables are operational infrastructure for cost, audit, lineage, and query history.

Non-Negotiables

• Production data uses Unity Catalog: three-part names, group/service-principal ownership, governed external locations or volumes.
• Catalog topology maps to business domain, environment, region/data residency, and workspace binding requirements before production objects are created.
• No persistent DBFS mounts (dbfs:/mnt/...) for governed production data.
• No personal access tokens in production automation.
• No account users grants, individual owners, or broad ALL PRIVILEGES grants without documented exception.
• BROWSE, APPLY TAG, and MANAGE are metadata-only capabilities; do not confuse them with data access.
• Standard access mode is the default. Dedicated access mode must state the technical reason.
• Deployable jobs and Lakeflow pipelines use Declarative Automation Bundles with databricks bundle validate in CI.
• Serverless SQL / jobs / Lakeflow are preferred where available and appropriate; classic clusters require a reason and a policy.
• PII/PCI/sensitive tables require tags, masks, row filters, and audit visibility before gold/serving publication.

---

Workflow 1 - Workspace Bootstrap

Stand up a new workspace with governance, security, and cost controls from day one.

Steps

1. Account-level setup:

• Account console used for identity, Unity Catalog metastore, and workspace provisioning
• SSO via Okta / Entra ID / Google with SCIM provisioning
• Admin role in a small, audited group

1. Unity Catalog metastore per region, tied to cloud storage location (one per region, one per env optionally)
2. Workspace deployment via Terraform (databricks provider):

• Network: private subnets, VPC endpoints / Private Link
• Customer-managed keys for notebooks and workspace storage
• Workspace bound to specific metastore

1. Groups and access (via SCIM from IdP):

• admins- - workspace admins
• data-engineers-, data-analysts-, data-scientists-, platform-
• Keep global admin list tiny

1. Cluster policies (see Workflow 4) for all user-facing compute
2. Catalog structure:

• Catalogs per environment (dev, stage, prod) or per domain (sales, marketing)
• Schemas within: raw, staging, curated, sandbox
• Default grants minimal; team-specific access roles

1. External locations + storage credentials (Unity Catalog) for cloud object storage; no direct S3/ADLS/GCS creds in notebooks
2. Budget and alerts:

• System tables for billing (system.billing.usage)
• Dashboards for cost by workspace / cluster / user / job
• Alerts on spend anomalies

Deliverable: Terraform modules (account, workspace, UC, groups, policies), bootstrap runbook, cost dashboard.

See [references/workspace-bootstrap.md](references/workspace-bootstrap.md).

---

Workflow 2 - Unity Catalog Migration

Migrate from Hive metastore (or no governance) to Unity Catalog.

Steps

1. Inventory:

• All databases / tables in Hive
• Locations (mount points, dbfs:/mnt/..., direct S3 paths)
• Access patterns (who reads / writes)
• Data pipelines depending on Hive paths

1. Metastore setup (if not already done).
2. External locations in Unity Catalog for the cloud storage backing the Hive tables.
3. Catalog and schema creation that mirrors the source structure (or a better one).
4. Migration options:

• Upgrade in place: SYNC command for schemas, or CREATE TABLE ... LIKE ... LOCATION ... to point UC tables at existing storage
• Copy: CREATE TABLE ... AS SELECT ... into new UC tables (physical re-write; expensive but clean)
• Dual-write during transition: write to both Hive and UC, read from UC, retire Hive

1. Lineage and ownership:

• Set UC owner to a group, not a person
• Add tags for sensitivity, owner, data steward

1. Access control:

• Grants at catalog / schema / table level
• No ANONYMOUS FUNCTION / ALL PRIVILEGES sprawl
• Column-level masking / row filters where needed

1. Cut over pipelines and BI:

• Update SQL Warehouses to use UC as default catalog
• Update notebooks / jobs to reference catalog.schema.table three-part names
• Disable Hive metastore access path after validation

1. Decommission Hive once all reads have moved.

Deliverable: Migration plan per database, cutover runbook, rollback plan, governance model (owners, stewards, tags, classifications).

---

Workflow 3 - Catalog Topology and Delegated Ownership

Translate organization structure into catalogs, schemas, workspaces, storage, and ownership.

Design dimensions

1. Metastore / region strategy: one metastore per cloud-region; use Delta Sharing for cross-region/cross-cloud sharing.
2. Catalog boundary: business unit, domain, environment, region, or shared data product.
3. Workspace binding: which workspaces can see/use each catalog.
4. Storage boundary: metastore, catalog, or schema managed location; external location for externally managed data.
5. Ownership and delegation: group/service-principal owners, MANAGE delegation, data stewards.

Topology patterns

| Pattern | Example | Use when | |---|---|---| | Direct separation | finance_prod, finance_stage, finance_dev | Environment isolation and workspace binding matter most | | Indirect separation | finance catalog with prod_*, stage_*, dev_* schemas | Fewer catalogs and team autonomy matter more | | Region + domain | emea_finance_prod, apac_sales_prod | Data residency or regional governance is a hard boundary | | Shared reference | shared_reference_prod | Cross-domain data products or common dimensions |

Ownership model

• OWNER: full control, unique per object, can transfer ownership. Use groups/service principals in production.
• MANAGE: can grant/revoke permissions without implicit data access or ownership transfer.
• Delegation pattern:

1. Metastore admin grants CREATE CATALOG to catalog owner groups.
2. Catalog owner groups create catalogs and schemas.
3. Catalog/schema owners grant MANAGE to domain lead groups.
4. Data stewards get APPLY TAG; broad users may get BROWSE for discovery.

Deliverable: topology decision record showing domain/env/region catalog names, workspace bindings, managed locations, owner groups, MANAGE delegates, and discovery/tagging groups.

---

Workflow 4 - Storage Governance and External Access

Design where data physically lives and how Unity Catalog mediates access.

Managed storage hierarchy

Unity Catalog resolves managed storage from most specific to least specific:

1. Schema managed location
2. Catalog managed location
3. Metastore default storage

Use schema-level managed locations for regulated data requiring physical isolation. Use catalog-level locations for domain/environment boundaries. Keep metastore default storage as a fallback.

Tables vs volumes

| Need | Use | |---|---| | Tabular data queried by SQL/Spark | Delta table (catalog.schema.table) | | Non-tabular files, PDFs, images, configs, ML artifacts | Unity Catalog volume (/Volumes/catalog/schema/volume/...) | | Existing cloud files where lifecycle remains outside Databricks | External table or external volume | | Legacy/external database without immediate copy | Foreign catalog / Lakehouse Federation |

External access chain

For external data, require the full chain:

1. Storage credential - cloud identity Unity Catalog can use.
2. External location - non-overlapping cloud path bound to the credential.
3. External table/volume - governed object using the external location.

Reject overlapping external locations, raw cloud credentials in notebooks, and direct cloud paths where UC volumes/external locations should be used.

Deliverable: storage map showing managed locations, storage credentials, external locations, volumes, foreign catalogs, owners, and allowed write paths.

---

Workflow 5 - Privileges, Discovery, FGAC, and ABAC

Implement least privilege and scalable fine-grained access.

Privilege chain

To query a table, users need:

1. USE CATALOG
2. USE SCHEMA
3. SELECT on the table/view

Higher-level grants inherit downward. Grant at the highest safe scope; avoid per-table grants unless the table is exceptional.

Metadata-only privileges

• BROWSE: discover objects and metadata without reading data.
• APPLY TAG: classify assets without data access.
• MANAGE: delegate permission administration without ownership or implicit data access.

FGAC choices

| Pattern | Use when | |---|---| | Row filter | Rows visible depend on user/group/context | | Column mask | Sensitive columns need redaction/transformation | | ABAC governed-tag policy | Same row/mask logic applies across many tables/columns | | Dynamic view | Legacy/special-case logic not expressible as table-attached policy |

Prefer ABAC for enterprise scale: governed tags + inheritable policies beat per-table copy/paste filters and masks.

Validation

Always validate outcomes:

SHOW GRANTS ON CATALOG finance_prod;
SHOW GRANTS `analysts` ON SCHEMA finance_prod.sales;
SHOW POLICIES ON CATALOG finance_prod;
SHOW POLICIES ON SCHEMA finance_prod.sales;
DESCRIBE TABLE EXTENDED finance_prod.sales.transactions;

Audit signals:

• grant/revoke actions
• ownership transfers
• policy and tag changes
• token creation
• admin role changes
• denied/failed access attempts

Deliverable: grants/policies matrix plus validation query outputs and system-table audit queries.

---

Workflow 6 - Lakeflow Spark Declarative Pipelines (formerly DLT)

Lakeflow Spark Declarative Pipelines for declarative, managed, observable pipelines. Existing DLT syntax still works; new Python should prefer from pyspark import pipelines as dp.

When Lakeflow

• Streaming / micro-batch SCD pipelines
• Data quality enforcement (expectations) required
• Lineage and observability wanted out of the box
• Team prefers declarative SQL / Python over hand-orchestrated jobs

When NOT Lakeflow

• One-off batch jobs
• Extensive custom Spark (Lakeflow constrains some APIs)
• Cost sensitivity with low duty cycle (pipeline compute has overhead)

Current Python naming

from pyspark import pipelines as dp

@dp.table
def raw_events():
    return spark.readStream.table("prod.raw.events")

@dp.materialized_view
def daily_event_counts():
    return spark.read.table("prod.silver.events").groupBy("event_date").count()

DLT compatibility mapping:

• import dlt -> from pyspark import pipelines as dp
• @dlt.table for streaming -> @dp.table
• @dlt.table for batch -> @dp.materialized_view
• @dlt.view -> @dp.temporary_view
• dlt.apply_changes(...) -> dp.create_auto_cdc_flow(...)

Design

1. Ingest to raw (bronze) from cloud storage via Auto Loader or Kafka
2. Stage (silver) with schema enforcement, de-dup, CDC via APPLY CHANGES INTO
3. Curated (gold) for business aggregates / dimensional model
4. Expectations on each layer:

• @expect_or_drop for quality-gate columns
• @expect_or_fail for must-pass invariants
• Track failure counts in DLT event log

1. Serverless Lakeflow where available for lower cost and managed sizing
2. Continuous vs triggered: continuous for streaming SLAs; triggered for batch windows
3. Target target_lag with Dynamic Tables equivalents or DLT settings
4. Naming and lineage: three-part UC names end-to-end; tags on every table

Guardrails

• Pipeline in Git; deployed via Declarative Automation Bundles or Terraform
• CI: unit tests for transformations, integration test against a dev catalog
• Alerting: event log -> metrics -> alerts on expectation failures and pipeline failures
• Cost budget per pipeline

---

Workflow 7 - Cluster Policies

Enforce cost, security, and compatibility at cluster-creation time.

Policies to define

| Policy | Audience | Shape | |---|---|---| | interactive-small | Data analysts | Single-node or 1-4 workers; auto-terminate 30-60 min; Standard access mode | | interactive-medium | Data engineers / scientists | 1-8 workers; auto-terminate 60 min; Standard access mode | | job-compute | Jobs / Lakeflow | Fixed instance types; no interactive attach; auto-scale caps | | ml-gpu | Data scientists | GPU-enabled types; restricted to ML group; auto-terminate 30 min | | shared-serverless-sql | SQL users | Serverless SQL Warehouse sizes S/M/L, governed by policy |

Policy content

• Enforced tags: cost_center, owner, env, pipeline (billable by tag)
• Runtime pinning to LTS + latest patch
• Init scripts disallowed or restricted to vetted list
• Access mode = Standard for most UC workloads; Dedicated only by documented exception
• DBFS / mount usage disallowed in UC-only workspaces
• Instance profiles / IAM roles via managed service credentials

Terraform

resource "databricks_cluster_policy" "interactive_small" {
  name       = "interactive-small"
  definition = jsonencode({
    "spark_version"       = { "type": "fixed", "value": "14.3.x-scala2.12" },
    "autotermination_minutes" = { "type": "range", "minValue": 15, "maxValue": 60 },
    # Provider-level field/value may vary by Databricks provider version.
    # Human-facing policy: Standard access mode for most UC workloads.
    "data_security_mode"  = { "type": "fixed", "value": "USER_ISOLATION" },
    "custom_tags.cost_center" = { "type": "unlimited" },
    # ...
  })
}

Then grant CAN_USE to the right group.

---

Workflow 8 - Cost and Performance Tuning

Levers (ordered by impact)

1. Right compute type:

• Serverless SQL for BI
• Job compute (not all-purpose) for jobs
• Serverless Lakeflow where eligible

1. Photon on for Spark SQL / DataFrame workloads - almost always faster and cheaper
2. Auto-scaling - set min low; max capped by

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: d-padmanabhan
• Source: d-padmanabhan/agent-engineering-handbook
• License: MIT

Install and usage instructions live in the source repository linked above.