Databricks Testing Validation

Testing & Validation in Databricks

Four Testing Levels

| Level | What | When | |-------|------|------| | 1. Source data tests | Profile raw data quality | Before Bronze write | | 2. Pipeline tests | Reconcile row counts, sums | After each layer write | | 3. Semantic model tests | Query accuracy, metric assertions | After Gold write | | 4. UAT / Report tests | Business stakeholder sign-off | Pre go-live |

Level 1: DLT Expectations (Inline Quality Gates)

import dlt
from pyspark.sql import functions as F

@dlt.table(name="silver_orders")
@dlt.expect_or_drop("non_null_order_id",   "order_id IS NOT NULL")       # drop invalid rows
@dlt.expect_or_drop("non_null_customer",    "customer_id IS NOT NULL")
@dlt.expect_or_drop("positive_revenue",    "revenue >= 0")
@dlt.expect_or_fail("no_future_dates",     "order_date = 0);

ALTER TABLE catalog.silver.orders
  ADD CONSTRAINT valid_status CHECK (order_status IN ('pending','confirmed','shipped','delivered','cancelled'));

-- List constraints
SHOW TBLPROPERTIES catalog.silver.orders;

-- Remove a constraint
ALTER TABLE catalog.silver.orders DROP CONSTRAINT positive_revenue;

Level 1: Spark DataFrame Assertions

from pyspark.sql import functions as F

def assert_no_nulls(df, columns: list[str], table: str = ""):
    for col in columns:
        count = df.filter(F.col(col).isNull()).count()
        assert count == 0, f"[{table}] Column '{col}' has {count} null values"

def assert_unique(df, columns: list[str], table: str = ""):
    total    = df.count()
    distinct = df.select(columns).distinct().count()
    assert total == distinct, f"[{table}] {total - distinct} duplicate rows on {columns}"

def assert_row_count_between(df, min_rows: int, max_rows: int, table: str = ""):
    count = df.count()
    assert min_rows = DATEADD(DAY, -7, CURRENT_DATE())
GROUP BY 1
ORDER BY pass_rate ASC

CLI Reference

databricks-agent sql validate --table catalog.silver.orders --checks not-null:order_id,unique:order_id
databricks-agent sql reconcile --source bronze.orders --target silver.orders --measure revenue
databricks-agent sql test --config tests/quality.yaml
databricks-agent pipelines events --name orders-pipeline --filter expectations  # DLT quality events

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: santoshkanthety
• Source: santoshkanthety/databricks-agent
• License: MIT

Install and usage instructions live in the source repository linked above.