# Databricks Testing Validation

> Data testing patterns for Databricks - DLT expectations, Delta constraints, Great Expectations, dbt tests, Spark unit tests, reconciliation, UAT. Use when the user mentions: testing, unit test, data quality, validate, assertion, UAT, reconciliation, Great Expectations, dbt test, DLT expectations, Delta constraints, pytest, data validation, quality checks, row count, null check.

- **Type:** Skill
- **Install:** `agentstack add skill-santoshkanthety-databricks-agent-databricks-testing-validation`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [santoshkanthety](https://agentstack.voostack.com/s/santoshkanthety)
- **Installs:** 0
- **Category:** [Databases](https://agentstack.voostack.com/c/databases)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [santoshkanthety](https://github.com/santoshkanthety)
- **Source:** https://github.com/santoshkanthety/databricks-agent/tree/main/skills/databricks-testing-validation

## Install

```sh
agentstack add skill-santoshkanthety-databricks-agent-databricks-testing-validation
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Testing & Validation in Databricks

## Four Testing Levels

| Level | What | When |
|-------|------|------|
| 1. Source data tests | Profile raw data quality | Before Bronze write |
| 2. Pipeline tests | Reconcile row counts, sums | After each layer write |
| 3. Semantic model tests | Query accuracy, metric assertions | After Gold write |
| 4. UAT / Report tests | Business stakeholder sign-off | Pre go-live |

## Level 1: DLT Expectations (Inline Quality Gates)

```python
import dlt
from pyspark.sql import functions as F

@dlt.table(name="silver_orders")
@dlt.expect_or_drop("non_null_order_id",   "order_id IS NOT NULL")       # drop invalid rows
@dlt.expect_or_drop("non_null_customer",    "customer_id IS NOT NULL")
@dlt.expect_or_drop("positive_revenue",    "revenue >= 0")
@dlt.expect_or_fail("no_future_dates",     "order_date = 0);

ALTER TABLE catalog.silver.orders
  ADD CONSTRAINT valid_status CHECK (order_status IN ('pending','confirmed','shipped','delivered','cancelled'));

-- List constraints
SHOW TBLPROPERTIES catalog.silver.orders;

-- Remove a constraint
ALTER TABLE catalog.silver.orders DROP CONSTRAINT positive_revenue;
```

## Level 1: Spark DataFrame Assertions

```python
from pyspark.sql import functions as F

def assert_no_nulls(df, columns: list[str], table: str = ""):
    for col in columns:
        count = df.filter(F.col(col).isNull()).count()
        assert count == 0, f"[{table}] Column '{col}' has {count} null values"

def assert_unique(df, columns: list[str], table: str = ""):
    total    = df.count()
    distinct = df.select(columns).distinct().count()
    assert total == distinct, f"[{table}] {total - distinct} duplicate rows on {columns}"

def assert_row_count_between(df, min_rows: int, max_rows: int, table: str = ""):
    count = df.count()
    assert min_rows = DATEADD(DAY, -7, CURRENT_DATE())
GROUP BY 1
ORDER BY pass_rate ASC
```

## CLI Reference

```bash
databricks-agent sql validate --table catalog.silver.orders --checks not-null:order_id,unique:order_id
databricks-agent sql reconcile --source bronze.orders --target silver.orders --measure revenue
databricks-agent sql test --config tests/quality.yaml
databricks-agent pipelines events --name orders-pipeline --filter expectations  # DLT quality events
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [santoshkanthety](https://github.com/santoshkanthety)
- **Source:** [santoshkanthety/databricks-agent](https://github.com/santoshkanthety/databricks-agent)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-santoshkanthety-databricks-agent-databricks-testing-validation
- Seller: https://agentstack.voostack.com/s/santoshkanthety
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.