Bauplan Data Assessment

Data Assessment

This skill takes a business question and determines whether available data in a Bauplan lakehouse can answer it. The output is a structured feasibility report with a clear verdict.

This skill is read-only. It explores and profiles data but never creates tables, imports data, runs pipelines, or merges branches.

If the user's request requires writing data, stop and suggest switching to the appropriate skill (bauplan-safe-ingestion, bauplan-data-pipeline, etc.).

Use the Bauplan Python SDK for all data operations. Create a working folder called data-assessment and iterate on a Python file named data_assessment.py inside it.

Environment Setup

Before writing any Python, check whether the project uses uv (look for pyproject.toml or uv.lock). If so, use uv run python to execute scripts and uv add to install packages. Otherwise, use the system python and pip install.

Ensure the required packages are installed:

• bauplan (the Bauplan Python SDK — required)
• polars (preferred for DataFrame operations — zero-copy Arrow interop)

Do not use pandas. Bauplan's client.query() returns a PyArrow table directly. Polars reads Arrow natively with zero-copy (pl.from_arrow(table)). Pandas requires a full data copy and is slower and more memory-intensive.

When to Use This Skill

Use this skill when:

• A user has a business question and wants to know if the data can answer it
• A user asks "can we figure out X from what we have?"
• A user wants to understand what data is available before committing to an analysis or pipeline
• A user needs to translate business language into concrete tables and columns

Do NOT use this skill for open-ended browsing with no question in mind (use bauplan-explore-data).

Required Inputs

Before starting, you need:

1. A business question (required): The user's question in their own words. If the question is vague, help them sharpen it before proceeding (see Phase 0).
2. Branch or ref to explore (required): Ask the user explicitly which branch contains the data they want to assess. Do not silently default to main — ask every time, even if main is the likely answer.
3. Namespace (optional): If the user works in a specific namespace. Default to bauplan.

CRITICAL RULES

1. Read-only. Never create tables, import data, run pipelines, or merge branches.
2. All queries must specify an explicit ref. Never rely on implicit defaults.
3. Always use LIMIT and explicit column selection. No unbounded queries.
4. Phases are sequential with gates. Do not skip ahead. If a gate blocks, stop and report.
5. Distinguish facts from assumptions. Every finding must be labeled as observed (from a query) or assumed (inferred by the agent). Never present assumptions as facts.
6. Profile only what matters. Do not run exhaustive profiling on every column. Profile only the columns that participate in the business question.
7. Show your work, then ask. At every decision point where the agent selects, excludes, or interprets data assets, it must present its reasoning to the user and get confirmation before proceeding. Do not commit to choices silently.

Workflow Overview

Phase 0: Decompose the business question
    │
    ⟳  CHECKPOINT: Present decomposition to user, get confirmation
    │
    ▼
Phase 1: Map concepts to data assets
    │
    ⟳  CHECKPOINT: Present ALL tables (candidates + rejected) with reasoning
    ⟳  CHECKPOINT: If multiple columns compete for same concept, ask user to choose
    │
    Gate: critical concepts have no backing → STOP
    │
    ▼
Phase 2: Assess data quality (scoped to the question)
    │
    Gate: critical tables are unusable → STOP
    │
    ▼
Phase 3: Validate semantic fit
    │
    ▼
Phase 4: Render verdict and write feasibility report
    │
    ▼
Handoff: offer to proceed to one-off analysis

---

Phase 0: Decompose the Business Question

Goal: Turn the user's question into concrete, checkable components — with the user's active participation. This phase is a conversation, not a form. Stay here until the user has a clear picture of what they're actually asking for.

Before touching any data, parse the business question into:

• Metric: What is being measured? ("churn rate," "average fulfillment time," "revenue per region")
• Grain: At what level? (per customer, per day, per region)
• Dimensions: What do we slice or group by? (region, time period, product category)
• Time scope: What period matters? (last quarter, trailing 12 months, since launch)
• Implicit filters: Who or what is included? (all customers or only active ones? all orders or only completed?)

Most business questions leave several of these unspecified. Do not fill gaps silently. Instead:

1. Identify what the user specified explicitly.
2. List what is unspecified or ambiguous.
3. Classify each gap as critical or deferrable (see below).
4. Present the gaps to the user and ask them to clarify.

Critical vs. Deferrable Gaps

Not all gaps are equal. Before moving on, classify each unspecified component:

Critical gaps — the analysis cannot proceed without resolving these, even approximately. These include:

• What the core entity is (e.g., what counts as a "customer," a "transaction," a "trip")
• What the metric actually measures (e.g., does "revenue" mean gross sales or profit?)
• What the grain is (per-day vs. per-month changes the entire query shape)

Deferrable gaps — these can be resolved later by inspecting what data actually exists. These include:

• Exact column choices when multiple candidates exist (e.g., which of three price columns to use — Phase 1 can surface options)
• Precise filter boundaries (e.g., exact date range — data coverage will constrain this anyway)
• Aggregation details (e.g., whether to use mean or median — can be decided after seeing distributions)

Rule: Critical gaps must be resolved in Phase 0. Deferrable gaps can become open questions for Phase 1.

When the User Says "I Don't Know"

This is the most important moment in the assessment. Do not move on. When a user cannot answer a question about their own business question, it means they need help thinking it through — not that the question should be parked.

Stay in Phase 0 and help them reason through it. Use their stated role and business context to offer concrete scenarios. The goal is not to give them the answer — it's to help them discover what they actually mean.

Example — the user asks "who are our top customers during bank holidays" and when asked "what is a customer?" they say "I don't know, I need help understanding that":

Good question to think through. Since you're in ecommerce, "customer"
could mean a few different things in your data:

  • A unique email address — simplest, but the same person might
    use multiple emails
  • An account ID — if your system requires login to purchase
  • A billing name or address — catches cases without accounts,
    but "John Smith" is ambiguous
  • A device/session — probably too granular for what you want

For "top customers by revenue during bank holidays," the answer
matters a lot. If someone buys once for $500 under one email and
once for $300 under another, are they one $800 customer or two
separate ones?

What does your ecommerce system use as the primary customer
identifier? If you're not sure, I can look at what identifiers
exist in the data and show you the options — but I want to make
sure we know what "same customer" means before I start counting.

Key behaviors when coaching:

• Use the user's stated role/domain to make examples concrete (an ecommerce manager gets ecommerce examples, not abstract ones)
• Explain why the distinction matters for their specific question (not in general)
• Offer 2–4 concrete options, not open-ended questions
• If the user still can't decide, offer to inspect the data together: "Let me show you what identifier columns exist, and we can decide together." This transitions to a joint exploration at the start of Phase 1, not a silent scan.

Presenting the Decomposition

Once gaps have been addressed (resolved or explicitly deferred with the user's knowledge), present the full decomposition and confirm the branch:

Before I start exploring, which branch should I look at?
(Default is main — but if you have a dev branch with newer
data or different tables, I should use that instead.)

Always ask explicitly. Do not silently default to main. The user may have data on a development branch that isn't on main yet, or main may have stale data. Getting this wrong wastes the entire assessment.

Then present the confirmed decomposition:

Here's what we've landed on:

- Metric: total revenue (sum of order values), ranked highest to lowest
- Grain: per customer (using account ID as the identifier)
- Dimensions: ranked list (top N — we said top 20)
- Time scope: all US federal holidays in 2024
- Filters: only completed orders (excluding cancellations and refunds)

Open questions we'll resolve by looking at the data:
- Which revenue column to use (we'll look at what's available)
- Whether "bank holiday" means the day itself or includes
  the surrounding weekend

Does this capture what you're after?

CRITICAL: Do not proceed past Phase 0 without presenting the decomposition to the user and getting confirmation. The user must see what you understood and agree before you start scanning data. This is not optional — skipping confirmation is the primary cause of wasted work in downstream phases.

Record the decomposition. It will form the first section of the feasibility report.

Proceed to Phase 1 once the user has confirmed the decomposition AND all critical gaps are resolved AND the target branch is confirmed. Deferrable gaps can remain as open questions.

---

Phase 1: Map Concepts to Data Assets

Goal: Determine which tables and columns could represent each business concept — and make the selection process fully transparent to the user.

1A. Discover ALL tables in scope

import bauplan

client = bauplan.Client()
ref = ""

# List all tables in the target namespace
tables = list(client.get_tables(ref=ref, filter_by_namespace="bauplan"))
for t in tables:
    print(t.name)

Record the complete list. Every table in the namespace is a candidate until explicitly excluded.

1B. Triage tables into candidates and non-candidates

For each table, get its schema and row count, then make a quick relevance judgment:

table = client.get_table(table="", namespace="bauplan", ref=ref)
fields = [(c.name, c.type) for c in table.fields]
num_records = table.records

Classify each table into one of:

• Strong candidate: Table name or columns clearly relate to a business concept from Phase 0. State which concept and why.
• Weak candidate: Table might be relevant but the connection is indirect or uncertain. State the reasoning.
• Not relevant: Table clearly does not relate to any business concept. State why (e.g., "contains geographic metadata with no connection to the metric").

CRITICAL: Present the full triage to the user before proceeding. Show all three categories with your reasoning. The user may know that a table you classified as "not relevant" is actually critical, or that a "strong candidate" is stale/deprecated. This is the most common point where silent assumptions cause problems.

Example presentation:

I found 8 tables in the bauplan namespace. Here's how I'd classify them
for your question about "average trip revenue by borough":

STRONG CANDIDATES:
  • taxi_fhvhv (14.2M rows) — contains trip records with fare columns
    (base_passenger_fare, total_amount, tips) and pickup location IDs.
    This looks like the primary fact table for trip revenue.
  • taxi_zones (265 rows) — location reference table with Borough, Zone,
    and LocationID. Needed to map location IDs to borough names.

WEAK CANDIDATES:
  • taxi_trips_summary (12K rows) — monthly pre-aggregated summary.
    Could work but we'd lose per-trip granularity.

NOT RELEVANT (for this question):
  • driver_licenses (45K rows) — driver data, no fare/trip info
  • vehicle_registry (82K rows) — vehicle metadata
  • weather_daily (3.6K rows) — weather data
  • zone_boundaries (265 rows) — GIS polygons
  • rate_codes (6 rows) — fare rate definitions

Do these classifications look right? Is there a table I should
reconsider, or one you know is the right source for this question?

Wait for the user's response. They may redirect you ("actually use taxitripsclean, not taxi_fhvhv — that one has known data issues") or confirm. Do not proceed until the user has acknowledged the candidate list.

1C. Deep-inspect confirmed candidates

Only after user confirmation, inspect the strong and weak candidates in detail:

table = client.get_table(table="", namespace="bauplan", ref=ref)
fields = [(c.name, c.type) for c in table.fields]
num_records = table.records

For each confirmed candidate, identify which columns could map to which business concepts. When multiple columns could serve the same concept (e.g., price, base_passenger_fare, total_amount for "revenue"), list all of them — do not pick one silently.

1D. Build the concept-to-column mapping

For each business concept (metric, dimensions, filters), identify:

• Which table contains a plausible column
• The column name and type
• Any transformation needed (e.g., "revenue" requires summing price where event_type = 'purchase')
• Whether a join is needed to connect this concept to the main entity
• If there are multiple candidate columns for the same concept, list all of them and ask the user which one to use

Present the mapping to the user. When alternatives exist, surface them explicitly:

| Business Concept | Table | Column | Type | Notes | |------------------|--------|---------------------|---------|-----------------------------------------------| | revenue | events | price | decimal | Only where eventtype='purchase' | | revenue (alt) | events | totalamount | decimal | Includes tax and fees — is this what you mean? | | region | zones | Borough | string | Requires join on LocationID |

When multiple columns compete for the same concept, explicitly ask the user to choose. This is the most important interaction in the entire skill. Example: "I found two columns that could represent 'revenue': base_passenger_fare (excludes tips and surcharges) and total_amount (includes everything). Which one matches what you mean by 'revenue'?"

1E. Identify gaps

Flag any business concept that has NO plausible column backing. These are blockers.

Also flag any open questions from Phase 0 that the data inspection has now clarified or complicated.

Gate: Can we proceed?

• If ALL critical concepts (the metric and at least one dimension) have confirmed column mappings → proceed to Phase 2.
• If ANY critical concept has no backing → STOP. Report what's missing. Tell the user which concepts cannot be mapped and suggest alternatives if any exist (e.g., "there's no 'region' column, but there's a LocationID that could be joined to a zones table").

---

Phase 2: Assess Data Quality (Scoped to the Question)

Goal: Determine whether the mapped columns are trustworthy enough for this specific analysis.

Profile ONLY the tables and columns identified in Phase 1. Do not profile the entire catalog.

2A. For each mapped table, run the targeted profiling battery

Freshness — How recent is the latest record?

q = "SELECT MAX(event_time) AS latest FROM bauplan."
res = client.query(q, ref=ref, max_rows=1)

Completeness — Null rate on the columns we need:

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: BauplanLabs
• Source: BauplanLabs/bauplan-skills
• License: MIT

Install and usage instructions live in the source repository linked above.