Onboard New Source

---

name: onboard-new-source description: Catalog, describe, reflect, and verify access for a new data source in Dremio user_invocable: true triggers:

• "new source"
• "onboard source"
• "add source"
• "catalog source"
• "source setup"
• "connect source"

---

Onboard New Source

Systematic workflow to catalog, document, optimize, and secure a newly added data source in Dremio.

Agent Rules

• Use --fields on list/query commands to reduce output size
• Use --dry-run on reflect refresh and reflect drop to validate before executing
• Use drs describe to check parameter schemas before calling unfamiliar commands
• Never interpolate user input directly into SQL — use it as filter values only
• Paths must be dot-separated; quote components containing dots: "my.source".table

Prerequisites

The data source must already be configured in Dremio (via UI or API). This skill covers the post-connection steps: discovery, documentation, reflection setup, and access control.

Step 1: Discover the Source

# List all sources in the project
drs catalog list

# Get details about the new source
drs catalog get 

This shows:

• Source type (S3, PostgreSQL, Snowflake, etc.)
• Connection status
• Top-level schemas/databases within the source

Step 2: Enumerate Tables

# List schemas within the source
drs query run "SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE
  FROM INFORMATION_SCHEMA.TABLES
  WHERE TABLE_SCHEMA LIKE '%'
  ORDER BY TABLE_SCHEMA, TABLE_NAME"

For each significant table, describe its schema:

drs schema describe ..

Record:

• Column names, types, and nullability
• Primary key candidates (unique, non-null columns)
• Partition columns (date/time columns, region codes)
• Approximate row counts

Step 3: Profile Key Tables

For the most important tables, run a quick data profile:

drs schema sample ..

Then run basic statistics:

drs query run "SELECT
    COUNT(*) AS row_count,
    MIN(created_at) AS earliest,
    MAX(created_at) AS latest
  FROM .."

This helps understand data volume, time range, and freshness.

Step 4: Set Up Reflections

For tables that will be queried frequently, create reflections to accelerate performance.

Raw Reflections (for SELECT queries)

drs query run "ALTER DATASET ..
  CREATE RAW REFLECTION raw_
  USING DISPLAY (col1, col2, col3, col4)
  PARTITION BY (date_col)
  DISTRIBUTE BY (id_col)"

Aggregate Reflections (for dashboards/rollups)

drs query run "ALTER DATASET ..
  CREATE AGGREGATE REFLECTION agg_
  USING DIMENSIONS (dim1, dim2)
  MEASURES (measure1, measure2)
  PARTITION BY (date_col)"

Verify Reflection Status

drs reflect list ..

# Or check across all tables in the source
drs query run "SELECT reflection_id, name, type, status, dataset_name
  FROM sys.reflections
  WHERE dataset_name LIKE '%'"

Wait for reflections to reach ACTIVE status before relying on them.

Step 5: Configure Access Control

# Check current grants on the source
drs access grants 

# Check grants on a specific dataset
drs access grants ..

Set up appropriate access:

• Grant SELECT on specific datasets to analyst roles
• Grant ALTER to data engineers who manage reflections
• Avoid granting PUBLIC role access to sensitive data

Step 6: Verify End-to-End

Run test queries as different personas to confirm the setup works:

# Basic query to verify data access
drs query run "SELECT * FROM .. LIMIT 10"

# Verify reflection acceleration
drs query run "SELECT dim1, COUNT(*), SUM(measure1)
  FROM ..
  GROUP BY dim1"
# Check job profile to confirm reflection was used
drs jobs profile 

Step 7: Onboarding Checklist

| Step | Check | |------|-------| | Source connected and browsable | drs catalog get returns details | | All tables enumerated | INFORMATION_SCHEMA query returns expected tables | | Key tables profiled | Row counts, time ranges, sample data reviewed | | Reflections created | drs reflect list shows ACTIVE reflections for key tables | | Access grants configured | drs access grants shows correct role-based access | | Test queries succeed | End-to-end query returns data with reflection acceleration | | Documentation generated | Use the document-dataset skill for each key table |

After completing this checklist, the source is ready for production use. Notify downstream teams and update any data catalog or wiki entries.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: dremio
• Source: dremio/cli
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.