# Run On Databricks

> Submit a training job to the Databricks GPU cluster, wait for results, and pull MLflow metrics back locally. Use when the user wants to train on GPU, run a full experiment, or execute code on the Databricks cluster. Triggers on "run on Databricks", "train on GPU", "submit job", "full training run".

- **Type:** Skill
- **Install:** `agentstack add skill-duonginspace-claude-code-databricks-ml-run-on-databricks`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [duonginspace](https://agentstack.voostack.com/s/duonginspace)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [duonginspace](https://github.com/duonginspace)
- **Source:** https://github.com/duonginspace/claude-code-databricks-ml/tree/master/skills/run-on-databricks

## Install

```sh
agentstack add skill-duonginspace-claude-code-databricks-ml-run-on-databricks
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Databricks training task

Script and args: $ARGUMENTS

## Steps

1. Read `CLAUDE.md` for cluster ID, experiment name, and environment setup.
2. Run a local smoke test first if the user hasn't confirmed the code runs locally:
   `uv run python scripts/train.py --datasets 1 --epochs 2 2>&1 | head -40`
   If it fails, stop and report the error. Don't submit broken code to the cluster.
3. Submit to Databricks (this builds a wheel, uploads it + scripts/train.py to DBFS, and runs):
   `uv run python scripts/submit_to_databricks.py scripts/train.py $ARGUMENTS`
   The script passes `--wheel-path` and `--experiment` args automatically.
   - To use an ephemeral job cluster (lower DBU rate) instead of the existing cluster, add `--job-cluster`
   - Job clusters have ~5-10min startup overhead but cost less per DBU
4. Pull MLflow results:
   `uv run python scripts/pull_results_on_databricks.py`
5. Read `mlflow_results/latest_run.json` and `mlflow_results/all_runs.csv`.
6. Report:
   - Run status (SUCCESS / FAILED)
   - Key metrics (loss, accuracy, or whatever the primary metric is)
   - Training time and GPU type used
   - Comparison with the previous best run from all_runs.csv
7. If the run failed, read `mlflow_results/job_logs.txt` and diagnose the error.
   Common DBR 15+ issues:
   - `OSError: Operation not supported` on Workspace paths — scripts must be on DBFS
   - `pydantic has no model_validator` — stale system pydantic; check that the bootstrap clears sys.modules
   - `cannot import Sentinel from typing_extensions` — same stale module issue
   - `BAD_REQUEST: For input string: "None"` — MLflow experiment name must be a `/Users/...` workspace path
   - `DBFS library installations are not supported on DBR 15` — wheel must be pip-installed at runtime, not via compute.Library

## Output
Summarize results and propose the next experiment with specific hyperparameter or architecture changes.
ultrathink

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [duonginspace](https://github.com/duonginspace)
- **Source:** [duonginspace/claude-code-databricks-ml](https://github.com/duonginspace/claude-code-databricks-ml)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-duonginspace-claude-code-databricks-ml-run-on-databricks
- Seller: https://agentstack.voostack.com/s/duonginspace
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.