Run On Databricks

Databricks training task

Script and args: $ARGUMENTS

Steps

1. Read CLAUDE.md for cluster ID, experiment name, and environment setup.
2. Run a local smoke test first if the user hasn't confirmed the code runs locally:

uv run python scripts/train.py --datasets 1 --epochs 2 2>&1 | head -40 If it fails, stop and report the error. Don't submit broken code to the cluster.

1. Submit to Databricks (this builds a wheel, uploads it + scripts/train.py to DBFS, and runs):

uv run python scripts/submit_to_databricks.py scripts/train.py $ARGUMENTS The script passes --wheel-path and --experiment args automatically.

• To use an ephemeral job cluster (lower DBU rate) instead of the existing cluster, add --job-cluster
• Job clusters have ~5-10min startup overhead but cost less per DBU

1. Pull MLflow results:

uv run python scripts/pull_results_on_databricks.py

1. Read mlflow_results/latest_run.json and mlflow_results/all_runs.csv.
2. Report:

• Run status (SUCCESS / FAILED)
• Key metrics (loss, accuracy, or whatever the primary metric is)
• Training time and GPU type used
• Comparison with the previous best run from all_runs.csv

1. If the run failed, read mlflow_results/job_logs.txt and diagnose the error.

Common DBR 15+ issues:

• OSError: Operation not supported on Workspace paths — scripts must be on DBFS
• pydantic has no model_validator — stale system pydantic; check that the bootstrap clears sys.modules
• cannot import Sentinel from typing_extensions — same stale module issue
• BAD_REQUEST: For input string: "None" — MLflow experiment name must be a /Users/... workspace path
• DBFS library installations are not supported on DBR 15 — wheel must be pip-installed at runtime, not via compute.Library

Output

Summarize results and propose the next experiment with specific hyperparameter or architecture changes. ultrathink

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: duonginspace
• Source: duonginspace/claude-code-databricks-ml
• License: MIT

Install and usage instructions live in the source repository linked above.