# Cultivar

> Drive the cultivar CLI to test whether an agent skill improves behavior — scaffold tasks, run with/without the skill across Claude/Copilot/Gemini (locally or on Modal), grade against a rubric, and read the results.

- **Type:** Skill
- **Install:** `agentstack add skill-pinecone-io-cultivar-cultivar`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [pinecone-io](https://agentstack.voostack.com/s/pinecone-io)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [pinecone-io](https://github.com/pinecone-io)
- **Source:** https://github.com/pinecone-io/cultivar/tree/main/skills/cultivar

## Install

```sh
agentstack add skill-pinecone-io-cultivar-cultivar
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# cultivar

cultivar is a CLI that measures whether an agent **skill** actually improves an agent's
behavior. For each task it runs the agent **with the skill** and **without** it (and
optionally **with the source docs**), then an LLM grader scores each run against a
natural-language rubric. Use this skill when the user wants to create, run, or interpret
cultivar evals.

## The loop

1. `cultivar init ` — scaffold `tasks/.yaml` + a `SKILL.md` stub.
2. Edit the task file (intent + PASS/FAIL criteria) and the skill.
3. `cultivar run -s  -r  --grade` — run all variants and grade.
4. `cultivar report` / `cultivar show latest -t ` — read the outcome.
5. Iterate on the skill; re-run; compare.

Always confirm the install first with `cultivar hello` (or `cultivar hello --no-grade`
when no `ANTHROPIC_API_KEY` is available) — it runs a packaged smoke task end-to-end.

## Commands

- `cultivar init  [--skills-dir DIR]` — scaffold task YAML + SKILL.md stub.
- `cultivar run -s  -r ` — run. Key flags:
  - `-t ` one task · `-v ` one variant
  - `--remote` run in isolated Modal sandboxes · `-n N` repeat · `-p N` parallelism
  - `--grade` grade after running · `--title NAME` label the run · `--dry-run` print the
    prompt + command without calling anything · `--timeout S` per-call budget (default 90)
- `cultivar grade  -s  [--report]` — (re)grade an existing run.
- `cultivar report [run]` — summary table across runners/variants.
- `cultivar show  -t  [--grader|--conversation-only|--workdir]` — inspect one run.

`--dry-run` is the safe way to preview exactly what will be sent before spending tokens.

## Variants (the controls)

- **with-skill** — skill loaded; prompt prefixed `Use the /`.
- **without-skill** — no skill; identical otherwise. The baseline.
- **with-docs** — no skill, but the task's `ground_truth.context_refs` files are prepended.
  Only runs for tasks that declare `context_refs`.

Read two deltas: with-skill vs without-skill ("does the skill do anything?") and
with-skill vs with-docs ("is the distilled skill better than dumping the raw docs?").

## Tasks

`tasks/.yaml` holds one or more tasks. Each task:

```yaml
tasks:
  - id: a-short-id
    intent: "what you'd ask the agent to do"
    category: cli            # or: code-gen
    # setup / teardown / verify: optional shell hooks
    # env: ["SOME_KEY"]      # required env vars, checked upfront
    ground_truth:
      criteria: |
        PASS requires .
        FAIL if .
      # context_refs: [docs/ref.md]   # activates the with-docs variant
```

Guidance:
- For **code-gen** tasks, the intent must say "write a file … in the current directory."
  Anything the agent writes to its cwd is captured and shown to the grader. A code-gen
  task that produces no file auto-fails.
- Write criteria as crisp PASS conditions + at least one concrete FAIL mode — vague
  criteria produce vague grades.

## Where skills live

cultivar tests exactly **one** skill per run (the `-s` one). It resolves the skills root
as: `--skills-dir` flag → `CULTIVAR_SKILLS_DIR` env → `./.claude/skills`. Keep
skills-under-test outside `.claude/` (e.g. `./skills`, via `CULTIVAR_SKILLS_DIR=skills`)
if you don't want your interactive coding agent to auto-load them.

## Local vs remote

- **Local** (default) — uses the runner CLI installed on your machine + its auth.
- **`--remote`** — each (task, variant, repeat) runs in its own Modal sandbox: clean
  isolation, parallelism, reproducibility. Requires a Modal account (`modal token new`)
  and a secret holding the agent's `ANTHROPIC_API_KEY` (default secret name
  `eval-sandbox-secrets`; override with `CULTIVAR_MODAL_SECRET`). Prefer `--remote` for
  rigorous comparisons. The grader always runs locally and needs `ANTHROPIC_API_KEY`.

## Reading results

`results/[__title]/` holds per-run `.json` (stats), `.md` (readable trace),
`.jsonl` (raw events), and `.workdir/` (files the agent wrote). `grades.json` holds the
verdicts. Use `cultivar report` for the table and `cultivar show … --grader` for the
grader's reasoning + suggestions on a failure.

## Gotchas

- Grading needs `ANTHROPIC_API_KEY` (loaded from a `.env` in the cwd). `hello --no-grade`
  and `run --dry-run` need no key.
- `tasks/`, `examples/`, and `results/` are cwd-relative and user-owned.
- One run is a sample, not a signal — use `-n 3` (or more) for anything you'll act on.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [pinecone-io](https://github.com/pinecone-io)
- **Source:** [pinecone-io/cultivar](https://github.com/pinecone-io/cultivar)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-pinecone-io-cultivar-cultivar
- Seller: https://agentstack.voostack.com/s/pinecone-io
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.