Skillgrade Graders

Skillgrade Grader Authoring

Procedures

Step 1: Identify the Grading Strategy

1. Determine whether the task requires objective verification (deterministic) or qualitative assessment (LLM rubric).
2. For most tasks, combine both: deterministic graders verify outcomes (weight 0.7), LLM rubrics assess approach quality (weight 0.3).

Step 2: Write a Deterministic Grader

1. Create a script in the skill's graders/ directory (bash or TypeScript).
2. The script must output a JSON object to stdout with the following structure:

``json {"score": 0.67, "details": "2/3 checks passed", "checks": [{"name": "check-name", "passed": true, "message": "Description"}]} ``

1. score (0.0–1.0) and details are required. checks is optional but recommended.
2. Read references/grader-output-schema.md for the full output specification.
3. Use awk for arithmetic in bash scripts — bc is not available in node:20-slim.
4. Reference the grader in eval.yaml:

```yaml

• type: deterministic

run: bash graders/check.sh weight: 0.7 ```

Step 3: Write an LLM Rubric Grader

1. Draft a rubric with explicit scoring criteria and point allocations.
2. Structure the rubric into weighted sections that sum to 1.0:

``` Workflow Compliance (0-0.5):

• Did the agent follow the mandatory workflow steps?

Efficiency (0-0.5):

• Completed in ≤5 commands without trial-and-error?

```

1. Reference the rubric in eval.yaml:

```yaml

• type: llm_rubric

rubric: | [rubric text or file path] weight: 0.3 provider: gemini # optional: gemini (default) | anthropic | openai model: gemini-3-flash-preview # optional, each provider has a default model ```

1. For long rubrics, store in a separate file and reference by path: rubric: rubrics/quality.md.

Step 4: Combine Multiple Graders

1. Assign weights to each grader based on importance. Weights are normalized automatically.
2. Final reward is calculated as: Σ (grader_score × weight) / Σ weight.
3. Example configuration:

```yaml graders:

• type: deterministic

run: bash graders/check.sh weight: 0.7

• type: llm_rubric

rubric: rubrics/quality.md weight: 0.3 ```

Step 5: Validate Graders

1. Create a reference solution script that produces the expected output.
2. Run skillgrade --validate to verify graders score the reference solution correctly.
3. Test only deterministic graders: skillgrade --grader=deterministic (skips LLM calls, faster iteration).
4. Test only LLM rubric graders: skillgrade --grader=llm_rubric.
5. Run a specific eval with a specific grader type: skillgrade --eval=my-eval --grader=deterministic.
6. If a grader returns unexpected scores, inspect the script output and adjust scoring logic.

Error Handling

• If a deterministic grader outputs non-JSON, ensure all echo/console.log statements except the final JSON result are redirected to stderr.
• If an LLM rubric grader returns 0.00 with a missing API key message, set the appropriate key for your provider: GEMINI_API_KEY (provider: gemini), ANTHROPIC_API_KEY (provider: anthropic), or OPENAI_API_KEY (provider: openai).
• To use a custom/self-hosted LLM endpoint, set ANTHROPIC_BASE_URL (for provider: anthropic) or OPENAI_BASE_URL (for provider: openai) — e.g. for Ollama or vLLM.
• If scores are inconsistent across trials, reduce rubric ambiguity by adding concrete examples of passing and failing behavior.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mgechev
• Source: mgechev/skillgrade
• License: MIT
• Homepage: https://blog.mgechev.com/2026/03/14/skillgrade/

Install and usage instructions live in the source repository linked above.