Scoring Checks

Adding a Scoring Check

Add a new deterministic check that evaluates a single aspect of AI agent config quality. All checks must be filesystem-based with no network calls or LLM inference.

Critical

• Check must be deterministic: Same filesystem state → same result every time. No randomness, no external APIs.
• Point values come from constants.ts: Every earnedPoints and maxPoints must reference POINTS_* from src/scoring/constants.ts. Do NOT hardcode numbers.
• Always return Check[] array: Export a function check(dir: string): Check[] where category is one of: existence, quality, grounding, accuracy, freshness, bonus.
• Every check must have: id (kebab-case, unique), name, category, maxPoints, earnedPoints, passed, detail, and optional suggestion/fix.
• Fix object fields: action (string describing what to do), data (context for the fix), instruction (user-facing guidance).
• Register in src/scoring/index.ts: Add the import and spread the result into the allChecks array in computeLocalScore().
• Target filtering: If the check is platform-specific (Claude-only, Cursor-only, etc.), add its ID to the appropriate *_ONLY_CHECKS set in constants.ts.

Instructions

Step 1: Define point constants in src/scoring/constants.ts

Verify before proceeding: Is your check measurable with a numeric point value?

Add constants below the appropriate category section (existence, quality, grounding, accuracy, freshness, bonus):

// In the appropriate CATEGORY section, e.g., Quality checks (25 pts):
export const POINTS_YOUR_CHECK_NAME = 4; // 1-12 pts typical

// If threshold-based, add a companion array:
export const YOUR_THRESHOLD_ARRAY = [
  { minValue: 10, points: 4 },
  { minValue: 5, points: 2 },
] as const;

Check existing patterns: Token budgets use TOKEN_BUDGET_THRESHOLDS, code blocks use CODE_BLOCK_THRESHOLDS, concreteness uses CONCRETENESS_THRESHOLDS.

Verify: Review CATEGORY_MAX object to ensure your check fits within its category's point budget.

Step 2: Create or edit check function in src/scoring/checks/

Choose the file based on category. Each file exports a check(dir: string): Check[] function:

• existence.ts — files/directories exist (CLAUDE.md, .cursorrules, skills, MCP servers)
• quality.ts — config structure, size, clarity (code blocks, token budget, concreteness, duplicates)
• grounding.ts — references to actual project files/directory structure
• accuracy.ts — validity of references, git-based config drift
• freshness.ts — git commit-based staleness, secrets, permissions
• bonus.ts — hooks, learned content, OpenSkills format
• sources.ts — source configuration and usage

Create the function following this structure:

import type { Check } from '../index.js';
import {
  POINTS_YOUR_CHECK,
  YOUR_THRESHOLD_ARRAY,
} from '../constants.js';
import { readFileOrNull } from '../utils.js'; // or other helpers

export function checkYourCategory(dir: string): Check[] {
  const checks: Check[] = [];

  // 1. Measure something concrete
  const yourMetric = /* e.g., countFiles(), validatePaths(), etc. */;
  const threshold = YOUR_THRESHOLD_ARRAY.find(t => yourMetric >= t.minValue);
  const earnedPts = threshold?.points ?? 0;

  checks.push({
    id: 'your_unique_check_id',
    name: 'Human-readable check name',
    category: 'quality', // matches function context
    maxPoints: POINTS_YOUR_CHECK,
    earnedPoints: earnedPts,
    passed: earnedPts >= Math.ceil(POINTS_YOUR_CHECK * 0.6), // or custom logic
    detail: `${earnedPts}/${POINTS_YOUR_CHECK} points — ${yourMetric} items found`,
    suggestion: earnedPts >= POINTS_YOUR_CHECK ? undefined : 'Action to improve',
    fix: earnedPts >= POINTS_YOUR_CHECK ? undefined : {
      action: 'verb_noun', // e.g., 'add_code_blocks', 'fix_references'
      data: { currentValue: yourMetric, targetValue: 10 },
      instruction: 'Specific, actionable guidance for the user.',
    },
  });

  return checks;
}

Verify ID uniqueness: Run grep -r "'your_unique_check_id'" src/scoring/checks/ — should return only your new check.

Step 3: Handle platform-specific filtering (if applicable)

If your check only applies to certain agents (Claude, Cursor, Codex, GitHub Copilot), register it in src/scoring/constants.ts:

// Add to the appropriate set:
export const CLAUDE_ONLY_CHECKS = new Set([
  'claude_md_exists',
  'your_new_check_id', // ← add here
  'claude_rules_exist',
]);

Available sets (update exactly one if applicable):

• CLAUDE_ONLY_CHECKS — Claude Code targets
• CURSOR_ONLY_CHECKS — Cursor targets
• CODEX_ONLY_CHECKS — Codex/OpenCode targets
• COPILOT_ONLY_CHECKS — GitHub Copilot targets
• BOTH_ONLY_CHECKS — Both Claude AND Cursor (cross-platform parity)
• NON_CODEX_CHECKS — Everything except Codex/OpenCode
• CLAUDE_OR_CODEX_CHECKS — Claude OR Codex

Verify filtering: Examine filterChecksForTarget() in src/scoring/index.ts to ensure your category will work correctly for your target agents.

Step 4: Register in src/scoring/index.ts

Import your function at the top:

import { checkYourCategory } from './checks/your-file.js';

Add to computeLocalScore() inside the allChecks array initialization:

export function computeLocalScore(dir: string, targetAgent?: TargetAgent): ScoreResult {
  const target = targetAgent ?? detectTargetAgent(dir);

  const allChecks: Check[] = [
    ...checkExistence(dir),
    ...checkQuality(dir),
    ...checkGrounding(dir),
    ...checkAccuracy(dir),
    ...checkYourCategory(dir), // ← ADD HERE IN ORDER
    ...checkFreshness(dir),
    ...checkBonus(dir),
    ...checkSources(dir),
  ];
  // ... rest of function
}

Verify registration: Run npm test src/scoring/__tests__/accuracy.test.ts (or similar) — all existing tests should still pass.

Step 5: Write deterministic unit tests

Create or edit src/scoring/checks/__tests__/your-file.test.ts:

import { describe, it, expect } from 'vitest';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { checkYourCategory } from '../your-file.js';
import { POINTS_YOUR_CHECK } from '../../constants.js';

describe('checkYourCategory', () => {
  it('awards full points when condition passes', () => {
    const dir = mkdtempSync('test-scoring-');
    try {
      // Set up the passing condition
      writeFileSync(join(dir, 'SOME_FILE.md'), 'content that satisfies check');
      
      const checks = checkYourCategory(dir);
      const check = checks.find(c => c.id === 'your_unique_check_id');
      
      expect(check).toBeDefined();
      expect(check?.passed).toBe(true);
      expect(check?.earnedPoints).toBe(POINTS_YOUR_CHECK);
    } finally {
      rmSync(dir, { recursive: true });
    }
  });

  it('awards zero points when condition fails', () => {
    const dir = mkdtempSync('test-scoring-');
    try {
      // Don't create the required condition
      const checks = checkYourCategory(dir);
      const check = checks.find(c => c.id === 'your_unique_check_id');
      
      expect(check?.passed).toBe(false);
      expect(check?.earnedPoints).toBe(0);
    } finally {
      rmSync(dir, { recursive: true });
    }
  });

  it('returns correct detail message', () => {
    const dir = mkdtempSync('test-scoring-');
    try {
      const checks = checkYourCategory(dir);
      const check = checks.find(c => c.id === 'your_unique_check_id');
      expect(check?.detail).toBeTruthy();
    } finally {
      rmSync(dir, { recursive: true });
    }
  });
});

Run tests: npm test src/scoring/checks/__tests__/your-file.test.ts. All must pass before shipping.

Examples

Example 1: Existence Check

Trigger: User says "Add a check to verify .claude/rules/ directory exists."

Actions:

1. Add export const POINTS_CLAUDE_RULES = 3; to constants.ts
2. In existence.ts: existsSync(join(dir, '.claude', 'rules')) → true/false
3. Import in index.ts and add ...checkExistence(dir) (already done)
4. Test with mkdtempSync; verify earnedPoints matches POINTSCLAUDERULES

Result: Check id: 'claude_rules_exist' returns earnedPoints: 3, passed: true when dir exists.

Example 2: Quality Check with Thresholds

Trigger: User says "Verify config has at least 3 code blocks with executable commands."

Actions:

1. Add to constants.ts:

``typescript export const CODE_BLOCK_THRESHOLDS = [ { minBlocks: 3, points: 8 }, { minBlocks: 2, points: 6 }, { minBlocks: 1, points: 3 }, ] as const; ``

1. In quality.ts:

• Parse CLAUDE.md with regex to count \\\` blocks
• Match against CODEBLOCKTHRESHOLDS
• Return points based on threshold match

1. In fix: Suggest which commands to add

Result: 3+ blocks = 8 pts, 2 blocks = 6 pts, 1 block = 3 pts, 0 blocks = 0 pts.

Example 3: Accuracy Check (Reference Validation)

Trigger: User says "Check that all file paths mentioned in config actually exist."

Actions:

1. Add export const POINTS_REFERENCES_VALID = 8; to constants.ts
2. In accuracy.ts:

• Extract backtick-quoted paths and dir patterns from config
• Check existence with existsSync(join(dir, path))
• Calculate ratio: valid / total
• Award partial points: Math.round(ratio * POINTS_REFERENCES_VALID)

1. In fix: List invalid paths the user should fix

Result: 80% valid refs = ~6 pts; 100% valid = 8 pts; 0% valid = 0 pts.

Common Issues

Issue: "My check doesn't appear in the score report." Fix: 1) Verify ID in *_ONLY_CHECKS if platform-specific. 2) Verify import and spread in index.ts allChecks array. 3) Run npm test to ensure no tsc errors. 4) Check detectTargetAgent() returns your target platform.

Issue: "Points are hardcoded but should use constants." Fix: Replace all literal numbers like earnedPoints: 5 with earnedPoints: POINTS_YOUR_CHECK. Constants are in src/scoring/constants.ts — use them consistently.

Issue: "Check makes an API call or network request." Fix: Scoring MUST be deterministic and offline. Use only: fs module (readFileSync, existsSync, readdirSync), path, execSync for git commands. No HTTP, no LLM calls, no external services.

Issue: "Platform-specific check appears for the wrong agent." Fix: 1) Verify check ID is in correct *_ONLY_CHECKS set. 2) Double-check filterChecksForTarget() handles your platform set. 3) Test with detectTargetAgent() on a real project.

Issue: "Test fails with 'Module not found' error." Fix: Ensure file is in src/scoring/checks/ (not nested). Use .js extension in imports (TypeScript transpiles to ES modules). Run npm run build to check for tsc errors.

Issue: "Detail message is confusing or too technical." Fix: Use friendly language: 3 code blocks found (need 3 for full points) instead of codeBlockCount=3. Make it clear WHY they got/lost points.

Issue: "Threshold-based check gives wrong points for edge cases." Fix: Test all boundaries: value=0, value=threshold, value>>threshold. Use .find() to match highest-to-lowest: find(t => value >= t.minValue).

Issue: "Two checks have the same ID." Fix: Run grep -r "'my_id'" src/scoring/checks/ to find duplicates. IDs must be globally unique across all check files. Use descriptive names like claude_md_exists, not check_1.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: caliber-ai-org
• Source: caliber-ai-org/ai-setup
• License: MIT
• Homepage: https://caliber-ai.dev

Install and usage instructions live in the source repository linked above.