Puzzletide Agent Evals

PuzzleTide Agent Evals

Generate reproducible, objectively gradable puzzle tasks for testing models and agents with the local PuzzleTide CLI.

Why puzzles: they are verifiable by construction. A sudoku answer either satisfies the rules and preserves the givens or it doesn't; a word search answer either spells the word along a straight line in the grid or it doesn't. Grading needs no LLM judge and no trusted answer key.

Prefer the local CLI. Check availability in this order:

ptide --version
puzzletide --version
npx puzzletide --version

If none of those work, ask the user before installing (npm install -g puzzletide).

Generate a task set

ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.json

The tuple (type, difficulty, n, seed) fully determines the task set, so it names a reproducible benchmark — same command, same tasks, on any machine.

Each task has id, instructions, and the puzzle payload:

• sudoku: puzzle (81 chars, . = empty). Expected answer: completed 81-char string.
• wordsearch: grid (array of row strings) and words. Expected answer: JSON array

of {word, startRow, startCol, endRow, endCol} (0-indexed).

Run the subject model

Send each task's instructions + payload to the model under test and collect answers as a JSON array of {id, answer}.

Grade

ptide eval check --tasks tasks.json --answers answers.json --json

Returns per-task pass/fail with reasons and a summary score. Grading is deterministic and local.

Links

• Source and docs: GitHub · npm · CLI reference
• From the makers of puzzletide.com — free word search, crossword, and sudoku puzzles

Safety

• Everything runs locally; no account, API key, or network access.
• Do not install packages without asking the user first.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Caravaca-Labs
• Source: Caravaca-Labs/puzzletide-cli
• License: MIT

Install and usage instructions live in the source repository linked above.