Puzzletide Agent Evals
Use this skill when the user wants verifiable reasoning tasks to benchmark or test an LLM or agent — reproducible puzzle task sets (sudoku, word search) with objective, by-construction grading. No answer key to trust: answers are verified against the rules and the grid.
Install
$ agentstack add skill-caravaca-labs-puzzletide-cli-puzzletide-agent-evals ✓ scanned · ✓ verified, works with Claude Code, Cursor, and more.
Security review
✓ PassedNo issues found. Passed automated security review. · v0.1.0 How review works →
- ✓ Prompt-injection patterns
- ✓ Secret / credential exfiltration
- ✓ Dangerous shell & filesystem operations
- ✓ Untrusted network calls
- ✓ Known-malicious package signatures
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ✓ Dynamic code execution No
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
Verified badge
Passed review? Show it. Paste this badge into your README, it links to the public security report.
Links to your public security report.[](https://agentstack.voostack.com/security/report/skill-caravaca-labs-puzzletide-cli-puzzletide-agent-evals) Reliability & compatibility
Declared compatibility
Compatibility is declared by the source manifest. End-to-end runtime verification is coming, see below.
We're building live execution health for every listing: tool-call success rate, median latency, uptime, and last-checked timestamps, measured, not self-reported. It isn't live yet, so we don't show numbers we can't stand behind.
How agent discovery & health will work →About
PuzzleTide Agent Evals
Generate reproducible, objectively gradable puzzle tasks for testing models and agents with the local PuzzleTide CLI.
Why puzzles: they are verifiable by construction. A sudoku answer either satisfies the rules and preserves the givens or it doesn't; a word search answer either spells the word along a straight line in the grid or it doesn't. Grading needs no LLM judge and no trusted answer key.
Prefer the local CLI. Check availability in this order:
ptide --version
puzzletide --version
npx puzzletide --versionIf none of those work, ask the user before installing (npm install -g puzzletide).
Generate a task set
ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.jsonThe tuple (type, difficulty, n, seed) fully determines the task set, so it names a reproducible benchmark — same command, same tasks, on any machine.
Each task has id, instructions, and the puzzle payload:
- sudoku:
puzzle(81 chars,.= empty). Expected answer: completed 81-char string. - wordsearch:
grid(array of row strings) andwords. Expected answer: JSON array
of {word, startRow, startCol, endRow, endCol} (0-indexed).
Run the subject model
Send each task's instructions + payload to the model under test and collect answers as a JSON array of {id, answer}.
Grade
ptide eval check --tasks tasks.json --answers answers.json --jsonReturns per-task pass/fail with reasons and a summary score. Grading is deterministic and local.
Links
- Source and docs: GitHub · npm · CLI reference
- From the makers of puzzletide.com — free word search, crossword, and sudoku puzzles
Safety
- Everything runs locally; no account, API key, or network access.
- Do not install packages without asking the user first.
Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- Author: Caravaca-Labs
- Source: Caravaca-Labs/puzzletide-cli
- License: MIT
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet, be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.