Skill Completeness V1

skillcompletenessv1

Purpose

• Grades a skill or verifier directory for required files, valid manifest structure, side-effect declarations, validation gates, paired-verifier resolution, fixtures, authoring hygiene, and derived lifecycle status.
• Use this before publishing or reviewing a skill-shaped artifact. Engineering verification only.
• Manifest I/O: inputs are target_skill; outputs are completeness_report.

Instructions

• Run scripts/grade.py on a target skills/ or verifiers/ directory.
• If a host agent exposes run_script, use run_script("scripts/grade.py", args=["skills/"]).
• Prefer the eval-engine command when you need a verifier evidence pack; use the direct Python command for quick local inspection.

Available Scripts

| Script | Purpose | Arguments | |---|---|---| | scripts/grade.py | Primary verifier entrypoint declared by skill_manifest.yaml. | TARGET_SKILL_OR_VERIFIER_DIR |

Prerequisites

• The target directory must contain SKILL.md and skill_manifest.yaml to receive a meaningful structural audit.
• Optional LLM review requires LLM_VERIFIER=1 and NV_INFER_TOKEN; leave it unset for deterministic local checks.
• Side effects: deterministic tiers are offline; optional LLM review contacts https://inference-api.nvidia.com.

Limitations

• Tier 3 LLM review is advisory and skipped by default.
• Tier 4 test-quality assessment is still deferred.
• This verifier checks skill shape and authoring hygiene; it does not prove the wrapped medical AI task succeeds.

Troubleshooting

| Error | Cause | Fix | |---|---|---| | not a directory | The argument is not a skill or verifier directory. | Pass skills/ or verifiers/. | | Missing manifest/frontmatter | The target is not skill-shaped or has malformed metadata. | Add the required files before rerunning. | | Advisory findings | The target is runnable but less agent-friendly. | Fix the listed SKILL.md, fixture, or manifest hygiene gap. |

Meta-verifier for spec quality.

python verifiers/skill_completeness_v1/scripts/grade.py skills/dicom-metadata-extract

python eval_engine/run.py verifiers/skill_completeness_v1 \
  --fixture skills/dicom-metadata-extract \
  --out runs/audit_dicom_metadata

Tier 1 checks required files, frontmatter, manifest fields, entrypoint paths, and output schemas. Tier 2 checks side effects, gate presence, paired verifier resolution, fixture presence, import declarations, nontrivial sanity checks, and SKILL.md hygiene. The report also derives a non-authoritative lifecycle status (draft, runnable, gated, verified, or published) from those checks plus paired-verifier, trusted-run, behavior-eval, benchmark-note, and curated-evidence signals.

Optional advisory LLM review is disabled by default. Set LLM_VERIFIER=1 and NV_INFER_TOKEN only when explicitly running that experiment.

This verifier is itself audited by the same eval_engine.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: NVIDIA-Medtech
• Source: NVIDIA-Medtech/medical-AI-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.