Repo To Skill

Repo-to-Skill Generator

Generate a complete, tested, and evaluated agent skill from any GitHub repository.

Command

/skill repo-to-skill 

Example:

/skill repo-to-skill https://github.com/vllm-project/vllm

Overview

Given a GitHub repo URL, this skill:

1. Clones and analyzes the repository
2. Extracts usage patterns from README, docs, examples, CLI help
3. Generates a working skill (SKILL.md + resources)
4. Tests the skill by running basic operations
5. Evaluates the skill with test prompts and assertions

Process

Step 1: Clone & Analyze

# Clone repo (shallow for speed)
git clone --depth 1  /tmp/repo-to-skill/

Analyze in this order (stop when you have enough):

1. README.md — primary source for installation, quickstart, features
2. docs/ or documentation/ — detailed usage guides
3. examples/ — concrete usage patterns (high value)
4. CLI help — if it's a CLI tool, check --help output
5. setup.py / pyproject.toml / package.json — dependencies and entry points
6. Source code — only if docs are insufficient; focus on public API

Extract:

• What it does (one sentence)
• Installation method (pip, npm, cargo, etc.)
• Core commands/API (the 5-10 most common operations)
• Configuration (env vars, config files, required setup)
• Input/Output formats (what goes in, what comes out)
• Common patterns (from examples/ or README)

Step 2: Classify the Tool

Determine the tool type to shape the skill structure:

| Type | Skill Focus | Example | |------|------------|---------| | CLI tool | Commands, flags, common workflows | vllm, ffmpeg, gh | | Python library | API patterns, code snippets | transformers, pandas | | Framework | Project setup, config, patterns | FastAPI, Next.js | | Service | API endpoints, auth, integration | Stripe, OpenAI |

Step 3: Generate Skill Structure

Create the skill in the user's preferred location (default: ~/.agents/skills//).

/
├── SKILL.md                    # Core instructions
├── references/
│   ├── api-reference.md        # Full API/CLI reference (if large)
│   ├── examples.md             # Curated usage examples
│   └── configuration.md        # Config options (if complex)
└── scripts/
    └── setup.sh                # Installation/setup script (if needed)

SKILL.md Template

---
name: 
description: 
---

# 

## Installation

## Quick Start

## Core Operations

## Common Patterns

## Troubleshooting

## References

- For full API details, see [references/api-reference.md](references/api-reference.md)
- For more examples, see [references/examples.md](references/examples.md)

Step 4: Test the Skill

Verify the skill works by actually using the tool:

1. Install test — run the installation command
2. Smoke test — run the quickstart example
3. Feature test — try 2-3 core operations from the skill

If tests fail, update the skill with corrections.

Document test results:

✅ Installation: pip install vllm → success
✅ Quick start: vllm serve model → server started
❌ Feature: offline batching → fixed: added --dtype auto flag

Step 5: Validate & Deliver

Before delivering:

• [ ] SKILL.md under 500 lines
• [ ] Frontmatter has name + description
• [ ] Description includes trigger phrases
• [ ] All referenced files exist
• [ ] Examples are tested and working
• [ ] No secrets or credentials in skill files

Present the skill to the user with a summary of what it covers.

Step 6: Evaluate the Skill

After generating and testing, run a structured evaluation. See [references/eval-schemas.md](references/eval-schemas.md) for full JSON schemas.

Create Test Prompts

Write 3-5 realistic prompts a user would send to an agent with this skill. Save to evals/evals.json:

{
  "skill_name": "vllm",
  "evals": [
    {
      "id": 1,
      "prompt": "Serve Llama-3-8B with vllm on port 8000",
      "expected_output": "Working vllm serve command with correct model and port",
      "expectations": [
        "Command includes 'vllm serve' or 'python -m vllm.entrypoints'",
        "Port 8000 is specified",
        "Model name is correct"
      ]
    }
  ]
}

Run Evals (with-skill vs baseline)

For each test prompt, spawn two runs in parallel:

1. With-skill run: Agent has the generated skill loaded
2. Baseline run: Same prompt, no skill

Save outputs to -workspace/iteration-/eval-/with_skill/ and without_skill/.

Grade Results

For each run, evaluate assertions and produce grading.json:

{
  "expectations": [
    {
      "text": "Command includes 'vllm serve'",
      "passed": true,
      "evidence": "Output contains: vllm serve meta-llama/Llama-3-8B"
    }
  ],
  "summary": {
    "passed": 3,
    "failed": 0,
    "total": 3,
    "pass_rate": 1.0
  }
}

For assertions that can be checked programmatically (file exists, command runs, output matches pattern), write and run a script instead of eyeballing.

Aggregate & Report

Produce a benchmark comparing with-skill vs baseline:

• pass_rate: mean ± stddev across runs
• time_seconds: execution time
• tokens: token usage
• delta: improvement from skill

Present results to user. If pass_rate < 0.7, iterate on the skill.

Guidelines

• Be concise — only include what the model doesn't already know
• Prefer examples over explanations — show, don't tell
• Test everything — never include untested commands
• Progressive disclosure — keep SKILL.md lean, put details in references/
• Version-aware — note the repo version/commit analyzed
• Installation-first — always verify the tool actually installs cleanly
• Evaluate — always run at least 3 test prompts before delivering

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: shuyhere
• Source: shuyhere/repo-to-skill
• License: MIT

Install and usage instructions live in the source repository linked above.