# Harbor Adapter Creator

> Create Harbor benchmark adapters that convert external benchmark datasets into Harbor task format. Use when porting an existing benchmark to Harbor, running parity experiments, registering a dataset to the Harbor registry, or debugging adapter validation failures. Covers: adapter class interface (generate_task, make_local_task_id), directory layout including YAML job configs, oracle verification,…

- **Type:** Skill
- **Install:** `agentstack add skill-harbor-framework-skills-harbor-adapter-creator`
- **Verified:** Pending review
- **Seller:** [harbor-framework](https://agentstack.voostack.com/s/harbor-framework)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [harbor-framework](https://github.com/harbor-framework)
- **Source:** https://github.com/harbor-framework/skills/tree/main/skills/harbor-adapter-creator

## Install

```sh
agentstack add skill-harbor-framework-skills-harbor-adapter-creator
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Creating Harbor Benchmark Adapters

Adapters convert external benchmarks (SimpleQA, GAIA, AiderPolyglot, CodePDE, spider2-dbt, etc.) into Harbor's standardized task directory format. Each adapter reads source benchmark data and generates many individual task directories, one per benchmark instance.

## When to Use an Adapter vs. Creating Tasks Directly

**Use an adapter when:**
- You have an existing benchmark dataset with many instances
- Tasks share the same structure but differ in data (questions, code, etc.)
- You want to track evaluation parity with the original benchmark

**Create tasks directly when:**
- You're authoring original evaluation challenges
- Each task has unique structure and environment
- There are fewer than ~10 tasks

## Adapter Directory Structure

```
adapters//
├── adapter.py              # Core conversion logic (adapter class)
├── run_adapter.py          # CLI entry point
├── run_.yaml   # Job config for oracle and parity experiment runs
├── README.md               # Benchmark docs, license, parity, citation
├── parity_experiment.json  # Parity tracking results (JSON array)
├── adapter_metadata.json   # Adapter metadata (JSON array)
└── template/               # Task template files
    ├── task.toml
    ├── instruction.md
    ├── environment/
    │   └── Dockerfile
    ├── tests/
    │   └── test.sh
    └── solution/
        └── solve.sh
```

All template files, parity_experiment.json, adapter_metadata.json, and README.md are required. The validator (`harbor adapters validate`) checks for all of them. The YAML job config is not validated but is expected for parity experiments.

## Scaffolding with harbor adapters init

```bash
harbor adapters init my-benchmark
```

The interactive `AdapterWizard` prompts for:
- **Benchmark name** -- Vanilla name (e.g., "SWE-bench")
- **Adapter ID** -- Lowercase, no spaces (e.g., `swebench`)
- **Class name** -- Python class name (auto-derived or overridden, e.g., `SwebenchAdapter`)
- **Description** -- One-line description for the README
- **Source URL** -- Link to original benchmark paper or repo
- **License** -- Dataset license for the README

If all fields are provided as CLI arguments, the wizard runs non-interactively. The wizard renders Jinja2 templates from `src/harbor/cli/template-adapter/` and places the result under `adapters//`.

After scaffolding, validate:

```bash
harbor adapters validate adapters/my-benchmark
```

## Writing adapter.py

The adapter class has no formal base class, but every adapter must implement the same interface: a `NAME` attribute, `make_local_task_id` static method, and `generate_task` method.

### Core Class Interface

```python
import shutil
from pathlib import Path
from loguru import logger

class MyBenchmarkAdapter:
    """Converts MyBenchmark instances into Harbor task directories."""

    NAME = "mybenchmark"

    @staticmethod
    def make_local_task_id(source_id: str) -> str:
        """Convert a source benchmark ID to a Harbor task directory name.

        Convention: lowercase, hyphens as separators, prefixed with
        the adapter ID to prevent collisions across benchmarks.
        """
        normalized = source_id.lower().replace("_", "-")
        return f"mybenchmark-{normalized}"

    def __init__(self, task_dir: Path, **kwargs: object):
        """Initialize the adapter.

        Args:
            task_dir: Root directory where generated tasks will be written.
                      Each task gets a subdirectory: task_dir / local_task_id
            **kwargs: Adapter-specific configuration (data paths, splits, etc.)
        """
        self.task_dir = Path(task_dir)
        self._config = kwargs

        # Locate template files relative to this file
        self.template_dir = Path(__file__).parent / "template"
        if not self.template_dir.exists():
            raise FileNotFoundError(f"Template directory not found: {self.template_dir}")

        # Load your benchmark data here.
        # Examples from real adapters:
        #   SimpleQA: pandas.read_csv() from a URL
        #   GAIA: HuggingFace datasets or local JSONL
        #   AiderPolyglot: walk language dirs, parse .meta/config.json
        #   CodePDE: clone repo, load PDE descriptions
        logger.info(f"Initializing MyBenchmarkAdapter -> {self.task_dir}")

    def _copy_template(self, output_dir: Path) -> None:
        """Copy baseline template files into the output directory."""
        shutil.copytree(self.template_dir, output_dir, dirs_exist_ok=True)

    def generate_task(self, source_id: str, local_task_id: str) -> None:
        """Generate a single Harbor task directory.

        This is the core method. It must create a complete, valid task directory
        with all required files: task.toml, instruction.md, environment/Dockerfile,
        tests/test.sh, and optionally solution/solve.sh.

        Args:
            source_id: Identifier from the original benchmark
            local_task_id: Harbor task directory name (from make_local_task_id)
        """
        output_dir = self.task_dir / local_task_id
        output_dir.mkdir(parents=True, exist_ok=True)

        # 1. Copy baseline template
        self._copy_template(output_dir)

        # 2. Look up the source instance data
        # instance = self.data[source_id]

        # 3. Write task-specific files:
        #    - instruction.md  (the prompt the agent sees)
        #    - task.toml       (metadata, timeouts, resource limits)
        #    - environment/Dockerfile
        #    - tests/test.sh   (MUST write to /logs/verifier/reward.txt)
        #    - solution/solve.sh (reference solution for OracleAgent)
        #    - Copy attachments into environment/ if needed

        logger.info(f"Generated task {local_task_id} in {output_dir.resolve()}")
```

### The generate_task Method in Detail

The `generate_task` method receives `source_id` (the original benchmark's identifier) and `local_task_id` (the Harbor directory name produced by `make_local_task_id`). It builds the output path from `self.task_dir / local_task_id`.

Typical steps inside `generate_task`:

1. **Create directory and copy template** -- `shutil.copytree` with `dirs_exist_ok=True`
2. **Load instance data** -- look up the specific benchmark record
3. **Write instruction.md** -- replace placeholders or write dynamically
4. **Write task.toml** -- set metadata, tags, timeouts
5. **Write or modify Dockerfile** -- add `COPY` commands for attachments
6. **Write tests/test.sh** -- the verification script (see test.sh requirements below)
7. **Write solution/solve.sh** -- reference solution for oracle agent
8. **Copy attachments** -- instance-specific files into `environment/workspace/` or `environment/`

### Data Loading Patterns (from real adapters)

| Adapter | Data Source | Loading Pattern |
|---------|-----------|----------------|
| SimpleQA | OpenAI public CSV | `pandas.read_csv(url)` in `__init__` |
| GAIA | HuggingFace or local JSONL | `datasets` library or line-by-line JSON |
| AiderPolyglot | Exercism repo | Walk language dirs, parse `.meta/config.json` |
| CodePDE | CodePDE repo | Clone repo, load PDE description files |
| spider2-dbt | Spider2-DBT repo | Clone repo, read dbt project structures |

### Template Filling Patterns

- **Simple string replacement**: `content.replace("{KEY}", value)` -- used by GAIA
- **Placeholder replacement**: `content.replace("{{PLACEHOLDER}}", value)` -- common pattern
- **Direct file writes**: Build content programmatically and write -- used by AiderPolyglot
- **Jinja2**: For complex templates with conditionals (not used by existing adapters but supported)

### Handling Attachments

When source benchmarks include per-instance files (images, CSVs, databases):

1. Copy files into `environment/workspace/` in the task directory
2. Add `COPY workspace/ /app/files/` to the Dockerfile (GAIA pattern)
3. Reference files from `instruction.md` using the container path (e.g., `/app/files/data.csv`)
4. Mention the attachment in the instruction so the agent knows about it

## Writing run_adapter.py

The CLI entry point follows a standard pattern with helper functions for argument parsing, ID collection, repo cloning, and benchmark processing. The default output directory is `datasets/`.

```python
#!/usr/bin/env python3
import argparse
import sys
from pathlib import Path
from loguru import logger

sys.path.insert(0, str(Path(__file__).parent.parent))
from mybenchmark.adapter import MyBenchmarkAdapter

def _parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Generate Harbor tasks from MyBenchmark",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    parser.add_argument(
        "--output-dir", type=Path,
        default=Path("datasets/mybenchmark"),
        help="Directory to write generated Harbor task directories",
    )
    parser.add_argument(
        "--task-ids", nargs="*",
        help="Specific source task IDs to generate (space-separated)",
    )
    parser.add_argument(
        "--ids-file", type=Path,
        help="Path to a file containing source task IDs, one per line",
    )
    parser.add_argument(
        "--limit", type=int,
        help="Limit the number of tasks to generate",
    )
    return parser.parse_args()

def _collect_ids(task_ids: list[str] | None, ids_file: Path | None) -> list[str]:
    """Collect task IDs from command line args or a file."""
    if task_ids:
        return task_ids
    if ids_file:
        return [line.strip() for line in ids_file.read_text().splitlines() if line.strip()]
    return []

def _process_benchmark(output_dir: Path, task_ids: list[str]) -> None:
    """Initialize adapter and generate all tasks."""
    output_dir.mkdir(parents=True, exist_ok=True)
    adapter = MyBenchmarkAdapter(task_dir=output_dir)

    if not task_ids:
        # Load all available IDs from the benchmark data
        task_ids = adapter.get_all_source_ids()

    errors = []
    for i, source_id in enumerate(task_ids, 1):
        local_task_id = MyBenchmarkAdapter.make_local_task_id(source_id)
        try:
            adapter.generate_task(source_id, local_task_id)
            logger.info(f"[{i}/{len(task_ids)}] {source_id} -> {local_task_id}")
        except Exception as e:
            errors.append((source_id, str(e)))
            logger.error(f"[{i}/{len(task_ids)}] {source_id} FAILED: {e}")

    logger.info(f"Generated {len(task_ids) - len(errors)}/{len(task_ids)} tasks")
    if errors:
        sys.exit(1)

def main() -> None:
    args = _parse_args()
    task_ids = _collect_ids(args.task_ids, args.ids_file)
    if args.limit and task_ids:
        task_ids = task_ids[:args.limit]
    _process_benchmark(args.output_dir, task_ids)

if __name__ == "__main__":
    main()
```

## test.sh Requirements

This is the most common source of adapter validation failures. The verifier uploads the entire `tests/` directory to `/tests` inside the container, makes `test.sh` executable, and runs it.

**test.sh MUST write a reward value to `/logs/verifier/reward.txt`** -- a single float (e.g., `0` or `1` or `0.75`). If this file is missing, the verifier raises `RewardFileNotFoundError` and the trial fails.

Optionally, test.sh can also write `/logs/verifier/reward.json` for structured reward data (e.g., `{"reward": 1.0, "quality": 0.8}`). The verifier checks `reward.txt` first, then falls back to `reward.json`.

### Verification Patterns

**String matching** (GAIA pattern):
```bash
#!/bin/bash
mkdir -p /logs/verifier
EXPECTED=$(cat /tests/expected_answer.txt)
if [ ! -f /app/answer.txt ]; then
    echo 0 > /logs/verifier/reward.txt
    exit 0
fi
ACTUAL=$(cat /app/answer.txt | tr '[:upper:]' '[:lower:]' | xargs)
EXPECTED_NORM=$(echo "$EXPECTED" | tr '[:upper:]' '[:lower:]' | xargs)
if [ "$ACTUAL" = "$EXPECTED_NORM" ]; then
    echo 1 > /logs/verifier/reward.txt
else
    echo 0 > /logs/verifier/reward.txt
fi
```

**Pytest-based** (CodePDE, hello-world pattern):
```bash
#!/bin/bash
# Install test dependencies inside test.sh, not in the Dockerfile
apt-get update && apt-get install -y curl
curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh
source $HOME/.local/bin/env
uvx --with pytest==8.4.1 pytest /tests/test_solution.py -rA
if [ $? -eq 0 ]; then
    echo 1 > /logs/verifier/reward.txt
else
    echo 0 > /logs/verifier/reward.txt
fi
```

**LLM judge** (SimpleQA pattern):
```bash
#!/bin/bash
mkdir -p /logs/verifier
cd /tests
python3 llm_judge.py  # reads ground_truth.json, writes reward.txt
```

## task.toml Configuration

```toml
version = "1.0"
source = "mybenchmark/instance-42"

[metadata]
author_name = "Your Name"
difficulty = "medium"           # easy, medium, hard
category = "qa"
tags = ["factual", "reasoning"]

[agent]
timeout_sec = 600.0             # max wall-clock seconds for the agent

[verifier]
timeout_sec = 600.0             # max seconds for test.sh
# env = { API_KEY = "..." }    # env vars injected into test.sh

[environment]
build_timeout_sec = 600.0       # max seconds for docker build
cpus = 1                        # CPU cores (default: 1)
memory_mb = 2048                # RAM in MB (default: 2048)
storage_mb = 10240              # disk in MB (default: 10240)
gpus = 0                        # GPUs (default: 0)
allow_internet = true           # network access (default: true)

[solution]
# env = { SECRET = "..." }     # env vars for solve.sh (OracleAgent)
```

## parity_experiment.json

A JSON **array** of experiment objects tracking how Harbor results compare to the original benchmark. Required fields per entry: `adapter_name`, `agent`, `model`, `date`, `metrics`.

```json
[
  {
    "adapter_name": "mybenchmark",
    "agent": "codex@0.77.0",
    "model": "openai/gpt-4o-2025-03-01",
    "date": "2026-01-15",
    "notes": "50 tasks; averaged over 3 trials",
    "original_parity_repo": "https://github.com/example/mybenchmark/tree/harbor-parity",
    "adapter_pr": [
      "https://github.com/harbor-framework/harbor/pull/123"
    ],
    "dataset_pr": [
      "https://github.com/harbor-framework/harbor-datasets/pull/45"
    ],
    "parity_pr": [
      "https://huggingface.co/datasets/harborframework/parity-experiments/discussions/10"
    ],
    "metrics": [
      {
        "benchmark_name": "MyBenchmark",
        "metric": "Accuracy",
        "original": "0.72 +/- 0.03",
        "harbor": "0.71 +/- 0.02",
        "original_trials": [0.73, 0.69, 0.74],
        "harbor_trials": [0.72, 0.70, 0.71]
      }
    ]
  }
]
```

The `adapter_pr`, `dataset_pr`, and `parity_pr` fields in `parity_experiment.json` must be arrays of URLs when present. Each metric object needs `benchmark_name`, `metric`, and at least one of `original`, `tb_adapter`, or `harbor`.

The `harbor_adapter` entry must include `parity_matching_agents` (array of `"agent@version+model"` strings). In the harbor-datasets `registry.json`, use `"version": "parity"` to enable `harbor jobs start -d mybenchmark@parity`.

## adapter_metadata.json

A JSON **array** describing the adapter and its relationship to the original benchmark. Required fields per entry: `adapter_name`, `adapter_builders`, `original_benchmark`, `harbor_adapter`.

```json
[
  {
    "adapter_name": "mybenchmark",
    "adapter_builders": [
      "Your Name (you@example.com)"
    ],
    "original_benchmark": [
      {
        "split": "test",
        "size": 500,
        "harness": "agent",
        "supported_agents": ["openhands"],
        "adaptable": true,
        "notes": "Only test split has ground-truth answers."
      }
    ],
    "harbor_adapter": [
      {
        "split": "test",
        "adapted_benchmark_size": 500,
        "parity_benchmark_size": 50,
        "parity_sampling_rate": 0.1,
        "registry_benchmark_size": 500,
        "parity_costs": "150",
        "parity_matching_agents": ["codex@0.77.0+openai/gpt-4o-2025-03-01"],
        "notes": "Full benchmark adapted. Parity on 10% sample."
      }
    ]
  }
]
```

## Validation

The `harbor adapters

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [harbor-framework](https://github.com/harbor-framework)
- **Source:** [harbor-framework/skills](https://github.com/harbor-framework/skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: flagged — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-harbor-framework-skills-harbor-adapter-creator
- Seller: https://agentstack.voostack.com/s/harbor-framework
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.