# Exp Lens Comparator Construction

> Create Comparator Construction experimental design analysis assessing whether baselines and controls are fair and relevant. Counterfactual lens answering "Is the comparator fair and relevant?

- **Type:** Skill
- **Install:** `agentstack add skill-trecek-useful-claude-skills-exp-lens-comparator-construction`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Trecek](https://agentstack.voostack.com/s/trecek)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Trecek](https://github.com/Trecek)
- **Source:** https://github.com/Trecek/useful-claude-skills/tree/main/.claude/skills/exp-lens-comparator-construction

## Install

```sh
agentstack add skill-trecek-useful-claude-skills-exp-lens-comparator-construction
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Comparator Construction Experimental Design Lens

**Philosophical Mode:** Counterfactual
**Primary Question:** "Is the comparator fair and relevant?"
**Focus:** Baseline Choice, Control Realism, Version Matching, Effort Symmetry, Baseline Drift

## When to Use

- Benchmark comparisons where baseline quality is questioned
- Ablation studies needing fair controls
- Claims of improvement over prior work
- User invokes `/exp-lens-comparator-construction` or `/make-experiment-diag comparator`

## Critical Constraints

**NEVER:**
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Accept at face value that baselines received symmetric treatment

**ALWAYS:**
- Build a fairness matrix covering all treatment-vs-comparator pairs
- Check for confounding differences in implementation, tuning, data access, and compute
- Assess whether each comparator is the best available alternative at the time of the experiment
- Identify temporal drift in baseline relevance
- BEFORE creating any diagram, LOAD the `/mermaid` skill using the Skill tool - this is MANDATORY

---

## Analysis Workflow

### Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

**Baseline/Control Definitions**
- Find what the proposed method is compared against
- Look for: baseline, control, comparison, prior, state-of-the-art, vanilla, default, reference

**Implementation Parity**
- Find whether baselines get equal engineering effort
- Look for: reproduce, reimplement, original, paper, author, tuned, optimized, hyperparameter

**Version & Environment Match**
- Find whether baselines use the same software/hardware environment
- Look for: version, library, framework, gpu, hardware, environment, checkpoint

**Tuning Protocol Symmetry**
- Find whether hyperparameter tuning is symmetric
- Look for: tune, search, grid, optuna, sweep, budget, trials, epochs

**Temporal Baseline Drift**
- Find whether baselines have been updated or are stale
- Look for: date, published, year, updated, latest, deprecated, legacy

### Step 2: Build the Comparator Inventory

For each comparator, assess:
1. Is it the best available alternative?
2. Is it given equal engineering effort?
3. Is it run in the same environment?
4. Is the tuning budget symmetric?
5. Has it drifted since originally published?

### Step 3: Construct the Fairness Matrix

**CRITICAL — Analyze Counterfactual Quality:**
For each treatment-vs-comparator pair:
- Does the comparison isolate the intended factor?
- Are there confounding differences in implementation, tuning, data access, or compute?

Build a fairness matrix with rows = comparators, columns = fairness dimensions.

### Step 4: Create the Optional Comparison Diagram

If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.

**Direction:** `LR` (treatment and comparator flow in parallel toward evaluation)

**Subgraphs:** "PROPOSED METHOD", "COMPARATOR(S)", "SHARED EVALUATION"

**Node Styling:**
- `cli` class: proposed method nodes
- `phase` class: comparator method nodes
- `handler` class: shared evaluation pipeline nodes
- `output` class: results nodes
- `gap` class: asymmetries flagged
- `detector` class: parity checks

### Step 5: Write Output

Write the analysis to: `temp/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md`

---

## Output Template

```markdown
# Comparator Construction Analysis: {Experiment Name}

**Lens:** Comparator Construction (Counterfactual)
**Question:** Is the comparator fair and relevant?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Comparator Inventory

| Comparator | Source | Reimplemented? | Same Environment? | Same Tuning Budget? |
|------------|--------|---------------|-------------------|---------------------|
| {name} | {paper/repo} | Yes / No / Partial | Yes / No | Yes / No / Unknown |

## Fairness Matrix

| Comparator | Best Available? | Equal Effort? | Same Env? | Symmetric Tuning? | Temporally Current? |
|------------|----------------|--------------|-----------|-------------------|---------------------|
| {name} | Yes / No | Yes / No | Yes / No | Yes / No | Yes / No |

## Comparison Diagram (Optional)

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Proposed ["PROPOSED METHOD"]
        METHOD["Proposed Method━━━━━━━━━━{method name}"]
    end

    subgraph Comparators ["COMPARATOR(S)"]
        COMP1["Comparator 1━━━━━━━━━━{name}"]
        COMP2["Comparator 2━━━━━━━━━━{name}"]
    end

    subgraph Evaluation ["SHARED EVALUATION"]
        EVAL["Evaluation Pipeline━━━━━━━━━━{dataset/benchmark}"]
        RESULTS["Results━━━━━━━━━━{metrics reported}"]
        PARITY["Parity Check━━━━━━━━━━{asymmetry found}"]
        ASYM["Asymmetry━━━━━━━━━━{description}"]
    end

    METHOD -->|"evaluated on"| EVAL
    COMP1 -->|"evaluated on"| EVAL
    COMP2 -->|"evaluated on"| EVAL
    EVAL --> RESULTS
    RESULTS --> PARITY
    PARITY -.->|"flagged"| ASYM

    class METHOD cli;
    class COMP1,COMP2 phase;
    class EVAL handler;
    class RESULTS output;
    class PARITY detector;
    class ASYM gap;
```

**Color Legend:**
| Color | Category | Description |
|-------|----------|-------------|
| Dark Blue | Proposed Method | The method being evaluated |
| Purple | Comparators | Baselines and controls |
| Orange | Evaluation | Shared evaluation pipeline |
| Dark Teal | Results | Reported outcomes |
| Red | Parity Checks | Fairness verification points |
| Yellow | Asymmetries | Flagged unfair differences |

## Asymmetry Register

| # | Asymmetry | Affects | Impact Assessment | Remediation |
|---|-----------|---------|-------------------|-------------|
| 1 | {description} | {comparator(s)} | High / Medium / Low | {how to fix} |

## Recommendations

1. {Most critical fairness fix — e.g., retune baseline with same budget}
2. {Version alignment or environment standardization needed}
3. {Additional comparator that should be included}
```

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-estimand-clarity` - For clarifying what the comparison is measuring
- `/exp-lens-fair-comparison` - For deeper analysis of evaluation protocol fairness

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Trecek](https://github.com/Trecek)
- **Source:** [Trecek/useful-claude-skills](https://github.com/Trecek/useful-claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-trecek-useful-claude-skills-exp-lens-comparator-construction
- Seller: https://agentstack.voostack.com/s/trecek
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.