# Exp Lens Validity Threats

> Create a validity threat matrix identifying alternative explanations and design mitigations. Adversarial lens answering "What alternative explanations survive?

- **Type:** Skill
- **Install:** `agentstack add skill-trecek-useful-claude-skills-exp-lens-validity-threats`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Trecek](https://agentstack.voostack.com/s/trecek)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Trecek](https://github.com/Trecek)
- **Source:** https://github.com/Trecek/useful-claude-skills/tree/main/.claude/skills/exp-lens-validity-threats

## Install

```sh
agentstack add skill-trecek-useful-claude-skills-exp-lens-validity-threats
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Validity Threats Experimental Design Lens

**Philosophical Mode:** Adversarial
**Primary Question:** "What alternative explanations survive?"
**Focus:** History Effects, Instrumentation Changes, Selection Effects, Co-interventions, Treatment Diffusion

## When to Use

- Quasi-experimental designs without full randomization
- System experiments with environmental changes
- Longitudinal comparisons
- User invokes `/exp-lens-validity-threats` or `/make-experiment-diag validity`

## Critical Constraints

**NEVER:**
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

**ALWAYS:**
- Apply Campbell & Stanley's full threat taxonomy — do not skip threats just because they seem unlikely
- For every observed difference, enumerate at least 3 alternative explanations
- Assess mitigation strength honestly — "partially mitigated" is better than false confidence
- Distinguish threats that are ruled out by design from those that remain plausible
- BEFORE creating any diagram, LOAD the `/mermaid` skill using the Skill tool - this is MANDATORY

---

## Analysis Workflow

### Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

**Temporal Changes (History)**
- Find whether the environment changed between conditions
- Look for: version, update, release, change, drift, before, after, time, date

**Instrumentation Changes**
- Find whether measurement tools or methods changed
- Look for: log, metric, instrument, measurement, format, schema, API, breaking

**Selection & Filtering**
- Find whether groups are compared after differential filtering
- Look for: filter, exclude, select, subsample, eligible, criteria, dropout, attrition

**Co-interventions**
- Find whether multiple changes occurred simultaneously
- Look for: also, simultaneously, together, combined, alongside, additionally, migration

**Treatment Diffusion**
- Find whether control units could access treatment
- Look for: access, share, copy, learn, adopt, contaminate, crossover, spillover

### Step 2: Apply Threat Taxonomy

Apply Campbell & Stanley's threat taxonomy to the specific experiment. For each threat category:
1. Is it plausible in this context?
2. What design feature mitigates it?
3. How strong is the mitigation?

Build the threat matrix.

### Step 3: Analyze Alternative Explanations

**CRITICAL — Analyze Alternative Explanations:**
For every observed difference between conditions:
- List at least 3 alternative explanations besides the intended causal claim
- For each: What evidence rules it out? What evidence remains consistent with it?

### Step 4: Create the Diagram

Use the mermaid skill conventions to create a threat-flow diagram with:

**Direction:** `TB` (threats flow from sources through mitigations to residual risk)

**Subgraphs:**
- THREAT SOURCES
- DESIGN MITIGATIONS
- RESIDUAL THREATS

**Node Styling:**
- `gap` class: Active unmitigated threats
- `detector` class: Design mitigations
- `stateNode` class: Threat sources
- `handler` class: Partially mitigated threats
- `output` class: Successfully mitigated threats

### Step 5: Write Output

Write the diagram to: `temp/exp-lens-validity-threats/exp_diag_validity_threats_{YYYY-MM-DD_HHMMSS}.md`

---

## Output Template

```markdown
# Validity Threat Analysis: {Experiment Name}

**Lens:** Validity Threats (Adversarial)
**Question:** What alternative explanations survive?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Threat Matrix

| Threat | Category | Plausibility | Mitigation | Residual Risk |
|--------|----------|-------------|------------|---------------|
| {threat} | {Campbell-Stanley category} | {Low/Medium/High} | {design feature} | {Low/Medium/High} |

## Alternative Explanations

| Observed Difference | Alternative Explanation | Ruling-Out Evidence | Consistent Evidence |
|--------------------|------------------------|---------------------|---------------------|
| {difference} | {explanation} | {evidence} | {evidence} |

## Threat-Flow Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Sources ["THREAT SOURCES"]
        direction TB
        HIST["History Effect━━━━━━━━━━Environmental changebetween conditions"]
        INST["Instrumentation━━━━━━━━━━Measurement toolchange"]
        SEL["Selection Effect━━━━━━━━━━Differential filteringof units"]
    end

    subgraph Mitigations ["DESIGN MITIGATIONS"]
        direction TB
        RAND["Randomization━━━━━━━━━━Controls selectionand history"]
        CTRL["Control Group━━━━━━━━━━Absorbs environmentalchanges"]
    end

    subgraph Residual ["RESIDUAL THREATS"]
        direction TB
        DIFF["Treatment Diffusion━━━━━━━━━━Control accessto treatment"]
        COINT["Co-interventions━━━━━━━━━━Simultaneouschanges"]
    end

    HIST --> CTRL
    INST --> CTRL
    SEL --> RAND
    RAND --> CTRL
    CTRL --> DIFF
    CTRL --> COINT

    %% CLASS ASSIGNMENTS %%
    class HIST,INST,SEL stateNode;
    class RAND,CTRL detector;
    class DIFF,COINT gap;
```

## Campbell-Stanley Checklist

| Threat | Applicable? | Assessment |
|--------|-------------|------------|
| History | {Yes/No/Partial} | {assessment} |
| Maturation | {Yes/No/Partial} | {assessment} |
| Testing | {Yes/No/Partial} | {assessment} |
| Instrumentation | {Yes/No/Partial} | {assessment} |
| Statistical Regression | {Yes/No/Partial} | {assessment} |
| Selection | {Yes/No/Partial} | {assessment} |
| Experimental Mortality | {Yes/No/Partial} | {assessment} |
| Selection-Maturation Interaction | {Yes/No/Partial} | {assessment} |
| Diffusion of Treatments | {Yes/No/Partial} | {assessment} |
| Co-interventions | {Yes/No/Partial} | {assessment} |

## Key Findings

- {Description of most critical surviving alternative explanations}
```

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-causal-assumptions` - For DAG-based causal structure analysis
- `/exp-lens-severity-testing` - For statistical severity and power analysis

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Trecek](https://github.com/Trecek)
- **Source:** [Trecek/useful-claude-skills](https://github.com/Trecek/useful-claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-trecek-useful-claude-skills-exp-lens-validity-threats
- Seller: https://agentstack.voostack.com/s/trecek
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.