# Exp Lens Pipeline Integrity

> Create Pipeline Integrity experimental design diagram showing data splits, leakage points, preprocessing order, and label contamination. Integrity lens answering "Could data handling create optimistic bias?

- **Type:** Skill
- **Install:** `agentstack add skill-trecek-useful-claude-skills-exp-lens-pipeline-integrity`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Trecek](https://agentstack.voostack.com/s/trecek)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Trecek](https://github.com/Trecek)
- **Source:** https://github.com/Trecek/useful-claude-skills/tree/main/.claude/skills/exp-lens-pipeline-integrity

## Install

```sh
agentstack add skill-trecek-useful-claude-skills-exp-lens-pipeline-integrity
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Pipeline Integrity Experimental Design Lens

**Philosophical Mode:** Integrity
**Primary Question:** "Could data handling create optimistic bias?"
**Focus:** Data Splits, Leakage Points, Preprocessing Order, Label Contamination, Pipeline Invariants

## When to Use

- ML pipeline with train/test splits
- Preprocessing before or after splitting is ambiguous
- Feature engineering touching labels
- User invokes `/exp-lens-pipeline-integrity` or `/make-experiment-diag pipeline`

## Critical Constraints

**NEVER:**
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

**ALWAYS:**
- Classify every pipeline stage as pre-split or post-split
- Trace whether transforms are fitted on full data or train-only
- Flag all label-touching feature engineering steps
- Document pipeline invariants that guard against leakage
- BEFORE creating any diagram, LOAD the `/mermaid` skill using the Skill tool - this is MANDATORY

---

## Analysis Workflow

### Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

**Data Loading & Sources**
- Find data ingestion code, raw data paths
- Look for: load, read, fetch, dataset, csv, parquet, download

**Preprocessing & Transforms**
- Find normalization, encoding, imputation steps
- Look for: transform, normalize, scale, encode, impute, clean, preprocess

**Split Logic**
- Find train/test/validation split code
- Look for: split, train_test, fold, cross_val, stratify, group

**Feature Engineering**
- Find feature creation, selection, extraction
- Look for: feature, extract, select, engineer, embed, vectorize

**Model Training & Evaluation**
- Find training loops and evaluation metrics
- Look for: fit, train, predict, evaluate, score, metric, loss

### Step 2: Map the Complete Pipeline

Map the full pipeline from raw data to reported metrics. For each stage, determine:
- What information flows in?
- What information flows out?
- Could any downstream information leak upstream?
- Classify each stage as pre-split or post-split.

### Step 3: Identify Leakage Risks

**CRITICAL — Analyze Leakage Direction:**
For every data transformation:
- Does it use information from the full dataset (leakage risk) or only from the training partition?
- Is normalization fitted on train-only or all data?
- Are features derived from labels?

Assign a severity level (High/Medium/Low) to each leakage risk based on whether it would invalidate reported metrics.

### Step 4: Create the Diagram

Use flowchart with:

**Direction:** `LR` (data flows left to right)

**Subgraphs:**
- RAW DATA
- PREPROCESSING
- SPLIT POINT
- TRAIN PATH
- TEST PATH
- EVALUATION

**Node Styling:**
- `cli` class: Data sources
- `handler` class: Transforms
- `detector` class: Split point and validation gates
- `stateNode` class: Data stores
- `gap` class: Leakage risks
- `output` class: Metrics and results
- `phase` class: Model training

**Edge Labels:** full data, train only, test only, LEAKAGE RISK

### Step 5: Write Output

Write the diagram to: `temp/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md`

---

## Output Template

```markdown
# Pipeline Integrity Diagram: {Experiment Name}

**Lens:** Pipeline Integrity (Integrity)
**Question:** Could data handling create optimistic bias?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Pipeline Stages

| Stage | Input | Output | Pre/Post Split | Leakage Risk? |
|-------|-------|--------|----------------|---------------|
| {stage} | {input} | {output} | {Pre/Post} | {Yes/No} |

## Pipeline Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Raw ["RAW DATA"]
        SRC["Raw Dataset━━━━━━━━━━Source pathN samples"]
    end

    subgraph Preprocessing ["PREPROCESSING"]
        PREP["Normalization / Encoding━━━━━━━━━━Fitted on: full/train?"]
        LEAK["Leaky Transform━━━━━━━━━━Uses full dataset"]
    end

    subgraph SplitPoint ["SPLIT POINT"]
        SPLIT["Train/Test Split━━━━━━━━━━Stratified? Ratio?"]
    end

    subgraph TrainPath ["TRAIN PATH"]
        TRAIN_DATA["Train Set━━━━━━━━━━N_train samples"]
        MODEL["Model Training━━━━━━━━━━fit()"]
    end

    subgraph TestPath ["TEST PATH"]
        TEST_DATA["Test Set━━━━━━━━━━N_test samples"]
    end

    subgraph Evaluation ["EVALUATION"]
        METRIC["Reported Metric━━━━━━━━━━score / loss"]
    end

    %% PIPELINE FLOWS %%
    SRC -->|"full data"| PREP
    PREP -->|"full data"| LEAK
    LEAK -.->|"LEAKAGE RISK"| METRIC
    PREP -->|"full data"| SPLIT
    SPLIT -->|"train only"| TRAIN_DATA
    SPLIT -->|"test only"| TEST_DATA
    TRAIN_DATA -->|"fit"| MODEL
    MODEL -->|"predict"| TEST_DATA
    TEST_DATA -->|"evaluate"| METRIC

    %% CLASS ASSIGNMENTS %%
    class SRC cli;
    class PREP handler;
    class LEAK gap;
    class SPLIT detector;
    class TRAIN_DATA,TEST_DATA stateNode;
    class MODEL phase;
    class METRIC output;
```

**Color Legend:**
| Color | Category | Description |
|-------|----------|-------------|
| Dark Blue | Data Source | Raw input datasets |
| Orange | Transform | Preprocessing and feature engineering steps |
| Red | Split / Gate | Split point and validation gates |
| Teal | Data Store | Partitioned data stores (train/test) |
| Purple | Training | Model training stages |
| Dark Teal | Output | Reported metrics and results |
| Amber | Leakage Risk | Transforms using full-dataset information |

## Leakage Assessment

| Risk | Stage | Mechanism | Severity |
|------|-------|-----------|----------|
| {risk name} | {stage} | {how leakage occurs} | {High/Medium/Low} |

## Pipeline Invariants

- [ ] All scalers/encoders fitted on train partition only
- [ ] Feature selection criteria computed from train partition only
- [ ] No label information used in feature construction
- [ ] Test set never seen by any fitting step
```

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-reproducibility-artifacts` - For artifact completeness audit
- `/exp-lens-measurement-validity` - For outcome measurement validity

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Trecek](https://github.com/Trecek)
- **Source:** [Trecek/useful-claude-skills](https://github.com/Trecek/useful-claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-trecek-useful-claude-skills-exp-lens-pipeline-integrity
- Seller: https://agentstack.voostack.com/s/trecek
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.