# Phack Taxonomy

> Name and classify p-hacking strategies, and quantify what each one does to the false-positive rate. Covers the twelve-strategy compendium of Stefan and Schoenbrodt (2023), thirteen econometrics-specific degrees of freedom (clustering doctrine, fixed-effect structure, RDD bandwidth, kernel and inference mode, IV instrument sets and first-stage screening, staggered-DiD estimator and comparison-grou…

- **Type:** Skill
- **Install:** `agentstack add skill-brycewang-stanford-p-hacking-skills-01-phack-taxonomy`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [brycewang-stanford](https://agentstack.voostack.com/s/brycewang-stanford)
- **Installs:** 0
- **Category:** [Search](https://agentstack.voostack.com/c/search)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [brycewang-stanford](https://github.com/brycewang-stanford)
- **Source:** https://github.com/brycewang-stanford/p-hacking-skills/tree/main/skills/01-phack-taxonomy

## Install

```sh
agentstack add skill-brycewang-stanford-p-hacking-skills-01-phack-taxonomy
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Strategy taxonomy

Read `references/taxonomy.md`. It is the substance of this skill: 27 strategies
across three layers, each with what is chosen, why it is defensible, and what
it costs in type I error — plus the *procedure* layer, because the
false-positive rate of a session depends on the order in which knobs are
turned and on when the searcher stops (`09-search-procedures`). The third
strategy layer is what happens *between* a pilot and a confirmatory analysis
(Adda, Decker & Ottaviani 2020): continuing only after a promising pilot is
selection, not p-hacking, and keeps its size on a fresh sample; pooling the
pilot into the confirmatory test, or registering only the significant stage,
is.

## Quantifying a strategy

```bash
python scripts/phack_cli.py simulate --strategy 07_transformation --n-sims 4000
python scripts/phack_cli.py simulate --workflow 09_alternative_tests,01_selective_dv,11_subgroup
python scripts/phack_cli.py simulate --n-sims 4000            # all thirteen simulated strategies
python scripts/phack_cli.py simulate --strategy 26_selective_continuation --report main    # 0.05: not p-hacking
python scripts/phack_cli.py simulate --strategy 26_selective_continuation --report pooled  # 0.17: it is now
```

Data are generated under a true null, so `fpr_hacked` is the probability the
strategy manufactures a false positive. `fpr_original` is the calibration
check and should land on 0.05.

## Using it to classify

When someone describes an analytical choice, the useful question is not "is
this p-hacking?" — almost nothing is p-hacking in isolation. It is:

1. **Which axis of the grid is this?** Map it to a numbered strategy.
2. **Was it fixed before the outcome was seen?** A choice made ex ante is a
   design; the same choice made ex post is a degree of freedom spent.
3. **How many alternatives were available and how many were tried?** This is
   the multiplicity that inference has to pay for.
4. **Is the alternative set disclosed?** A disclosed search is a multiverse
   analysis. An undisclosed one is a p-hacked result.

Only question 2 and question 4 separate legitimate work from misconduct.
Questions 1 and 3 are just accounting — and the accounting is what this suite
automates.

## What not to conclude

A high false-positive rate for a strategy does not mean anyone using that
strategy is hacking. Outlier exclusion, covariate adjustment and imputation are
all *necessary* in real data. The rates in the table are what happens when the
choice is made **after** seeing the result, repeatedly, and reported as one.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [brycewang-stanford](https://github.com/brycewang-stanford)
- **Source:** [brycewang-stanford/p-hacking-skills](https://github.com/brycewang-stanford/p-hacking-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-brycewang-stanford-p-hacking-skills-01-phack-taxonomy
- Seller: https://agentstack.voostack.com/s/brycewang-stanford
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.