Phack Taxonomy

Strategy taxonomy

Read references/taxonomy.md. It is the substance of this skill: 27 strategies across three layers, each with what is chosen, why it is defensible, and what it costs in type I error — plus the procedure layer, because the false-positive rate of a session depends on the order in which knobs are turned and on when the searcher stops (09-search-procedures). The third strategy layer is what happens between a pilot and a confirmatory analysis (Adda, Decker & Ottaviani 2020): continuing only after a promising pilot is selection, not p-hacking, and keeps its size on a fresh sample; pooling the pilot into the confirmatory test, or registering only the significant stage, is.

Quantifying a strategy

python scripts/phack_cli.py simulate --strategy 07_transformation --n-sims 4000
python scripts/phack_cli.py simulate --workflow 09_alternative_tests,01_selective_dv,11_subgroup
python scripts/phack_cli.py simulate --n-sims 4000            # all thirteen simulated strategies
python scripts/phack_cli.py simulate --strategy 26_selective_continuation --report main    # 0.05: not p-hacking
python scripts/phack_cli.py simulate --strategy 26_selective_continuation --report pooled  # 0.17: it is now

Data are generated under a true null, so fpr_hacked is the probability the strategy manufactures a false positive. fpr_original is the calibration check and should land on 0.05.

Using it to classify

When someone describes an analytical choice, the useful question is not "is this p-hacking?" — almost nothing is p-hacking in isolation. It is:

1. Which axis of the grid is this? Map it to a numbered strategy.
2. Was it fixed before the outcome was seen? A choice made ex ante is a

design; the same choice made ex post is a degree of freedom spent.

1. How many alternatives were available and how many were tried? This is

the multiplicity that inference has to pay for.

1. Is the alternative set disclosed? A disclosed search is a multiverse

analysis. An undisclosed one is a p-hacked result.

Only question 2 and question 4 separate legitimate work from misconduct. Questions 1 and 3 are just accounting — and the accounting is what this suite automates.

What not to conclude

A high false-positive rate for a strategy does not mean anyone using that strategy is hacking. Outlier exclusion, covariate adjustment and imputation are all necessary in real data. The rates in the table are what happens when the choice is made after seeing the result, repeatedly, and reported as one.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: brycewang-stanford
• Source: brycewang-stanford/p-hacking-skills
• License: MIT

Install and usage instructions live in the source repository linked above.