# Fairness And Bias Analysis

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-openmatter-network-agent-io-skills-fairness-and-bias-analysis`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [OpenMatter-Network](https://agentstack.voostack.com/s/openmatter-network)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [OpenMatter-Network](https://github.com/OpenMatter-Network)
- **Source:** https://github.com/OpenMatter-Network/agent-io-skills/tree/main/personnel-selection/skills/fairness-and-bias-analysis

## Install

```sh
agentstack add skill-openmatter-network-agent-io-skills-fairness-and-bias-analysis
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Fairness and bias analysis

Two distinct ideas that are routinely conflated. Keep them separate.

## Fairness is a social concept, not a single statistic

"Fairness" has **no single agreed definition** (statistical, psychometric, or social). Recognized
meanings include:
- **Equal group outcomes** (e.g., equal passing rates). The *Standards* **reject** this as the
  definition of fairness: group outcome differences alone **do not indicate bias** — though they
  should **trigger heightened scrutiny** for possible bias.
- **Equitable treatment** of all examinees (testing conditions, access to practice materials,
  feedback, retest opportunities, reasonable accommodation, mode of administration).
- **Comparable access to the construct** — accessible testing so all candidates can show their
  standing without being advantaged/disadvantaged by construct-irrelevant characteristics (age,
  race, ethnicity, gender, SES, cultural/linguistic background, disability).
- **Lack of bias.**

There is broad agreement that equitable treatment, access, bias, and scrutiny when subgroup
differences appear are important — but **no agreement that "fairness" can be uniquely defined** in
terms of any one of them.

## Bias is a technical concept — two forms

**Bias** = systematic error that differentially affects the performance of different subgroups.

- **Predictive bias** — slope and/or intercept of the predictor→criterion regression differs across
  groups (a *predictor–criterion* relationship issue).
- **Measurement bias** — construct-irrelevant variance producing systematically higher/lower
  **scores** for a subgroup (a *score* issue, for predictors or criteria).

### Crucial point on consequences
A subgroup-mean difference (adverse impact) is a **negative consequence**, but it is evidence against
validity **only if it traces to a measurement property** of the procedure (i.e., bias). If the group
difference on the procedure mirrors a real difference in the work-relevant outcome (i.e., **no
predictive bias**), the consequence is a **policy issue** for the user, not a validity defect.

## Predictive bias / differential prediction

Test via **moderated multiple regression (MMR)**: regress the criterion on the predictor, subgroup
membership, and their **interaction**. Slope and/or intercept differences signal predictive bias.
MMR is preferred over comparing separate subgroup correlation coefficients.

- Frame the question as **"is the subgroup's performance *underpredicted*?"** — only
  **underprediction** signals bias against that group. Simply knowing slopes/intercepts differ
  doesn't answer it. (In U.S. cognitive-ability research, slope differences are rare; when intercept
  differences occur they typically take the form of **overprediction** of minority performance —
  Schmidt, Pearlman, & Hunter, 1980; and corrected analyses, e.g., Berry & Zhao, 2015, still find
  little underprediction.)
- Consider **effect sizes** as well as statistical significance (Nye & Sackett, 2017; Dahlke &
  Sackett, 2017).

### Technical cautions (predictive bias)
1. Analyze predictors **as operationally used** (e.g., test the **composite** when selection uses a
   composite, not each test separately).
2. A **confident, unbiased criterion** is a prerequisite.
3. **Statistical power is a chronic problem** — small total/subgroup samples, unequal subgroup sizes,
   range restriction, and predictor unreliability all reduce power to detect slope/intercept
   differences.
4. Check the **homogeneity-of-error-variance** assumption; use alternative tests when it's violated.
5. Use an **unbiased estimate** of the intercept difference and operational validity parameters
   (not observed parameters).

Predictive bias and mean differences can exist **independently**; analyze predictive bias when
there's compelling reason to question whether predictor and criterion relate comparably across
subgroups *and* appropriate data exist. Where relevant research exists, generalized evidence can
inform the question.

## Measurement bias

Construct-irrelevant variance raising/lowering scores for a subgroup — hard to detect because it
requires comparing an observed score to a **true** score. Approaches:

- **Item sensitivity review** — diverse reviewers examine items (and instructions to candidates and
  scorers) for language/content that could carry **differing meaning** across subgroups or be
  **demeaning/offensive**. Value depends on content; use is a matter of professional judgment.
- **Differential item functioning (DIF)** — identifies items on which members of different subgroups
  with the **same total score (or same IRT true score)** perform differently. Notes:
  - Needs **large samples** for stable results.
  - Domains where DIF is common have **rarely** shown sizable, replicable DIF (Sackett et al., 2001);
    for cognitive tests it's common to find roughly equal numbers of items favoring each subgroup,
    netting to little test-level bias.
  - DIF is **not a routine/expected** part of selection development; explore it when appropriate data
    exist. Especially useful in **cross-cultural / linguistically different** testing.

## Pitfalls

- Equating adverse impact with bias, or "no bias" with "fair."
- Testing each component instead of the operational composite.
- Running underpowered bias analyses and reading a null as "no bias."
- Comparing subgroup correlations instead of using MMR.
- Treating any slope/intercept difference as bias without asking about *under*prediction direction.

## Checklist

- [ ] "Fairness" meaning(s) at issue named explicitly
- [ ] Subgroup differences treated as a scrutiny trigger, not a verdict
- [ ] Predictive bias tested with MMR on the **operational** predictor/composite
- [ ] Underprediction direction (not mere difference) interpreted; effect sizes reported
- [ ] Power, range restriction, unreliability, error-variance homogeneity addressed
- [ ] Unbiased parameter estimates used
- [ ] Measurement bias considered (item sensitivity review and/or DIF) where data/justification exist
- [ ] Equitable treatment and access to the construct addressed (see accommodations skill)

## See also

`criterion-related-validation` · `selection-decisions-and-scoring` (composites & subgroup tradeoffs)
· `candidate-accommodations` (equitable treatment/access) · `internal-structure-validation` ·
`technical-validation-report`

*Source: Principles (5th ed., 2018), "Fairness and Bias."*

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [OpenMatter-Network](https://github.com/OpenMatter-Network)
- **Source:** [OpenMatter-Network/agent-io-skills](https://github.com/OpenMatter-Network/agent-io-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-openmatter-network-agent-io-skills-fairness-and-bias-analysis
- Seller: https://agentstack.voostack.com/s/openmatter-network
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.