# Validation Planning

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-openmatter-network-agent-io-skills-validation-planning`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [OpenMatter-Network](https://agentstack.voostack.com/s/openmatter-network)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [OpenMatter-Network](https://github.com/OpenMatter-Network)
- **Source:** https://github.com/OpenMatter-Network/agent-io-skills/tree/main/personnel-selection/skills/validation-planning

## Install

```sh
agentstack add skill-openmatter-network-agent-io-skills-validation-planning
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Validation planning

The front end of any selection project. Validation should begin with a **clear statement of the
proposed uses and the intended interpretations and outcomes** — *then* a design is chosen to
support those inferences. Getting this wrong wastes the whole study: a poorly executed effort can
lead an employer to reject a beneficial procedure or adopt an invalid one. "Misleading, poorly
designed validation efforts should not be undertaken."

## What validity means here

Validity is the degree to which evidence and theory support the **interpretation of scores for a
proposed use**. You validate an *inference* (e.g., "scores predict job performance"), not a test.
Plan around the inference you must defend.

The *Standards* name five sources of validity evidence; for employment the first three usually
carry the argument:
1. **Relationships with other variables** (test–criterion) — see `criterion-related-validation`
2. **Content** — see `content-based-validation`
3. **Internal structure** — see `internal-structure-validation`
4. **Response processes** (often irrelevant to employment use)
5. **Consequences** (relevant to validity only when a negative consequence traces to a measurement
   property of the procedure — e.g., bias; mere subgroup-mean differences are *not*, by themselves,
   evidence against validity)

Sources are not competing "types" of validity; they are complementary evidence. Plan to combine
them where the inference is complex, novel, or high-stakes.

## Steps

1. **Define the organization's needs, objectives, and constraints.** Work collaboratively with HR,
   recruiting, legal/compliance, IT, labor relations, and affected stakeholders. Different units
   have competing objectives (rigor vs. applicant flow). Run a cost–benefit lens over candidate
   procedures.
2. **Specify the proposed use(s) precisely.** Target job(s) or job family, candidate pool
   (experienced vs. inexperienced; applicants vs. incumbents), the decision (hire, promote, place,
   certify), and how scores will be used (rank order, cutoff, band). The *use* drives the design.
3. **Characterize the setting.** One homogeneous organization or many? Stable work or rapidly
   changing? Workforce size and applicant availability — these constrain whether a local study is
   even feasible and which strategies (local criterion study, VG, synthetic, content) are open.
4. **Inventory existing evidence.** Is there sufficient accumulated validity evidence (meta-analytic
   bases for cognitive ability and a growing set of noncognitive measures; prior local studies;
   transportable studies) to support the proposed use without new data? Weigh the *informational
   value* of new evidence against its cost. Existing evidence alone may or may not suffice — judge
   the strength of the generalization to your setting. See `generalizing-validity-evidence`.
5. **Apply the requirements of sound inference.** Reliable and relevant measures, representative
   samples, appropriate analyses, controls over plausible confounds, and people qualified for the
   tasks they perform in the study.
6. **Assess feasibility.** Time, cost, sample size/statistical power, organizational disruption,
   union–management relations. Constraints may legitimately narrow the scope of defensible
   generalizations — but never justify a misleading design.
7. **Choose a validation strategy that fits objectives, constraints, and the procedures/criteria.**
   Often more than one source of evidence is valuable. Three illustrative fits:
   - Small population, strong cumulative evidence in similar settings → **validity generalization**.
   - Same job extended from one business unit to another → **transportability**.
   - Position unique to the organization, no external evidence → **content-based** strategy.
8. **Communicate the plan.** Management and workers need the purpose, the research plan, and their
   roles. Decide and communicate confidentiality (e.g., concurrent study data kept out of
   employment decisions). Document the plan.

## Decision guidance: which strategy?

| Situation | Lean toward |
|-----------|-------------|
| Adequate sample, relevant criterion obtainable, need local proof of prediction | criterion-related |
| KSAOs/behaviors sampled directly from work; content closely mirrors the job | content-based |
| Strong external cumulative evidence; small local sample | meta-analytic validity generalization |
| Same job already validated in a comparable unit | transportability |
| Many jobs share common work components | synthetic / job-component validity |
| Multidimensional procedure whose internal structure must be defended | internal structure (as support) |

## Pitfalls

- Designing the study before pinning down the inference and use.
- Treating subgroup-mean differences as automatic evidence against validity (they trigger
  *scrutiny* for bias, not a verdict).
- Underpowered local studies: adequate power while controlling Type I error often needs samples
  that are hard to obtain — consider this *before* committing to a criterion-related design.
- Letting convenience (available criterion, available sample, available software) drive design.
- Promising legal defensibility — the *Principles* address psychological method, not the law.

## Checklist

- [ ] Proposed use(s), target job(s), and candidate pool stated explicitly
- [ ] Inference(s) to be supported written down
- [ ] Existing evidence inventoried; gap (if any) justified
- [ ] Feasibility (power, cost, time, access) assessed honestly
- [ ] Strategy chosen and rationale documented
- [ ] Stakeholders identified; communication & confidentiality plan set
- [ ] Plan documented (feeds `technical-validation-report`)

## See also

`work-analysis` · `criterion-related-validation` · `content-based-validation` ·
`internal-structure-validation` · `generalizing-validity-evidence` · `technical-validation-report`

*Source: Principles (5th ed., 2018), "Overview of the Validation Process" and "Operational
Considerations → Initiating a Validation Effort / Selecting the Validation Strategy."*

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [OpenMatter-Network](https://github.com/OpenMatter-Network)
- **Source:** [OpenMatter-Network/agent-io-skills](https://github.com/OpenMatter-Network/agent-io-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-openmatter-network-agent-io-skills-validation-planning
- Seller: https://agentstack.voostack.com/s/openmatter-network
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.