Validation Planning

Validation planning

The front end of any selection project. Validation should begin with a clear statement of the proposed uses and the intended interpretations and outcomes — then a design is chosen to support those inferences. Getting this wrong wastes the whole study: a poorly executed effort can lead an employer to reject a beneficial procedure or adopt an invalid one. "Misleading, poorly designed validation efforts should not be undertaken."

What validity means here

Validity is the degree to which evidence and theory support the interpretation of scores for a proposed use. You validate an inference (e.g., "scores predict job performance"), not a test. Plan around the inference you must defend.

The Standards name five sources of validity evidence; for employment the first three usually carry the argument:

1. Relationships with other variables (test–criterion) — see criterion-related-validation
2. Content — see content-based-validation
3. Internal structure — see internal-structure-validation
4. Response processes (often irrelevant to employment use)
5. Consequences (relevant to validity only when a negative consequence traces to a measurement

property of the procedure — e.g., bias; mere subgroup-mean differences are not, by themselves, evidence against validity)

Sources are not competing "types" of validity; they are complementary evidence. Plan to combine them where the inference is complex, novel, or high-stakes.

Steps

1. Define the organization's needs, objectives, and constraints. Work collaboratively with HR,

recruiting, legal/compliance, IT, labor relations, and affected stakeholders. Different units have competing objectives (rigor vs. applicant flow). Run a cost–benefit lens over candidate procedures.

1. Specify the proposed use(s) precisely. Target job(s) or job family, candidate pool

(experienced vs. inexperienced; applicants vs. incumbents), the decision (hire, promote, place, certify), and how scores will be used (rank order, cutoff, band). The use drives the design.

1. Characterize the setting. One homogeneous organization or many? Stable work or rapidly

changing? Workforce size and applicant availability — these constrain whether a local study is even feasible and which strategies (local criterion study, VG, synthetic, content) are open.

1. Inventory existing evidence. Is there sufficient accumulated validity evidence (meta-analytic

bases for cognitive ability and a growing set of noncognitive measures; prior local studies; transportable studies) to support the proposed use without new data? Weigh the informational value of new evidence against its cost. Existing evidence alone may or may not suffice — judge the strength of the generalization to your setting. See generalizing-validity-evidence.

1. Apply the requirements of sound inference. Reliable and relevant measures, representative

samples, appropriate analyses, controls over plausible confounds, and people qualified for the tasks they perform in the study.

1. Assess feasibility. Time, cost, sample size/statistical power, organizational disruption,

union–management relations. Constraints may legitimately narrow the scope of defensible generalizations — but never justify a misleading design.

1. Choose a validation strategy that fits objectives, constraints, and the procedures/criteria.

Often more than one source of evidence is valuable. Three illustrative fits:

• Small population, strong cumulative evidence in similar settings → validity generalization.
• Same job extended from one business unit to another → transportability.
• Position unique to the organization, no external evidence → content-based strategy.

1. Communicate the plan. Management and workers need the purpose, the research plan, and their

roles. Decide and communicate confidentiality (e.g., concurrent study data kept out of employment decisions). Document the plan.

Decision guidance: which strategy?

| Situation | Lean toward | |-----------|-------------| | Adequate sample, relevant criterion obtainable, need local proof of prediction | criterion-related | | KSAOs/behaviors sampled directly from work; content closely mirrors the job | content-based | | Strong external cumulative evidence; small local sample | meta-analytic validity generalization | | Same job already validated in a comparable unit | transportability | | Many jobs share common work components | synthetic / job-component validity | | Multidimensional procedure whose internal structure must be defended | internal structure (as support) |

Pitfalls

• Designing the study before pinning down the inference and use.
• Treating subgroup-mean differences as automatic evidence against validity (they trigger

scrutiny for bias, not a verdict).

• Underpowered local studies: adequate power while controlling Type I error often needs samples

that are hard to obtain — consider this before committing to a criterion-related design.

• Letting convenience (available criterion, available sample, available software) drive design.
• Promising legal defensibility — the Principles address psychological method, not the law.

Checklist

• [ ] Proposed use(s), target job(s), and candidate pool stated explicitly
• [ ] Inference(s) to be supported written down
• [ ] Existing evidence inventoried; gap (if any) justified
• [ ] Feasibility (power, cost, time, access) assessed honestly
• [ ] Strategy chosen and rationale documented
• [ ] Stakeholders identified; communication & confidentiality plan set
• [ ] Plan documented (feeds technical-validation-report)

See also

work-analysis · criterion-related-validation · content-based-validation · internal-structure-validation · generalizing-validity-evidence · technical-validation-report

Source: Principles (5th ed., 2018), "Overview of the Validation Process" and "Operational Considerations → Initiating a Validation Effort / Selecting the Validation Strategy."

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: OpenMatter-Network
• Source: OpenMatter-Network/agent-io-skills
• License: MIT

Install and usage instructions live in the source repository linked above.