Selection Decisions And Scoring

Selection decisions and scoring

How scores become hiring/promotion decisions. Every choice here affects validity, expected performance of those selected, and subgroup passing rates — so each requires a documented rationale. There are few absolutes; professional judgment driven by organizational goals governs.

Combining procedures into a system

When multiple procedures form the basis of a decision, both the individual components and the combination must be supported by validity evidence. Document the method and rationale for combining and sequencing. Organizations weight differently depending on whether they emphasize maximizing validity, minimizing subgroup differences, or balancing the two — state which.

Recall from criterion-related-validation: effective weights ≠ nominal weights (they depend on component variances/covariances, and differential range restriction can distort them).

Compensatory vs. multiple hurdle

• Compensatory — candidates achieve a specified total across assessments; a high score on

one can offset a low score on another.

• Multiple hurdle — candidates must clear a score on each assessment (often sequential).
• Hybrids are possible (a hurdle on one critical predictor, compensatory among the rest).

No model is universally correct. The method of combining scores can affect the overall reliability of the process and subgroup passing rates (Sackett & Roth, 1996). Present the rationale and supporting evidence for the model recommended.

Cutoff scores vs. rank order

Two common strategies: a cutoff score (reject below a point) or rank-order / top-down selection.

• There is no single best method for setting cutoffs. Options include criterion-referenced

cutoffs (when the predictor links to a meaningful performance threshold) and others (Mueller, Norris, & Oppler, 2007).

• With valid predictors showing linearity, cutoffs may be set as high or low as needed to

meet organizational requirements. Nonmonotonicity in the predictor–criterion relation should inform how scores are used.

• When local data can't establish linearity/monotonicity, consult past research and its

implications (e.g., cognitive ability tends to relate linearly to performance; linearity for other predictors such as personality is less settled).

• When setting any cutoff, consider the conditional standard error of measurement at the cutoff

region; document the SEM model used. Consider reporting the percentage of applicants classified the same way (pass/fail) across replications at the cutoff (Haertel, 2006).

Bands

Bands are score ranges within which candidates are treated alike (a form of cutoff that defines ranges). Methods vary (Cascio, Outtz, Zedeck, & Goldstein, 1991; Campion et al., 2001).

• Rationale can be psychometric (imprecision/SEM of scores) and/or **administrative/

organizational**.

• Tradeoff: because banded candidates with different scores are treated alike, banding generally

yields lower expected criterion performance and utility than top-down selection — but may be balanced by administrative ease and possibly increased workforce diversity, depending on how within-band selection is done.

• Document the basis for development and the decision rules for administering the band.

Top-down vs. cutoff vs. bands — the driving factors

Decisions are typically driven by organizational goals and factors such as: estimated cost–benefit ratio, number of vacancies, selection ratio, labor market, expectancy of success vs. failure, consequences of selection errors, relative emphasis on performance vs. diversity goals, judgments about the level of KSAO/performance required, and the procedure's utility. Some organizations choose a cutoff over rank order to increase diversity, accepting possible reductions in performance and utility. Whatever the decision, document the rationale.

Norms

Present normative information for the applicant pool and incumbent population when appropriate. Describe the normative group's relevant demographic/occupational characteristics and the time frame. Note that large discrepancies between incumbents and the applicant pool can make incumbent-based cutoffs too high (or otherwise inappropriate).

Communicating effectiveness

• Expectancy charts relate score ranges to work performance; Taylor–Russell tables show the

proportion of hires who will be successful under combinations of validity, selection ratio, and base rate.

• Utility estimates project productivity gains (in dollars, output %, or reductions in

accidents/person-hours). Utility values rest on assumptions and uncertain parameters — report them as estimates, and present minimal and maximal point estimates to reflect uncertainty.

Appropriate use

Use a procedure only for purposes with validity evidence. Changing the mix or combination of components (especially in a compensatory system) can fundamentally change the supported inference — the original validation evidence may no longer suffice. Don't repurpose a procedure (e.g., diagnostic use, or an education-designed test for employment) without supporting evidence.

Pitfalls

• Combining procedures without validating the combination, not just the parts.
• Setting cutoffs without considering linearity/monotonicity or the conditional SEM.
• Presenting utility as precise dollar truth rather than an assumption-laden estimate.
• Using incumbent norms to set cutoffs when the applicant pool differs markedly.
• Silently changing weights/combination and assuming prior validity carries over.

Checklist

• [ ] Combination & sequencing method documented with rationale
• [ ] Compensatory/hurdle/hybrid choice justified; reliability & passing-rate effects considered
• [ ] Cutoff/rank/band choice tied to org goals (selection ratio, vacancies, error costs, diversity)
• [ ] Linearity/monotonicity and conditional SEM addressed for cutoffs
• [ ] Band development basis and within-band decision rules documented
• [ ] Norms described (group, demographics, time frame); incumbent-vs-applicant gap considered
• [ ] Effectiveness communicated (expectancy/utility) with uncertainty ranges
• [ ] Validity evidence covers the actual operational use

See also

criterion-related-validation (weights, composites, corrections) · fairness-and-bias-analysis (subgroup tradeoffs; analyze the operational composite) · technical-validation-report · administration-documentation

Source: Principles (5th ed., 2018), "Operational Considerations → Data Analyses (Combining procedures, Multiple hurdles vs. compensatory, Cutoff scores vs. rank orders, Bands, Norms) and Communicating the Effectiveness / Appropriate Use of Selection Procedures."

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: OpenMatter-Network
• Source: OpenMatter-Network/agent-io-skills
• License: MIT

Install and usage instructions live in the source repository linked above.