Technical Validation Report

Technical validation report

The durable record of a validation effort. Standard: enough detail that a competent professional not involved in the study could understand what was done, evaluate it, replicate it, and make recommendations. Report findings, interpretations, and the decisions based on them accurately, including findings that qualify the conclusions or limit generalizability.

Keep it separate from the administration manual and from secure operational content (don't put scoring keys or live items in it). See administration-documentation.

Required sections

Use [report-outline.md](report-outline.md) as a fill-in template. Each section below maps to it.

1. Identifying information — authors, credentials, affiliations, dates, and what's needed to know

who conducted the original research.

1. Statement of purpose — the purpose of the validation research.
2. Analysis of work — description of the analysis, participant characteristics, SME judgments,

instructions given to participants for their tasks, analyses and results including reliability/precision. If any materials were translated, describe the translation/adaptation.

1. Search for alternative selection procedures — document any search for procedures (or alternate

combinations) showing substantially equal or greater validity with less subgroup difference.

1. Identification or development of selection procedures — names, editions, forms of published

procedures; for proprietary tools, the construct(s) measured, the content, and how content was developed. Do not include scoring algorithms/keys or live items (protect security); do document scoring procedures enough to ensure accurate, consistent scoring. Give the rationale for each statistical procedure and analysis.

1. Establishing validity — describe the validation studies so another professional could

reproduce analyses and results; describe methods used to determine the procedure is statistically and practically related to a criterion and/or representative of a job-content domain.

• Criterion studies: report criterion measures; the rationale for them; data-collection

procedures; and a discussion of relevance, reliability, possible deficiency, possible contamination, and freedom from/control of biasing variance. If you developed the criterion, give the rationale and steps.

1. Research sample — sampling procedure and sample characteristics relative to interpretation;

the population the sample represents; sampling biases; significance of deviations from representativeness; statistical power results. Note range restriction in scores. State whether psychometrics refer to candidates or incumbents. Don't present concurrent results as predictive.

1. Results — all summary statistics tied to the conclusions and recommendations; complete

statistical results (not just significant/supportive ones), clearly labeled. Present both uncorrected and corrected values when correcting for artifacts (range restriction, criterion unreliability).

1. Scoring and transformation of raw scores — methods/algorithms used to score content; rationale

for weighted/derived/composite/categorical scores; for judgment-based scoring (work samples, performance tasks), describe rater training and scoring criteria.

1. Normative information — normative parameters and how to interpret them; demographic/

occupational characteristics, data-collection time frame, and status of test takers (candidates/ incumbents/students); central tendency, variability (and skewness when appropriate); clear description of the normative data (percentiles, standard scores); percent passing; expectancy tables.

1. Recommendations — recommended use and the rationale (rank order, bands, cutoffs; how

information is combined). Note that some implementation rules may change over time; place subsequent modifications in an addendum.

1. Caution regarding interpretations — help readers interpret data correctly and **warn against

common misuses**.

1. Technology-enabled selection procedures — if technology-enabled, document the technology

requirements and any technology-based accommodations administrators can provide for test takers with disabilities.

1. References — complete references for all published literature and technical reports cited.

(Note: proprietary/confidential technical reports may not be generally available.)

Cross-cutting reporting expectations

• Report the rationale for each statistical procedure and analysis performed.
• If raters are integral (e.g., some work samples), determine and document **rater

reliability/agreement**.

• Document decision rules used in preparing data (handling of missing data, outliers,

inconsistencies across sources) and justify them.

• For transportability support, include the relationship between the original validation sample

and the population for which use is now proposed.

Pitfalls

• Reporting only significant/favorable results.
• Reporting only corrected coefficients (always pair with uncorrected) or applying naive significance

tests to corrected values.

• Presenting concurrent results as if predictive.
• Putting secure content (keys, live items) in the report.
• Omitting power analysis, range-restriction notes, or the criterion's deficiency/contamination

discussion.

• Failing to document the search for less-adverse alternatives.

Checklist

• [ ] All 14 sections present (use report-outline.md)
• [ ] Replicable: methods/analyses described in enough detail
• [ ] Corrected and uncorrected results; power; range restriction noted
• [ ] Criterion relevance/reliability/deficiency/contamination discussed
• [ ] Sample representativeness, population, incumbent-vs-candidate status stated
• [ ] Alternatives search documented
• [ ] Secure content excluded; scoring documented enough for consistency
• [ ] Data-handling decision rules justified
• [ ] Misuse cautions and interpretation guidance included
• [ ] References complete

See also

administration-documentation (the separate operational manual) · every evidence skill (criterion-related-validation, content-based-validation, internal-structure-validation, generalizing-validity-evidence) · fairness-and-bias-analysis · selection-decisions-and-scoring

Source: Principles (5th ed., 2018), "Operational Considerations → Technical Validation Report."

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: OpenMatter-Network
• Source: OpenMatter-Network/agent-io-skills
• License: MIT

Install and usage instructions live in the source repository linked above.