# Ai Validity Evidence > >- - **Type:** Skill - **Install:** `agentstack add skill-openmatter-network-agent-io-skills-ai-validity-evidence` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [OpenMatter-Network](https://agentstack.voostack.com/s/openmatter-network) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [OpenMatter-Network](https://github.com/OpenMatter-Network) - **Source:** https://github.com/OpenMatter-Network/agent-io-skills/tree/main/ai-selection-legal-ethical/skills/ai-validity-evidence ## Install ```sh agentstack add skill-openmatter-network-agent-io-skills-ai-validity-evidence ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # AI validity evidence (Concern 5) Among I-O psychologists there is **no doubt** about the importance of validity. Evidence of validity is necessary to demonstrate a selection procedure's effectiveness, and it is a **legal requirement when adverse impact is observed.** For business and legal-defensibility reasons, **validity is a *sine qua non* of selection research** — including for AI tools. This is the AI-specific framing; for *how* to gather validity evidence, use `criterion-related-validation`, `content-based-validation`, and `generalizing-validity-evidence`. ## AI tools are "tests" — so they need validity AI-based predictors, game-oriented tools, and evaluations of facial features and voice qualities used to decide whom to hire are **tests under both the law and professional guidelines** (UGESP §2B: a test is any selection procedure used for an employment decision). Therefore: - In the **U.S.**, their use must be supported by **validity evidence if adverse impact results.** - In **all countries**, validity evidence **provides the business justification** for use. ## Which validity strategy — and the AI wrinkle - **Criterion-related** — a statistical relationship between scores and a relevant criterion (correlation/regression). Report strength via **size, confidence interval, and significance**; regression should indicate **imprecision** (CIs, bootstrapping, cross-validation). Traditional selection rarely models **nonlinearities** (linearity affords interpretability and needs less power); ML **does** model and cross-validate nonlinearities, but interpretation is difficult and power may or may not be an issue depending on the number of variables and cases. - **Content** — SMEs relate the **content of the test to job requirements.** Because of the nature of AI-based tests, **content validation evidence is often not used at all, or only to supplement** a criterion-related study. (When content can't carry the argument, criterion evidence and job analysis must.) ### When overall model fit is the only handle Some ML methods are based on regression (logistic, lasso, ridge, elastic net) and yield **coefficients** — a "demonstration of a relationship." But complex models (random forests, neural nets) or **clustering** approaches don't express relationships as apparent, direct coefficients. There, **overall model fit (analogous to R² in regression) may be the only basis** for establishing the predictor–criterion relationship — which makes **job analysis all the more important** to ensure job-relevant predictors and criteria are the ones in the model, even when their weighting is opaque. ## Comparative data and the alternatives duty Because U.S. employers are obligated to search for **alternative procedures with equal or greater validity and less adverse impact** (UGESP §3B), **comparative data are particularly important.** The empirical, meta-analytic history of validities in I-O psychology can serve as a **reasonable baseline the ML model should have to beat** — but vendor claims of "higher validity / lower adverse impact" are frequently **unaccompanied by the evidence needed to make the comparison.** ## Fundamental requirements and documentation A key open question: should users of algorithm-based procedures be **required to provide specific conceptual and empirical evidence of predictor–criterion relationships**, in addition to overall model metrics? The *Uniform Guidelines*, *Principles*, and *Standards* all enumerate the components a technical report should document — yet critical information is **often omitted**, and the problem is **acute for AI** because so many decisions are made in selecting and tuning the model. Documentation should let an external party **understand, evaluate, and reproduce** the work. Ask: - How were data **obtained, prepared, cleaned, transformed, and combined**? What approach handled **missing data**? - With text mining / written responses, were effects of **vocabulary knowledge or verbal fluency controlled** where the text wasn't meant to measure verbal ability? - How was **conflicting applicant data** weighted, discarded, or cleaned? - Was **one** algorithmic approach taken or **several**, and could an **external party reproduce** the analysis? - Were algorithm choices **a priori** or **post hoc**? Were predictions **combined across algorithms**? - Was there **post hoc filtering** — inspecting results to pick the algorithm that "looked good" (a **suspect** practice)? ## Questions to ask (from the article) - What is **sufficient evidence of validity** when ML models are used? - What details of the AI research **must be documented**? - What details of the AI research **must be shared with users**? ## Pitfalls - Treating "no adverse impact" as a substitute for validity (see `ai-selection-legal-landscape`). - Accepting overall model fit (R²/AUC) without job-analytic grounding of the predictors/criterion. - Accepting "more valid, less adverse impact" claims without the comparative evidence. - Tolerating post hoc algorithm selection / filtering without cross-validation safeguards. - Thin documentation that no external party could reproduce. ## Checklist - [ ] Tool treated as a "test" requiring validity (esp. if adverse impact) - [ ] Validity strategy chosen (criterion-related / content / generalization) and justified for AI - [ ] If only overall model fit is available, predictor/criterion job-relevance secured via job analysis - [ ] Imprecision reported (CIs, bootstrapping, cross-validation) - [ ] Comparative data vs. traditional/meta-analytic baselines and less-adverse alternatives provided - [ ] Data prep, missing-data, weighting, and algorithm-selection decisions documented and reproducible - [ ] Post hoc filtering ruled out or controlled; evidence shared with users ## See also `criterion-related-validation` · `content-based-validation` · `generalizing-validity-evidence` · `technical-validation-report` · `ai-selection-tech-data-algorithms` (why a technology is not "universally valid") · `ai-ml-methodology-evaluation` (model-fit metrics) · `ai-job-analysis-and-relevancy` · `ai-selection-legal-landscape` · `ai-model-outputs-audit` *Source: Tippins, Oswald & McPhail (2021), Concern: "Validity."* ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [OpenMatter-Network](https://github.com/OpenMatter-Network) - **Source:** [OpenMatter-Network/agent-io-skills](https://github.com/OpenMatter-Network/agent-io-skills) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-openmatter-network-agent-io-skills-ai-validity-evidence - Seller: https://agentstack.voostack.com/s/openmatter-network - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.