# Ai Dynamic Models And Revalidation > >- - **Type:** Skill - **Install:** `agentstack add skill-openmatter-network-agent-io-skills-ai-dynamic-models-and-revalidation` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [OpenMatter-Network](https://agentstack.voostack.com/s/openmatter-network) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [OpenMatter-Network](https://github.com/OpenMatter-Network) - **Source:** https://github.com/OpenMatter-Network/agent-io-skills/tree/main/ai-selection-legal-ethical/skills/ai-dynamic-models-and-revalidation ## Install ```sh agentstack add skill-openmatter-network-agent-io-skills-ai-dynamic-models-and-revalidation ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # AI dynamic models & revalidation (Concern 7) ML-derived selection procedures enable analysis that wasn't feasible before, and many vendors **refresh the algorithm frequently — sometimes after every test administration.** Such "dynamic" procedures update validation evidence and normative data in **near real time.** That creates a dilemma and several operational problems. ## The core dilemma When a dynamic model changes, the change could reflect either: - a **real change** in the nature of the applicant sample and/or the job (which *should* drive an update), **or** - **sample idiosyncrasies or instabilities** over time that **should not be capitalized on.** Distinguishing the two is hard — and capitalizing on instability degrades the model. ## Why "dynamic" raises problems 1. **Documentation must keep up.** Validity evidence must be documented in a **technical report.** So every update to the underlying selection process requires **updating the technical report** (validity *and* normative-data characteristics). **Out-of-date reports are problematic** for legal defensibility and for HR records maintenance over time. (See `technical-validation-report`.) 2. **Score adjustments & grandparenting.** Each new selection process may require **adjusting scores** already in the database, or a **policy** on how to treat scores produced by different processes at different times. Continual adjustment creates administration problems: a candidate **qualified today may be unqualified tomorrow** (or vice versa). Changes can include **altering predictor weights, adding/removing predictors, and changing interpretations** (e.g., adjusting **cutoff scores**). **Grandparenting** policies are administratively hard to manage in high-volume programs. (See `selection-decisions-and-scoring`.) 3. **Disparate-treatment / disparate-impact risk.** A result of frequent change is that **different candidates are evaluated on different variables depending on when they applied.** If that variation relates to a protected characteristic, there's a **disparate-impact** specter; even **without** group-level disparate impact, **disparate-treatment** concerns can arise — and the **appearance** of such treatment can trigger applicant dissatisfaction and complaints. 4. **Applicant-pool shifts.** Large shifts in the applicant pool can materially affect **reliability, validity, range restriction, or range enhancement** on big data — so **monitor key applicant-pool characteristics** (demographic, educational). Future shifts in available technologies/data/algorithms may change the applicant database and **indicate the ML model should be updated.** 5. **Defining group comparisons.** Frequent algorithm changes make it **harder to define relevant applicant pools** for adverse-impact analysis or appropriate **normative groups** for comparison. ## Revalidation & norms updating Employers have **always** revalidated and updated norms; with AI the difference is the **frequency.** Traditionally, revalidation was triggered when the job changed, the test was compromised, the applicant pool shifted substantially, or enough time elapsed to question validity in a legal challenge — and it was undertaken at **well-spaced intervals** because it was laborious. Today's computing power makes **continuous updating** far less laborious, which raises a genuinely open question: **how often should validation be refreshed** to accommodate the nature of new applicant data? ## Questions to ask (from the article) - Are **dynamic algorithms and norms useful?** - How should results from dynamic algorithms be **documented to comply with existing and future legal and administrative requirements?** - How **frequently** should tests be revalidated and norms updated? - What are the **indicators** that revalidation and updates to norms are needed? ## Pitfalls - Capitalizing on sample instability and mistaking it for real change. - Letting technical reports/normative documentation fall out of date as the model drifts. - Unmanaged score adjustments → candidates flipping qualified/unqualified; messy grandparenting. - Evaluating different applicants on different variables → disparate-treatment exposure (and optics). - Not monitoring applicant-pool shifts that invalidate prior validity/range assumptions. - Unstable pool/normative definitions that make adverse-impact analysis incoherent. ## Checklist - [ ] Each model/norm update justified as real change, not instability - [ ] Technical report and normative documentation updated with every change - [ ] Score-adjustment / grandparenting policy defined and administrable - [ ] Risk that candidates are scored on different variables (disparate treatment) assessed - [ ] Applicant-pool characteristics monitored for shifts affecting validity/range - [ ] Revalidation/norms-update cadence and trigger indicators defined - [ ] Adverse-impact pools and normative groups remain definable under the update regime ## See also `ai-validity-evidence` · `ai-selection-legal-landscape` · `generalizing-validity-evidence` · `technical-validation-report` · `administration-documentation` (review/updating, records) · `selection-decisions-and-scoring` (cutoffs, norms) *Source: Tippins, Oswald & McPhail (2021), Concern: "Changes to Technologically Enhanced Systems" (Dynamic models and norms; Revalidation and norms updating).* ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [OpenMatter-Network](https://github.com/OpenMatter-Network) - **Source:** [OpenMatter-Network/agent-io-skills](https://github.com/OpenMatter-Network/agent-io-skills) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-openmatter-network-agent-io-skills-ai-dynamic-models-and-revalidation - Seller: https://agentstack.voostack.com/s/openmatter-network - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.