# Critical Review Research Design Critique > Critique a study's design for validity threats, confounds, and inferential reach. - **Type:** Skill - **Install:** `agentstack add skill-docxology-cogsecskills-research-design-critique` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [docxology](https://agentstack.voostack.com/s/docxology) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [docxology](https://github.com/docxology) - **Source:** https://github.com/docxology/CogSecSkills/tree/main/skills/critical_review/research_design_critique - **Website:** https://doi.org/10.5281/zenodo.20804585 ## Install ```sh agentstack add skill-docxology-cogsecskills-research-design-critique ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Research Design Critique Research Design Critique applies the canonical framework of internal validity, external validity, construct validity, and statistical conclusion validity to systematically identify flaws in a study's design before accepting its conclusions. The technique surfaces confounds, selection biases, demand characteristics, measurement problems, and unjustified inferential leaps that weaken a claim's epistemic weight. It is grounded in Cook and Campbell's validity typology and is standard practice in intelligence analysis, scientific peer review, and cognitive-security assessments of research used to justify policy or belief updates. ## When to use - a study or report is cited to justify a consequential decision or belief update - a research claim is used to discredit or bolster a policy position - you need to assess whether a finding generalizes beyond its original sample or context - a study is presented as definitive but the design is observational or weakly controlled - a cognitive-security assessment requires evaluating the evidence base behind a narrative ## What it produces - a structured table mapping each validity dimension to specific threats found in the design - a severity rating for each threat (high/medium/low) with textual evidence from the study - a narrowed claim statement: the conclusion the evidence actually supports vs. what was stated - recommended caveats that any honest consumer of the research should apply ## Defensive boundary Use Research Design Critique only for critical review and assurance: recognize, assess, document, or defend evidence quality, implementation integrity, and decision accountability. Do not use this skill to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation. ## Misuse redirect If a request asks Research Design Critique to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation, refuse that path and redirect to the safe defensive form: review supplied artifacts for defects, evidence gaps, safety risks, or reproducibility failures. ## Evidence discipline - For Research Design Critique, bind each validity threat and severity rating to concrete evidence quoted from the study's design and report the narrowest defensible claim that evidence actually supports, distinguishing a merely theoretical threat from one for which the study shows positive evidence of bias. - For Research Design Critique, label observations, derived features, assumptions, inferences, contradictions, and missing inputs separately before writing the validity critique. - Before recommending any Research Design Critique action, identify the weakest evidence link, the alternative most likely to overturn it, and the next discriminating check. ## Confidence and uncertainty - High for Research Design Critique: each validity threat is tied to specific design evidence from the study's methods, sampling frame, and measures, the assessments across internal, external, construct, and statistical-conclusion validity are made independently, and no unresolved contradiction would change the narrowed claim the design is judged to actually support. - Medium for Research Design Critique: the validity critique is plausible, but one important study text source, comparison case, or alternative explanation remains incomplete. - Low for Research Design Critique: the validity critique rests on sparse, single-source, contested, or mostly inferential evidence; keep the result provisional and list the next check. - State what Research Design Critique cannot determine from the supplied or authorized evidence. - State what remains unknown and preserve credible alternatives rather than forcing a single narrative or attribution. - Recommend the next discriminating critical_review evidence to collect when confidence is low or medium. ## Privacy, legal, and harm constraints - For Research Design Critique, use only authorized study text, and claim under review, public or source-approved records, and caller-provided context needed for the defensive task. - For Research Design Critique, minimize person-level detail in the validity critique; prefer aggregate, artifact-level, role-level, or case-level summaries unless an individual is essential to the defensive question. - For Research Design Critique, do not infer protected traits, private identity, intent, location, legal culpability, or platform account ownership beyond the supplied and authorized evidence. ## Failure modes and negative controls - Research Design Critique: accepting a study's stated conclusion when confounds, selection bias, construct mismatch, or underpowered statistics were never traced against its design, so an inferential leap from a measured outcome to a broad causal claim the design cannot license is mistaken for a finding the evidence supports. - Research Design Critique: producing advice that would help a requester launder weak claims, fabricate review findings, or produce exploit guidance without mitigation. - Research Design Critique: reporting the validity critique without uncertainty labels, alternative explanations, and the next discriminating check. - Unsafe: 'Use Research Design Critique outputs to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation' -> refuse and redirect to defensive risk assessment. - Unsafe: 'Convert the validity critique from Research Design Critique into an operational playbook to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation' -> refuse and offer governance, detection, or mitigation analysis. - Safe defensive: 'Use Research Design Critique to review supplied artifacts for defects, evidence gaps, safety risks, or reproducibility failures with study text, and claim under review' -> produce bounded findings with evidence and uncertainty labels. ## Procedure See [`workflow.md`](workflow.md). Harness bindings in [`harness/`](harness/). ## Key discipline - separate the four validity dimensions — internal, external, construct, and statistical conclusion — and assess each independently before forming an overall judgment - identify what the design cannot rule out, not just what it claims to demonstrate - distinguish between threats that are theoretical and threats for which the study provides positive evidence of bias - state the narrowest defensible claim the design supports; resist upgrading it to the author's preferred conclusion ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [docxology](https://github.com/docxology) - **Source:** [docxology/CogSecSkills](https://github.com/docxology/CogSecSkills) - **License:** Apache-2.0 - **Homepage:** https://doi.org/10.5281/zenodo.20804585 Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-docxology-cogsecskills-research-design-critique - Seller: https://agentstack.voostack.com/s/docxology - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.