Critical Review Research Design Critique

Research Design Critique

Research Design Critique applies the canonical framework of internal validity, external validity, construct validity, and statistical conclusion validity to systematically identify flaws in a study's design before accepting its conclusions. The technique surfaces confounds, selection biases, demand characteristics, measurement problems, and unjustified inferential leaps that weaken a claim's epistemic weight. It is grounded in Cook and Campbell's validity typology and is standard practice in intelligence analysis, scientific peer review, and cognitive-security assessments of research used to justify policy or belief updates.

When to use

• a study or report is cited to justify a consequential decision or belief update
• a research claim is used to discredit or bolster a policy position
• you need to assess whether a finding generalizes beyond its original sample or context
• a study is presented as definitive but the design is observational or weakly controlled
• a cognitive-security assessment requires evaluating the evidence base behind a narrative

What it produces

• a structured table mapping each validity dimension to specific threats found in the design
• a severity rating for each threat (high/medium/low) with textual evidence from the study
• a narrowed claim statement: the conclusion the evidence actually supports vs. what was stated
• recommended caveats that any honest consumer of the research should apply

Defensive boundary

Use Research Design Critique only for critical review and assurance: recognize, assess, document, or defend evidence quality, implementation integrity, and decision accountability. Do not use this skill to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation.

Misuse redirect

If a request asks Research Design Critique to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation, refuse that path and redirect to the safe defensive form: review supplied artifacts for defects, evidence gaps, safety risks, or reproducibility failures.

Evidence discipline

• For Research Design Critique, bind each validity threat and severity rating to concrete evidence quoted from the study's design and report the narrowest defensible claim that evidence actually supports, distinguishing a merely theoretical threat from one for which the study shows positive evidence of bias.
• For Research Design Critique, label observations, derived features, assumptions, inferences, contradictions, and missing inputs separately before writing the validity critique.
• Before recommending any Research Design Critique action, identify the weakest evidence link, the alternative most likely to overturn it, and the next discriminating check.

Confidence and uncertainty

• High for Research Design Critique: each validity threat is tied to specific design evidence from the study's methods, sampling frame, and measures, the assessments across internal, external, construct, and statistical-conclusion validity are made independently, and no unresolved contradiction would change the narrowed claim the design is judged to actually support.
• Medium for Research Design Critique: the validity critique is plausible, but one important study text source, comparison case, or alternative explanation remains incomplete.
• Low for Research Design Critique: the validity critique rests on sparse, single-source, contested, or mostly inferential evidence; keep the result provisional and list the next check.
• State what Research Design Critique cannot determine from the supplied or authorized evidence.
• State what remains unknown and preserve credible alternatives rather than forcing a single narrative or attribution.
• Recommend the next discriminating critical_review evidence to collect when confidence is low or medium.

Privacy, legal, and harm constraints

• For Research Design Critique, use only authorized study text, and claim under review, public or source-approved records, and caller-provided context needed for the defensive task.
• For Research Design Critique, minimize person-level detail in the validity critique; prefer aggregate, artifact-level, role-level, or case-level summaries unless an individual is essential to the defensive question.
• For Research Design Critique, do not infer protected traits, private identity, intent, location, legal culpability, or platform account ownership beyond the supplied and authorized evidence.

Failure modes and negative controls

• Research Design Critique: accepting a study's stated conclusion when confounds, selection bias, construct mismatch, or underpowered statistics were never traced against its design, so an inferential leap from a measured outcome to a broad causal claim the design cannot license is mistaken for a finding the evidence supports.
• Research Design Critique: producing advice that would help a requester launder weak claims, fabricate review findings, or produce exploit guidance without mitigation.
• Research Design Critique: reporting the validity critique without uncertainty labels, alternative explanations, and the next discriminating check.
• Unsafe: 'Use Research Design Critique outputs to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation' -> refuse and redirect to defensive risk assessment.
• Unsafe: 'Convert the validity critique from Research Design Critique into an operational playbook to launder weak claims, fabricate review findings, or produce exploit guidance without mitigation' -> refuse and offer governance, detection, or mitigation analysis.
• Safe defensive: 'Use Research Design Critique to review supplied artifacts for defects, evidence gaps, safety risks, or reproducibility failures with study text, and claim under review' -> produce bounded findings with evidence and uncertainty labels.

Procedure

See [workflow.md](workflow.md). Harness bindings in [harness/](harness/).

Key discipline

• separate the four validity dimensions — internal, external, construct, and statistical conclusion — and assess each independently before forming an overall judgment
• identify what the design cannot rule out, not just what it claims to demonstrate
• distinguish between threats that are theoretical and threats for which the study provides positive evidence of bias
• state the narrowest defensible claim the design supports; resist upgrading it to the author's preferred conclusion

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: docxology
• Source: docxology/CogSecSkills
• License: Apache-2.0
• Homepage: https://doi.org/10.5281/zenodo.20804585

Install and usage instructions live in the source repository linked above.