Robustness Checker

Purpose

Design and run robustness, sensitivity, error, and baseline comparison checks for mathematical modeling contest solutions.

This skill runs the perturbation, sensitivity, error, and baseline computations that test whether model conclusions are stable enough to support paper claims. It compares baseline and main model results, perturbs key parameters or inputs, checks error behavior, identifies fragile assumptions, and records the boundary conditions under which conclusions remain valid.

The computation is AI-owned: running perturbations, computing deltas, comparing against the baseline, and producing the ≥ 5 pass-item report are this skill's job. The stability verdict itself — "is the model robust enough" and "how confident are we" — is NOT. That is a modeling judgment the human makes at the gate. This skill lays out the computed evidence plus an explicit ai_suggestion reading of it, then emits a PENDING decision artifact and STOPS. It never renders a graded stability verdict on its own.

A key output is the per-subquestion robustness report (robustness/Qx/qx_robustness_report.md), which serves the paper materials chain: the report's findings are referenced by the final result analysis, incorporated into the solution package, and ultimately written into the paper's robustness section.

This skill does not choose the original modeling route, rewrite model code broadly, fabricate experiment results, or write final paper sections.

When to use

Use this skill:

• After code-reviewer has confirmed that modeling code is runnable or has clear remaining risks.
• After baseline and main model outputs exist.
• After the final method has been selected (not during early experimentation rounds — basic error checks during experimentation are done by the code or experiment report).
• Before figure-table-planner.
• Before paper writing uses model conclusions.
• When the solution needs sensitivity analysis, baseline comparison, error analysis, or stability evidence.
• When final claims depend on parameter choices, weights, random seeds, sampling, constraints, or uncertain assumptions.

Preconditions

The following should already exist or be provided:

• A validated candidate method pool.
• Final method explanation for the target subquestion (if the final method is locked; otherwise, basic checks only).
• Reviewed modeling code.
• Run instructions.
• Baseline outputs where applicable.
• Main model outputs.
• Cleaned data or approved non-tabular inputs.
• Data audit report and code review report if available.
• Known parameters, weights, thresholds, assumptions, constraints, and random seeds.

If model outputs do not exist, hand back to code-reviewer, python-model-code-generator, or matlab-model-code-generator.

If the main model has no baseline and no justified exception, hand back to method-selector.

Inputs

Use or request:

• Candidate method pool files.
• Final method explanation files (if available).
• Reviewed scripts under code/.
• Cleaned data under workspace/data/data_clean/.
• Model outputs under results/Qx/experiments/.
• Experiment reports under results/Qx/experiments/roundN/.
• Figures under results/Qx/experiments/*/figures/.
• Code review report.
• Data audit report.
• Baseline result files.
• Main model result files.
• User-specified robustness requirements or contest scoring criteria if available.

Workflow

1. Identify claims that need support.

• Extract key conclusions from the final result analysis or experiment reports.
• Identify which outputs will likely become paper claims.
• Do not test irrelevant quantities.

1. Map checks to model type.

• Evaluation models → weight sensitivity, normalization sensitivity, ranking stability.
• Prediction models → error metrics, split sensitivity, residual checks, extrapolation limits.
• Optimization models → constraint perturbation, parameter sensitivity, feasibility checks, baseline comparison.
• Simulation models → random seed stability, trial count sensitivity, distribution summaries.
• Mechanism models → parameter sensitivity, boundary condition checks, unit or scale consistency.

1. Compare baseline and main model.

• Confirm that the main model improves, clarifies, or complements the baseline.
• If the main model does not outperform the baseline, report that honestly.
• Do not hide baseline failure or main model weakness.

1. Design robustness checks.

• Perturb key parameters, weights, thresholds, or assumptions.
• Perturb data where meaningful (noise injection, sample deletion, resampling, missing-value strategy comparison).
• Test extreme but reasonable scenarios.
• Test random seed stability when randomness exists.
• Check feasibility under changed constraints for optimization tasks.

1. Run or specify checks.

• Run checks when code and data are available.
• If execution is not possible, produce an executable robustness plan and mark it as not yet run.
• Do not invent numerical robustness results.

1. Summarize the computed evidence (facts), and offer a reading (suggestion) — do NOT render a verdict.

• The computed numbers stay AI-owned facts: perturbation deltas, baseline comparison values, error metrics, ranking changes, variance under seeds.
• You MAY group findings into a suggested stable/fragile split and state which conclusions you read as supported vs. requiring caution — but label this as your ai_suggestion, not a graded verdict.
• State the boundary conditions under which conclusions appear valid, as evidence the human will weigh.
• Do NOT auto-render an overall Status: passed / needs_caution. The overall stability verdict and the confidence level are decided by the human in step 8.5, not here.

1. Determine figure placement.

• For each robustness figure, recommend: main paper (Type 3 论文图) or appendix (Type 4 附录图).
• Criteria: if the result materially affects the main conclusion → main paper. If supplementary detail → appendix.

1. Produce per-subquestion robustness report.

• Save to robustness/Qx/qx_robustness_report.md.
• Keep the report self-contained and citable from the final result analysis and solution package.

8.5. Emit the modeler decision artifact, then STOP (Gate G4.5).

• This is the load-bearing change: whether the model is "robust enough" — and how confident we are in its stability — is a graded modeling judgment. The AI must not make it. You ran the perturbations and computed the deltas; the human reads them and commits the verdict.
• Create methods/Qx/decisions/robustness-checker_modeler_decision.md following the Human Decision Artifact Convention in CLAUDE.md, with:
• schema_version, skill: robustness-checker, scope: Qx, decision_id: qx_stability_verdict, decision_point: confidence.
• status: PENDING, decided_by: human (left for the human to confirm), decided_at left blank.
• ai_suggestion: your reading of the perturbation / sensitivity / baseline results — your single best stability verdict (high / medium / needs_caution) plus the one number that drives it. This is YOUR view, in its own field — it does not count as the decision. The computed numbers themselves (perturbation deltas, baseline comparison) stay AI-owned facts in the report, not in this field.
• choice (the human's confidence ∈ {high, medium, needs_caution}), every rejected_alternatives[*].reason, and the ## Modeler's rationale body left as >> sentinels for the modeler to fill. The human rationale MUST cite at least one specific number from the robustness report (e.g. a perturbation delta, a baseline improvement, an error metric) — you cannot trust stability without naming the perturbation result that justifies it. This evidence-citation is the strong lever.
• evidence_refs: point at the real computed numbers under robustness/Qx/ — the report, the perturbation/sensitivity/baseline CSVs and figures — so the human's rationale resolves to concrete numbers.
• Then STOP and tell the modeler exactly what to fill in. Do NOT proceed to figure-table-planner or hand the verdict downstream. Gate G4.5 (part of the results→freeze human layer) keeps the stability verdict from flowing into the solution package / freeze until this artifact's status is DECIDED with a non-empty, non-copied human rationale that cites a report number.
• In learning mode (if planning/session_config.json says so): withhold ai_suggestion until after the human writes their rationale, to avoid anchoring. In speed mode: show ai_suggestion alongside. Either way the human rationale field, its char floor, and the copy/evidence checks are identical.
• modeler-decision-logger collects this artifact into methods/Qx/qx_decision_log.md (append-only).

1. Recommend next step.

• If robustness evidence is sufficient AND the human has committed the stability verdict (decision artifact status: DECIDED), hand off to figure-table-planner.
• If model or code issues are discovered, hand back to the relevant upstream skill.

Outputs

Produce per-subquestion robustness artifacts:

• robustness/Q1/q1_robustness_report.md
• robustness/Q2/q2_robustness_report.md
• robustness/Q3/q3_robustness_report.md
• robustness/Q4/q4_robustness_report.md
• Sensitivity data files: robustness/Qx/weight_sensitivity.csv, robustness/Qx/error_metrics.csv, robustness/Qx/constraint_sensitivity.csv, etc.
• Robustness figures: robustness/Qx/figures/weight_sensitivity.png, robustness/Qx/figures/residual_plot.png, robustness/Qx/figures/baseline_comparison.png, etc.

Output format

Each robustness/Qx/qx_robustness_report.md must follow this structure:

# Qx Robustness and Sensitivity Report

## 1. Summary

- **Computed evidence (AI-owned facts)**: [one-sentence factual summary of what the perturbations / baseline comparison produced — the numbers, not a verdict]
- **AI-suggested stability reading** (`ai_suggestion`, NOT a verdict): high / medium / needs_caution — [the one number that drives this reading]
- **Verdict status**: PENDING — decided by the human in `methods/Qx/decisions/robustness-checker_modeler_decision.md` (Gate G4.5). This skill does NOT render the stability verdict.
- **Recommended next skill** (after the human decides): `figure-table-planner`

## 2. Claims Under Test

| # | Claim | Source | Importance |
|---|-------|--------|------------|
| 1 | [claim from final result analysis or experiment report] | [source artifact] | high / medium |

## 3. Baseline Comparison

### 3.1 Comparison Setup
- **Baseline**: [method name]
- **Main model**: [method name]
- **Comparison metric**: [metric name and formula]

### 3.2 Comparison Results

| Metric | Baseline | Main Model | Improvement | Meaningful? |
|--------|----------|------------|-------------|-------------|
| [metric] | [value] | [value] | [delta] | yes / no / marginal |

### 3.3 Baseline Comparison Reading (`ai_suggestion`, NOT a verdict)
- The comparison **numbers** above (baseline value, main-model value, delta) are AI-owned facts.
- Whether the improvement is "meaningful enough" is the AI's *suggested reading* — label it as such. The human commits whether this counts as a meaningful improvement in the decision artifact.
- [If the AI reads the main model as not improving, honest factual statement of the gap]

## 4. Robustness Checks

### 4.1 [Check Type: e.g., Weight Sensitivity]

- **Description**: [what was perturbed, how, and by how much]
- **Perturbation range**: [e.g., ±5%, ±10%, ±20%]
- **Status**: completed / planned / not applicable
- **Artifact**: `robustness/Qx/weight_sensitivity.csv`
- **Figure**: `robustness/Qx/figures/weight_sensitivity.png`
- **Finding**: [specific finding]
- **Conclusion stability**: stable under this perturbation / fragile / breaks
- **Figure recommendation**: main paper / appendix / not needed

### 4.2 [Next Check Type]
[same structure]

## 5. Supported Conclusions

These conclusions are supported by robustness evidence:

| # | Conclusion | Supporting Check | Confidence |
|---|-----------|-----------------|------------|
| 1 | [conclusion] | [check name + artifact] | high / medium |

## 6. Fragile Conclusions

These conclusions require caution or qualification:

| # | Conclusion | Why Fragile | Boundary Condition | Recommended Qualification |
|---|-----------|-------------|-------------------|--------------------------|
| 1 | [conclusion] | [which check revealed fragility] | [under what conditions it holds] | [how to qualify in the paper] |

## 7. Conclusion Boundaries

For each major conclusion, state the conditions under which it remains valid:

| Conclusion | Valid When | May Fail When | Paper Guidance |
|-----------|------------|--------------|----------------|
| [conclusion] | [conditions] | [conditions] | [how to frame in paper] |

## 8. Figure Placement Recommendations

| Figure | Content | Recommended Placement | Reason |
|--------|---------|----------------------|--------|
| `robustness/Qx/figures/weight_sensitivity.png` | Weight perturbation effect on ranking | Main paper (Type 3) | Directly supports key claim about ranking stability |
| `robustness/Qx/figures/full_sensitivity_curves.png` | All indicators, full perturbation | Appendix (Type 4) | Supplementary detail |

## 9. Remaining Risks

| Risk | Impact | Mitigation |
|------|--------|------------|
| [risk not covered by current checks] | [potential impact on conclusions] | [how to address or acknowledge in paper] |

## 10. Generated Artifacts

- `robustness/Qx/qx_robustness_report.md`
- `robustness/Qx/weight_sensitivity.csv`
- `robustness/Qx/figures/weight_sensitivity.png`
- `methods/Qx/decisions/robustness-checker_modeler_decision.md` (PENDING — the human commits the stability verdict here at Gate G4.5)

## 11. Handoff

- **Stability verdict**: PENDING — awaiting the human in `methods/Qx/decisions/robustness-checker_modeler_decision.md` (Gate G4.5).
- **Next skill** (after the human decides): `figure-table-planner`
- **AI-suggested supported conclusions** (`ai_suggestion`, for the human to weigh): [list]
- **AI-suggested fragile conclusions** (`ai_suggestion`, for the human to weigh): [list]

Plus, separately, the decision artifact methods/Qx/decisions/robustness-checker_modeler_decision.md (PENDING, awaiting the human) — see workflow step 8.5.

Robustness check types

Baseline comparison

Use when: a baseline and main model both exist; a paper claim says the main model is better.

Check:

• Whether outputs differ materially.
• Whether improvement is meaningful.
• Whether the main model adds interpretability, accuracy, feasibility, or stability.
• Whether the baseline already solves the task sufficiently.

Parameter sensitivity

Use when: results depend on parameters, thresholds, coefficients, decay rates, penalties, or hyperparameters.

Check:

• Small perturbations (±5%, ±10%, ±20%).
• Monotonicity of output changes.
• Tipping points where conclusions change.
• Parameters that dominate results.

Weight sensitivity

Use when: evaluation, multi-objective optimization, or weighted scoring is used.

Check:

• Weight perturbation.
• Alternative weighting schemes.
• Ranking stability (do top-K alternatives stay the same?).
• Score stability.

Data perturbation

Use when: results may depend on noisy, incomplete, or uncertain data.

Check:

• Noise injection.
• Missing-value strategy comparison.
• Sample deletion.
• Bootstrap or resampling if appropriate.
• Effect of outliers.

Error analysis

Use when: prediction, fitting, classification, or estimation is used.

Check:

• MAE, RMSE, MAPE, accuracy, F1, residuals, confusion matrix.
• Train-test split or cross-validation.
• Error distribution.
• Failure cases.
• Extrapolation limits.

Constraint perturbation

Use when: optimization constraints affect feasibility or final decisions.

Check:

• Resource limits.
• Budget/capacity

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: zhnnky329
• Source: zhnnky329/MathModeling-skills
• License: MIT

Install and usage instructions live in the source repository linked above.