Mer Paper Reproduction

MER Paper Reproduction Skill

This skill turns a single MER paper PDF into a working, benchmarked PyTorch reproduction. It is standalone — the generated project does not depend on any platform code, and the skill itself can be driven equally well by Claude Code or Codex CLI.

The skill enforces a strict workflow:

1. Extract the paper into a structured fingerprint (≥ 3 verification passes).
2. Search GitHub for the authors' official implementation.
3. Plan → generate → review the PyTorch project.
4. Fabricate per-dataset features.
5. Run the canonical 13-experiment matrix on a GPU box.
6. Aggregate results, check quality gates, optionally tune.
7. Emit a reproduction report.

The skill never lets the LLM "improve" or guess what the paper does not say. Every code path is checked against the fingerprint by gate_fidelity_code, and every code-generation cycle ends in a /codex-review (or codex review --uncommitted under Codex CLI).

---

When to Activate

Activate this skill when the user:

• Hands you a MER paper PDF and asks you to reproduce it.
• Mentions "13 experiments / 7 datasets" reproduction in the MER domain.
• Invokes /mer-reproduce or runs mer-reproduce from a shell.

Do not activate this skill for non-MER tasks, for ablation studies on existing models, or for plotting / paper-writing tasks.

---

Prerequisites

Hard requirements (checks/preflight.py enforces these):

• Python ≥ 3.10
• torch, torchvision, dlib, opencv-python, pandas, numpy

(the mer-skill[runtime] extras)

• At least one CUDA-capable GPU visible to nvidia-smi
• ≥ 200 GB free on the user-supplied output directory
• ≥ 1 dataset path provided (the skill will skip experiments whose

datasets are missing rather than fail)

• dlib 68-point landmark model on disk (skill prompts to download

it on first use)

Soft requirements (the skill nags if missing but does not block):

• gh CLI logged in for GitHub search
• codex CLI installed for the Codex driver path

---

Domain Knowledge (hardcoded in mer_skill/)

The 13 experiments

The canonical matrix is exactly:

| # | Dataset | Classification | Protocol | Quality gate (UF1) | |---:|---------|----------------|----------------------|-------------------:| | 1 | CASME2 | 5-class default | LOSO (26 subjects) | ≥ 0.60 | | 2 | SAMM | 5-class default | LOSO (27 subjects) | ≥ 0.60 | | 3 | SMIC | 3-class default | LOSO (16 subjects) | ≥ 0.70 | | 4 | CASME3 | 5-class default | LOSO (~90 subjects) | ≥ 0.60 | | 5 | MMEW | 5-class default | LOSO (30 subjects) | ≥ 0.60 | | 6 | 4DME | 5-class single-label default | LOSO (~44 subjects) | ≥ 0.60 | | 7 | DFME | 7-class default | Fixed train/testA/testB | ≥ 0.45 | | 8 | CASME2 | 3-class | LOSO | ≥ 0.70 | | 9 | SAMM | 3-class | LOSO | ≥ 0.70 | | 10 | CASME3 | 3-class | LOSO | ≥ 0.70 | | 11 | MMEW | 3-class | LOSO | ≥ 0.70 | | 12 | 4DME | 3-class | LOSO | ≥ 0.70 | | 13 | DFME | 3-class | Fixed train/testA/testB | ≥ 0.70 |

SMIC has no separate 3-class row because its default is already 3-class.

3-class remap convention

(Hardcoded in mer_skill/datasets/class_mappings.py.)

• Drop others / other.
• happy / happiness → positive.
• surprise → surprise.
• Every remaining negative emotion (anger, contempt, disgust, fear,

sadness, repression) → negative.

• 4DME additionally drops every multi-label sample (label contains +).

---

Implementation status

As of v0.1.0 every phase below is implemented end-to-end and covered by pytest tests/. The CLI subcommands listed under each phase are real — there are no not yet implemented stubs left in mer_skill.cli.

| Phase | Subcommands | Gate | Status | |------:|-------------|------|:------:| | 0 | preflight init status matrix phases | — | ✅ | | 1 | extract --pass {A,B,C,merge} | gate fidelity-paper | ✅ | | 2 | hunt-github | — | ✅ | | 3 | plan generate review | gate fidelity-code | ✅ | | 4 | fabricate | — | ✅ | | 5 | run | — | ✅ | | 6 | collect | gate artifact + gate quality | ✅ | | 7 | tune (10-attempt global budget) | — | ✅ | | 8 | report | — | ✅ |

A second entry point — mer-reproduce-gate — exposes the gate group at the top level for CI integration.

---

State

Every run writes to {output_dir}/state.json. The dispatcher, generator, and reviewer all read and update the same file via mer_skill.experiments.state.update_state (atomic + locked). If the agent is interrupted, the next invocation reads state.json and resumes from the first phase whose status is not completed.

Never delete state.json — that is how the skill knows what is already done.

---

Workflow

The phase numbers and statuses are mirrored 1:1 in PIPELINE_PHASES / PHASE_STATUSES in mer_skill/experiments/state.py. Each phase ends in an exit gate; the agent only proceeds when the gate is green.

Phase 0 — Preflight

Goal: fail fast.

Inputs: nothing yet — the user just invoked the skill.

Action:

1. Run python -m mer_skill.cli preflight. It checks Python /

torch / dlib / nvidia-smi / disk space.

1. Ask the user (using AskUserQuestion under Claude Code, or the

prompt-toolkit prompts under Codex CLI):

• Path to the paper PDF.
• Path to each of the 7 datasets (any can be omitted; missing

datasets are skipped, not failed).

• Output root directory for everything the skill will write.
• GPU list (default: probe nvidia-smi).
• Slots per GPU (default: 4 — empirically stable on a 32-core /

2× RTX 3090 machine).

1. Initialize state.json with mer_skill.experiments.state.init_state.

Exit gate: preflight script exit 0 + state.json present.

Failure recovery: print the failing check verbatim and stop. Do not auto-fix environment problems.

---

Phase 1 — PDF extraction with 3-pass cross-verification

Goal: a fingerprint that is demonstrably faithful to the paper. The user requirement is verbatim: "至少循环核对 3 次，确保 真实复现，而不是胡乱改造". This phase implements that.

The three passes are deliberately orthogonal — re-reading the same section three times catches nothing.

Pass A — Structure (mer_skill/pdf/pass_structure.py)

Extract: table of contents, section boundaries, every numbered equation (preserve LaTeX where possible), every figure / table caption. Output: pass_a.yaml.

Under Claude Code: use the Read tool with pages to walk the PDF. Under Codex CLI: shell out to pdftotext + pdfplumber.

Pass B — Method (mer_skill/pdf/pass_method.py)

Re-read the method section only. For every layer / module: input shape, output shape, activation, initialization, losses (with formulas), optimizer, LR schedule, augmentations, preprocessing pipeline (face alignment, crop size, normalization). Output: pass_b.yaml.

Pass C — Experiments (mer_skill/pdf/pass_experiments.py)

Re-read the experiments section + appendices. Per dataset: validation protocol (LOSO / fixed), class mapping, reported UF1 / UAR, epochs, batch size, seeds. Output: pass_c.yaml.

Conflict detection (mer_skill/pdf/conflicts.py)

Cross-check the three passes:

• Every tensor symbol that appears in a Pass A equation must appear

in a Pass B layer name.

• Pass B's optimizer / epochs must match Pass C's training protocol.
• Every reported number in Pass C must be traceable to a Pass A

table reference.

Any mismatch → write to fingerprint.conflicts.yaml.

User adjudication

If fingerprint.conflicts.yaml is non-empty, stop and ask the user which version to keep. Use AskUserQuestion with one question per conflict — never silently pick one.

Merge the three passes + adjudications into fingerprint.yaml.

Exit gate: python -m mer_skill.cli gate fidelity-paper exits 0 (fingerprint complete + zero unresolved conflicts).

Failure recovery: if a field cannot be resolved even after user input, append it to fingerprint.unresolved.yaml and stop. Never let the LLM fill in a missing value.

---

Phase 2 — GitHub official-implementation hunt

Goal: find authoritative code before writing any.

Action:

mer-reproduce hunt-github --fingerprint fingerprint.yaml \
                          --out github_candidates.yaml

The hunter uses paper title / authors / arxiv id (whichever the fingerprint records). Under Claude Code it can also use the mcp__github__search_code and mcp__github__search_repositories tools; under Codex CLI it falls back to gh search.

Candidates are scored on (author match × star count × recent commit × license compatibility). Top 3 are written to github_candidates.yaml.

Ask the user which candidate (if any) to adopt. Three options:

(a) Fork-style port of the chosen candidate. (b) Hand-write from the fingerprint only. (c) Hybrid — use parts of the candidate as a template, write the rest.

Record the decision in state.implementation_path.

Exit gate: state.github_candidate populated (or set to "" if the user chose hand-written).

---

Phase 3 — Plan → Implement → Review

Goal: a self-contained PyTorch project that faithfully reproduces the paper.

3.1 Plan

The agent reads fingerprint.yaml and the chosen GitHub candidate (if any), then writes a plan.md describing every file it intends to generate and how it maps to fingerprint fields. The user confirms (AskUserQuestion) before any code is written.

3.2 Implement

Each generator under mer_skill/codegen/ fills a Jinja2 template under templates/code_templates/. The generated layout under the user's output directory is:

{output_dir}/{paper_short_name}/
├── model.py             # the paper's nn.Module
├── dataset.py           # per input_type loader
├── fabricate.py         # face align + crop + feature extraction
├── train.py             # LOSO loop + DFME 2-phase + spawn DataLoader
├── configs/
│   ├── {paper}_casme2_5class.yaml
│   ├── {paper}_casme2_3class.yaml
│   ├── ... (13 total)
│   └── README.md
├── run_all.sh
├── requirements.txt
└── README.md

The templates have already absorbed every relevant OpenAffective "踩坑教训" (test set drop_last guard, spawn DataLoader context, fold resource cleanup, DFME split_name propagation, stdout=DEVNULL on subprocess, accuracy as LOSO best-epoch metric, early stop at 100% fold accuracy). Generators must not turn these guards off.

3.3 Review

mer-reproduce review --files {generated files}

Under Claude Code this delegates to /codex-review. Under Codex CLI it calls codex review --uncommitted directly. The wrapper parses Codex's [P1] / [P2] / [P3] priorities. P1 issues must be fixed before exiting Phase 3. P2 issues are escalated to the user.

After every fix, also re-run gate_fidelity_code — it greps each fingerprint field (layer name, loss name, optimizer name) against the generated code. Fields missing from the code are P1.

Loop 3.2 → 3.3 up to 3 times. If still failing after 3 cycles, stop and ask the user for guidance.

Exit gate: fidelity-code green + review verdict PASS or "only P3 issues remaining".

---

Phase 4 — Data fabrication

Goal: turn each user-supplied raw dataset into the paper-specific input tensors (face-aligned, cropped, optical-flow or whatever the fingerprint declares).

Action: invoke the generated fabricate.py once per (dataset, classification) combination the user has data for:

python {output_dir}/{paper}/fabricate.py \
    --dataset CASME2 --classification 5class \
    --root {user_path} --out fabricated/ \
    --landmarks /path/to/shape_predictor_68_face_landmarks.dat

Subprocess is launched with stdout=DEVNULL (PIPE deadlock lesson from tmp/dfgm_comparison.py). After each job, validate the generated index_*.csv: row count > 0, no zero-population class.

Exit gate: every required fabricated/.../index_*.csv exists and validates.

Failure recovery: failed datasets are auto-retried once, then marked skipped so the dispatcher will not launch their experiments.

---

Phase 5 — Run the 13 experiments

Goal: drive the matrix to completion within the GPU budget.

mer-reproduce run --output-dir {output_dir} --gpus 0,1 --slots-per-gpu 4

This invokes mer_skill.experiments.dispatcher.Dispatcher. The dispatcher:

• Reads state.experiments and submits all pending rows.
• Pins each child to one GPU via CUDA_VISIBLE_DEVICES.
• Captures stdout/stderr to {output_dir}/logs/{key}.log (NEVER a

PIPE).

• After each child exits, reads {output_dir}/results/{key}.json

and writes pooled UF1 / UAR / accuracy back to state.json.

• Persists state on every transition; restarts pick up where they

left off.

Run the dispatcher in the background (Bash run_in_background=true under Claude Code; nohup under Codex CLI). The agent re-checks status every 10 minutes by calling mer-reproduce status.

Exit gate: python -m mer_skill.cli gate artifact confirms every planned experiment is in completed or skipped state.

---

Phase 6 — Result aggregation + quality gate

Goal: emit results_table.md, decide whether tuning is needed.

mer-reproduce collect --output-dir {output_dir} \
                      --out {output_dir}/results_table.md
mer-reproduce gate quality --output-dir {output_dir}

The collector reads each results/{key}.json, joins with fingerprint.datasets[*].reported_uf1, and renders a markdown table with a Gap column. The quality gate emits failures.yaml listing every (dataset, classification, current_uf1, target, gap) row that missed its threshold.

Exit gate: table generated. (The quality gate may fail; that just routes us to Phase 7.)

---

Phase 7 — Tuning (conditional)

Trigger: failures.yaml is non-empty.

Budget: 10 attempts globally, period. The user explicitly chose this in the design phase.

Strategy (mer_skill/tuning/strategy.py):

Sort failures by gap × dataset_weight (heavier failures first). For each, generate up to 4 candidate hyperparameter overrides:

1. lr × 0.5 / lr × 2
2. batchsize × 0.5 / batchsize × 2
3. epochs × 1.5
4. augmentation flags toggled

Submit through the same dispatcher with experiment_name += "_tuned_v{N}". Stop after 10 total submissions, regardless of how many failures remain.

After exhausting the budget: run mer_skill/tuning/pattern_analysis.py. It writes tuning_report.md summarizing per-dataset hyperparameter sensitivity and recommends the most-robust configuration. Then ask the user:

• (a) Accept current results, proceed to final report.
• (b) Re-run all 13 experiments with the recommended config.
• (c) Specify custom overrides and re-run.

This branch is the user's call — never auto-decide.

---

Phase 8 — Final report

Goal: hand the user a complete, self-contained reproduction package.

mer-reproduce report --output-dir {output_dir} produces:

• {output_dir}/REPRODUCTION_REPORT.md containing:
• paper metadata + reproduction date
• full fingerprint.yaml as an appendix
• decision log (each phase's user adjudication)
• results table + gap-vs-paper analysis
• tuning record (if Phase 7 ran)
• {output_dir}/{paper_short_name}/ is left exactly as is — the

user can git init it as a standalone reproduction project.

Run /codex-review one last time on the final code (not on the report).

Exit gate: report file exists and the review is PASS.

---

Harness Compatibility

| Capability | Claude Code path | Codex CLI path | |--------------------------|----------------------------------------------------------|-----------------------------------------------------| | User questions | AskUserQuestion | prompt_toolkit interactive prompt in `

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: LanEinstein
• Source: LanEinstein/MER-Reimplementation-Skill
• License: MIT
• Homepage: https://github.com/LanEinstein/MER-Reimplementation-Skill

Install and usage instructions live in the source repository linked above.