Stop Chasing The Optimizer Reduce Instead

Stop chasing the optimizer; reduce the repro instead

When to use

You have a hot function that passes on one toolchain (typically MSVC) but miscompiles or crashes under GCC / Clang at -O2 (or higher), and you're on attempt three of sprinkling anti-optimization decorations:

• volatile on a local / a parameter
• __attribute__((noinline)) on the callee
• __attribute__((optimize("O0"))) on the callee
• -fno-strict-aliasing on the TU
• asm volatile("" ::: "memory") barriers

Each one "almost works" — the symptom moves by a few bytes or into a different test case — but never fully goes away. That's the signal to stop.

Problem

Adding anti-optimization pragmas is a local fix applied to a global belief: "the optimizer is wrong." Three things go wrong in practice:

1. The decorations change the bug, not the cause. noinline in one

spot shifts the inlining decision elsewhere; the miscompile now lives in a different frame. You spend another iteration finding it.

1. Some decorations are themselves traps. __attribute__((optimize("O0")))

on a single function inside an -O2 TU is known to produce ABI mismatches between the -O0 callee and the -O2 caller on GCC (frame pointer, red-zone, stack alignment). The "fix" introduces a new SIGSEGV on the first call.

1. **You never learn whether it's a compiler bug, your UB, or your

aliasing.** All three present identically — "works on MSVC, breaks on GCC -O2" — and have very different fixes. Patching blindly leaves the root cause unknown, so the same bug returns the next time someone touches the file.

The heuristic: after two anti-optimization patches in a row have failed to fully fix the symptom, the next step is not a third patch. It's reduction.

Solution

Switch from "patch" mode to "reduce" mode:

1. Extract a standalone repro. Rip the suspect function out of the

project into a single .cpp of ≤ 200 lines that links with nothing but libc. Must reproduce the divergence between MSVC and GCC/Clang -O2. If it doesn't reproduce standalone, the bug is in how the function is called, not the function itself — go up a frame.

1. Diff the codegen. g++ -O2 -S -masm=intel repro.cpp vs

clang++ -O2 -S -masm=intel repro.cpp vs MSVC /FAs. Look for loads from offsets you never wrote, or stores that the compiler elided. This usually tells you within minutes whether it's UB (compiler is within its rights) or a real miscompile.

1. Shrink with creduce / cvise. Feed the standalone repro plus

a predicate script (g++ -O2 x.cpp && ./a.out; [ $? -ne 0 ]) to cvise. 200 lines typically collapses to 20.

1. Decide once, fix once.

• UB in your code → fix the UB (e.g. replace x >> 64 with a

branch, use memcpy instead of pointer-punning, add an explicit bounds check). No volatile needed.

• Real compiler bug → file upstream with the reduced repro, then

put one narrow workaround (guarded by compiler-version macros) with a link to the bug report.

• Aliasing assumption → use memcpy or __attribute__((may_alias))

at the type, not -fno-strict-aliasing on the whole TU.

The key is that step 4 is a single, documented change, not another round of sprinkling.

Example

Real case (the one this skill came from): a bit-blit routine appBitsCpyFast passed all MSVC tests but produced zeros on Linux GCC -O2. Sequence that didn't work, in order:

// Attempt 1: mark the output volatile at call sites. Symptom moves.
// Attempt 2: __attribute__((noinline)) on the inner helper. Passes
//            aligned cases, still fails unaligned.
// Attempt 3: __attribute__((optimize("O0"))) on the helper.
//            Now SIGSEGVs on the first call (ABI mismatch). Worse.

At this point the right move is not attempt 4 (yet another decoration). It is to switch modes: stop patching, start reducing. The concrete recipe:

# 1. Rip the function into a standalone repro.cpp that links only
#    against libc and still reproduces the MSVC-vs-GCC divergence.

# 2. Write a predicate script that exits 0 iff the bug reproduces.
cat > check.sh  Status note: on the project that inspired this skill, the reduction
> step above has not yet been performed at the time of writing. The
> skill is deliberately published before the fix lands, because the
> lesson ("after two failed decorations, reduce") is independent of
> which root cause reduction eventually uncovers. If you want the
> specific root cause, check the project's issue tracker rather than
> trusting a plausible-sounding example in a skill file.

## Pitfalls

- **`__attribute__((optimize("O0")))` on a single function inside an
  `-O2` TU on GCC can crash.** The `-O0` callee and `-O2` caller
  disagree on frame-pointer / red-zone / alignment convention. If you
  really need a function at lower optimization, put it in its own TU
  and compile that whole TU at `-O1` (not `-O0`) via the build system.
- **"Works on MSVC" is weak portability evidence.** MSVC tends to fold
  undefined shifts, undefined unions, and out-of-range enum values
  into "the obvious answer". GCC and Clang exploit them. Do not use
  MSVC green as a justification to ship.
- **`-fno-strict-aliasing` is a TU-wide sledgehammer.** If you reach
  for it as a fix, first check whether a targeted `memcpy` (or
  `std::bit_cast` in C++20) expresses what you meant. The sledgehammer
  also disables legitimate optimizations for every other function in
  the file.
- **The optimizer is almost never "wrong."** In ~20 years of shipping
  C++, roughly 1 in 50 "GCC miscompile" reports survives reduction.
  Assume UB until reduction says otherwise.
- If `cvise` is unavailable, a manual binary-chop of the function body
  (delete the second half, does it still fail?) gets you 80% of the
  way in 15 minutes.

## See also

- [detect-tool-vendor-by-query](../detect-tool-vendor-by-query/SKILL.md)
  — another "don't guess the compiler, query it" lesson.
- [doc-code-consistency-check](../doc-code-consistency-check/SKILL.md)
  — the mirror-image pitfall: patching the doc instead of the code.
- `cvise` project: 
- `creduce` project:

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [chen3feng](https://github.com/chen3feng)
- **Source:** [chen3feng/agent-skills](https://github.com/chen3feng/agent-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.