Reviewing Ai Generated Code

Reviewing AI-generated code: fluent is not correct

Model-written code reads well, which is exactly the risk: it is optimized for plausibility, and a reviewer's guard drops when the code is clean and confident. The security failures cluster in predictable places, dependencies that may not exist or may be attacker-registered, defaults copied from insecure examples, validation quietly omitted, and logic that looks right but is not. Reviewing it means aiming at those clusters, not skimming for style.

When to use

• You are reviewing an AI-authored change, an assistant's suggestion, or a large

generated diff.

• Generated infrastructure, config, or access-control code is entering the codebase.
• You are setting a review bar for machine-assisted contributions.

Scope check

Review code for projects you own or contribute to with authorization. If you can't name the authorization, stop.

The loop

1. Verify every dependency the code introduces. For each package the change adds,

confirm it exists, is the established package (not a lookalike or a name the model may have invented), and is the one you intend. A hallucinated package name an attacker later registers turns "the model suggested it" into installed attacker code. Do not let a plausible import in unverified.

1. Check the defaults and the omissions. Generated code tends to reproduce the

most common pattern, which is often the insecure-by-default one: permissive cross-origin rules, disabled verification, a broad permission, a missing auth check, a hardcoded or example secret, an unparameterized query. Read for what should be there and is not, not only for what is.

1. Test the claim against the behavior. Model code often states an intent in a

comment or a name the code does not fulfill: a function named validate that does not reject, a check that is computed and then ignored, error handling that swallows and continues. Confirm the security-relevant logic actually does what its surface promises.

1. Look for propagated vulnerable patterns. If the prompt or the surrounding code

contained an insecure idiom, the model likely extended it consistently. One confirmed weak pattern is a reason to sweep the whole diff for its siblings, the same way you would hunt variants of any bug.

1. Check the permissions and scope the code assigns. Generated infrastructure,

config, and access code frequently over-grants: a wildcard role, a public bucket, an all-origins rule, a token with more scope than used. Treat every permission the code sets as a claim to minimize, not to accept.

1. Adjudicate and record. Confirm each issue against the real behavior and the

real dependency, exactly as you would a hand-written finding; fluency is not evidence. Record confirmed issues with the minimal fix, and note patterns worth a broader sweep, in the schema.

Where generated code fails

• It optimizes for plausible, not correct. The failure mode is confident

wrongness, which defeats a skim.

• Dependencies are unverified input. A suggested package name is a claim until

you confirm it resolves to the real thing.

• Insecure defaults are the training-data mean. The most common example online

becomes the generated default.

• Names lie more than usual. A reassuring function name is not evidence the

function is safe.

Worked example (a confirm and a kill)

> Confirm. A generated data-access change adds an import for a package whose name > is a near-match to a popular one but is not the established package, and configures a > client with certificate verification disabled "for simplicity." The import is > unverified and the default is insecure. Confirmed, high, remediation = replace > with the verified real dependency, enable verification, and sweep the diff for the > same disabled-verification pattern. > > Kill. A generated function adds input parsing; review confirms every dependency > it uses is the intended, established package, the query is parameterized, validation > rejects the documented bad inputs (tested), and no permission is widened. It does > what its name claims. Killed, kill_reason = "dependencies verified, validation > confirmed by test, no insecure default or over-grant introduced."

Rationalizations to reject

• "The model is good, the code looks clean." → Clean is the risk. It is optimized

for looking right. Verify behavior.

• "It imported a package, so the package exists." → Not necessarily, and not

necessarily the real one. Confirm the name.

• "The comment says it validates." → The comment is generated too. Test that the

code rejects what it should.

• "It compiled and the tests passed." → Existing tests rarely cover the security

default or the missing check. Read for the omission.

Executing this in practice

You need the generated diff, the ability to resolve and verify each new dependency against a known-good source, and a way to test that the security-relevant logic behaves as named. A call graph over the change helps you find every site of a propagated pattern; the dependency verification and the claim-versus-behavior check are the method.

Related

• hunting-supply-chain-risks - the hallucinated-dependency and slopsquatting risk

this review catches at the source.

• hunting-bug-variants - sweeping the diff for siblings of one confirmed weak

pattern.

• adjudicating-taint-paths - confirming whether a missing validation is actually

reachable by untrusted input.

• [FINDING-SCHEMA.md](../../FINDING-SCHEMA.md) - source = the generated construct

(dependency, default, missing check), sink = the behavior it enables.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: UnboundCompute
• Source: UnboundCompute/security-agent-skills
• License: MIT
• Homepage: https://security.unboundcompute.com

Install and usage instructions live in the source repository linked above.