Ai Check

AI-Check Skill

Forensic analysis of text for AI-generation signals. Grounded in the published detection literature (Wu et al. 2025, Mitchell et al. 2023, Kujur 2025, AAAI 2025 shared task).

The output is a structured report, not a vague judgment. Every fired signal cites evidence.

---

The nine signal categories

Score each category 0–3:

• 0 = No signal detected (human-consistent)
• 1 = Weak signal (possible AI, could be human)
• 2 = Moderate signal (likely AI pattern)
• 3 = Strong signal (near-certain AI pattern)

Severity-to-score mapping (use for every category):

| Evidence in category | Score | |---|---| | No flagged instances | 0 | | One weak instance, or vague unease without a specific quote | 1 | | One moderate instance, or two or more weak instances | 2 | | One strong instance, or two or more moderate instances, or four or more weak instances | 3 |

Double-counting policy: a single phrase can fire at most two distinct signals when the phrase is genuinely diagnostic for both. Example: "it is important to note that" is both Signal A (banned vocabulary) and Signal C (institutional hedge). Log it under both, but the same phrase cannot count as two separate weak instances inside the same category.

Total score cap: 9 categories × 3 = 27 maximum.

Signal A: Perplexity (word predictability)

Look for vocabulary that is maximally safe and expected — words that are technically correct but never the most precise or interesting choice a knowledgeable human would make.

Flags:

• Generic verbs where domain-specific ones belong ("address" instead of "untangle", "implement" instead of "wire up")
• Adjectives that describe without adding information ("significant improvements", "notable progress", "key challenges")
• Hedged assertions that swap specificity for safety ("can often lead to", "may result in", "tends to")
• Any of the canonical AI vocabulary list:

delve, leverage (verb), utilize, robust, comprehensive, streamline, foster, facilitate, pivotal, nuanced, notable, notably, enduring, garner, it is worth noting, it is important to note, multifaceted, in the realm of, the landscape of, a myriad of, a plethora of

Cite the exact word or phrase that fired.

Signal B: Burstiness deficit (sentence uniformity)

Measure the variation in sentence length across the text.

Flags:

• Three or more consecutive sentences within 5 words of the same length
• No sentence shorter than 8 words in any 150-word block
• Metronomic rhythm — reading the passage aloud produces a steady pulse rather than natural variation
• No fragments used for emphasis

Report: list the sentence lengths in sequence (e.g. "14, 16, 13, 15, 17 — five consecutive sentences within 4 words of each other").

Signal C: Hedge density

Count the softening and epistemic hedge words.

Flags:

• "often", "generally", "typically", "in many cases", "it can be argued" appearing where direct assertion is warranted
• "it is important to note that", "it is worth mentioning", "one might consider"
• Diplomatic framing of obvious tradeoffs: "while X has benefits, it also presents challenges"
• Uncertainty expressed as institutional hedging rather than personal ("results may vary") vs human ("I'm not sure this holds when...")

Report: quote each hedge and note whether it was warranted by genuine uncertainty or reflexive softening.

Signal D: Structural tells

Look for document architecture patterns AI imposes regardless of content.

Flags:

• Bullet list where prose would serve better
• Topic sentence + evidence + restatement of topic sentence (humans skip the restatement)
• "In conclusion / To summarize / In summary" openers on closing paragraphs
• "In this [post/article/section] I will..." openers
• Numbered steps for content that isn't genuinely sequential
• Three-part structure imposed on every paragraph (intro, body, conclusion at micro-scale)
• Tricolon parallel structure: three examples or beats with identical grammatical shape

e.g. "You X. Y. Does Z? You X. Y. Does Z? You X. Y. Does Z?" — perfectly symmetrical triplets in prose are AI-constructed. Real writers use two examples or vary the shape. Severity: strong.

• Perfect paragraph-per-idea arc: every paragraph does exactly one narrative job and

advances the arc cleanly (setup → tension → lesson → evidence → reflection). Real personal writing has a paragraph that meanders, does two jobs, or doesn't fully resolve. A piece where every paragraph lands cleanly is architecturally perfect in a way human writing isn't. Severity: moderate in isolation, strong combined with other signals.

• Three-act Slack/update structure: for informal async messages, accomplishment → caveat

→ next steps maps directly to intro/body/conclusion. Real updates loop back, add a mid-message second thought, or end with something that doesn't fit the structure.

• Strawman pivot: "The case for X isn't about Y, it's about Z" / "It's not about X,

it's about Y." Leading with what something is NOT before saying what it IS. Real writers lead with the actual point. Severity: moderate.

Signal E: Specificity deficit

Measure whether claims are grounded in concrete detail.

Flags:

• Abstract claim with no number, name, time reference, or example: "Many organizations have adopted..."
• Passive constructions obscuring the actor: "it has been found that", "research suggests"
• Universalist framing: "teams often find", "developers frequently encounter" (applicable to everyone, specific to no one)
• Named examples that are suspiciously generic or perfectly illustrative (AI picks canonical examples: "Netflix", "Amazon", "Stripe" without context)

Report: quote each unanchored claim.

Signal F: Transition word fingerprint

Catalog the connective tissue between sentences and paragraphs.

Flags (strong AI signals):

• "Furthermore," as paragraph opener
• "Moreover," as paragraph opener
• "Additionally," as paragraph opener
• "It is clear that"
• "This highlights / underscores / demonstrates the importance of"
• "As previously mentioned"
• "In addition to the above"
• "It goes without saying"
• "Needless to say"

Flags (moderate signals):

• "However," used more than once per 200 words
• "Therefore," used as a mechanical logical connector rather than earned conclusion
• "Turns out" / "it turns out that" as a pivot or reveal. AI uses this to create

the illusion of a discovery narrative. "Turns out the config had a lower timeout" → "The config had a lower timeout." Quote each instance. Severity: moderate.

• Tutorial-voice transitions: "The standard fix is...", "The common approach is...",

"Simple enough on paper" — these frame what follows as received wisdom, not personal experience. Strong signal in technical writing.

• Announcement-colon patterns: "The rule I use:", "The key insight:", "The approach

here:", "The other thing I'd say:" — announcing before revealing. Severity: moderate. Also fires without a colon: "What I didn't expect was...", "What surprised me was...", "The thing I realized was..." — these are announcement sentences even without the colon. The colon isn't the tell; the announcement structure is.

• Pattern announcement: stating that a pattern exists before describing it.

"The pattern is almost always the same" followed by the pattern. Real writers just describe the pattern.

Signal G: Punctuation fingerprint

Count the three AI punctuation tells:

Em dashes: Count total em dashes. More than 1 per 300 words is a signal. Specific sub-patterns:

• Double em dash wrapping (— like this —) is a near-certain AI pattern
• Em dash as pivot ("not mid-sprint — and the on-call rotation") — list-joiner em dash

connecting two items within a sentence

• Em dash as dramatic aside ("X — which is worth noting — Y")

Report exact count, location, and which sub-pattern.

Semicolons: Any semicolon linking two independent clauses in non-academic prose is a flag. Report exact count. Exception: comma-containing lists ("Austin, TX; Denver, CO").

Mid-sentence colons: A colon preceded by an incomplete clause ("The problem: nobody tests this" / "The answer: start earlier") is an AI structural pattern. Report each instance.

Signal H: Voice and register

Look for absence of human traces.

Flags:

• No first-person perspective anywhere in a piece where first-person would be natural
• No second-person direct address in instructional or opinionated content
• Consistent "polished neutral tone" — no personality variance, no roughness, no informality spikes
• No rhetorical questions used as transitions
• No self-correction or mid-thought qualification ("actually, that's not quite right")
• Opening sentence is a thesis, definition, or contextual framing rather than mid-thought or scene

Register collapse (Slack / informal writing): The most commonly missed signal in casual-register text. AI writes Slack messages that read like polished status reports with informal markers sprinkled in. Look for:

• Complete, well-formed sentences throughout — real Slack has fragments
• Topic-per-paragraph structure even in a short message
• Formal vocabulary underneath casual markers (~60% and lmk but the sentences

themselves are well-constructed prose)

• No self-corrections mid-message ("oh also. just realized...")
• Three-act arc (accomplishment / caveat / next steps) intact beneath the informality
• Numbers written as words ("three incidents") rather than numerals with approximations

("~3 incidents", " 0, list each specific instance with a short quote or description. Format: SIGNAL-[LETTER] | "[exact quote or pattern description]" | severity: weak/moderate/strong]

WHAT GAVE IT AWAY ----------------- [2–4 sentences identifying the strongest signals in plain language. Be specific about which phrases, patterns, or absences were most diagnostic. This section is written for a human who wants to understand the tell, not just see a score.]

RECOMMENDED FIXES ----------------- [Only present if score > 6. Concrete rewrites or changes for the top 3 signals.]

---

## Scoring thresholds

| Total score | Verdict |
|---|---|
| 0–4 | Human |
| 5–8 | Likely Human |
| 9–13 | Uncertain |
| 14–19 | Likely AI |
| 20–27 | AI |

## Calibration notes

- Short texts (90% on
  direct AI text fail catastrophically on text that has been iteratively paraphrased through one or
  more LLMs. If the user mentions the text was paraphrased or rewritten, down-weight all signals.
- **Stylistic cues are corpus-conditional.** SHAP-based explainability analysis (arXiv 2603.23146) shows that
  surface stylistic features detectors rely on are dataset-specific, not stable authorship signals.
  This applies most strongly to Signal I (rhetorical scaffolding). Do not over-anchor on any single
  signal; require corroboration across categories.
- **Multilingual text needs language-matched calibration.** AI detectors badly misclassify non-English text — they wrongly flag lightly-polished
  human Arabic as AI, with one commercial detector dropping from 92% to 12% accuracy (arXiv 2511.16690). Refuse High confidence on non-English text
  unless calibration is known.

### Reference detector landscape (for context)

If the user asks "what would tool X say?", these are the current characteristics:

- **GPTZero (2025)** uses RL adversarial self-training plus a learned classifier ensemble, not just
  perplexity + burstiness. Produces a 4-class output (human / slight / moderate / full AI-assist).
  Older "GPTZero relies on perplexity + burstiness" framing is stale.
- **Binoculars** is a strong zero-shot baseline but has the Claude blind spot above.
- **Pangram 3.0** claims 99.98% accuracy with 1-in-10,000 FPR and 97% on humanized text per vendor
  benchmarks (independent replication pending).
- **EditLens** estimates AI-edit fraction rather than binary authorship (94.7 F1 binary, 90.4 F1 ternary).
- **Ghostbuster** is the canonical black-box (no token probs needed) detector — 99 F1 in-domain,
  degrades out-of-domain.
- **DependencyAI** uses syntactic dependency n-grams + LightGBM, cross-lingual without LLM access.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [harshaneel](https://github.com/harshaneel)
- **Source:** [harshaneel/humanize](https://github.com/harshaneel/humanize)
- **License:** MIT

Install and usage instructions live in the source repository linked above.