Enterprise Readiness Scorer

Enterprise Agentic-Readiness Scorer

Overview

Ch1 opens with a promise and a trap. The promise: an agent that pursues goals instead of answering questions. The trap: "you fire up your favorite LLM, add it to your agentic framework of choice, and connect it to a vector-based RAG system. Should be easy, right? Wrong." A naive vector-only approach creates five fatal flaws that are not bugs but an architectural failure preventing the system from becoming truly agentic.

This skill turns that diagnosis into a score. It checks four things the chapter argues are required for enterprise agency:

1. The five fatal flaws are cured. Each flaw is cured only by a

specific graph capability (context amnesia by evolving memory, relationship blindness by entity relationships, temporal ignorance by temporal evolution, reasoning paralysis by multi-hop reasoning, tool chaos by tool orchestration).

1. The three agency dimensions are calibrated. Autonomy, action, and

authority are sliding scales, not binary — and Ch1's point is calibration, not maximization (a real-estate agent has high autonomy but deliberately low pricing authority).

1. The four emergent capabilities are present. Autonomous

decision-making, contextual understanding, strategic tool utilization, memory persistence.

1. The decision-trace test passes. Per Marple's test in the Enterprise

Context Graphs section: can the system tell you not just what happened, but what alternatives were considered and rejected?

When to Use

• Before greenlighting an enterprise agent for production deployment
• Reviewing a vendor's "context graph" claim against the rejected-alternatives test
• Comparing a naive-vector prototype to a graph-augmented redesign
• Architecture review where someone proposes "just add a bigger vector store"

Phrases: "is this agent production-ready", "enterprise agentic readiness", "score my RAG architecture", "are we naive vector RAG", "context graph vs search index".

When NOT to Use

• Ranking LLMs by quality — Ch1 is explicit that the flaws are

architectural, not model-quality ("cannot be addressed by expanding context windows or refining embedding techniques")

• Consumer FAQ / support bots where queries map to a text snippet — Ch1

names these as a great fit for plain vector RAG

• Single-turn Q&A with no actions, state, or temporal evolution

Process

| Step | Input | Action | Output | Verification | |------|-------|--------|--------|--------------| | 1 | profile JSON (graphcapabilities map) | lib.score_flaws(caps) | (points, cured, openflaws) | each flaw cured only by its mapped capability | | 2 | profile JSON (agency map) | lib.score_agency(agency) | (points, missing dims) | scores coverage/calibration, not magnitude | | 3 | profile JSON (capabilities map) | lib.score_capabilities(caps) | (points, missing) | proportional to capabilities present | | 4 | capturesrejectedalternatives bool | lib.decision_trace_test(b) | (15 or 0, note) | binary test per Marple | | 5 | full profile | lib.assess(profile) | score + band + recommendations | score bounded 0-100; band matches thresholds |

Rationalizations

| Agent rationalization | Documented rebuttal | |------------------------|--------------------| | "Our LLM is frontier-grade, so we are production-ready." | Ch1: the five flaws "aren't bugs. They represent an architectural failure" and "cannot be addressed by optimizing search or tweaking embedding models." Model quality does not cure an architecture gap. | | "We have a vector store, that covers retrieval." | A vector store cures none of the five flaws by itself. Score it: all five stay open, band is NAIVE-VECTOR. The flaws are cured by graph capabilities, not by a vector index. | | "We logged everything, so we have a context graph." | Marple's test (Ch1, The Context Graph): can it tell you what alternatives were rejected? Logging final states is read-time data. The decision-trace test is worth 15 points precisely to catch relabeled search indexes. | | "Max out autonomy and authority for a powerful agent." | Ch1: agency dimensions are sliding scales and must be calibrated, not maximized. The real-estate agent has high autonomy, near-zero pricing authority. This scorer rewards calibration coverage, not magnitude. |

Red Flags

• Score is high but decision_trace is 0. The agent may pass the

capability checklist while recording only outcomes; it will become "a high-fidelity log of failure" if it ever hallucinates (Ch1 counter-thesis).

• All five flaws open but band is not NAIVE-VECTOR. Scoring bug —

open flaws should dominate; recheck the FLAW_CURE mapping.

• Agency magnitude drives the score. Misreads Ch1: a deliberately

low-authority agent is correct design, not a deficiency.

Non-Negotiable Verification

1. Run the benchmark battery. python cli.py benchmark must report:

• all-open flaws score 0; all-cured score the full 40
• each flaw cured only by its mapped graph capability
• decision-trace test is binary 15/0
• a perfect profile is exactly 100 and PRODUCTION-READY; empty is NAIVE-VECTOR
• agency scores calibration coverage, not magnitude

1. Verify CLI help. python cli.py --help exits 0 and prints the SKILL.md description.

Security Posture

• Prompt injection. The profile JSON is untrusted input - a vendor can

self-report flattering capability booleans to inflate the score. The scorer only reads fixed keys against a fixed rubric; adversarial field values can skew the score but never execute, so verify claimed capabilities (especially captures_rejected_alternatives) against evidence before trusting a band.

• Data exfiltration. No network calls, no file writes. Architecture profiles

may describe internal systems; they stay in-process and surface only in the stdout report the caller owns.

• Privilege escalation. No shell invocation, no eval, no dynamic import. A

PRODUCTION-READY band is advisory - it authorizes nothing; the deployment gate that consumes the score owns the actual go/no-go decision.

Source Attribution

Distilled from Agentic GraphRAG (O'Reilly) by Anthony Alcaraz and Sam Julien — Ch1: The Crisis of Agentic AI, specifically the five-fatal-flaws opening (the naive-vector failure), the "Defining Agentic AI" agency dimensions and capabilities, and the "Enterprise Context Graphs" decision-trace test. Supporting references: Singhal 2012 (strings to things), Lilian Weng 2023 (LLM-as-brain), Arvind Jain / Glean context-data-platform, Kirk Marple / Graphlit rejected-alternatives test.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: AnthonyAlcaraz
• Source: AnthonyAlcaraz/agentic-graph-rag-skills
• License: MIT
• Homepage: https://www.oreilly.com/library/view/agentic-graphrag/9798341623163/

Install and usage instructions live in the source repository linked above.