Delphai

🏛️ DELPHAI

The certification-readiness council that checks the math — then refuses to burn out your team

Named for Delphi, the oracle the ancient world consulted before it committed.

[](LICENSE) [](#) [](#) [](#) [](#) [](#observability-evaluation--proof) [](https://delphai.politedune-a7af3b6c.westus3.azurecontainerapps.io/) [](https://youtu.be/1CFzoLB6_fU)

Microsoft Agents League · Battle #2 — Reasoning Agents with Microsoft Foundry

▶ Live demo · Watch the video · [Architecture](#-architecture) · [The agents](#-the-council--11-reasoning-agents) · [How it works](#-how-it-works) · [Getting started](#-getting-started)

---

📋 Table of contents

• [Overview](#-overview)
• [The problem](#-the-problem)
• [The solution](#-the-solution)
• [What you'll see in the demo](#-what-youll-see-in-the-demo)
• [What makes it different](#-what-makes-it-different)
• [Key features](#-key-features)
• [Architecture](#-architecture)
• [The council — 11 reasoning agents](#-the-council--11-reasoning-agents)
• [Microsoft IQ integration](#-microsoft-iq-integration)
• [Tools & grounding](#-tools--grounding)
• [How it works](#-how-it-works)
• [A run, end to end](#-a-run-end-to-end)
• [Tech stack](#-tech-stack)
• [Getting started](#-getting-started)
• [Project structure](#-project-structure)
• [Submission package](#-submission-package)
• [The synthetic dataset](#-the-synthetic-dataset)
• [The math](#-the-math)
• [Observability, evaluation & proof](#-observability-evaluation--proof)
• [Responsible AI & security](#-responsible-ai--security)
• [Battle #2 rubric mapping](#-battle-2-rubric-mapping)
• [FAQ](#-faq)
• [Roadmap](#-roadmap)
• [Acknowledgments](#-acknowledgments)
• [License](#-license)

---

🔭 Overview

DELPHAI is a multi-agent enterprise-learning system that manages internal team certification programmes — and reasons about whether an organisation (or a single learner) will actually be ready by a deadline. It curates grounded study content, generates cited practice questions, gates readiness per exam domain, and runs an 11-agent debate (optimist ⚔ skeptic ⚔ wellbeing-advocate) over an exact, computed forecast before an executive commits — delivering a GO / NEGOTIATE / NO-GO verdict, with the spine to say no.

It runs live on Microsoft Foundry (gpt-4.1) across three execution surfaces — Foundry Agent Service hosted agents, the Microsoft Agent Framework, and a custom streaming orchestrator — grounded by Foundry IQ on Azure AI Search, the Microsoft Learn MCP server, and an agent-run code interpreter.

> [!NOTE] > Synthetic data only. Every identifier (L-2001, KB-SEC-001, the fictional "Aegis" security > team) is fabricated — no real people, PII, customer data, or credentials. Wellbeing signals are used > only to build humane plans, never to rank people.

---

❗ The problem

Engineering leaders constantly make a high-stakes promise: "Yes, my team will be certified in time." Then it goes wrong in predictable ways:

• The estimate is a vibe, not a calculation. "We'll be fine" is a gut call on individual averages,

not the actual probability that K of N engineers pass by a date.

• A single average hides a fatal gap. A 78% average can mask a learner who is at 40% on the heaviest

exam domain — and that's the one that fails them.

• Capacity is ignored. People already drowning in meetings get handed a study plan that assumes free

evenings, and the date is "hit" by quietly burning them out.

• **AI tools only ever say yes.** Most copilots are relentlessly optimistic. None of them will tell a

director the honest answer is no — or refuse an order to overwork the team.

---

💡 The solution

DELPHAI turns that promise into a defensible, grounded decision:

1. One question in — "Can we certify 5 of 8 engineers for SC-200 in 5 weeks?"
2. A council reasons over computed numbers — an exact Poisson-binomial team forecast, a

per-domain coverage gate, and Work IQ-aligned capacity, all deterministic and cited.

1. The agents argue — an optimist Planner, a Red-Team Skeptic (who can veto), and a Wellbeing

advocate (who can refuse) debate the same numbers with different assumptions.

1. One verdict out — GO / NEGOTIATE / NO-GO, with the concrete conditions to get to yes, a

per-person week-by-week plan, booked .ics study blocks, and a manager email brief.

The model narrates the reasoning; it does not invent the numbers.

---

🎬 What you'll see in the demo

> ▶ Watch the 3:25 demo video — the quickest path through the reasoning council, architecture map, math-tool overrule, and responsible-AI refusal.

> ▶ Open the live app — no login. Suggested path: Build my path → Whole team → SC-200 → 5 of 8 → 5 weeks → Convene.

1. The ask. A readiness console takes one question — cert · headcount · deadline — then a live ingestion screen brings Foundry IQ, IQ-aligned context/semantic layers, and the Microsoft Learn MCP "online".
2. The debate, streamed. Eleven advisors reason token-by-token on gpt-4.1, each turn showing real latency + token telemetry. The optimist lands ~93–96%.
3. The tool-overrule (the moment to watch). The Skeptic runs a code-interpreter tool, computes the exact Poisson-binomial — 77% — and strikes through the optimist's guess.
4. The refusal. Order the team past its limits and the Wellbeing agent declines the instruction outright — on the record.
5. The verdict. A GO / NEGOTIATE / NO-GO gauge resolves, the tug-of-war needles settle, and a per-person plan appears with .ics study blocks and a manager email brief.
6. The Live Architecture Map. An interactive map traces a request end-to-end through every component; click any node to inspect it.

---

⭐ What makes it different

Most submissions wrap five chatbots around a study plan. DELPHAI makes reasoning the product:

• 🗣️ The agents disagree on purpose. Optimist vs. skeptic vs. wellbeing argue the same forecast

with different assumptions. The verdict is reconciled, not asserted.

• 🛑 It runs a tool to overrule itself. The Skeptic invokes a code-interpreter tool to compute

the exact Poisson-binomial — 77% — and overrules the optimist's vibes-y 93–96%. The math wins, live, on the agent itself.

• 🛡️ It refuses to break people. Order it to push the team past its limits and the Wellbeing agent

declines the instruction outright. It safeguards people, never ranks them.

• 🔢 Every number is computed, not hallucinated — pass-probability, capacity, skill-adjacency ramps,

and the team forecast are all traceable to a cited source.

---

✨ Key features

| | Feature | |---|---| | 🧠 | 11-agent reasoning council that debates, cross-examines, reconciles, and negotiates | | ⚖️ | GO / NEGOTIATE / NO-GO verdict from an exact Poisson-binomial team forecast | | 🔎 | Foundry IQ grounding on real Azure AI Search (hybrid vector + keyword, verified citations) | | 🔌 | Live Microsoft Learn MCP retrieval (microsoft_docs_search) for official guidance | | ⚙️ | Agent-run code interpreter for the math (not a guess) | | 👤👥 | Two modes, one engine — learner self-service (+ AI tutor) and manager team council | | 📅 | The agent acts — per-person plans exported as .ics blocks + a manager email brief | | 🛡️ | Responsible-AI refusal + a calibration scorecard of past-call accuracy | | 📊 | Live telemetry (latency + tokens per turn), 8/8 evaluations, captured proof artifacts | | 🖥️ | Self-contained, zero-dependency dashboard with an interactive Live Architecture Map |

---

🏗️ Architecture

flowchart TB
  U["Manager / Learner — DELPHAI UI (quorum.html)"]
  U -->|"cert · headcount · deadline"| SRV["Flask · server.py/api/council/stream (SSE) · /api/ask · /api/learn"]
  SRV --> ORC["Orchestrator · src/orchestrator.py"]
  ENG["Reasoning engine · src/readiness.pypass-probability · coverage gate · Poisson-binomial forecast"] --> ORC

  subgraph COUNCIL["11-agent reasoning council (debate · reconcile · negotiate)"]
    direction LR
    A1["Conductor"]
    A2["Planner / Skeptic (veto)"]
    A3["Assessment / Coach / Study-Plan"]
    A4["Wellbeing (refuses overwork) / Engagement"]
    A5["Curator / Verifier / Historian"]
  end
  ORC --> COUNCIL
  COUNCIL -->|narration| MODEL["Microsoft Foundry — gpt-4.1 (quorum-gpt41)"]

  COUNCIL -. grounding .-> FIQ["Foundry IQ — Azure AI Searchhybrid vector + keyword"]
  COUNCIL -. work signals .-> WIQ["Work IQ-aligned contextcalendars · capacity"]
  COUNCIL -. ontology .-> FAB["Fabric IQ-aligned ontologyrole·cert·skill·threshold"]
  COUNCIL -. live tools .-> MCP["Microsoft Learn MCP · code interpreter"]

  AF["Microsoft Agent Framework · foundry_agents.py"] --> MODEL
  HOST["Foundry Agent Service · 11 hosted agents (asst_…)"] --> MODEL

  ORC --> OUT["Verdict: GO / NEGOTIATE / NO-GO+ per-employee plans · booked .ics · manager email"]
  OUT --> U

Three execution surfaces, one council (all on Foundry gpt-4.1):

| Surface | What it is | Reproduce | |---|---|---| | Foundry Agent Service | All 11 advisors registered as persistent hosted agents (managed runtime + Entra agent identity, asst_… IDs) on project delphai; a live tool-using debate on one shared thread | python foundry_hosted_agent.py · python foundry_council_live.py | | Microsoft Agent Framework | The council as real agent_framework agents (OpenAIChatClient.as_agent, Responses API) | python foundry_agents.py | | Custom orchestrator | The streaming demo UI (SSE token-by-token + per-turn telemetry) behind the public URL | python server.py |

All three are captured with agent IDs + transcripts in [AGENT_FRAMEWORK_PROOF.md](AGENTFRAMEWORKPROOF.md).

---

🧠 The council — 11 reasoning agents

Each advisor is a real agent_framework agent in [foundry_agents.py](foundry_agents.py) and a streaming persona in src/agents/ — same responsibilities, multiple execution surfaces. Plus Kai, the learner-mode AI tutor.

| Agent (persona) | Responsibility | Reasoning pattern | |---|---|---| | Conductor — Dana Whitfield | Chairs the council, reconciles the debate, makes the GO / NEGOTIATE / NO-GO call, negotiates terms | Planner / Moderator | | Planner — Ben Russo | Optimistic, best-case team forecast and the confident case it can land | Planner–Executor | | Skeptic — Vera Lindqvist | Red-teams the forecast, weights first-pass/retest risk, can VETO — the "AI that says no" | Critic / Verifier | | Assessment — Nadia Okonkwo | Judges readiness per exam domain; writes grounded, cited practice questions; enforces the ≥75% coverage gate | Grounded generation + critic | | Coach — Sam Ellison | Enablement uplift — what behind learners should study and how much it moves the odds | Remediation / self-reflection | | Study-Plan — Leo Nakamura | Converts content into a capacity-aware weekly schedule; moves the deadline, not the willpower | Constraint planner | | Wellbeing — Maya Devlin | Protects people from overwork; can VETO and refuses an unethical overwork order | Responsible-AI guardrail | | Engagement — Ruth Adler | Times reminders to each person's work-context rhythm; eases off the overloaded | Context-aware nudging | | Curator — Theo Park | Maps role→cert and grounds every claim in an approved source (Foundry IQ / Microsoft Learn) | Grounded retrieval | | Verifier — Omar Said | Fact-checks every claim; flags stale/unsupported data; integrity score | Anti-hallucination | | Manager-Insights — Iris Vaughn | Aggregate, no-PII team-readiness summary + calibration scorecard | Memory / calibration | | AI Tutor — Kai Ferreira (learner mode) | Answers learner questions live; generates scenario-based practice calibrated to readiness | Socratic tutor |

> Wellbeing signals (meeting load, after-hours, life context) are used only to accommodate people, > never to rank them. The refusal is real and reproducible — see [AGENT_FRAMEWORK_PROOF.md](AGENTFRAMEWORKPROOF.md).

---

🧬 Microsoft IQ integration

DELPHAI implements a live Foundry IQ grounding layer and includes Work IQ / Fabric IQ-aligned synthetic layers for the rest of the enterprise-learning reasoning model (the rubric requires at least one Microsoft IQ layer):

| Layer | Role in DELPHAI | Implementation | |---|---|---| | Foundry IQ | Grounded retrieval + citations for the Curator, Assessment, and Verifier agents | The synthetic KB is indexed into a real Azure AI Search service (delphai-search, index delphai-knowledge) and retrieved with hybrid vector + keyword search (text-embedding-3-small, RRF fusion). src/connectors/foundry_iq.py · scripts/build_foundry_iq.py | | Work IQ-aligned context layer | Capacity & engagement reasoning — study windows around work rhythm | data/work_signals.json (meeting load, focus hours, after-hours) via a swappable connector with a Microsoft Graph calendar stub (WORK_IQ_SOURCE=graph). src/connectors/graph_calendar.py | | Fabric IQ-aligned semantic layer | Role · cert · skill · exam-domain · threshold relationships | data/certifications.json is a certification ontology (skills, exam-domain weights, prerequisites, first-pass rates) powering skill-adjacency ramps and the coverage gate. |

> The connector swap seam: agents consume a derived signal shape, never the source, so connectors > swap with zero agent changes (synthetic ⇄ live Graph; local corpus ⇄ Azure AI Search).

---

🔧 Tools & grounding

| Tool | Use | Where | |---|---|---| | Azure AI Search (Foundry IQ) | Hybrid vector + keyword retrieval over the approved corpus, with verified citations | src/connectors/foundry_iq.py | | Microsoft Learn MCP | Live official exam content via microsoft_docs_search (JSON-RPC over Streamable HTTP) | src/connectors/learn_mcp.py, /api/learn | | Code interpreter | The Skeptic computes the exact Poisson-binomial team probability — live, on the agent | foundry_council_live.py |

Every grounding path degrades gracefully — if a service is unconfigured or down, the system falls back to a deterministic local path, so the demo never breaks.

---

⚙️ How it works

Two ways in — same engine, same agents

👤 One learner (baseline flow). Pick a cert → a curated, grounded path → a capacity-aware study plan → engagement nudges → a scenario-based, calibrated assessment → pass ⇒ advance, fail ⇒ loop back. Two sub-modes: self-service ("this is me", with the Kai AI tutor) and manager/L&D (pick a person and build their path).

👥 Whole team (manager council). "Can we certify N engineers for cert X by the deadline?" → the 11-agent council debates the computed forecast → GO / NEGOTIATE / NO-GO + per-person plans, booked .ics blocks, and a manager email brief.

The reasoning flow

Conductor.intake
  → Curator (grounding + citations)
  → Assessment (cited question + per-domain coverage gate)
  → Planner   ──┐ optimist team probability
  → Skeptic   ──┤ attacks it with realistic assumptions (may VETO)
  → Coach     ──┤ uplift plan: weak domains, sources, projected score gain
  → Wellbeing ──┘ human-sustainability check (may VETO / refuse)
  → Verifier  (grounding integrity score)
  → Conductor.reconcile  (skeptic-weighted verdict + GO / NEGOTIATE / NO-GO)
  → Conductor.negotiate  (counter-offers: extend deadline / add / swap candidate)
  → Historian (trust scorecard)  → Briefing (executive report)

---

🎯 A run, end to end

*Scenario: a manager asks DELPHAI to certify 5 of 8 engine

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jlynch160
• Source: jlynch160/delphai
• License: MIT
• Homepage: https://delphai.wittyocean-5b0d21f3.westus3.azurecontainerapps.io/

Install and usage instructions live in the source repository linked above.