# Foundry Doc Vision Speech

> >

- **Type:** Skill
- **Install:** `agentstack add skill-aiappsgbb-awesome-gbb-foundry-doc-vision-speech`
- **Verified:** Pending review
- **Seller:** [aiappsgbb](https://agentstack.voostack.com/s/aiappsgbb)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [aiappsgbb](https://github.com/aiappsgbb)
- **Source:** https://github.com/aiappsgbb/awesome-gbb/tree/main/skills/foundry-doc-vision-speech

## Install

```sh
agentstack add skill-aiappsgbb-awesome-gbb-foundry-doc-vision-speech
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Foundry Doc / Vision / Speech

This skill wires the **non-chat AI services** every threadlight process eventually
needs — image understanding, document parsing, voice — into the agent runtime.

It reads SPEC § 7b (AI Services & Model Selection) from `threadlight-design`,
selects the right Azure resource per modality, generates the tool contracts,
and emits the Bicep module selectors that `threadlight-deploy` Phase 6 + 
`azd-patterns` consume.

## When to Use

Use this skill when the SPEC declares ANY of:
- Image / photo / video frame analysis (returns triage damage photos, KYC selfie,
  blueprint annotation, supplier facility photos)
- Structured-document extraction (invoices, claim forms, KYC IDs, quotes, BOMs)
- Voice / audio (FNOL voice intake, call recording transcription, IVR responses)

If the SPEC only needs chat (text-in, text-out), you do NOT need this skill —
the default `gpt-5.4-mini` model from `threadlight-deploy` is sufficient.

## When NOT to Use

- **Pure chat agents** — `threadlight-deploy` handles the chat model
- **Knowledge retrieval over documents** — that's `foundry-iq` (semantic search
  over already-indexed content). Use this skill for the **extraction step that
  produces searchable text from unstructured docs**, then hand off to `foundry-iq`
  for retrieval.
- **MCP server deployment** — that's `foundry-mcp-aca`

---

## ⚠️ GPT-4o is LEGACY (May 2026)

GPT-4o and GPT-4o Vision are explicitly **forbidden** as defaults. SPEC § 7b
hardcodes the modern decision tree:

| Use case | Model | Notes |
|----------|-------|-------|
| Default chat reasoning | `gpt-5.4-mini` (2026-03-17) | 400K context, vision-capable, fastest, cheapest |
| Vision (default) | `gpt-5.4-mini` | Multimodal in same model — usually no separate vision deployment |
| Vision (high-stakes, large frames) | `gpt-5.4` (2026-03-05) | 1M context, more accurate on dense visual content |
| Vision + reasoning premium | `gpt-5.4-pro` | When the answer must reason about the image (e.g., "is this a structural defect?") |
| Bulk/cheap vision | `gpt-5.4-nano` | Background batch jobs, throwaway analysis |
| Code multimodal | `gpt-5.3-codex` | Screenshot → code, error UI → diagnosis |
| Structured docs | Document Intelligence v4 prebuilt | Field extraction with confidence scores; faster than vision LLM for known forms |
| Custom forms | Document Intelligence v4 custom | Train on 5+ samples of customer-specific forms |
| Voice → text | Azure Speech-to-Text (Whisper option) | Fast, batch, real-time options |
| Text → voice | Azure Speech-to-Text (neural voices) | For IVR, audio briefings |

If anyone proposes `gpt-4o`, push back: it doesn't support encrypted content,
has a 128K context vs 400K-1M, and is on a deprecation glide path. There is no
business case for new processes on legacy models.

---

## Decision Tree: Vision vs Document Intelligence vs Speech

```
What kind of unstructured input?
├── IMAGE
│   ├── Known structured form (invoice, ID, claim form, BOM page)
│   │   → Document Intelligence (prebuilt or custom)
│   ├── Free-form photo (damage, facility, blueprint, screenshot, selfie)
│   │   → gpt-5.4-mini vision (or gpt-5.4 if dense / high-stakes)
│   └── Bulk batch (1000s of images, low-stakes)
│       → gpt-5.4-nano vision
│
├── DOCUMENT (PDF / DOCX / scanned)
│   ├── Structured (forms, tables, key-value)
│   │   → Document Intelligence v4 prebuilt
│   ├── Customer-specific (their proprietary form)
│   │   → Document Intelligence v4 custom (train on 5+ samples)
│   └── Free-form / mixed → render page-by-page + gpt-5.4 vision
│
└── AUDIO
    ├── Real-time stream (live call, IVR turn) → Azure Speech-to-Text streaming
    ├── Recorded file (FNOL voicemail, recording)
    │   → Azure Speech-to-Text batch (or Whisper-equivalent endpoint)
    └── TTS for response → Azure Speech neural voices
```

---

## Pattern 0 — Foundry Toolbox (recommended starting point, May 2026)

Before wiring SDK calls per modality, **check whether a Foundry Toolbox can
do the job**. The Foundry Toolbox is a server-side, versioned bundle of
hosted tool configurations curated in the Foundry portal and exposed as a
single MCP-compatible endpoint. As of May 2026, the catalog includes
**Speech, Document Intelligence, Vision, Content Understanding, Translator,
Language, Content Safety, Custom Vision, Azure AI Search**, plus generic
hosted tools (web search, code interpreter, file search, image generation).

**Why prefer Toolbox over per-modality SDK code:**

- **Centralized config**: connection strings, SAS containers, model IDs all
  live in the portal — no env-var fan-out across containers
- **Versioning**: change toolbox config, test with `version="v3"`, promote
  to default — no agent redeploy
- **Auth handled server-side** for the upstream services (Speech / DocIntel
  account credentials are configured **once** at toolbox creation; the
  consumer only needs an Entra token to the Toolbox endpoint itself)
- **MCP-compatible consumption**: works with any agent runtime (MAF, GHCP
  SDK via bridge, LangGraph, custom code)
- **Same code across processes**: a healthcare KYC and an FSI claim agent
  can both consume the same `vision_doc_speech_toolbox`

**When NOT to use Toolbox** (fall back to direct SDK in Pattern A/B):

- You need fine-grained control over a SDK feature not exposed by the toolbox
  tool wrapper (e.g., DocIntel custom-trained model, Speech batch transcription
  with diarization, real-time streaming audio frames)
- Network-secured Foundry: **Azure Speech MCP doesn't support network-secured
  Foundry projects** as of May 2026 — you must use direct SDK in Pattern A/B
- Toolbox is preview (`azure-ai-projects>=2.1.0`); customer policy may forbid
  preview-tier features in prod
- High-volume batch (e.g., 100k invoices/night) — toolbox introduces a network
  hop the SDK doesn't

### Pattern 0a — DEFAULT: MCP consumption (works with ALL runtimes)

**Always start here.** `MCPStreamableHTTPTool` (or your runtime's equivalent
MCP client) talks to the Toolbox endpoint over HTTPS. This is the path the
Foundry team officially documents for MAF, GHCP (via bridge), LangGraph,
and custom code.

**Working sample (MAF — direct, with mandatory workarounds):**

```python
import os, asyncio
from typing import Any
# Use the SYNC azure.identity here — get_bearer_token_provider in the sync
# module returns a callable that returns a string when invoked. Using the
# .aio variant here returns a coroutine that gets f-string-formatted as
# "Bearer " and the server returns 401.
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from agent_framework import Agent, MCPStreamableHTTPTool
from agent_framework.foundry import FoundryChatClient

# 1. Build a refreshing Entra token provider for the Toolbox MCP endpoint.
#    Scope MUST be https://ai.azure.com/.default — wrong scope = 401.
credential = DefaultAzureCredential()  # reads AZURE_CLIENT_ID for UAMI
token_provider = get_bearer_token_provider(credential, "https://ai.azure.com/.default")

# 2. Build the MCP tool.
#    Endpoint format (note the "toolboxes/.../versions/.../mcp" shape — pin a version):
#    https://.services.ai.azure.com/api/projects//toolboxes//versions//mcp?api-version=v1
mcp_tool = MCPStreamableHTTPTool(
    name="vision_doc_speech_mcp",
    url=os.environ["TOOLBOX_MCP_ENDPOINT"],
    # GOTCHA: MAF calls header_provider(kwargs) — it MUST accept one positional
    # arg, not zero. Verified against agent_framework/_mcp.py in 1.x.
    # Use header_provider (NOT static `headers=`) so tokens refresh on expiry.
    header_provider=lambda kwargs: {"Authorization": f"Bearer {token_provider()}"},
    load_prompts=False,         # GOTCHA: Foundry MCP returns 500 on prompts/list
)

# 3. GOTCHA: MAF's MCPStreamableHTTPTool._ensure_connected() calls send_ping(),
#    which the Foundry MCP server rejects with 500. Override to a no-op.
async def _no_ping(*args: Any, **kwargs: Any) -> None: return None
mcp_tool._ensure_connected = _no_ping  # type: ignore[assignment]

async def main() -> None:
    async with mcp_tool, Agent(
        client=FoundryChatClient(credential=credential, model="gpt-5.4-mini"),
        name="ClaimsIntake",
        tools=[mcp_tool],
    ) as agent:
        # GOTCHA: Toolbox MCP requires stream=True on tools/call.
        # MAF Agent.run streams by default — explicitly avoid stream=False overrides.
        result = await agent.run("Transcribe the call and extract claim details.")
        print(result.text)

asyncio.run(main())
```

**Working sample (GHCP — via the official MCP bridge):**

GHCP rejects dots in tool names; Foundry MCP returns names as
`{server_label}.{tool_name}`. The official sample at
`https://aka.ms/foundry-toolbox-copilotsdk` provides an MCP bridge that
replaces `.` → `_` automatically. Use that bridge — do NOT roll your own.

> **GHCP cannot use Pattern 0b (native).** Native toolbox attachment
> (`tools=toolbox` directly) is a MAF-specific surface that has no
> equivalent in the GitHub Copilot SDK. For GHCP, **Pattern 0a is the
> only option** — and you must use the published bridge.

**MCP gotcha cheat-sheet** (from official troubleshoot, all confirmed
May 2026 — these WILL bite if ignored):

| Symptom | Cause | Fix |
|---------|-------|-----|
| `401` on MCP calls | Wrong/expired token | Scope MUST be `https://ai.azure.com/.default`; use `header_provider` (refreshes), not static `headers` |
| `500` on `send_ping()` | Foundry MCP doesn't implement `ping` | Override `MCPStreamableHTTPTool._ensure_connected` to a no-op (see sample) |
| `500` on `prompts/list` | Foundry MCP doesn't implement prompts | Pass `load_prompts=False` |
| `500` on `tools/call` | Non-streaming not supported | Use `stream=True` (MAF default; explicitly verify if you override) |
| `400 Multiple tools without identifiers` | Two unnamed tools of same type in toolbox | Toolbox creator must add `server_label`/`name` to each MCP tool |
| Tool name not found / GHCP error | Foundry returns `{server_label}.{tool_name}`; GHCP rejects dots | GHCP: use the bridge that swaps `.` → `_`. MAF: use the dotted name as-is |
| Custom env var silently overwritten | Platform reserves `FOUNDRY_*` prefix | Rename your env vars (e.g. `TOOLBOX_MCP_ENDPOINT`, NOT `FOUNDRY_TOOLBOX_ENDPOINT`) |

### Pattern 0b — Native MAF (`MCPStreamableHTTPTool`)

Use `MCPStreamableHTTPTool` to connect a MAF agent directly to the
toolbox MCP endpoint. This replaces the removed `client.get_toolbox()`
convenience helper (removed in MAF 1.3.0).

- **GHCP SDK does NOT support this** — use Pattern 0a with the bridge instead
- The `allowed_tools` parameter filters to specific tools from the toolbox

```python
# MAF 1.3.0+ — MCPStreamableHTTPTool replaces the removed get_toolbox()
import os
from agent_framework import Agent, MCPStreamableHTTPTool
from agent_framework.foundry import FoundryChatClient
from azure.identity.aio import AzureCliCredential

async with AzureCliCredential() as credential:
    client = FoundryChatClient(credential=credential)
    toolbox_tool = MCPStreamableHTTPTool(
        url=os.environ["TOOLBOX_MCP_ENDPOINT"],
        load_prompts=False,
        headers={"Foundry-Features": "Toolboxes=V1Preview"},
        allowed_tools=["azure_speech", "document_intelligence"],
    )
    async with Agent(client=client, name="ClaimsIntake", tools=[toolbox_tool]) as agent:
        result = await agent.run("Transcribe and extract claim details.")
```

If something silently misbehaves (tools listed but never invoked, partial
schemas, schema-inference errors), **fall back to Pattern 0a (MCP)** — it
exercises the same upstream Toolbox endpoint with strictly fewer SDK
abstractions in between.

### Keyless RBAC for Toolbox consumption

Everything below assumes **keyless** auth — no API keys are passed anywhere
in the threadlight chain (we set `disableLocalAuth: true` on every Cognitive
Services resource at provisioning time per `azd-patterns`).

| Identity | Role | Scope | Why |
|----------|------|-------|-----|
| **Consuming agent's UAMI** | `Azure AI User` | Foundry project | Required to mint tokens for `https://ai.azure.com/.default` and read the toolbox MCP endpoint |
| **Toolbox creator (one-time, not runtime)** | `Azure AI Project Manager` | Foundry project | Needed to create/update toolbox versions and assign `Azure AI User` to consumers |
| **Foundry project's own MI** | `Azure AI User` | Foundry account | Project proxies inference + reads upstream tool credentials configured in toolbox |
| **Upstream service identities** (Speech / DocIntel / Vision MI) | per-service role (see Pattern A/B below) | each Cognitive Services resource | Configured ONCE at toolbox creation — consumers never see these |

> **Why `Azure AI User` and not `Azure AI Developer`?** `Azure AI Developer`
> is scoped to the legacy AML / Foundry-hub world; hosted-agent + toolbox
> resources need `Azure AI User` (or higher: `Azure AI Project Manager`).

**Toolbox catalog reference**: [Foundry tool catalog](https://learn.microsoft.com/azure/foundry/agents/concepts/tool-catalog)
· [Toolbox how-to (Python)](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox)
· [Toolbox troubleshoot](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/toolbox#troubleshoot)
· [Hosted-agent permissions](https://learn.microsoft.com/azure/foundry/agents/concepts/hosted-agent-permissions)
· [Azure Speech MCP](https://learn.microsoft.com/azure/foundry/agents/how-to/tools/azure-ai-speech)
· [GHCP toolbox bridge sample](https://aka.ms/foundry-toolbox-copilotsdk)
· [MAF toolbox MCP sample](https://aka.ms/foundry-toolbox-maf)

If Toolbox doesn't fit (network-secured project, custom model, high-volume
batch), continue with Pattern A/B per modality below.

---

## ⚠️ Keyless / RBAC Matrix (every modality, every pattern)

Threadlight pilots are **keyless by default**. Every Cognitive Services
resource (`AIServices`, `DocumentIntelligence`, `SpeechServices`) is
provisioned with `properties.disableLocalAuth: true` per `azd-patterns`,
and runtime auth is via UAMI + Entra token. **No `KEY1` / `subscription_key`
appears anywhere in the chain.** When a customer needs to roll back to keyed
auth (rare — typically air-gapped lab), they explicitly opt in.

### The matrix

| Modality | Identity | Role assignment | Token scope | Notes |
|----------|----------|-----------------|-------------|-------|
| **Foundry chat (Responses endpoint)** | Agent's UAMI | `Azure AI User` on the **Foundry project** | `https://ai.azure.com/.default` | Project proxies inference using its own MI; this is the threadlight default |
| **Direct AOAI account endpoint** (bypassing project) | Agent's UAMI | `Cognitive Services OpenAI User` on the **Foundry account** | `https://cognitiveservices.azure.com/.default` | Only when you must hit the account endpoint — rare |
| **Foundry Toolbox via MCP** | Agent's UAMI | `Azure AI User` on Foundry project | `https://ai.azure.com/.default` | Toolbox creator (one-time) needs `Azure AI Project Manager` |
| **Vision via Foundry Responses** | Same as chat | Same as chat | Same as chat | Vision is a content-part on the chat call — no extra role |
| **Document Intelligence (direct SDK)** | Agent's UAMI | `Cognitive Services User` on the **DocIntel resource** | `https://cognitiveservices.azure.com/.default` | **NOT `Cognitive Services Contributor`** — Contributor only allows listing keys, and is denied at runtime when `disableLocalAuth: true`. Verified May 2026. |
| **Azure Speech (direct SDK)** | Agent's UAMI | `Cognitive Services Speech User` (or `Speech Contributor`) on the **Speech resource** | `https://cognitiveservices.azure.com/.default` | **REQUIRES custom subdomain** (one-time, **IRREVERSIBLE** — see Speech section) |
| **Azure AI Search (foundry-iq RAG)** | Agent's UAMI | `Search Index Data Reader` on Search service | `https://search.azure.com/.default` | Add `Search Service Contributor` for index management roles only |
| **Foundry contin

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [aiappsgbb](https://github.com/aiappsgbb)
- **Source:** [aiappsgbb/awesome-gbb](https://github.com/aiappsgbb/awesome-gbb)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** yes

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: flagged — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-aiappsgbb-awesome-gbb-foundry-doc-vision-speech
- Seller: https://agentstack.voostack.com/s/aiappsgbb
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.