# Foundry Network Runbook

> >

- **Type:** Skill
- **Install:** `agentstack add skill-aiappsgbb-awesome-gbb-foundry-network-runbook`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [aiappsgbb](https://agentstack.voostack.com/s/aiappsgbb)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [aiappsgbb](https://github.com/aiappsgbb)
- **Source:** https://github.com/aiappsgbb/awesome-gbb/tree/main/skills/foundry-network-runbook

## Install

```sh
agentstack add skill-aiappsgbb-awesome-gbb-foundry-network-runbook
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Foundry Network Runbook — Diagnose connectivity failures after the deploy succeeds

## 1. Goal

This runbook is for the on-call engineer who just got the page **"Foundry
agent is up but inference / project connection / model call is failing
over the network."** The `foundry-vnet-deploy` succeeded (or the
`foundry-caphost-lifecycle` create returned `Succeeded`), the resources
exist in the portal — but something at the **network layer** is wrong:
DNS resolves to the wrong IP, an NSG drops the packet, an RBAC role is
missing for a private DNS zone, a VNet peering went `Disconnected`, the
caphost subnet is wedged. You have ≤ 30 minutes to isolate the layer
before escalating.

This runbook is **not** a telemetry guide (`foundry-observability`
covers App Insights + OTel wiring), not a deploy guide
(`foundry-vnet-deploy` covers Day-0; `foundry-caphost-lifecycle` covers
Day-2 lifecycle), not an auth / JWT troubleshoot for the Citadel APIM
gateway (that's `citadel-spoke-onboarding`), and not an SDK / quota
troubleshoot. If the symptom is not a network-layer symptom, jump to
§ 8 — the cross-reference index points you at the right skill.

## 2. When to use this runbook

Trigger this runbook when **all** of the following are true:

1. The Foundry deployment (or hosted-agent create, or caphost create)
   reported `Succeeded` in ARM and the resources are visible in the
   portal.
2. Something fails downstream at the network layer: 503 from inference,
   DNS resolving to a public IP, project connection showing red, a
   private endpoint missing or not approved, a fresh redeploy hitting
   `Subnet already in use`.
3. You have a baseline of "this worked yesterday" or "the deploy just
   completed, why doesn't it work now?" — i.e., the failure mode is
   **operational** (post-deploy), not a code or SDK bug.

If the failure is during the Bicep `az deployment group create` itself,
use `foundry-vnet-deploy § 10b` (safe retry). If the failure is a
hosted-agent-creation 403 with `MI provisioning failed` or
`NIC provisioning failed`, this runbook's § 4 matrix points you back to
`foundry-vnet-deploy § 8b` for the RBAC fix.

## 3. Pre-flight checklist

Capture this baseline **before** you start diagnosing. Without it you
will mis-attribute symptoms (e.g., blaming DNS when the real failure is
a peering that went `Disconnected` 10 minutes earlier).

Run all 7 commands in order. Each block is read-only — none of them
mutate Azure state.

```bash
# 1. VNet peerings (spoke side): all MUST be Connected
az network vnet peering list \
  -g "$RG" --vnet-name "$VNET" \
  --query "[].{name:name, state:peeringState, remote:remoteVirtualNetwork.id}" \
  -o table
```

```bash
# 2. Private DNS zones reachable from the spoke
az network private-dns zone list \
  -g "$DNS_RG" \
  --query "[].{name:name, vnetLinks:numberOfVirtualNetworkLinks, records:numberOfRecordSets}" \
  -o table
```

```bash
# 3. Private endpoints in the spoke RG and their approval status
az network private-endpoint list -g "$RG" \
  --query "[].{name:name, target:privateLinkServiceConnections[0].privateLinkServiceId, status:privateLinkServiceConnections[0].privateLinkServiceConnectionState.status}" \
  -o table
```

```bash
# 4. Foundry account network state (public access + injection subnet)
az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUB}/resourceGroups/${RG}/providers/Microsoft.CognitiveServices/accounts/${ACCT}?api-version=2025-04-01-preview" \
  --query "{public:properties.publicNetworkAccess, injections:properties.networkInjections}" -o json
```

```bash
# 5. Project MI role assignments (the principal the agent runtime runs as)
PRINCIPAL_ID=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUB}/resourceGroups/${RG}/providers/Microsoft.CognitiveServices/accounts/${ACCT}/projects/${PROJ}?api-version=2025-04-01-preview" \
  --query "identity.principalId" -o tsv)

az role assignment list --assignee "$PRINCIPAL_ID" --all \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table
```

```bash
# 6. Agent subnet binding (serviceAssociationLink + delegation)
az network vnet subnet show \
  -g "$VNET_RG" --vnet-name "$VNET" -n "$AGENT_SUBNET" \
  --query "{name:name, sal:serviceAssociationLinks, delegations:delegations[].serviceName}"
```

```powershell
# 7. DNS resolution FROM INSIDE the spoke (run on a Bastion VM, peered VM,
#    or VPN-connected host — NOT from your laptop over the public internet)
Resolve-DnsName "${ACCT}.cognitiveservices.azure.com"
# MUST return a 10.x / 172.16-31.x / 192.168.x private IP.
# Public IP back = private endpoint missing OR DNS zone not VNet-linked.
```

If any of the 7 baseline checks already shows an anomaly, jump straight
to the matching row in § 4.

## 4. Symptom → cause → fix matrix

Network-layer symptoms only. For application-layer / SDK / quota issues
see § 8 (cross-references).

| Symptom | Likely cause | Diagnostic | Fix |
|---|---|---|---|
| `InvalidPrivateDnsZoneIds` at `az deployment group create` time | Deployment principal lacks `Private DNS Zone Contributor` on one of the 6 hub PDZs | `az role assignment list --assignee  --scope /subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones/ -o table` | Grant `Private DNS Zone Contributor` on each zone in the hub subscription (see § 5). Cap: per-zone, not RG-wide. |
| `Subnet 'agent-subnet' is already in use by capability host` on a second deploy into the same VNet | Caphost subnet still bound by a `serviceAssociationLink` from a prior caphost / account; soft-delete or caphost-only DELETE does **not** release it | `az network vnet subnet show … --query serviceAssociationLinks` returns non-empty | Full account purge — see `foundry-caphost-lifecycle § 8` for the verified GA CLI sequence and § 9 for the redeploy guard. |
| Agent inference returns HTTP 503 within ~1s, no trace in AppInsights | No private endpoint to the AI Services account from the spoke OR PE exists but DNS zone not VNet-linked | Run § 3 step 3 (enumerate PEs) and step 7 (Resolve-DnsName). PE missing = no row for AIServices in step 3. Zone unlinked = step 7 returns a public IP. | Add the missing PE via `az network private-endpoint create --group-id account --private-connection-resource-id `, then `az network private-dns link vnet create` for `privatelink.cognitiveservices.azure.com`, `privatelink.openai.azure.com`, `privatelink.services.ai.azure.com`. |
| Agent inference hangs ≥ 30s then times out (no HTTP status, no body) | NSG on the agent subnet (or PE subnet) drops egress 443 to the PE private IP | NSG flow logs (§ 6 query A) or `az network nsg rule list -g $RG --nsg-name $NSG -o table` looking for explicit `Deny *:443` outbound | Add an `Allow *:443 outbound to ` rule **above** any deny. Re-test from a Bastion VM with `Test-NetConnection -Port 443 -ComputerName .cognitiveservices.azure.com`. |
| `Resolve-DnsName` from inside the spoke returns the public Foundry IP (cloud edge) | PDZ exists somewhere but no VNet link to the spoke VNet, OR link is on the wrong zone name (e.g. `privatelink.openai.azure.com` linked but `privatelink.cognitiveservices.azure.com` not) | `az network private-dns link vnet list --zone-name  -g  -o table` for each of the 6 expected zones | `az network private-dns link vnet create --zone-name  -g  --name -link --virtual-network  --registration-enabled false` for the missing zone(s). 6 zones expected: `services.ai.azure.com`, `openai.azure.com`, `cognitiveservices.azure.com`, `search.windows.net`, `blob.core.windows.net`, `documents.azure.com`. |
| Hosted-agent create 403 with `MI provisioning failed` | The caller user / SP lacks `Managed Identity Operator` on the Foundry account | `az role assignment list --assignee  --scope /subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.CognitiveServices/accounts/$ACCT -o table` | Grant per `foundry-vnet-deploy § 8b`. Propagation: 5-15 min. |
| Hosted-agent create 403 with `NIC provisioning failed` | The caller lacks `Network Contributor` on the agent injection subnet | Same as above, but scope is `/subscriptions/$SUB/resourceGroups/$VNET_RG/providers/Microsoft.Network/virtualNetworks/$VNET/subnets/$AGENT_SUBNET` | Grant `Network Contributor` per `foundry-vnet-deploy § 8b`. **Do not** grant at VNet scope — minimum needed is the subnet. |
| APIM call (from inside Foundry, through Citadel hub) returns 404 / 503 | `privatelink.azure-api.net` PDZ not linked to the spoke VNet, so the APIM hostname resolves to a public IP unreachable from the private-only spoke | `Resolve-DnsName .azure-api.net` from the spoke → public IP = unlinked. `az network private-dns link vnet list --zone-name privatelink.azure-api.net -g ` confirms. | Link the zone to the spoke VNet (one-shot `az network private-dns link vnet create`). See `citadel-spoke-onboarding` for the canonical command including the JWT-product-policy follow-up. |
| Project connection (Storage / Cosmos / AISearch) shows red in portal, agent runtime returns 403 on access | Target resource still has `publicNetworkAccess: Enabled` OR PE connection is `Pending` (not Approved) OR the project MI lacks the data-plane role | `az storage account show -n $STG --query "{public:publicNetworkAccess, peConns:privateEndpointConnections[].privateLinkServiceConnectionState.status}"` (analogous for Cosmos / Search) | Set `publicNetworkAccess: Disabled` once the PE is approved. Approve any `Pending` PE with `az network private-endpoint-connection approve …`. RBAC: per `foundry-vnet-deploy § 11.10` (6 role assignments). |
| VNet peering shows `Disconnected` after a recent change | Other-side peering was deleted, OR an address-space update on one side broke the contract (overlap, or shrink), OR cross-tenant peering credential expired | `az network vnet peering show -g $RG --vnet-name $VNET -n $PEER --query peeringState` | Both sides must re-peer. Spoke side: delete + recreate `az network vnet peering`. Hub side: ask hub team to re-run their reverse-peering command (`foundry-vnet-deploy § 8d` emits the `hubReversePeeringCommand` output for this). Verify address-space prefixes haven't drifted. |
| `Subnet 'agent-subnet' delegation 'Microsoft.App/environments' not allowed` on first deploy | Region doesn't yet have `Microsoft.App` registered, or the subnet was pre-created without the delegation | `az provider show -n Microsoft.App --query registrationState` and `az network vnet subnet show … --query delegations` | `az provider register -n Microsoft.App` (idempotent, takes 1-5 min). Add the delegation: `az network vnet subnet update … --delegations Microsoft.App/environments`. |

> **Matrix discipline.** The matrix is **deliberately capped at 10
> rows**. If a symptom is application-layer (model returns wrong text,
> SDK raises a TypeError, AppInsights shows the call but with bad data),
> it goes in § 8, not here.

## 5. Pre-flight at scale (cross-subscription DNS)

When the spoke is deployed into an enterprise hub-and-spoke topology,
the 6 Foundry private DNS zones are typically owned by the **platform
team in a separate connectivity hub subscription**. The deployment
principal needs RBAC in **both** subscriptions — one for the spoke
resources, one for the hub PDZs — or the deploy fails with
`InvalidPrivateDnsZoneIds`. This is the canonical hub-and-spoke pattern
documented in [CAF — Private Link and DNS integration at scale](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/azure-best-practices/private-link-and-dns-integration-at-scale#private-link-and-dns-integration-in-hub-and-spoke-network-architectures).

| Subscription | Resource | Required role | Why |
|---|---|---|---|
| Spoke (workload) | Spoke resource group | `Contributor` (or finer-grained `Cognitive Services Contributor` + `Network Contributor`) | Deploy the AI Services account, project, model deployment, spoke VNet, PEs, NICs |
| Spoke (workload) | Spoke VNet | `Network Contributor` | Subnet delegation `Microsoft.App/environments`, PE NIC injection, agent NIC creation |
| Hub (connectivity) | Each of the 6 `privatelink.*` PDZs (or `privatelink.azure-api.net` if Citadel) | `Private DNS Zone Contributor` (per-zone, NOT RG-wide) | Create the VNet link from each zone to the spoke VNet so `Resolve-DnsName` from the spoke returns private IPs |
| Hub (connectivity) | Hub VNet | `Network Contributor` (on the hub side, for the hub team) | Create the **reverse** hub→spoke peering (Citadel only — see `foundry-vnet-deploy § 8d`) |
| Foundry account scope | The AI Services account | `Managed Identity Operator` (caller of hosted-agent create) | Per-user grant; provisioning of agent-instance MIs |
| Agent subnet scope | The injection subnet | `Network Contributor` (caller of hosted-agent create) | Per-user grant; agent NIC creation in the delegated subnet |

If the deploy fails with `InvalidPrivateDnsZoneIds`, run this in the
**hub subscription** to confirm the grant exists:

```bash
az account set -s "$HUB_SUB"
az role assignment list \
  --assignee "$DEPLOYER_OBJECT_ID" \
  --scope "/subscriptions/${HUB_SUB}/resourceGroups/${HUB_DNS_RG}/providers/Microsoft.Network/privateDnsZones/privatelink.cognitiveservices.azure.com" \
  -o table
```

If empty, ask the hub team to grant on all 6 zones (or all 7 if Citadel
APIM is in scope). Propagation: 5-15 min before retry.

## 6. Diagnostic Kusto queries

All four queries assume a Log Analytics workspace receives diagnostic
logs from the network resources (NSG flow logs v2, Azure Firewall, the
Cognitive Services account). If a query returns "table not found", the
diagnostic setting is missing — that is a Day-0 setup gap, not a
runtime failure. See `foundry-observability` for diagnostic-setting
wiring.

### A. NSG flow log denies on the agent subnet

```kusto
AzureNetworkAnalytics_CL
| where TimeGenerated > ago(1h)
| where SubType_s == "FlowLog"
| where FlowStatus_s == "D"            // 'D' = denied by NSG
| where SrcIP_s startswith "10."        // adjust to the agent subnet CIDR
   or DestIP_s startswith "10."
| project TimeGenerated, NSGRule_s, SrcIP_s, SrcPort_d,
          DestIP_s, DestPort_d, L7Protocol_s, FlowDirection_s
| order by TimeGenerated desc
| take 50
```

> **Fallback when NSG flow logs v2 isn't enabled** — use
> `AzureDiagnostics | where Category == 'NetworkSecurityGroupFlowEvent'`
> and parse the embedded JSON. v1 is significantly noisier; enabling v2
> is preferred (`az network watcher flow-log create --location 
> --enabled-nsg  --workspace  --version 2`).

### B. Azure Firewall denies for Foundry-bound traffic

If the agent subnet egresses through an Azure Firewall (hub-and-spoke
topology), denies show up here:

```kusto
AzureDiagnostics
| where TimeGenerated > ago(1h)
| where Category in ("AzureFirewallApplicationRule", "AzureFirewallNetworkRule")
| where OperationName == "AzureFirewallRuleLog"
| where msg_s contains "Deny"
| where msg_s contains "cognitiveservices.azure.com"
   or msg_s contains "services.ai.azure.com"
   or msg_s contains "openai.azure.com"
| project TimeGenerated, msg_s, Category
| order by TimeGenerated desc
| take 50
```

If you see denies, add an Application Rule allowing the matching FQDNs
(prefer the FQDN form over the IP — the private endpoint IP can rotate
on the platform side without notice).

### C. Foundry account `Failed` events from the activity log

When a hosted-agent create or a model deployment fails at the ARM layer
(403, conflict, validation error), this query surfaces the activity-log
trail:

```kusto
AzureActivity
| where TimeGenerated > ago(24h)
| where ResourceProvider == "Microsoft.CognitiveServices"
| where ActivityStatusValue == "Failed"
| project TimeGenerated, OperationNameValue, ActivitySubstatusValue,
          Caller, ResourceId, Properties
| order by TimeGenerated desc
| take 20
```

### D. Role-assignment audit for a given principal (JMESPath, not KQL)

Activity-log-based RBAC audits are slow and lag by minutes. For
real-time RBAC introspection, use the ARM

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [aiappsgbb](https://github.com/aiappsgbb)
- **Source:** [aiappsgbb/awesome-gbb](https://github.com/aiappsgbb/awesome-gbb)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-aiappsgbb-awesome-gbb-foundry-network-runbook
- Seller: https://agentstack.voostack.com/s/aiappsgbb
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.