Foundry Network Runbook

Foundry Network Runbook — Diagnose connectivity failures after the deploy succeeds

1. Goal

This runbook is for the on-call engineer who just got the page "Foundry agent is up but inference / project connection / model call is failing over the network." The foundry-vnet-deploy succeeded (or the foundry-caphost-lifecycle create returned Succeeded), the resources exist in the portal — but something at the network layer is wrong: DNS resolves to the wrong IP, an NSG drops the packet, an RBAC role is missing for a private DNS zone, a VNet peering went Disconnected, the caphost subnet is wedged. You have ≤ 30 minutes to isolate the layer before escalating.

This runbook is not a telemetry guide (foundry-observability covers App Insights + OTel wiring), not a deploy guide (foundry-vnet-deploy covers Day-0; foundry-caphost-lifecycle covers Day-2 lifecycle), not an auth / JWT troubleshoot for the Citadel APIM gateway (that's citadel-spoke-onboarding), and not an SDK / quota troubleshoot. If the symptom is not a network-layer symptom, jump to § 8 — the cross-reference index points you at the right skill.

2. When to use this runbook

Trigger this runbook when all of the following are true:

1. The Foundry deployment (or hosted-agent create, or caphost create)

reported Succeeded in ARM and the resources are visible in the portal.

1. Something fails downstream at the network layer: 503 from inference,

DNS resolving to a public IP, project connection showing red, a private endpoint missing or not approved, a fresh redeploy hitting Subnet already in use.

1. You have a baseline of "this worked yesterday" or "the deploy just

completed, why doesn't it work now?" — i.e., the failure mode is operational (post-deploy), not a code or SDK bug.

If the failure is during the Bicep az deployment group create itself, use foundry-vnet-deploy § 10b (safe retry). If the failure is a hosted-agent-creation 403 with MI provisioning failed or NIC provisioning failed, this runbook's § 4 matrix points you back to foundry-vnet-deploy § 8b for the RBAC fix.

3. Pre-flight checklist

Capture this baseline before you start diagnosing. Without it you will mis-attribute symptoms (e.g., blaming DNS when the real failure is a peering that went Disconnected 10 minutes earlier).

Run all 7 commands in order. Each block is read-only — none of them mutate Azure state.

# 1. VNet peerings (spoke side): all MUST be Connected
az network vnet peering list \
  -g "$RG" --vnet-name "$VNET" \
  --query "[].{name:name, state:peeringState, remote:remoteVirtualNetwork.id}" \
  -o table

# 2. Private DNS zones reachable from the spoke
az network private-dns zone list \
  -g "$DNS_RG" \
  --query "[].{name:name, vnetLinks:numberOfVirtualNetworkLinks, records:numberOfRecordSets}" \
  -o table

# 3. Private endpoints in the spoke RG and their approval status
az network private-endpoint list -g "$RG" \
  --query "[].{name:name, target:privateLinkServiceConnections[0].privateLinkServiceId, status:privateLinkServiceConnections[0].privateLinkServiceConnectionState.status}" \
  -o table

# 4. Foundry account network state (public access + injection subnet)
az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUB}/resourceGroups/${RG}/providers/Microsoft.CognitiveServices/accounts/${ACCT}?api-version=2025-04-01-preview" \
  --query "{public:properties.publicNetworkAccess, injections:properties.networkInjections}" -o json

# 5. Project MI role assignments (the principal the agent runtime runs as)
PRINCIPAL_ID=$(az rest --method GET \
  --url "https://management.azure.com/subscriptions/${SUB}/resourceGroups/${RG}/providers/Microsoft.CognitiveServices/accounts/${ACCT}/projects/${PROJ}?api-version=2025-04-01-preview" \
  --query "identity.principalId" -o tsv)

az role assignment list --assignee "$PRINCIPAL_ID" --all \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table

# 6. Agent subnet binding (serviceAssociationLink + delegation)
az network vnet subnet show \
  -g "$VNET_RG" --vnet-name "$VNET" -n "$AGENT_SUBNET" \
  --query "{name:name, sal:serviceAssociationLinks, delegations:delegations[].serviceName}"

# 7. DNS resolution FROM INSIDE the spoke (run on a Bastion VM, peered VM,
#    or VPN-connected host — NOT from your laptop over the public internet)
Resolve-DnsName "${ACCT}.cognitiveservices.azure.com"
# MUST return a 10.x / 172.16-31.x / 192.168.x private IP.
# Public IP back = private endpoint missing OR DNS zone not VNet-linked.

If any of the 7 baseline checks already shows an anomaly, jump straight to the matching row in § 4.

4. Symptom → cause → fix matrix

Network-layer symptoms only. For application-layer / SDK / quota issues see § 8 (cross-references).

| Symptom | Likely cause | Diagnostic | Fix | |---|---|---|---| | InvalidPrivateDnsZoneIds at az deployment group create time | Deployment principal lacks Private DNS Zone Contributor on one of the 6 hub PDZs | az role assignment list --assignee --scope /subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones/ -o table | Grant Private DNS Zone Contributor on each zone in the hub subscription (see § 5). Cap: per-zone, not RG-wide. | | Subnet 'agent-subnet' is already in use by capability host on a second deploy into the same VNet | Caphost subnet still bound by a serviceAssociationLink from a prior caphost / account; soft-delete or caphost-only DELETE does not release it | az network vnet subnet show … --query serviceAssociationLinks returns non-empty | Full account purge — see foundry-caphost-lifecycle § 8 for the verified GA CLI sequence and § 9 for the redeploy guard. | | Agent inference returns HTTP 503 within ~1s, no trace in AppInsights | No private endpoint to the AI Services account from the spoke OR PE exists but DNS zone not VNet-linked | Run § 3 step 3 (enumerate PEs) and step 7 (Resolve-DnsName). PE missing = no row for AIServices in step 3. Zone unlinked = step 7 returns a public IP. | Add the missing PE via az network private-endpoint create --group-id account --private-connection-resource-id , then az network private-dns link vnet create for privatelink.cognitiveservices.azure.com, privatelink.openai.azure.com, privatelink.services.ai.azure.com. | | Agent inference hangs ≥ 30s then times out (no HTTP status, no body) | NSG on the agent subnet (or PE subnet) drops egress 443 to the PE private IP | NSG flow logs (§ 6 query A) or az network nsg rule list -g $RG --nsg-name $NSG -o table looking for explicit Deny *:443 outbound | Add an Allow *:443 outbound to rule above any deny. Re-test from a Bastion VM with Test-NetConnection -Port 443 -ComputerName .cognitiveservices.azure.com. | | Resolve-DnsName from inside the spoke returns the public Foundry IP (cloud edge) | PDZ exists somewhere but no VNet link to the spoke VNet, OR link is on the wrong zone name (e.g. privatelink.openai.azure.com linked but privatelink.cognitiveservices.azure.com not) | az network private-dns link vnet list --zone-name -g -o table for each of the 6 expected zones | az network private-dns link vnet create --zone-name -g --name -link --virtual-network --registration-enabled false for the missing zone(s). 6 zones expected: services.ai.azure.com, openai.azure.com, cognitiveservices.azure.com, search.windows.net, blob.core.windows.net, documents.azure.com. | | Hosted-agent create 403 with MI provisioning failed | The caller user / SP lacks Managed Identity Operator on the Foundry account | az role assignment list --assignee --scope /subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.CognitiveServices/accounts/$ACCT -o table | Grant per foundry-vnet-deploy § 8b. Propagation: 5-15 min. | | Hosted-agent create 403 with NIC provisioning failed | The caller lacks Network Contributor on the agent injection subnet | Same as above, but scope is /subscriptions/$SUB/resourceGroups/$VNET_RG/providers/Microsoft.Network/virtualNetworks/$VNET/subnets/$AGENT_SUBNET | Grant Network Contributor per foundry-vnet-deploy § 8b. Do not grant at VNet scope — minimum needed is the subnet. | | APIM call (from inside Foundry, through Citadel hub) returns 404 / 503 | privatelink.azure-api.net PDZ not linked to the spoke VNet, so the APIM hostname resolves to a public IP unreachable from the private-only spoke | Resolve-DnsName .azure-api.net from the spoke → public IP = unlinked. az network private-dns link vnet list --zone-name privatelink.azure-api.net -g confirms. | Link the zone to the spoke VNet (one-shot az network private-dns link vnet create). See citadel-spoke-onboarding for the canonical command including the JWT-product-policy follow-up. | | Project connection (Storage / Cosmos / AISearch) shows red in portal, agent runtime returns 403 on access | Target resource still has publicNetworkAccess: Enabled OR PE connection is Pending (not Approved) OR the project MI lacks the data-plane role | az storage account show -n $STG --query "{public:publicNetworkAccess, peConns:privateEndpointConnections[].privateLinkServiceConnectionState.status}" (analogous for Cosmos / Search) | Set publicNetworkAccess: Disabled once the PE is approved. Approve any Pending PE with az network private-endpoint-connection approve …. RBAC: per foundry-vnet-deploy § 11.10 (6 role assignments). | | VNet peering shows Disconnected after a recent change | Other-side peering was deleted, OR an address-space update on one side broke the contract (overlap, or shrink), OR cross-tenant peering credential expired | az network vnet peering show -g $RG --vnet-name $VNET -n $PEER --query peeringState | Both sides must re-peer. Spoke side: delete + recreate az network vnet peering. Hub side: ask hub team to re-run their reverse-peering command (foundry-vnet-deploy § 8d emits the hubReversePeeringCommand output for this). Verify address-space prefixes haven't drifted. | | Subnet 'agent-subnet' delegation 'Microsoft.App/environments' not allowed on first deploy | Region doesn't yet have Microsoft.App registered, or the subnet was pre-created without the delegation | az provider show -n Microsoft.App --query registrationState and az network vnet subnet show … --query delegations | az provider register -n Microsoft.App (idempotent, takes 1-5 min). Add the delegation: az network vnet subnet update … --delegations Microsoft.App/environments. |

> Matrix discipline. The matrix is deliberately capped at 10 > rows. If a symptom is application-layer (model returns wrong text, > SDK raises a TypeError, AppInsights shows the call but with bad data), > it goes in § 8, not here.

5. Pre-flight at scale (cross-subscription DNS)

When the spoke is deployed into an enterprise hub-and-spoke topology, the 6 Foundry private DNS zones are typically owned by the platform team in a separate connectivity hub subscription. The deployment principal needs RBAC in both subscriptions — one for the spoke resources, one for the hub PDZs — or the deploy fails with InvalidPrivateDnsZoneIds. This is the canonical hub-and-spoke pattern documented in CAF — Private Link and DNS integration at scale.

| Subscription | Resource | Required role | Why | |---|---|---|---| | Spoke (workload) | Spoke resource group | Contributor (or finer-grained Cognitive Services Contributor + Network Contributor) | Deploy the AI Services account, project, model deployment, spoke VNet, PEs, NICs | | Spoke (workload) | Spoke VNet | Network Contributor | Subnet delegation Microsoft.App/environments, PE NIC injection, agent NIC creation | | Hub (connectivity) | Each of the 6 privatelink.* PDZs (or privatelink.azure-api.net if Citadel) | Private DNS Zone Contributor (per-zone, NOT RG-wide) | Create the VNet link from each zone to the spoke VNet so Resolve-DnsName from the spoke returns private IPs | | Hub (connectivity) | Hub VNet | Network Contributor (on the hub side, for the hub team) | Create the reverse hub→spoke peering (Citadel only — see foundry-vnet-deploy § 8d) | | Foundry account scope | The AI Services account | Managed Identity Operator (caller of hosted-agent create) | Per-user grant; provisioning of agent-instance MIs | | Agent subnet scope | The injection subnet | Network Contributor (caller of hosted-agent create) | Per-user grant; agent NIC creation in the delegated subnet |

If the deploy fails with InvalidPrivateDnsZoneIds, run this in the hub subscription to confirm the grant exists:

az account set -s "$HUB_SUB"
az role assignment list \
  --assignee "$DEPLOYER_OBJECT_ID" \
  --scope "/subscriptions/${HUB_SUB}/resourceGroups/${HUB_DNS_RG}/providers/Microsoft.Network/privateDnsZones/privatelink.cognitiveservices.azure.com" \
  -o table

If empty, ask the hub team to grant on all 6 zones (or all 7 if Citadel APIM is in scope). Propagation: 5-15 min before retry.

6. Diagnostic Kusto queries

All four queries assume a Log Analytics workspace receives diagnostic logs from the network resources (NSG flow logs v2, Azure Firewall, the Cognitive Services account). If a query returns "table not found", the diagnostic setting is missing — that is a Day-0 setup gap, not a runtime failure. See foundry-observability for diagnostic-setting wiring.

A. NSG flow log denies on the agent subnet

AzureNetworkAnalytics_CL
| where TimeGenerated > ago(1h)
| where SubType_s == "FlowLog"
| where FlowStatus_s == "D"            // 'D' = denied by NSG
| where SrcIP_s startswith "10."        // adjust to the agent subnet CIDR
   or DestIP_s startswith "10."
| project TimeGenerated, NSGRule_s, SrcIP_s, SrcPort_d,
          DestIP_s, DestPort_d, L7Protocol_s, FlowDirection_s
| order by TimeGenerated desc
| take 50

> Fallback when NSG flow logs v2 isn't enabled — use > AzureDiagnostics | where Category == 'NetworkSecurityGroupFlowEvent' > and parse the embedded JSON. v1 is significantly noisier; enabling v2 > is preferred (az network watcher flow-log create --location > --enabled-nsg --workspace --version 2).

B. Azure Firewall denies for Foundry-bound traffic

If the agent subnet egresses through an Azure Firewall (hub-and-spoke topology), denies show up here:

AzureDiagnostics
| where TimeGenerated > ago(1h)
| where Category in ("AzureFirewallApplicationRule", "AzureFirewallNetworkRule")
| where OperationName == "AzureFirewallRuleLog"
| where msg_s contains "Deny"
| where msg_s contains "cognitiveservices.azure.com"
   or msg_s contains "services.ai.azure.com"
   or msg_s contains "openai.azure.com"
| project TimeGenerated, msg_s, Category
| order by TimeGenerated desc
| take 50

If you see denies, add an Application Rule allowing the matching FQDNs (prefer the FQDN form over the IP — the private endpoint IP can rotate on the platform side without notice).

C. Foundry account Failed events from the activity log

When a hosted-agent create or a model deployment fails at the ARM layer (403, conflict, validation error), this query surfaces the activity-log trail:

AzureActivity
| where TimeGenerated > ago(24h)
| where ResourceProvider == "Microsoft.CognitiveServices"
| where ActivityStatusValue == "Failed"
| project TimeGenerated, OperationNameValue, ActivitySubstatusValue,
          Caller, ResourceId, Properties
| order by TimeGenerated desc
| take 20

D. Role-assignment audit for a given principal (JMESPath, not KQL)

Activity-log-based RBAC audits are slow and lag by minutes. For real-time RBAC introspection, use the ARM

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aiappsgbb
• Source: aiappsgbb/awesome-gbb
• License: MIT

Install and usage instructions live in the source repository linked above.