# Azure Ai Content Safety

> Guidance for Azure AI Content Safety — programmatic content moderation for text, images, multimodal, and generative AI guardrails. Covers Content Safety categories (hate, violence, sexual, self-harm) with severity levels, Prompt Shields for jailbreak and indirect prompt injection detection, groundedness detection (hallucination check vs grounding sources), protected material detection (text and c…

- **Type:** Skill
- **Install:** `agentstack add skill-vinayaklatthe-microsoft-security-skills-azure-ai-content-safety`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vinayaklatthe](https://agentstack.voostack.com/s/vinayaklatthe)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [vinayaklatthe](https://github.com/vinayaklatthe)
- **Source:** https://github.com/vinayaklatthe/microsoft-security-skills/tree/main/skills/azure-ai-content-safety

## Install

```sh
agentstack add skill-vinayaklatthe-microsoft-security-skills-azure-ai-content-safety
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Azure AI Content Safety

Azure AI Content Safety is the platform for **real-time content moderation and generative
AI safety**. It analyzes text, images, and multimodal content for harm categories, blocks
prompt-injection / jailbreak attempts (Prompt Shields), detects ungrounded responses
(Groundedness), and flags protected text/code reproduction.

It pairs with Azure OpenAI as built-in **content filters** (default safety net) and as
configurable guardrails for any AI/LLM application (including non-Microsoft models).

## When to use
Building or hardening any generative AI application — chat assistant, agent, RAG app,
multimodal product — that needs guardrails on inputs and outputs.

**Do not use this skill** for SOC threat detection on AI workloads
(`defender-for-cloud-ai`), end-user AI usage governance (`purview-ai-hub`), or generic AI
application architecture.

## Capability map

| Capability | What it does | Where to apply |
|---|---|---|
| **Text moderation** | Classifies text into hate, violence, sexual, self-harm (severity 0–7) | Input + output |
| **Image moderation** | Same categories on images | User uploads, generated images |
| **Multimodal moderation** | Text+image combined context | Multimodal chat |
| **Prompt Shields — User Prompt** | Detects jailbreak attempts in the user prompt | Input |
| **Prompt Shields — Indirect** | Detects injection in untrusted grounding content (RAG, tool results, web pages) | Pre-grounding step |
| **Groundedness detection** | Checks LLM response against grounding sources for hallucination | Output |
| **Protected material — text** | Detects verbatim copyrighted text | Output |
| **Protected material — code** | Detects reproduction of public-repo code | Code-generation output |
| **Custom categories** | Customer-defined harm categories (e.g., off-brand topics) | Input + output |
| **Blocklists** | Exact-match + regex blocklists | Input + output |

## Approach

1. **Start with default Azure OpenAI content filters.** Every Azure OpenAI deployment
   has a default Content Safety filter at *medium severity* across all four harm
   categories. Don't deploy without it.

2. **Tune severity thresholds per use case.**
   - Children's product: low threshold (block at severity 2).
   - Adult creative app: high threshold (block only at severity 6).
   Author a custom content filter in Azure OpenAI Studio; assign to deployment.

3. **Enable Prompt Shields for any user-facing chat or agent.**
   - **User Prompt Shield** on the incoming prompt.
   - **Indirect Prompt Shield** on grounding content (RAG document chunks, tool
     outputs, fetched web pages) **before** they hit the model. Indirect injection is
     the higher-impact, lower-visibility risk.

4. **Groundedness detection for RAG apps.** After the model produces a response, call
   groundedness detection with the source chunks. Flag/reject ungrounded claims. Adds
   latency (one extra model call) but is the main defense against hallucination in
   high-stakes apps (medical, legal, financial).

5. **Protected material detection** for products that generate text or code at scale:
   - Text: SaaS copywriting tools.
   - Code: developer assistants (Copilot-style).
   Flag and either suppress or annotate the output.

6. **Custom categories + blocklists** for brand- or domain-specific risks (competitor
   mentions, regulated disclosure language, internal codenames). Train custom
   categories via Content Safety Studio with ~100 examples per category.

7. **Architecture pattern (sidecar / middleware).**
   - Request → Prompt Shield → (LLM call) → Output classifiers (harm, groundedness,
     protected material) → Response.
   - Fail-closed: if Content Safety call errors, do not return uncontrolled LLM output.
   - Log every decision (allowed/blocked/flagged) for audit and tuning.

8. **Performance.** Each guardrail adds 50–300ms. Parallelize input and output checks;
   short-circuit on first hard block. Budget for the latency in user expectation.

9. **Operate.** Weekly review of blocked-content samples (with privacy controls).
   Tune thresholds, custom categories, and blocklists. Track false positive / false
   negative rates per category.

## Guardrails
- **Default filters are baseline, not a complete safety strategy.** Prompt Shields,
  groundedness, and custom categories close the real gaps.
- **Don't expose harm-category scores to end users.** A "you said X violence score 6"
  message helps attackers tune jailbreaks.
- **Fail-closed on Content Safety errors.** A 500 from the service should not result in
  unmoderated LLM output reaching users.
- **Indirect prompt injection is the under-appreciated risk.** A user's prompt is
  visible; a malicious string in a fetched web page or document is not. Always shield
  grounding content.
- **Groundedness detection requires you to send the grounding sources.** RAG apps
  natively have them; pure chat apps don't.
- **Custom categories are not a substitute for product policy.** They detect; they
  don't prevent the user from generating harmful content with another tool.
- **Protected material detection is best-effort.** Copyright/IP review is a legal
  process, not an API result.
- **Content Safety is per-call billed.** High-traffic apps need cost modeling; not all
  features run on every call (groundedness is expensive — gate on use case).
- **Pair with Defender for Cloud AI** for SOC-side detection of attempts and patterns
  (`defender-for-cloud-ai`).

## Common anti-patterns
- **"Default filter is enough"** — no jailbreak/injection protection.
- **"Indirect prompt shield skipped because 'all our sources are trusted'"** — they're
  trusted to be there; not trusted to be uncompromised. RAG corpora get poisoned.
- **"Groundedness on every chat call regardless of grounding"** — burns cost on pure
  chat without grounding sources to check against.
- **"Logged user prompts in plaintext for tuning"** — privacy/PII exposure. Pseudonymize.
- **"Returned the Content Safety severity score in error messages"** — gives attacker
  a tuning oracle.
- **"Custom categories trained on 5 examples"** — no model lift. Need ~100 minimum.
- **"Blocklists used as the main strategy"** — misses paraphrases. Combine with
  classifier categories.
- **"Content Safety only on output"** — input attacks (jailbreaks) reach the model.

## Example prompts
- `Architect Content Safety guardrails for a RAG-based customer support assistant on
  Azure OpenAI.`
- `Configure Prompt Shields for a Foundry agent with internet-grounded tools.`
- `Tune content filter severity thresholds for a teen-audience product.`
- `Add groundedness detection to flag hallucinations in our medical Q&A assistant.`
- `Train a custom category for "competitor mention" and integrate as output blocklist.`
- `Cost and latency model: Content Safety calls per chat turn on a 10K-user app.`
- `IR runbook: how SOC handles "prompt shield blocked attempted jailbreak" pattern
  (paired with Defender for Cloud AI alert).`

## Microsoft Learn
- Content Safety overview: https://learn.microsoft.com/azure/ai-services/content-safety/overview
- Text moderation: https://learn.microsoft.com/azure/ai-services/content-safety/concepts/harm-categories
- Image moderation: https://learn.microsoft.com/azure/ai-services/content-safety/quickstart-image
- Prompt Shields: https://learn.microsoft.com/azure/ai-services/content-safety/concepts/jailbreak-detection
- Groundedness detection: https://learn.microsoft.com/azure/ai-services/content-safety/concepts/groundedness
- Protected material: https://learn.microsoft.com/azure/ai-services/content-safety/concepts/protected-material
- Custom categories: https://learn.microsoft.com/azure/ai-services/content-safety/concepts/custom-categories
- Azure OpenAI content filtering: https://learn.microsoft.com/azure/ai-services/openai/concepts/content-filter
- Content Safety Studio: https://learn.microsoft.com/azure/ai-services/content-safety/studio-quickstart

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vinayaklatthe](https://github.com/vinayaklatthe)
- **Source:** [vinayaklatthe/microsoft-security-skills](https://github.com/vinayaklatthe/microsoft-security-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-vinayaklatthe-microsoft-security-skills-azure-ai-content-safety
- Seller: https://agentstack.voostack.com/s/vinayaklatthe
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.