# Observability Datadog

> Datadog log and metrics analysis. Use when querying Datadog logs, metrics, or APM data. Provides scripts and query syntax reference.

- **Type:** Skill
- **Install:** `agentstack add skill-swapnildahiphale-opensre-observability-datadog`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [swapnildahiphale](https://agentstack.voostack.com/s/swapnildahiphale)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [swapnildahiphale](https://github.com/swapnildahiphale)
- **Source:** https://github.com/swapnildahiphale/OpenSRE/tree/main/sre-agent/.claude/skills/observability-datadog
- **Website:** https://opensre.in

## Install

```sh
agentstack add skill-swapnildahiphale-opensre-observability-datadog
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Datadog Analysis

## Authentication

**IMPORTANT**: Credentials are injected automatically by a proxy layer. Do NOT check for `DATADOG_API_KEY` or `DATADOG_APP_KEY` in environment variables - they won't be visible to you. Just run the scripts directly; authentication is handled transparently.

Configuration environment variables you CAN check (non-secret):
- `DATADOG_SITE` - Datadog site (e.g., `us5.datadoghq.com`, `datadoghq.eu`)

---

## MANDATORY: Statistics-First Investigation

**NEVER dump raw logs.** Always follow this pattern:

```
STATISTICS → SAMPLE → PATTERNS → CORRELATE
```

1. **Statistics First** - Know volume, error rate, and top patterns before sampling
2. **Strategic Sampling** - Choose the right strategy based on statistics
3. **Pattern Extraction** - Cluster similar errors to find root causes
4. **Context Correlation** - Investigate around anomaly timestamps

## Available Scripts

All scripts are in `.claude/skills/observability-datadog/scripts/`

### PRIMARY INVESTIGATION SCRIPTS

#### get_statistics.py - ALWAYS START HERE
Comprehensive statistics with pattern extraction.
```bash
python .claude/skills/observability-datadog/scripts/get_statistics.py [--service SERVICE] [--time-range MINUTES]

# Examples:
python .claude/skills/observability-datadog/scripts/get_statistics.py --time-range 60
python .claude/skills/observability-datadog/scripts/get_statistics.py --service payment
```

Output includes:
- Total count, error count, error rate percentage
- Status distribution (info, warn, error)
- Top services by log volume
- **Top error patterns** (crucial for quick triage)
- Actionable recommendation

#### sample_logs.py - Strategic Sampling
Choose the right sampling strategy based on statistics.
```bash
python .claude/skills/observability-datadog/scripts/sample_logs.py --strategy STRATEGY [--service SERVICE] [--limit N]

# Strategies:
#   errors_only   - Only error logs (default for incidents)
#   warnings_up   - Warning and error logs
#   around_time   - Logs around a specific timestamp
#   all           - All log levels

# Examples:
python .claude/skills/observability-datadog/scripts/sample_logs.py --strategy errors_only --service payment
python .claude/skills/observability-datadog/scripts/sample_logs.py --strategy around_time --timestamp "2026-01-27T05:00:00Z" --window 5
```

---

## Datadog Query Language (DQL)

### Basic Filters
```
# Service filter
service:payment

# Status filter
status:error
status:warn

# Host filter
host:web-server-01

# Combine with AND (space) or OR
service:payment status:error
service:payment OR service:checkout
```

### Facet Filters
```
# Tag filter
env:production
version:1.2.3

# Attribute filter
@http.status_code:>=500
@duration:>1000

# Wildcard
service:payment-*
```

### Time Ranges
```
# Relative
@timestamp:[now-1h TO now]

# Absolute
@timestamp:[2026-01-27T00:00:00Z TO 2026-01-27T12:00:00Z]
```

### Common Patterns
```
# All errors in last hour
status:error

# Errors for specific service
service:api-gateway status:error

# Slow requests (>1s)
@duration:>1000000

# HTTP 5xx errors
@http.status_code:>=500

# Exceptions
*exception* OR *error* OR *failed*
```

---

## Investigation Workflow

### Standard Incident Investigation

```
┌─────────────────────────────────────────────────────────────┐
│ 1. STATISTICS FIRST (mandatory)                              │
│    python get_statistics.py --service               │
│    → Know volume, error rate, top patterns                   │
└─────────────────────────────────────────────────────────────┘
                             │
                             ▼
                     High Error Rate?
               ┌─────────────┴─────────────┐
               │                           │
       YES (>5%)                           NO
               │                           │
               ▼                           ▼
┌─────────────────────────────┐  ┌───────────────────────────────────────────┐
│ 2. FAST PATH                │  │ 2. TARGETED INVESTIGATION                 │
│    Sample errors directly   │  │    Filter by specific criteria            │
│    python sample_logs.py    │  │    python sample_logs.py --strategy all   │
│    --strategy errors_only   │  │    → Look for anomalies                   │
└─────────────────────────────┘  └───────────────────────────────────────────┘
```

### Quick Commands Reference

| Goal | Command |
|------|---------|
| Start investigation | `get_statistics.py --service X` |
| Sample errors only | `sample_logs.py --strategy errors_only --service X` |
| Investigate spike | `sample_logs.py --strategy around_time --timestamp T` |
| All logs | `sample_logs.py --strategy all --service X --limit 20` |

---

## Metrics Query Syntax

### Basic Structure
```
aggregation:metric_name{tag_filters}
```

### Aggregations
```
avg:   - Average across series
sum:   - Sum across series
min:   - Minimum value
max:   - Maximum value
p50:   - 50th percentile (APM)
p95:   - 95th percentile (APM)
p99:   - 99th percentile (APM)
```

### Common Metrics
```
# System
avg:system.cpu.user{service:X}
avg:system.mem.used{service:X}

# APM (traces)
sum:trace.http.request.hits{service:X}.as_rate()
sum:trace.http.request.errors{service:X}.as_rate()
p95:trace.http.request.duration{service:X}
```

---

## Anti-Patterns to Avoid

1. ❌ **NEVER skip statistics** - `get_statistics.py` is MANDATORY first step
2. ❌ **Unbounded queries** - Always specify time ranges and limits
3. ❌ **Fetching all logs** - Use sampling strategies, not unbounded searches
4. ❌ **Ignoring error rate** - High error rate means immediate investigation
5. ❌ **Missing service filter** - For multi-service apps, always filter by service

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [swapnildahiphale](https://github.com/swapnildahiphale)
- **Source:** [swapnildahiphale/OpenSRE](https://github.com/swapnildahiphale/OpenSRE)
- **License:** Apache-2.0
- **Homepage:** https://opensre.in

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-swapnildahiphale-opensre-observability-datadog
- Seller: https://agentstack.voostack.com/s/swapnildahiphale
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.