Dt Obs Services

Application Services Skill

Monitor application service performance, health, and runtime-specific metrics using DQL.

---

Core Capabilities

1. Service Performance (RED Metrics)

Monitor service Rate, Errors, Duration using metrics-based timeseries queries.

Key Metrics:

• dt.service.request.response_time - Response time (microseconds)
• dt.service.request.count - Request count
• dt.service.request.failure_count - Failed request count

Common Use Cases:

• Response time monitoring (avg, p50, p95, p99)
• Error rate tracking and spike detection
• Traffic analysis (throughput, peaks, growth)
• Performance degradation detection
• Multi-cluster comparison

Quick Example:

timeseries {
  p95 = percentile(dt.service.request.response_time, 95),
  total_requests = sum(dt.service.request.count),
  failures = sum(dt.service.request.failure_count)
}, by: {dt.service.name}
| fieldsAdd p95_ms = p95[] / 1000, error_rate_pct = (failures[] * 100.0) / total_requests[]

→ For detailed queries: See [references/service-metrics.md](references/service-metrics.md)

2. Advanced Service Analysis

Span-based queries for complex scenarios requiring flexible filtering and custom aggregations.

Use Cases:

• SLA compliance tracking with custom thresholds
• Service health scoring (multi-dimensional)
• Operation/endpoint-level performance analysis
• Custom error classification
• Failure pattern detection with error details

Quick Example:

fetch spans, from: now() - 1h | filter request.is_root_span == true
| fieldsAdd meets_sla = if(request.is_failed == false AND duration  = (), by: {dimensions}`
- **Files:** service-metrics.md, all runtime-specific files

**2. Span-based (fetch spans)**
- **Use for:** Complex filtering, custom logic, detailed analysis
- **Pattern:** `fetch spans | filter request.is_root_span == true | fieldsAdd ... | summarize ...`
- **Files:** service-metrics.md (Advanced Service Analysis section)

**3. Comparison queries**
- Use `append` for baseline comparison
- Use `shift: -15m` for time-shifted baselines
- **Example:** Performance degradation detection

### Response Construction Guidelines

**Always include:**
1. **Metric name(s)** - Clear metric identifiers
2. **Aggregation** - How data is aggregated (avg, sum, percentile)
3. **Grouping** - Dimensions used (`dt.service.name`, `k8s.workload.name`, etc.)
4. **Unit conversion** - Convert microseconds to milliseconds where appropriate
5. **Filtering** - Relevant thresholds or conditions

**When referencing runtime-specific content:**
- **Check** user's technology stack first
- **Provide** only relevant runtime queries (don't overwhelm with all 6 runtimes)
- **Explain** runtime-specific metrics (e.g., "OPcache hit ratio" measures PHP opcode cache efficiency)

---

## Common Workflows

### Workflow: Service Health Check

1. Check response time (RED metrics)
2. Check error rate (RED metrics)
3. Check traffic patterns (RED metrics)
4. If runtime-specific issues suspected → Load runtime-specific reference

### Workflow: SLA Monitoring

1. Define SLA criteria (e.g., < 3s response time AND < 1% error rate)
2. Use span-based query for custom SLA logic
3. Calculate compliance percentage
4. Filter non-compliant services

### Workflow: Service Mesh Analysis

1. Check mesh response time
2. Compare mesh vs direct performance
3. Calculate mesh overhead
4. Analyze mesh failure rates

### Workflow: Runtime Troubleshooting
1. Identify technology stack → Load runtime-specific reference
2. Check memory/GC metrics → threads/goroutines → runtime features

---

## Troubleshooting

| Problem | Cause | Solution |
|---------|-------|----------|
| Response time values look too large | Metric is in microseconds | Divide by 1000 to convert to milliseconds |
| No data for service mesh metrics | Service mesh not configured | Verify mesh sidecar injection is enabled |
| Runtime metrics missing | Wrong technology or no OneAgent | Confirm the runtime is supported and OneAgent is active |
| `dt.smartscape.service` returns SmartscapeId, not name | Need entity name resolution | Use `getNodeName(dt.smartscape.service)` |
| Error rate always zero | Using wrong failure metric | Use `dt.service.request.failure_count`, not custom fields |

---

## References

**Core Service Monitoring:**
- [references/service-metrics.md](references/service-metrics.md) - Complete RED metrics, SLA tracking, service mesh queries

**Runtime-Specific Monitoring:**
- [references/java.md](references/java.md) - Java/JVM monitoring
- [references/nodejs.md](references/nodejs.md) - Node.js monitoring  
- [references/dotnet.md](references/dotnet.md) - .NET CLR monitoring
- [references/python.md](references/python.md) - Python monitoring
- [references/php.md](references/php.md) - PHP monitoring
- [references/go.md](references/go.md) - Go runtime monitoring

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Dynatrace](https://github.com/Dynatrace)
- **Source:** [Dynatrace/dynatrace-for-ai](https://github.com/Dynatrace/dynatrace-for-ai)
- **License:** Apache-2.0
- **Homepage:** https://www.dynatrace.com

Install and usage instructions live in the source repository linked above.