# Nexus Observability

> >

- **Type:** Skill
- **Install:** `agentstack add skill-aayushostwal-nexus-observability`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [aayushostwal](https://agentstack.voostack.com/s/aayushostwal)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [aayushostwal](https://github.com/aayushostwal)
- **Source:** https://github.com/aayushostwal/nexus/tree/main/skills/observability
- **Website:** https://nexus-ai.aayushostwal.com

## Install

```sh
agentstack add skill-aayushostwal-nexus-observability
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Nexus Observability — API Failure Correlation Engine

Systematically correlate failures across distributed services to identify causal chains, blast radius, and the origin service — before proposing any fix.

---

## Compatibility
- Supporting files: `checklists/investigation-checklist.md`, `anti-patterns/common-mistakes.md`, `validation/output-validation.md`
- Required tools: Read, Bash, Grep
- Optional tools: WebSearch (vendor-specific error lookups)
- Hands off to: `nexus:debugging` once origin service is identified; the `roadmap-planner` agent after fix is agreed upon

---

## Core Principle

**Never assign blame before building the full cross-service timeline.** A cascade always has one origin — fixing a victim while the cause is live means the failure recurs.

---

## Workflow

### Step 1 — Context Acquisition

Collect before reading any logs. Require items 1–5 minimum; ask for all in one message:

| # | Collect | Why |
|---|---------|-----|
| 1 | Verbatim error logs from **all** affected services | Paraphrased logs lose exact timestamps and error codes |
| 2 | Distributed traces (Jaeger/Zipkin/X-Ray/Tempo) for ≥3 failing requests | Shows exact call path and which hop introduced error/latency |
| 3 | Metrics: error rate, p99 latency, RPS, CPU/mem/conn-pool per service | Distinguishes saturation from errors from cascades |
| 4 | Exact first-elevated-error timestamp per service | Required for timeline in Step 2 |
| 5 | Dependency map (direct + indirect call relationships) | Required for blast radius in Step 3 |
| 6 | Changes in the 4h window: deploys, config, feature flags, cron, infra | Most failures are change-triggered |
| 7 | Alert firing history and order | Alert order mirrors propagation direction |

---

### Step 2 — Timeline Construction

Align all timestamps to UTC. Use **first occurrence** per service, not peak. Trust trace timestamps over log timestamps when they differ. Note data gaps explicitly — they are blind spots, not clean windows.

```
| Time (UTC) | Service     | Event                              | Error Rate |
|------------|-------------|------------------------------------|------------|
| T+00:00    | database    | connection pool at 95% utilization | 0%         |
| T+00:47    | database    | connection pool exhausted          | 0%         |
| T+01:03    | api-gateway | upstream timeouts begin            | 2%         |
| T+01:15    | auth-service| DB queries failing, JWT slow       | 8%         |
| T+01:22    | user-service| auth check timeouts                | 34%        |
| T+01:31    | api-gateway | circuit breaker opens on auth      | 78%        |
```

---

### Step 3 — Identify the Origin Service

The origin has the **earliest** elevated error signal. Confirm with all three tests:

| Test | Method | What it proves |
|------|--------|----------------|
| Earliest timestamp | Lowest T+XX:XX across services | Candidate origin |
| No upstream errors at that time | Check if anything calling the candidate was already erroring | Confirms not a victim |
| Error type is local | DB timeout with no downstream = local origin; HTTP 503 from a pass-through = victim | Distinguishes originator from propagator |
| Metrics confirm saturation | CPU/mem/conn-pool maxed at origin time | Saturation precedes errors at the origin |

State the origin as: *"[Service X] is the origin. Its [metric] saturated at [T+XX:XX], [N]s before the first downstream error in [Service Y]."*

If two candidates remain ambiguous, state both — do not guess.

---

### Step 4 — Map Dependency Graph and Failure Mode

Classify using this table:

| Failure Mode | Signature | Meaning |
|-------------|-----------|---------|
| **Linear cascade** | Errors propagate A → B → C with delays matching timeout settings | Upstream timeout drives downstream retry storm |
| **Fan-out cascade** | One service fails; N downstream services spike simultaneously | Shared dependency (DB, cache, auth) failed |
| **Independent failures** | Multiple services spike with no call relationship | Shared infra failure: network, DNS, load balancer, cloud event |
| **Thundering herd** | Error rate drops then re-spikes in waves | Circuit breaker retry storm after brief recovery |
| **Brownout** | Latency rises gradually, error rate follows slowly | Resource saturation (pool, thread, memory) near limit |

Draw the dependency graph and mark each node: `ORIGIN`, `PROPAGATOR`, or `VICTIM`.

```
[database] ← [auth-service] ← [api-gateway] ← [user-service]
                              ↑
                          [billing-service]
```

---

### Step 5 — Root Cause + Blast Radius

**Root cause format (required):**
> *"[Service X] failed because [mechanism] when [trigger], causing [failure mode] that propagated to [affected services], with [N]% of [user-facing operation] requests failing by [T+XX:XX]."*

**Blast radius table:**
```
| Service         | Role        | Peak Error Rate | User Impact            |
|-----------------|-------------|-----------------|------------------------|
| database        | ORIGIN      | 100% timeouts   | Internal only          |
| auth-service    | PROPAGATOR  | 45%             | Login failures         |
| api-gateway     | PROPAGATOR  | 78%             | All authenticated reqs |
| user-service    | VICTIM      | 34%             | Profile page errors    |
```

**Fix table (one action per service):**
```
| Service      | Immediate Action                          | Prevention                          |
|--------------|-------------------------------------------|-------------------------------------|
| database     | Increase pool limit; kill idle conns      | Alert at 80% pool utilization       |
| auth-service | Add circuit breaker on DB calls           | Cache JWT verify results for 30s    |
| api-gateway  | Tune circuit breaker thresholds           | Add fallback for auth timeout       |
```

---

## Signal Patterns

| Observed pattern | Most likely cause |
|-----------------|-------------------|
| All services spike simultaneously (< 1s apart) | Shared infra: network partition, DNS, load balancer, cloud event |
| Errors propagate in waves matching timeout delays | Request-driven cascade — upstream timeout → downstream retry storm |
| Periodic error spikes (regular interval) | Cron job contention, batch overloading shared resource, pool TTL recycling |
| p99 latency rises **before** error rate | Resource saturation building (pool, thread, memory near limit) |
| Error rate rises **before** latency | Hard failure (process crash, network drop) not saturation |
| Error rate drops and re-spikes in waves | Thundering herd — circuit breaker open/close/retry cycle |
| Single service high error rate, all others fine | Isolated failure — route to `nexus:debugging`, do not correlate |
| Error rate high, latency normal | Fast-failing errors (circuit breaker open, auth rejection) |
| Memory/CPU climbs gradually then all services degrade | GC pauses or OOM kills on shared nodes |

---

## Specialization Notes

**Service meshes (Istio/Linkerd):** Check control plane health first — a sick control plane degrades all data plane services simultaneously. `503 upstream connect error` and `upstream reset before response started` are cascade signatures. Envoy proxy logs often contain the causal chain before application logs do.

**Databases (PostgreSQL/MySQL):** Check `max_connections`/`pg_stat_activity` — connection exhaustion is the most common DB-originated cascade. Check slow query log (one long query can block all connection slots). Check replication lag — replica failures appear before primary errors.

**Caches (Redis/Memcached):** Check `rejected_connections` — full connection pool causes DB fallback storms. Check memory/evictions — `maxmemory-policy` evictions cause cache misses that look like cache failure. Check `slowlog` — an expensive command (LRANGE, SMEMBERS) blocks all other operations.

**API gateways/load balancers:** Check backend health check logs — misconfigured health checks mark healthy backends as down. Check connection timeout vs. read timeout mismatch — causes partial failure modes that are hard to correlate.

---

## Output Contract

Every investigation closes with this report. Fill every field; write `none` if not applicable — never leave blank:

```
## Failure Correlation Report

**Outcome:**          [one-line summary]
**Failure Mode:**     [Linear cascade | Fan-out | Independent | Thundering herd | Brownout]
**Origin Service:**   [service name + specific metric/event that triggered failure]
**Trigger:**          [what caused the origin service to fail]
**Timeline:**         [inline table T+00:00 to resolution]
**Causal Chain:**     [A → B → C with failure mode at each hop]
**Blast Radius:**     [table: service | role | peak error rate | user impact]
**Fix — Immediate:**  [per-service immediate actions]
**Fix — Prevention:** [per-service prevention measures]
**Blind Spots:**      [services or time windows with no data]
**Next Step:**        [one clear action for the user]
```

---

## Anti-Patterns

Read `anti-patterns/common-mistakes.md` before starting any investigation.

- Never blame a downstream service without proving its upstream was healthy at failure time.
- Never construct a timeline from summarized logs — raw timestamps only.
- Never propose a fix before completing Steps 1–5.
- Never skip the dependency graph — fan-out cascades look like independent failures without it.
- Never mark complete if `Blind Spots` is empty when data is missing — state what is unknown.
- Never conflate correlation with causation — simultaneous spikes do not imply causation.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [aayushostwal](https://github.com/aayushostwal)
- **Source:** [aayushostwal/nexus](https://github.com/aayushostwal/nexus)
- **License:** MIT
- **Homepage:** https://nexus-ai.aayushostwal.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-aayushostwal-nexus-observability
- Seller: https://agentstack.voostack.com/s/aayushostwal
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.