# Rai Health

> Guides diagnosis of RAI engine performance, failed transactions, CDC/data-stream health, and CDC engine management. Use when a reasoner is slow or queuing, a transaction or batch has failed, a CDC stream is suspended or quarantined, or CDC engine sizing/recovery is needed.

- **Type:** Skill
- **Install:** `agentstack add skill-relationalai-rai-agent-skills-rai-health`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [RelationalAI](https://agentstack.voostack.com/s/relationalai)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [RelationalAI](https://github.com/RelationalAI)
- **Source:** https://github.com/RelationalAI/rai-agent-skills/tree/main/plugins/rai/skills/rai-health
- **Website:** https://relational.ai

## Install

```sh
agentstack add skill-relationalai-rai-agent-skills-rai-health
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

## Summary

**What:** A process skill for diagnosing RAI operational health across four domains: reasoner
performance (memory/CPU/demand), failed transactions, CDC / data-stream health, and CDC engine
management. Each domain has its own step with decision tables and remediation actions.

**When to use:**
- Reasoner is slow, stuck, or queuing; need to check memory, CPU, or demand metrics
- Observability views need setup, role grants, or dashboard/alerting work
- A transaction was aborted; `get_transaction_problems`, `get_own_transaction_problems`,
  or `get_load_errors` must be called
- Batch processing has failed and load errors need inspection
- A CDC task is suspended, a data stream is quarantined, or `resume_cdc` is needed
- CDC engine needs resizing (`alter_cdc_engine_size`) or force-deletion

**When NOT to use:**
- Writing PyRel models or query logic → see `rai-pyrel-coding`
- Configuring authentication or initial RAI setup → see `rai-setup`
- Managing solver optimization problems → see `rai-prescriptive-solver-management`

**Overview (process steps):**
1. Verify observability is set up (events view registered and healthy)
2. Query the three metric views: memory, CPU, demand
3. Apply threshold-based decision rules and prescribe the exact remediation action
4. Diagnose a failed transaction (get_transaction, get_load_errors, owner-restriction pitfall)
5. Diagnose CDC / data stream health (errors, batches, quarantine recovery, resume_cdc)
6. Manage the CDC engine (alter_cdc_engine_size, force delete, cdc_status)

> **Navigation:** Steps 1–3 cover reasoner health only. For CDC/stream issues go directly to
> **Step 5**. For transaction failures go directly to **Step 4**. For CDC engine sizing or
> force-delete go directly to **Step 6**.

---

## Quick Reference

### The Three Metric Views (all in `OBSERVABILITY_PREVIEW`)

| View | Key Column | Healthy Signal |
|------|-----------|----------------|
| `logic_reasoner__memory_utilization` | `MEMORY_UTILIZATION` (0.0–1.0) |  1.0 = queuing) |

**Quickest health check — all three metrics joined, last hour:**
```sql
SELECT
  m.REASONER_NAME,
  m.TIMESTAMP,
  m.MEMORY_UTILIZATION,
  c.CPU_UTILIZATION,
  d.DEMAND,
  d.REASONER_CAPACITY
FROM relationalai.observability_preview.logic_reasoner__memory_utilization m
JOIN relationalai.observability_preview.logic_reasoner__cpu_utilization c
  ON m.REASONER_ID = c.REASONER_ID AND m.TIMESTAMP = c.TIMESTAMP
JOIN relationalai.observability_preview.logic_reasoner__demand d
  ON m.REASONER_ID = d.REASONER_ID AND m.TIMESTAMP = d.TIMESTAMP
WHERE m.TIMESTAMP >= DATEADD(hour, -1, CURRENT_TIMESTAMP())
ORDER BY m.TIMESTAMP DESC;
```

> **Always include a time-range filter.** Querying without `WHERE timestamp >= ...` scans the entire
> Event Table and incurs high Snowflake compute costs.

---

## Step 1 — Verify Observability Is Active

Before querying metrics, confirm the events view is registered and data is flowing.

```sql
CALL relationalai.app.CHECK_EVENTS_VIEW_STATUS();
```

| Status | Meaning | Action |
|--------|---------|--------|
| `Events view active` | Healthy, events flowing | None |
| `No events view registered` | Setup not done | Follow setup in `references/setup-guide.md` |
| `ERROR` | Configuration broken | Fix per error message reported |

> Run `CHECK_EVENTS_VIEW_STATUS()` whenever observability views return unexpected or empty results —
> it diagnoses most configuration issues automatically.

---

## Step 2 — Query Each Metric

### Memory Utilization
```sql
SELECT REASONER_NAME,
       AVG(MEMORY_UTILIZATION) AS avg_mem,
       MAX(MEMORY_UTILIZATION) AS peak_mem
FROM relationalai.observability_preview.logic_reasoner__memory_utilization
WHERE timestamp >= DATEADD(hour, -24, CURRENT_TIMESTAMP())
GROUP BY REASONER_NAME;
```

### CPU Utilization
```sql
SELECT REASONER_NAME,
       AVG(CPU_UTILIZATION) AS avg_cpu,
       MAX(CPU_UTILIZATION) AS peak_cpu
FROM relationalai.observability_preview.logic_reasoner__cpu_utilization
WHERE timestamp >= DATEADD(hour, -24, CURRENT_TIMESTAMP())
GROUP BY REASONER_NAME;
```

### Demand (Queue Pressure)
```sql
SELECT REASONER_NAME,
       AVG(DEMAND)   AS avg_demand,
       MAX(DEMAND)   AS peak_demand,
       REASONER_CAPACITY
FROM relationalai.observability_preview.logic_reasoner__demand
WHERE timestamp >= DATEADD(hour, -24, CURRENT_TIMESTAMP())
GROUP BY REASONER_NAME, REASONER_CAPACITY;
```

> **Interpret patterns, not isolated spikes.** Utilization is naturally spiky. The key question is:
> does *almost every* workload run exceed the threshold? Isolated peaks are normal; consistent
> exceedance across runs is the signal to act.

---

## Step 3 — Health Verdicts and Actions

### OK — HEALTHY — No Action
**Signals:** `MEMORY_UTILIZATION`  0.80 on most workload runs
- `CPU_UTILIZATION` consistently > 0.95
- `CPU_UTILIZATION` = 1.0 AND `MEMORY_UTILIZATION` = 1.0 AND `DEMAND` > 1.0 (all hard limits hit)

**Action:** Upgrade reasoner size. No in-place resize exists — delete and recreate.
```bash
rai reasoners:suspend --type Logic --name 
rai reasoners:delete  --type Logic --name 
rai reasoners:create  --type Logic --name  --size 
```

---

### ELEVATED — PLAN TO RESIZE — Proactive Warning
**Signals:** `CPU_UTILIZATION` consistently 0.85–0.95 (below critical but limited headroom for bursts).

**Action:** Schedule a resize during a low-traffic window before the next traffic spike. No immediate
action required.

---

### WARNING — QUEUING — Review Job Volume / Split Across Reasoners
**Signals:** `DEMAND` consistently > 1.0 (more jobs than available queue slots).

**Action:**
1. Investigate root cause: is a batch job or burst flooding the queue?
2. If higher concurrency is genuinely needed, route different job types to separate reasoner instances
   (send different jobs to different reasoners — do not simply upsize).
3. Upsizing is not the default fix for queuing — it is a demand and routing problem.

---

### INFORMATIONAL — UNDERUTILIZED — Downsize to Save Cost
**Signals:** `CPU_UTILIZATION` 
rai reasoners:delete  --type Logic --name 
rai reasoners:create  --type Logic --name  --size 
```

---

### NOMINAL — IDLE — Suspend or Lower Auto-Suspend Threshold
**Signals:** `DEMAND` = 0 for extended periods.

**Action:** Suspend the reasoner or reduce its `auto_suspend` threshold to stop billing for idle time.
```bash
rai reasoners:suspend --type Logic --name 
```

---

## Step 4 — Diagnose a Failed Transaction

Use these procedures when a transaction appears stuck, aborted, or when load errors are reported.

### Fetch a Transaction by ID

```sql
CALL relationalai.api.get_transaction('');
```

Returns the full transaction record including status, owner, start/end timestamps, and error detail.

### Get Transaction Problems

```sql
-- Problems for any transaction (requires admin-level role)
CALL relationalai.api.get_transaction_problems('');

-- Problems for transactions you own (end-user role)
CALL relationalai.api.get_own_transaction_problems('');
```

| Procedure | Accessible by | Returns |
|-----------|--------------|---------|
| `get_transaction_problems` | Admin roles | All transactions |
| `get_own_transaction_problems` | Any role | Only caller-owned transactions |

### Get Load Errors

```sql
CALL relationalai.api.get_load_errors('');
```

Returns row-level load errors associated with a transaction: source object, error message, and
affected row count.

> **WARNING — Owner-restriction pitfall:** If `get_transaction_problems` returns **HTTP 400**, check the
> transaction owner before assuming a permissions misconfiguration:
> ```sql
> CALL relationalai.api.get_transaction('');
> ```
> Then use the table below to interpret the result.

| `get_transaction` result | Meaning | Next step |
|--------------------------|---------|-----------|
| `owner` = `cdc.scheduler@erp` | Expected behavior — CDC-owned transactions are not visible to end-user roles by design | Use `SELECT * FROM relationalai.api.cdc_status` or an admin role |
| `owner` = any other identity; called `get_transaction_problems` without an admin role | Permission issue — `get_transaction_problems` requires an admin role | Grant the admin role, or switch to `get_own_transaction_problems` if you own the transaction |
| `owner` = any other identity; caller has an admin role | Genuine API failure — not a permissions problem | Open a support ticket with the transaction ID and full error response |
| `get_transaction` itself returns 400 | Invalid transaction ID, or insufficient role to read any transactions | Verify the transaction ID; if correct, confirm read access to `relationalai.api` |

When `get_transaction` returns an unexpected column or state code, or when inspecting load errors per row, see [transaction-debug.md](references/transaction-debug.md).

---

## Step 5 — Diagnose CDC / Data Stream Health

> **WARNING — Auto-quarantine gotcha:** A stream that has been in `SUSPENDED` state for
> **approximately one month** will be automatically promoted to `QUARANTINED` —
> **without creating any rows in `data_stream_errors`**. The absence of error rows does not
> mean the stream is healthy. Always confirm stream status from `cdc_status` or
> `data_stream_batches` before treating an empty errors result as a clean bill of health.

### Find Your Streams (Start Here)

```sql
SELECT * FROM relationalai.api.cdc_status;
```

Key columns: `stream_name`, `stream_status`, `engine_name`, `engine_status`. Use the `stream_name`
values from this output as `''` in the queries below.

### Check Batch-Level Status

```sql
SELECT stream_name, batch_id, status, error_message, created_at
FROM relationalai.api.data_stream_batches
WHERE stream_name = ''
  AND created_at >= DATEADD(day, -7, CURRENT_TIMESTAMP())
ORDER BY created_at DESC;
```

> Use a 7-day window rather than 24 hours — a quarantined or long-suspended stream may have
> had no batches for days, and a 24-hour filter returns empty output indistinguishable from a
> healthy-but-idle stream.

### Check Stream-Level Errors

```sql
SELECT *
FROM relationalai.api.data_stream_errors
WHERE stream_name = ''
ORDER BY created_at DESC
LIMIT 50;
```

For auto-quarantined streams this may return empty — that is expected. Use
`data_stream_batches` `status` as the authoritative source.

### Stream State Verdicts

| Status | Meaning | Action |
|--------|---------|--------|
| `ACTIVE` | Healthy, batches flowing | None |
| `SUSPENDED` | Paused; no new batches | Call `resume_cdc()` — see below |
| `QUARANTINED` | Permanently paused; data integrity issue | [Follow quarantine recovery flow](references/cdc-recovery.md#quarantine-recovery-runbook) |
| `FAILED` | Batch or load error | Check `data_stream_errors` and `get_load_errors` |

### Resume a Suspended Stream

```sql
CALL relationalai.app.resume_cdc('');
```

### Quarantine Recovery

See [cdc-recovery.md — Quarantine Recovery Runbook](references/cdc-recovery.md#quarantine-recovery-runbook)
for the full step-by-step recovery checklist, schema reference, and official docs link.

---

## Predictive reasoner stuck in data-index init

A freshly created or freshly resumed Predictive reasoner can sit in `STATUS=PROVISIONING` for many minutes while the in-pod data index hydrates from CDC streams — and `gnn.fit()` running against it appears to hang in Step 1 (dataset prep) before reaching "Training job submitted." This is the same per-table CDC stream-sync compounding that affects unwarmed Logic reasoners on first model query, surfacing on the Predictive side as a silent prep-step stall.

### Diagnose (in order)

1. **Reasoner status** — is the pod still hydrating or actually ready?
   ```sql
   CALL RELATIONALAI.API.GET_REASONER('predictive', '');
   ```
   `STATUS=PROVISIONING` → wait (1–3 min typical for warm pools; longer on first-time Snowflake-stream attach). `STATUS=READY` plus a hang → continue to (2) and (3).

2. **CDC stream health** — is a stream the predictive reasoner depends on suspended or quarantined?
   ```sql
   SELECT * FROM relationalai.api.cdc_status;
   ```
   `engine_status != READY` or `stream_status` not `RUNNING` → see § Step 5 — Diagnose CDC / Data Stream Health for the quarantine-recovery / `resume_cdc` runbook. Predictive jobs cannot proceed until the upstream streams are healthy.

3. **Transaction problems** — if a specific transaction id appeared in client logs (often the `Failed to pull data into index: transaction was aborted` wrapper), pull its problems:
   ```sql
   CALL RELATIONALAI.API.GET_OWN_TRANSACTION_PROBLEMS('');
   -- or with admin role: GET_TRANSACTION_PROBLEMS('')
   ```
   See § Step 4 — Diagnose a Failed Transaction for the schema reference and owner-restriction pitfall.

### Recovery

If CDC is healthy and the reasoner has been `PROVISIONING` for more than ~5 minutes with no client-side progress, treat it the same as a stuck worker — `SUSPEND_REASONER` + `RESUME_REASONER_ASYNC` (see § Predictive train jobs stuck QUEUED below). Persistent failure: rebuild on a fresh GPU reasoner via `DELETE_REASONER` + `CREATE_REASONER_ASYNC('predictive', ..., 'GPU_NV_S', OBJECT_CONSTRUCT())`.

---

## Predictive train jobs stuck QUEUED

A predictive train job submitted via `gnn.fit()` can sit in `STATE='QUEUED'` in `RELATIONALAI.API.JOBS` indefinitely while `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` still reports `STATUS='READY'`. The SDK only checks reasoner-pod status before submitting — the in-pod worker queue can be out of sync with that status, and the SDK has no way to detect it (`relationalai_gnns/core/connector.py::_check_engine_availability`).

### Diagnostic ladder

Long-running predictive jobs are usually fine — distinguish stuck from slow before suspending anything. Use the same ladder as `rai-predictive-training` § "Training appears stuck":

1. `CALL RELATIONALAI.API.GET_REASONER('predictive', '')` → `STATUS=READY`?
2. `client.jobs.list("Predictive", name="")` → is there a `RUNNING` train job (with rising `AGE_MIN`), or a `QUEUED` one going stale?
3. `SHOW EXPERIMENTS IN SCHEMA .` → did a new experiment row append within ~60s of the `RUNNING` train job?

A `QUEUED` train job that won't advance while the reasoner reports `READY` is the worker-queue desync this section addresses. Genuine long runs progress through (1) READY → (2) RUNNING → (3) new experiment row.

### Recovery

Suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate `GNN(...)` and resubmit (`gnn.fit()` is idempotent — see `rai-predictive-training` § `gnn.fit()` is idempotent). Use only the supported `RELATIONALAI.API.*` surface — do not invoke `RELATIONALAI.EXPERIMENTAL.*` procedures as a workaround.

```sql
-- 1. Confirm a stuck train job
SELECT ID, STATE, DATEDIFF('minute', CREATED_ON, CURRENT_TIMESTAMP()) AS AGE_MIN
FROM RELATIONALAI.API.JOBS
WHERE STATE IN ('QUEUED','RUNNING')
  AND PAYLOAD LIKE '%"job_type": "train"%'
ORDER BY CREATED_ON ASC;

-- 2. Recycle the worker
CALL RELATIONALAI.API.SUSPEND_REASONER('predictive', '');
CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', '');

-- 3. Wait for STATUS=READY, kill the stuck client, then resubmit (re-instantiate GNN(...))
CALL RELATIONALAI.API.GET_REASONER('predictive', '');
```

If recycling doesn't unstick the worker, **rebuild on a fresh GPU predictive reasoner** — the documented path:

```sql
CALL RELATIONALAI.API.DELETE_REASONER('predictive', '');
CALL RELATIONALAI.API.CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', OBJECT_CONSTRUCT());
-- Poll until STATUS=READY:
CALL RELATIONALAI.API.GET_REASONER('predictive', '');
```

See `rai-predictive-training` § Worker not ready to accept jobs for the matching client-side symptom and § Stalled train job: SDK polls without a timeout for stalled-job forensics.

---

## Step 6 — CDC Engine Management

> **CDC engine ≠ reasoner engine.** The CDC pipeline runs on a dedicated managed engine
> distinct from Logic reasoner engines. `alter_cdc_engine_size` targets only the CDC engi

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [RelationalAI](https://github.com/RelationalAI)
- **Source:** [RelationalAI/rai-agent-skills](https://github.com/RelationalAI/rai-agent-skills)
- **License:** Apache-2.0
- **Homepage:** https://relational.ai

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-relationalai-rai-agent-skills-rai-health
- Seller: https://agentstack.voostack.com/s/relationalai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.