Rai Health

Summary

What: A process skill for diagnosing RAI operational health across four domains: reasoner performance (memory/CPU/demand), failed transactions, CDC / data-stream health, and CDC engine management. Each domain has its own step with decision tables and remediation actions.

When to use:

• Reasoner is slow, stuck, or queuing; need to check memory, CPU, or demand metrics
• Observability views need setup, role grants, or dashboard/alerting work
• A transaction was aborted; get_transaction_problems, get_own_transaction_problems,

or get_load_errors must be called

• Batch processing has failed and load errors need inspection
• A CDC task is suspended, a data stream is quarantined, or resume_cdc is needed
• CDC engine needs resizing (alter_cdc_engine_size) or force-deletion

When NOT to use:

• Writing PyRel models or query logic → see rai-pyrel-coding
• Configuring authentication or initial RAI setup → see rai-setup
• Managing solver optimization problems → see rai-prescriptive-solver-management

Overview (process steps):

1. Verify observability is set up (events view registered and healthy)
2. Query the three metric views: memory, CPU, demand
3. Apply threshold-based decision rules and prescribe the exact remediation action
4. Diagnose a failed transaction (gettransaction, getload_errors, owner-restriction pitfall)
5. Diagnose CDC / data stream health (errors, batches, quarantine recovery, resume_cdc)
6. Manage the CDC engine (altercdcenginesize, force delete, cdcstatus)

> Navigation: Steps 1–3 cover reasoner health only. For CDC/stream issues go directly to > Step 5. For transaction failures go directly to Step 4. For CDC engine sizing or > force-delete go directly to Step 6.

---

Quick Reference

The Three Metric Views (all in OBSERVABILITY_PREVIEW)

| View | Key Column | Healthy Signal | |------|-----------|----------------| | logic_reasoner__memory_utilization | MEMORY_UTILIZATION (0.0–1.0) | 1.0 = queuing) |

Quickest health check — all three metrics joined, last hour:

SELECT
  m.REASONER_NAME,
  m.TIMESTAMP,
  m.MEMORY_UTILIZATION,
  c.CPU_UTILIZATION,
  d.DEMAND,
  d.REASONER_CAPACITY
FROM relationalai.observability_preview.logic_reasoner__memory_utilization m
JOIN relationalai.observability_preview.logic_reasoner__cpu_utilization c
  ON m.REASONER_ID = c.REASONER_ID AND m.TIMESTAMP = c.TIMESTAMP
JOIN relationalai.observability_preview.logic_reasoner__demand d
  ON m.REASONER_ID = d.REASONER_ID AND m.TIMESTAMP = d.TIMESTAMP
WHERE m.TIMESTAMP >= DATEADD(hour, -1, CURRENT_TIMESTAMP())
ORDER BY m.TIMESTAMP DESC;

> Always include a time-range filter. Querying without WHERE timestamp >= ... scans the entire > Event Table and incurs high Snowflake compute costs.

---

Step 1 — Verify Observability Is Active

Before querying metrics, confirm the events view is registered and data is flowing.

CALL relationalai.app.CHECK_EVENTS_VIEW_STATUS();

| Status | Meaning | Action | |--------|---------|--------| | Events view active | Healthy, events flowing | None | | No events view registered | Setup not done | Follow setup in references/setup-guide.md | | ERROR | Configuration broken | Fix per error message reported |

> Run CHECK_EVENTS_VIEW_STATUS() whenever observability views return unexpected or empty results — > it diagnoses most configuration issues automatically.

---

Step 2 — Query Each Metric

Memory Utilization

SELECT REASONER_NAME,
       AVG(MEMORY_UTILIZATION) AS avg_mem,
       MAX(MEMORY_UTILIZATION) AS peak_mem
FROM relationalai.observability_preview.logic_reasoner__memory_utilization
WHERE timestamp >= DATEADD(hour, -24, CURRENT_TIMESTAMP())
GROUP BY REASONER_NAME;

CPU Utilization

SELECT REASONER_NAME,
       AVG(CPU_UTILIZATION) AS avg_cpu,
       MAX(CPU_UTILIZATION) AS peak_cpu
FROM relationalai.observability_preview.logic_reasoner__cpu_utilization
WHERE timestamp >= DATEADD(hour, -24, CURRENT_TIMESTAMP())
GROUP BY REASONER_NAME;

Demand (Queue Pressure)

SELECT REASONER_NAME,
       AVG(DEMAND)   AS avg_demand,
       MAX(DEMAND)   AS peak_demand,
       REASONER_CAPACITY
FROM relationalai.observability_preview.logic_reasoner__demand
WHERE timestamp >= DATEADD(hour, -24, CURRENT_TIMESTAMP())
GROUP BY REASONER_NAME, REASONER_CAPACITY;

> Interpret patterns, not isolated spikes. Utilization is naturally spiky. The key question is: > does almost every workload run exceed the threshold? Isolated peaks are normal; consistent > exceedance across runs is the signal to act.

---

Step 3 — Health Verdicts and Actions

OK — HEALTHY — No Action

Signals: MEMORY_UTILIZATION 0.80 on most workload runs

• CPU_UTILIZATION consistently > 0.95
• CPU_UTILIZATION = 1.0 AND MEMORY_UTILIZATION = 1.0 AND DEMAND > 1.0 (all hard limits hit)

Action: Upgrade reasoner size. No in-place resize exists — delete and recreate.

rai reasoners:suspend --type Logic --name 
rai reasoners:delete  --type Logic --name 
rai reasoners:create  --type Logic --name  --size 

---

ELEVATED — PLAN TO RESIZE — Proactive Warning

Signals: CPU_UTILIZATION consistently 0.85–0.95 (below critical but limited headroom for bursts).

Action: Schedule a resize during a low-traffic window before the next traffic spike. No immediate action required.

---

WARNING — QUEUING — Review Job Volume / Split Across Reasoners

Signals: DEMAND consistently > 1.0 (more jobs than available queue slots).

Action:

1. Investigate root cause: is a batch job or burst flooding the queue?
2. If higher concurrency is genuinely needed, route different job types to separate reasoner instances

(send different jobs to different reasoners — do not simply upsize).

1. Upsizing is not the default fix for queuing — it is a demand and routing problem.

---

INFORMATIONAL — UNDERUTILIZED — Downsize to Save Cost

Signals: CPU_UTILIZATION rai reasoners:delete --type Logic --name rai reasoners:create --type Logic --name --size

---

### NOMINAL — IDLE — Suspend or Lower Auto-Suspend Threshold
**Signals:** `DEMAND` = 0 for extended periods.

**Action:** Suspend the reasoner or reduce its `auto_suspend` threshold to stop billing for idle time.
```bash
rai reasoners:suspend --type Logic --name 

---

Step 4 — Diagnose a Failed Transaction

Use these procedures when a transaction appears stuck, aborted, or when load errors are reported.

Fetch a Transaction by ID

CALL relationalai.api.get_transaction('');

Returns the full transaction record including status, owner, start/end timestamps, and error detail.

Get Transaction Problems

-- Problems for any transaction (requires admin-level role)
CALL relationalai.api.get_transaction_problems('');

-- Problems for transactions you own (end-user role)
CALL relationalai.api.get_own_transaction_problems('');

| Procedure | Accessible by | Returns | |-----------|--------------|---------| | get_transaction_problems | Admin roles | All transactions | | get_own_transaction_problems | Any role | Only caller-owned transactions |

Get Load Errors

CALL relationalai.api.get_load_errors('');

Returns row-level load errors associated with a transaction: source object, error message, and affected row count.

> WARNING — Owner-restriction pitfall: If get_transaction_problems returns HTTP 400, check the > transaction owner before assuming a permissions misconfiguration: > ``sql > CALL relationalai.api.get_transaction(''); > `` > Then use the table below to interpret the result.

| get_transaction result | Meaning | Next step | |--------------------------|---------|-----------| | owner = cdc.scheduler@erp | Expected behavior — CDC-owned transactions are not visible to end-user roles by design | Use SELECT * FROM relationalai.api.cdc_status or an admin role | | owner = any other identity; called get_transaction_problems without an admin role | Permission issue — get_transaction_problems requires an admin role | Grant the admin role, or switch to get_own_transaction_problems if you own the transaction | | owner = any other identity; caller has an admin role | Genuine API failure — not a permissions problem | Open a support ticket with the transaction ID and full error response | | get_transaction itself returns 400 | Invalid transaction ID, or insufficient role to read any transactions | Verify the transaction ID; if correct, confirm read access to relationalai.api |

When get_transaction returns an unexpected column or state code, or when inspecting load errors per row, see [transaction-debug.md](references/transaction-debug.md).

---

Step 5 — Diagnose CDC / Data Stream Health

> WARNING — Auto-quarantine gotcha: A stream that has been in SUSPENDED state for > approximately one month will be automatically promoted to QUARANTINED — > without creating any rows in data_stream_errors. The absence of error rows does not > mean the stream is healthy. Always confirm stream status from cdc_status or > data_stream_batches before treating an empty errors result as a clean bill of health.

Find Your Streams (Start Here)

SELECT * FROM relationalai.api.cdc_status;

Key columns: stream_name, stream_status, engine_name, engine_status. Use the stream_name values from this output as '' in the queries below.

Check Batch-Level Status

SELECT stream_name, batch_id, status, error_message, created_at
FROM relationalai.api.data_stream_batches
WHERE stream_name = ''
  AND created_at >= DATEADD(day, -7, CURRENT_TIMESTAMP())
ORDER BY created_at DESC;

> Use a 7-day window rather than 24 hours — a quarantined or long-suspended stream may have > had no batches for days, and a 24-hour filter returns empty output indistinguishable from a > healthy-but-idle stream.

Check Stream-Level Errors

SELECT *
FROM relationalai.api.data_stream_errors
WHERE stream_name = ''
ORDER BY created_at DESC
LIMIT 50;

For auto-quarantined streams this may return empty — that is expected. Use data_stream_batches status as the authoritative source.

Stream State Verdicts

| Status | Meaning | Action | |--------|---------|--------| | ACTIVE | Healthy, batches flowing | None | | SUSPENDED | Paused; no new batches | Call resume_cdc() — see below | | QUARANTINED | Permanently paused; data integrity issue | [Follow quarantine recovery flow](references/cdc-recovery.md#quarantine-recovery-runbook) | | FAILED | Batch or load error | Check data_stream_errors and get_load_errors |

Resume a Suspended Stream

CALL relationalai.app.resume_cdc('');

Quarantine Recovery

See [cdc-recovery.md — Quarantine Recovery Runbook](references/cdc-recovery.md#quarantine-recovery-runbook) for the full step-by-step recovery checklist, schema reference, and official docs link.

---

Predictive reasoner stuck in data-index init

A freshly created or freshly resumed Predictive reasoner can sit in STATUS=PROVISIONING for many minutes while the in-pod data index hydrates from CDC streams — and gnn.fit() running against it appears to hang in Step 1 (dataset prep) before reaching "Training job submitted." This is the same per-table CDC stream-sync compounding that affects unwarmed Logic reasoners on first model query, surfacing on the Predictive side as a silent prep-step stall.

Diagnose (in order)

1. Reasoner status — is the pod still hydrating or actually ready?

``sql CALL RELATIONALAI.API.GET_REASONER('predictive', ''); ` STATUS=PROVISIONING → wait (1–3 min typical for warm pools; longer on first-time Snowflake-stream attach). STATUS=READY` plus a hang → continue to (2) and (3).

1. CDC stream health — is a stream the predictive reasoner depends on suspended or quarantined?

``sql SELECT * FROM relationalai.api.cdc_status; ` enginestatus != READY or streamstatus not RUNNING → see § Step 5 — Diagnose CDC / Data Stream Health for the quarantine-recovery / resume_cdc` runbook. Predictive jobs cannot proceed until the upstream streams are healthy.

1. Transaction problems — if a specific transaction id appeared in client logs (often the Failed to pull data into index: transaction was aborted wrapper), pull its problems:

``sql CALL RELATIONALAI.API.GET_OWN_TRANSACTION_PROBLEMS(''); -- or with admin role: GET_TRANSACTION_PROBLEMS('') `` See § Step 4 — Diagnose a Failed Transaction for the schema reference and owner-restriction pitfall.

Recovery

If CDC is healthy and the reasoner has been PROVISIONING for more than ~5 minutes with no client-side progress, treat it the same as a stuck worker — SUSPEND_REASONER + RESUME_REASONER_ASYNC (see § Predictive train jobs stuck QUEUED below). Persistent failure: rebuild on a fresh GPU reasoner via DELETE_REASONER + CREATE_REASONER_ASYNC('predictive', ..., 'GPU_NV_S', OBJECT_CONSTRUCT()).

---

Predictive train jobs stuck QUEUED

A predictive train job submitted via gnn.fit() can sit in STATE='QUEUED' in RELATIONALAI.API.JOBS indefinitely while CALL RELATIONALAI.API.GET_REASONER('predictive', '') still reports STATUS='READY'. The SDK only checks reasoner-pod status before submitting — the in-pod worker queue can be out of sync with that status, and the SDK has no way to detect it (relationalai_gnns/core/connector.py::_check_engine_availability).

Diagnostic ladder

Long-running predictive jobs are usually fine — distinguish stuck from slow before suspending anything. Use the same ladder as rai-predictive-training § "Training appears stuck":

1. CALL RELATIONALAI.API.GET_REASONER('predictive', '') → STATUS=READY?
2. client.jobs.list("Predictive", name="") → is there a RUNNING train job (with rising AGE_MIN), or a QUEUED one going stale?
3. SHOW EXPERIMENTS IN SCHEMA . → did a new experiment row append within ~60s of the RUNNING train job?

A QUEUED train job that won't advance while the reasoner reports READY is the worker-queue desync this section addresses. Genuine long runs progress through (1) READY → (2) RUNNING → (3) new experiment row.

Recovery

Suspend then resume the predictive reasoner to force a worker recycle, kill any stuck client, then re-instantiate GNN(...) and resubmit (gnn.fit() is idempotent — see rai-predictive-training § gnn.fit() is idempotent). Use only the supported RELATIONALAI.API.* surface — do not invoke RELATIONALAI.EXPERIMENTAL.* procedures as a workaround.

-- 1. Confirm a stuck train job
SELECT ID, STATE, DATEDIFF('minute', CREATED_ON, CURRENT_TIMESTAMP()) AS AGE_MIN
FROM RELATIONALAI.API.JOBS
WHERE STATE IN ('QUEUED','RUNNING')
  AND PAYLOAD LIKE '%"job_type": "train"%'
ORDER BY CREATED_ON ASC;

-- 2. Recycle the worker
CALL RELATIONALAI.API.SUSPEND_REASONER('predictive', '');
CALL RELATIONALAI.API.RESUME_REASONER_ASYNC('predictive', '');

-- 3. Wait for STATUS=READY, kill the stuck client, then resubmit (re-instantiate GNN(...))
CALL RELATIONALAI.API.GET_REASONER('predictive', '');

If recycling doesn't unstick the worker, rebuild on a fresh GPU predictive reasoner — the documented path:

CALL RELATIONALAI.API.DELETE_REASONER('predictive', '');
CALL RELATIONALAI.API.CREATE_REASONER_ASYNC('predictive', '', 'GPU_NV_S', OBJECT_CONSTRUCT());
-- Poll until STATUS=READY:
CALL RELATIONALAI.API.GET_REASONER('predictive', '');

See rai-predictive-training § Worker not ready to accept jobs for the matching client-side symptom and § Stalled train job: SDK polls without a timeout for stalled-job forensics.

---

Step 6 — CDC Engine Management

> CDC engine ≠ reasoner engine. The CDC pipeline runs on a dedicated managed engine > distinct from Logic reasoner engines. alter_cdc_engine_size targets only the CDC engi

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: RelationalAI
• Source: RelationalAI/rai-agent-skills
• License: Apache-2.0
• Homepage: https://relational.ai

Install and usage instructions live in the source repository linked above.