Sqlclusterlog Review

WSFC Cluster Log Review Skill

Purpose

Analyze Windows Server Failover Cluster (WSFC) CLUSTER.LOG files to diagnose Always On Availability Group failures at the cluster level — the layer below SQL Server DMVs. Applies 30 checks (L1–L30) across five categories:

• L1–L8 — File-wide patterns: lease timeouts, health check failures, RHS crashes, error

bursts, repeated failover cycling, quorum loss, node eviction, log time gaps

• L9–L17 — AG resource checks: offline transitions, SQL connectivity loss, forced failovers,

long pending states, DLL init failures, API timeouts, cascade failures, primary role loss, replica disconnection

• L18–L22 — Network and node: partition/split-brain, NIC failure, heartbeat timeout,

witness failure, node isolation

• L23–L25 — Configuration signals: VerboseLogging=0, SeparateMonitor absent, incomplete

node coverage

• L26–L30 — Modern cluster features: Cloud Witness timeout, Azure Arc agent disconnect,

Contained AG system database offline, cross-subnet probe failure, spserverdiagnostics warning

Input

Accept any of:

• File path — path to CLUSTER.LOG (live log: C:\Windows\Cluster\cluster.log; log generated via Get-ClusterLog: C:\Windows\Cluster\Reports\CLUSTER.LOG)
• Inline paste — raw CLUSTER.LOG content pasted directly into chat
• Natural language description — describe symptoms ("the AG went offline at 14:32,

SQL error log shows lease expiry")

For full analysis, the log should cover at least the 10 minutes before the incident and include entries from all cluster nodes. If only a partial extract is available, note which time range and nodes are covered and flag L25 if node coverage appears incomplete.

Log Entry Format

WSFC log entries follow this pattern:

.:://-::.  [] 

Key components:

• [RES] — Resource DLL host (hadrres.dll operations)
• [hadrag] — AG-specific resource agent inside RES
• [RHS] — Resource Hosting Subsystem (manages RES process lifecycle)
• [RCM] — Resource Control Manager (orchestrates state transitions)
• [NM] — Network Manager
• [NODE] — Node membership and heartbeat
• ERR / WARN / INFO — Severity prefixes in log lines

---

Thresholds Reference

| Threshold | Value | Used by | |-----------|-------|---------| | Error burst window | >10 ERR lines in 5 min → Critical; >5 → Warning | L4 | | Failover cycling | ≥3 group moves in 30 min → Critical; ≥2 → Warning | L5 | | Log time gap | >30 min → Critical; >5 min → Warning | L8 | | Pending state duration | >120 sec → Critical; >30 sec → Warning | L12 | | Lease timeout | 20 sec (SQL Server default LeaseTimeout — distinct from HealthCheckTimeout) | L1 | | Health check timeout | 30 sec (SQL Server default HealthCheckTimeout for spserverdiagnostics) | L2 | | Heartbeat timeout | Missed heartbeats reach the threshold — WSFC default SameSubnetThreshold/CrossSubnetThreshold is version-dependent: 20 (Windows Server 2019), 10 (WS2016), 5 (WS2008–2012 R2) | L20 |

---

File-Wide Pattern Checks (L1–L8)

Evaluate these first — they reveal root causes that explain all downstream AG failures.

L1 — Lease Timeout

• Trigger: Log contains [hadrag] Lease Thread terminated, lease time expired, HealthCheckTimeout associated with a lease expiry message, or LeaseExpired in [RES] or [hadrag] context
• Severity: Critical — lease expiry causes an immediate AG resource failure with no grace period
• Fix: Lease timeout indicates the SQL Server health check thread did not respond within the lease window (see Thresholds Reference). Root causes: (1) SQL Server scheduler starvation — check for long-running queries blocking the health thread; (2) memory pressure causing paging — review sys.dmosmemory_clerks; (3) storage I/O latency > 10 ms on the system drive — check Windows Performance Monitor; (4) if underlying cause is none of the above, increase sp_server_diagnostics timeout via CLUSTER_DIAGNOSTICS_TIMEOUT or raise HealthCheckTimeout in the AG resource properties.

L2 — Health Check Failure

• Trigger: Log contains IsAlive check failed, LooksAlive check failed, HealthCheckTimeout, or sp_server_diagnostics returning a failure state (STATE = 3 or STATE = 4) in [RES]/[hadrag] messages
• Severity: Critical — consecutive health check failures trigger resource restart or failover
• Fix: Identify whether LooksAlive or IsAlive failed. LooksAlive failures (process-level ping) indicate SQL Server process termination or severe hangs. IsAlive failures (spserverdiagnostics query) indicate scheduler starvation, I/O hangs, or insufficient health check timeout. Capture sys.dm_os_ring_buffers for the incident time and review the SQL ERRORLOG for the matching SPID N error.

L3 — RHS Process Crash

• Trigger: Log contains RHS process terminated, RHS.EXE terminated unexpectedly, creating new RHS process, rhs.exe exit in [RHS] context, or RHS exiting / unhandled exception in RHS
• Severity: Critical — RHS crash causes all resources hosted in that process to go offline
• Fix: RHS crash is a Windows-level failure, not SQL Server. Capture the Windows Application and System event logs at the incident time. Look for Event ID 1146 (RHS terminated) and corresponding Dr. Watson / crash dump. Common causes: a resource DLL (hadrres.dll or another DLL) threw an unhandled exception, or a third-party DLL was loaded into RHS and faulted. Enable SeparateMonitor on the AG resource to isolate hadrres.dll in its own RHS process (see L24).

L4 — Error Burst Density

• Trigger: More than 5 ERR-level lines appear within any 5-minute window in the log
• Severity: Warning if >5 ERR lines in 5 min; Critical if >10 ERR lines in 5 min (see Thresholds Reference)
• Fix: An error burst signals that multiple subsystems are failing simultaneously — often the symptom of a single root cause (L1, L6, L18). Identify the first ERR line in the burst — that is the originating failure. All subsequent ERRs in the burst are usually cascades. Fix the root cause first; the cascade errors will stop.

L5 — Repeated Failover Cycling

• Trigger: The log contains ≥2 [RCM] or [hadrag] group Move, Online, or Offline events for the same AG resource within any 30-minute window
• Severity: Warning if ≥2 moves in 30 min; Critical if ≥3 moves in 30 min (see Thresholds Reference)
• Fix: Rapid cycling means the AG resource goes online, fails again, and attempts recovery repeatedly. This exhausts MaxRestarts and eventually leaves the AG permanently offline. Root cause is almost always L1, L2, or L18 — a recurring condition that fails every recovery attempt. Identify and fix the root cause before the next failover window. Consider temporarily suspending the AG resource to prevent further cycling while the root cause is resolved.

L6 — Quorum Loss

• Trigger: Log contains quorum loss, quorum not achieved, no quorum, lost quorum, or cluster service stopping — no quorum in any component
• Severity: Critical — quorum loss stops the cluster service on all nodes, taking all AG resources offline
• Fix: Identify how many votes were lost. Review which nodes are in the log — if a majority-node-set configuration lost too many nodes, quorum fails. Immediate action: if a node is temporarily isolated, restore network connectivity. If a witness (disk/FSW/cloud) is unavailable, fix the witness first (see L21). Long-term: review quorum configuration — avoid even-node clusters without a witness.

L7 — Node Eviction

• Trigger: Log contains node eviction, removing node, node was removed from membership, evicted from cluster, or NodeMembership showing a node leaving in [NODE] or [NM] context
• Severity: Critical — node eviction means the cluster forcibly removed a node from membership, ending all resources that were primary on that node
• Fix: Node eviction is caused by sustained communication failure (heartbeat timeout, network partition) or manual eviction. Check [NODE] entries immediately before the eviction for heartbeat failures (L20) or network partition signals (L18). If eviction was manual, the cluster is operating as expected. If unexpected, review NIC bonding configuration and cross-subnet latency.

L8 — Log Time Gap

• Trigger: Consecutive log entries have a timestamp gap larger than 5 minutes with no intervening entries from any component
• Severity: Warning if >5 min; Critical if >30 min (see Thresholds Reference)
• Fix: A large time gap means the cluster log was not recording events — either the cluster service stopped, the node was powered off, or VerboseLogging was too low to capture events at the right frequency (see L23). If the gap coincides with the incident time, critical diagnostic data is missing. Retrieve CLUSTER.LOG from all nodes — one node may have continued logging while another was silent. Enable verbose logging proactively on all cluster nodes.

---

AG Resource Checks (L9–L17)

These checks fire on SQL Server AG-specific resource events within the WSFC log layer.

L9 — AG Offline Transition

• Trigger: Log contains TransitionToState ... Online-->Offline, TransitionToState ... OnlinePending-->Offline, OfflineCallIssued, or resource going offline in [RCM] or [hadrag] for an AG resource
• Severity: Critical if transition source is Online (unexpected); Warning if transition source is OnlinePending (resource never completed initialization)
• Fix: The AG resource transitioned to Offline. Identify whether the transition was initiated by WSFC (health check failure — see L2) or by SQL Server itself (the AG resource DLL called OfflineResource). If initiated by WSFC: fix L1 or L2. If initiated by the DLL: capture the [hadrag] log entries immediately before the transition — they explain the SQL-side reason (replica disconnect, data sync failure, or SQL Server error).

L10 — SQL Connectivity Loss

• Trigger: Log contains Disconnect from SQL Server, SQL Server connection failed, ODBC error, or SqlConnect failed in [hadrag] or [RES] context
• Severity: Critical — loss of connectivity between hadrres.dll and the local SQL Server instance means health checks cannot run and the AG resource will fail
• Fix: The resource DLL connects to SQL Server on the local loopback to run health checks. Failure means: (1) SQL Server service is stopped or starting — check Windows Service Control Manager; (2) SQL Server is overloaded and not accepting new connections — check for max connections reached (sys.dm_exec_sessions vs. max connections); (3) the dedicated admin connection (DAC) is in use — hadrres.dll uses a regular connection, not DAC, so this is not a DAC issue. Restart SQL Server if terminated unexpectedly.

L11 — Forced Failover

• Trigger: Log contains forced failover in [hadrag], [RCM], or [RES] context, or FAILOVER_MODE = MANUAL paired with an Online event on a formerly secondary node
• Severity: Warning — forced failover may result in data loss if the secondary was not synchronized
• Fix: Determine whether the forced failover was administrator-initiated or automatic. If automatic: this should not happen in a normal AG — a forced failover implies the primary failed and automatic failover was configured. Check whether data loss occurred by comparing last_commit_lsn on the former secondary (now primary) with the last confirmed last_hardened_lsn on the old primary. If administrator-initiated: document the reason and verify the replica is in synchronized state.

L12 — Long Pending State

• Trigger: An AG resource remains in OnlinePending or OfflinePending state for longer than the pending state thresholds (see Thresholds Reference) — calculated from the timestamp of the Pending entry to the next state transition entry
• Severity: Warning if >30 sec in pending; Critical if >120 sec (see Thresholds Reference)
• Fix: Pending states longer than expected indicate the resource DLL's Online or Offline call is not returning promptly. For OnlinePending: hadrres.dll is waiting for SQL Server to complete AG initialization — check SQL ERRORLOG for slow database recovery or role change. For OfflinePending: the DLL is waiting for the AG to gracefully suspend — if the wait is very long, a KILL or forced offline may be issued by WSFC, causing a dirty shutdown.

L13 — hadrres.dll Init Failure

• Trigger: Log contains DLL load failure, hadrres.dll initialization error, failed to initialize, or DLL could not be loaded in [RES] or [RHS] context
• Severity: Critical — the AG resource DLL cannot run, so the AG resource cannot come online on this node
• Fix: DLL init failure is usually caused by a missing dependency (Visual C++ runtime, Windows Server feature) or a corrupted hadrres.dll. Steps: (1) verify SQL Server is fully installed and the path in the cluster resource properties points to the correct hadrres.dll version; (2) check the Windows Application event log for DLL load errors; (3) run sfc /scannow to check for corrupted system files; (4) if after a SQL Server patch, the cluster resource DLL path may need to be updated manually.

L14 — Resource DLL API Timeout

• Trigger: Log contains API call timed out, Resource DLL returned ... after ... ms, or Dll timeout in [RCM] or [RHS] context for an AG resource
• Severity: Warning — if the DLL API timeout repeats, WSFC will declare the resource failed
• Fix: The resource DLL took longer than the configured DllWatchdogTimeout to respond to a WSFC API call (Online, Offline, LooksAlive, IsAlive). Caused by the same conditions as L1 and L2 — SQL Server scheduler starvation or I/O hangs. Correlate the timeout timestamp with SQL Server ERRORLOG. If timeouts recur, raise the cluster resource DllWatchdogTimeout value or fix the underlying SQL Server performance issue.

L15 — Cascade Across AGs

• Trigger: Multiple distinct AG resource names appear in ERR lines within the same 5-minute window — indicating more than one AG failed concurrently
• Severity: Warning — simultaneous multi-AG failure indicates a shared infrastructure failure (network partition, node failure) rather than an AG-specific issue
• Fix: Multiple AGs failing at the same time rules out AG-specific tuning as the solution. Focus on the shared infrastructure: network (L18, L19), node health (L7), or quorum (L6). Identify which AG failed first — that is the originating AG; the others are cascades caused by the same underlying event. Fix the root infrastructure issue.

L16 — Primary Role Loss

• Trigger: Log contains [hadrag] messages showing the primary replica transitioning to Resolving or Secondary role without a corresponding planned failover command
• Severity: Warning — unexpected primary role loss means the AG is momentarily without a primary; all write workloads will fail
• Fix: Unexpected primary role loss is caused by WSFC declaring the AG resource offline (see L1, L2, L9) or by a network split that caused the primary to lose quorum. Check whether a secondary promoted to primary simultaneously — if so, a failover completed. If no secondary promoted, the AG is in a Resolving state and requires manual intervention to bring a replica online. Review L6 for quorum loss.

L17 — Replica Disconnection

• Trigger: Log contains DISCONNECTED, replica disconnected, or connectivity failure messages in [hadrag] or [RES] context that refer to a remote replica endpoint
• Severity: Warning — a disconnected replica means data is not flowing to that secondary; during failover, the disconnected replica will have a stale copy
• Fix: Replica disconnection is a network-layer event — the AG mirroring endpoint (typically TCP 5022) cannot reach the remote replica. Check: (1) firewall rules on port 5022; (2) SQL Server Database Mirroring End

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: vanterx
• Source: vanterx/mssql-performance-skills
• License: MIT

Install and usage instructions live in the source repository linked above.