Error Resilience

Error Resilience Patterns

Overview

Production code fails. Networks drop, services crash, databases timeout. This skill teaches patterns that keep your system running when dependencies don't.

When to Use

• Any HTTP/API call to an external service
• Database operations that can timeout
• Background job processing
• Event/message consumers
• Any I/O operation in production code
• User asked "how do I handle errors properly"
• User asked "how do I retry failed operations"

Quick Start: The Minimum Viable Resilience

Every external call needs AT MINIMUM:

async function resilientCall(
  fn: () => Promise,
  options: { retries?: number; timeout?: number; fallback?: T } = {}
): Promise {
  const { retries = 3, timeout = 5000, fallback } = options;
  
  for (let attempt = 1; attempt  controller.abort(), timeout);
      const result = await fn();
      clearTimeout(timer);
      return result;
    } catch (error) {
      if (attempt === retries) {
        if (fallback !== undefined) return fallback;
        throw error;
      }
      await sleep(exponentialBackoff(attempt));
    }
  }
  throw new Error('Unreachable');
}

function exponentialBackoff(attempt: number): number {
  const base = 1000;
  const jitter = Math.random() * 500;
  return Math.min(base * Math.pow(2, attempt - 1) + jitter, 30000);
}

function sleep(ms: number): Promise {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Pattern 1: Retry with Exponential Backoff

When: Transient failures (network blips, rate limits, temporary unavailability)

// Configuration
const RETRY_CONFIG = {
  maxAttempts: 3,
  baseDelay: 1000,      // 1s, 2s, 4s, 8s...
  maxDelay: 30000,      // Never wait > 30s
  retryableErrors: [
    'ECONNRESET', 'ETIMEDOUT', 'ECONNREFUSED',
    'EPIPE', 'EAI_AGAIN', 'EHOSTUNREACH'
  ],
  retryableStatuses: [408, 429, 500, 502, 503, 504]
};

async function withRetry(
  fn: () => Promise,
  config = RETRY_CONFIG
): Promise {
  let lastError: Error;
  
  for (let attempt = 1; attempt (fn: () => Promise, fallback?: () => T): Promise {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailure > this.resetTimeout) {
        this.state = 'half-open';
      } else {
        if (fallback) return fallback();
        throw new Error('Circuit breaker is OPEN — service unavailable');
      }
    }
    
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      if (fallback) return fallback();
      throw error;
    }
  }
  
  private onSuccess() {
    this.failures = 0;
    this.state = 'closed';
  }
  
  private onFailure() {
    this.failures++;
    this.lastFailure = Date.now();
    if (this.failures >= this.threshold) {
      this.state = 'open';
      console.error(`Circuit OPEN after ${this.failures} failures`);
    }
  }
}

// Usage:
const paymentCircuit = new CircuitBreaker(3, 60000);
const result = await paymentCircuit.call(
  () => stripe.charges.create(params),
  () => ({ status: 'queued', message: 'Payment queued for retry' })
);

Pattern 3: Timeout + Abort

When: Operations that can hang forever (DNS, DB connections, external APIs)

async function withTimeout(
  fn: (signal: AbortSignal) => Promise,
  ms: number,
  errorMessage = `Operation timed out after ${ms}ms`
): Promise {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), ms);
  
  try {
    const result = await fn(controller.signal);
    return result;
  } catch (error: any) {
    if (error.name === 'AbortError') {
      throw new Error(errorMessage);
    }
    throw error;
  } finally {
    clearTimeout(timer);
  }
}

// Usage with fetch:
const data = await withTimeout(
  (signal) => fetch('https://api.example.com/data', { signal }).then(r => r.json()),
  5000,
  'API did not respond within 5 seconds'
);

Pattern 4: Graceful Degradation

When: Feature is nice-to-have, core functionality must continue without it

async function withGracefulDegradation(
  primary: () => Promise,
  fallback: T | (() => T | Promise),
  options: { log?: boolean; metric?: string } = {}
): Promise {
  try {
    return await primary();
  } catch (error) {
    if (options.log !== false) {
      console.warn(`Degraded: ${options.metric || 'unknown'}`, error);
    }
    return typeof fallback === 'function' ? (fallback as Function)() : fallback;
  }
}

// Examples:
const recommendations = await withGracefulDegradation(
  () => mlService.getRecommendations(userId),
  [],  // Show no recommendations rather than error page
  { metric: 'recommendations.degraded' }
);

const userAvatar = await withGracefulDegradation(
  () => avatarService.get(userId),
  '/default-avatar.png',
  { metric: 'avatar.degraded' }
);

Pattern 5: Dead Letter Queue (DLQ)

When: Background job processing — failed jobs need investigation, not silent drops

interface DLQEntry {
  originalPayload: any;
  error: string;
  failedAt: string;
  attempts: number;
  queue: string;
}

class JobProcessor {
  private maxAttempts = 3;
  
  async process(job: any, handler: (job: any) => Promise): Promise {
    for (let attempt = 1; attempt  {
    // Store in DLQ table/queue for manual investigation
    console.error('Job moved to DLQ:', entry);
    // await db.insert('dead_letter_queue', entry);
  }
}

Decision Matrix

| Situation | Pattern | Why | |-----------|---------|-----| | API call might fail temporarily | Retry + Backoff | Transient errors resolve on retry | | Service is completely down | Circuit Breaker | Stop wasting resources on dead service | | Operation might hang forever | Timeout + Abort | Prevent thread/connection exhaustion | | Feature failure shouldn't crash app | Graceful Degradation | Core UX survives | | Background job fails repeatedly | Dead Letter Queue | Don't lose data, investigate later | | Multiple services, any might fail | Combine all above | Defense in depth |

Common Mistakes

• Retrying non-retryable errors (400/401/404 — your bug, not theirs)
• No jitter in backoff (all clients retry at same time = thundering herd)
• Infinite retries (always set maxAttempts — infinite = memory leak + useless)
• Swallowing errors silently (catch without logging = invisible failures)
• Same timeout for all operations (DNS: 2s, DB query: 5s, file upload: 60s — tune per operation)
• No circuit breaker for cascading calls (Service A → B → C, if C is down, A and B pile up)
• Retrying inside a retry (nested retries = exponential attempts: 3 × 3 × 3 = 27 attempts)

Verification

After implementing resilience patterns:

• [ ] Non-retryable errors (400, 401, 404) are NOT retried
• [ ] Retries have exponential backoff WITH jitter
• [ ] Timeouts are set for every external call
• [ ] Circuit breaker prevents cascade when service is down
• [ ] Graceful degradation tested (kill dependency, verify app still works)
• [ ] Failed jobs land in DLQ with full context for debugging
• [ ] Logs show retry attempts and final outcomes
• [ ] Metrics track failure rates per dependency

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Adit-Jain-srm
• Source: Adit-Jain-srm/skill-forge
• License: MIT
• Homepage: https://github.com/Adit-Jain-srm/skill-forge#quick-start

Install and usage instructions live in the source repository linked above.