Performance Testing

Performance Testing

Load and performance testing validates that your system meets latency and throughput requirements under realistic and extreme traffic conditions.

When to Activate

• Load testing an API before a product launch
• Setting up k6 or Locust for a project
• Writing Go benchmark functions for critical code paths
• Defining SLO-based pass/fail thresholds for load tests
• Identifying bottlenecks under load (pool exhaustion, N+1, GC pressure)
• Adding performance regression detection to a CI/CD pipeline

Test Type Decision Table

| Type | Description | Load shape | Goal | When to run | |------|-------------|-----------|------|-------------| | Load | Simulate expected traffic | Ramp to normal, hold | Verify baseline meets SLO | Pre-launch, nightly | | Stress | Push beyond capacity | Ramp past normal | Find breaking point | Before scaling decisions | | Soak | Sustained load over time | Constant for 1–4 hours | Detect memory leaks, pool exhaustion | Weekly | | Spike | Sudden burst | 0 → peak instantly | Test autoscaling, queue buffering | Before planned events | | Volume | Large datasets, normal load | Normal rps, huge data | Find data-size bottlenecks | When data volume increases |

k6

Script Structure

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const errorRate = new Rate('errors');
const paymentDuration = new Trend('payment_duration');

export const options = {
  stages: [
    { duration: '2m', target: 50 },   // ramp up
    { duration: '5m', target: 50 },   // hold
    { duration: '2m', target: 100 },  // ramp up further
    { duration: '5m', target: 100 },  // hold
    { duration: '2m', target: 0 },    // ramp down
  ],
  thresholds: {
    // SLO-based pass/fail: test fails if these are breached
    'http_req_duration': ['p(95) r.status === 201,
    'response time  r.timings.duration  before.txt
# ... make the change ...
go test -bench=. -count=10 -benchmem ./... > after.txt
benchstat before.txt after.txt

SLO-Based Pass/Fail Criteria

Defining Thresholds from SLOs

Base thresholds on your production SLOs — not arbitrary numbers.

// If SLO: p99  20% from baseline
4. Set SLO threshold: fail if p99 exceeds SLO target

### Bottleneck Identification Under Load

| Symptom | Likely cause | How to confirm | Fix |
|---------|-------------|---------------|-----|
| Latency climbs with VU count | Connection pool exhausted | Check pool wait metric | Increase pool / add PgBouncer |
| Error spikes at N rps | Thread / goroutine limit | Check active connections | Tune concurrency config |
| Memory grows during soak | Memory leak / large cache | Heap profile during test | Fix leak, tune GC |
| High latency, low CPU | N+1 queries | Count DB queries per request | Add eager loading |
| CPU > 90% | Compute bottleneck | CPU flame graph | Optimize hot path, add cache |
| Latency spikes periodically | GC pause (JVM/Go) | GC log analysis | Tune GC, reduce allocations |

## CI Integration

### When to Run

| Type | Frequency | Trigger | Failure action |
|------|-----------|---------|---------------|
| Smoke perf (5 VUs, 1 min) | Every PR | PR CI | Fail PR if p99 > 2× baseline |
| Full load test | Nightly | Cron | Alert on Slack |
| Stress test | Weekly | Cron | Report only |

### GitHub Actions Example

```yaml
jobs:
  load-test:
    runs-on: ubuntu-latest
    if: github.event_name == 'schedule'
    steps:
      - uses: actions/checkout@v4

      - name: Run k6 load test
        uses: grafana/k6-action@v0.3.0
        with:
          filename: tests/load/payment.js
        env:
          API_TOKEN: ${{ secrets.LOAD_TEST_TOKEN }}
          K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: k6-results-${{ github.run_id }}
          path: results/

> See also: performance, observability, ci-cd

Red Flags

• Symmetric ramp-up/ramp-down without a sustained plateau — spike-then-ramp-down misses memory leaks and GC pressure; hold at target RPS for ≥10 min in steady state
• Asserting only on HTTP 200 — a cached error page or open circuit breaker returns 200; use check() to assert on specific response body fields, not just the status code
• Single load generator machine for high VU counts — one machine saturates its NIC before the target; use distributed execution (k6 cloud, multiple Locust workers) above ~500 VUs
• No baseline before the test — without a pre-change baseline you can't tell whether 300ms p99 is a regression or always was that way
• Load test traffic escaping into production — test traffic that bypasses rate limits can trigger real customer alerts; isolate by dedicated API key, IP allowlist, or a separate environment
• Zero think time between requests — real users pause between actions; 0ms think time inflates effective concurrency 5–10×, producing false bottlenecks that don't exist in production
• Setting SLO thresholds from the first test run — first-run numbers are noisy; run 3+ tests under stable conditions before codifying a regression threshold

Checklist

• [ ] Test type chosen (load/stress/soak/spike) matches the specific question being answered
• [ ] k6 / Locust thresholds tied to SLO values — not made-up numbers
• [ ] Baseline measured before setting regression thresholds
• [ ] Test users and data isolated from production
• [ ] Think time (sleep) included in VU scripts for realistic simulation
• [ ] k6 check() used for per-request assertions (not just global thresholds)
• [ ] Go benchmarks include b.ReportAllocs() and b.ResetTimer()
• [ ] benchstat used to compare before/after for Go performance changes
• [ ] Bottleneck identification checklist followed when tests fail
• [ ] Load test results stored as CI artifacts for trending over time

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kid-sid
• Source: kid-sid/claude-spellbook
• License: MIT

Install and usage instructions live in the source repository linked above.