Distributed Tracing Patterns

Distributed Tracing Patterns

Overview

Distributed tracing records the journey of a request across services as a tree of spans. Without proper instrumentation, diagnosing latency in a microservices system requires guesswork. Done poorly — missing context propagation, over-instrumentation, or span explosion — tracing creates noise and cost without insight.

When to use: Instrumenting a new service, reviewing tracing code, debugging latency across service boundaries, linking traces to metrics, or designing a sampling strategy for high-throughput systems.

Prerequisite skills: observability-patterns, microservices-resilience-patterns, performance-anti-patterns.

Quick Reference

| Topic | Core Idea | Primary Red Flag | |-------|-----------|-----------------| | Auto-instrumentation | Framework hooks inject spans with zero code change | Forgetting to add SDK; auto-spans are too coarse | | Manual instrumentation | Code-level spans for business operations | Spans that duplicate auto-instrumented work | | Span naming | using a stable taxonomy | Dynamic IDs in span names cause cardinality explosion | | Span attributes | Semantic conventions for keys; bounded value sets | Free-text attribute values create unbounded cardinality | | Span status | OK / ERROR / UNSET with description only on ERROR | Setting ERROR on expected 404s pollutes error rates | | Context propagation | W3C TraceContext (traceparent) or B3 headers | Fire-and-forget calls that drop context | | Head sampling | Decision made at trace root, propagated downstream | Sampling before context is set loses correlated spans | | Tail sampling | Decision made after all spans collected | Requires stateful collector; complex to operate | | Probabilistic sampling | Fixed rate (e.g., 1%) applied uniformly | Low-rate sampling misses rare slow paths | | Rate-limiting sampling | Fixed span count per second | Spiky traffic bursts still get sampled proportionally | | Exemplars | Attach trace ID to a metric data point | Metrics and traces in separate systems with no link | | Over-instrumentation | More spans than needed obscures signal | Every line of code wrapped in a span | | Span explosion | Dynamic span names or attributes create millions of series | span.name = "GET /users/" + userId |

---

Patterns in Detail

1. OpenTelemetry Instrumentation — Auto vs. Manual

Auto-instrumentation: Framework hooks (HTTP servers, DB drivers, message brokers) with zero code changes. Manual instrumentation: Add spans for business logic the framework cannot see — long transactions, batch loops, background jobs.

TypeScript (Node.js) — auto + manual:

// sdk-setup.ts — register before any other imports
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
    new PgInstrumentation(),
  ],
});
sdk.start();

// app.ts — manual span for business logic
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service', '1.0.0');

async function processOrder(orderId: string): Promise {
  await tracer.startActiveSpan('order.process', async (span) => {
    span.setAttribute('order.id', orderId);
    try {
      await validateInventory(orderId);
      await chargePayment(orderId);
      await dispatchFulfillment(orderId);
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Go — manual span (context passing is idiomatic):

func processOrder(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "order.process",
        trace.WithAttributes(attribute.String("order.id", orderID)),
    )
    defer span.End()

    if err := chargePayment(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return fmt.Errorf("processOrder: %w", err)
    }
    return nil
}

Java — auto via javaagent + annotation for manual:

// Auto: attach -javaagent:opentelemetry-javaagent.jar at startup
// JVM: -Dotel.service.name=order-service -Dotel.exporter.otlp.endpoint=http://collector:4317

@WithSpan("order.validate-inventory")
public void validateInventory(@SpanAttribute("order.id") String orderId) { }

---

2. Span Design — Naming, Attributes, and Status Codes

Naming conventions — in snake_case or dot.notation:

| Operation type | Convention | Example | |---------------|-----------|---------| | HTTP server/client | HTTP | HTTP GET | | Database | . | SELECT orders.payments | | Message consumer | receive | payments.events receive | | Business operation | . | order.process |

Red flags: "GET /users/" + userId (cardinality), "step1" (meaningless), "handleRequest" (too generic).

Attribute taxonomy:

// CORRECT — semantic convention keys, bounded values
span.setAttributes({
  'http.method': 'POST',
  'http.status_code': 200,
  'db.system': 'postgresql',
  'db.operation': 'INSERT',
  'order.id': orderId,
  'order.items.count': 3,
});

// WRONG — PII, unbounded values
span.setAttributes({
  'user.email': email,                          // PII
  'response.body': JSON.stringify(resp),        // unbounded size
});

Status codes:

| Status | When to use | |--------|------------| | UNSET | Default; span completed normally | | OK | Explicitly verified success (use sparingly) | | ERROR | Unexpected failure the caller would investigate |

Set ERROR only on 5xx, network errors, and SLA-breaking business violations. NOT on 404s or 400s.

---

3. Context Propagation — W3C TraceContext and B3 Headers

W3C TraceContext (recommended):

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ version  ^^ trace-id (128 bit)           ^^ parent-span-id  ^^ flags
tracestate:  vendor-specific additional state (optional)

B3 (legacy Zipkin):

X-B3-TraceId: 463ac35c9f6413ad48485a3953bb6124
X-B3-SpanId:  a2fb4a1d1a96d312
X-B3-Sampled: 1

Go — composite propagator for mixed environments:

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},  // W3C — preferred
    propagation.Baggage{},
    b3.New(),                    // B3 — for legacy Zipkin consumers
))

Manual propagation (message queues, async tasks):

HTTP frameworks inject/extract automatically. For Kafka, SQS, and job queues, propagate manually:

// Producer — inject context into message headers
import { propagation, context } from '@opentelemetry/api';

function publishEvent(topic: string, payload: object): void {
  const carrier: Record = {};
  propagation.inject(context.active(), carrier);
  kafkaProducer.send({ topic, messages: [{ headers: carrier, value: JSON.stringify(payload) }] });
}

// Consumer — extract context from message headers
function consumeEvent(message: KafkaMessage): void {
  const ctx = propagation.extract(context.active(), message.headers ?? {});
  context.with(ctx, () => {
    tracer.startActiveSpan('payments.events receive', (span) => {
      handleEvent(JSON.parse(message.value!.toString()));
      span.end();
    });
  });
}

Python — manual propagation for Celery tasks:

from opentelemetry.propagate import inject, extract

def dispatch_task(payload: dict) -> None:
    carrier: dict = {}
    inject(carrier)
    celery_task.apply_async(args=[payload], headers=carrier)

@app.task(bind=True)
def process_task(self, payload: dict) -> None:
    ctx = extract(self.request.headers or {})
    with trace.use_span(trace.get_tracer(__name__).start_span("task.process", context=ctx)):
        handle_payload(payload)

Red Flags:

• Fire-and-forget async calls that never extract the incoming context
• HTTP clients that strip traceparent before forwarding
• Manual trace_id logging without linking to the trace context object
• Propagating context across trust boundaries without scrubbing tracestate

---

4. Sampling Strategies

Head sampling — decision at trace root:

// Go — probabilistic 10% + priority override for checkout flows
tp := trace.NewTracerProvider(
    trace.WithSampler(trace.TraceIDRatioBased(0.10)),
)

type prioritySampler struct{ base trace.Sampler }
func (s prioritySampler) ShouldSample(p trace.SamplingParameters) trace.SamplingResult {
    if strings.HasPrefix(p.Name, "checkout.") {
        return trace.AlwaysSample().ShouldSample(p)
    }
    return s.base.ShouldSample(p)
}

Tail sampling — decision after span collection (OTEL Collector):

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow-traces
        type: latency
        latency: { threshold_ms: 2000 }
      - name: probabilistic-base
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Probabilistic (Python):

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
sampler = TraceIdRatioBased(0.05)  # 5% of traces

Rate-limiting sampler (Java — unique custom Sampler interface):

public class RateLimitingSampler implements Sampler {
    private final RateLimiter limiter;
    public RateLimitingSampler(double tracesPerSecond) {
        this.limiter = RateLimiter.create(tracesPerSecond);
    }
    @Override
    public SamplingResult shouldSample(Context parentContext, String traceId,
            String name, SpanKind kind, Attributes attributes, List links) {
        return limiter.tryAcquire()
            ? SamplingResult.recordAndSample()
            : SamplingResult.drop();
    }
    @Override public String getDescription() { return "RateLimitingSampler"; }
}

Strategy selection guide:

| Strategy | Best for | Avoid when | |----------|---------|-----------| | Head probabilistic | High-volume, homogeneous traffic | Need to capture all errors or slow traces | | Head rate-limiting | Bursty traffic, cost control | Need representative statistical sample | | Tail sampling | Error and latency capture guarantees | Simple setups; adds collector complexity | | Always-on | Low-volume, critical paths (payments) | High-volume endpoints — cost explosion | | Parent-based | Respecting upstream sampling decision | You are the root service |

---

5. Trace-Based Testing and Debugging

Integration test with in-memory exporter (TypeScript):

import { InMemorySpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';

let exporter: InMemorySpanExporter;

beforeEach(() => {
  exporter = new InMemorySpanExporter();
  const provider = new NodeTracerProvider();
  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
  provider.register();
});

afterEach(() => exporter.reset());

test('processOrder creates child spans with correct attributes', async () => {
  await processOrder('order-42');

  const spans = exporter.getFinishedSpans();
  const root = spans.find(s => s.name === 'order.process');
  expect(root).toBeDefined();
  expect(root!.attributes['order.id']).toBe('order-42');
  expect(root!.status.code).toBe(SpanStatusCode.OK);

  const chargeSpan = spans.find(s => s.name === 'payment.charge');
  expect(chargeSpan!.parentSpanId).toBe(root!.spanContext().spanId);
});

Debugging workflow:

1. Find the trace ID from APM (Jaeger, Tempo, Honeycomb, Datadog) via a log correlation or Exemplar link.
2. In the trace waterfall, locate the longest span or first ERROR span.
3. Inspect span attributes (db.statement, http.url, order.id) to narrow scope.
4. Check gap between parent span start and first child — often network, serialization, or lock contention.
5. Missing expected child span → context was likely dropped; check async call sites and message producer/consumer pairs.

---

6. Exemplars — Linking Traces to Metrics

Exemplars embed a trace ID into a metric data point — one click in Grafana goes from a p99 spike to the exact trace.

TypeScript — OTEL Prometheus exporter (trace ID auto-attached from active context):

import { MeterProvider } from '@opentelemetry/sdk-metrics';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';

const exporter = new PrometheusExporter({ appendTimestamp: true });
const meter = new MeterProvider({ readers: [exporter] }).getMeter('order-service');
const requestDuration = meter.createHistogram('http.request.duration', { unit: 'ms' });

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    requestDuration.record(Date.now() - start, {
      'http.method': req.method,
      'http.status_code': res.statusCode,
    });
    // OTEL SDK auto-attaches trace_id + span_id from active context
  });
  next();
});

Java — Spring @Timed with Micrometer-OTEL bridge:

// Requires: micrometer-tracing-bridge-otel
@Timed(value = "order.process.duration", histogram = true)
public void processOrder(String orderId) { /* trace_id linked automatically */ }

Grafana PromQL:

histogram_quantile(0.99, rate(http_request_duration_bucket[5m]))
# Click exemplar dot → linked trace in Tempo/Jaeger

Requires metric and trace SDKs to share the same active context. Recording metrics outside a span = no exemplar attached (silent gap).

---

7. Distributed Tracing Anti-Patterns

Anti-Pattern: Over-Instrumentation

Detection: >50 spans/request; span names like getField, setProperty. Fix: Span only at I/O boundaries and significant business operations. Internal pure functions do not need spans.

Anti-Pattern: Missing Context Propagation

// WRONG — goroutine loses context
go func() { processNotification(orderId) }()

// CORRECT — pass context explicitly
go func(ctx context.Context) { processNotification(ctx, orderId) }(ctx)

Anti-Pattern: Span Explosion (Cardinality)

# WRONG — orderId in span name = unbounded cardinality
with tracer.start_as_current_span(f"process-order-{order_id}"): ...

# CORRECT — stable name, dynamic value as attribute
with tracer.start_as_current_span("order.process") as span:
    span.set_attribute("order.id", order_id)

Anti-Pattern: Logging the Trace ID Instead of Using Context

// WRONG — manual string, not linked in APM
logger.info(`trace_id=${traceId} order processed`);

// CORRECT — inject from active span context
const span = trace.getActiveSpan();
const { traceId, spanId } = span?.spanContext() ?? { traceId: '', spanId: '' };
logger.info('order processed', { traceId, spanId });

Anti-Pattern: Incorrect Span Status

| HTTP Status | Span status | |------------|------------| | 2xx | OK or UNSET | | 4xx (client error) | UNSET (expected — not a service failure) | | 5xx (server error) | ERROR | | Network timeout | ERROR |

Anti-Pattern: Unbounded Attribute Values

// WRONG
span.setAttribute("db.statement", rawSql);        // may contain passwords
span.setAttribute("response.body", responseJson); // unbounded size

// CORRECT
span.setAttribute("db.operation", "SELECT");
span.setAttribute("db.table", "orders");
span.setAttr

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [mickeyyaya](https://github.com/mickeyyaya)
- **Source:** [mickeyyaya/refactoring-skills](https://github.com/mickeyyaya/refactoring-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.