In a monolithic application, debugging a request is straightforward — one log file, one thread. In microservices, a single user request spans multiple services, each with its own logs, on different machines. When something fails, finding the root cause requires manually correlating log entries across services — a tedious, near-impossible task.
Distributed tracing tracks a single request across multiple services by propagating a unique trace ID through every service call. Each service records spans (units of work) with timing, status, and metadata. A tracing system (Zipkin, Jaeger) collects and visualizes these spans, showing the full request path, timing breakdown, and errors.
- Trace ID generation: When a request enters the system (at the API Gateway), a unique trace ID is generated
- Propagation: The trace ID and span ID are propagated via HTTP headers (
X-B3-TraceId,X-B3-SpanId) to downstream services - Span recording: Each service creates spans for the work it does — e.g., controller method, database query, HTTP call
- Annotation: Spans include annotations for key events (client send, server receive, error)
- Collection: Services send completed spans to a collector (Zipkin server) asynchronously
- Visualization: The Zipkin UI shows a waterfall diagram: one row per span, with timing and parent-child relationships
- Trace ID: Unique identifier for an entire request across all services
- Span: A named, timed unit of work within a service (e.g., “SELECT from orders”)
- Parent span: Spans form a tree — parent-child relationships show the call hierarchy
- Propagation: Trace context passed via HTTP headers (B3 format: X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId)
- Sampling: Not every request is traced — sampling rate configurable (e.g., trace 10% of requests)
- Visualization: Zipkin Jaeger, or Grafana Tempo show trace timelines as waterfall diagrams
- Built from: Java Microservices — Distributed tracing is essential when requests span multiple services
- Related: Spring Cloud — Spring Cloud Sleuth (deprecated) / Micrometer Tracing provides auto-configuration
- Related: URL to Rendering — Full Flow Analysis — Both trace end-to-end request flows
- Contrasts with: JTA and JTS — JTA propagates transaction context; tracing propagates trace context
- Performance overhead: Generating and reporting spans adds overhead — use sampling, especially at high throughput
- Async boundaries: Trace context doesn’t propagate automatically across async boundaries — use
@Async+ trace context propagation - Sensitive data: Spans may inadvertently capture sensitive data (SQL queries, request bodies) — sanitize span tags
- Clock skew: Services on different machines have slightly different clocks — Zipkin adjusts using client/server send/receive timestamps
- End-to-end setup: Requires all services to participate — one service without tracing breaks the trace chain