1 min read

Debugging Latency Spikes with Jaeger Traces

Visualizing a request bottleneck using Grafana Tempo/Jaeger
Street photograph by Stalin Pereira
Photo by Stalin Pereira

Where Did the Request Die?

Debugging latency without logging into 10 servers.

The 'Needle in a Haystack'

  • The Complaint — "The checkout is slow." (It takes 5 seconds).
  • The Architecture — Checkout talks to User Service, Inventory Service, and Payment Gateway.
  • The Mystery — Which one is slow? Logs won't tell you the network time between them.

Enter Distributed Tracing

  • Trace ID — A unique ID attached to the request header at the start.
  • Propagation — This ID is passed along to every internal service call.
  • Spans — Each service reports: "I worked on Trace ID 123 from 10:00 to 10:01."

Visualizing with Jaeger / Tempo

The Waterfall Chart

  • The Long Bar — You instantly see the longest bar in the chart. That's your bottleneck.
  • The Gap — Empty space between bars means network latency or queuing.

Common Culprits

  • Serial Calls — Service A calls B, waits, then calls C. (Should be parallel).
  • DB Locks — A span that takes 2s but does simple work? Likely waiting for a lock.
  • N+1 HTTP — Calling a microservice inside a loop. The waterfall looks like stairs.

Zero Code Instrumentation

  • OpenTelemetry — The industry standard. Auto-instruments HTTP and DB drivers.
  • Ease of Use — Often just a config change or a library import. No rewriting logic.

Summary: Don't Guess

Visualize the flow.

Logs show you what happened. Traces show you where and how long.

Can you trace a request across 5 services?