Debugging Latency Spikes with Jaeger Traces
Visualizing a request bottleneck using Grafana Tempo/Jaeger
Where Did the Request Die?
Debugging latency without logging into 10 servers.
The 'Needle in a Haystack'
- The Complaint — "The checkout is slow." (It takes 5 seconds).
- The Architecture — Checkout talks to User Service, Inventory Service, and Payment Gateway.
- The Mystery — Which one is slow? Logs won't tell you the network time between them.
Enter Distributed Tracing
- Trace ID — A unique ID attached to the request header at the start.
- Propagation — This ID is passed along to every internal service call.
- Spans — Each service reports: "I worked on Trace ID 123 from 10:00 to 10:01."
Visualizing with Jaeger / Tempo
The Waterfall Chart
- The Long Bar — You instantly see the longest bar in the chart. That's your bottleneck.
- The Gap — Empty space between bars means network latency or queuing.
Common Culprits
- Serial Calls — Service A calls B, waits, then calls C. (Should be parallel).
- DB Locks — A span that takes 2s but does simple work? Likely waiting for a lock.
- N+1 HTTP — Calling a microservice inside a loop. The waterfall looks like stairs.
Zero Code Instrumentation
- OpenTelemetry — The industry standard. Auto-instruments HTTP and DB drivers.
- Ease of Use — Often just a config change or a library import. No rewriting logic.
Summary: Don't Guess
Visualize the flow.
Logs show you what happened. Traces show you where and how long.
Can you trace a request across 5 services?