Distributed Tracing

Traces and spans

A trace is one request’s entire journey, sharing a single trace ID. A span is one timed unit of work within it — handling the request, calling a downstream service, running a query — each with a span ID, a parent, a duration, and attributes. Spans nest into a waterfall:

graph TD
A["span: GET /checkout<br/>(gateway) — 240ms"] --> B["span: auth.Verify<br/>15ms"]
A --> C["span: orders.Create<br/>200ms"]
C --> D["span: db.INSERT<br/>30ms"]
C --> E["span: payments.Charge<br/>160ms ← the slow hop"]

That waterfall instantly shows payments.Charge ate most of the 240ms — something neither a metric (“/checkout is slow”) nor a log (one service’s events) could pinpoint.

See it: context propagation builds the trace

The mechanism is context propagation: the trace/span IDs ride in context.Context locally and in headers across the network. This runs here — a simplified tracer threading IDs through nested calls to build the span tree:

package main

import (
"context"
"fmt"
)

type traceKey struct{}
type spanCtx struct {
traceID string
spanID  string
depth   int
}

// startSpan derives a child span from whatever is in the context.
func startSpan(ctx context.Context, name string) (context.Context, func()) {
parent, _ := ctx.Value(traceKey{}).(spanCtx)
if parent.traceID == "" {
	parent.traceID = "trace-7f3a" // root: would be random in real code
}
child := spanCtx{traceID: parent.traceID, spanID: name, depth: parent.depth + 1}
fmt.Printf("%*sspan start: %-16s trace=%s\n", child.depth*2, "", name, child.traceID)
ctx = context.WithValue(ctx, traceKey{}, child)
return ctx, func() { fmt.Printf("%*sspan end:   %s\n", child.depth*2, "", name) }
}

func main() {
ctx, end := startSpan(context.Background(), "GET /checkout")
defer end()

ctx2, end2 := startSpan(ctx, "orders.Create")
// payments.Charge is a CHILD of orders.Create — same trace ID propagates.
_, end3 := startSpan(ctx2, "payments.Charge")
end3()
end2()
}

Every span shares the one trace=trace-7f3a because the ID propagates through ctx. Across the network, the client injects it into the W3C traceparent header and the server extracts it — so spans from different services join the same trace. This is the formalized version of the correlation IDs from logging.

OpenTelemetry does this for real

You don’t hand-roll tracers. OpenTelemetry (opentelemetry.io) is the vendor-neutral standard: instrument once, export to any backend (Jaeger, Tempo, Datadog). Its Go libraries auto-instrument net/http, gRPC, and database/sql (fenced — third-party SDK):

// otelhttp wraps a handler: it extracts traceparent, starts a server span,
// and puts the context (with trace IDs) into r.Context() for you.
handler := otelhttp.NewHandler(mux, "api")

// In code, create child spans and add attributes:
ctx, span := tracer.Start(ctx, "payments.Charge")
defer span.End()
span.SetAttributes(attribute.Int("amount", amount))
// pass ctx to the next call — propagation is automatic.

At scale you sample: keep ~1% of traces plus 100% of slow/errored ones, so you keep the signal without the storage cost. The sampling decision rides in the trace context so all services agree.

See also

• Structured logging — correlation IDs, the precursor to spans.
• Metrics with Prometheus — the aggregate that tells you to look at a trace.
• gRPC & service comms — the boundaries traces cross.
• context (concurrency) — the carrier of trace context.

Next: looking inside one process when it’s the bottleneck — profiling in production.

Related topics