Three Pillars & OpenTelemetry

Three pillars, three questions

Observability rests on three kinds of telemetry, and the skill is knowing which one answers which question. They are not redundant; each trades detail against cost differently.

Metrics are cheap numeric aggregates over time (counters, gauges, histograms). They cost almost nothing per data point, retain for a long time, and are what you alert on and trend on. They answer "how many, how fast, how full, right now and over the last quarter?" What they cannot tell you is why one specific request was slow, because they have thrown away the individual events.

Logs are structured records of discrete events. They carry the detail metrics discarded: the exact error, the parameters, the code path. They answer "what exactly happened to this request?" But logs are expensive at volume and painful to correlate across services unless they are structured (JSON with consistent fields) and carry shared ids. A wall of free-text log lines from twelve services with no common id is nearly useless for a distributed problem.

Traces capture the causal path of a single request as it fans across services. A trace is a tree of spans; each span is one unit of work (an HTTP handler, a DB query, a cache lookup) with a start time, duration, and attributes. Traces answer the question metrics and logs alone cannot: "this checkout took 1.4s, where did the time go across the 12 hops?" That is the pillar most teams under-invest in and most regret skipping.

  Trace: checkout (1.4s)
  |-- api-gateway            [ 20ms ]
  |-- order-svc              [ ============ 200ms ]
  |     |-- postgres write   [ ==== 40ms ]
  |-- payment-svc            [ ================================ 1.1s ]
  |     |-- fraud-check gRPC [ ============================ 1.0s ]  <- culprit
  |-- notify-svc            [ 30ms ]

Context propagation ties traces together

The thing that makes traces work across service boundaries is context propagation. Each incoming request carries a traceparent header (the W3C Trace Context standard) holding the trace id and the parent span id. Each service reads it, starts a child span, and passes the updated header to its own downstream calls. That shared trace id is also what you stamp onto every log line and (as an exemplar) onto metrics, so you can pivot: a metric spike -> an exemplar trace id -> the full trace -> the correlated logs for exactly that request. Without propagated context, "why is it slow?" is unanswerable in a distributed system, which is the single most common wrong turn in this space.

OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard that ties all three pillars together. It gives you: SDKs (per language) that produce metrics, logs, and traces with a common data model and automatic context propagation; instrumentation libraries that trace popular frameworks with near-zero code; and the OTel Collector, a separate process that receives your telemetry, processes it (batching, sampling, redaction, adding resource attributes), and exports it to whatever backends you choose (Prometheus for metrics, Loki/Elasticsearch for logs, Jaeger/Tempo for traces, or a vendor like Datadog/Honeycomb). The payoff is decoupling: your application code emits OTel and knows nothing about the backend, so you can switch vendors or fan out to several by editing Collector config, not redeploying every service.

Interview nuance: be ready to talk cost control, because that is where these designs are won or lost. Cardinality drives metric cost, so keep labels bounded. Trace volume is enormous at scale, so you sample: head-based sampling decides at the first span (cheap, simple, but may drop the one trace you needed), while tail-based sampling buffers whole traces at the Collector and keeps the interesting ones (all errors, all slow requests) plus a small percentage of normal traffic (accurate, but the Collector must hold traces in memory). Logs get tiered storage: hot in a fast index for a few days, then rolled to cheap object storage (S3) for compliance retention.

Recap: metrics for cheap aggregates and alerting, logs for structured per-event detail, traces for the causal cross-service path; propagate W3C trace context and share the trace id across all three; standardize on OpenTelemetry SDKs plus a Collector to decouple apps from backends; and control cost with bounded cardinality, trace sampling (tail-based keeps errors/slow), and tiered log retention.