The Four Golden Signals & RED/USE

A small, dependable set of numbers

When you own a service at 3am and it is misbehaving, you do not have time to stare at forty dashboards. You need a small, dependable set of numbers that tells you whether the service is healthy and which direction it is failing. Google's SRE book distills this to the four golden signals: latency, traffic, errors, and saturation. Instrument these four for every service and you can answer "is it up, is it fast, is it failing, is it about to fall over?" without guessing.

Latency: how long a request takes. Measure it as a distribution, not a mean. A mean of 40ms can hide a p99 of 900ms that is torching 1% of your users. Alert and dashboard on p50, p95, p99, and often p99.9. Traffic: demand on the system, typically requests per second (QPS) for an API, or bytes per second for a pipe. Errors: the rate of failing requests, split by explicit failures (HTTP 500) and implicit ones (a 200 with a wrong or empty body, or a request that blew its latency budget). Saturation: how full the most constrained resource is (CPU, memory, connection pool, queue depth). Saturation is your leading indicator: latency and errors tell you the house is on fire, saturation tells you the wiring is overheating before it ignites.

Interview nuance: interviewers love to ask why you separate the latency of successful requests from the latency of failed ones. Fast failures (a validation 400 returning in 2ms) drag your aggregate latency down and make a struggling service look healthy; slow failures (a request that times out at 30s then 500s) can hide inside an aggregate that averages them with fast successes. Always chart success latency and error latency as separate series, or a bad deploy that fails fast will look like a latency improvement.

RED and USE

• RED (Rate, Errors, Duration) is for request-driven services: an API, a gRPC endpoint, a web handler. Per endpoint you emit request rate, error rate, and duration distribution. This is the workhorse for microservices.
• USE (Utilization, Saturation, Errors) is for resources: a CPU, a disk, a NIC, a connection pool, a thread pool. Per resource you emit how busy it is (utilization), how much work is queued beyond what it can serve (saturation), and its error count.

They are complementary, not competing. RED tells you the checkout API's p99 doubled; USE tells you it is because the Postgres connection pool is saturated and requests are queuing for a connection.

  request-driven service   ->  RED   (Rate, Errors, Duration)
  underlying resource       ->  USE   (Utilization, Saturation, Errors)
  every service, always     ->  4 golden signals

The cardinality trap

The trap that quietly bankrupts observability budgets is cardinality. A metric's cost scales with the number of unique label combinations (time series), not the number of data points. Add a user_id label to a request counter and a service with 5 million users creates up to 5 million time series per metric; Prometheus will OOM and your bill explodes. Keep labels bounded: endpoint, method, status_class (2xx/4xx/5xx), region. Never put unbounded values (user id, order id, full URL with ids, raw error message) in a metric label. High-cardinality identifiers belong in logs and traces, not metrics.

The other common wrong turn is building dashboards nobody watches instead of signal-based alerting. A wall of graphs does not page anyone. Alert on symptoms the golden signals expose (error rate over budget, p99 over SLO, saturation climbing), keep dashboards for diagnosis after the page fires, and delete the ones that have not been opened in a quarter.

Recap: instrument latency, traffic, errors, saturation on every service; use RED for request-driven services and USE for resources; split success vs error latency; treat saturation as your early warning; and guard cardinality by keeping unbounded ids out of metric labels.