Inter-Service Communication

How services talk decides whether the system is resilient

Once you have services, how they talk determines whether the system is resilient or a house of cards. Two axes: synchronous versus asynchronous, and orchestration versus choreography.

Sync vs async

Synchronous request-response (REST or gRPC) is right when the caller needs the answer now to proceed: checkout must know if the payment authorized. Asynchronous messaging (events on Kafka, RabbitMQ, or SQS) is right when the caller does not need to wait and you want to decouple: after an order is placed, notifications, analytics, and the loyalty service should each react without checkout waiting on any of them. The failure-mode difference is the whole point. A synchronous chain A -> B -> C -> D means if D is slow, A is slow, and threads pile up all the way back, so one slow service stalls the entire flow and can cascade into total outage. An async hop absorbs the slowness in a queue; D drains it when it recovers.

Protocol choice

For internal east-west traffic between services, gRPC with protobuf is the default: binary, strongly typed, HTTP/2 multiplexed, and much faster than JSON over HTTP/1. For external north-south traffic to browsers and third parties, REST or GraphQL over HTTP/JSON wins on ubiquity and tooling. So a common shape is REST at the edge, gRPC inside.

Orchestration vs choreography

 Orchestration                     Choreography
 [ Coordinator ] --> Payment       Order -event-> Payment -event-> Inventory
   (saga)        --> Inventory     each service reacts to the
                 --> Shipping      previous one's event; no central brain
 central visibility, one           low coupling, but flow is
 place to change, but a            implicit and hard to trace
 coupling hotspot

Orchestration puts a central coordinator (a saga orchestrator like Temporal or a workflow service) in charge of calling each service in order and handling failures. You get one place to see and change the flow, at the cost of a component that knows about everyone. Choreography has each service emit events and react to others' events with no central brain: lowest coupling, but the end-to-end flow lives nowhere and is painful to debug and reason about. Rule of thumb: choreography for simple 2 to 3 step flows, orchestration once a workflow has real branching, compensation, and needs auditability.

Consistency without 2PC

You cannot run an ACID transaction across services, and two-phase commit does not scale and blocks on failure. The pattern is the saga: a sequence of local transactions where each step has a compensating action. If Shipping fails after Payment succeeded, you run the compensation for Payment (refund) rather than rolling back a distributed transaction. Sagas give you eventual consistency, not atomicity, and you must design the compensations explicitly.

Interview nuance: resilience primitives are almost always probed. Every sync call needs a timeout (never infinite), retries with exponential backoff and jitter (to avoid retry storms), a circuit breaker (stop calling a dead dependency so it can recover and you fail fast), and idempotency keys (so a retry does not double-charge). Backpressure and bounded queues stop a fast producer from drowning a slow consumer. These four, timeout plus retry-with-jitter plus circuit breaker plus idempotency, are what prevent one failure from cascading.

Recap: sync for must-know-now reads (gRPC inside, REST at the edge), async events to decouple and absorb slowness, orchestration for complex flows and choreography for simple ones, sagas with compensations for cross-service consistency, and timeouts plus jittered retries plus circuit breakers plus idempotency to stop cascades.