Stream Processing: Windowing, Watermarks & State

Batch answers a bounded set; a stream never completes

Batch processing sees a whole bounded dataset and computes an answer. Stream processing computes continuously over an unbounded, never-complete dataset, so the hard question is not "what is the answer" but "when do I emit it, given that more data for the period I am aggregating might still arrive." Everything in stream processing flows from three clocks and a promise about lateness.

Three clocks

Event time is when the thing actually happened (stamped by the producing device). Ingestion time is when the broker received it. Processing time is when your operator sees it. These differ because of network delay, mobile clients going offline, retries, and partition skew. A phone can buffer events for 30 seconds in a tunnel, then flush them. If you aggregate by processing time, that burst lands in the wrong 5-minute bucket and your per-user counts are silently wrong. Correct real-time analytics almost always uses event time.

Windows

You cut the infinite stream into finite chunks. Tumbling windows are fixed, non-overlapping (every event in exactly one 5-minute bucket). Sliding windows overlap (a 5-minute window advancing every 1 minute, so each event is in five windows: this is what you want for "rolling 5-minute rate"). Session windows group bursts separated by a gap of inactivity (great for user sessions, sized dynamically).

Watermarks

A watermark is the engine's assertion "I believe I have now seen all events with event time <= T." It is a heuristic, usually max_event_time_seen - allowed_lateness. When the watermark passes a window's end, the window fires and emits its result. This is the mechanism that lets an unbounded stream produce bounded, timely output. You tune the lateness bound: a tight bound (say 5s) gives low latency but drops stragglers; a loose bound (say 5 min) waits longer but captures late data. Allowed lateness additionally keeps a window's state around after firing so a late event can trigger an updated (retracted/corrected) result instead of being dropped.

Interview nuance: the single most common wrong answer is "just use processing time and drop late events." Say out loud what you do with late data: route to a side output / dead-letter, or emit a correction. Silent drops are a correctness bug that never pages anyone.

Fault-tolerant state

Aggregations are stateful (a per-user counter lives somewhere). Flink keeps this in an embedded RocksDB state backend on each task's local disk, and periodically takes a checkpoint: a consistent snapshot of all operator state plus the source offsets, written to durable storage (S3/HDFS) using the Chandy-Lamport barrier algorithm. On failure it restores the last checkpoint and rewinds Kafka to the checkpointed offsets, giving exactly-once state semantics (each event affects state once, even though it may be reprocessed). Kafka Streams does the same idea with a compacted changelog topic backing each local store.

Engine choice

Flink: richest event-time/state/CEP support, true exactly-once via checkpoints, best for complex low-latency work. Kafka Streams: a library (no cluster), great when you already live in Kafka and want per-partition local state. Spark Structured Streaming: micro-batch, best if you already run Spark and can tolerate slightly higher latency. Joins matter too: stream-stream joins need windowed state on both sides; stream-table joins enrich events against a materialized KTable (a changelog folded into current-value-per-key).

Recap: aggregate by event time, use watermarks to bound lateness and fire windows, keep local state fault-tolerant via RocksDB plus checkpoints/changelogs for exactly-once, and never drop late data silently.