Event Time, Watermarks, and Late Events

Two clocks, and why they drift apart

Every event carries two timestamps. Event time is when the thing happened, stamped by whatever produced it. Processing time is when your pipeline saw it. In the previous lesson you bucketed on event_time_ms and ignored processing_time_ms entirely; this lesson is about the gap between them.

The gap is never zero and it is rarely small. A phone goes through a tunnel and buffers 20 minutes of taps. A retry queue drains after an outage. A partition rebalances and a consumer pauses. A client's clock is simply wrong. Every one of those makes an event arrive long after it happened, and the difference is the event's lateness:

processing_time_ms - event_time_ms

Windowing on processing time makes lateness invisible and the numbers wrong: a tunnel full of 17:05 taps lands at 17:25 and inflates the 17:25 window while the 17:05 window under-reports forever. Windowing on event time puts every event in the bucket where it belongs, which is correct but raises a new question with no free answer: when is a window done?

A watermark is a promise about completeness

A watermark is the engine's assertion that all events with an event time at or before time t have arrived. It advances behind the largest event time observed, trailing it by a bounded out-of-orderness you choose. Flink names that setting forBoundedOutOfOrderness, and its SQL form is the - INTERVAL '10' MINUTE inside a WATERMARK clause. With a 10-minute watermark, an event whose lateness exceeds 10 minutes arrives after the engine already declared its window complete, so the engine calls it late.

Do not confuse that trailing interval with allowed lateness, which is a different knob covered below. Bounded out-of-orderness decides when a window is declared complete; allowed lateness decides how long its state survives afterwards.

That single number is the completeness-versus-latency knob, and it has no correct setting. Set it to 1 minute and dashboards update fast but drop stragglers. Set it to an hour and almost nothing is missed but every window emits an hour behind. Interviewers ask what you would pick precisely because there is no right answer, only a tradeoff you can defend.

Table

The watermark is a line sweeping through processing time. Lateness is measured against event time, so the verdict depends on the gap, not on when the event happened.

event_id	event time	processing time	lateness	10-minute watermark
8003	17:02:00	17:02:06	6 s	on time
8004	17:03:10	17:11:10	8 min	on time
8016	17:12:20	17:22:20	10 min exactly	on time, the boundary
8008	17:05:50	17:17:50	12 min	LATE
8013	17:09:30	17:34:30	25 min	LATE

Note the boundary row. The watermark asserts "everything at or before t has arrived", so an event exactly at the allowed lateness is still inside the promise. Late means strictly greater. Writing >= instead of > misclassifies exactly the events that sit on the line, which is exactly where a grader looks.

One thing to notice before you write the queries: they test processing_time_ms - event_time_ms > 600000 rather than comparing an event time against a moving watermark line. That substitution is only valid because the on-time events in this seed arrive within seconds, so the largest event time observed tracks the wall clock and the watermark sits a flat 10 minutes behind both. On a real stream the watermark lags the maximum event time rather than the clock, and the two come apart the moment a source goes idle and stops advancing it.

Where late events go

This is asked openly and word for word, and there are four canonical answers. Naming all four with a tradeoff each is the strong response.

1. Drop them. Simplest, cheapest, and the right call when the metric tolerates a small undercount. It is also silent, so nobody notices the loss.
2. Allowed lateness updates. Keep the window's state alive past its emission and re-emit a corrected result when a straggler shows up. This is the second knob: allowed lateness is the grace period that begins AFTER the watermark has already passed the window, during which the window's state is retained so a straggler can trigger a restatement instead of being dropped. Correct, but every downstream consumer must accept a restatement of a number it already read.
3. Side-output to a dead-letter queue. Route late events to a separate stream or table with the record, the reason, and a timestamp, then alert on the volume. Nothing is lost, and someone gets to decide what to do about it.
4. Append-only storage plus a windowed reprocess. Land every event raw, then recompute a bounded rolling window (the last few days) on a schedule so late arrivals self-heal without any streaming state at all. This is the pattern most batch-leaning teams actually ship.

Flink, in two words a junior needs

Awareness level, not implementation. Checkpointing is Flink's periodic consistent snapshot of operator state: barriers flow through the stream alongside the records, and on failure the job restarts from the last completed checkpoint. That mechanism is what makes its exactly-once state claim true, and it is the reason a Flink job's recovery story is "resume from a snapshot" rather than "reprocess from the beginning". Backpressure is a slow operator forcing the stages upstream of it to throttle so buffers do not blow up; from outside the job it looks like consumer lag climbing steadily with no error in the log. Say those two words correctly and stop there. Tuning either one is senior scope. On AWS, stateful Flink jobs run on Amazon Managed Service for Apache Flink, which was renamed from Kinesis Data Analytics in 2023; the old name still circulates in prep material and should not be the one you say out loud.

Common mistake: subtracting in the wrong direction and getting negative lateness, then filtering it away. Lateness is processing_time_ms - event_time_ms, arrival minus occurrence, and it is positive by construction unless a client clock is ahead of the platform's, which is itself a real finding worth reporting rather than hiding.

Interview nuance: event time versus processing time is the single most transferable streaming concept, and the way to prove you own it is with the incident story rather than the definition. "Yesterday's number changed overnight" is almost always a window that was emitted before its late events arrived, and the fix is stated as a policy choice from the four options above, not as a bug.

On a real platform this differs. Flink declares a watermark strategy on the source (WATERMARK FOR event_time AS event_time - INTERVAL '10' MINUTE) and tracks it per partition, taking the minimum across partitions so one idle partition can stall the whole job. Spark Structured Streaming spells the same idea withWatermark("event_time", "10 minutes") and uses it to decide when state can be dropped. Here the watermark is a constant in a predicate, which makes the arithmetic visible; on a platform it is a moving line the engine maintains for you, and the thing you actually configure is the out-of-orderness budget, with allowed lateness configured separately on the window.