Real-Time Recommendation Systems

A latency-constrained funnel

A recommender is a latency-constrained funnel that turns a catalog of millions into an ordered list of a dozen, personalized to what the user did seconds ago, in under 100ms. You cannot run a heavy ranking model over millions of items per request, so the entire design is about narrowing the set cheaply before spending compute where it matters.

millions of items
   -> candidate generation   (two-tower + ANN, ~5ms)      -> ~1000 candidates
   -> ranking                (deep model on candidates)    -> ~100 scored
   -> re-ranking             (diversity, freshness)        -> ~20
   -> business rules         (dedup, blocklist, ads)       -> final feed

Candidate generation with two-tower + ANN

Candidate generation must be sublinear in catalog size. Train two encoders: a user tower that maps user features (history, context) to a vector, and an item tower that maps item features to a vector in the same space, so that dot product approximates relevance. Precompute all item vectors offline and load them into an ANN index (HNSW or IVF). At request time you compute only the user vector and do an ANN lookup for its nearest item vectors. That is how you retrieve the top 1000 relevant items from millions in a few milliseconds. Item vectors refresh nightly (batch), while the user vector can be computed fresh per request from recent activity, which is what makes it react to the last few clicks.

Ranking and real-time signals

Ranking then runs a heavier model (gradient-boosted trees or a deep network) on the ~1000 candidates, using richer features and cross-features that would be too expensive at retrieval scale. Modern rankers are multi-task: they jointly predict click, watch-time or dwell, and conversion, then combine those into one score, because optimizing clicks alone trains clickbait. Calibrated probabilities matter when you blend objectives or mix in ads priced by expected value.

The user's last few clicks reach the recommender within seconds via Kafka plus Flink, updating either the user embedding or fast counter features. The common split is near-line (compute embeddings and features within seconds of an event, store them) versus online (per-request scoring), which keeps the request path fast while still reacting quickly.

Interview nuance: the evaluation answer separates senior from junior. Your logs are biased: users can only click what you showed them (position bias) and popular items get shown more (popularity bias), so naively training on click logs makes the model recommend what it already recommends. You break the loop with exploration (bandits or epsilon-random slots) to gather counterfactual data, and you evaluate with offline replay plus a real online A/B test, not just offline AUC.

Cold start

Both new users (fall back to popularity, context, or onboarding signals until you have history) and new items (rely on content features in the item tower so a brand-new item still gets an embedding without interaction data) need an explicit answer.

Recap: cascade two-tower plus ANN candidate generation into a multi-task ranker into diversity re-ranking to hit p99 under 100ms, feed recent clicks through Kafka/Flink for real-time reaction, handle cold start with content features and popularity, and evaluate with exploration plus online A/B to escape feedback-loop bias.