Schema Management & Evolution

An event schema is a public contract, not a private struct

An event stream outlives every version of every service that touches it. A Kafka topic with 30-day retention holds messages written by last month's producer that today's consumer must still read, and a new consumer that spins up next quarter will replay events from long-dead producers. That is the core insight: an event schema is a public contract, not a private struct. The moment a second team subscribes to your topic, you have lost the freedom to change the shape at will. Rename a field and you break every consumer that still reads the old name, at deploy time, in production, with no compiler to catch it.

The schema registry

The tool that makes this safe is a schema registry (Confluent Schema Registry, Karapace, AWS Glue). Producers register the schema; the registry assigns a numeric schema ID and, critically, rejects any new version that violates the topic's configured compatibility rule before the producer is allowed to publish. On the wire the message carries the compact binary payload plus the schema ID (a few bytes), not the full schema, so you get self-describing evolution without paying to embed the schema on every record. Consumers fetch the writer schema by ID and reconcile it against their own reader schema. Avro, Protobuf, and JSON Schema all support this; Avro's explicit writer-plus-reader resolution is the classic model, Protobuf leans on field numbers and reserved tags.

Compatibility modes are the heart of the interview

• Backward (the common default): new schema can read old data. You may add a field with a default and delete an optional field. Consumers upgrade first, then producers.
• Forward: old schema can read new data. You may add a field and remove one that had a default. Producers upgrade first, then consumers.
• Full: both directions hold. The safe intersection: add or remove only optional fields with defaults, never touch required ones.

The compatibility mode literally dictates your deploy order, which is why it matters operationally and not just in theory.

The rule that follows: add fields with defaults, never remove or rename a required field. A field with a default lets an old consumer that never heard of it simply fall back, and lets a new consumer read old records that lack it. Renaming is the classic trap, because a rename is a delete plus an add and breaks both directions at once.

Making a genuinely breaking change

You do not mutate in place. Three options: upcasting (a transform step that reads v1 and rewrites it to v2 shape on read), the tolerant reader pattern (consumers ignore unknown fields and tolerate missing optional ones, so most additive change needs no coordination), or, for a real break, publish a new topic (user.v2) and run both until every consumer has migrated, then retire v1. New topic is the honest answer when the change cannot be made additive.

Interview nuance: the wrong turn is treating event schemas like internal database columns you can ALTER freely. Downstream teams you have never met are parsing your bytes. The senior move is to name the compatibility mode, show that it fixes deploy order, and reach for a new topic (not a mutation) when a break is unavoidable.

Recap: an event schema is a versioned public contract enforced at produce time by a registry; evolve additively with defaults under a stated compatibility mode, and make true breaks with upcasting, tolerant readers, or a new topic, never an in-place mutation.