What Iceberg Solves: Snapshots, Manifests, and the Metadata Tree

Where "files plus a catalog" runs out

Level 6 built a table out of two things: Parquet files in a bucket, and a catalog entry saying where they live and what the columns are. That gets you a queryable table for almost nothing, and it is genuinely how a lake starts. It also has three holes that show up the first week a second person touches the data.

• No transactions. A job that rewrites a partition deletes some files and writes others. A reader that lands in the middle sees half the old data and half the new. There is no commit, so there is nothing to be atomic about.
• No safe concurrent writers. Two jobs writing the same table race on the same prefix. The catalog has no idea a write is in progress, so last-writer-wins silently destroys work.
• A frozen layout, and no way back. The partition scheme is baked into the folder paths, so changing from daily to hourly means rewriting every path in the table. And once a file is overwritten, yesterday's answer is gone. You cannot reproduce the number you put in a dashboard last Tuesday.

An open table format fixes exactly these. Files stay Parquet in object storage, and any engine can still read them, but a layer of metadata turns the pile of files into a table with commits.

The metadata tree

Iceberg's whole idea is that a table is defined by metadata, not by whatever happens to be sitting under a prefix. The catalog stores one pointer: the current metadata file. Everything else hangs off it.

Order of evaluation

1. Catalogone pointer: current metadata file
2. Snapshotone commit; schema + partition spec
3. Manifest listthe manifests in this snapshot
4. Manifestdata files + per-file stats
5. Data filesordinary Parquet in the bucket

Read it top down and the properties fall out:

• A snapshot is one commit: the complete set of data files that made up the table at that instant. Committing means writing new metadata and then swapping the catalog pointer to it in a single atomic operation. That swap is the transaction. Readers already holding the old pointer keep reading a consistent older snapshot, so a writer never tears a reader in half.
• A manifest lists data files together with per-file statistics: row count, file size, and column lower and upper bounds. Those stats are the reason a planner can throw away files before opening any of them.
• Readers never list the bucket. On a plain lake, planning a query means asking object storage for every key under a prefix, which is slow and gets slower as the table grows. Iceberg reads a manifest instead, so planning cost tracks the number of files in the metadata, not the number of objects in the bucket.

Hidden partitioning and schema evolution

Two features come up constantly and each is one paragraph of understanding.

Hidden partitioning. On a Hive-style lake the partition value is part of the path, so the query has to filter on the partition column by hand (WHERE dt = '2026-07-01'), and a user who filters on the raw timestamp instead gets a full scan. Iceberg stores the partition rule as a transform of a real column (for example days(event_ts)), records the derived value in the manifest, and applies it for you. You filter on event_ts and pruning still happens. Because the rule lives in metadata rather than in paths, you can also change it later (partition evolution): new data is written under the new scheme, old data keeps the old one, and both are still readable.

Schema evolution by field id. Every column carries a numeric field id, and data files reference ids, not names. Renaming a column is a metadata edit. Adding, dropping, and reordering columns never require rewriting files, and a dropped-then-re-added column cannot resurrect old values, because the new column gets a new id.

Delta, Hudi, and who won

Iceberg is the de facto open standard: vendor-neutral governance and the broadest engine support (Spark, Flink, Trino, Snowflake, BigQuery, DuckDB, AWS S3 Tables). The v3 spec was ratified in mid-2025, adding deletion vectors, a variant type, and geospatial types. Delta Lake remains strong inside Databricks, and its UniForm feature emits Iceberg-readable metadata beside Delta's own. Hudi holds a streaming and CDC niche. The pattern worth naming in an interview is the consumption gravity: every non-Iceberg format now ships an Iceberg-readable surface, and the reverse investment does not exist.

Worth keeping in proportion: in the 2026 State of Data Engineering Survey (n = 1,101), 43.8% of teams run a cloud warehouse, 26.8% a lakehouse, and 11.7% a hybrid. The warehouse still dominates entry-level work. Lakehouse literacy is expected, not lakehouse everywhere.

Reading it with SQL

The payoff for this level is that Iceberg exposes its own metadata as ordinary tables you can query. snapshots gives you the commit history. files gives you every data file, the snapshot that added it, and the snapshot that stopped pointing at it. A file with no deleting snapshot is live right now, which makes "what would this query scan" a plain WHERE clause.

Common mistake: reading the files table without filtering deleted_snapshot_id IS NULL and reporting the result as the table's size. That set includes every file the table ever wrote, including the ones replaced by later commits, so the number can be several times the live footprint.

Interview nuance: the junior bar for "what does Iceberg actually solve" is the problem statement, not the spec. One sentence covers it: ACID transactions, schema evolution, and time travel over Parquet files in object storage, with a catalog pointer telling engines which files are current. Catalog wars and manifest internals are senior awareness, and volunteering them before the one-sentence answer reads as memorized.

On a real platform this differs. These seeds mirror Iceberg's real metadata tables. In Spark or Athena you query them directly with SELECT * FROM analytics.orders.snapshots and SELECT * FROM analytics.orders.files, and snapshot_id, parent_id, committed_at, operation, file_path, record_count, and file_size_in_bytes carry exactly these names. Four columns here are a teaching simplification, and knowing which ones saves you a column-not-found error on your first real query. Real snapshot ids are large random longs, so production code walks the parent_id chain or compares committed_at rather than using <. The real snapshots table has no added_files or deleted_files column; those counts are string entries in its summary map, read as summary['added-data-files'] and summary['deleted-data-files']. The real files table exposes partition as a struct of the partition-spec fields rather than a flat partition_dt. And it carries no added_snapshot_id or deleted_snapshot_id at all: it shows one snapshot's live files at a time, and the commit that added or dropped a file is reachable through the entries and all_entries metadata tables.