Lazy Evaluation: Transformations Build a Plan, Actions Run It

Nothing happens until you ask for something

Write df.filter(...).select(...).join(...) in a notebook and Spark does no work at all. Those calls are transformations: each one returns a new DataFrame that records what you asked for, and the recording is all that happens. Work starts only when you call an action: count(), collect(), show(), write(), take(), foreach(). An action submits a job, and one action is at least one job in the Jobs tab, usually exactly one. A write is the common exception: the data write and the commit of the written files can be scheduled as separate jobs, so one write.parquet often shows up as two rows.

The tell is the return type, and it is worth saying out loud in an interview because it is exact. A transformation returns another DataFrame. An action returns something that is not a DataFrame: a number, a list of rows, or nothing at all because it wrote to storage.

That rule settles the gotcha juniors get caught on most. df.count() returns a number, so it is an action and it submits a job. df.groupBy("x").count() returns a DataFrame, so it is a transformation and it submits nothing. Same method name, opposite answer, and the return type is what tells you which one you are looking at.

Table

The line every Spark question hangs on. Transformations record intent and return a DataFrame. Actions return a result and submit a job.

call	kind	returns	submits a job?
filter, select, withColumn	transformation	a DataFrame	no
join, groupBy().agg()	transformation	a DataFrame	no
groupBy().count()	transformation	a DataFrame	no, despite the name
cache, persist	transformation	a DataFrame	no, it only marks the plan
df.count(), collect, take	action	a number or rows	yes
show	action	nothing, it prints	yes
write.parquet	action	nothing, it writes	yes, sometimes more than one

Why laziness is a feature, not a delay

If Spark ran each transformation the moment you typed it, it would read the whole table for the filter, write the result, read it back for the select, and so on. Because it waits, the optimizer gets to see the entire chain before a single byte is read, and it can rewrite the chain:

• Column pruning. You select three columns out of forty, so it reads three columns from the Parquet files.
• Predicate pushdown. Your filter becomes a PushedFilters entry on the scan, so row groups that cannot match are skipped in the file reader rather than loaded and discarded.
• Operator fusion. Consecutive narrow steps run in one pass over each partition instead of materializing between them.

None of that is possible if each step executes on arrival. This is also why laziness makes debugging feel strange: your typo in a join condition does not raise anything until the count() twenty lines later, because that is the first moment Spark actually looks.

Reading a physical plan at the only level you need

df.explain() prints the physical plan, and it prints it root first: the last operator to run is the top line, each +- below it is that operator's input, and the file scan is at the bottom. So you read an explain output from the bottom up, and the tables in this lesson number the steps in that reading order, with the scan as step 1.

You are not expected to parse all of it in a junior interview. You are expected to recognize one operator: Exchange. An Exchange is a shuffle, a shuffle is where data moves across the network to co-locate by key, and every shuffle is a stage boundary. That gives you an arithmetic you can do at a glance:

stages = Exchange operators + 1

A plan with no Exchange runs in one stage: scan, filter, project, done, no data ever moves between machines. A sort-merge join shows an Exchange on each side, and a group-by after it shows a third, so that plan runs in four stages. Counting them is the fastest read of "how expensive is this query" available to you.

The operator you count is the plain shuffle Exchange. Plans also print BroadcastExchange, which you will meet in the next module: it ships one small side of a join to every executor instead of repartitioning both sides by key, so it does not cut the plan into stages the same way and the arithmetic above does not apply to it.

Common mistake: believing that only actions ever submit jobs. Almost true, and the exception is worth knowing: reading CSV or JSON with schema inference scans the file to work out the types, and that scan shows up as a job in the UI before you have called any action. Reading Parquet does not, because the schema is in the file footer. That is why the Jobs tab of a fresh notebook sometimes has a job in it and nobody can find the action that caused it.

Interview nuance: "why is Spark lazy" is a Tier A question and a one-sentence answer scores: because deferring execution lets the optimizer see the whole plan and prune columns and push filters down before reading any data. If you are then asked about .explain(), do not try to narrate the tree. Say that you look for Exchange operators, because each one is a shuffle and a stage boundary, and that tells you where the cost is.

On a real platform this differs. Here you query a saved copy of the plan and a saved copy of the Jobs tab. In production you call df.explain(True) for the parsed, analyzed, optimized and physical plans, or read the SQL tab of the Spark UI, which draws the same plan as a graph with per-operator row counts. The Python course makes this executable a different way: a roughly 100-line pure-Python MiniSpark, where .map and .filter build a plan, .collect() runs it, and .explain() prints the stages with Exchange markers, so you can watch laziness happen instead of reading about it. One currency note: Spark 4.0 shipped in mid-2025 and is generally available on EMR, and it turns ANSI mode on by default, so a divide by zero or a bad cast now fails fast instead of quietly returning NULL. Answers built on the old silent-null behavior are out of date.