Module 10.1: Spark's Execution Model
The vocabulary and mechanics every Spark conversation presupposes, taught by querying the simulated Spark UI: driver and executors, lazy evaluation, and the job to stage to task decomposition.
The Driver, the Executors, and One Task per Core
The two kinds of process in every Spark application, how work lands on core slots one task at a time, and the driver-versus-executor OOM fork read straight off the executor metrics.
Lazy Evaluation: Transformations Build a Plan, Actions Run It
Why Spark refuses to do anything until you ask for a result, how to tell a transformation from an action, and how to count stages off a physical plan by counting its Exchange operators.
Job, Stage, Task: Decomposing a Run You Have Never Seen
Rebuild the job to stage to task hierarchy of an unfamiliar run from its metrics alone, then name where the time actually went.
Module 10.2: Shuffle, Skew & Tuning
From knowing the shuffle exists to tuning it: the broadcast band at 10 MB and 100 MB and the driver-memory bet behind it, shuffle-partition arithmetic against the 2-to-4-per-core heuristic, skew measured as max over median with AQE's real cutoffs, and the full Amazon slow-job investigation on a snapshot you have never seen.
Narrow, Wide, and the Broadcast Decision at 10 MB and 100 MB
Read narrow from wide off the shuffle metrics, pick auto-broadcast, hinted broadcast, or sort-merge with the real byte thresholds, and flag the stages whose shuffle write crosses the 10 GB line.
Shuffle Partition Count: Why 200 Is Wrong for Your Job
Size spark.sql.shuffle.partitions with the 2-to-4-per-core heuristic, spot over- and under-partitioned stages from task metrics, and say what AQE now does for you automatically.
Skew Diagnosis with Percentiles: Max vs Median, Not Max vs Mean
Compute a median and a percentile in plain SQL, apply the max-over-median skew signal and AQE's 5x-median rule, and name the fix ladder in the order an interviewer expects.
Caching Decisions, and the Slow-Job Investigation
Decide when caching pays from the evidence in the metrics, then run the whole Amazon slow-job walk on a snapshot you have never seen and check your diagnosis against the model answer.
Module 10.3: Data Quality & Observability
Where quality results go and how production watches them: the append-only check_results ledger with warn-versus-error severity and a run verdict derived from severity rather than from a failure count, the five observability pillars implemented as always-on monitors over catalog metadata, and the composite incident walk that diffs schema snapshots and follows lineage upstream to the run that succeeded while writing zero rows.
From One Check to a Severity-Ranked Check Ledger
Run a whole check suite in one script, record every result in an append-only check_results ledger with its severity, and derive the run verdict from severity rather than from a raw failure count.
The Five Pillars: Freshness and Volume Monitors in SQL
Name the five observability pillars, then implement the freshness, volume, and distribution monitors as ordinary SQL over table metadata rather than over the data itself.
Schema Drift, Lineage, and the Incident Walk
Detect breaking schema drift by diffing two catalog snapshots, then triage a silent-success incident by walking lineage upstream with a recursive CTE until you find the run that succeeded and wrote nothing.
Module 10.4: Cost & Governance Operations
FinOps and DEA-C01 Domain 4 as audits you run in SQL: attribute spend, quantify the Parquet lever, prove least privilege, and audit PII masking.
FinOps: Cost per Query, Cost per Pipeline
Attribute serverless spend to queries, users, and pipelines from usage metadata, then find the repeated-scan leak that quietly doubles a bill.
Quantifying the Parquet and Partitioning Lever
Compute, rather than assert, the cost gap between scanning raw CSV and a partitioned Parquet layout, then price the small-files problem that erodes it in dollars per month.
IAM Least Privilege for a Pipeline, by Audit
Evaluate a policy set the way the platform does (default deny, explicit deny wins), audit granted-but-unused permissions, and explain the canonical Athena Access Denied from policy data alone.
PII Tags, Masking Policies, and the Governance Audit
Audit a catalog for untagged and unmasked PII, then produce the exposure report a governance review asks for, unaided.