Module 8.1: Orchestration Fundamentals
What an orchestrator actually stores and why the order matters: runs bound to an immutable data interval, the task-instance lifecycle, the dependency edges that turn one failure into a blast radius, sensors that wait and time out, and the retry-versus-SLA-versus-silent-success split that decides what deserves a page. All read with SQL over a simulated scheduler metadata catalog.
Runs, Task States, and the Data Interval
Read an orchestrator's own metadata to tell which runs and tasks are healthy, and explain why a run is responsible for a time range rather than for the moment it happened to execute.
Dependencies, Blocked Tasks, and Sensors
Trace why a task did not run by walking the scheduler's dependency edges to whatever blocked it, and read sensor waits as first-class tasks that can time out.
Retries, SLAs, and the Alert You Page On
Decide what deserves a page: separate the transient failures a retry absorbed from exhausted retries and SLA misses, then build the on-call triage report unaided.
Module 8.2: Idempotency, the Core Property
The spine of the level: the three idempotent-write implementations every 2025-2026 source converges on, each graded by literally running your script twice. Partition replacement driven by an injected run parameter instead of the wall clock, a keyed merge that survives in-batch duplicates whose timestamps tie and a batch that redelivers what you already applied, and an append-only landing zone whose idempotency contract is scoped to the projection built on top of it.
Delete + Insert: Replacing the Run's Partition
Write a batch load that owns exactly one date partition, driven by a run parameter instead of the wall clock, and prove it survives a rerun.
Merge Semantics: In-Batch Duplicates, Tied Versions, and Redelivery
Apply a redelivered change batch to a dimension when two changes for one key carry the same timestamp, and keep the merge idempotent.
Append-Only Bronze, Dedup at Read
Keep the write path a blind append and make the read path idempotent instead, then choose between all three idempotency patterns without guidance.
Module 8.3: Incremental Loads, Late Data & Backfills
The half of incremental loading the shipped SQL lessons stop short of: quantifying the rows a strict watermark silently dropped before you fix anything, healing them with a bounded rolling window, running a manifest-driven backfill that also rebuilds the downstream aggregate it invalidated, and composing the whole daily pipeline into one script you can defend out loud.
Watermark Review: The Control Table as ADF Bookmark
Re-run the incremental loop you already know against a control table keyed by pipeline, and name the pattern in the two cloud vocabularies interviewers use.
Late Data: Quantify the Silent Loss, Then Heal It
Measure exactly how many rows a strict watermark load dropped without raising an error, then repair the pipeline with a rolling reprocessing window that self-heals.
Backfills: Manifest-Driven Reruns and the Downstream Trap
Reprocess a date range from a backfill manifest, then rebuild the downstream aggregate for exactly the affected weeks so the mart stops lying.
The Daily Ingestion Pipeline, End to End
Build the whole daily pipeline in one script, stage then load then audit, and rehearse the spoken design answer against a written model answer.
Module 8.4: The Transformation Workflow
The analytics-engineering workflow behind every dbt line on a job description: models as SELECTs built in ref order, data tests and unit tests as queries that pass at zero rows, lineage as a queryable graph, and a dev-to-prod capstone that survives a rerun.
Models Are SELECTs, Built in Ref Order
Materialize a staging-to-mart model chain in the order its ref graph dictates, and derive that order from the graph itself with a recursive CTE.
Data Tests, Unit Tests, and the Grader Behind the Curtain
Collapse the four generic tests into one violation list, then build a model against a hand-written fixture the way a unit test does.
Lineage, Impact Analysis, and Dev vs Prod
Answer what breaks downstream when a source changes, separate dev builds from prod builds, then ship the level capstone unaided.