Module 6.1: The Cloud Data Platform
What the cloud is and how a data platform is shaped: regions and managed services, object vs block vs file storage, storage classes and lifecycle, and the files-plus-catalog-equals-a-table idea, all queried as platform metadata.
What the Cloud Is, and the Shape of a Data Platform
Regions, availability zones, and managed services, then the five decoupled layers every data platform is built from: storage, catalog, compute, orchestration, serving.
Object vs Block vs File Storage, and Why the Lake Lives on S3
The three storage shapes every cloud offers (S3, EBS, EFS), the immutable-blob-over-HTTP object model, eleven-nines durability, and why a data lake is files in a bucket.
Storage Classes and Lifecycle: Paying Less for Colder Data
Hot, warm, cold, and archive tiers, the retrieval-latency and minimum-duration trade, lifecycle policies, and computing the bill (and the saving from a transition) with a join.
Files Plus a Catalog Equals a Table: Lake, Warehouse, Lakehouse
How a pile of files in a bucket becomes a queryable table (the Hive metastore / Glue catalog), the lake vs warehouse vs lakehouse split, and auditing the catalog with SQL.
Module 6.2: File Formats and Why Columnar Wins
Why Parquet beats CSV for analytics: column projection reads only the columns you select, columnar compression and encodings shrink each column hard, and row-group min/max stats let predicate pushdown skip data unread. Plus when a row format (Avro) is the right call.
Rows vs Columns on Disk, and Column Projection
Why a CSV is row-oriented and Parquet is column-oriented, and the first thing columnar buys you: a query reads only the columns it selects, not every byte of every row.
Compression and Encoding: Why Columnar Shrinks So Much
Dictionary, run-length, delta, and bit-packing encodings plus a codec (Snappy, Zstd, Gzip) shrink a column far more than a row-interleaved CSV, and you can measure the ratio in SQL.
Row Groups and Predicate Pushdown, plus Parquet vs ORC vs Avro
Inside a Parquet file: row groups, the footer, and the min/max stats that let a filter skip whole row groups without reading them. Then when a row format (Avro) is the right call.
Module 6.3: Partitioning a Large Table
The single most-asked 'make a big table fast and cheap' topic: how dt=value partition folders let a filter prune to a fraction of the data, how to choose the partition key without shredding the table into tiny files, and how bucketing and the full-scan trap fit in.
What a Partition Is, and Why Pruning Makes a Big Table Cheap
Hive-style dt=value folders put the partition key in the path, so a filter on it reads only the matching folders (pruning) and scans a fraction of the bytes, which on a pay-per-scan engine is directly cheaper.
Choosing a Partition Key, and the Small-Files Problem
Partition on the low-cardinality column you filter on (usually date). Partition on a high-cardinality key and you shred the table into millions of tiny files, where per-file overhead dominates.
Bucketing vs Partitioning, and the Full-Scan Trap
Bucket (or cluster) a high-cardinality join column into a fixed number of files instead of partitioning it, and keep the partition key bare in a filter so the engine can actually prune.
Module 6.4: Distributed Processing & Pipelines
How big data is actually computed and how it flows: an engine splits data into partitions and tasks and pays for the shuffle, data skew creates stragglers, a broadcast join avoids shuffling a huge table, a pipeline of scheduled, idempotent jobs (the medallion) moves data raw to useful under a freshness SLA, and data-quality checks and gates stop a green-but-wrong run from publishing bad data.
How a Distributed Engine Splits Work, and Why the Shuffle Is Expensive
Data splits into partitions, a job into stages, a stage into one task per partition. Narrow steps stay put; wide steps (group-by, join) force a shuffle across the network, the dominant cost of a big job.
Data Skew, Stragglers, and Broadcast vs Shuffle Joins
One hot key overloads one task and the whole stage waits on that straggler. For a huge-to-tiny join, broadcast the small side so the big table never shuffles (Spark auto-broadcasts under 10 MB).
Pipelines, Orchestration, and Idempotency
A DAG of scheduled jobs moves data raw to useful (medallion Bronze/Silver/Gold, ELT). The reliability ideas that matter: idempotent re-runs, backfills, and freshness SLAs, all queryable from a run log.
Data Quality Checks and Quality Gates
A pipeline can succeed and still write garbage. The check families a junior is expected to name (uniqueness, completeness, validity, volume, referential integrity) and the quality gate that stops bad data, all written as plain SQL.