What the Cloud Is, and the Shape of a Data Platform
Regions, availability zones, and managed services, then the five decoupled layers every data platform is built from: storage, catalog, compute, orchestration, serving.
What "the cloud" really is
The cloud is renting other people's computers and services over the internet, billed by usage, instead of buying and racking your own hardware. For a data engineer that reframes cost as a design choice: you pay for bytes stored, bytes a query scans, requests made, and data moved, so the cheapest query is the one that reads the fewest bytes. That single idea drives everything in this level.
Two words you will hear on day one:
- A Region is a geographic area where a cloud provider clusters data centers (for example
us-east-1in North Virginia). Regions are fully independent, and data in a Region stays in that Region unless you copy it out. - An Availability Zone (AZ) is one or more discrete data centers inside a Region, each with its own power and networking. Every Region has at least three AZs, physically separated but linked by fast private networking. One AZ can fail, so spreading data and compute across several is how a platform stays up.
A managed service is the other half of the mental model. Instead of running the server yourself, you consume a capability through an API and the provider handles the machines, patching, scaling, and failover. Object storage (Amazon S3), serverless SQL (Amazon Athena), and a warehouse (Amazon Redshift) are all managed: you rent the capability, not the box.
The shape of a data platform
Almost every cloud data platform is the same five layers, and knowing the layers is how you place any new tool you meet.
- Storagefiles on object storage (S3)
- Catalogschema for those files (Glue)
- Computeruns the SQL (Athena, Spark)
- Orchestrationschedules the jobs (Airflow)
- Servingdashboards, APIs, cache
- Storage holds the raw and processed data as files, cheaply and at any scale (the data lake on object storage).
- Catalog records what those files mean: table names, columns, types, and where the files live. Without it, a folder of files is just bytes.
- Compute is the engine that reads the files and runs your SQL or jobs (Athena, Spark, a warehouse). Compute is separate from storage, so you can scale them independently and point many engines at the same data.
- Orchestration schedules the jobs and wires their dependencies into a pipeline that runs on time.
- Serving is where cleaned results land for people and apps: dashboards, an API, a cache.
The defining feature of the modern platform is that these layers are decoupled. Storage is just files, the catalog gives them a schema, and compute is rented by the query. That separation is why "run SQL over a folder of files" is a normal sentence, and it is the backdrop for the rest of this level.
Reading the platform with SQL
You do not just build these layers, you operate them, and operating them means asking questions: what does each layer cost, which services are compute versus storage, where is the money going. Those questions are answered by querying a catalog of the platform itself. The exercises in this level do exactly that: you query a stand-in for each layer's own metadata, which is a real data-engineering skill (teams query cost-and-usage reports, storage inventories, and job logs with SQL every day).
Common mistake: assuming storage is the expensive part. On most platforms the compute layer (query engines and warehouses) dominates the bill, which is why reading fewer bytes per query is the lever that matters.
Interview nuance: when an interviewer asks you to "describe a data platform," the five-layer storage / catalog / compute / orchestration / serving spine is a clean, complete answer, and naming the decoupling of storage from compute is the detail that shows you understand cloud economics.
On a real platform this differs. Here you query one small
platform_servicestable in SQLite. On AWS the same reasoning runs against the Cost and Usage Report (queried in Athena or loaded into a warehouse), where each row is a service, usage type, and cost. The query shape (group by a layer or service, sum the cost, sort) is identical.
CREATE TABLE platform_services (
service TEXT,
layer TEXT, -- storage | catalog | compute | orchestration | serving
monthly_cost_usd INTEGER
);
INSERT INTO platform_services (service, layer, monthly_cost_usd) VALUES
('S3 data lake', 'storage', 1200),
('Glue Data Catalog', 'catalog', 150),
('Athena', 'compute', 900),
('EMR Spark cluster', 'compute', 2600),
('Redshift warehouse', 'compute', 3100),
('Airflow (MWAA)', 'orchestration', 400),
('QuickSight BI', 'serving', 350),
('API cache (Redis)', 'serving', 220);-- The platform catalog: one row per service, tagged by the layer it belongs to.
SELECT service, layer, monthly_cost_usd
FROM platform_services
ORDER BY monthly_cost_usd DESC;Apply
Your turn
The task this lesson builds to.
Write a query that returns the total monthly cost of each platform layer, as (layer, monthly_cost), most expensive layer first, over platform_services(service, layer, monthly_cost_usd).
Group the services by their layer and sum monthly_cost_usd. Alias the summed column exactly monthly_cost, ordered descending.
3 hints and 1 automated check are waiting in the workspace.
Practice
Make it stick
A second problem on the same idea, plus 4 bonus drills.
Write a query that returns each platform layer running more than one service, with its service count and total monthly cost, as (layer, service_count, monthly_cost), most expensive first, over the same platform_services table.
Keep only layers with more than one service. Alias the columns exactly service_count and monthly_cost, ordered by monthly_cost descending.
3 hints and 1 automated check are waiting in the workspace.