Storage Classes and Lifecycle: Paying Less for Colder Data

Level 6: Level 6: Cloud & Data Engineering Foundationsmedium26 minS3 storage classeslifecycle policiescost modelingJOINaggregationROUND

Hot, warm, cold, and archive tiers, the retrieval-latency and minimum-duration trade, lifecycle policies, and computing the bill (and the saving from a transition) with a join.

Hot, warm, cold, and archive

Not all data is read equally. Yesterday's events are queried constantly; last year's are touched once a quarter for a compliance check. Object storage gives you storage classes so you can pay less for colder data, trading away instant, free retrieval as the data gets colder. Every class is designed for the same eleven nines of durability. What changes is the price per gigabyte, the retrieval latency, and a minimum billing duration.

Table

Colder classes cost less per GB but add retrieval latency and a minimum billing duration. Illustrative US prices.
storage class	$/GB-month	retrieval	min duration	use for
S3 Standard	0.023	instant (ms)	none	hot, active tables
Standard-IA	0.0125	instant (ms)	30 days	rarely read, needs instant
Glacier Instant Retrieval	0.004	instant (ms)	90 days	archive, rare but instant
Glacier Deep Archive	0.00099	hours	180 days	once-a-year compliance

Colder classes cost less per GB but add retrieval latency and a minimum billing duration. Illustrative US prices.

The pattern to internalize:

S3 Standard is for hot, frequently read data (your active lake and warehouse tables): highest per-GB price, instant retrieval, no minimum.
Standard-Infrequent Access (Standard-IA) is for data you still need instantly but read rarely (backups, older-but-live tables): cheaper per GB, but it bills a 30-day minimum and adds a per-GB retrieval fee.
Glacier Instant Retrieval is archive priced, still milliseconds to read, with a 90-day minimum.
Glacier Flexible Retrieval reads take minutes to hours (with a 90-day minimum). Glacier Deep Archive, the cheapest "look at it once a year" compliance tier, takes hours to restore (roughly 12 hours standard, up to 48 hours bulk, with a 180-day minimum).

Lifecycle policies do this for you

You do not move objects between classes by hand. An S3 Lifecycle policy is a set of rules that transition objects to a colder class after N days (Standard to Standard-IA at 30 days, to Glacier at 365) and eventually expire (delete) them. It is how a data engineer automates hot to cold to archive to gone without a cron job.

There is a real trap here, and it is a favorite interview follow-up: moving cold data to a cheaper class does not always save money. The infrequent and archive classes have minimum storage durations and per-GB retrieval fees, so transitioning tiny or short-lived objects, or objects you will actually read often, can cost more than leaving them in Standard. The saving is real only for large objects that genuinely sit cold.

Doing the math with SQL

Cost is just arithmetic over the inventory: size in gigabytes times a per-GB price, summed. That is a join between the object inventory and a small price list, and it is exactly the kind of query a DE writes to justify a lifecycle change. The exercises here compute the current bill per class and the saving from a proposed transition.

Common mistake: assuming a colder class is always cheaper. Infrequent-access and archive tiers add per-GB retrieval fees and minimum storage durations, so moving small or frequently read objects there can cost more than leaving them in Standard.

Interview nuance: asked "you have 10 TB of logs you rarely read but must keep seven years, how do you store it cheaply," the answer is a lifecycle policy that transitions to Glacier Flexible or Deep Archive, and the detail that completes the answer is naming the retrieval-latency and minimum-duration trade rather than just saying "use Glacier."

On a real platform this differs. Real per-GB prices vary by Region and change over time, and AWS bills a "GB" as a specific byte count. The storage_pricing table here uses round, illustrative US prices and treats 1 GB as 1,000,000,000 bytes so the arithmetic stays clean. The query shape (join inventory to price, sum, compare) is the real one.

Sample data for this example

CREATE TABLE s3_inventory (
  object_key       TEXT,
  size_bytes       INTEGER,
  storage_class    TEXT,
  last_access_days INTEGER   -- days since the object was last read
);
INSERT INTO s3_inventory (object_key, size_bytes, storage_class, last_access_days) VALUES
  ('raw/events/2026-01-02/part-0.parquet',    300000000000, 'STANDARD',      1),
  ('raw/events/2025-08-01/part-0.parquet',    300000000000, 'STANDARD',    150),
  ('raw/logs/2025-11-15/app.log',             400000000000, 'STANDARD',    210),
  ('curated/daily_active/2026-01-01.parquet',  90000000000, 'STANDARD',      1),
  ('raw/logs/2025-10-01/app.log',             600000000000, 'STANDARD_IA', 260),
  ('archive/2024/full.tar',                  1000000000000, 'GLACIER_DEEP',800),
  ('exports/report-2026-01.csv',              50000000000, 'STANDARD',      5);
CREATE TABLE storage_pricing (
  storage_class   TEXT,
  usd_per_gb_month REAL   -- illustrative US price; 1 GB = 1,000,000,000 bytes here
);
INSERT INTO storage_pricing (storage_class, usd_per_gb_month) VALUES
  ('STANDARD',     0.023),
  ('STANDARD_IA',  0.0125),
  ('GLACIER_IR',   0.004),
  ('GLACIER_DEEP', 0.00099);

Worked example (SQL)

-- Each object's own monthly storage cost: size in GB times its class price.
SELECT i.object_key, i.storage_class,
       ROUND(i.size_bytes / 1000000000.0, 0) AS size_gb,
       ROUND(i.size_bytes * p.usd_per_gb_month / 1000000000.0, 2) AS monthly_cost_usd
FROM s3_inventory i
JOIN storage_pricing p ON p.storage_class = i.storage_class
ORDER BY monthly_cost_usd DESC;

Apply

Your turn

The task this lesson builds to.

Write a query that returns the total gigabytes and total monthly cost of each storage class present in the inventory, as (storage_class, total_gb, monthly_cost_usd), most expensive first, over s3_inventory joined to storage_pricing(storage_class, usd_per_gb_month).

Treat 1 GB as 1,000,000,000 bytes. Round total_gb and monthly_cost_usd to 2 decimals, and order by monthly_cost_usd descending.

3 hints and 1 automated check are waiting in the workspace.

Practice

Make it stick

A second problem on the same idea, plus 4 bonus drills.

Write a query that returns, for each cold STANDARD object, the monthly saving from moving it to Standard-IA, as (object_key, current_cost_usd, ia_cost_usd, monthly_saving_usd), largest saving first, over the same s3_inventory table.

A cold object is in the STANDARD class with last_access_days greater than 90. Standard costs 0.023 per GB-month and Standard-IA costs 0.0125 per GB-month (1 GB = 1,000,000,000 bytes). Round every dollar column to 2 decimals, ordered by monthly_saving_usd descending. This is the storage-price saving only: Standard-IA also adds a per-GB retrieval fee and a 30-day minimum, which erode it for small or frequently read objects.

3 hints and 1 automated check are waiting in the workspace.