Retries, SLAs, and the Alert You Page On

A retry that worked is not an incident

Every task carries a retry budget. Two fields hold it: try_number, the attempt that produced the row's final state, and max_tries, how many attempts the task was allowed. The canonical setting is about 2 (Research §5.2), and the number is deliberately small. Retries exist to absorb the transient stuff, a network blip, a briefly locked table, a rate limit. They do not exist to hide a bug. Ten retries turn a five-minute failure into an hour of the pipeline quietly re-running broken code before anyone hears about it.

One arithmetic warning before you write the SQL. Real Airflow stores max_tries as the number of RETRIES, not the number of attempts, so a task configured with retries=2 runs as try 1, try 2, try 3 and lands its terminal failure at try_number = max_tries + 1. This lesson's table uses max_tries as the total attempt budget instead, so the exhausted test is the plain try_number = max_tries and you can spend your attention on the reasoning rather than on an off-by-one. When you query a real metadata database, check which convention that version uses before you trust the comparison. In the seed here every task is allowed 2 attempts except load_facts, which is allowed 3 because it writes to a remote warehouse.

That gives you two very different events sharing one word:

• A retry save. state = 'success' and try_number > 1. Something hiccuped and the machine handled it. This is the system working. It belongs in a weekly report, not on a phone at 3am.
• An exhausted retry. state = 'failed' and try_number = max_tries. Every attempt was used and the task is still broken. This is the incident.

Not every red task spent its budget, which is why that second condition is not decoration. A task that raises a fail-fast error, Airflow's AirflowFailException, or that an operator marks failed by hand, stops with attempts still on the clock: state = 'failed' and try_number below max_tries. It is a real failure, but the retry policy is not the thing to argue about, because retrying was never going to help.

Alert fatigue is what happens when you treat them the same. Fire on every attempt and the person on call learns that the pipeline channel is noise, and then they miss the one page that mattered.

The SLA miss is a separate failure

A service level agreement on a task is a freshness promise: "load_facts finishes within 8 minutes". Missing it is its own kind of failure, and the thing that makes it worth teaching is that the task can succeed and still miss. A green run that took three times as long still means the dashboard was stale when the business opened. Alerting only on failures misses this entirely.

Table

Four signals from the same task_instances table. Two are noise-adjacent, two are incidents, and only one of the four is visible on a status dashboard.

signal	how you detect it	does it page	why
retry save	state = 'success' AND try_number > 1	no	the system already recovered; it belongs in a trend, not a page
exhausted retry	state = 'failed' AND try_number = max_tries	yes	every attempt was spent and the data did not land
SLA miss	duration_sec > sla_sec	yes	the data is late even though the run is green
silent success	state = 'success' AND rows_written far below normal	yes	nothing is red and the data is missing anyway

Silent success: green pipeline, empty dashboard

The most-missed production failure has no red anywhere (Research §5.7). Every task returns success, the run is green, and the table is empty or a tenth of its normal size. An upstream API returned an empty page, a filter got tightened, a partition path changed and the read matched nothing. Nothing raised, so nothing alerted.

The only defense is to check the work, not the status. Compare each task's rows_written against what that same task normally writes, and treat a collapse as a failure whatever the state column says. That is the bridge into Module 8.4: a test is a query that returns violating rows, and it passes when it returns none.

Common mistake: counting failures without splitting exhausted ones from rescued ones. That single number cannot tell you whether your platform is flaky and self-healing or actually broken, and it is the number most junior dashboards show.

Interview nuance: "how many retries would you configure and why" has a real answer: around two, because retries are for transient faults, and anything that survives two attempts needs a human rather than a third attempt. And "the DAG is green but the dashboard is empty" is the silent-success story. Answer it with rows written, compared against that task's own history, walked upstream to the first task whose volume broke.

On a real platform this differs. The detection queries are the same everywhere; the delivery is not. Real alert routing lives in the orchestrator's callbacks and a pager tool such as PagerDuty or Opsgenie, with severity, escalation policies, and on-call rotations attached. The alerts table here is the outbox those callbacks write to, and querying the outbox is how you prove to yourself that your alerting is signal rather than noise. One vendor detail is worth flagging because pre-2025 material gets it wrong: Airflow 3.0 removed task SLAs outright, the sla parameter, sla_miss_callback, and the sla_miss table with it, and Airflow 3.1 replaced them with Deadline Alerts (AIP-86), which fix the old ambiguity about when the clock starts by letting you anchor the deadline to the queue time or the run's start explicitly. Configure an sla= on Airflow 3 and nothing happens. The duration_sec > sla_sec reasoning below is unchanged; only the vendor mechanism moved.