The Curation Funnel and Acceptance-Rate Monitoring

Seven stages, and every one of them is allowed to delete your data

The two previous lessons each took one stage of a curation pipeline. Here is the whole thing, in the order NVIDIA's NeMo Curator runs it, which is the reference architecture worth being able to recite:

Order of evaluation

1. extractraw crawl to text, about 96% accepted
2. lang_idkeep the target languages, about 84%
3. heuristic_filterword count, boilerplate, repetition, about 80%
4. dedupexact then fuzzy, about 75%
5. quality_classifiera model scores each doc, about 61%
6. pii_redactionregex plus NER, about 99%
7. decontaminationremove eval-set overlap, about 99%

Read it as a funnel, because that is what it is. A million extracted documents become fewer than three hundred thousand training documents, and the losses are intentional at every step. Which raises the operational question this lesson exists for: if every stage is supposed to delete data, how do you notice when one of them starts deleting the wrong amount?

Acceptance rate is the health metric of a stage

Every stage has exactly one number that describes its behaviour, and it is not a runtime and not a row count:

acceptance_pct = 100.0 * docs_out / docs_in

A raw output count cannot tell you anything, because a stage that emitted fewer documents this week might simply have been handed fewer. The ratio is comparable across weeks of different sizes, which is exactly what makes it monitorable. Multiply by 100.0 rather than 100, because docs_out / docs_in in integer arithmetic is 0 for every stage that accepts less than everything.

Healthy acceptance rates are remarkably stable week over week, and that stability is the whole basis of the monitor. Dedup removes about a quarter of what it is given, every single week, because the web does not suddenly stop mirroring things. When dedup accepts 95 percent one week, the crawl did not get cleaner, your hashing broke.

The worst failure is a successful run

Level 10's observability module named the failure mode: silent success. A job that crashes gets a red status, a page, and somebody looking at it within minutes. A job that finishes cleanly while quietly destroying the corpus gets a green tick and nobody looks at it for a month, by which time you have trained on it.

The run log below has no status column at all, deliberately, because every one of these runs succeeded. The engine had nothing to complain about. Somebody edited the quality classifier's score threshold, fat-fingered a decimal, and the stage now rejects almost everything it sees. It reports success on every run, because rejecting documents is literally its job.

So the monitor cannot watch statuses. It has to watch the rates, and the comparison it makes is against that stage's own recent history, not against a global constant. Every stage has a different healthy rate, so a single threshold like "alert below 50 percent acceptance" would page you every week for the quality classifier at a perfectly normal 61 percent while missing a redaction stage that fell from 99 to 80.

LAG is the natural tool: partition by stage, order by run date, and each row can see what the same stage did last time.

LAG(acceptance_pct) OVER (PARTITION BY stage ORDER BY run_dt) AS prior_pct

Then the alert condition is a comparison between two columns of the same row, and the threshold is expressed in percentage points, not percent. A fall from 61 to 8.9 is a drop of 52.1 points. Calling it "an 85 percent decrease" is also true and much harder to reason about when you are setting the alerting threshold at three in the morning.

Common mistake: alerting on the change in docs_out. Output counts move for a completely innocent reason all the time, which is that the stage above moved. When the language filter has a bad week, every stage below it emits fewer documents and every one of them fires. The acceptance rate is immune to that, because it divides the stage's output by its own input, so a stage whose upstream shrank keeps reporting the same healthy ratio. That is exactly what happens on the middle run in the seed below.

Interview nuance: "how would you know your pipeline silently broke" is a stock reliability question, and the strong answer has three beats. Name the metric that is invariant to volume (a rate, not a count). Say what you compare it against (the same stage's own trailing history, not a global constant). Say what the threshold is expressed in (percentage points, with a floor on the input volume so a tiny run does not produce a noisy ratio). The pattern transfers directly: the same LAG-against-your-own-history shape is how you monitor conversion funnels, null rates, and job durations.

On a real platform this differs. This run log is 21 rows you can read at a glance. A real curation pipeline emits these counters per stage per shard into a metrics store or a run-log table in the warehouse, and the monitor is a scheduled query with the alert wired to a channel rather than a query you run by hand. Some teams go further and put a data test on the rate itself so a bad run blocks the publish instead of merely reporting itself, which is the write-audit-publish gate from SQL Level 5 pointed at a percentage. The shape of the query does not change.