Module 11.1: Data Quality for AI
Training-data curation as a funnel of graded SQL stages shaped after NVIDIA's NeMo Curator: exact-hash deduplication with the ROW_NUMBER keep-one every SQL interview asks for, a two-detector PII scrubbing funnel whose disposition comes from severity rather than volume, and per-stage acceptance-rate monitoring that catches the run where a classifier silently started eating the corpus while every job still reported success.
Deduplicating a Training Corpus with Hashes
Dedup a crawled corpus by exact content hash with ROW_NUMBER and a composite tiebreaker, materialize the result idempotently, and place fuzzy and semantic dedup as the same idea at higher cost.
PII Scrubbing Funnels Before Training
Drive a redact-or-drop decision from a two-detector PII scan table, find the entities one detector misses, and place benchmark decontamination as the stage that runs right after.
The Curation Funnel and Acceptance-Rate Monitoring
Read the seven-stage curation funnel as acceptance rates, then catch the run where one stage silently started eating the corpus while every job still reported success.
Module 11.2: Semantics, Text-to-SQL & the AI-Era Role
The semantic model as the text-to-SQL accuracy lever, graded by putting you in the loop: implement the governed metric, then catch and disprove a plausible-but-wrong AI query. Closes the level and the course with the commoditize-versus-appreciate framing.
Semantic Models: the Text-to-SQL Accuracy Lever
The governed metric contract is the single biggest accuracy lever on AI-written SQL, and reading one and implementing it exactly is the data engineer's half of the job.
Reviewing AI-Written SQL, and What Appreciates in the AI Era
A generated query can parse perfectly and still be wrong in three ways at once, so the closing skill of the course is reviewing one against its contract and proving the divergence in SQL.