LLM Evaluation & Guardrails

Eval and guardrails are first-class production components

LLMs are non-deterministic and sensitive: a one-word prompt tweak or a model version bump can silently break outputs that worked yesterday. So eval and guardrails are not QA afterthoughts, they are first-class production components, the CI/CD and the WAF of an LLM feature. The rule is: no prompt or model change ships to users without passing an eval gate, and no user input or model output flows unfiltered.

Offline eval (the pre-ship gate)

You maintain golden datasets: representative inputs paired with expected outputs or with scoring criteria. On every prompt or model change you run the candidate against the golden set in CI and compare scores to the current production version. Scoring methods, in order of reliability: exact/programmatic checks (does the JSON parse, does the SQL run, does the answer contain the required id) are cheapest and most trustworthy; similarity metrics for freer text; and LLM-as-judge, where a strong model grades outputs against a rubric. LLM-as-judge scales but has real biases (it favors longer answers, its own style, and the first option in a pair), so you calibrate it against human labels, use it for relative comparison more than absolute scores, and never let it grade safety-critical outputs alone. A regression suite of past failures runs every time so fixed bugs stay fixed.

Online eval (post-ship)

Offline sets never cover real traffic, so you also evaluate in production. Canary a new prompt/model to 1 to 5 percent of traffic and watch live quality and guardrail metrics before ramping. A/B test prompt variants on business and quality metrics. Capture implicit signals (thumbs up/down, retries, edits, escalations) and explicit feedback. This is the loop that catches the drift offline eval missed.

Guardrails (the runtime filters)

input  -> [PII redaction] [prompt-injection / jailbreak detection] -> model
model  -> [schema validation] [toxicity / moderation] [PII scan] [groundedness check] -> user
         (fail -> block, redact, or safe fallback; never ship the raw bad output)

Input guardrails redact PII before it hits a third-party model and detect prompt-injection and jailbreak attempts. Output guardrails validate structure (the response must be valid JSON matching a schema, else reject and retry), run moderation for toxicity, scan for leaked PII, and for RAG verify groundedness. On failure you block, redact, or return a safe fallback, never the raw bad output. For RAG, score groundedness (is each claim supported by the retrieved context) and verify every citation resolves to a real retrieved chunk. An unsupported claim or a fabricated citation fails the guardrail.

Production failures and human labels feed back into the golden and regression sets, so eval coverage grows toward real usage over time. This human-in-the-loop labeling is what keeps eval from going stale.

Interview nuance: when asked "how do you know it works," a weak answer is "we tried some prompts." The strong answer is a golden set scored in CI, a canary with live metrics, runtime guardrails, and a feedback loop that grows the eval set.

Recap: gate every change with offline golden-set eval (programmatic checks, calibrated LLM-as-judge, regression suite) plus online canary/A-B, enforce input and output guardrails at runtime (PII, injection, schema, moderation, groundedness), and close the loop by feeding production failures back into the eval sets.