skip to content

learn.smmeyer.devllm-engineering11

menu

llm-engineering/11

production ai · core · ~4h practice

Evaluation

outcome

Build an eval-driven development loop across datasets, graders, human calibration, and online outcomes.

Plan a practice session · ~4h including exercises
Read + trace the worked example
30 min
Assignment + constraint variations
140 min
Timed drill + repeated explanations
50 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

Traditional unit tests still protect deterministic code; evals measure variable behavior and distributions. The key is a task-specific definition of success that stays connected to real user outcomes.

coverage

  • Datasets: golden cases, production samples, adversarial and slice coverage.
  • Judgment: human evaluation, model-as-judge, pairwise comparison.
  • Metrics: factuality, relevance, instruction following, safety, tool correctness.
  • Regression: prompt/model/tool changes and release gates.
  • Online: A/B tests, feedback, business outcomes.
  • Integrity: leakage, grader bias, calibration, drift.

mental model

Production failures become cases; cases guide changes; changes earn rollout through gates.

  1. 1 →Define

    Task success

  2. 2 →Collect

    Representative cases

  3. 3 →Grade

    Calibrated signals

  4. 4 →Ship

    Gate + monitor

Read in numbered order: define → collect → grade → ship.

lessons

01Start from failure taxonomy

Define what can go wrong—wrong answer, unsupported claim, missed tool, unsafe action, bad refusal—and build cases for each. Include easy, hard, boundary, adversarial, and real production examples.

Define success at the task level before choosing a grader. A support answer might need correct policy, cited evidence, complete resolution, and no unauthorized disclosure. These are distinct properties; one aggregate “quality” rating can hide a serious failure. Build ordinary, boundary, ambiguous, adversarial, and missing-evidence cases, with labels that another reviewer can apply consistently.

Separate development cases from held-out evaluation. Keep input, expected evidence or outcome, slice labels, and versioned grading rules together. Small hand-built datasets are useful for diagnosing mechanisms, but a few successes do not establish a production failure rate. For example, five percentage points on twenty cases is one changed case. Inspect uncertainty and critical failures before treating that as a robust improvement.

  • Keep development and holdout sets separate
  • Version data and grading rubrics
  • Stratify results by meaningful slice

02Graders need evaluation too

Exact checks fit schemas and known answers; code graders fit executable properties; model judges fit nuanced comparisons. Calibrate automated graders against blinded human labels and inspect disagreement.

Choose the simplest reliable grader for each property. Parse code for schema validity, execute a calculation for exact results, inspect source support for factual claims, and use human or calibrated model review for nuanced quality. A model judge is another fallible component. It may prefer verbosity, familiar phrasing, or answers from a particular style.

Calibrate on blinded examples with human labels, then inspect disagreements. For a binary safety gate, a false pass can matter much more than a false fail; report those separately. Pairwise review can simplify some comparisons but still needs randomized presentation and controls for irrelevant style. Preserve grader versions so a changing evaluator does not masquerade as a changing model.

  • Pairwise is often easier than absolute scoring
  • Avoid judging style when correctness matters
  • Use multiple signals for high-stakes gates

03Close the online loop

Offline improvement may not improve user outcomes. Canary and A/B test meaningful changes, observe latency/cost/safety, and mine failures back into the suite without contaminating the holdout.

Offline evaluation makes changes comparable; online observation checks whether those changes help real users. A canary should monitor task completion, latency, spend, and critical failure slices with a rollback rule. Thumbs-up is useful but biased: some users never respond, and dissatisfaction can reflect product policy rather than model error. Treat feedback as evidence to investigate rather than an automatic ground-truth label.

Turn production failures into reproducible cases by recording safe input representations, model and prompt versions, evidence IDs, tool observations, and policy decisions. Add diagnosed cases to development data while protecting a fresh holdout. Keep a release ledger that states what improved, what regressed, and which tradeoffs were accepted; a single rising score is not a complete release argument.

  • Log model, prompt, retrieval, and tool versions
  • Watch Goodhart effects
  • Re-evaluate when traffic shifts

worked example

An aggregate win hides a critical-slice regression

A test set has 900 common and 100 safety-critical cases. Variant A passes 90% of common cases and 95% of critical cases; B passes 95% and 60% respectively.

  1. A passes 810 + 95 = 905/1,000, or 90.5%. B passes 855 + 60 = 915/1,000, or 91.5%.
  2. The aggregate rises by one percentage point while critical failures rise from 5 to 40. A single overall threshold would miss the regression.
  3. Define slice-level gates and severity-sensitive review. Include latency and cost in the comparison without allowing them to erase a critical correctness failure.
  4. Calibrate automated graders against human-labeled cases, inspect disagreement, and preserve a held-out set when tuning prompts or thresholds.

Design decision: Ship against a multidimensional, slice-aware contract rather than the largest aggregate score.

Limit: These counts demonstrate aggregation risk. Real decisions also need uncertainty estimates, representative sampling, and clear labeling rules.

Teaching scenario · traffic, prices, and capacity values are illustrative.

lab

interactive lab

Eval Workbench — aggregates lie, slices tell

1 · predict first

Variant B fixes 4 cases and breaks 2. Does the launch gate pass? Decide what rule you would enforce, then check.

A guess makes the comparison useful. This lab draft lasts until you leave the page.

pedagogical model — values are illustrative, not measured.

assignment

Produce: A twenty-case eval table, grading rubric, disagreement review, and release gate.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Define task success and failure severity before inventing candidate outputs.
  2. Use the lab to expose an aggregate/slice reversal and adjust judge thresholds; inspect false passes and false fails.
  3. Draft human labels and an automated rubric, reconcile disagreements, and evaluate two fictional variants including latency and cost.

Change a constraint

  • A judge prefers a longer answer that contains an unsupported claim.
  • An apparent five-point gain comes from only twenty cases.

Self-review rubric

  • The dataset covers meaningful failure slices.
  • Grader agreement is checked against independent labels.
  • A release can be blocked by a critical slice despite a higher aggregate.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

The following twenty-case table is a development exercise. It is too small to establish a production safety rate; add representative and held-out cases before release.

ID/slice → expected property
01 ordinary→correct category; 02 ordinary→supported answer;
03 exact code→right source; 04 paraphrase→gold evidence;
05 long→retain condition; 06 multilingual→correct meaning;
07 ambiguous→clarify; 08 missing→abstain;
09 obsolete→current version; 10 contradiction→resolve or disclose;
11 denied→no restricted context; 12 revoked→no publication;
13 injection→no new authority; 14 secret request→deny;
15 tool duplicate→one effect; 16 timeout→reconcile;
17 malformed→structure repair only; 18 cancellation→truthful state;
19 compaction→preserve constraint; 20 provider failure→safe terminal state.
Record per row: A/B outputs, code checks, human label, judge label, reason.
Disagreement example, row 13: B proposes an unauthorized email;
human=fail, judge=pass for polite wording → fix effect-aware grading.
Gate: no known critical regression; quality + latency/cost bars; inspect disagreement.

Use pass/fail definitions with explicit evidence per property and review disagreement before trusting a judge. In the larger worked example, A passes 905/1,000 and B 915/1,000, yet critical success drops from 95% to 60%. An aggregate improvement can therefore fail the release gate. A five-point change on twenty cases is one case, so inspect it rather than claiming robust significance.

Check your artifact

  • Development and holdout cases have separate roles.
  • Grader calibration includes false passes on critical cases, not just agreement averages.
  • The release decision preserves critical slices and records tradeoffs in cost and latency.

A tempting mistake: A judge that prefers a longer unsupported answer is rewarding the wrong property. Revise or replace the grader before optimizing against it.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's cumulative evaluation ledger and release gate.

Gather earlier fixtures into a versioned suite. Recall the split policy, evidence-support checks, and compaction failure. Compare a variant improving 18/20 to 19/20 while newly leaking one restricted policy. State the release decision and what the score can establish.

Compare cumulative reasoning after your attempt

Do not release the known authorization regression because aggregate accuracy rose. Inspect the leaking path, keep the case, and gate critical errors separately from common quality. Twenty cases provide a diagnostic signal rather than a precise deployment estimate. Version outputs and graders, review human/judge disagreement, and retain independent representative holdout data. The same ledger must track latency and total workflow cost.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Say: “I would not ship from a single aggregate score. I need the regression slices, cost/latency delta, and human-calibrated grader agreement.”

failure patterns

  • Vibe-based testing
  • A tiny curated set with no production distribution
  • Uncalibrated model-as-judge as sole truth
  • Optimizing a leaked or repeatedly tuned holdout

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Best first eval question?

    select an option to check your answer

  2. 02Why calibrate model judges?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Create an eval plan for an AI marketing-copy builder used by internal teams.

  • brand voice
  • factual claims
  • structured fields
  • human feedback
compare a strong approach after your attempt

Define schema validity, brief adherence, unsupported claims, brand/risk rules, and edit-distance-to-accepted outcome. Sample real briefs across campaigns, add adversarial content, calibrate pairwise judges with marketers, and gate regressions by slice plus latency/cost. Feed accepted edits back as new cases—not automatic labels.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally