skip to content

learn.smmeyer.devllm-engineering14

menu

llm-engineering/14

product systems · core · ~3h practice

Building LLM products

outcome

Choose models and UX patterns across intelligence, latency, cost, streaming, feedback, and failure.

Plan a practice session · ~3h including exercises
Read + trace the worked example
25 min
Assignment + constraint variations
100 min
Timed drill + repeated explanations
35 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

An AI feature succeeds as a product, not a demo. Users experience waiting, uncertainty, corrections, citations, and partial failure—not benchmark scores in isolation.

coverage

  • Selection: model intelligence, latency, cost, modality, tool support.
  • Routing: task/risk-based routes and fallbacks.
  • UX: streaming, conversation state, citations, explainability.
  • Jobs: async/background work and progress.
  • Control: guardrails, rate limits, cost budgets.
  • Learning: feedback, observability, product metrics.

mental model

Match model effort and user experience to task value, risk, and time horizon.

  1. 1 →Classify

    Task + risk

  2. 2 →Route

    Model + tools

  3. 3 →Experience

    Stream or async

  4. 4 →Learn

    Feedback + eval

Read in numbered order: classify → route → experience → learn.

lessons

01Choose on a measured frontier

Compare candidates on your task dataset across quality, latency, token cost, tool correctness, context, and operational limits. Route simple/low-risk work cheaply and escalate hard or high-value cases.

Compare model configurations on the actual user task. For code review, a useful result may be an accepted actionable finding, not a grammatically polished paragraph. Measure missed important defects, false alarms, time to useful output, and spend across representative languages and repository sizes. Include a deterministic baseline so you know what value the probabilistic component adds.

Cost per success changes the ranking. If configuration A costs 0.02 illustrative units per attempt and succeeds 50% of the time, its cost per successful attempt is 0.04. Configuration B at 0.03 and 90% success yields about 0.033. Include retries, tools, and human correction in the real comparison. A cheaper call can produce a more expensive workflow.

  • Model families evolve; isolate provider-specific details
  • Cache and shorten context before indiscriminate downgrades
  • Budget cost per successful outcome

02Design uncertainty into UX

Stream when partial text is useful; use progress/checkpoints for long jobs; expose citations and editable outputs where verification matters. Provide cancel, retry, and recovery without duplicate effects.

Show uncertainty through useful product states. Distinguish processing, partial findings, unsupported answer, denied action, failed run, and completed result. A streaming interface can reduce waiting anxiety but should not imply the first plausible sentence is verified. Give users a clear way to inspect evidence, correct inputs, cancel work, or escalate a consequential decision.

Cancellation has semantics. It may stop future model generation while a provider action is already in flight and cannot be undone. Tell users what was cancelled and what still needs reconciliation. Persist enough state for reconnection so refreshing a page does not lose the result or create a duplicate run. Product behavior and backend state must describe the same reality.

  • Do not stream irreversible decisions
  • Preserve partial artifacts
  • Make fallbacks and limitations legible

03Feedback needs interpretation

Thumbs signals are sparse and biased. Pair them with edits, task completion, abandon rates, escalation, latency, and qualitative review. Turn recurrent failures into eval cases.

A rejected suggestion can mean the model was wrong, the user disagreed with policy, the result arrived too late, or the feature was irrelevant. Collect enough context to distinguish these causes while minimizing sensitive data. Accepted edits are also imperfect labels: users may accept a plausible mistake or modify text for stylistic reasons.

Close the loop with diagnosed cases. Sample accepted and rejected outcomes, review a subset, add representative failures to development evaluations, and compare the next version with the previous one. Measure abandonment and time saved alongside model scores. A feature succeeds when it improves a workflow under realistic cost and reliability constraints, not merely when the model appears impressive in a demo.

  • Optimize user outcome, not answer length
  • Watch quality by segment
  • Include safety and cost in launch gates

worked example

Compare cost per useful result

In a fictional test, model A costs $0.01 per attempt and succeeds on 60% of tasks. B costs $0.02 and succeeds on 90%. Assume one attempt per task.

  1. Across 100 tasks, A costs $1 for 60 successes: about $0.0167 per success. B costs $2 for 90: about $0.0222 per success.
  2. Add latency, escalation cost, error severity, and user acceptance. Token price alone leaves those product outcomes out.
  3. If failures trigger retries or human review, measure the complete workflow cost rather than simply dividing the first-call price.
  4. Expose progress and recoverable errors in the UI, and avoid streaming claims that later checks routinely retract.

Design decision: Choose an operating point using task success, user experience, risk, latency, and total cost together.

Limit: The prices and success rates are fabricated for arithmetic. A task with a costly wrong answer can reverse the apparent value comparison.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: A model-selection scorecard and UX flow for an AI code-review feature.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Define accepted useful findings, noise, misses, severity, response time, and cost per accepted finding.
  2. Compare three fictional configurations, including a deterministic baseline and a staged fast/deep workflow.
  3. Sketch loading, cancellation, partial results, correction, and escalation; review the flow using the eval cases.

Change a constraint

  • Users abandon after three seconds.
  • One configuration is cheaper but routinely produces unsupported security claims.

Self-review rubric

  • Cost uses a meaningful success denominator.
  • Failure and cancellation are visible and recoverable.
  • The choice is supported by task-level evidence.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Use the worked example's one-attempt costs: A=$0.01 at 60% success, B=$0.02 at 90%. Suppose the product requires at least 85% success on this fictional task set.

Configuration      cost/attempt   success   cost/success
A                  $0.01          60%       ~$0.0167
B                  $0.02          90%       ~$0.0222
Rule-only baseline measured first; staged fast/deep workflow evaluated too
Choice under 85% bar: B; A is cheaper/success but fails quality requirement
UX: submitted → findings streaming (partial) → verified complete
    ↘ cancel/status ↘ unsupported/escalate ↘ failure/retry same run
Code-review metrics: useful accepted findings, false alarms, misses,
                    time to useful finding, total workflow spend

The example does not show B winning on every dimension: it clears the required quality threshold at higher cost per success. Include tools, retries, and human review in the real denominator. Under a three-second abandonment threshold, compare early useful output and a staged design rather than hiding minutes of work behind a spinner.

Check your artifact

  • The success definition measures useful task completion, not fluent text.
  • Unsupported security claims fail a critical slice even if the configuration is cheap.
  • Cancellation and partial results reflect actual backend state.

A tempting mistake: Choosing the lowest per-call price can ignore failure cost; choosing the highest score can ignore product latency and spend constraints.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's model/UX decision record tied to the evaluation ledger.

Recall average phase demand and critical release gates. Compare a fast configuration that gives more unsupported policy claims with a slower supported one. Define the quality floor, useful-latency measure, and cost denominator before choosing, then sketch partial/cancelled/unresolved states.

Compare cumulative reasoning after your attempt

Reject configurations below the evidence/security floor before trading latency against spend. Measure time to useful supported output and total cost per resolved ticket, including retries and human correction. A staged response can acknowledge progress without publishing an unverified policy claim. Show durable run status and uncertain effects accurately so UI responsiveness does not imply false completion.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Frame model choice as an empirical product decision: task distribution × acceptable failure × latency/cost budget.

failure patterns

  • Choosing the biggest model by default
  • Streaming without cancellation semantics
  • Treating thumbs-up as ground truth
  • Hiding fallback-induced capability changes

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Best model-selection metric?

    select an option to check your answer

  2. 02Useful feedback signal beyond thumbs?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Design the product experience for an AI code review assistant on pull requests.

  • developer trust
  • 10-min deep review
  • false positives
  • cost budget
compare a strong approach after your attempt

Run cheap deterministic checks immediately, stream only stable findings, and make deep semantic review an async job with progress. Cite exact code and rationale, let developers resolve/feedback, route complex diffs to stronger models, and evaluate accepted findings, misses, noise, latency, and cost per useful finding.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally