skip to content

learn.smmeyer.devllm-engineering12

menu

llm-engineering/12

production ai · core · ~3h practice

Reliability & failure modes

outcome

Wrap probabilistic and upstream failures in deterministic recovery and safe degradation.

Plan a practice session · ~3h including exercises
Read + trace the worked example
25 min
Assignment + constraint variations
100 min
Timed drill + repeated explanations
35 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

AI applications add model variability, invalid outputs, context overflow, retrieval misses, tool errors, loops, and provider limits to ordinary distributed-system failure.

coverage

  • Model: hallucination, nondeterminism, instruction conflicts.
  • Interface: invalid schema, tool hallucination, context overflow.
  • Agent: loops, non-progress, duplicate side effects.
  • Knowledge: retrieval misses, stale/conflicting context.
  • Provider: rate limits, timeouts, outages.
  • Recovery: retries, fallback, routing, degradation, guardrails.

mental model

Classify the failure before choosing retry, repair, fallback, abstain, or human review.

  1. 1 →Detect

    Typed failure

  2. 2 →Contain

    Budget + cancel

  3. 3 →Recover

    Repair or route

  4. 4 →Record

    Case + metric

Read in numbered order: detect → contain → recover → record.

lessons

01Different failures need different responses

Transient transport errors may retry; invalid structured output may use constrained repair; a missing document needs retrieval fallback; unsafe or low-confidence high-impact output may require abstention or review.

Separate transport failure, invalid structure, unsupported content, denied action, and ambiguous external effect. Retrying a rate-limited read after bounded backoff can help; retrying a denied action should not bypass policy; retrying a timed-out booking can create a duplicate. Recovery starts by identifying which boundary failed and whether state may already have changed.

Use a stable operation record with states such as proposed, authorized, executing, succeeded, failed, and unresolved. A timeout after an external commit belongs in unresolved until reconciliation establishes the result. A generated message is not the source of truth for that state. Return a run ID and truthful status so a client does not accidentally submit a new operation while waiting.

  • Do not retry deterministic bad inputs
  • Preserve one operation ID across retries
  • Bound total attempts inside deadline/cost

02Fallback is a product decision

A smaller or different model can change quality, tool support, context, or safety behavior. Validate compatibility and disclose material degradation. Sometimes a queued or explicit failure is safer than silent fallback.

A fallback must satisfy the same essential contract. A text-only provider cannot transparently replace one required to return a particular tool mode. A cheaper model may pass common tasks but fail a critical safety or language slice. Record required capabilities and a minimum evaluated quality bar, then choose whether to reroute, delay, degrade, or fail clearly.

Mid-stream fallback is particularly difficult: the user may already have seen part of an answer, and tools may have executed. Do not append a second provider's incompatible continuation as if nothing changed. Define restart or partial-result behavior, preserve effect identity, and distinguish already published content from a new attempt. Reserve fallback capacity so a provider incident does not overwhelm the alternate path.

  • Circuit-break unhealthy routes
  • Hedge only when cost and duplicate effects are controlled
  • Test fallback paths continuously

03Guardrails surround, not replace, the model

Validate schemas, permissions, tool arguments, token budgets, citation requirements, and state transitions in code. Track unknown outcomes after timeouts and reconcile durable effects.

Validation and guardrails belong at multiple boundaries. Validate request size before expensive work, retrieved scope before context construction, proposed arguments before execution, and final output before publication where feasible. A check late in the path cannot undo a leaked secret or a committed side effect. Use controls suited to the consequence at that stage.

Bound repair loops. If a malformed response is reparsed or regenerated, retain the original input and failure evidence and limit attempts. Repairing JSON must not re-execute an already committed action. Monitor recovery success and cost separately from first-pass success; otherwise repeated hidden attempts can make a feature look reliable while creating unacceptable latency or spend.

  • Cancellation must propagate to model and tools
  • Use resumable checkpoints for long tasks
  • Turn every incident into an eval case

worked example

A retry repairs JSON but repeats an external action

An assistant creates a calendar event successfully, then returns malformed final JSON. A caller retries the whole request.

  1. Separate the durable action result from final answer formatting. A parser failure must not erase the fact that the event was already created.
  2. Resume from the recorded event ID and repair only the final representation when possible.
  3. Use deterministic checks for required fields and business invariants; bound repair attempts within the original run budget.
  4. Record a failure category and reproduce it in an eval that checks both output validity and effect count.

Design decision: Recover at the smallest safe boundary and preserve durable evidence of committed effects.

Limit: A fallback model can change behavior and may not support the same tools or output contract. It must earn its role through evaluation.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: A failure taxonomy and recovery table for an event-booking assistant.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Cover malformed output, unsupported claim, missing evidence, tool rejection, partial stream, and provider timeout.
  2. For each failure, choose retry, repair, fallback, abstain, or escalation and identify the durable state needed.
  3. Trace a duplicate request and a formatting repair; write regression cases that assert effect count and user-visible status.

Change a constraint

  • A fallback provider lacks the requested tool contract.
  • The stream ends after showing an event ID but before a success marker.

Self-review rubric

  • Recovery does not blindly replay completed effects.
  • Retries and repairs are bounded.
  • The UI distinguishes failure, ambiguity, and successful completion.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

An event-booking effect succeeded before the final response became malformed. The application stores operation identity and provider status.

Failure                         recovery
invalid request                 deterministic reject; no model/tool effect
provider read timeout           bounded retry within deadline
malformed final JSON            repair representation using stored receipt
unauthorized booking            deny; do not try a broader tool
booking timeout-after-commit    unresolved → reconcile same operation
missing terminal stream event  inspect durable run status
fallback lacks required tool   delay/degrade/fail explicitly
Gate recovery: no repeated effect; bounded retries; truthful user state

The business operation and its presentation are separate. Reusing the receipt can repair the response without creating another calendar event. A final event ID seen in a partial stream is not necessarily a terminal success marker; consult durable status. A fallback needs equivalent essential capability and evaluated behavior.

Check your artifact

  • Repeated client requests preserve operation identity.
  • Each error category has a distinct action and retry budget.
  • Recovery metrics include latency, spend, and unresolved outcomes, not only eventual valid JSON.

A tempting mistake: Retrying the entire agent run to repair syntax can repeat an external action that already succeeded.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's recovery table and regression cases for uncertain effects.

Reuse the action executor and run state. A refund succeeded, the final JSON failed, and the client disconnected. Recover the effect identity from memory, choose the response-repair path, and add a gate proving this path does not duplicate the effect.

Compare cumulative reasoning after your attempt

Read the persisted receipt or reconcile the stable operation ID, then repair only the presentation. Reconnection queries the same run. If status is still uncertain, report unresolved rather than a fabricated success or failure. Include a crash/retry fixture whose counted external effects remain one under the stated provider contract and whose recovery stays within budget.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Produce a failure matrix: failure → detection → user impact → automatic action → audit/metric. It shows operational maturity fast.

failure patterns

  • Retrying every failure
  • Silent downgrade to an incompatible model
  • Assuming schema validity equals semantic validity
  • No reconciliation for ambiguous tool timeouts

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01When should a model outage silently fall back?

    select an option to check your answer

  2. 02Best response to an ambiguous timed-out write?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Design the failure policy for a customer-support agent that reads accounts and can issue credits.

  • money effect
  • rate limits
  • model outage
  • partial streams
compare a strong approach after your attempt

Separate read/generate/write stages. Retry safe reads, route generation through a tested fallback, and never repeat credit issuance without the same idempotency key. Require approval or strict policy for credits, preserve an audit record, and tell the user when outcome is unknown. Reconcile later by operation ID.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally