skip to content

learn.smmeyer.devllm-engineering15

menu

llm-engineering/15

product systems · patterns · ~3h practice

LLM application architectures

outcome

Select the smallest architecture pattern that fits the task and autonomy level.

Plan a practice session · ~3h including exercises
Read + trace the worked example
25 min
Assignment + constraint variations
100 min
Timed drill + repeated explanations
35 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

Most LLM products fit a small set of shapes. Recognizing them keeps designs simple and helps you identify where determinism, retrieval, tools, and approval belong.

coverage

  • Direct: simple chat, classification, structured extraction.
  • Grounded: search + RAG and citation workflows.
  • Assistive: coding, support, research, and data-analysis assistants.
  • Interactive: browser/computer-use agents.
  • Workflow: automation and multi-step business processes.
  • Autonomy: deterministic, approval-gated, bounded autonomous.

mental model

Add complexity only when the task needs new information, actions, or adaptive planning.

  1. 1 →Generate

    One response

  2. 2 →Retrieve

    External knowledge

  3. 3 →Tool

    Deterministic action

  4. 4 →Agent

    Adaptive loop

Read in numbered order: generate → retrieve → tool → agent.

lessons

01Start at the left

Classification or extraction often needs one constrained model call. Add retrieval when knowledge is external/fresh, tools when deterministic capability or action is needed, and an agent loop only when the path adapts to observations.

Start with a deterministic baseline and the smallest useful model step. A classification task may need one constrained call followed by validation. Add retrieval only when external evidence is needed, tools when exact data or effects are needed, and adaptive control flow only when the sequence cannot be known in advance. Each added stage introduces latency, failure, state, and evaluation requirements.

Compare synchronous and durable asynchronous execution. A fast bounded extraction can return directly. A long report involving several providers may need submit/status/result APIs and durable work state. Disconnecting a client should not lose accepted work, while replaying a submit should not create another run. These requirements justify job infrastructure more concretely than simply calling the application an agent.

  • Batch offline work when latency is loose
  • Use schemas at machine interfaces
  • Keep final effects deterministic and authorized

02Architecture follows verification

A research system needs citation coverage; a coding assistant needs tests/diffs; support needs account and policy checks; data analysis needs executable computation and artifact inspection.

Choose completion evidence before architecture. For a report, preserve cited evidence and the generated artifact. For a code change, preserve the patch and relevant verification. For a booking, preserve the external receipt and operation identity. A verifier may be deterministic code, source inspection, a human decision, or several signals. Design where that evidence is produced and persisted.

Draw the run state machine with recovery transitions. A worker crash before a read can retry; a crash around an external write may require reconciliation. A failed verification should not automatically rerun an irreversible action. Separating proposal, execution, verification, and publication makes these differences visible and reduces accidental coupling to the conversation format.

  • Design evidence of completion
  • Capture intermediate artifacts
  • Choose approval around consequence, not buzzwords

03Workflow engines and agents can coexist

An explicit workflow can call a model inside known steps and hand a bounded subproblem to an agent. This hybrid often gives flexibility without surrendering control.

An outer workflow can handle authentication, durable state, deadlines, retries, and approval while a bounded agent investigates a subproblem. Give the subproblem a scoped goal, allowed tools, budget, and expected artifact. The workflow consumes that artifact through validation rather than treating an arbitrary agent message as authority to advance every state.

Version state and contracts so long-running work can finish across deployments. On resume, check whether the original inputs, permissions, and tool capabilities are still valid. If migration is unsafe, pause or terminate clearly. A workflow engine can make scheduling durable, but business recovery, idempotency, and cancellation rules still belong to the application design.

  • State transitions stay durable
  • Agent receives scoped goal and tools
  • Workflow owns retries, deadlines, and compensation

worked example

Move a slow review into a durable job

A repository review takes several minutes and uses tools. The user closes the browser after 20 seconds.

  1. Create a durable run with an owner, input snapshot, status, and idempotency key before starting expensive work.
  2. Return a run ID quickly. Stream or poll status through a read path that can reconnect without creating a second run.
  3. Checkpoint tool results and effect identities so a worker restart resumes safely.
  4. Define cancel, retry, expiry, and artifact access. Enforce authorization when fetching both status and results.

Design decision: Separate the request lifecycle from the durable work lifecycle when tasks outlive a connection.

Limit: A queue alone does not supply resume correctness, cancellation, or exactly-once external effects. Those are application contracts.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: A run API, state machine, data model, and three recovery timelines.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Define create, inspect, cancel, resume, and result-fetch APIs with authorization and idempotency.
  2. Choose durable versus ephemeral state and show checkpoint boundaries.
  3. Trace disconnect, worker crash, and duplicate delivery; compare a synchronous version and justify the extra job infrastructure.

Change a constraint

  • A tool action commits immediately before a worker crash.
  • The user cancels while a non-cancellable provider call is in flight.

Self-review rubric

  • Reconnection does not create another run.
  • Resume semantics preserve effects and input identity.
  • Cancellation has an explicit best-effort or guaranteed boundary.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

A repository review takes minutes and must survive browser disconnects. Some provider calls cannot be cancelled after acceptance.

POST /runs + idempotency key → runId
GET /runs/:id → state + event cursor; GET /runs/:id/result → artifact
POST /runs/:id/cancel; POST /runs/:id/resume (only allowed states)
Run: accepted → running → awaiting approval → completed/failed/cancelled
     executing uncertain effect → unresolved → reconcile
Durable: caller/tenant, input hash, versions, steps, attempts, receipts
Disconnect → work/status survives; crash → checkpoint + reconcile;
duplicate submit → same run; cancellation → stop future work,
record in-flight non-cancellable effect until resolved.

The client can reconnect with its event cursor or fetch a durable result. A job-based design is justified by duration, restart, and status requirements, but adds storage and worker operations compared with a short synchronous call. Resume checks inputs and current authority; it does not bypass denial or replay completed effects.

Check your artifact

  • A reconnect does not create another run.
  • A crash after effect commit has a reconciliation path.
  • Cancelled means a defined boundary, not a claim that every in-flight external action was undone.

A tempting mistake: A durable queue can restart a handler, but business state and effect semantics are still your responsibility.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's durable run API and integrated state machine.

Promote the earlier workflow sketch into APIs and persisted records. Reconstruct retry, disconnect, and cancellation behavior without notes. Justify which support requests remain synchronous and which need durable jobs, using the quality and latency contract you already chose.

Compare cumulative reasoning after your attempt

Fast bounded triage can return directly; multi-step investigation or approved actions benefit from stable run identity, status, event replay, and stored receipts. A disconnect does not revoke every effect automatically. Checkpoint safe steps, reconcile uncertain ones, and apply current authorization on resume. Extra job infrastructure is warranted by those recovery requirements rather than by the presence of an LLM alone.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

For any AI prompt, say which of four levels applies: generate, retrieve, tool, or agent—and why the simpler previous level is insufficient.

failure patterns

  • Calling every multi-step workflow an agent
  • Using RAG when the missing need is a calculator/API
  • Giving autonomous writes where draft-and-approve works
  • No evidence-based completion check

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01When does RAG help most?

    select an option to check your answer

  2. 02A fixed multi-step business process should default to…

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Classify and sketch architectures for invoice extraction, policy Q&A, refund support, and web research.

  • 4 systems
  • 5 min each
  • smallest pattern
  • verification
compare a strong approach after your attempt

Invoice: schema extraction + validation. Policy Q&A: ACL-aware RAG + citations. Refund: workflow + read tools + gated idempotent write. Research: bounded agent + web tools + claim/source state + citation verification.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally