skip to content

learn.smmeyer.devllm-engineering20

menu

llm-engineering/20

interview mode · studio · ~6h practice

AI-native design exercises

outcome

Practice full AI-native designs across storage, inference, agents, retrieval, memory, evals, routing, voice, and abuse.

Plan a practice session · ~6h including exercises
Read + trace the worked example
30 min
Assignment + constraint variations
210 min
Timed drill + repeated explanations
100 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

These exercises combine the entire course. Rotate which dimension you deep-dive so you can handle interviewer steering rather than replay one prepared speech.

coverage

  • State: ChatGPT conversation storage and persistent memory.
  • Serving: scalable inference API and model gateway.
  • Agents: coding, research, tool execution, long-running tasks.
  • Knowledge: enterprise RAG with permissions.
  • Quality: evaluation platform.
  • Realtime: voice assistant.
  • Security: abuse-resistant public API and tenant isolation.

mental model

Every AI design should cover state, intelligence, action, control, and learning.

  1. 1 →State

    Durable context

  2. 2 →Model

    Route + serve

  3. 3 →Action

    Tools + sandbox

  4. 4 →Learn

    Evals + feedback

Read in numbered order: state → model → action → learn.

lessons

01Vary the deep dive

For conversation storage, go deep on ordering/deletion. For inference, scheduling/KV/admission. For agents, state/checkpoints/tools. For RAG, ACLs/freshness/evaluation. For voice, interruption and latency.

Prepare a common packet structure but let the hardest path vary. Enterprise RAG needs document lifecycle and authorized evidence; inference needs token-shaped admission and memory; tool execution needs effect identity and reconciliation. Use the same headings—contract, estimates, APIs, ownership, path, failure, quality, cost—so differences become visible instead of hiding behind different diagram styles.

For each design, trace one success, one denial, and one uncertain outcome. Name durable state and the evidence of completion. A model-quality failure and a request-availability failure can occur independently. Quantify the dominant resource using explicit assumptions and explain which measurement would validate the capacity decision.

  • Reuse the execution framework
  • State model vs system failure separately
  • Always include cost and safety

02Add constraint cards

After 20 minutes, inject one change: a region fails, one tenant drives 40% of traffic, context doubles, a provider rate-limits, a tool has side effects, or retention law changes.

Inject a constraint after the first design is coherent. If a tenant loses access while a run is queued, revisit retrieval, execution, and publication checks. If output length doubles, revisit decode demand, active memory, admission, and spend. State the changed premise before proposing a mechanism so the revision can be assessed against the original invariant.

Preserve useful work. Do not erase the whole diagram when one provider fails. Identify what is durable, what output has already been shown, what effect may have committed, and what a safe fallback can still do. Explain migration or reconciliation when the system already contains state; a new happy-path sketch does not resolve an ambiguous old operation.

  • Revise the smallest boundary
  • Preserve the product invariant
  • Explain migration and operational impact

03Score evidence, not confidence

Use five dimensions from 1–4: framing, correctness, depth, tradeoffs, and communication/recovery. Keep one concrete note and one next drill per dimension.

Anchor a score in observable evidence. “Correctness = 2 because at minute 24 the retry path can duplicate a refund” is more useful than “felt unclear.” Choose one repair and rehearse that segment, then retry with a changed case. Good feedback identifies a mechanism, its consequence, and a next practice action.

Compare several design packets after practice. Check whether you quantified different dominant resources, chose different failure responses, and used verification suited to each task. If every conclusion is “add a larger model, cache, and queue,” return to the invariant. Improvement should transfer to an unfamiliar constraint rather than only make one recorded speech smoother.

  • Record at least half the mocks
  • Repeat weak designs after 48 hours
  • Seek adversarial follow-up questions

worked example

Defend the critical path of a tool executor

A public tool-execution service runs expensive actions for many tenants. Requests may be duplicated, permissions can change, and provider calls can time out after committing.

  1. Define submit, status, cancel, and result APIs with authenticated tenant ownership and durable operation identity.
  2. Admit work against tenant quotas, validate arguments, and bind approval to the exact action before execution.
  3. Persist state and reconcile ambiguous outcomes using provider idempotency or status support. Isolate execution environments and credentials.
  4. Evaluate unauthorized effects, duplicates, unresolved outcomes, latency, and cost per successful operation; test the region-loss path.

Design decision: Build around durable effect identity and authorization, then add model-facing proposals as a constrained input path.

Limit: Different actions have different cancellation and replay semantics. A generic tool interface must preserve those differences rather than promise universal exactly-once execution.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: Three design packets: an enterprise RAG assistant, an inference API, and a tool executor; include two recorded mocks.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. For each packet, define users, estimates, APIs, data ownership, hardest path, failure matrix, evals, cost, and rollout.
  2. Run two 50-minute mocks with different constraint cards and score each dimension with evidence.
  3. Compare the packets and redo the weakest path with a new failure; keep the revision and its rationale beside the original.

Change a constraint

  • The primary provider fails mid-stream.
  • A tenant loses permission while a run is queued.

Self-review rubric

  • Every design quantifies its dominant resource.
  • Trust and effect boundaries are explicit.
  • Quality, operations, and rollout gates belong to the full workflow.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Use these condensed packets as a completeness comparison. Estimates and capacity limits must be supplied and justified in your own artifacts.

RAG packet: ingest/query APIs; versioned sources/tombstones; ACL scope;
 hybrid retrieval → evidence → cited answer; recall/support/freshness gates
Inference packet: submit/stream/status; phase demand + active KV;
 tenant admission → scheduler → typed events; latency/fairness/cost gates
Executor packet: propose/approve/execute/status; exact approval + operation ID;
 durable attempt → effect → reconcile; authorization/duplicate-effect gates
All three: workload table, owner, hardest trace, failure matrix,
           accounting, release/rollback; two recorded changed-constraint mocks

A provider failure mid-stream changes publication and recovery; permission loss while queued changes eligibility. Revise the affected boundaries without discarding valid state. Score a mock with timestamped evidence for framing, correctness, depth, tradeoffs, and recovery, then repeat the weakest path under a different constraint.

Check your artifact

  • The packets quantify different dominant resources and use task-appropriate completion evidence.
  • All paths preserve trusted tenant and durable operation identity.
  • Two mocks retain diagrams, changed premises, scores with evidence, and a specific revised weak segment.

A tempting mistake: A polished architecture that omits reconciliation, evaluation, or cost has not yet covered the full workflow.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor capstone: complete packet and two recorded constraint-change reviews.

Rebuild Harbor from a blank page. Include task contract, demand, APIs/state, retrieval, scoped tools, security, quality, cost, and rollout. In one review fail the provider mid-stream; in the other revoke access while queued. Compare your results with the chapter-by-chapter project history.

Compare cumulative reasoning after your attempt

The packet must connect authorized evidence and controlled effects to durable runs, token-aware admission, typed events, evaluation gates, and truthful product states. Provider failure preserves prior effects and published output; revocation invalidates relevant evidence/action authority. Neither implies restarting everything. Use the existing fixtures to demonstrate these boundaries and retain one concrete next practice action from each review.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

A strong AI design closes with: eval gate, production SLOs, cost attribution, abuse boundary, and safe fallback.

failure patterns

  • Practicing only your favorite agent prompt
  • No capacity estimates
  • No distinction between quality and availability
  • Stopping once the happy-path diagram is complete

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Which closing topic is easy to miss but highly differentiating?

    select an option to check your answer

  2. 02Why inject constraints mid-mock?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Complete four mocks: inference API, enterprise RAG, coding agent, and realtime voice. Draw and narrate each.

  • 4 × 50 min
  • constraint injection
  • score 1–4
  • 48h redo
compare a strong approach after your attempt

Use the standard time boxes. At minute 20, randomly inject a failure/scale/security constraint. Finish with the top three risks and signals. Redo the lowest-scoring deep dive after 48 hours without reviewing the old diagram first.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally