skip to content

learn.smmeyer.devsystem-design12

menu

system-design/12

ai systems · deep · ~5h practice

AI / LLM–flavored system design

outcome

Design an end-to-end AI service across streaming, GPUs, retrieval, tools, safety, cost, and evals.

Plan a practice session · ~5h including exercises
Read + trace the worked example
30 min
Assignment + constraint variations
190 min
Timed drill + repeated explanations
60 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

This is where both tracks meet. The model is one dependency inside a system that must schedule scarce work, stream partial outcomes, run untrusted tools, retrieve authorized context, and improve measurably.

coverage

  • Inference path: streaming, prefill/decode, batching, GPU utilization, limits.
  • Routing: model selection, fallback, admission, tenant quotas.
  • Context: prompt budgets, embeddings, vector search, retrieval.
  • Agents: long tasks, tool execution, sandboxing, checkpoints.
  • Control: caching, cost budgets, safety/moderation.
  • Learning: evaluation, telemetry, feedback, regression gates.

mental model

An AI request is a controlled loop, not a single model call.

  1. 1 →Ground

    Policy + context

  2. 2 →Generate

    Route + stream

  3. 3 →Act

    Tool sandbox

  4. 4 →Evaluate

    Outcome + trace

Read in numbered order: ground → generate → act → evaluate.

lessons

01Separate planes and stages

The edge authenticates and reserves quota; orchestration assembles context and policy; the model gateway routes; inference workers batch and stream; tool runners execute behind isolation; telemetry and eval pipelines learn from outcomes.

Trace an enterprise assistant in stages. The application authenticates the caller, assembles authorized evidence, asks the model for text or a proposed tool call, validates any action, and publishes the result. The model is one stage in a larger system. Distinguish a bad answer from a failed request, an unauthorized retrieval, and a duplicated action; each needs different telemetry and recovery.

Keep policy and durable run state outside the generated conversation. Record tenant, input identity, selected versions, tool attempts, and terminal status in application-owned storage. A control plane changes model routing, quotas, or tool policy; the data plane serves runs under a known policy version. A control-plane outage should have an explicit effect on existing and new runs.

Streaming improves perceived responsiveness but exposes partial state. Label output that is incomplete, record a terminal result, and let clients inspect a run after disconnection. Do not assume that displaying a plausible final sentence means an external action succeeded. Persist effects and authoritative tool receipts separately from generated prose.

  • Track time-to-first-token separately from total time
  • Cancel work when the client disconnects
  • Keep authorization attached to retrieved context and tools

02Optimize tokens as the core resource

Input tokens consume prefill compute and KV memory; output tokens consume sequential decode time. Prefix caching, continuous batching, smaller context, and model routing change both latency and cost.

Model request cost depends on work shape: input length, generated length, model choice, retries, tool loops, and reused prefixes. Two requests can differ by orders of magnitude in work while counting equally in RPS. Reserve estimated tokens and spend at admission, bound output and step counts, then reconcile actual usage including failed attempts.

Separate queue delay, prefill latency, and decode speed. A faster first token does not guarantee faster completion if generation is long. A cache hit on a reusable prefix can help prefill without making a stale retrieved document safe. Cache identity must include the inputs and versions that determine behavior; permission checks remain a separate application responsibility.

When capacity is scarce, choose degradation by user need. A long research task may become an asynchronous job, while a short interactive extraction may be prioritized. A smaller model is a valid fallback only if it satisfies the required capability and quality bar. Make these choices visible in the product contract and evaluation slices.

  • Batch compatible work without excessive waiting
  • Budget context among instructions, history, retrieval, and tool results
  • Cache only when privacy and semantic equivalence are clear

03Bound probabilistic behavior

Wrap model decisions in deterministic checks: schemas, policies, permission gates, time/cost budgets, loop detection, and idempotent tools. Evals test end-to-end outcomes, not only prose quality.

An LLM can propose a tool argument; deterministic code must authorize the principal, resource, and effect. Bind approval to the exact action parameters and recheck relevant permissions before execution. If access changes while a run waits, the old conversation is not a fresh grant. The same rule applies before publishing restricted evidence to the user.

Define completion with evidence: cited passages for a knowledge answer, a durable operation receipt for an action, or an inspected artifact for a report. Then build an evaluation plan with unsupported answers, stale sources, denied permissions, duplicate calls, and provider failure. Test the whole path because a strong model cannot compensate for a broken authorization filter.

Before drawing a complicated agent architecture, compare a fixed workflow. If steps are known—retrieve policy, check account, propose reply—a workflow offers explicit state and recovery. Use a bounded adaptive loop only where observations genuinely change the path. Its budgets, stop conditions, and escalation behavior belong in the design from the start.

  • Fallback quality change must be product-visible when material
  • Moderation has input/output/tool surfaces
  • Telemetry needs model and prompt versions

worked example

A knowledge assistant loses permission midway through a run

An assistant searches tenant documents, drafts a cited answer, and can create a support ticket. The caller's document access is revoked during execution.

  1. Retrieve only evidence authorized for the caller and tenant; preserve source IDs, versions, and permission context with the evidence.
  2. Treat retrieved text as data. Its instructions cannot grant tool permissions or change the requested task.
  3. Before publishing a sensitive answer or executing the ticket action, revalidate the relevant access and action policy; define how a revoked run stops.
  4. Trace retrieval relevance, citation support, tool authorization, latency, and cost separately so an incident becomes a reproducible eval.

Design decision: Put probabilistic generation inside a durable application workflow with explicit authorization and publication boundaries.

Limit: Citations and structured outputs do not by themselves prove factual support or permission safety.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: An enterprise assistant design with a read path, action path, run state machine, and eval plan.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Specify ingestion freshness, retrieval ACLs, evidence construction, streaming policy, and action approvals.
  2. Trace permission revocation, missing evidence, provider timeout, and duplicate tool delivery.
  3. Write ten evaluation cases across correctness, access, action safety, and abstention; use them to review and revise the design.

Change a constraint

  • The user requests an irreversible external action.
  • The newest source contradicts a document already included in context.

Self-review rubric

  • Authorization persists through retrieval and effects.
  • The workflow has bounded retries and recoverable state.
  • Evaluation measures end-to-end success and separate failure stages.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

The assistant answers from private documents and may create a support ticket. Revocation may happen after retrieval but before publication or execution.

Read: authenticate → authorized retrieve → source/version context
      → draft → recheck publication access → cite or abstain
Action: proposal → schema + current permission → exact approval
        → durable operation ID → execute/reconcile → receipt
Run: queued → retrieving → drafting → awaiting approval → executing
     → completed | denied | unresolved | failed
Eval rows: missing evidence; stale policy; revoked access; duplicate tool;
           timeout-after-commit; unsupported citation; provider outage

Keep run and effect state outside model prose. Revoked evidence must not be published merely because it was fetched earlier. Bind approval to exact ticket parameters; a changed action requires a new decision. Fresh contradictory evidence calls for source/version policy and possibly abstention, not silently increasing top-k.

Check your artifact

  • The read and action paths have different evidence of completion.
  • A duplicate action returns the same receipt or unresolved operation, not another ticket.
  • Quality metrics, request availability, unauthorized effects, latency, and cost are measured separately.

A tempting mistake: Treating the model's chosen tool or final text as authority skips the deterministic permission and effect boundary.

cumulative practice

Relay · build and evolve a private file service

Project brief

Keep one design document for Relay, a tenant-aware file service. Start with 100k daily users, ten downloads per user/day, 10k uploads/day, 1 MB typical bodies, 30-day retention, and a 5× download peak. Files are private to workspaces. Treat these as fictional planning assumptions. Later chapters add previews, recovery, regional constraints, compute, and an assistant. Record each change rather than replacing the earlier version; the final review combines the whole packet. These activities share the chapter's existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Relay's permission-aware file assistant and action/recovery trace.

Add cited answers over files and an optional support-ticket tool. Recall cache, fencing/idempotency, and token budgets. Revoke file access after retrieval and time out a ticket write after commit. Show the two independent recovery decisions.

Compare cumulative reasoning after your attempt

Do not publish revoked evidence; recheck access and abstain or retrieve permitted alternatives. Reconcile the ticket using stable operation identity before retrying; file permission failure does not prove the external ticket failed. Bound tokens and run steps, preserve source versions and receipts, and record terminal or unresolved status outside the conversation. Test both cases together so one recovery does not bypass the other's security boundary.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Always draw the non-model control loop: authenticate → reserve budget → retrieve with ACLs → invoke → validate → act with least privilege → evaluate.

failure patterns

  • Treating the model API as the entire architecture
  • Retrying model/tool calls without tracking duplicate effects
  • Ignoring cancellation after streaming begins
  • Using vector similarity as authorization

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Which metric isolates queue + prefill responsiveness?

    select an option to check your answer

  2. 02Where should retrieval authorization be enforced?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Design a multi-tenant coding-agent backend that can run for an hour, edit repositories, and recover from worker loss.

  • 1h tasks
  • repo tools
  • GPU scarce
  • worker loss
compare a strong approach after your attempt

Persist a run state machine and append-only action log. Schedule model calls through a quota-aware gateway and execute tools in ephemeral sandboxes with scoped credentials. Checkpoint repository state and compact conversation history. Resume from the last committed checkpoint, deduplicate tool actions, gate risky operations, and evaluate task/test outcomes.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally