skip to content

learn.smmeyer.devllm-engineering17

menu

llm-engineering/17

agent systems · deep · ~4h practice

LLM infrastructure at scale

outcome

Design gateways, queues, caches, quotas, tenant isolation, and capacity around scarce model compute.

Plan a practice session · ~4h including exercises
Read + trace the worked example
30 min
Assignment + constraint variations
140 min
Timed drill + repeated explanations
50 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

This chapter is system design through an AI lens: requests differ by tokens and features, streams stay open, providers/fleets vary, and cost attribution matters per tenant and model.

coverage

  • Gateway: model abstraction, policy, multi-model routing.
  • Admission: quotas, rate limits, concurrency, token reservation.
  • Scheduling: queues, priority, batch inference, fairness.
  • Delivery: streaming connections, cancellation, backpressure.
  • Efficiency: prefix/output caches and batching.
  • Operations: telemetry, cost attribution, capacity, regional availability.
  • Scarcity: GPU allocation and graceful degradation.

mental model

A model gateway converts tenant policy and workload shape into an admitted, routed, observable request.

  1. 1 →Identify

    Tenant + task

  2. 2 →Reserve

    Tokens + spend

  3. 3 →Route

    Model + region

  4. 4 →Account

    Actual + outcome

Read in numbered order: identify → reserve → route → account.

lessons

01Rate limits need token awareness

Request count hides a 100-token extraction versus a 100k-token agent turn. Combine request, token, concurrency, and spend budgets. Reserve estimated usage on admission, then reconcile actual input/output/tool cost.

Two tenants each sending ten requests/s may impose very different load if one uses 100-token prompts and the other 100k-token prompts. Add generated length, tool steps, and concurrency to the model. Reserve estimated input/output work and spend on admission, then reconcile actual usage. Bound estimation error with output and step limits rather than accepting unlimited future work.

Apply limits at tenant and shared-fleet levels. A tenant under its quota can still arrive during global overload. Keep interactive and batch classes separate where needed, expose retry timing truthfully, and avoid repeated immediate retries. Admission should reject or defer before scarce compute and memory are committed.

  • Separate tenant and global fleet limits
  • Expose truthful retry/queue semantics
  • Protect high-priority capacity

02The gateway owns cross-cutting policy

Centralize supported-model mapping, retries, routing, fallbacks, usage, tracing, and safety hooks without hiding capability differences. Avoid a lowest-common-denominator interface that erases streaming or tool semantics.

An internal gateway can centralize authentication, capability-aware routing, quotas, typed events, usage, and tracing. Keep provider differences visible in types where they affect the product: supported modalities, tool behavior, state semantics, and cancellation. A single text-returning wrapper is too weak if the application depends on tool lifecycle or usage events.

Version routing policies and preserve provider request IDs. Before rerouting a failed request, determine whether partial output or external effects already exist. A circuit breaker can stop calls to an unhealthy backend, but the fallback still needs evaluated capability and reserved capacity. Record the chosen backend and policy so a quality regression can be traced to a routing change.

  • Version routing policy
  • Circuit-break unhealthy models/regions
  • Preserve provider request IDs for debugging

03Capacity is token-shaped

Forecast queued input/output tokens, context distributions, cache hit rates, and model mix—not only RPS. Regional headroom and fallback capacity must handle correlated demand or fleet loss.

Forecast input and output demand independently. At 30 requests/s with 2,000 input and 300 output tokens, demand is 60k input and 9k output tokens/s before retries. These phases have different performance characteristics. Also model active context memory and length distributions: the average can hide a small group of very long requests that consumes most cache capacity.

Test fairness and headroom by removing capacity and changing tenant mix. Equal request slots are not equal resource shares. Use per-tenant concurrency, size classes, or weighted resource allocation with an explicit priority policy. Track waiting time by class, cancellations, rejected work, and cost per useful completion. A high fleet utilization score can coexist with an unacceptable interactive service.

  • Cancel disconnected work
  • Batch offline traffic
  • Degrade by context, model, priority, or async path

worked example

A noisy tenant changes the token mix

A gateway sees unchanged 100 RPS, but one tenant raises average input length from 1k to 20k tokens for half the requests.

  1. Input demand becomes 50 × 1k + 50 × 20k = 1.05M tokens/s, compared with 100k previously: a 10.5× increase at unchanged RPS.
  2. Long prompts affect prefill demand and active KV memory. Decode demand depends on output lengths and must be budgeted separately.
  3. Use tenant-aware admission and bounded interactive/batch queues. Reserve estimated work and reconcile actual usage after completion.
  4. Measure per-slice TTFT, token latency, queue age, rejection, and cost before changing batch size or adding a fallback.

Design decision: Account for token-shaped work and fairness at the gateway, then route within measured fleet limits.

Limit: Token count alone is an imperfect resource predictor. Model, cache reuse, batching, and architecture affect actual service cost.

Teaching scenario · traffic, prices, and capacity values are illustrative.

lab

interactive lab

GPU capacity — token-shaped demand meets fixed supply

1 · predict first

120 RPS × (800 in + 300 out) tokens on 8 GPUs: over or under water? Guess the utilization first.

A guess makes the comparison useful. This lab draft lasts until you leave the page.

pedagogical model — values are illustrative, not measured.

assignment

Produce: A tenant capacity model, scheduling policy, and overload playbook.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Use the shared capacity lab for three traffic mixes; calculate input/output demand independently.
  2. Allocate interactive and batch queues, token reservations, concurrency caps, and cancellation accounting.
  3. Run a tabletop with a noisy tenant and a quarter-fleet loss; define fairness, degradation, and the measurements required to validate them.

Change a constraint

  • Retries double demand during a provider incident.
  • A high-priority tenant has a strict latency contract but limited reserved capacity.

Self-review rubric

  • The model distinguishes requests from work.
  • One tenant cannot consume unbounded shared capacity.
  • Admission accounts for memory, latency, and failure headroom.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

There are 100 RPS. Half now use 20k input tokens and half retain 1k. Output demand, KV layout, and measured fleet capacity still require their own inputs.

Before: 100 × 1k = 100k input tokens/s
After: 50 × 20k + 50 × 1k = 1.05M input tokens/s (10.5×)
Reservation: tenant + input estimate + bounded output + active memory
Queues: protected interactive share / bounded batch / tenant caps
Reconcile: actual tokens, tool spend, cancellation, failed attempts
Tabletop: long-prompt tenant + 25% fleet loss + provider retry wave
Observe: TTFT/wait by class, active KV, rejects, fairness, cost/success

Unchanged RPS conceals a large work change. Request slots alone may let long prompts dominate memory and input processing. Fleet sizing must use representative benchmarks at the latency target. A high-priority tenant can consume its reserved share, but strict latency cannot be promised for unlimited overflow without additional capacity or admission limits.

Check your artifact

  • The input calculation is 10.5× rather than 20× because only half the traffic changes.
  • Retries pass through the same shared and tenant budgets.
  • Cancelling work releases or reconciles reservations without declaring unknown usage to be zero.

A tempting mistake: Autoscaling on HTTP request count alone can miss a token and memory overload until queues have already grown.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's token-aware gateway and overload playbook.

Recover the original 20 requests/minute demand. Half the inputs rise from 1,500 to 15,000 tokens while output stays 200. Calculate the new input rate, compare output demand, and show what reservations, queues, and cancellation accounting must change.

Compare cumulative reasoning after your attempt

Ten long and ten ordinary requests/minute give 165k input tokens/minute, or 2,750/s, versus 500/s before: 5.5×. Output remains about 66.7/s before retries. Reassess prefill and active-memory capacity, not just decode. Bound per-tenant reservations and context, protect interactive work, defer bulk investigation, and reconcile actual usage for completed, failed, or cancelled runs.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Estimate tokens/sec and KV occupancy alongside RPS. This is the bridge from classic capacity planning to inference systems.

failure patterns

  • One RPS limit for every request size
  • Opaque routing that changes capabilities silently
  • No cost attribution by tenant/task
  • Fallback fleet sharing the same failure domain

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Why is RPS alone insufficient?

    select an option to check your answer

  2. 02What should admission reserve?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Design a model gateway serving 10k enterprise tenants across hosted and self-run models.

  • 10k tenants
  • multi-model
  • regional
  • cost attribution
compare a strong approach after your attempt

Authenticate tenant, classify capability/risk, reserve token/spend/concurrency quotas, route by policy and health, stream typed events, and reconcile usage. Maintain capability metadata, per-route circuit breakers, weighted fairness, regional failover, prompt/model version telemetry, and tested degradation tiers.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally