llm-engineering/04

serving · deep · ~4h practice

Inference & serving

outcome

Trace an API request through prefill, decode, KV cache, batching, parallelism, and scheduling.

Plan a practice session · ~4h including exercises

Read + trace the worked example: 30 min
Assignment + constraint variations: 140 min
Timed drill + repeated explanations: 50 min
Closed-book recall + self-review: 20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

Inference is a scheduling and memory problem wrapped around matrix computation. Understanding stage-specific costs lets you explain latency, throughput, and why serving infrastructure is hard.

coverage

Stages: request queue, prefill, decode, streaming.
Metrics: TTFT, inter-token latency, tokens/sec.
Batching: static and continuous batching.
Memory: weights, activations, KV cache, quantization.
Parallelism: tensor and pipeline parallelism.
Acceleration: speculative decoding, prefix caching, request scheduling.
Tradeoff: cost, latency, throughput, quality.

mental model

Input-heavy prefill and sequential decode stress different resources.

1 →Queue
Admit + batch
2 →Prefill
Process prompt
3 →Decode
One token/step
4 →Stream
Cancel + account

Read in numbered order: queue → prefill → decode → stream.

lessons

01Prefill and decode are different

Prefill processes the prompt in parallel and creates KV state; it drives time to first token and grows with input length. Decode generates sequentially and drives inter-token latency and total completion time.

A request first waits for admission and scheduling. Prefill then processes the input and constructs state used for generation. Decode repeatedly computes the next token, selects it, and extends the sequence. Time to first token includes queueing, prefill, and first-token work; inter-token latency describes subsequent delivery. Time to complete also depends strongly on how many output tokens are generated.

A large prompt can have slow TTFT even with a short answer. A short prompt with a long answer can start quickly yet occupy a slot for a long time. Benchmark both shapes and their mix. If you report only total tokens/s, you cannot tell whether a throughput improvement preserved the interactive latency contract or simply admitted more concurrent work.

Large prompts compete for memory and prefill capacity
Long outputs occupy decode slots
Track latency by stage and token count

02KV cache is the serving currency

Caching prior keys/values avoids recomputing the entire sequence on every output token, but memory grows with batch size and sequence length. Memory pressure limits concurrency even when raw compute remains.

For a common cache layout, bytes per sequence = 2 × layers × cached tokens × KV heads × head dimension × bytes per element. The factor two stores keys and values. With 32 layers, 8,192 tokens, eight KV heads, dimension 128, and two-byte elements, the result is 1 GiB. Use KV heads, which may differ from query heads in grouped-query attention.

Multiply by active sequences only under the stated sharing assumptions, then add weights, temporary workspace, activations, fragmentation, and safety headroom. Sixteen one-GiB caches do not mean a 16 GiB device can serve the workload. Precision, prefix sharing, windowed attention, and architecture alter the estimate. The formula is a planning model to compare with actual allocation and out-of-memory behavior.

Quantization reduces weight memory and may affect quality/speed
Prefix caching reuses shared prompt work
Eviction policy changes latency predictability

03Scheduling creates the frontier

Continuous batching admits new requests as others finish, improving utilization. Tensor parallelism splits operations across devices; pipeline parallelism splits layers/stages. Speculative decoding uses a faster draft path that the target verifies.

Batching combines requests to use hardware more efficiently. Waiting to form a large batch can improve aggregate throughput while worsening request latency. Continuous batching can replace completed sequences with new work, but requests still have different lengths and memory needs. An admission policy needs input/output estimates and concurrency limits, not just a maximum number of HTTP connections.

Parallelism also has distinct costs. Splitting one model across devices adds communication; replicating a model adds independent serving capacity but requires another weight copy. Quantization can reduce memory and bandwidth demand while requiring quality and performance checks. Choose an operating point from measured quality, TTFT, decode latency, and memory under your traffic distribution rather than maximizing utilization alone.

Batching improves throughput but may add queue delay
Parallelism adds communication overhead
Route by workload shape and SLO

worked example

Calculate the KV memory of one sequence

Assume 32 layers, 8 KV heads, head dimension 128, 8,192 cached tokens, and two bytes per cached element.

Keys plus values require 2 × layers × tokens × KV heads × head dimension × bytes per element.
Substitution gives 2 × 32 × 8,192 × 8 × 128 × 2 = 1,073,741,824 bytes, or 1 GiB for one sequence.
Sixteen such sequences require about 16 GiB of KV data before allocator overhead, model weights, activations, or workspace.
Separate prefill, which processes the prompt, from sequential decode, which extends each sequence. Benchmark both phases under the intended concurrency and lengths.

Design decision: Bound context and concurrency using KV memory as well as compute demand; reserve headroom for the rest of the serving process.

Limit: This formula assumes the stated cache layout and precision. Quantization, sharing, sliding windows, and architecture changes alter the footprint.

Teaching scenario · traffic, prices, and capacity values are illustrative.

lab

interactive lab

KV Cache — why long context costs memory

1 · predict first

80 layers × 8 KV heads × 128 head-dim × fp16 × 4k tokens: roughly how many GB per request? Guess an order of magnitude first.

pedagogical model — values are illustrative, not measured.

assignment

Produce: A capacity worksheet for three context lengths and two concurrency levels, plus an admission policy.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

Reproduce the worked calculation in the lab and separate decimal GB from binary GiB.
Add model weights, overhead, input/output length distributions, TTFT, and inter-token latency to the capacity plan.
Design a representative load test and an overload policy; explain the consequences of rejecting, queueing, or truncating a request.

Change a constraint

Double cached context with fixed concurrency.
Switch from eight KV heads to two while keeping attention-query heads unchanged.

Self-review rubric

The KV formula uses KV heads rather than hidden width.
The estimate includes more than weights.
Latency and admission decisions are tied to the workload.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Use 32 layers, eight KV heads, head dimension 128, two bytes per cached element, no shared prefixes, and the common layout in the worked example.

Context    KV/sequence   concurrency 1   concurrency 16
4,096      0.5 GiB       0.5 GiB         8 GiB
8,192      1 GiB         1 GiB           16 GiB
16,384     2 GiB         2 GiB           32 GiB
Formula: 2 × layers × tokens × KV heads × headDim × bytes
Total device plan = weights + KV + workspace/overhead + headroom
Admission: context/output bounds + memory reservation + latency budget
Load test: vary prompt/output distributions, concurrency, and batching

These are KV values, not total device memory. Switching from eight to two KV heads quarters this component under otherwise identical assumptions. Measure input processing and output generation separately and report TTFT and inter-token latency as well as throughput. Reject or defer work that cannot fit the required memory and latency envelope.

Check your artifact

The eight-KV-head, 8,192-token calculation equals 1,073,741,824 bytes.
GB and GiB are not silently interchanged.
The worksheet includes weights and runtime allocations and does not infer a real fleet size from one cache formula.

A tempting mistake: Using query-head count or hidden width in place of KV-head width can substantially overstate cache size for grouped-query attention.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's phase-demand worksheet and long-ticket policy.

At the brief's 20 requests/minute, 1,500 input and 200 output tokens each, derive average phase demand. Recall the cache formula and explain why a few 16k-token tickets can alter concurrency limits even at unchanged RPS.

Compare cumulative reasoning after your attempt

20/60 requests/s gives 500 input and about 66.7 output tokens/s before retries. These are separate phase demands, not a universal throughput requirement. Under the example's layout, 16,384 tokens need 2 GiB KV per active sequence. Bound context and concurrency, reserve output headroom, and load-test bursts and long-ticket slices. Average demand alone does not establish a safe serving configuration.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Draw two latency bars: TTFT = admission + queue + prefill; completion = TTFT + decode steps. Then show which optimization attacks each part.

failure patterns

Using tokens/sec as the only user-experience metric
Calling batching free throughput
Ignoring KV memory in concurrency estimates
Assuming quantization only affects file size

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

01What most directly determines TTFT?
select an option to check your answer
02Why is decode hard to parallelize across time?
select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

1. In the example, what happens to KV memory when context doubles to 16,384 tokens?
2. Why can a throughput improvement make interactive latency worse?

interview drill

A model service has good average latency but p99 TTFT spikes when long prompts arrive. Diagnose and redesign scheduling.

mixed prompt lengths
p99 TTFT
fixed GPU fleet
streaming

compare a strong approach after your attempt

Break latency into queue and prefill. Introduce size-aware queues or prefill chunking, bound huge contexts, reserve capacity for short interactive traffic, and enforce tenant fairness. Monitor queued tokens and KV occupancy, not only request count.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Kwon et al.: PagedAttention and LLM serving ↗reviewed 2026-10

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally