llm-engineering/02

foundations · core · ~3h practice

Transformer fundamentals

outcome

Explain attention and the transformer stack intuitively, including long-context cost.

Plan a practice session · ~3h including exercises

Read + trace the worked example: 25 min
Assignment + constraint variations: 100 min
Timed drill + repeated explanations: 35 min
Closed-book recall + self-review: 20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

You do not need to derive gradients, but you should be able to trace information through embeddings, attention, feed-forward blocks, residuals, and causal masking.

coverage

Inputs: token embeddings and positional information.
Attention: queries, keys, values, similarity weights.
Capacity: multi-head attention and feed-forward layers.
Stability: residual connections and layer normalization.
Generation: causal masking and stacked blocks.
Scaling: why naive attention grows quadratically with sequence length.

mental model

Each token reads a weighted mixture of earlier token information, then transforms it locally.

1 →Embed
Token + position
2 →Attend
Q·K weights V
3 →Transform
Feed-forward
4 →Repeat
Residual stack

Read in numbered order: embed → attend → transform → repeat.

lessons

01Attention is content-addressed mixing

Each position creates a query describing what it seeks, keys describing what positions offer, and values containing information to mix. Query–key similarity becomes weights over values.

At each position, learned projections produce query, key, and value vectors. A query is compared with keys to produce scores; scores over permitted positions are normalized into weights; the output is a weighted sum of values. For weights 0.25 and 0.75 over [4,0] and [0,4], the mixture is [1,3]. The model learns the projections rather than using handwritten semantic rules.

In a causal decoder, a position cannot attend to later positions. Apply the mask before normalization so future positions get zero probability, then normalize over allowed positions. Several heads can form different mixtures, but a visible attention pattern does not by itself prove a head understands a grammatical relationship. Output behavior arises from the whole computation, not one highlighted link.

Causal masks block future positions
Multiple heads learn different interaction patterns
Attention moves information; it is not a symbolic database query

02The block has two major jobs

Attention combines information across positions; the feed-forward network transforms each position independently. Residual paths preserve and refine representations, while normalization stabilizes computation.

Token embeddings turn IDs into vectors; positional mechanisms provide order information. Attention mixes information between positions. Feed-forward layers transform each position's representation. Residual connections add a layer's contribution to an existing representation, while normalization helps regulate the computation. Repeating blocks builds representations used to predict the next token. This is a useful information-flow model without deriving every matrix operation.

Do not confuse attention heads with separate little agents or the KV cache with learned knowledge. The cache stores intermediate keys and values for a sequence during serving. Model weights persist across requests; a request's cached state is associated with its input. This separation explains why longer context increases serving memory even though the model's parameter count has not changed.

Depth composes many small transformations
Position is required because attention alone is permutation-insensitive
Outputs become logits over vocabulary

03Long context has real systems cost

A dense attention map compares many token pairs, so prefill work and memory rise quickly with sequence length. During decoding, a KV cache avoids recomputing past keys and values but consumes accelerator memory.

A dense attention pattern contains roughly n² query-key pairs for n positions. Doubling n creates four times as many pairs, which motivates careful attention implementations. It does not prove that every implementation allocates an n-by-n matrix in device memory: algorithms can tile computation and avoid materializing the full map. Distinguish a mathematical operation count from a measured memory or latency result.

Serving also distinguishes prompt processing from incremental generation. Prefill processes many known prompt positions; decode extends a sequence one token at a time using cached state. Longer input increases prefill work and cache footprint, while longer output adds more decode steps. These mechanisms are why a large advertised context window is a capacity limit, not a promise of equal quality, latency, or cost at every length.

Long capacity does not guarantee perfect retrieval
Context selection remains an application concern
Architecture optimizations can change the exact scaling constants

worked example

Compute one causal attention mixture

A token at position 3 has illustrative attention weights 0.2, 0.3, 0.5 over positions 1–3. Their two-dimensional value vectors are [1,0], [0,2], and [2,2].

The weighted value sum is 0.2[1,0] + 0.3[0,2] + 0.5[2,2] = [1.2,1.6]. Attention mixes representations across positions.
A causal mask excludes position 4 and all later positions before the weights are normalized. Future content contributes zero.
Feed-forward layers then transform each position's representation; residual paths carry information across the stack.
Doubling sequence length gives four times as many token pairs in a dense attention map. Implementations can avoid materializing that whole map, so pair count is not a direct memory benchmark.

Design decision: Use the block structure to reason about information flow and context cost without assigning a literal semantic role to every head.

Limit: This mixture omits learned projections, scaling, normalization, and multiple heads. Attention weights alone are not a full explanation of an output.

Teaching scenario · traffic, prices, and capacity values are illustrative.

lab

interactive lab

Attention Head Explorer — where “it” looks

1 · predict first

Before clicking: which prior token do you think “it” attends to most — “street” (recent) or “animal” (correct)? Commit. Note the constraint: nothing may look into the future.

pedagogical model — values are illustrative, not measured.

assignment

Produce: A four-token causal attention diagram, one calculated mixture, and an annotated transformer block.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

Compute two weighted mixtures by hand and mark the masked entries.
Use the attention lab with different query positions; explain what changes and what remains illustrative.
Trace embeddings through attention, feed-forward, residual paths, and logits; rehearse a three-minute explanation and repair the weakest step.

Change a constraint

Remove positional information and discuss the lost ordering signal.
Double context length and separate pair-count growth from implementation memory behavior.

Self-review rubric

Future positions receive zero weight.
The weighted sum and normalization are correct.
The explanation distinguishes cross-position mixing from local transformation.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Four positions use causal attention. At position 3 the allowed weights are 0.2, 0.3, 0.5 with values [1,0], [0,2], [2,2].

Allowed positions by query: 1→{1}; 2→{1,2}; 3→{1,2,3}; 4→{1,2,3,4}
Position 3: 0.2[1,0] + 0.3[0,2] + 0.5[2,2] = [1.2,1.6]
Position 2 example: 0.25[1,0] + 0.75[0,2] = [0.25,1.5]
Block: token + position representation → attention mixing
       → residual/normalization + position-local feed-forward
       → repeated blocks → logits → next-token selection
Future-position weight at query 3: position 4 = 0

The diagram is an information-flow sketch, not an exact ordering for every transformer architecture. Learned projections create scores and values; the illustration supplies weights directly. Removing positional information loses the intended ordering signal. Doubling a dense attention length quadruples pair count but does not establish the allocated memory of a tiled implementation.

Check your artifact

Every row normalizes over allowed positions only.
The weighted sums match the displayed vectors.
You distinguish cross-position attention from per-position transformations and do not assign a proven semantic role to one head.

A tempting mistake: Masking after an already normalized distribution without renormalizing leaves the wrong total mass on allowed positions.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's generation diagram and context-cost explanation.

Reconstruct the tokenize/predict/sample loop without notes. Add a causal mask and explain why a long ticket changes prefill work and cached state even if the answer remains short. Which part of this diagram supplies the company's actual refund policy?

Compare cumulative reasoning after your attempt

Token embeddings and positional information feed causal mixing and local transformations, then logits and selection repeat. Longer input adds prompt processing and sequence state. No attention mechanism supplies an absent private policy by itself: the application must provide authorized evidence. A visible attention link is an illustration, not proof that the final deadline claim is supported.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Use an example: in “the animal didn't cross the street because it was tired,” one head can weight earlier tokens to refine what “it” represents.

failure patterns

Calling attention literal human attention
Saying every head has a known interpretable role
Forgetting causal masking in autoregressive generation
Claiming context windows are free storage

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

01What prevents an autoregressive token from reading future tokens?
select an option to check your answer
02Attention's core operation produces…
select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

1. Using weights 0.25 and 0.75 over values [4,0] and [0,4], what is the mixture?
2. Does a visible attention link prove that one head understands a pronoun?

interview drill

Explain queries, keys, and values using a search-within-a-meeting analogy, then state where the analogy fails.

3 min
draw 4 tokens
show weights
name limitation

compare a strong approach after your attempt

Each current token emits a query; each prior token advertises a key and carries a value. Similarity weights decide the mixture. The analogy fails because learned vector operations are distributed and repeated across many heads/layers, not discrete document lookup.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Vaswani et al.: Attention Is All You Need ↗reviewed 2026-10
Kwon et al.: PagedAttention and LLM serving ↗reviewed 2026-10

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally