skip to content

learn.smmeyer.devllm-engineering18

menu

llm-engineering/18

llm platforms · current · ~4h practice

Modern LLM platform concepts

outcome

Navigate the durable primitives exposed by modern LLM platforms and reason about provider evolution without memorizing a release snapshot.

Plan a practice session · ~4h including exercises
Read + trace the worked example
30 min
Assignment + constraint variations
140 min
Timed drill + repeated explanations
50 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

Platform details change. Engineering fluency means understanding durable concepts—model requests, typed content, streaming, tools, retrieval, realtime sessions, state, and evaluation—while checking each provider's current official documentation for models, limits, deprecations, and capabilities.

coverage

  • Core API: model requests, typed content/items, multimodal input/output, and provider adapters.
  • Delivery: semantic streaming events and conversation state.
  • Control: structured outputs, function calling, built-in tools.
  • Knowledge: files, retrieval/vector stores, embeddings.
  • Realtime: low-latency audio sessions and server controls.
  • Agents: long-running/background work, tool loops, tracing.
  • Platform: safety layers, rate/usage limits, model strategy, compatibility.

mental model

Treat every model platform as a set of composable primitives behind an application-owned policy, state, and evaluation layer.

  1. 1 →Model API

    Typed content + state

  2. 2 →Tools

    Built-in + custom

  3. 3 →Realtime

    Live sessions

  4. 4 →Evaluation

    Measure behavior

Read in numbered order: model api → tools → realtime → evaluation.

lessons

01Understand the provider's application primitive

Modern provider APIs represent messages, content, actions, and tool calls through structured request and response types. OpenAI's Responses API is one concrete example; other providers expose similar capabilities through different contracts. Build around the durable concepts rather than one vendor's naming.

Build an internal protocol around meaning: text deltas, tool proposals, tool observations, usage, errors, and terminal completion. A provider-specific payload maps to these events through an adapter. Preserve required capability information rather than flattening everything into text. Treat “stream closed” separately from “operation completed successfully,” because network closure can also mean failure.

Persist application run identity and business state even if a provider stores conversation state. The product still owns access, retention, approval, and recovery semantics. Record the provider mapping and versions used for a run so a later migration can reproduce or diagnose behavior. Learn the durable boundaries while checking volatile endpoint details in current official documentation.

  • Keep an internal adapter around vendor specifics
  • Use typed streaming lifecycle events
  • Persist your own product state and audit semantics

02Choose real-time architecture by interaction

Realtime sessions keep a low-latency connection open for live audio, conversation events, and tools. Request-based APIs remain simpler for bounded transcription or speech tasks. Server-side controls keep business logic and credentials away from clients.

A live voice session has a different latency budget from a queued transcription job. Track capture, transport, inference, first audible output, and interruption response. Cancelling audio playback is not necessarily cancelling an in-flight tool action. Use session and turn identities so late events from an interrupted turn do not become the current answer.

Keep sensitive controls on the trusted service side. A client connection can carry media without granting unrestricted tool or credential access. Handle reconnects, duplicate events, and partial turns explicitly. Compare a persistent realtime session with a simpler request-based path using measured interaction needs; long-lived connections add state and operational responsibilities.

  • WebRTC fits client media; WebSocket fits server connections
  • Handle interruption and cancellation
  • Measure audio/session cost separately

03Documentation is part of the design process

Model availability, tool support, rate limits, and deprecations are update-sensitive. Verify them in the selected provider's official documentation at implementation time; protect product behavior with capability checks, adapters, evals, and staged migrations.

Maintain a dated capability matrix for the configurations you actually support: input/output types, tool contract, streaming events, cancellation, usage reporting, and required limits. An adapter can normalize syntax but cannot manufacture a missing capability. Routing must check the matrix before accepting a task that depends on it.

Protect migrations with contract probes, replayed evaluation cases, staged traffic, and rollback. If usage arrives only in a final event and the stream breaks, accounting should remain unresolved until reconciled rather than becoming zero. Separate assumptions you verified from ones you still need to test. That makes changing provider documentation an operational input rather than a reason to memorize a release snapshot.

  • Do not couple product contracts to one model alias
  • Pin/version behavior where needed
  • Re-run evals before model or prompt migration

worked example

A provider switch breaks a valid-looking tool response

A gateway routes to another provider after an outage. Its adapter expects text deltas, but the response contains interleaved tool-call argument events.

  1. Represent text, tool proposals, tool results, usage, errors, and terminal states as distinct application events.
  2. Accumulate partial tool arguments according to the provider contract. Do not execute an action merely because a partial fragment parses as JSON.
  3. Check capability compatibility before routing: required tools, structured output behavior, cancellation, context handling, and data controls.
  4. Run adapter contract tests and end-to-end evals for normal, interrupted, and multi-tool streams before enabling fallback traffic.

Design decision: Own a stable application protocol and make provider adapters verify their current supported capabilities.

Limit: This is a hypothetical protocol mismatch. Provider support and event shapes must be checked in the selected provider's current official documentation.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: A provider-neutral event schema, capability matrix, and migration checklist.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Define text, tool, usage, error, and completion events, including partial and terminal states.
  2. Inspect the linked official docs for one provider and record a dated mapping; use fictional adapters for a second if access is unavailable.
  3. Trace fallback during a partial stream, cancellation, and an unsupported capability; design a staged rollout and rollback.

Change a constraint

  • The fallback supports text but lacks the required tool mode.
  • Usage arrives only in a final event that never reaches the client.

Self-review rubric

  • The application protocol preserves typed event meaning.
  • Routing checks required capabilities before execution.
  • Update-sensitive assumptions have dated sources and contract tests.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Use fictional adapters unless you inspect a provider's current official contract. No capabilities or prices in this sample assert a current vendor guarantee.

Internal event: {runId, sequence, type, itemId, payload}
Types: text_delta | tool_arguments_delta | tool_ready | usage
       | error | completed; transport close ≠ completed
Capability matrix: input modes / tool mode / stream meaning /
                   cancellation / state / usage, each verified + dated
Adapter A: maps named provider events → typed internal events
Adapter B: preserves interleaved tool items; validates complete args
Fallback: required capability check → safe new attempt, never merge
          unrelated partial tool arguments across providers
Migration: contract fixtures + eval replay → canary → rollback

The adapter preserves semantics rather than hiding meaningful differences. A text-only fallback cannot satisfy a required tool contract. If the final usage event is missing, mark usage unresolved and reconcile through the available accounting path; do not treat a broken stream as free work. Add a dated mapping from documentation and a contract probe before calling a provider supported.

Check your artifact

  • Tool fragments are associated with the correct item ID and not executed before complete validation.
  • Only a terminal application event or durable status establishes completion.
  • Capability and version changes trigger contract and task-level regression checks.

A tempting mistake: A lowest-common-denominator text wrapper can silently erase the event semantics the product relies on.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's provider adapter contract and migration evidence.

Recall the run state, effect identity, and usage reservation. Fail a provider after text and a tool proposal have streamed. State what a compatible fallback must preserve, what it must not replay, and how unknown final usage is represented.

Compare cumulative reasoning after your attempt

Persist already published content and proposed/executed tool identities. Check required capabilities before routing a new attempt; never replay a committed effect or splice unrelated provider fragments into one proposal. The internal event stream identifies items and completion separately from network closure. Keep usage unresolved until reconciled, preserve reservations appropriately, and gate migration with contract fixtures plus the cumulative task suite.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Say: “I know the durable platform primitives; I would verify current model and tool support in the provider's official documentation before locking the implementation.”

failure patterns

  • Memorizing current model names as architecture
  • Assuming every model supports every tool or modality
  • Letting provider conversation state become your only source of truth
  • Migrating models without evals and rollback

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Where should current model/tool compatibility be verified?

    select an option to check your answer

  2. 02How should an application handle provider-specific request and streaming formats?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Sketch a vendor-resilient LLM application layer for chat, tools, file retrieval, and streaming.

  • one initial provider
  • streaming
  • tools
  • future evolution
compare a strong approach after your attempt

Define an internal request/event/tool contract, map it to the provider's structured content and streaming events, store durable application state, and isolate capability differences. Keep tool execution and authorization in your service, record versions and usage, verify current limits in official docs, and gate migrations with offline evals plus canary rollback.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally