01Workload shape determines the pool
Online inference values low queue time and predictable latency. Batch training or embeddings can wait, fill troughs, and tolerate preemption. Separate pools or reservations prevent batch work from consuming interactive headroom.
Describe a job by the resources it needs and how long it holds them. A CPU batch transform, a memory-heavy index build, and a GPU decode request are not interchangeable work units. Track duration distributions, memory footprint, accelerator compatibility, and whether interruption is safe. A queue containing ten jobs may represent seconds or hours of work.
Separate scheduling from admission. Admission decides whether the service can promise to accept a job; scheduling decides which accepted job runs next on which resource. Reserve scarce resources before accepting work when necessary, and release or reconcile reservations at completion. If you accept unlimited work then schedule it fairly, users can still wait beyond every promised deadline.
Use workload classes where sharing harms predictability. Interactive requests need low waiting time; batch jobs may tolerate delay and benefit from larger batches. Separate pools or priority reservations can protect interactive latency. Explain the cost: idle reserved capacity may reduce utilization, and strict priority can starve batch work unless you allocate a minimum share.
- Stateless frontends scale cheaply
- Stateful workers need checkpoint and ownership semantics
- Autoscaling lag makes admission control essential
02The bottleneck moves
A service may be GPU-compute bound during decode, memory-capacity bound by KV caches, CPU-bound during tokenization, or network-bound during distributed collectives. Measure per stage.
Compute can be limited by arithmetic, memory bandwidth, device memory, network transfer, or queueing. Increasing one resource helps only while that resource is the bottleneck. A model may fit in memory yet serve too slowly because each generated token requires reading substantial model state. Conversely, enough compute does not help if long contexts exhaust cache memory.
For an illustrative inference workload, 20 requests/s with 1,000 input and 200 output tokens require 20k input tokens/s and 4k output tokens/s. Those rates describe different serving phases and cannot simply be divided by one universal tokens/s rating. Measure capacity under representative context lengths, concurrency, batching, and latency targets; treat calculator coefficients as assumptions to test.
Little's Law relates average work in the system to arrival throughput and average time: L = λW in a stable population. At 20 requests/s and two seconds average residence time, about 40 requests are present on average. This is not a p99 guarantee or a memory estimate by itself. Use it to check consistency between observed concurrency and latency, then inspect distribution tails.
- Bin-pack without fragmenting scarce accelerator shapes
- Data/model locality reduces transfer cost
- Queue age exposes insufficient service rate
03Fairness is explicit policy
FIFO can let one huge request block many small ones. Fair-share scheduling, tenant weights, size classes, and concurrency caps make service predictable. Cost-aware routing may choose a smaller model or batch path.
FIFO is easy to understand but a long job at the front can delay many short ones. Size classes, preemption where supported, or separate queues reduce this head-of-line blocking. A scheduler also needs a fairness unit: jobs, compute time, tokens, or reserved capacity. Equal request counts can give very unequal resource shares.
Give each tenant bounded concurrency and a resource budget, then allocate remaining capacity with a policy such as weighted fair sharing. Protect a minimum batch share from permanent interactive pressure. If you estimate job cost incorrectly, reconcile actual usage and adjust future admission; otherwise a tenant can repeatedly obtain more capacity than its reservation suggests.
Test overload as a normal operating case. Remove part of the fleet, double job size, or introduce one tenant with giant prompts. Specify what is rejected, queued, downgraded, or cancelled, and measure waiting time by class. A healthy average utilization is not enough if one class misses deadlines or one tenant monopolizes device memory.
- Reserve capacity for critical traffic
- Preemption requires checkpoint or restart policy
- Charge/attribute shared resource cost