Overview

The unit of work is a token, not a request; the request's total cost is unknown when it arrives (you cannot know output length in advance); and the dominant resource is memory held for the duration of the generation, not CPU consumed at the start.

Because the model already has the entire prompt in hand before generation starts -- there is no sequential dependency between prompt tokens the way there is between generated tokens, so prefill is a parallel, compute-bound pass.

Because each new token depends on every token generated before it -- token N+1 cannot be computed until token N exists. This sequential dependency is structural, not a hardware limitation.

Each decoded token requires reading the full weight set (and the growing cache) from GPU memory, so decode throughput tracks memory bandwidth, not arithmetic throughput. A bandwidth-bound operation gets no benefit from more FLOPs.

Without a cache, generating the Nth token would require re-projecting all N-1 prior tokens' key/value tensors every single step, scaling quadratically with sequence length. Caching avoids that re-projection -- but not attention scoring itself, which still costs O(t) per step even with the cache in place.

Sequence length, batch size, number of layers, and number of attention heads (along with the precision's bytes-per-value). Computing this precisely is the prerequisite to every later capacity decision.

Because available cache memory, not compute, usually sets the maximum number of concurrent requests a GPU can serve -- once cache is exhausted, no more requests can be admitted even if the GPU's arithmetic units have spare capacity.

Variable and unpredictable generation lengths force a contiguous allocator to reserve the worst-case length for every request up front -- most requests finish well short of that reservation, and the unused remainder cannot be reclaimed for anyone else.

Operating-system virtual-memory paging -- allocating fixed-size blocks instead of contiguous regions. It fixes co-08's fragmentation by bounding waste to at most one partially-full block per request, instead of the whole worst-case-minus-actual gap.

Prefix sharing only deduplicates the literal, token-for-token identical portion at the start of two sequences -- if that shared prefix has zero length, there is nothing to deduplicate, and both requests need their own full, independent cache blocks.

It stays reserved and idle until every other request in the batch also finishes -- static batching waits for the whole group, so an early-finishing request's slot is wasted for the remainder of the batch's duration.

Requests are admitted into a free slot and retired out of it the instant they finish, checked at every single decode step -- not once per whole batch. This keeps the batch close to full throughout instead of idling on early finishers.

Because the scheduling policy decides the ORDER requests are served in, not the requests' own cost -- first-come-first-served can make many cheap requests wait behind one expensive one, while a shortest-first policy processes cheap requests first, producing a very different tail with zero change to the underlying work.

It is discarded, not paused -- the request's cache is evicted and its generation must be recomputed from scratch later. Preemption trades that one request's latency for the scheduler's ability to admit other work right now.

Because they move in the same direction from the same lever -- a larger batch raises aggregate throughput and simultaneously raises each individual request's inter-token latency. Every configuration is a deliberate point on that one trade-off, never a free win on both axes.

Time-to-first-token (TTFT), inter-token latency (ITL), per-user tokens per second, and aggregate throughput -- four distinct numbers, each optimized by opposing serving choices.

Every in-flight request's decode step must wait for that prefill to finish, worsening their ITL by roughly the full prefill duration -- the new request's TTFT improves at every in-flight request's expense.

Weights, KV cache, activations, and framework overhead. The remainder after weights, activations, and overhead is what actually buys concurrency via the cache-size arithmetic (co-06).

Because the quality cost is task-dependent -- a published degradation figure on one benchmark does not transfer reliably to a different task or workload. The trade (lower precision buys memory and speed at a quality cost) is durable; the specific measured cost is not.

Per-layer interconnect traffic between devices (which can dominate decode-step latency on a slow link) and all fault tolerance -- a single device's failure fails the entire request, since there is no redundancy.

Because real traffic is a distribution with a long tail -- a handful of very long generations can dominate resource consumption far more than their share of request count suggests. Planning against the average hides exactly that tail.

It measures a workload that does not exist -- the resulting throughput or capacity number can meaningfully understate (or overstate) the real wall-clock cost, because the real distribution's mix of short and long requests batches differently than a uniform one.

Because loading multi-gigabyte weights takes seconds, not milliseconds -- a policy that waits until the queue is already deep before scaling out will still be mid-cold-start by the time the queue has grown well past the threshold. A GPU-aware policy must project queue growth across the cold start.

Weights, runtime, and configuration -- versioned together so the served artifact is a single, identifiable, reproducible unit, not three independently-changing pieces.

Because it has quality consequences just like any code change -- a staged rollout with a guardrail bounds the blast radius of a bad model to a small fraction of traffic, and a rollback rule reverses it automatically the moment a regression is detected, rather than propagating a bad model to 100% of traffic on a single all-or-nothing swap.

Queue depth, batch occupancy, cache utilization, preemption rate, and the latency metrics from co-16 (especially ITL). Read together, this small set is usually enough to diagnose an incident without inspecting individual request traces.

It wins on sustained high utilization, data residency, latency floors, and model control; it loses on operational burden, idle cost, and elasticity. The right decision depends entirely on which side of that trade your traffic and organization sit on.

Idle hours, engineering time, on-call burden, and -- most importantly -- the utilization you will REALISTICALLY achieve, not full load. The identical GPU at the identical hourly rate can be cheaper or far more expensive than the API purely as a function of that realistic utilization.

The load test used a uniform, unrealistically cheap workload instead of the real prompt/output length distribution (co-21, co-22) -- 100 identical short prompts batch far more efficiently than a realistic mix containing occasional long generations. The reported number describes a workload that does not exist in production; the fix is to characterize the real length distribution and load-test against it directly.

No -- this is exactly co-07's point: available cache memory, not compute, usually sets the concurrency ceiling. A GPU can be compute-idle at 40% while completely full on KV cache, refusing new admissions correctly. The "bug" is a wrong mental model, not a real defect; the fix is to dashboard cache utilization alongside compute utilization (co-26).

The specific mechanism is co-11/co-12: the old server idled every finished request's slot until the whole batch completed, while the new engine admits and retires at token granularity, eliminating that idle-slot waste. "The new framework is faster" is not an explanation; "continuous batching eliminates static batching's idle-slot waste" is -- and it is directly measurable by comparing slot-steps used versus slot-steps wasted on the identical workload.

A staged rollout with BOTH an error-rate and a latency guardrail (co-25) -- a model swap is a production change requiring the same staged discipline as any release, and error rate alone (which stayed clean here) is not sufficient; and serving observability tracking ITL/p99 latency continuously (co-26), so a slow-but-correct regression is visible immediately rather than discovered days later through user reports.

The comparison used full-load utilization instead of realistic utilization (co-27, co-28) -- real traffic is spiky, and a GPU bills for every hour it is on regardless of how busy it actually is during that hour. The honest comparison divides the same hourly rate by REALISTIC (not peak) effective throughput, which this topic's own Tensions & trade-offs section calls out as "almost always wrong" when done at full load.

This is starvation (co-13): strict priority with no fairness guarantee can starve a lower class completely if higher-priority work keeps arriving. The minimal fix is a fairness mechanism -- e.g. weighted round-robin -- that guarantees every class a nonzero share of slots every round, priority only breaking ties within that guaranteed share.

The policy is reactive, not proactive (co-23) -- it ignores the multi-second cold start required to load model weights onto a new GPU replica. By the time the naive threshold fires and a new replica finishes loading, the queue has grown well past what the threshold predicted. The fix projects queue growth across the measured cold-start window and scales out earlier.

Evaluating the quantized candidate on the team's own eval suite and measuring an actual quality delta (co-19), rather than a small number of hand-picked prompts -- this is the exact same "vibe check" failure mode that applies to any model change: five manually-tried cases cannot reveal a regression on a case class nobody thought to try. A quantization decision record should cite a measured delta against a real, fixed eval set, never an impression from ad hoc spot checks.

Because co-07 establishes that cache memory -- not compute -- is what actually limits concurrency, any technique that wants to increase how many requests a GPU can serve must ultimately act on memory, not arithmetic throughput. Quantization shrinks the bytes each cached value costs; paged allocation shrinks the bytes wasted to fragmentation. They attack the identical bottleneck from two different angles (how much each unit costs vs. how much is wasted per unit), which is exactly why a real deployment typically uses both together rather than treating them as competing choices.

Continuous batching's admit/retire-at-token-granularity mechanism works regardless of how cache is allocated, but it becomes dramatically more efficient once cache is addressed in fixed blocks (co-09) rather than contiguous per-request regions -- fixed blocks let two requests reference the SAME block for a shared prefix (co-10) instead of each needing its own identical contiguous memory. Without paging, continuous batching still eliminates idle-slot waste, but it cannot additionally deduplicate shared prefixes, leaving real capacity on the table.

Chunking bounds the WORST-CASE stall to one chunk's cost instead of the whole prefill's cost, but it does not eliminate the tension entirely -- there is still a real trade between chunk size (smaller chunks bound the stall more tightly but add scheduling overhead and slightly delay the new request's own TTFT) and how aggressively prefill is interleaved with decode. Co-17's underlying trade-off is durable; chunking is a mechanism that makes the trade's worst case much more tolerable, not a way to make the trade disappear.

A hosted API bills strictly per token actually consumed -- there is no such thing as an "idle" API hour costing money. A self-hosted GPU, by contrast, is a fixed-cost resource that bills for every hour it exists, whether it served one token or a million during that hour. This asymmetry means the self-hosted side of the comparison is extremely sensitive to how much of its billed time is genuinely productive, while the hosted-API side is not sensitive to utilization at all -- which is exactly why plugging in a full-load throughput number for the self-hosted side, when real traffic is spiky, systematically favors self-hosting in a way that does not survive contact with real operations.

A well-designed preemption policy evicts occasionally, under genuine cache pressure, to admit specific higher-priority or more-urgent work -- the cost (recomputed progress) is real but bounded and purposeful. Starvation is what happens when a scheduling rule systematically and repeatedly disadvantages the SAME class of request with no bound -- not an occasional, purposeful trade but a structural exclusion. The fix for starvation (a fairness guarantee, co-13) and the fix for preemption thrashing (a minimum-progress guard, co-14) are the same shape of fix -- both bound how much any one class of request can be repeatedly disadvantaged -- because both failure modes are the identical underlying problem: an unbounded, repeated cost imposed on the same victim.

Co-01's three properties are specifically about RUNNING a model that already exists, with weights already fixed -- they say nothing about how good the model's answers are, how its weights were produced, or how a chain of calls is orchestrated into an agent. Everything this course teaches (the KV cache, batching, scheduling, capacity planning, deployment) follows directly from those three serving-specific properties; none of it changes if you swap in a better-fine-tuned model or a smarter prompt. That is exactly the boundary this course's scope guard draws: if changing the question changes co-01's three properties, it's out of scope; if it doesn't, it's this course's job.