https://gen-ai.fyi/tags/pagedattention/ Recent content in Pagedattention on Gen-AI Hugo -- gohugo.io en Sun, 27 Sep 2026 00:00:00 +0000 https://gen-ai.fyi/one-layer-down/pagedattention-kv-cache-allocator/ Sun, 27 Sep 2026 00:00:00 +0000 https://gen-ai.fyi/one-layer-down/pagedattention-kv-cache-allocator/ My first GPU-backed run used vLLM’s offline LLM.generate API. The second run put vLLM behind its OpenAI-compatible server and measured streaming TTFT and TPOT. Those experiments showed the runtime from a caller’s point of view. They did not show how serving engines manage memory. To understand the memory problem, I first needed to follow one request through generation: how the model reads its prompt, produces each new token, and retains useful state from earlier tokens in that same request.