One Layer Down

One Layer Down is my lab notebook for moving below LLM application code into inference infrastructure and agent runtimes.

The posts here start from hands-on work: running open-source models, reading serving papers, wiring small repros, and measuring behavior that is easy to miss in ordinary chat demos. The recurring questions are systems questions: where KV cache lives, how prefill and decode get scheduled, what gets copied or recomputed, how latency changes under load, and what a useful result costs.

Paper notes stay in the Paper Notes section. General AI application essays stay in Posts. This section is for the work that connects those ideas to commands, traces, result manifests, and the parts that broke while trying to make them real.

Measuring TTFT and TPOT with a Local vLLM Server

In the first One Layer Down run, I loaded TinyLlama with vLLM on a Colab T4 and generated 100 tokens. That was useful because it proved the basic path: GPU, CUDA, PyTorch, vLLM, model weights, tokenizer, and one successful generation. But it was not the latency measurement I actually needed. When an application calls a model, it can wait for the full answer and show it all at once, or it can stream partial output as the model generates....

My First GPU-Backed LLM Inference Run

I have been reading plenty about inference systems: KV cache, PagedAttention, prefill, decode, batching, and disaggregated serving. But reading about the machinery is different from running it and watching the machine do work. This post is the first in the series that closes that gap. The goal was intentionally small: run one open-source model on a GPU, record what environment I actually got, generate 100 tokens, and capture enough numbers to make the next experiment less hand-wavy....