Measuring TTFT and TPOT with a Local vLLM Server
In the first One Layer Down run, I loaded TinyLlama with vLLM on a Colab T4 and generated 100 tokens. That was useful because it proved the basic path: GPU, CUDA, PyTorch, vLLM, model weights, tokenizer, and one successful generation. But it was not the latency measurement I actually needed. When an application calls a model, it can wait for the full answer and show it all at once, or it can stream partial output as the model generates....