My First GPU-Backed LLM Inference Run

I have been reading plenty about inference systems: KV cache, PagedAttention, prefill, decode, batching, and disaggregated serving. But reading about the machinery is different from running it and watching the machine do work.

This post is the first in the series that closes that gap. The goal was intentionally small: run one open-source model on a GPU, record what environment I actually got, generate 100 tokens, and capture enough numbers to make the next experiment less hand-wavy.

This is not a benchmark. It is a first measured smoke test.

The Setup

I used Google Colab as the GPU machine. Google Colab is the easiest way to get hands-on access to a T4 GPU for free.

The model was:

TinyLlama/TinyLlama-1.1B-Chat-v1.0

I started with TinyLlama for a practical reason: it is ungated on Hugging Face and small enough to fit on the GPU Colab usually gives me. I originally tried a Llama 3.2 model, but that repository requires Hugging Face access approval and authentication. That is not the problem I wanted to solve in this first run. TinyLlama is also Llama-shaped, which keeps the architecture question relevant.

The serving engine was vLLM. I used vLLM here because I want the learning path to move toward real inference infrastructure, one layer down from a high-level text generation API.

Why an Inference Engine?

The simplest way to expose a model is to load it with PyTorch or Transformers and put a FastAPI endpoint in front of it. That is useful for proving that a model can produce text, but it does not answer the systems questions that show up when many requests share the same GPU.

An inference engine is the runtime layer that manages that pressure. It is responsible for things like batching requests, scheduling prefill and decode work, managing KV-cache memory, choosing attention kernels, streaming tokens, and keeping GPU utilization high without blowing up latency.

That is why this run starts with vLLM instead of a hand-rolled API wrapper. vLLM, SGLang, TensorRT-LLM, and llama.cpp are examples of inference runtimes with different design points. They are not just text generation APIs. They are where serving behavior lives: memory allocation, queueing, token scheduling, kernel choices, and tradeoffs between throughput and latency.

For this first post, I only used vLLM’s offline LLM.generate API. That is still the shallow end of the pool, but it puts the experiment on the right runtime layer. The next step is to run vLLM as an OpenAI-compatible server and measure streaming behavior directly.

Step 1: Check the GPU

Before installing anything, I checked what Colab assigned:

$ nvidia-smi

NVIDIA-SMI 580.82.07
Driver Version: 580.82.07
GPU: Tesla T4
Memory: 0 MiB / 15360 MiB

This matters because the output of a model run is not portable without the hardware context. A 100-token generation on a T4, L4, A100, or H100 means very different things.

At this point I only captured the hardware and Python version. I left PyTorch and vLLM version checks until after the install/restart, so the model process starts from a clean runtime.

import sys

print(f"Python: {sys.version.split()[0]}")

The important habit is not the exact code. The habit is to record the runtime before believing the result.

For this first run, nvidia-smi was enough for GPU and VRAM measurements. If this becomes a repeatable Python measurement harness later, I will use nvidia-ml-py for NVIDIA GPU telemetry and psutil for CPU/process telemetry. I am not installing either yet because the first version does not need them.

Step 2: Install vLLM

The install command was:

pip install -q -U uv
uv pip install --system vllm --torch-backend=auto
pip uninstall -y -q torchaudio || true

The --torch-backend=auto flag is not decoration. vLLM and PyTorch wheels are tied to CUDA backends. In Colab, the assigned runtime can change, and hard-coding a backend can install a wheel that imports cleanly until it looks for the wrong CUDA runtime library.

I hit that once already: vLLM tried to load libcudart.so.13, but the Colab runtime did not provide it. Letting uv choose the torch backend avoids turning the first inference experiment into a CUDA wheel debugging session.

I removed torchaudio after installing vLLM because this text-generation run does not use audio, and a mismatched audio wheel can break imports.

One Colab-specific wrinkle: I ran the vLLM code as a normal Python process:

python phase0_vllm_run.py

vLLM 0.28 initializes CUDA and distributed runtime machinery even for a single-GPU run. In that path it can call sys.stdout.fileno(). Colab’s captured notebook stream does not always provide a real file descriptor, which can fail with UnsupportedOperation: fileno. Launching a small script gives vLLM the process boundary it expects.

Step 3: Write the Runner

The runner keeps model loading, generation, timing, and result capture together:

import subprocess
import time

from vllm import LLM, SamplingParams

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

load_start = time.time()
llm = LLM(
    model=model_name,
    dtype="half",
    max_model_len=1024,
    gpu_memory_utilization=0.8,
    enforce_eager=True,
    disable_log_stats=True,
)
load_seconds = time.time() - load_start

prompt = "KV caching helps inference servers because"
sampling = SamplingParams(max_tokens=100, temperature=0.7)

generate_start = time.time()
outputs = llm.generate([prompt], sampling)
generate_seconds = time.time() - generate_start

There are a few choices packed into those lines.

dtype="half" tells vLLM to use FP16 weights where possible. That keeps memory lower than FP32 and is normal for this kind of small GPU smoke test.

max_model_len=1024 caps the context length. I did not need a long-context run yet, and keeping the maximum sequence length small reduces memory pressure.

gpu_memory_utilization=0.8 tells vLLM how much of the GPU memory it is allowed to plan around. This matters because vLLM pre-allocates and manages memory for serving, including KV-cache blocks.

enforce_eager=True disables CUDA graph capture and torch compilation. That is not the fastest serving configuration, but it keeps this first Colab run simpler and avoids spending the first experiment debugging compile and graph-capture behavior.

Does the Model Need to Be Written for vLLM?

This was my first architecture question after the run. vLLM printed:

Resolved architecture: LlamaForCausalLM

That line matters. vLLM does not need every individual checkpoint to be written specifically for it. It needs support for the model architecture.

TinyLlama works because its Hugging Face config maps to LlamaForCausalLM, and vLLM already has an execution path for that architecture. The model repository provides weights, config, tokenizer files, and generation settings. vLLM provides the serving runtime: model execution, batching, attention kernels, and KV-cache management.

PagedAttention is part of the serving engine. The checkpoint does not say “use PagedAttention.” vLLM decides how to store and schedule KV-cache blocks at runtime.

The practical rule is:

supported architecture + compatible weights/tokenizer = vLLM can usually run it
unsupported architecture or custom layers = vLLM needs model support

That distinction helped me separate the model artifact from the serving engine. TinyLlama was not built for this notebook. It worked because it is Llama-shaped, and vLLM knows how to serve that shape.

Step 4: Generate 100 Tokens

The generation prompt was deliberately boring:

prompt = "KV caching helps inference servers because"
sampling = SamplingParams(max_tokens=100, temperature=0.7)
outputs = llm.generate([prompt], sampling)

The prompt was chosen because it points at the system concept I am trying to understand. The model’s answer was not the important result. The important result was that a serving engine loaded weights, accepted a prompt, generated tokens, and produced measurable runtime behavior.

What I Measured

The saved result is intentionally small:

Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
Load time: 102.7s
Generation time: 3.97s
Generated tokens: 100
Throughput: 25.2 tok/s

I am treating this as a successful smoke test, not a benchmark. The load path includes model download/cache behavior and vLLM initialization. The generation number is a single offline request, not a serving latency distribution.

What I Had to Fix

Four things broke before the first clean run.

First, I installed a vLLM/PyTorch combination that expected a CUDA runtime library that Colab did not have. That is why I switched to:

uv pip install --system vllm --torch-backend=auto

Second, I tried to load a gated Llama model from Hugging Face without authenticating. The serving stack was working, but the model download failed with a 401. That is why the first run now uses TinyLlama.

Third, torchaudio was compiled for a different CUDA version than the PyTorch wheel selected for vLLM. This run does not use audio, so the simplest fix was to remove torchaudio instead of trying to align an unused package.

Fourth, running vLLM directly inside Colab’s notebook process hit UnsupportedOperation: fileno. That was not a model problem or a CUDA problem. It was a process/runtime boundary problem: vLLM expected normal stdout behavior while the notebook kernel was intercepting output. Running the same code through python phase0_vllm_run.py is the cleaner Colab path.

None of these failures were deep. But they are exactly the sort of setup details that disappear from polished inference papers and then eat the first few hours of hands-on work.

What This Run Does Not Measure Yet

This run used offline LLM.generate from a Python process, not the OpenAI-compatible vLLM server with streaming responses.

That means I measured:

total generation elapsed time
generated token count
aggregate throughput
basic VRAM movement

I did not yet measure:

TTFT: time to first token
TPOT: time per output token after the first token
p95/p99 latency
goodput under an SLO
KV-cache hit rate
memory fragmentation

That distinction matters. A first model run is useful, but it is not a serving benchmark. The next step is to run vLLM as a server, send a streaming request, and record TTFT and TPOT directly.

The Systems Lesson

The model output made a common mistake: it described KV cache like a distributed key-value store of input/output pairs. That is not what KV cache means in Transformer inference.

In this context, KV cache is cached attention state. During prefill, the model processes the prompt and stores key/value tensors for prior tokens. During decode, each new token can attend over those cached tensors instead of recomputing the whole prefix.

That is why the systems questions become concrete:

Where is the KV cache stored?
How much memory does it consume?
When does it grow?
Can multiple requests share it?
What happens when it does not fit?
Can prefill and decode run on different machines without too much transfer cost?

This is the bridge from “I ran a model” to inference infrastructure.

Next

The next run should use the vLLM OpenAI-compatible server and a streaming client. That will turn this smoke test into a real latency measurement:

TTFT: request start -> first streamed token
TPOT: average gap between streamed output tokens

After that, the next useful experiment is PagedAttention as a memory allocator: compare naive contiguous KV-cache reservation against fixed-size paged blocks under variable prompt and output lengths.