Shlok Vaishnav

mini-vLLM

A from-scratch LLM inference engine. It builds the two systems ideas behind vLLM: continuous batching and a paged KV-cache. It serves an OpenAI-compatible API, and it is checked token for token against Hugging Face transformers.

Code

325 tok/s
mini-vLLM with all 8 prompts in one batch, on a T4 GPU
42 tok/s
Hugging Face generate(), one prompt at a time, same prompts
50 of 50
GPU tests pass, checked against Hugging Face
4× smaller
weights with INT4, at 0.98 cosine similarity

How it works

Continuous batching

Standard batching waits for a full batch, then releases the whole batch when the slowest sequence finishes. That wastes GPU time whenever sequences differ in length. Continuous batching runs a decode step every tick. New sequences join, and finished ones leave, without stalling the rest. The scheduler decides at each step which sequences run, which wait, and which get preempted under memory pressure. A preempted sequence has its blocks freed, and it is re-prefilled from its prompt and the tokens generated so far when it is admitted again.

Paged KV-cache

The key and value tensors that every attention layer accumulates grow with sequence length. A contiguous buffer per sequence fragments memory and caps the batch size. The paged cache works like virtual memory in an operating system: a fixed pool of physical blocks, each holding a set number of token slots, and a block table per sequence that maps positions to blocks. Sequences of very different lengths share one pool without fragmentation. On CUDA, a custom Triton kernel does the paged attention.

The pieces

  1. API layer. POST /v1/completions in FastAPI, with streaming.
  2. AsyncLLMEngine. One background task steps the engine and sends tokens to each request.
  3. LLMEngine. A scheduler that admits and preempts at every step, and a block manager that owns the pool of physical blocks.
  4. Runner. A dense KV cache for CPU and tests, or the paged runner with the Triton kernel for Llama models on CUDA.
  5. Model and weights. TinyLlama-1.1B and GPT-2.

Results

All GPU numbers come from a Kaggle T4 (16 GB) with TinyLlama-1.1B in fp16. The benchmarks were last re-run on 8 October 2026.

Throughput against Hugging Face

8 prompts, 32 output tokens each. With all 8 prompts in one batch, mini-vLLM matches batched Hugging Face within the noise from run to run. With a batch size of 4, only half the prompts run at a time, so it is slower. Compare rows within one run, not across runs: the Hugging Face baselines moved by about 15% between the last two runs. The comparison with vLLM itself is opt-in in the benchmark script, and it was not run.

Under concurrent load

Throughput rises up to 8 concurrent requests, which is the server's default maximum batch size, and then stops growing.
Time to first token stays low up to 8 requests. At 16, requests queue and it rises sharply. The server was warmed up before the test.

Quantization

Model size

Decode speed

The weights are quantized per channel, and they are converted back to fp16 before each matrix multiplication. The memory saving is real: INT8 halves the size and INT4 cuts it to a quarter, with a cosine similarity of 1.0000 and 0.9785 to fp16. Decode gets slower, though, because the conversion runs on every step. A real speed-up needs a fused low-precision kernel, which is not implemented. Size and quality are exact. Speed varies by about 10% from run to run.

Correctness

Decoding is checked token for token against Hugging Face transformers. For quantization, the check is a calibrated error bound. All 50 tests pass on the GPU, with none skipped. On a CPU, 39 pass and 11 are skipped, because the Triton kernel and the real-model paged decode need CUDA.

Test suites
SuiteTestsWhat it shows
Dense runner3 of 3Single, batched and continuous-batch decode match model.generate() exactly.
Block manager, scheduler, paged attention23 of 23Block allocation, preemption under memory pressure, and paged attention (with grouped-query heads) match dense attention.
Paged runner on a real model4 of 4The full paged decode path matches Hugging Face on the GPU. Batched sampling equals per-sequence sampling.
Quantization6 of 6INT8 and INT4 round trips and model-level error bounds hold.
API server8 of 8Streaming equals non-streaming. Concurrent requests match solo runs. Bad input is rejected. Stop strings cut the output.
Engine robustness6 of 6Recompute after preemption matches an unconstrained run. Aborts and engine failures are handled.

What these numbers do not show