Reference
Performance & Troubleshooting
Tuning knobs for when things are slow, and fixes for the errors you'll hit most often.
Performance tuning
- GPU layer offloading (
-nglin llama.cpp) — if the whole model doesn't fit in VRAM, offload only some transformer layers to GPU and leave the rest on CPU. More layers on GPU = faster, until you run out of VRAM. - Context length vs. memory — every doubling of context roughly doubles KV cache size at a given batch size. Don't request a bigger context window than you need.
- Quantization level — drop from Q5/Q6 to Q4_K_M if memory constrained; the quality loss is usually smaller than expected, especially for straightforward tasks.
- Batch size / concurrent requests — irrelevant for single-user chat, critical for a server — this is the whole reason vLLM exists over naive serving.
- Speculative decoding — an advanced technique (supported by llama.cpp and vLLM) where a small "draft" model proposes several tokens and the large model verifies them in one pass, often 1.5–2x faster with no quality loss.
Troubleshooting
"CUDA out of memory"
Drop to a smaller quantization, reduce --max-model-len/-c context size, or offload fewer layers to GPU.
Extremely slow generation on CPU
Expected — CPU inference is 10–50x slower than GPU for the same model. Use a smaller model or get GPU offload working.
Model output looks garbled or repetitive
Check you're using the correct prompt template for that model — chat models expect a specific format wrapping user/assistant turns (tokenizer.apply_chat_template in transformers, or automatic in Ollama/llama-server when using the right GGUF).
Ollama model won't load / "model not found"
Run ollama list to confirm the exact tag; tags are case-sensitive and versioned.
vLLM won't start
Almost always a CUDA/driver version mismatch or insufficient --gpu-memory-utilization headroom; check nvidia-smi first.