Acceleration backend#
The hardware path the inference engine computes on — Metal on Apple Silicon, CUDA on NVIDIA, ROCm or Vulkan on AMD, plain CPU otherwise. It decides which engines can run here at all.
Benchmarking#
Runs a short generation on a small installed model and measures the throughput, then rescales every prediction to your machine's real effective bandwidth. Takes about a minute.
Context length#
How much conversation the model holds at once, in tokens — roughly 0.75 words each. Longer context costs memory, so pick the smallest that suits your work.
Continuous batching#
Slotting new requests into a running batch as older ones finish, rather than waiting for the whole batch to drain. Keeps a busy GPU from idling between requests.
Fit verdict#
Whether the model fits alongside everything else you're running. Comfortable means room to grow; tight means a long conversation could push it over; too big means it won't fit in fast memory.
GGUF#
The file format most local models ship in — weights, quantization, tokenizer and metadata in a single file. A tool that reads GGUF will generally read any GGUF.
Inference#
Running a trained model forward: input in, output out. Not a piece of software — it's the process every engine and runtime exists to carry out.
Inference engine#
The software that does the actual maths — kernels, hardware backends, quantized arithmetic, memory layout. llama.cpp and MLX are engines; Ollama is a runtime built over one.
KV cache#
Working memory for the conversation so far, so the model doesn't recompute it for each new token. It grows with context length and comes out of the same budget as the weights.
Measured vs estimated#
Estimates come from your chip's rated bandwidth, which is a ceiling no machine quite reaches. A short real benchmark replaces it with what your machine actually does.
Memory bandwidth#
How fast your machine can read from memory. Generating text means reading the whole model for every token, so bandwidth — not GPU compute — sets the speed ceiling.
Memory needed#
The model's weights, plus the KV cache for your chosen context length, plus about a gigabyte of runtime overhead. All of it has to stay resident while the model is loaded.
Mixture of experts#
An architecture that holds many parameters but activates only a fraction of them per token — so it costs the memory of a large model and runs nearer the speed of a small one.
Offloading#
Splitting a model between GPU and system memory when it won't fit in the fast one. It runs, but every layer left behind drags the whole generation onto the slower path.
Paged attention#
Handing out KV cache in small blocks as a conversation actually grows, instead of reserving the maximum up front. It's how a serving engine fits far more sessions on one card.
Parameters#
The learned numbers a model is made of — the 8 in "Llama 8B" means eight billion. They set both the memory bill and the speed ceiling, because every one is read again for every token.
Prefill and decode#
The two phases of an answer. Prefill reads your prompt in parallel and sets the wait before the first word; decode writes the reply one token at a time and sets tokens per second.
Quantization#
How many bits each parameter is stored in. Q4_K_M is 4-bit and about a quarter the size of the original with very little quality lost — which is why it's the usual pick.
Runtime#
The layer that manages running a model: downloading weights, choosing settings, and serving an API. Ollama is one, wrapped around an inference engine that does the actual maths.
Runtime manager#
The layer above a runtime: it reads the hardware, works out which models and backends suit it, and configures the rest. It's the category ModelFit is in.
Score#
Quality and speed combined into one 0–100 number, weighted for the objective you picked. It's for ranking models against each other on this machine, not an absolute rating.
Serving engine#
A runtime tuned for many simultaneous users rather than one. vLLM and SGLang are the common ones, and the tricks they use only pay off under real concurrency.
Tokens per second#
Generation speed, in word-fragments per second. Around 10 keeps pace with reading; 30 or more feels immediate and is what coding and agent work want.
Unified memory#
On Apple Silicon the CPU and GPU share one pool at full speed, so the whole machine's RAM is available to a model — minus what macOS and your other apps are using.
VRAM#
Memory on the graphics card itself. On a machine with a discrete GPU this is the budget that matters — system RAM sits behind a much slower bus and can't substitute for it.
No term matches that. Try a broader word — the definitions are searched too, not just the titles.