Local LLM fundamentals

Parameters: the B in 7B#

When a model is called Llama 3.1 8B or Qwen 32B, that number is its parameter count — 8 billion, 32 billion. A parameter is one number the model learned during training. Running the model means multiplying your input through all of them.

Two consequences follow, and almost everything else in this article is a detail of one of them:

• Parameters have to be in memory. All of them, for the whole time the model is loaded.
• Parameters have to be read to produce each word. Every single token the model writes requires a pass over the weights.

So parameter count sets both your memory bill and your speed limit. A model twice the size needs roughly twice the memory and runs roughly half as fast on the same hardware. In exchange it is usually more capable — better at reasoning, less prone to losing the thread, more reliable on code.

Stored at their native training precision (16-bit, two bytes each), 8 billion parameters occupy about 16 GB. That is already more than many laptops can spare, which is why nobody runs models that way locally.

Both consequences, on one model:

Llama 3.1 8B at 16-bit precision

  8,000,000,000 × 2 bytes  =  16 GB of weights

  in memory    16 GB, for as long as it is loaded
  per token    16 GB, read start to finish
  at 30 tok/s  480 GB read every second

That last line is the one to keep. Generating at a readable pace means moving most of a gigabyte every few milliseconds, over and over — which is why the number that limits you turns out to be memory bandwidth rather than anything about the GPU.

Quantization#

Quantization stores each parameter in fewer bits. Instead of a 16-bit number per weight, you keep a 4-bit or 5-bit approximation plus a small scaling factor shared across a block of weights. The model gets three to four times smaller and, because there is less to read, correspondingly faster.

You lose some accuracy. The important and slightly surprising fact is how little you lose down to about 4 bits, and how quickly it falls apart below that:

Read the names left to right: Q4_K_M is 4-bit, using the K-quant scheme, in its Medium variant (_S and _L are the smaller and larger siblings). The _0 and _1 suffixes on older types like Q8_0 are earlier, simpler schemes.

The memory equation#

What a loaded model actually costs is three things added together:

total memory  =  quantized weights
              +  KV cache (grows with context length)
              +  runtime overhead (~0.5–1 GB)

For an 8B model at Q4_K_M with an 8k context window, that is roughly:

The weights are fixed the moment you pick a model and a quantization. The overhead is small and roughly constant. The KV cache is the part under your control, and the part people forget.

Context length#

Context length is how much text the model can hold in mind at once — your prompt, any documents you pasted, and everything said so far in the conversation. It is measured in tokens, which are word fragments: roughly 0.75 words per token in English, so 8k tokens is about 6,000 words.

A model has a maximum context it was trained for, but you choose what to actually allocate when you load it. That choice costs memory, because of the KV cache.

Pick the smallest context that fits your work. Chat and quick questions are comfortable at 4k–8k. Reading a long document or a large source file wants 32k or more. Allocating 128k "just in case" can cost more memory than the model itself, and it will not make short conversations any better.

For an 8B model, the price of that choice looks like this:

context      KV cache      vs 4.9 GB of weights
  4k          0.5 GB       a tenth of the model
  8k          1.0 GB       a fifth
 32k          4.0 GB       most of a second model
128k         16.0 GB       three times the model

Which is the whole argument in one table. Going from 8k to 128k costs you 15 GB — enough memory to have run a far better model instead.

The KV cache#

As the model reads your text it computes, for every token, a pair of vectors called the key and the value. It keeps them so it does not have to recompute the whole conversation each time it writes one more word. That store is the KV cache, and it grows linearly with the number of tokens in play.

Its size is set by the model's architecture:

KV bytes = 2 × layers × kv_heads × head_dim × bytes_per_value × tokens

The practical upshot is that the same context length costs wildly different amounts on different models. Modern architectures use grouped-query attention, which shares keys and values across attention heads and cuts the cache by four to eight times compared with older designs. It is entirely normal for a well-designed 14B to need less context memory than a badly-aged 7B.

Unified memory vs VRAM#

Where the model has to fit depends on the shape of your machine, and there are two shapes.

Discrete GPU (most PCs)

The graphics card has its own memory, and that pool is the one that counts. A 24 GB card with 128 GB of system RAM behind it still gives you roughly 24 GB to work with — minus one or two for the display and the desktop. System RAM does not help; it is on the far side of a slow bus.

Unified memory (Apple Silicon)

CPU and GPU share one pool of memory at full speed, so a 32 GB Mac has something close to 32 GB available for a model. macOS reserves a share for itself and everything else you have open, so plan on about 70% being genuinely usable — roughly 22 GB on that machine. This is the reason modest-looking Macs run models that embarrass much more expensive gaming PCs: it is not that the GPU is faster, it is that the memory pool is bigger.

Drawn out, the difference is the whole story:

DISCRETE GPU (most PCs)

  ┌───────────────┐     ┌───────────────┐
  │  128 GB RAM   │ ──▶ │  24 GB VRAM   │
  └───────────────┘     └───────────────┘
      slow bus           the only pool
                         that counts

UNIFIED MEMORY (Apple Silicon)

  ┌─────────────────────────────────────┐
  │     32 GB, shared by CPU + GPU      │
  └─────────────────────────────────────┘
          about 70% usable  =  ~22 GB

The PC in that picture has five times the total memory and less of it available to a model. Adding system RAM to the left-hand machine changes nothing at all.

Offloading and the cliff#

When a model does not fit in GPU memory, runtimes will split it: some layers on the GPU, the rest in system RAM on the CPU. This is offloading, and it does work — the model loads and answers.

It is also a cliff rather than a slope. CPU memory bandwidth is roughly ten to twenty times lower than GPU memory bandwidth, and every token has to traverse every layer, so the slow layers dominate the total. Pushing 20% of a model to the CPU does not cost you 20% of your speed — it can easily cost you 70%.

The arithmetic is worth seeing, because the result is so unintuitive. Take a 32-layer model and a GPU 15 times faster than the CPU path:

all 32 layers on GPU     32 × 1              =  32 time units
26 on GPU, 6 on CPU      26 × 1  +  6 × 15   = 116 time units

Six layers — under a fifth of the model — and it now takes 3.6 times as long per token. You kept 81% of the model in fast memory and lost 72% of your speed.

A model that fits entirely in fast memory at a lower quantization will nearly always beat the same model at a higher quantization spilling onto the CPU. When you have to choose, choose to fit.

Memory bandwidth#

Here is the fact that reorganizes everything else: generating text is memory-bandwidth-bound, not compute-bound.

To produce one token, the hardware must read every active weight out of memory and multiply your one-token-wide input by it. That is a tiny amount of arithmetic against an enormous amount of data movement. The arithmetic units spend most of their time idle, waiting for weights to arrive. The bottleneck is the pipe, not the engine.

Which is why bandwidth — measured in gigabytes per second — predicts local model speed better than any teraflops figure:

Note how well this explains the CPU offloading cliff, and how badly raw GPU compute would have explained any of it.

Tokens per second#

Because each token requires reading the weights once, you can estimate generation speed with arithmetic rather than folklore:

tokens/sec  ≈  memory bandwidth ÷ model size in memory  ×  efficiency

Efficiency lands between 0.6 and 0.85 on real hardware — no memory system achieves its rated peak. An 8B model at Q4_K_M (4.9 GB) on an M4 Pro (273 GB/s) works out to 273 ÷ 4.9 ≈ 56 theoretical, so expect somewhere around 35–45 tokens per second in practice. That estimate is usually within 20% of reality, which is close enough to decide whether a model is worth downloading.

What the numbers feel like:

• Under 5 tok/s — painful. Fine for a batch job you walk away from.
• 5–10 tok/s — usable. Around adult reading speed; you wait, but not unbearably.
• 10–30 tok/s — comfortable conversation. Output arrives faster than you read.
• 30+ tok/s — fast enough for coding assistance and agent loops, where you are waiting on whole responses rather than reading along.

Two different speeds

Generation is only half the story. Processing your prompt — the pause before the first word appears — is compute-bound, not bandwidth-bound, because the whole prompt goes through at once. This is why pasting a long document produces a long silence and then fluent output. If time-to-first-token matters to you, that is a different measurement from tokens per second.

Measured vs estimated#

Spec-sheet bandwidth is a ceiling, not a promise. What your machine actually achieves depends on thermals, what else is running, the runtime's build options, and how well the model's shape maps onto your hardware. Two identical laptops can differ by 20%; one of them plugged in and one on battery can differ by more.

So an estimate answers "is this plausible?" and a measurement answers "what will I get?". Running a short real generation on a small model and extrapolating from the observed throughput replaces the spec figure with your machine's actual effective bandwidth, and every prediction downstream of it gets sharper. It costs about a minute. It is worth the minute.

What that minute actually changes:

M2 Pro running an 8B at Q4_K_M (4.9 GB)

  spec sheet                200 GB/s
  estimate, 55% of spec     110 GB/s  →  22 tok/s
  measured on the machine    96 GB/s  →  20 tok/s

Close, in this case — and you only know it is close because you measured. On a thermally throttled laptop, or one with real work running alongside, the same two rows can differ by half.

Mixture of experts#

Some models have two parameter counts, written like 30B-A3B: 30 billion total, 3 billion active per token. These are mixture-of-experts models. The network is divided into many expert sub-networks, and a router picks a small handful for each token.

This decouples the two consequences from the top of the article, and you have to reason about them separately:

• Memory follows the total. All 30B of weights must be resident — any expert might be needed for the next token.
• Speed follows the active count. Only 3B of weights are read per token, so it generates at something like 3B pace.

The result is a model that is expensive to hold and cheap to run — excellent on a large-memory machine, and impossible on a small one no matter how fast it would have been.

The two numbers pull in opposite directions:

Qwen3 30B-A3B

  memory  ████████████████████  30 B held
  speed   ██                     3 B per token

You pay for the long bar and you are served at the speed of the short one. If you have unified memory to spare, these are often the best quality-per-second available to you.

GGUF and model files#

GGUF is the file format the local ecosystem settled on. One file holds the quantized weights, the tokenizer, and the metadata a runtime needs — context limits, architecture, prompt template — so there is nothing to assemble and no Python environment to get wrong. The quantization is usually right there in the filename: Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf.

You will also meet safetensors, which is what unquantized models are published as on Hugging Face, and MLX, Apple's own format for its silicon. For running a model locally today, GGUF is the one you will actually download.

One naming detail worth knowing: a model tagged Instruct or Chat has been tuned to follow instructions and hold a conversation. A base model has not — it continues text rather than answering questions, and will feel broken if you try to chat with it. Take the instruct variant unless you know why you want otherwise.

Which makes a GGUF filename readable end to end:

Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
│    │         │  │        │      │
│    │         │  │        │      └ format
│    │         │  │        └─────── quantization
│    │         │  └──────────────── tuned to chat
│    │         └─────────────────── parameters
│    └───────────────────────────── model family
└────────────────────────────────── who made it

Every field you need in order to choose is in the name. Nothing has to be opened to find out what it is.

Runtimes#

A runtime is the program that loads the weights and does the maths. Almost everything you will encounter is built on llama.cpp, the C++ inference engine that also gave GGUF its name.

Ollama wraps it in something usable: a background service, a one-command model library, sensible defaults for your hardware, and an HTTP API on localhost:11434 that most local AI tools speak. It is the reason running a model is now ollama run rather than an afternoon of build flags. LM Studio is the equivalent with a graphical interface; llama.cpp directly gives you the most control and the earliest features.

They all read the same GGUF files and hit the same memory and bandwidth walls. Choosing between them is a question of interface, not capability.

What inference actually involves, where the engine ends and the runtime begins, and how the rest of the landscape sorts out, is in LLM runtimes and inference.

Reading a fit verdict#

"Does it fit" is not the same question as "does it load". Your operating system, your browser and your editor were using memory before the model arrived, and they will keep asking for more. Fitting means fitting alongside them.

That is why ModelFit sorts models into three verdicts rather than a yes and a no:

• Runs comfortably — the model plus its KV cache sit well inside usable memory, with room for the conversation to grow and for you to keep working while it runs.
• Runs, but tight — it fits now. A long conversation, a big paste, or a few more browser tabs may push it over, at which point the system starts swapping and speed collapses.
• Too big — it will not fit in fast memory. It may still load by offloading to the CPU, at the speed cost described above.

On a 16 GB Mac — roughly 11 GB usable once macOS and your open apps have taken their share — the same three verdicts fall out of simple addition. Each total includes the same 0.7 GB of runtime overhead:

Comfortable means landing under about 9 GB here; anything up to roughly 10 GB still fits but has nowhere to grow. The 13B is not a worse model than the 8B — it is a worse fit on this particular machine, on this particular afternoon.

Headroom is not caution for its own sake. It is what stops a model that worked this morning from failing this afternoon.

The score#

ModelFit puts a number from 0 to 100 on each model. It is a ranking device for one machine, not a rating of the model — and it is worth knowing exactly what goes into it, because two of the four things you might expect are missing.

It combines two terms, each scaled to 0–1:

score = (quality_weight × quality + speed_weight × speed) × 100

Quality is a curated figure from the model registry, not something measured on your machine — a judgement about how capable the model is, with a separate number for coding. It is rescaled over 5 to 10 rather than 0 to 10, because no model worth listing scores below 5 and stretching the bottom half of the range would make a real one-point quality gap look smaller than a speed difference you would never notice.

Speed is the estimated tokens per second from the section above — measured, if you have run the benchmark. It counts fully up to 20 tok/s and then stops: past comfortable reading speed, faster stops mattering for interactive use, so a model at 60 tok/s gets no more credit than one at 20.

The objective you pick sets the weights, and two other things besides:

Below the speed floor a model is excluded outright rather than scored badly. So is one that will not fit, one whose context overflows, and — under the coding objective — one with no coding capability. A score of 0 means excluded, not bad.

None of which survives a change of machine. The quality half is constant, but the speed half depends on your bandwidth and on which quantization fits in your memory, so the same model scores differently on a laptop and a workstation. Compare scores down a list, never across two computers.

Worth working one through, because the objective does more than nudge the order. Two models on the same machine — a fast average one and a slow good one:

            quality  →  qn      speed     →  sn
model A       7.2    →  0.44    12 tok/s  →  0.60
model B       8.4    →  0.68     6 tok/s  →  0.30

Overall (0.6 / 0.4)          Speed (0.3 / 0.7)
A  0.6×0.44 + 0.4×0.60 = 50  A  0.3×0.44 + 0.7×0.60 = 55
B  0.6×0.68 + 0.4×0.30 = 53  B  0.3×0.68 + 0.7×0.30 = 41

Under Overall the slower, better model wins by three points. Switch to Speed and the order reverses by fourteen. Same machine, same two models, same arithmetic — the objective is not a tiebreaker, it is the question you are asking.

Choosing, in practice#

Everything above compresses into a short procedure:

1. Find your real memory budget. VRAM on a discrete GPU; about 70% of total RAM on Apple Silicon.
2. Subtract your context. Decide the context length you actually need and leave room for its KV cache — one to several gigabytes.
3. Spend the rest on parameters at 4-bit. The largest model that fits at Q4_K_M is usually the best model you can run. Do not go below 4-bit to squeeze in a bigger one.
4. Check the speed it implies. Bandwidth divided by model size, times about 0.7. If that lands under 10 tok/s and you want to converse, step down a size.
5. Measure instead of trusting the estimate. One short benchmark replaces the spec sheet with your machine.

The recurring mistake is optimizing the wrong variable — hunting for a faster GPU when memory capacity was the constraint, or paying for 8-bit precision that buys nothing you can perceive. Memory decides what you can run. Bandwidth decides how fast it runs. Almost nothing else about your machine matters nearly as much.