A model must fit its weights plus runtime state, context cache and supporting software into available memory. The model-file size is a useful starting point, not the complete requirement.
Start with the weight calculation
A simplified theoretical estimate for weight data is:
parameter count × average bits per weight ÷ 8For example, eight billion parameters at exactly four bits per weight would equal roughly four billion bytes of raw weight data. Real quantised formats can use mixed tensor types and include metadata, so the actual GGUF file size is a better immediate reference than the simplified formula.
Theoretical weight-only examples
| Parameter count | At 4 bits | At 5 bits | At 8 bits |
|---|---|---|---|
| 7B | 3.5 GB | 4.4 GB | 7 GB |
| 8B | 4 GB | 5 GB | 8 GB |
| 14B | 7 GB | 8.8 GB | 14 GB |
| 32B | 16 GB | 20 GB | 32 GB |
| 70B | 35 GB | 43.8 GB | 70 GB |
Interpretation These are arithmetic examples, not download sizes or measured runtime requirements. A specific file can differ because quantisation schemes do not necessarily represent every tensor at one uniform bit width.
What uses memory beyond the weights?
- Context or KV cache: memory used to retain information needed during generation. It grows with context length and depends on model architecture and cache settings.
- Runtime buffers: working memory used by the inference engine.
- Operating system and applications: Windows, browser tabs and other software continue to use RAM.
- GPU offloading: some or all model layers may use VRAM instead of system RAM, depending on runtime and configuration.
- Multimodal components: vision or audio models may load additional projectors or encoders.
Leave usable headroom
Planning rule Do not choose a model simply because its file size is a few megabytes below your installed RAM. A system that constantly swaps memory to disk may become extremely slow or unstable.
For a first attempt, choose a GGUF file comfortably below the memory remaining after Windows and normal applications are running. Then start with a moderate context size and observe actual memory use.
RAM versus VRAM
System RAM is attached to the CPU; VRAM is attached to a discrete GPU. A runtime may execute entirely on the CPU, entirely on supported GPU resources, or split work between them. The amount of VRAM needed therefore depends on how many layers are offloaded, the model, context and runtime configuration.
A GPU with insufficient VRAM can still be useful for partial offloading, but performance must be measured on the actual system. Do not infer tokens per second from memory capacity alone.
A reliable selection process
- Record installed RAM and currently available RAM.
- Record GPU model and usable VRAM, if present.
- Check the exact GGUF file size and quantisation.
- Confirm runtime support for the model architecture.
- Leave capacity for Windows, context and runtime overhead.
- Start with a lower context size.
- Measure memory and response speed with representative prompts.
- Keep the exact model filename and settings with any result.
Loco Free shows system hardware and model-fit guidance, but that guidance remains an estimate. See the Loco model documentation before choosing a large download.
Sources and methodology
- The weight-only table is direct arithmetic using decimal parameter counts and stated bit widths. It is not a Loco benchmark.
- llama.cpp GGUF implementation — tensor types and file structure.
- llama.cpp model documentation — GGUF model operation and conversion.