Model format
The runtime must support the model architecture and file format. GGUF is commonly associated with llama.cpp-based runtimes.
Runtimes and tools
Understand the software that runs local models.
An inference runtime loads model weights, manages memory and executes the calculations that turn a prompt into generated tokens.
Choose by use caseA desktop interface, command-line runtime, local API server and multi-user web interface solve different problems. Compare tools by what you need to operate.
Runtime decisions
Check compatibility before features.
Hardware acceleration
CPU instruction support, GPU backends, drivers and offload settings determine which hardware can accelerate inference.
Interface and integration
Decide whether you need a desktop chat, command-line control, a local API, document tools or shared access.