MMLU (Massive Multitask Language Understanding)
Covers 57 subjects across STEM, humanities, and social sciences. A high score (70%+) indicates strong broad encyclopedic knowledge and few factual hallucinations.
Performance & Evaluation
Local AI Benchmarks & Model Comparison.
A benchmark number only matters if you know what was tested and how the model behaves on real workstation hardware. Below is the verified publisher benchmark comparison for the top 5 models recommended for Loco.
Official Publisher Data
Scores shown are compiled directly from technical reports published by Meta, Alibaba Cloud, DeepSeek, Mistral AI, and Google DeepMind on base and instruction-tuned checkpoints.
Official Benchmark Matrix: Top 5 Local Models
Scores represent standard evaluation protocols (0-shot to 8-shot as documented by publishers). Higher percentages indicate superior capability in that domain.
| Model Variant | Developer | MMLU (General QA) |
GSM8K (Math 8-shot) |
HumanEval (Python Code) |
MATH / MATH-500 (Competition Math) |
Context Window | Q4_K_M RAM Fit | Primary Superpower |
|---|---|---|---|---|---|---|---|---|
| Llama 3.1 8B Instruct | Meta AI | 73.0% | 84.5% | 72.6% | 51.9% | 128,000 tokens | 8 GB System RAM | All-round conversation & JSON formatting |
| Qwen 2.5 7B Instruct | Alibaba Cloud | 74.2% | 88.3% | 79.9% | 75.5% | 128,000 tokens | 8 GB System RAM | Coding champion & structured workflow tool use |
| DeepSeek-R1-Distill-7B | DeepSeek AI | 70.8% | 89.2% | 78.7% | 92.8% | 128,000 tokens | 8 GB System RAM | Step-by-step reasoning & mathematical proofs |
| Mistral 7B Instruct v0.3 | Mistral AI | 62.7% | 52.2% | 40.2% | 38.0% | 32,768 tokens | 8 GB System RAM | Ultra-fast token generation & function calling |
| Gemma 2 9B Instruct | Google DeepMind | 71.3% | 79.0% | 51.8% | 42.6% | 8,192 tokens | 12 GB System RAM | Factual accuracy, prose nuance & dense recall |
Understanding the Tests
What These Benchmarks Actually Measure
Don't look at scores in isolation. Different benchmarks test completely different cognitive and generative domains.
HumanEval (Python Code Synthesis)
164 hand-crafted programming problems evaluated by functional unit tests. Models scoring near 80% (like Qwen 2.5) can reliably write clean code, debug scripts, and build modules in Loco Code.
GSM8K (Grade School Math 8K)
Multi-step arithmetic and word problem solving. Demonstrates whether a model can perform basic chain-of-thought calculation without losing context midway through.
MATH / MATH-500 (Competition STEM)
Challenging high school and university mathematics competition problems. DeepSeek-R1 Distill's 92.8% on MATH-500 represents groundbreaking automated reasoning capability in an open model.
Local Execution Reality
The Trade-Off: Accuracy vs. Tokens Per Second
Higher benchmark scores are useless if the model runs at 1 token per second on your workstation.
Memory BandwidthInference speed on CPU is strictly bottlenecked by system RAM bandwidth (DDR4 vs DDR5).
GPU OffloadingEven a modest 6 GB or 8 GB VRAM GPU will accelerate token velocity by 3x–5x by offloading transformer layers.
Context OverheadFull 128k context windows require substantial KV-cache memory. Loco allows you to dial context size from 2k to 32k+ dynamically.
Quantization (Q4_K_M)4-bit Medium quantization preserves 98.5%+ of full FP16 benchmark performance while slashing memory footprint by 70%.