A field guide to freely downloadable models across ten hardware classes — Raspberry Pi to a half-terabyte Graviton4 box. Quantized GGUF (llama.cpp) as the common denominator; ONNX / vLLM noted where relevant.
Included: models with weights freely downloadable (Hugging Face) and runnable on open-source inference stacks — llama.cpp/GGUF first, plus ONNX Runtime GenAI, Ollama, vLLM. Best 2–3 per class chosen on benchmark quality per GB, community adoption, and measured or estimable throughput on the class hardware. Q4_K_M-class quantization assumed unless noted (MXFP4 for gpt-oss). All t/s figures are single-stream decode; treat as planning numbers — build, context fill and RAM tuning swing results ±2×.
AI datacenter buildout absorbed the memory supply: LPDDR4 up ~7× in a year (Raspberry Pi), contract DRAM forecast +up to 95% in Q1'26 (TrendForce), NVIDIA cut RTX-50 output 20–40% and let the 5090 drift from $1,999 MSRP to $3.7–4.3K street. Pi 5 16 GB went $120 → $205 (Feb) with further rises in April. Consequence: picking a model is now mostly a memory-price question — which is exactly how this report is organized.
Qwen 3.5 (Feb, Apache 2.0, 0.8B→397B) and Qwen 3.6 (Apr: 27B dense + 35B-A3B) · Gemma 4 (Apr 2, first Gemma under Apache 2.0; E2B→31B) · DeepSeek V4 (Apr 24, MIT: Flash 284B-A13B, Pro 1.6T-A49B) · GLM-5.2 (Jun, MIT — top open GPQA) · Kimi K2.7 Code (Jun, Modified-MIT, ~1T) · MiniMax M3 · Kimi K3 (2.8T) weights due Jul 27. Frontier-adjacent quality now ships MIT/Apache, and everything ≤ ~13B active runs on consumer gear.
Each card: hardware definition and Jul-'26 price, the 2–3 models to run, memory footprint at ~4-bit, expected single-stream decode speed on the class hardware (• measured, ≈ estimated), and why. Required anchor classes — Raspberry Pi 4, i9 at 16 GB, RTX 30/40/50 — are C1, C3, C5, C6, C7.
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| Llama 3.2 3B Instruct | ~2.0 GB | • 8–9 (Pi 5) · ~3–4 (Pi 4) | Best-tested small model on Pi; reliable structure/tool output; reading-speed generation on a Pi 5. |
| Gemma 4 E2B | ~2.5–3 GB | ≈ 6–10 (Pi 5) | The only multimodal option in class: image + native audio input, 128K ctx, Apache-2.0. |
| Qwen3.5-4B | ~2.5 GB | ≈ 4–7 (Pi 5) | Highest quality that stays usable; thinking-mode toggle; 119-language coverage. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| gpt-oss-20b (MXFP4) | 12.9 GB file · 14–16 GB RAM | ≈ 10–20 | The 16-GB headline act: o3-mini-class reasoning + native tool use on a stock desktop, no dGPU. |
| Qwen3.5-9B | ~5 GB | ≈ 8–14 | Default dense all-rounder; leaves RAM for the rest of the machine. |
| Gemma 4 12B Unified | ~7 GB | ≈ 6–10 | 2026's laptop model: text+image+video+audio input under Apache-2.0. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| Qwen3.6-35B-A3B | ~21 GB · needs 32 GB | • 12–25 (30+ tuned) | CPU MoE king: GPQA-Diamond 86.0 while reading only ~3B params per token. |
| gpt-oss-20b (MXFP4) | 12.9 GB · fits 16 GB | ≈ 15–25 | The pick at exactly 16 GB; adjustable reasoning effort, agent-grade function calling. |
| Gemma 4 26B (A4B MoE) | ≈16 GB | ≈ 12–20 | Apache MoE with vision input; top-10 open model on Arena (text). |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| Qwen3.5-4B | ~2.5 GB | ≈ 60–90 | 2024-flagship quality in toy VRAM; room for 16K+ ctx even on 6 GB. |
| Qwen3.5-9B / Llama 3.1 8B | ~5 GB (8 GB cards) | • 40–55 | The workhorse tier: chat, RAG, light coding — fully resident on an 8 GB card. |
| Gemma 4 E4B | ~3 GB | ≈ 45–70 | Vision + audio input in ~3 GB; Apache-2.0. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| Qwen3.6-35B-A3B | 21–22 GB (3090) | • 101 (3090) · • 38 (3060 + expert offload) | The best perf-per-dollar trick of 2026: a $200 Ampere runs a 35B at chat speed. |
| Qwen3.6-27B | ~16–17 GB | ≈ 30–35 (3090) | SWE-bench Verified 77.2 — best open dense coder, fully resident in 24 GB. |
| gpt-oss-20b (MXFP4) | 12.9 GB · wants ≥15 GB VRAM | ≈ 150–180 (3090) · • 30 (3060, spills) | Reasoning at interactive speed on 3080 Ti / 3090-class cards. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| gpt-oss-20b (MXFP4) | 12.9 GB | • 225 (4090) | Agentic loops in real time; throughput champion of the 24 GB tier. |
| Qwen3.6-27B | ~16–17 GB | ≈ 38–45 (4090) | Daily-driver coder; 256K ctx stays cheap (hybrid linear attention → small KV). |
| Gemma 4 31B | ≈19 GB | ≈ 33–40 (4090) | MMLU-Pro 85.2 · AIME'26 89.2 — strongest open generalist that fits in 24 GB. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| Qwen3 32B (dense) | ~19–20 GB | • 61 (5090) | The 32-GB payoff: a dense 32B fully in VRAM with ~32K ctx at interactive speed. |
| gpt-oss-120b (MXFP4) | 60.8 GB · experts → DDR5 | ≈ 25–45 (5090 + fast RAM) | o4-mini-class at home; full 128K ctx workable via MoE expert offload. |
| Qwen3.6-35B-A3B | 21–22 GB | ≈ 120–190 (5090) | Plus • 282 t/s on gpt-oss-20b — the fastest single-card local stack, period. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| gpt-oss-20b (MXFP4) | 12.9 GB | • 102 (XTX) | CUDA-free reasoning at ~36% of a 5090's speed for ~22% of its street price. |
| Qwen3.5-9B / Llama 3.1 8B | ~5 GB | • 96 (XTX) | Daily driver on ROCm 7.2 or the zero-setup Vulkan backend. |
| Qwen3.6-35B-A3B | 21–22 GB | ≈ 75–90 (XTX) | 24 GB holds the 2026 default MoE fully resident. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| Qwen3.5-9B / Llama 3.1 8B | ~5 GB | • 35–55 (A770, SYCL) · ≈ 50–80 (B580) | The budget 8B box; B580 numbers vary strongly by stack. |
| Qwen3 14B / Qwen2.5-14B | ~9 GB | • 21 (A770) | Meaningfully smarter than 8B, with ~6 GB headroom for 8K ctx on the A770. |
| gpt-oss-20b (MXFP4) | 12.9 GB → A770 16 GB | ≈ 25–40 | The reasoning tier on a $279 card. |
| Model | Footprint (~Q4) | Est. decode t/s | Why this pick |
|---|---|---|---|
| gpt-oss-120b (MXFP4) | 60.8 GB file · 64 GB+ RAM | ≈ 15–35 | The big-RAM sweet spot: 5.1B active keeps CPU decode interactive; Apache-2.0. |
| DeepSeek V4-Flash | Q3 103 GB → Q8 162 GB | ≈ 4–10 | Frontier-adjacent (GPQA-D 88.1) with a 1M-token ctx under MIT — huge RAM is the cheapest way in. |
| Qwen3.5-122B-A10B | ~70 GB | ≈ 10–20 | Mid-size Apache MoE shaped exactly for many-core, huge-RAM boxes. |
| MODEL | PARAMS (tot / act) | ARCH | TYPE · MODALITY | BEST AT | ORG | CTRY | LICENSE | MIN MEM ~Q4 | KEY BENCHMARKS & NOTES |
|---|---|---|---|---|---|---|---|---|---|
| Llama 3.2 3B | 3B dense | Transformer | instruct · text | edge chat, tools | Meta | US | Llama Community (700M-MAU cap) | ~2 GB | Best-tested Pi-class model; the largest ecosystem of fine-tunes. |
| Gemma 4 E2B | 5.1B / 2.3B eff. | edge-optimized | instruct · text+img+aud | on-device multimodal | Google DeepMind | US | Apache-2.0 | ~2.5 GB | 128K ctx; native audio input; 140+ languages. |
| Phi-4-mini | 3.8B dense | Transformer | instruct · text | CPU / edge reasoning | Microsoft | US | MIT | ~2.5 GB | 128K ctx on a 3.8B; runs on effectively anything. |
| Qwen3.5-4B | 4B dense | hybrid (Gated DeltaNet) | instruct+think · text | small all-rounder | Alibaba (Qwen) | CN | Apache-2.0 | ~2.5 GB | Vendor claim: 4B-class matches prior-gen 72B on many evals. |
| Qwen3.5-9B | 9B dense | hybrid lin-attn | instruct+think · text | 8-GB-class default | Alibaba (Qwen) | CN | Apache-2.0 | ~5 GB | The workhorse for 8–16 GB machines; ~5 GB at Q4. |
| Gemma 4 12B Unified | 12B dense | Transformer | instruct · text+img+vid+aud | multimodal laptop | Google DeepMind | US | Apache-2.0 | ~7 GB | Added Jun '26; MTP drafters give up to ~3× decode. |
| gpt-oss-20b | 20.9B / 3.6B | MoE 32e · top-4 | reasoning · text+tools | agents, function calls | OpenAI | US | Apache-2.0 | 14–16 GB | ≈ o3-mini quality; adjustable reasoning effort; 128K ctx. |
| Gemma 4 26B | 26B / 3.8B | MoE | instruct · text+img | fast consumer / server | Google DeepMind | US | Apache-2.0 | ~16 GB | Top-10 open model on Arena (text). |
| Qwen3.6-27B | 27B dense | hybrid lin-attn | instruct+think · text | coding | Alibaba (Qwen) | CN | Apache-2.0 | 16–17 GB | SWE-bench Verified 77.2 — best open dense coder; 256K ctx. |
| Qwen3.6-35B-A3B | 35B / 3B | hybrid + MoE | instruct+think · text | everyday driver | Alibaba (Qwen) | CN | Apache-2.0 | 21–22 GB | GPQA-Diamond 86.0 at ~3B active; 256K ctx. |
| Gemma 4 31B | 31B dense | Transformer | instruct · text+img | open generalist | Google DeepMind | US | Apache-2.0 | ~19 GB | MMLU-Pro 85.2 · AIME'26 89.2 · #3 open on Arena. |
| gpt-oss-120b | 116.8B / 5.1B | MoE 128e · top-4 | reasoning · text+tools | reasoning at scale | OpenAI | US | Apache-2.0 | 61–66 GB | ≈ o4-mini quality; one 80-GB GPU — or any big-RAM CPU. |
| Qwen3.5-122B-A10B | 122B / 10B | MoE | instruct+think · text | mid-size server | Alibaba (Qwen) | CN | Apache-2.0 | ~70 GB | The Apache ladder step between 35B and the 397B flagship. |
| DeepSeek V4-Flash | 284B / 13B | MoE + CSA/HCA sparse attn | reasoning · text | long-ctx, high value per $ | DeepSeek | CN | MIT | 103 GB (Q3) | MMLU-Pro 86.2 · GPQA-D 88.1 · TerminalBench-2 56.9 · 1M ctx. |
| GLM-5.2 | ~750B-class | MoE | reasoning · text | open-SOTA reasoning | Zhipu / Z.ai | CN | MIT | ~400+ GB | GPQA-Diamond 91.2 — highest open score; server-scale only. |
ALSO NOTABLE (outside these ten classes): DeepSeek V4-Pro (1.6T / 49B act, MIT — SWE-bench Verified 80.6, open-SOTA coder, ~862 GB) · Kimi K2.7 Code (~1T MoE, Modified-MIT, long-horizon coding + vision) · Kimi K3 (2.8T; weights due Jul 27 '26) · MiniMax M3 · Nemotron 3 (open weights + data + recipes) · Mistral Large 3 · Llama 4 Scout (109B / 17B act, Llama license). Benchmark scores are vendor-reported unless noted; community Q4 builds typically land slightly lower.
llama.cpp (GGUF) the lingua franca: CPU (AVX-512/AMX, NEON/SVE), CUDA, ROCm/HIP, SYCL, Vulkan, Metal. Every model in this report ships GGUF (stock + Unsloth dynamic quants). The 2026 flag that matters: --n-cpu-moe / --fit for MoE expert offload.
Ollama / LM Studio llama.cpp frontends; first-party libraries for gpt-oss, Gemma 4, Qwen 3.6, DeepSeek V4-Flash.
ONNX Runtime GenAI int4 ONNX builds (Phi, Gemma, Llama, Qwen small) on CPU / DirectML / CUDA; the app-embedding path.
vLLM / SGLang production serving; day-0 support for DeepSeek V4, GLM-5, Qwen 3.6; tensor/expert parallelism for multi-GPU.
OpenVINO the Intel Arc / iGPU path now that IPEX-LLM is archived.
Quant ladder: Q8_0 ≈ lossless · Q4_K_M default (≈0.55–0.6 bytes/param) · Q3/IQ3 to squeeze (V4-Flash → 103 GB) · MXFP4 is native for gpt-oss. Min-mem rule of thumb: GGUF file size + 1–3 GB KV/overhead at 8–16K ctx. Hybrid-attention Qwen 3.5/3.6 and V4's sparse attention keep long-context KV unusually small.
Decode is bandwidth-bound: t/s ≈ effective GB/s ÷ (active params × quant bytes) × 0.2–0.6 efficiency. Prefill is compute-bound and 10–100× faster on GPUs — long prompts favor GPUs far more than these decode numbers suggest.
• = measured / community-reported (sources in §6). ≈ = bandwidth-scaled estimate. Expect ±2× from llama.cpp build, context fill, and RAM config — one documented case tripled throughput just by enabling XMP.
Reasoning costs wall-clock: thinking models burn hundreds–thousands of hidden tokens; at 20 t/s a hard answer is 1–2 minutes.
Prices are a Jul-2026 snapshot taken mid-DRAM-crisis; direction is up, and the used market is the value play. AWS figures are us-east-1 on-demand.
Country column reflects the developing organization's HQ (US / CN dominate; nothing else made the cut this cycle). License ≠ jurisdiction: Apache-2.0/MIT weights are usable anywhere.
· gumloop.com — open-weight model field guide (Jul '26: DeepSeek V4, Kimi K2.7, Qwen 3.5)
· taskade.com — open-source LLM rankings + dated release timeline (GLM-5.2, Kimi K3, Qwen 3.6/3.7)
· morphllm.com — open-model benchmark tables; DeepSeek V4 / V4-Flash deep-dives
· onyx.app — open LLM leaderboard (GLM-5.2 GPQA-D 91.2; per-category leaders)
· wikipedia + ghacks + aurigait + lmstudio + layer3labs — Gemma 4 sizes, Apache-2.0 relicense, benchmarks
· codersera.com + insiderllm.com — Qwen 3.5 / 3.6 family guides, sizes, footprints, scores
· huggingface + unsloth.ai + api-docs.deepseek.com — V4 params, GGUF sizes, scores, MIT license
· mindstudio.ai + Vertex model card — gpt-oss architecture and o3-mini/o4-mini positioning
· localaimaster.com — Pi 5 12-model bench; Arc A770; AMD ROCm 7.2 guide (XTX 96 t/s)
· tinyweights + sitepoint + stratosphereips — Raspberry Pi 5/4 numbers, 17 GB/s ceiling
· runaihome.com — gpt-oss-20b consumer-GPU table (4090 225 t/s, 3060 30 t/s); Qwen3.6-35B guide
· hardware-corner.net — RTX 5090 benchmarks (32B @ 61 t/s, gpt-oss-120b @ 128K), GPU ranking, Qwen3 hardware
· insiderllm.com — Qwen3.6-35B-A3B measured: 3090 101 t/s, 3060 + expert offload 38 t/s
· techradar.com (NVIDIA/llama.cpp data) — gpt-oss-20b: 5090 282 · M3 Ultra 116 · 7900 XTX 102 t/s
· huggingface.co/ubergarm + arsturn.com — CPU-only 30B-A3B MoE: 12–15 t/s stock, 30+ tuned @ 88 GB/s
· cybernews + medium/@massimozito + carteakey.dev — gpt-oss-120b on CPUs: 5–8 t/s dual-ch DDR5, ~30 t/s @ 256 GB/s
· aws-graviton-getting-started + whileone.in + llamafile#503 — Graviton4 llama.cpp; 70B dense ~4.6 t/s
· techtactician + markaicode + propelrc — Intel Arc stacks post-IPEX-LLM archive
· videocardprices.com — Jul '26 street: 5090 $3.68–4.5K · 5080 $1.09–1.35K · 5070 Ti $775–970 · 7900 XTX $0.8–1.1K
· techtimes + tech-insider + techpowerup — GPU memory crisis: 5090 to $4,329; RTX-50 output cut 20–40%
· alibaba electronics guide — 4090 refurb $2.5–2.8K; 4070 Ti Super $799
· raspberrypi.com + tomshardware + jeffgeerling + IEEE Spectrum — Pi price rises: 8 GB $80→$125; 16 GB $120→$205→~$300
· instances.vantage.sh + compute.doit.com + Tom's GPU tracker — r8g on-demand/spot; RX 9070 XT $649