Field Log
Pi 5 LLM Bench Notes
Testing local LLM inference on a Raspberry Pi 5, including CPU baselines, Vulkan surprises, and what the next benchmark round needs to prove.
I have been testing local LLM inference on a Raspberry Pi 5 because it sits in the exact zone that makes this fun: cheap enough to experiment with, powerful enough to surprise you, and constrained enough that every assumption gets punished quickly. The test box is an 8 GB Pi 5 running llama.cpp at commit 22397c31a, with CPU/OpenBLAS and Vulkan builds.
The first useful baseline was Qwen3 1.7B Q4_K_M. On CPU/OpenBLAS, the Pi 5 hit 55.35 tok/s for prompt processing and 10.11 tok/s for generation. That is not workstation-fast, but it is genuinely usable for short local tasks, classifications, tool decisions, and little appliance-style workflows.
The 4B test made things more interesting. Qwen3 4B Q4_K_M is a 2.32 GiB GGUF, and the warm CPU run produced:
| Configuration | pp512 | tg128 | Peak RSS |
|---|---|---|---|
| CPU/OpenBLAS | 21.46 | 4.52 | ~4.84 GiB |
Vulkan build, forced CPU with --device none | 21.38 | 4.53 | ~4.89 GiB |
Vulkan with -ngl 0 | 8.06 | 0.80 | ~343 MiB |
Vulkan with GGML_VK_PREFER_HOST_MEMORY=1 -ngl 0 | 7.84 | 0.80 | ~344 MiB |
The CPU result is the practical one: around 4.5 generated tokens per second from a 4B Q4 model on a Pi 5 is slow, but usable if the job is small and local-first matters more than speed. The Vulkan result is the weird one. It looked like a massive memory win if you only watched process RSS, but it was also roughly five to six times slower for generation.
The interesting finding is that the memory did not vanish. During the Vulkan run, the process showed only about 345 MiB RSS, while /proc/meminfo showed roughly 2.42 GiB of system Shmem, suspiciously close to the 2.32 GiB model file. The current working theory is that the Pi 5’s V3DV Vulkan path is allocating model-related buffers through DRM/GEM shared-memory-backed GPU objects. Same physical LPDDR4X, different accounting bucket.
That means the original “Vulkan uses way less RAM” idea is only partly true. It can absolutely make process RSS look tiny, but that does not mean the weights stopped occupying memory. The useful question is subtler: can those driver-managed pages behave differently enough under pressure to let a larger model remain usable without ordinary Linux swap pain?
That is where the next tests come in. The 1.7B and 4B runs are good controls, but the real answer needs an 8B-class Q4 model around 4.5 to 5 GiB, with swap disabled before the run. If the CPU path pushes the 8 GB Pi into ugly memory pressure while Vulkan keeps enough MemAvailable headroom to limp along, that is useful even if it is slow. If both paths hit the same wall, then the Vulkan memory trick is mostly prettier accounting.
My current take: Raspberry Pi 5 CPU inference is more practical than expected for small local models, especially around 1.7B to 4B. Vulkan on the Pi is not faster for this workload, but it exposes a memory-accounting behavior worth understanding. The next bench session is less about chasing tokens per second and more about finding out whether an 8 GB Pi can stretch into models that should be right on the edge.