1. Install llama.cpp

sudo apt update && sudo apt install build-essential cmake git -y
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j4

Build from source. Prebuilt binaries rarely match your kernel and flags, and a mismatched build can cost you 20% throughput.

2. Pick models that actually fit

A Pi 5 has 8 GB of RAM shared with the OS. Stay under ~5 GB for model + context or you'll swap and your numbers are garbage. Reliable picks in Q4: TinyLlama 1.1B, Qwen 1.5B/3B, Phi-3-mini (tight but works), Gemma 2B. Skip 7B+ unless you enjoy watching swap thrash.

3. Measure honestly

Use llama-bench with a fixed prompt and generation length. Log three numbers per run: prompt processing tok/s, generation tok/s, and SoC temperature at start and end. A run without a temperature is not a benchmark - thermal throttling on an uncooled Pi 5 shows up around 85°C and can cut throughput by a third mid-run.

./build/bin/llama-bench -m models/qwen-3b-q4.gguf -p 128 -n 256 -r 3

4. Keep runs repeatable

• Fix the CPU governor (performance) for the run and note it down.
• Same prompt set every time. Different prompts = different numbers.
• Cool the board between runs, or say so in the log.
• Record ambient temperature when comparing across days.

Going further

I run this protocol often enough that I packaged it: a Benchmark Workbook with the log sheets and test matrix pre-built, and a Report Generator that turns raw logs into a shareable report.

Full guide with more detail: Local AI on Raspberry Pi - guides hub

What are you running on your Pi? Curious what tok/s people are seeing on 3B-class models.