Skip to content

llama.cpp on Raspberry Pi: Build and Benchmark

llama.cpp runs quantised GGUF language models locally on Raspberry Pi. A useful setup starts with a model that fits comfortably in RAM, builds the current project with CMake, measures prompt processing and text generation separately, and changes only one parameter per benchmark.

Quick start

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
sudo apt update
sudo apt install -y build-essential cmake git

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j "$(nproc)"

./build/bin/llama-cli --version

Run a local GGUF model:

1
2
3
4
5
6
./build/bin/llama-cli \
  -m /srv/models/model.gguf \
  -p "Explain what a Raspberry Pi GPIO pin does." \
  -n 128 \
  -t 4 \
  -c 2048

The project changes quickly. Check the upstream llama.cpp repository and build documentation before copying old commands. Current builds use CMake and place tools such as llama-cli, llama-server, and llama-bench under build/bin/; old make and ./main instructions are obsolete.

Hardware and model sizing

The model file is only part of the memory requirement. llama.cpp also needs memory for its context/KV cache, compute buffers, the operating system, and any server concurrency.

Raspberry Pi Practical starting point Notes
Pi 4, 4 GB Small 1B–3B-class quantised model, short context Keep desktop applications closed and watch swap
Pi 4, 8 GB Small model or carefully tested 7B-class quantisation Generation is CPU-bound; active cooling helps consistency
Pi 5, 8 GB Small models for responsiveness; test 7B-class models Use active cooling and fast storage for repeatable tests
Pi 5, 16 GB Larger context or model experiments More RAM does not make CPU generation proportionally faster

These are starting ranges, not guarantees. Architecture, quantisation, context length, batch size, llama.cpp revision, and background processes change the result.

Before loading a model:

1
2
3
4
free -h
df -h /srv/models
vcgencmd measure_temp
vcgencmd get_throttled

Keep at least several hundred MiB free for the OS, and preferably more for services and filesystem cache. If the kernel kills llama-cli, choose a smaller model or context instead of treating swap as unlimited RAM.

Choose a GGUF quantisation

GGUF files commonly include a quantisation label such as Q4_K_M or Q5_K_M. Lower-bit quantisation usually reduces RAM and storage use but can reduce model quality. Higher-bit files require more RAM and storage and may be slower to load.

Use this selection process:

  1. Confirm that the model licence permits your use.
  2. Choose a model architecture supported by your current llama.cpp revision.
  3. Start with a moderate quantisation such as a Q4 variant.
  4. Start with a short context such as 2048 tokens.
  5. Measure quality, memory, prompt speed, and generation speed.
  6. Change one dimension—model, quantisation, or context—at a time.

Do not infer RAM requirements from parameter count alone. Inspect the actual file and test the complete runtime configuration.

Build options for Raspberry Pi

Standard CPU build

The standard build is the best baseline:

1
2
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j "$(nproc)"

Record the exact source revision with benchmark results:

1
2
git rev-parse HEAD
./build/bin/llama-cli --version

Native CPU build

When compiling on the same Raspberry Pi that will run the binary, native optimisation can be tested with:

1
2
3
4
cmake -S . -B build-native \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_NATIVE=ON
cmake --build build-native --config Release -j "$(nproc)"

A native binary may not be portable to a different Arm CPU. Benchmark it against the standard build rather than assuming it is faster.

OpenBLAS build

OpenBLAS can improve prompt processing for suitable batch sizes, but upstream documentation notes that it does not improve token generation speed.

1
2
3
4
5
6
7
sudo apt install -y libopenblas-dev

cmake -S . -B build-openblas \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_BLAS=ON \
  -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build-openblas --config Release -j "$(nproc)"

Compare build/bin/llama-bench with build-openblas/bin/llama-bench using the same model and parameters.

Tune the important runtime parameters

Option What it controls Start with
-t, --threads CPU threads used for generation Test 1, 2, and 4
-c, --ctx-size Maximum context/KV cache size 2048, then increase only if needed
-b, --batch-size Prompt-processing batch size Default, then test measured alternatives
-n, --n-predict Maximum generated tokens 128 for a quick test
--mlock Prevent eligible model pages from being swapped Only when the model fits RAM comfortably
--no-mmap Load without memory mapping Test only for a measured reason

More threads are not automatically faster. Test thread counts because memory bandwidth, background work, cooling, and model size affect scaling.

Context size and RAM

A larger -c reserves more memory for context. Do not set a large context simply because the model advertises one. Start small and observe peak resident memory:

1
2
3
4
/usr/bin/time -v ./build/bin/llama-cli \
  -m /srv/models/model.gguf \
  -p "Summarise Raspberry Pi thermal throttling." \
  -n 128 -t 4 -c 2048

Look at Maximum resident set size and repeat with only -c changed.

What --mlock does

--mlock asks the operating system to lock model memory so it cannot be swapped. It can reduce latency spikes after memory pressure, but it does not make matrix multiplication faster.

Check the current locked-memory limit:

1
ulimit -l

Use --mlock only when:

  • The complete model and runtime buffers fit in physical RAM.
  • Other required services retain enough memory.
  • The process has permission to lock the required memory.
  • A repeated benchmark shows that swap-related latency is a real problem.

Do not combine an oversized model with --mlock. The result may be a startup failure or an out-of-memory kill instead of slower swapping.

For a systemd service, set a deliberate limit in its unit rather than silently granting every process unlimited locked memory:

1
2
[Service]
LimitMEMLOCK=6G

Then reload and restart the service:

1
2
sudo systemctl daemon-reload
sudo systemctl restart llama-server

Benchmark prompt and generation speed

Use the project's llama-bench instead of parsing human-oriented CLI output:

1
2
3
4
5
6
7
./build/bin/llama-bench \
  -m /srv/models/model.gguf \
  -p 512 \
  -n 128 \
  -t 1,2,4 \
  -r 5 \
  -o md

The benchmark reports prompt processing (pp) and text generation (tg) separately. The upstream llama-bench documentation notes that its timings do not include tokenisation or sampling, so also measure end-to-end application latency when that matters.

Record:

Field Example of what to capture
Board Raspberry Pi model and RAM
OS /etc/os-release and kernel
llama.cpp Git commit and build flags
Model Exact filename, quantisation, and checksum
Runtime Threads, context, batch, mmap/mlock
Storage microSD, USB SSD, or NVMe
Cooling Cooler, fan curve, ambient temperature
Result pp tokens/s, tg tokens/s, peak RAM, max temperature
Health vcgencmd get_throttled before and after

Run the tests after the system reaches a stable idle state. Avoid comparing a cold first run with a cached repeat without labelling them.

See the site-wide Raspberry Pi benchmark methodology for a reusable test record.

Run the local API server

Bind to loopback by default so the unauthenticated API is not exposed to the LAN:

1
2
3
4
5
6
./build/bin/llama-server \
  -m /srv/models/model.gguf \
  -t 4 \
  -c 2048 \
  --host 127.0.0.1 \
  --port 8080

Test it locally:

1
curl http://127.0.0.1:8080/health

If another machine needs access, put an authenticated TLS reverse proxy or a private VPN in front of the service. Do not bind an unauthenticated model server to 0.0.0.0 and expose the port directly to the internet.

Minimal systemd service

Create a dedicated unprivileged account and model directory:

1
2
sudo useradd --system --home /var/lib/llama --create-home llama
sudo install -d -o llama -g llama /srv/models

Example /etc/systemd/system/llama-server.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
[Unit]
Description=Local llama.cpp API
After=local-fs.target

[Service]
Type=simple
User=llama
Group=llama
ExecStart=/opt/llama.cpp/build/bin/llama-server -m /srv/models/model.gguf -t 4 -c 2048 --host 127.0.0.1 --port 8080
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/srv/models

[Install]
WantedBy=multi-user.target

Adjust the executable and model paths to match your installation, then verify before enabling it at boot:

1
2
3
4
5
sudo systemctl daemon-reload
sudo systemctl start llama-server
systemctl status llama-server
journalctl -u llama-server -n 100 --no-pager
sudo systemctl enable llama-server

Troubleshooting

make: No rule to make target

Current llama.cpp uses CMake. Remove old instructions that call make LLAMA_OPENBLAS=1; configure with cmake -B build and build with cmake --build build.

./main: No such file or directory

The current CLI binary is normally build/bin/llama-cli. The server and benchmark tools are build/bin/llama-server and build/bin/llama-bench.

The process is killed while loading

Check the kernel log:

1
2
journalctl -k -b | grep -i -E 'out of memory|oom|killed process'
free -h

Use a smaller quantisation or model, reduce context size and concurrency, and stop unrelated processes. Do not hide an OOM problem by adding large amounts of slow swap.

mlock fails

Check ulimit -l, the process user, systemd LimitMEMLOCK, and available physical RAM. Run without --mlock unless a benchmark demonstrates a need.

Performance drops during a long run

Monitor temperature, frequency, and throttling:

1
watch -n 2 'vcgencmd measure_temp; vcgencmd measure_clock arm; vcgencmd get_throttled'

Use active cooling, fix undervoltage, and repeat the same benchmark after the system returns to a comparable starting temperature.

The API works locally but not from another computer

This guide intentionally binds to 127.0.0.1. Use a private VPN or an authenticated reverse proxy instead of exposing the raw server. If you deliberately bind to a LAN address, restrict it with a firewall and do not forward the port from your router.

FAQ

Does --mlock make llama.cpp faster on Raspberry Pi?

It can prevent swap-related stalls, but it does not accelerate inference computation. If the model already stays resident and the system is not under memory pressure, it may provide no measurable benefit.

Should I use all four Raspberry Pi CPU cores?

Test it. Four threads are a reasonable starting point on Pi 4 and Pi 5, but memory bandwidth and background services can make a lower count competitive.

Does OpenBLAS improve generation speed?

Upstream documentation says BLAS may improve prompt processing with suitable batch sizes but does not improve generation performance. Measure pp and tg separately.

Can the Raspberry Pi GPU accelerate llama.cpp?

Backend support changes rapidly and results depend on firmware, drivers, build options, and model operations. Treat CPU as the reproducible baseline and test Vulkan only with the current upstream build documentation and a matching benchmark.

Which model is fastest?

Smaller models and lower-bit quantisations generally need less memory, but quality and speed depend on the exact build and prompt. Publish measurements with the model checksum, context, threads, cooling, and llama.cpp commit.