Skip to content

llama.cpp on Raspberry Pi: Build, Memory Locking, and Benchmark

Run a small local language model on your Raspberry Pi before tuning its performance. This guide builds llama.cpp, downloads a specific GGUF model, and generates a short answer; memory locking, benchmarks, and a local API come afterwards.

Before you start

Run the commands in a terminal on the Pi, including through SSH. Use a Raspberry Pi 4 or 5 with a 64-bit Linux installation, an internet connection, and several GB of free storage for source, compiler output, and the model. The first example uses a tiny English-language model on the CPU, two threads, and a short context.

By the end of the required steps, the CLI should load the model, print a short response, and return to your shell prompt. Then you can try the optional benchmark and API sections.

Check the system before installing anything:

1
2
3
uname -m
free -h
df -h "$HOME"

The architecture should be aarch64. On a 32-bit installation, use the Getting Started course to prepare a separate 64-bit OS card first. Keep your current card and data until the new installation works. In free, look at available memory rather than only the free column; the model also needs working buffers and space for the OS.

Stage What you will check
Build llama-cli --version prints a version
Download The GGUF checksum matches the selected file
First response A single short turn finishes
Optional benchmark Prompt processing and generation have separate results
Optional API A loopback health request returns status: ok

Quick start

1. Build llama.cpp on Raspberry Pi

Install the compiler, CMake, Git, and download tools. APT asks before installing packages; read its proposed changes.

sudo apt update
sudo apt install build-essential cmake git curl ca-certificates

Create a fresh working directory in your home directory. Keep using this terminal so that llama_work and your current directory remain available.

# Create a new directory without replacing an existing checkout.
llama_work=$(mktemp -d "$HOME/llama-start.XXXXXX")
printf 'Working directory: %s\n' "$llama_work"
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git "$llama_work/llama.cpp"
cd "$llama_work/llama.cpp"

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF
cmake --build build --config Release \
  --target llama-cli llama-bench llama-server -j 2

./build/bin/llama-cli --version
git rev-parse HEAD

Wait for each command to finish successfully before continuing. The first build can take many minutes on a Pi 4; progress percentages can restart as CMake moves between targets. Two build jobs limit parallel compiler memory use. If compilation is killed, inspect memory and retry the same build command with -j 1; there is no need to delete the checkout.

The version and Git commit identify your build. The upstream build instructions use CMake; old make LLAMA_OPENBLAS=1 and ./main instructions refer to older versions. From a new terminal, change into the printed working directory's llama.cpp subdirectory first.

2. Download a small GGUF model

Use SmolLM2-135M-Instruct, Q4_K_M for the first run. This particular file is 105,454,432 bytes (about 101 MiB). It makes download and loading checks practical; it is not a recommendation for answer quality or every language.

The original model is published by HuggingFaceTB under Apache 2.0. The GGUF conversion used here is published separately by bartowski. The URL selects a fixed repository revision, and the checksum checks that exact file.

1
2
3
4
5
# Run inside the llama.cpp checkout from step 1.
mkdir -p models
curl --fail --location --connect-timeout 15 --max-time 300 \
  --output models/SmolLM2-135M-Instruct-Q4_K_M.gguf.part \
  'https://huggingface.co/bartowski/SmolLM2-135M-Instruct-GGUF/resolve/09816acd5d99df7be770d85ea30822623dab342c/SmolLM2-135M-Instruct-Q4_K_M.gguf'

Only continue if curl succeeds. Verify the completed download before giving it its final name:

1
2
3
4
5
6
printf '%s  %s\n' \
  '2e8040ceae7815abe0dcb3540b9995eaa1fa0d2ca9e797d0a635ae4433c68c2d' \
  'models/SmolLM2-135M-Instruct-Q4_K_M.gguf.part' \
  | sha256sum --check \
  && mv models/SmolLM2-135M-Instruct-Q4_K_M.gguf.part \
    models/SmolLM2-135M-Instruct-Q4_K_M.gguf

Expect a line ending in : OK. If the checksum fails or the transfer times out, stop and repeat the download; do not load the partial file. curl --output replaces the .part file on the next attempt.

3. Generate one short response

1
2
3
4
5
6
./build/bin/llama-cli \
  -m models/SmolLM2-135M-Instruct-Q4_K_M.gguf \
  --device none \
  --single-turn \
  -p "Write one short sentence about a garden." \
  -n 32 -t 2 -c 512

Loading messages are normal. After loading, expect generated text and a return to the shell prompt. The wording varies and the model can make mistakes. -n 32 limits generated tokens, while --single-turn prevents another interactive turn. Tokens are pieces of text, not necessarily whole words. Press Ctrl+C to stop a run early.

Option Purpose in this first run
-m Path to the downloaded model
--device none Keep inference on the CPU
--single-turn Finish after the supplied prompt's response
-n 32 Generate at most 32 tokens
-t 2 Use two generation threads
-c 512 Use a small context for this short request

Keep this file and command working before changing the model or enabling additional features. If an option is rejected, use ./build/bin/llama-cli --help and confirm you are running the executable you just built.

Hardware and model sizing

The GGUF file is only part of the memory requirement. Context/KV cache, compute buffers, the OS, and server concurrency need additional RAM. A 4 GB Pi does not have 4 GB available for model weights.

Start with the small example above on both Pi 4 and Pi 5. For the next model, choose a supported instruction-tuned GGUF, check its licence and actual file size, then keep context short. Increase one dimension at a time and observe memory. More RAM allows larger workloads; it does not proportionally increase CPU generation speed.

Change What to watch
More parameters Weight storage, loading time, and generation latency
Different quantisation File size and answer quality; Q4_K_M is not the same as every other Q4 format
Larger context (-c) KV cache and peak RAM, even with the same model file
More concurrent requests Combined server buffers and active contexts

Do not choose an architecture-specific quantisation just because its filename starts with Q4. It may require CPU instructions absent on your Pi. Consult the conversion's model card and your llama.cpp build.

Build options for Raspberry Pi

The first build is the CPU baseline. Upstream enables GGML_NATIVE by default, so adding -DGGML_NATIVE=ON is not a separate optimisation experiment. Build on the Pi you will use; a native executable built for a different CPU may fail with Illegal instruction.

OpenBLAS is optional. Upstream describes benefits for prompt processing with suitable batch sizes, not a general improvement to token generation. To compare it, install its development package and use a separate build directory:

1
2
3
4
5
6
7
sudo apt install libopenblas-dev
cmake -S . -B build-openblas \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLAMA_BUILD_TESTS=OFF \
  -DGGML_BLAS=ON \
  -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build-openblas --config Release --target llama-bench -j 2

Compare the two llama-bench executables with the same model and benchmark settings. Installing OpenBLAS alone does not change an already-built executable.

Tune the important runtime parameters

Keep the model fixed when changing runtime settings. Try -t 1, -t 2, and -t 4 separately; four threads can lose to fewer threads when memory bandwidth or background work limits throughput. Increase -c only when your prompt and output need more space. A 512-token context is for this brief exercise, not a long conversation.

For a finite run's peak resident memory, install the time package if /usr/bin/time is missing, then run:

1
2
3
4
5
/usr/bin/time -v ./build/bin/llama-cli \
  -m models/SmolLM2-135M-Instruct-Q4_K_M.gguf \
  --device none --single-turn \
  -p "Write one short sentence about a garden." \
  -n 32 -t 2 -c 512

Look for Maximum resident set size (kbytes). This is process resident memory, not a complete measure of system memory use. Repeat with only one setting changed; do not change the model and context while trying to isolate a thread-count improvement.

What --mlock does

Memory locking asks Linux to keep model pages resident. Current builds expose --load-mode; mmap+mlock requests mapping and locking. Older builds used --mlock. Neither form accelerates the inference calculation itself.

Leave the loading mode at its default for the first run. Consider locking only if the complete workload fits comfortably in physical RAM and a repeated measurement identifies paging-related stalls. Check your actual executable's options and the shell's limit:

./build/bin/llama-cli --help
ulimit -l

Do not solve a warning by granting unlimited locked memory or copying a service limit from a different Pi. Follow the llama.cpp memory-lock failure guide to distinguish an obsolete option, a lock warning, and a kernel OOM kill; it also explains bounded systemd limits and rollback.

Benchmark prompt and generation speed

This short, optional benchmark uses the same small model, two threads, and one repetition:

1
2
3
./build/bin/llama-bench \
  -m models/SmolLM2-135M-Instruct-Q4_K_M.gguf \
  -p 32 -n 16 -t 2 -r 1 -o md

Read the test and t/s columns:

Test Meaning
pp32 Process a 32-token prompt; prompt-processing throughput
tg16 Generate 16 tokens; generation throughput

One repetition is a smoke test, not a stable performance result. For comparisons, repeat under similar idle, cooling, and power conditions and report the variation. The upstream benchmark documentation explains that tokenisation and sampling are excluded, so these numbers are not the full time a user waits for an answer.

Record the board/RAM, OS/kernel, Git commit, model filename/checksum, build flags, threads, context, and storage. Check temperature and throttling before and after longer tests. Use the benchmark methodology and throttling decoder to interpret the conditions rather than reporting tokens/s alone.

Run the local API server

Keep the CLI working first. In the same checkout, start an optional foreground server bound to this Pi's loopback address:

1
2
3
4
5
./build/bin/llama-server \
  -m models/SmolLM2-135M-Instruct-Q4_K_M.gguf \
  --device none \
  -t 2 -c 512 --parallel 1 \
  --host 127.0.0.1 --port 8080

Leave that terminal open. In a second terminal on the same Pi, check health after the model has loaded:

curl --fail --connect-timeout 3 --max-time 10 \
  http://127.0.0.1:8080/health

An HTTP 200 response containing {"status":"ok"} means the server is ready. A loading response means to wait; Connection refused means no listener is reachable at that address and port. Check the first terminal's messages. If port 8080 is occupied, choose another free port in both commands.

Return to the first terminal and press Ctrl+C to stop the server. This exercise does not enable startup at boot. Before turning it into a persistent service, plan a dedicated account, actual executable/model paths, resource limits, and recovery using the service-isolation lesson. Keep the API private; access from another machine needs an appropriately authenticated connection or proxy.

Troubleshooting

Symptom First check Next action
CMake cannot find a compiler or build tool Read the first configure error Complete prerequisite package installation, then reconfigure
Build is killed Check available memory and kernel logs Retry with one build job; keep the same checkout
./main or make instructions fail Check the tutorial/version Use CMake and build/bin/llama-cli as above
Model file not found Check pwd and the filename below Return to your checkout and confirm the download step completed
GGUF loading fails Check the checksum and first error Redownload a bad file; for a different model, check architecture support
unknown argument: --mlock Inspect this executable's --help Use the version-aware memory-lock guide
Killed or exit 137 Inspect kernel logs for an OOM entry Investigate; the exit alone is not proof of OOM

These checks are read-only:

1
2
3
4
pwd
ls -lh models/SmolLM2-135M-Instruct-Q4_K_M.gguf
free -h
journalctl -k -b --no-pager | grep -i -E 'out of memory|oom|killed process'

An empty grep result is not proof that memory was sufficient; log access and retention vary. With a confirmed memory shortage, reduce model size, context, or concurrency. Adding slow swap does not make an oversized workload responsive.

If generation slows over time, use the temperature guide and check vcgencmd get_throttled. Do not attribute a changing result to the model before checking power, cooling, and other running processes.

Check your understanding

  • Why can a model need more RAM than its GGUF file size?
  • Which option makes the supplied prompt a single turn, and which limits output tokens?
  • Why is pp32 not a prediction of generation speed?
  • Why would a memory-lock warning need a different response from a confirmed OOM kill?

Next, use the benchmark methodology to compare one setting at a time. For an actual locking error, go directly to the mlock diagnostic.

Donate