llama.cpp on Raspberry Pi: Build, Memory Locking, and Benchmark¶
Run a small local language model on your Raspberry Pi before tuning its performance. This guide builds llama.cpp, downloads a specific GGUF model, and generates a short answer; memory locking, benchmarks, and a local API come afterwards.
Before you start¶
Run the commands in a terminal on the Pi, including through SSH. Use a Raspberry Pi 4 or 5 with a 64-bit Linux installation, an internet connection, and several GB of free storage for source, compiler output, and the model. The first example uses a tiny English-language model on the CPU, two threads, and a short context.
By the end of the required steps, the CLI should load the model, print a short response, and return to your shell prompt. Then you can try the optional benchmark and API sections.
Check the system before installing anything:
The architecture should be aarch64. On a 32-bit installation, use the Getting Started course to prepare a separate 64-bit OS card first. Keep your current card and data until the new installation works. In free, look at available memory rather than only the free column; the model also needs working buffers and space for the OS.
| Stage | What you will check |
|---|---|
| Build | llama-cli --version prints a version |
| Download | The GGUF checksum matches the selected file |
| First response | A single short turn finishes |
| Optional benchmark | Prompt processing and generation have separate results |
| Optional API | A loopback health request returns status: ok |
Quick start¶
1. Build llama.cpp on Raspberry Pi¶
Install the compiler, CMake, Git, and download tools. APT asks before installing packages; read its proposed changes.
Create a fresh working directory in your home directory. Keep using this terminal so that llama_work and your current directory remain available.
Wait for each command to finish successfully before continuing. The first build can take many minutes on a Pi 4; progress percentages can restart as CMake moves between targets. Two build jobs limit parallel compiler memory use. If compilation is killed, inspect memory and retry the same build command with -j 1; there is no need to delete the checkout.
The version and Git commit identify your build. The upstream build instructions use CMake; old make LLAMA_OPENBLAS=1 and ./main instructions refer to older versions. From a new terminal, change into the printed working directory's llama.cpp subdirectory first.
2. Download a small GGUF model¶
Use SmolLM2-135M-Instruct, Q4_K_M for the first run. This particular file is 105,454,432 bytes (about 101 MiB). It makes download and loading checks practical; it is not a recommendation for answer quality or every language.
The original model is published by HuggingFaceTB under Apache 2.0. The GGUF conversion used here is published separately by bartowski. The URL selects a fixed repository revision, and the checksum checks that exact file.
Only continue if curl succeeds. Verify the completed download before giving it its final name:
Expect a line ending in : OK. If the checksum fails or the transfer times out, stop and repeat the download; do not load the partial file. curl --output replaces the .part file on the next attempt.
3. Generate one short response¶
Loading messages are normal. After loading, expect generated text and a return to the shell prompt. The wording varies and the model can make mistakes. -n 32 limits generated tokens, while --single-turn prevents another interactive turn. Tokens are pieces of text, not necessarily whole words. Press Ctrl+C to stop a run early.
| Option | Purpose in this first run |
|---|---|
-m |
Path to the downloaded model |
--device none |
Keep inference on the CPU |
--single-turn |
Finish after the supplied prompt's response |
-n 32 |
Generate at most 32 tokens |
-t 2 |
Use two generation threads |
-c 512 |
Use a small context for this short request |
Keep this file and command working before changing the model or enabling additional features. If an option is rejected, use ./build/bin/llama-cli --help and confirm you are running the executable you just built.
Hardware and model sizing¶
The GGUF file is only part of the memory requirement. Context/KV cache, compute buffers, the OS, and server concurrency need additional RAM. A 4 GB Pi does not have 4 GB available for model weights.
Start with the small example above on both Pi 4 and Pi 5. For the next model, choose a supported instruction-tuned GGUF, check its licence and actual file size, then keep context short. Increase one dimension at a time and observe memory. More RAM allows larger workloads; it does not proportionally increase CPU generation speed.
| Change | What to watch |
|---|---|
| More parameters | Weight storage, loading time, and generation latency |
| Different quantisation | File size and answer quality; Q4_K_M is not the same as every other Q4 format |
Larger context (-c) |
KV cache and peak RAM, even with the same model file |
| More concurrent requests | Combined server buffers and active contexts |
Do not choose an architecture-specific quantisation just because its filename starts with Q4. It may require CPU instructions absent on your Pi. Consult the conversion's model card and your llama.cpp build.
Build options for Raspberry Pi¶
The first build is the CPU baseline. Upstream enables GGML_NATIVE by default, so adding -DGGML_NATIVE=ON is not a separate optimisation experiment. Build on the Pi you will use; a native executable built for a different CPU may fail with Illegal instruction.
OpenBLAS is optional. Upstream describes benefits for prompt processing with suitable batch sizes, not a general improvement to token generation. To compare it, install its development package and use a separate build directory:
Compare the two llama-bench executables with the same model and benchmark settings. Installing OpenBLAS alone does not change an already-built executable.
Tune the important runtime parameters¶
Keep the model fixed when changing runtime settings. Try -t 1, -t 2, and -t 4 separately; four threads can lose to fewer threads when memory bandwidth or background work limits throughput. Increase -c only when your prompt and output need more space. A 512-token context is for this brief exercise, not a long conversation.
For a finite run's peak resident memory, install the time package if /usr/bin/time is missing, then run:
Look for Maximum resident set size (kbytes). This is process resident memory, not a complete measure of system memory use. Repeat with only one setting changed; do not change the model and context while trying to isolate a thread-count improvement.
What --mlock does¶
Memory locking asks Linux to keep model pages resident. Current builds expose --load-mode; mmap+mlock requests mapping and locking. Older builds used --mlock. Neither form accelerates the inference calculation itself.
Leave the loading mode at its default for the first run. Consider locking only if the complete workload fits comfortably in physical RAM and a repeated measurement identifies paging-related stalls. Check your actual executable's options and the shell's limit:
Do not solve a warning by granting unlimited locked memory or copying a service limit from a different Pi. Follow the llama.cpp memory-lock failure guide to distinguish an obsolete option, a lock warning, and a kernel OOM kill; it also explains bounded systemd limits and rollback.
Benchmark prompt and generation speed¶
This short, optional benchmark uses the same small model, two threads, and one repetition:
Read the test and t/s columns:
| Test | Meaning |
|---|---|
pp32 |
Process a 32-token prompt; prompt-processing throughput |
tg16 |
Generate 16 tokens; generation throughput |
One repetition is a smoke test, not a stable performance result. For comparisons, repeat under similar idle, cooling, and power conditions and report the variation. The upstream benchmark documentation explains that tokenisation and sampling are excluded, so these numbers are not the full time a user waits for an answer.
Record the board/RAM, OS/kernel, Git commit, model filename/checksum, build flags, threads, context, and storage. Check temperature and throttling before and after longer tests. Use the benchmark methodology and throttling decoder to interpret the conditions rather than reporting tokens/s alone.
Run the local API server¶
Keep the CLI working first. In the same checkout, start an optional foreground server bound to this Pi's loopback address:
Leave that terminal open. In a second terminal on the same Pi, check health after the model has loaded:
An HTTP 200 response containing {"status":"ok"} means the server is ready. A loading response means to wait; Connection refused means no listener is reachable at that address and port. Check the first terminal's messages. If port 8080 is occupied, choose another free port in both commands.
Return to the first terminal and press Ctrl+C to stop the server. This exercise does not enable startup at boot. Before turning it into a persistent service, plan a dedicated account, actual executable/model paths, resource limits, and recovery using the service-isolation lesson. Keep the API private; access from another machine needs an appropriately authenticated connection or proxy.
Troubleshooting¶
| Symptom | First check | Next action |
|---|---|---|
| CMake cannot find a compiler or build tool | Read the first configure error | Complete prerequisite package installation, then reconfigure |
| Build is killed | Check available memory and kernel logs | Retry with one build job; keep the same checkout |
./main or make instructions fail |
Check the tutorial/version | Use CMake and build/bin/llama-cli as above |
| Model file not found | Check pwd and the filename below |
Return to your checkout and confirm the download step completed |
| GGUF loading fails | Check the checksum and first error | Redownload a bad file; for a different model, check architecture support |
unknown argument: --mlock |
Inspect this executable's --help |
Use the version-aware memory-lock guide |
Killed or exit 137 |
Inspect kernel logs for an OOM entry | Investigate; the exit alone is not proof of OOM |
These checks are read-only:
An empty grep result is not proof that memory was sufficient; log access and retention vary. With a confirmed memory shortage, reduce model size, context, or concurrency. Adding slow swap does not make an oversized workload responsive.
If generation slows over time, use the temperature guide and check vcgencmd get_throttled. Do not attribute a changing result to the model before checking power, cooling, and other running processes.
Check your understanding¶
- Why can a model need more RAM than its GGUF file size?
- Which option makes the supplied prompt a single turn, and which limits output tokens?
- Why is
pp32not a prediction of generation speed? - Why would a memory-lock warning need a different response from a confirmed OOM kill?
Next, use the benchmark methodology to compare one setting at a time. For an actual locking error, go directly to the mlock diagnostic.