Skip to content

Offline Voice Assistant on Raspberry Pi OS Trixie

Build a local voice pipeline in testable stages: microphone capture, voice activity or keyword detection, speech-to-text, a strict intent dispatcher, and text-to-speech. Raspberry Pi OS Trixie can run this on 64-bit ARM without sending recordings to a cloud service, but model size, microphone quality, and cooling determine the actual latency.

This guide uses the actively maintained sherpa-onnx project as the common runtime because it supports ARM64 Linux, Raspberry Pi, streaming and non-streaming recognition, keyword spotting, VAD, and speech synthesis. It does not use the retired pattern of installing the old Google Assistant SDK or copying OAuth credentials into a generic script.

Architecture and trust boundaries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
microphone
    ↓
audio device selection and level test
    ↓
VAD or keyword spotter
    ↓
offline speech recognition
    ↓
exact intent allowlist
    ↓
local action and offline speech response

Treat every recognised phrase as untrusted input. Do not pass it to shell=True, eval, a shell pipeline, sudo, or an unrestricted home-automation API. The dispatcher should map a small known phrase to a fixed argument list.

Choose the first useful version

Version Components Use it when
Push-to-talk prototype Record button, ASR, printed response Proving the microphone and model
Endpointed assistant VAD, ASR, allowlisted actions, TTS A desk or workshop assistant
Wake-word assistant Keyword spotter, VAD, ASR, actions, TTS Hands-free use after false-positive testing
Conversational assistant Above plus a local or remote language model You have measured latency, privacy, and failure behavior

Start with push-to-talk. A wake word adds a continuously running model, false accepts, false rejects, and more audio-device state. An LLM adds nondeterministic output and should not be allowed to create shell commands.

Hardware and OS baseline

Recommended starting point:

  • Raspberry Pi 5 or Pi 4 with a 64-bit Raspberry Pi OS image
  • USB microphone or a supported audio HAT
  • USB, HDMI, HAT, or analogue audio output appropriate to the model
  • official-quality power supply and active cooling for sustained inference
  • at least several gigabytes of free storage for environments, models, logs, and recordings used during testing

Record the system before installing anything:

1
2
3
4
5
cat /etc/os-release
uname -m
python3 --version
df -h /
vcgencmd get_throttled

Expect aarch64 from uname -m for the ARM64 Python wheel path. Do not infer userspace architecture from the board model alone.

Understand Raspberry Pi audio layers

Raspberry Pi OS Desktop normally exposes applications through PipeWire-compatible interfaces, while Raspberry Pi OS Lite can use ALSA directly. The official Raspberry Pi audio guidance warns that a global ~/.asoundrc can interfere with the desktop's view of devices; create one only for an application that truly needs direct ALSA configuration.

List devices without changing configuration:

1
2
3
4
arecord -l
aplay -l
arecord -L | sed -n '1,120p'
aplay -L | sed -n '1,120p'

On a desktop session, also inspect the user audio server when the commands exist:

1
2
wpctl status
pactl info

Do not run the assistant as root to “fix” microphone access. A root process has a different environment and creates a much larger impact if recognition triggers the wrong action.

Install dependencies in a virtual environment

Install OS packages:

1
2
3
4
5
sudo apt update
sudo apt full-upgrade
sudo apt install --no-install-recommends \
  python3-venv python3-pip alsa-utils ffmpeg \
  libportaudio2 portaudio19-dev git

Create an application directory owned by your normal user:

1
2
3
4
5
6
7
mkdir -p ~/offline-voice/{models,recordings,logs}
cd ~/offline-voice
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install sherpa-onnx sounddevice
python -m pip freeze > requirements.lock

On 6 August 2026, PyPI published sherpa-onnx ARM64 Linux wheels for current CPython versions including 3.13. Always verify the package source, resolved version, and wheel architecture on the device instead of copying that observation indefinitely:

1
2
python -m pip show sherpa-onnx sounddevice
python -c 'import platform; from importlib.metadata import version; print(platform.machine(), version("sherpa-onnx"))'

Do not use sudo pip, and do not disable Debian's externally-managed-environment protection. Recreate the virtual environment when changing Python major versions.

Prove capture and playback first

Record five seconds as 16 kHz, 16-bit, mono PCM. Replace the device after checking arecord -l:

1
2
3
4
5
arecord -D plughw:CARD=<card>,DEV=<device> \
  -f S16_LE -r 16000 -c 1 -d 5 recordings/mic-test.wav

file recordings/mic-test.wav
aplay recordings/mic-test.wav

Inspect the signal without uploading it:

1
2
ffmpeg -hide_banner -i recordings/mic-test.wav \
  -af volumedetect -f null - 2>&1 | tail -20

Listen for clipping, hum, echo, and automatic-gain pumping. Software cannot recover words that the input clipped or never captured clearly. Move the microphone, reduce speaker feedback, and set a repeatable distance before comparing models.

Select and verify an offline model

sherpa-onnx supports multiple model families. Choose one from its official pre-trained model documentation that explicitly lists:

  • supported language
  • streaming or non-streaming operation
  • sample rate
  • model licence
  • required tokens, encoder, decoder, or joiner files
  • CPU and memory expectations

Avoid copying a model URL from an unrelated tutorial. Download into a versioned directory and keep a checksum manifest:

1
2
3
4
5
6
mkdir -p ~/offline-voice/models/<model-name>
cd ~/offline-voice/models/<model-name>

# Download the exact files from the selected official model page, then:
sha256sum * > SHA256SUMS.local
sha256sum -c SHA256SUMS.local

The locally generated checksum detects later corruption but does not authenticate the first download. Compare publisher-provided hashes when available and record each source URL and licence.

Useful official entry points:

Use the command shown by the chosen model page because argument names differ between transducer, Paraformer, Whisper, SenseVoice, Moonshine, and other model families. Confirm available options in the installed version:

1
2
sherpa-onnx --help
python -c 'from importlib.metadata import version; print(version("sherpa-onnx"))'

Build a safe intent dispatcher

Test command policy separately from speech recognition. Create dispatcher.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/usr/bin/env python3
from __future__ import annotations

import re
import subprocess
import sys


ACTIONS = {
    "system status": ["systemctl", "--failed", "--no-pager"],
    "disk space": ["df", "-h", "/"],
    "ip address": ["hostname", "-I"],
    "temperature": ["vcgencmd", "measure_temp"],
}


def normalize(text: str) -> str:
    return re.sub(r"[^a-z0-9 ]+", " ", text.lower()).strip()


def dispatch(text: str) -> tuple[bool, str]:
    phrase = normalize(text)
    command = ACTIONS.get(phrase)
    if command is None:
        return False, "No approved action matches that exact phrase."
    try:
        result = subprocess.run(
            command,
            check=False,
            capture_output=True,
            text=True,
            timeout=10,
        )
    except (OSError, subprocess.TimeoutExpired) as error:
        return False, f"Action failed safely: {error}"
    output = (result.stdout or result.stderr).strip()
    return result.returncode == 0, output[:2000]


if __name__ == "__main__":
    ok, response = dispatch(" ".join(sys.argv[1:]))
    print(response)
    raise SystemExit(0 if ok else 1)

Test accepted and rejected input:

1
2
3
4
source ~/offline-voice/.venv/bin/activate
python dispatcher.py temperature
python dispatcher.py 'system status'
python dispatcher.py 'temperature; reboot now'

The last input must be rejected. subprocess.run() receives a fixed list and never invokes a shell. Keep write operations, door locks, alarms, purchases, and destructive administration out of the first version.

Connect recognition to the dispatcher

The recognition callback should pass only the final recognised text to dispatch(). Keep partial transcripts for UI feedback, not actions. Use an explicit state machine:

1
IDLE → LISTENING → FINAL_TEXT → CONFIRM_OR_DISPATCH → RESPONDING → IDLE

Required safeguards:

  1. Discard empty and partial results.
  2. Limit maximum utterance duration.
  3. Require exact phrases for privileged or state-changing actions.
  4. Add a physical or spoken confirmation for actions with consequences.
  5. Apply a cooldown so echo from the speaker cannot immediately retrigger the assistant.
  6. Log the action identifier and outcome, not raw audio by default.

Do not ask a language model to return a command that is then executed. If you later add conversational generation, place it outside the deterministic action boundary.

Add keyword spotting only after ASR works

sherpa-onnx includes keyword spotting, and openWakeWord is another local option. Model licences and commercial-use terms differ, so review both code and model licences.

Measure at least:

  • true activations in quiet speech
  • missed activations at intended distances
  • false activations during television, music, and conversation
  • CPU and memory use while idle
  • time from keyword end to listening state

Keep a visible microphone or listening indicator. Add a hardware mute switch that electrically or logically disables capture when privacy matters.

Add offline text-to-speech

sherpa-onnx provides ARM and ARM64 speech-synthesis packages and model-specific examples. Keep TTS separate from ASR so each stage can be timed and replaced.

Test generated audio before integrating it:

1
sherpa-onnx-offline-tts --help

Follow the exact command on the selected model page, write to a WAV file, then play it through the verified device. Do not let recognised text become a filename or command-line option without validation.

Prevent speaker output from retriggering recognition by pausing capture during playback or using echo cancellation with a measured cooldown.

Run as an unprivileged user service

First make the complete assistant loop work interactively. Then create ~/.config/systemd/user/offline-voice.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
[Unit]
Description=Offline voice assistant
After=pipewire.service wireplumber.service

[Service]
Type=simple
WorkingDirectory=%h/offline-voice
ExecStart=%h/offline-voice/.venv/bin/python %h/offline-voice/assistant_loop.py
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=%h/offline-voice/logs

[Install]
WantedBy=default.target

The example assumes assistant_loop.py already exists and writes only to logs. A desktop user service can access the user's audio session more naturally than a system service. Raspberry Pi OS Lite may need different ordering because it can use ALSA directly and have no PipeWire user service.

Validate before enabling:

1
2
3
4
5
systemd-analyze --user verify ~/.config/systemd/user/offline-voice.service
systemctl --user daemon-reload
systemctl --user start offline-voice.service
systemctl --user status offline-voice.service --no-pager
journalctl --user -u offline-voice.service -b --no-pager

Enable it only after an interactive start succeeds:

1
systemctl --user enable offline-voice.service

Measure latency and resource use

Do not publish model performance without measurements from the stated hardware. Record timestamps at:

1
2
3
4
5
6
speech start
speech end / endpoint detected
final transcript ready
intent selected
action complete
audio response begins

Monitor the process:

1
2
3
4
5
pid=$(systemctl --user show -p MainPID --value offline-voice.service)
ps -o pid,etimes,%cpu,%mem,rss,cmd -p "$pid"
pidstat -r -u -p "$pid" 1
vcgencmd measure_temp
vcgencmd get_throttled

Use a result table with real values:

Pi / cooling Model Audio Endpoint-to-text Response start CPU RSS Notes
Record your system Record exact files 16 kHz mono — — — — Do not fill with estimates

Run enough repeated phrases to show median and slow-tail behavior. Include misses and false activations; averages alone hide failures.

Troubleshooting

PortAudioError: Error querying device

List devices from the same user and environment that runs the assistant:

1
2
3
source ~/offline-voice/.venv/bin/activate
python -c 'import sounddevice as sd; print(sd.query_devices())'
arecord -l

Do not assume device index 0 remains stable after adding HDMI, USB, Bluetooth, or HAT audio. Prefer a verified device name where the API supports it.

Works in a terminal but not under systemd

Compare the user, environment, working directory, audio session, and model paths. Use absolute paths and inspect the user journal. Do not solve it by switching to a root system service.

Recognition is slow

Check throttling and cooling, then compare a smaller or quantised model from the same official family. Measure endpointing separately from inference; waiting for silence can look like slow recognition.

Recognition is inaccurate

Replay the saved test WAV through the recogniser. If the recording itself is clipped, quiet, reverberant, or dominated by speaker echo, fix capture before changing models.

The ARM64 package does not install

Verify all three layers:

1
2
3
uname -m
dpkg --print-architecture
python3 -c 'import platform; print(platform.machine())'

Recreate the virtual environment after confirming aarch64/arm64. Do not install an AMD64 wheel or use --break-system-packages.

The assistant triggers itself

Pause recognition during TTS, lower speaker-to-microphone coupling, add a post-playback cooldown, and test echo cancellation. Keep the action dispatcher idempotent where possible.

Privacy and maintenance checklist

  • process audio locally unless the interface clearly indicates a cloud mode
  • do not retain raw recordings by default
  • protect logs from transcripts, tokens, addresses, and personal names
  • verify model and package licences before distribution or commercial use
  • pin and record working dependency versions, but review security updates
  • keep actions allowlisted and unprivileged
  • provide a visible listening state and physical mute option
  • test failure after network loss, audio-device removal, model corruption, and reboot
  • back up configuration, not captured speech

FAQ

Can this work without Internet access?

Yes, after OS packages, Python wheels, and model files are installed. Test by disconnecting the network and confirming that recognition, dispatch, and speech generation still work.

Is Raspberry Pi 5 required?

No. Smaller streaming models can run on older 64-bit Pi models, but latency and model choice must be measured. Pi Zero-class systems need especially careful model and audio choices.

Should I use Vosk instead?

Vosk remains an offline recogniser with small models suitable for Raspberry Pi. Its current public Python package and documentation should be tested against the exact Trixie Python version. This guide uses sherpa-onnx because its current ARM64 wheels and documentation cover ASR, VAD, keyword spotting, and TTS in one actively updated project.

Can a local LLM control shell commands?

Do not execute generated shell text. Map confirmed intents to fixed, reviewed argument lists outside the model, and require additional confirmation for consequential actions.