eBPF (Extended Berkeley Packet Filter) - Kernel Programming for Raspberry Pi
Introduction
eBPF (Extended Berkeley Packet Filter) is a revolutionary Linux kernel technology that allows you to run sandboxed programs directly in the kernel without changing kernel source code or loading kernel modules. Originally designed for packet filtering, eBPF has evolved into a powerful platform for observability, security, networking, and performance analysis.
With eBPF, you can:
- Monitor System Performance: Track CPU, memory, disk I/O, and network in real-time
- Network Packet Analysis: Capture, filter, and analyze network traffic at kernel level
- Security Monitoring: Detect malicious behavior, syscall anomalies, and intrusions
- Dynamic Tracing: Profile application performance without recompilation
- Custom Networking: Implement load balancers, firewalls, and DDoS protection
- Resource Control: Fine-grained control over process behavior and resource usage
This comprehensive guide covers:
- eBPF Architecture: How eBPF works in the Linux kernel
- Development Tools: BCC, bpftrace, libbpf, and modern eBPF toolchains
- Practical Examples: Network monitoring, system tracing, security enforcement
- Performance Analysis: CPU profiling, I/O analysis, latency tracking
- Security Applications: Intrusion detection, syscall filtering, malware detection
- Network Programming: Packet filtering, load balancing, XDP (eXpress Data Path)
- Production Deployment: Monitoring Raspberry Pi clusters and IoT devices
- Raspberry Pi Optimization: ARM64-specific considerations and performance tuning
Perfect for:
- System Administrators: Deep visibility into system behavior
- DevOps Engineers: Production monitoring and troubleshooting
- Security Researchers: Runtime security monitoring and enforcement
- Network Engineers: High-performance packet processing
- IoT Developers: Lightweight monitoring for edge devices
- Performance Engineers: Identifying bottlenecks and optimization opportunities
- Kubernetes/Container Users: Container-aware observability
Understanding eBPF Architecture
Traditional Kernel Development vs eBPF
Traditional Approach:
| ┌──────────────────────────┐
│ User Application │
└────────────┬─────────────┘
│ syscalls
▼
┌──────────────────────────┐
│ Linux Kernel │
│ │
│ ┌────────────────────┐ │
│ │ Custom Module │ │ <- Requires kernel compilation
│ │ (your code) │ │ Root access needed
│ └────────────────────┘ │ Kernel crashes possible
│ │ Hard to debug
└──────────────────────────┘
|
Problems:
- Requires kernel source modification or module compilation
- Risk of kernel crashes from bugs
- Difficult to debug and test
- Security risks (full kernel access)
- Distribution challenges (kernel version dependencies)
eBPF Approach:
| ┌──────────────────────────┐
│ User Application │
│ (writes eBPF program) │
└────────────┬─────────────┘
│ bpf() syscall
▼
┌──────────────────────────┐
│ Linux Kernel │
│ │
│ ┌────────────────────┐ │
│ │ eBPF Verifier │ │ <- Safety checks
│ └──────────┬─────────┘ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ JIT Compiler │ │ <- Compile to native code
│ └──────────┬─────────┘ │
│ ▼ │
│ ┌────────────────────┐ │
│ │ eBPF VM (runs in │ │ <- Sandboxed execution
│ │ kernel context) │ │ Cannot crash kernel
│ └────────────────────┘ │ Verified safe
│ │
│ Attach points: │
│ • Kprobes/Kretprobes │
│ • Tracepoints │
│ • XDP (network) │
│ • Cgroups │
│ • Perf events │
└──────────────────────────┘
|
Advantages:
- Safe: Verifier ensures programs cannot crash kernel
- No Compilation: No kernel rebuild required
- Dynamic: Load/unload programs at runtime
- Portable: Works across kernel versions (within limits)
- Efficient: JIT-compiled to native machine code
- Non-Intrusive: No kernel source modifications
eBPF Program Lifecycle
| 1. Write eBPF Program (C or bpftrace syntax)
↓
2. Compile to eBPF bytecode
↓
3. Load into kernel via bpf() syscall
↓
4. Verifier checks safety:
- No infinite loops
- No out-of-bounds memory access
- No unsafe pointer operations
- Stack size limits
↓
5. JIT Compile to ARM64 machine code (on Raspberry Pi)
↓
6. Attach to kernel hook point:
- kprobe (any kernel function)
- tracepoint (stable kernel events)
- XDP (network ingress)
- tc (traffic control)
- cgroup (container events)
↓
7. Execute on events
↓
8. Collect data via:
- Maps (key-value store shared with userspace)
- Perf events
- Ring buffers
↓
9. User program reads results
|
eBPF Maps: Kernel-Userspace Communication
| ┌─────────────────────────────────────┐
│ Kernel Space │
│ │
│ eBPF Program → BPF_MAP_UPDATE │
│ ↓ │
│ ┌─────────┐ │
│ │ eBPF Map│ │
│ │(shared) │ │
│ └─────────┘ │
│ ↑ │
└──────────────────────┼──────────────┘
│
┌──────────────────────┼──────────────┐
│ User Space │ │
│ │ │
│ User Program → BPF_MAP_LOOKUP │
│ │
└─────────────────────────────────────┘
Map Types:
- BPF_MAP_TYPE_HASH: Hash table
- BPF_MAP_TYPE_ARRAY: Array (fixed size)
- BPF_MAP_TYPE_PERF_EVENT_ARRAY: Performance events
- BPF_MAP_TYPE_RINGBUF: Ring buffer (efficient)
- BPF_MAP_TYPE_LRU_HASH: LRU cache
- BPF_MAP_TYPE_STACK_TRACE: Stack traces
|
Check Kernel Support
| # Check kernel version (need 4.x+ for basic eBPF, 5.x+ recommended)
uname -r
# Should show: 6.1.x or newer on Raspberry Pi OS
# Check if eBPF is enabled
cat /boot/firmware/config.txt | grep -i bpf
zcat /proc/config.gz | grep -i bpf
# Should show: CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y
# Verify BPF JIT compiler
sudo sysctl net.core.bpf_jit_enable
# Should show: net.core.bpf_jit_enable = 1
# Check available eBPF features
sudo bpftool feature probe
|
Install BCC (BPF Compiler Collection)
| # Update system
sudo apt update
sudo apt upgrade -y
# Install dependencies
sudo apt install -y \
build-essential \
cmake \
python3-dev \
python3-pip \
linux-headers-$(uname -r) \
libelf-dev \
llvm \
clang \
libbpf-dev \
bpfcc-tools \
linux-headers-generic
# Install BCC tools (may take 30-60 minutes on Raspberry Pi)
sudo apt install -y bpfcc-tools python3-bpfcc
# Verify installation
ls /usr/share/bcc/tools/
# Should show many tools: execsnoop, opensnoop, tcpconnect, etc.
|
Install bpftrace
| # Install bpftrace (high-level tracing language)
sudo apt install -y bpftrace
# Verify
bpftrace --version
# Should show: bpftrace v0.x.x
# Test with simple one-liner
sudo bpftrace -e 'BEGIN { printf("Hello, eBPF!\n"); exit(); }'
# Should print: Hello, eBPF!
|
| # Install libbpf and development headers
sudo apt install -y libbpf-dev
# Install bpftool (BPF introspection and debugging)
sudo apt install -y linux-tools-common linux-tools-generic
# Or compile from source for latest version
# Verify bpftool
sudo bpftool version
sudo bpftool prog list
sudo bpftool map list
|
Optional: Install from Source (Latest Versions)
| # For cutting-edge features, compile from source
# Warning: Takes 1-2 hours on Raspberry Pi
# Install more dependencies
sudo apt install -y \
libdebuginfod-dev \
libcap-dev \
binutils-dev
# Clone BCC
git clone https://github.com/iovisor/bcc.git
cd bcc
mkdir build && cd build
# Configure and build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr
make -j$(nproc)
sudo make install
# Clone bpftrace
cd ~
git clone https://github.com/iovisor/bpftrace
cd bpftrace
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)
sudo make install
|
BCC includes dozens of pre-built tools for common monitoring tasks.
Process and CPU Monitoring
execsnoop - Monitor Process Execution:
| # Watch all new processes
sudo /usr/share/bcc/tools/execsnoop
# Sample output:
# PCOMM PID PPID RET ARGS
# ls 12345 1234 0 /bin/ls -la
# python3 12346 1234 0 /usr/bin/python3 script.py
# grep 12347 12346 0 /bin/grep error log.txt
# Filter by specific command
sudo /usr/share/bcc/tools/execsnoop -n python3
# Include timestamps
sudo /usr/share/bcc/tools/execsnoop -t
|
cpudist - CPU Run Queue Latency:
| # Measure time processes spend waiting for CPU
sudo /usr/share/bcc/tools/cpudist
# Sample output:
# usecs : count distribution
# 0 -> 1 : 234 |***** |
# 2 -> 3 : 891 |******************** |
# 4 -> 7 : 1023 |************************|
# 8 -> 15 : 456 |********** |
# 16 -> 31 : 123 |*** |
# Monitor for 10 seconds
sudo /usr/share/bcc/tools/cpudist 10 1
|
profile - CPU Profiling:
| # Sample CPU stack traces at 49 Hz
sudo /usr/share/bcc/tools/profile -F 49
# Profile for 30 seconds
sudo /usr/share/bcc/tools/profile -F 99 30
# Generate flame graph data
sudo /usr/share/bcc/tools/profile -F 99 -f 30 > profile.txt
# Filter specific process
sudo /usr/share/bcc/tools/profile -p $(pgrep python3)
|
File System Monitoring
opensnoop - Monitor File Opens:
| # Watch all file open operations
sudo /usr/share/bcc/tools/opensnoop
# Sample output:
# PID COMM FD ERR PATH
# 12345 python3 3 0 /etc/passwd
# 12346 nginx 4 0 /var/log/nginx/access.log
# 12347 mysql -1 2 /tmp/mysql.sock (No such file)
# Filter by process name
sudo /usr/share/bcc/tools/opensnoop -n nginx
# Filter by PID
sudo /usr/share/bcc/tools/opensnoop -p 1234
# Show failed opens only
sudo /usr/share/bcc/tools/opensnoop -x
|
filetop - File I/O by Process:
| # Show top processes by file I/O
sudo /usr/share/bcc/tools/filetop
# Sample output:
# TID COMM READS WRITES R_Kb W_Kb T FILE
# 12345 python3 125 0 512 0 R /var/log/app.log
# 12346 postgres 89 234 356 1024 W /var/lib/postgres/data
# 12347 nginx 456 123 2048 512 R /var/www/html/index.html
# Update every 5 seconds
sudo /usr/share/bcc/tools/filetop 5
|
filelife - File Lifetime:
| # Track short-lived files (temp files, etc.)
sudo /usr/share/bcc/tools/filelife
# Sample output:
# TIME PID COMM AGE(s) FILE
# 12:34:56 12345 python3 0.123 /tmp/tmpfile_abc123
# 12:34:57 12346 make 0.456 /tmp/ccXYZ.o
# 12:34:58 12347 bash 0.089 /tmp/sh-thd.xyz
|
Network Monitoring
tcpconnect - TCP Connection Tracing:
| # Monitor all outgoing TCP connections
sudo /usr/share/bcc/tools/tcpconnect
# Sample output:
# PID COMM IP SADDR DADDR DPORT
# 12345 curl 4 192.168.1.10 93.184.216.34 80
# 12346 ssh 4 192.168.1.10 192.168.1.100 22
# 12347 python3 4 192.168.1.10 8.8.8.8 443
# Filter by port
sudo /usr/share/bcc/tools/tcpconnect -P 443
# Count connections
sudo /usr/share/bcc/tools/tcpconnect -c
|
tcpaccept - TCP Server Connection Acceptance:
| # Monitor incoming TCP connections
sudo /usr/share/bcc/tools/tcpaccept
# Sample output:
# PID COMM IP RADDR RPORT LADDR LPORT
# 12345 nginx 4 192.168.1.50 54321 192.168.1.10 80
# 12346 sshd 4 192.168.1.60 51234 192.168.1.10 22
# 12347 python3 4 192.168.1.70 48765 192.168.1.10 8080
|
tcpretrans - TCP Retransmission Tracing:
| # Detect network quality issues
sudo /usr/share/bcc/tools/tcpretrans
# Sample output:
# TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE
# 12:34:56 12345 4 192.168.1.10:54321 R> 93.184.216.34:80 ESTABLISHED
# 12:34:57 12346 4 192.168.1.10:12345 R> 8.8.8.8:443 ESTABLISHED
# Retransmissions indicate:
# - Network congestion
# - Packet loss
# - Poor WiFi signal
# - Network misconfiguration
|
tcptop - TCP Traffic Summary:
| # Show TCP send/receive by process
sudo /usr/share/bcc/tools/tcptop
# Sample output:
# PID COMM LADDR RADDR RX_KB TX_KB
# 12345 firefox 192.168.1.10:54321 93.184.216.34:443 1234 56
# 12346 nginx 192.168.1.10:80 192.168.1.50:12345 89 5678
# 12347 rclone 192.168.1.10:44123 142.250.80.78:443 9876 543
# Update every 5 seconds
sudo /usr/share/bcc/tools/tcptop 5
|
Memory Monitoring
memleak - Memory Leak Detection:
| # Detect memory leaks in running processes
sudo /usr/share/bcc/tools/memleak -p $(pgrep python3)
# Sample output (after 10 seconds):
# [12:34:56] Top 10 stacks with outstanding allocations:
# addr = 7f1234567890 size = 4096
# addr = 7f1234568000 size = 8192
# ...
# 1234 bytes in 56 allocations from stack
# 0x00007f123456789a malloc+0x1a
# 0x00007f12345678bc my_function+0x2c
# ...
# Monitor for 60 seconds
sudo /usr/share/bcc/tools/memleak -p $(pgrep python3) 60
|
cachestat - Page Cache Statistics:
| # Monitor page cache hit/miss ratio
sudo /usr/share/bcc/tools/cachestat
# Sample output:
# HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
# 1234 56 89 95.65% 123 456
# 2345 67 91 97.22% 124 458
# 3456 78 92 97.79% 125 460
# Good hit ratio (>90%) = efficient caching
# Low hit ratio = consider more RAM or optimization
|
Disk I/O Monitoring
biolatency - Block I/O Latency:
| # Measure disk I/O latency distribution
sudo /usr/share/bcc/tools/biolatency
# Sample output:
# usecs : count distribution
# 0 -> 1 : 0 | |
# 2 -> 3 : 12 |* |
# 4 -> 7 : 234 |****************** |
# 8 -> 15 : 345 |************************|
# 16 -> 31 : 123 |********* |
# 32 -> 63 : 45 |*** |
# Monitor specific disk
sudo /usr/share/bcc/tools/biolatency -D /dev/mmcblk0
# Millisecond histograms
sudo /usr/share/bcc/tools/biolatency -m
|
biotop - Block I/O Top:
| # Top processes by disk I/O
sudo /usr/share/bcc/tools/biotop
# Sample output:
# PID COMM D MAJ MIN DISK I/O Kbytes AVGms
# 12345 postgres R 179 0 mmcblk0 123 4096 8.45
# 12346 python3 W 179 0 mmcblk0 89 2048 12.34
# 12347 rsync R 8 0 sda 456 16384 5.67
|
biosnoop - Individual Block I/O Events:
| # Trace every disk I/O operation
sudo /usr/share/bcc/tools/biosnoop
# Sample output:
# TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms)
# 0.000000 postgres 12345 mmcblk0 W 12345678 4096 8.45
# 0.012345 python3 12346 mmcblk0 R 23456789 8192 6.78
# 0.023456 rsync 12347 sda R 34567890 16384 4.56
# Filter by process
sudo /usr/share/bcc/tools/biosnoop | grep python3
|
bpftrace: High-Level Tracing Language
bpftrace provides a concise, awk-like syntax for writing eBPF programs.
Basic Syntax
| # General format
bpftrace -e 'probe_type:probe_name /filter/ { actions }'
# BEGIN/END probes
sudo bpftrace -e 'BEGIN { printf("Starting...\n"); }
END { printf("Exiting...\n"); }'
# Interval probe (every second)
sudo bpftrace -e 'interval:s:1 { printf("tick\n"); }'
# Kprobe (kernel function entry)
sudo bpftrace -e 'kprobe:do_sys_open { printf("File opened\n"); }'
# Kretprobe (kernel function return)
sudo bpftrace -e 'kretprobe:do_sys_open { printf("Return value: %d\n", retval); }'
# Tracepoint
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("openat() called\n"); }'
|
Practical bpftrace One-Liners
System Call Counting:
| # Count syscalls by type
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[args->id] = count(); }'
# Count syscalls by process name
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# Count read() syscalls per second
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @ = count(); }
interval:s:1 { print(@); clear(@); }'
|
Process Monitoring:
| # Trace new processes with arguments
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
printf("%s: %s\n", comm, str(args->filename));
}'
# Process lifetime (creation to exit)
sudo bpftrace -e 'tracepoint:sched:sched_process_fork { @start[args->child_pid] = nsecs; }
tracepoint:sched:sched_process_exit /@start[pid]/ {
printf("%s (PID %d) lifetime: %d ms\n",
comm, pid, (nsecs - @start[pid]) / 1000000);
delete(@start[pid]);
}'
|
File I/O Tracing:
| # Files opened by process
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
printf("%s opened: %s\n", comm, str(args->filename));
}'
# Read/write size distribution
sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read /args->ret > 0/ {
@read_bytes = hist(args->ret);
}
tracepoint:syscalls:sys_exit_write /args->ret > 0/ {
@write_bytes = hist(args->ret);
}'
# Slow file operations (>10ms)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /@start[tid]/ {
$dur = nsecs - @start[tid];
if ($dur > 10000000) {
printf("Slow read: %d ms\n", $dur / 1000000);
}
delete(@start[tid]);
}'
|
Network Tracing:
| # TCP connections by destination IP
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect {
printf("%s connecting to %s\n", comm, ntop(args->addr));
}'
# TCP send/receive size
sudo bpftrace -e 'kprobe:tcp_sendmsg { @send = hist(arg2); }
kprobe:tcp_recvmsg { @recv = hist(arg2); }'
# Network latency (DNS queries)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_sendto /comm == "systemd-resolve"/ {
@start[tid] = nsecs;
}
tracepoint:syscalls:sys_exit_recvfrom /@start[tid]/ {
printf("DNS query latency: %d ms\n", (nsecs - @start[tid]) / 1000000);
delete(@start[tid]);
}'
|
CPU Profiling:
| # Sample on-CPU stack traces
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'
# Sample user + kernel stacks
sudo bpftrace -e 'profile:hz:99 { @[ustack, kstack] = count(); }'
# CPU time by process
sudo bpftrace -e 'profile:hz:99 { @[comm] = count(); }'
# CPU time by function (top kernel functions)
sudo bpftrace -e 'profile:hz:99 { @[kfunc] = count(); }'
|
Memory Allocation:
| # Track memory allocations
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @bytes = sum(args->bytes_alloc); }'
# Allocation size distribution
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @alloc_size = hist(args->bytes_alloc); }'
# Top allocating processes
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @[comm] = sum(args->bytes_alloc); }'
|
Complete bpftrace Scripts
TCP Connection Monitor:
| sudo tee tcp_monitor.bt <<'EOF'
#!/usr/bin/env bpftrace
BEGIN {
printf("Monitoring TCP connections... Hit Ctrl-C to end.\n");
printf("%-8s %-16s %-6s %-16s %-16s %-6s\n",
"TIME", "COMM", "PID", "SADDR", "DADDR", "DPORT");
}
tracepoint:syscalls:sys_enter_connect /args->family == AF_INET/ {
@connect[tid] = args;
}
tracepoint:syscalls:sys_exit_connect /@connect[tid]/ {
$args = @connect[tid];
$sa = (struct sockaddr_in *)$args->addr;
time("%H:%M:%S ");
printf("%-16s %-6d %-16s %-16s %-6d\n",
comm, pid,
ntop(AF_INET, 0), // Local IP (simplified)
ntop(AF_INET, $sa->sin_addr.s_addr),
($sa->sin_port >> 8) | (($sa->sin_port << 8) & 0xff00));
delete(@connect[tid]);
}
END {
clear(@connect);
}
EOF
chmod +x tcp_monitor.bt
sudo ./tcp_monitor.bt
|
Slow I/O Detector:
| sudo tee slow_io.bt <<'EOF'
#!/usr/bin/env bpftrace
BEGIN {
printf("Detecting slow I/O (>50ms)...\n");
}
tracepoint:syscalls:sys_enter_read,
tracepoint:syscalls:sys_enter_write {
@start[tid] = nsecs;
@fd[tid] = args->fd;
}
tracepoint:syscalls:sys_exit_read,
tracepoint:syscalls:sys_exit_write
/@start[tid]/ {
$duration_ms = (nsecs - @start[tid]) / 1000000;
if ($duration_ms > 50) {
time("%H:%M:%S ");
printf("%s (PID %d) slow I/O on fd %d: %d ms\n",
comm, pid, @fd[tid], $duration_ms);
}
delete(@start[tid]);
delete(@fd[tid]);
}
END {
clear(@start);
clear(@fd);
}
EOF
chmod +x slow_io.bt
sudo ./slow_io.bt
|
Writing Custom eBPF Programs with BCC
Simple Python BCC Example
Count Syscalls:
| #!/usr/bin/env python3
# syscall_counter.py
from bcc import BPF
from time import sleep
# eBPF program (C code)
bpf_program = """
#include <uapi/linux/ptrace.h>
BPF_HASH(syscall_count, u64);
TRACEPOINT_PROBE(raw_syscalls, sys_enter) {
u64 syscall_id = args->id;
u64 *count = syscall_count.lookup(&syscall_id);
if (count) {
(*count)++;
} else {
u64 initial = 1;
syscall_count.update(&syscall_id, &initial);
}
return 0;
}
"""
# Load eBPF program
b = BPF(text=bpf_program)
print("Counting syscalls... Hit Ctrl-C to stop.")
try:
while True:
sleep(5)
print("\n=== Top 10 Syscalls (last 5 seconds) ===")
syscall_count = b["syscall_count"]
# Get top 10
items = sorted(syscall_count.items(), key=lambda x: x[1].value, reverse=True)[:10]
for syscall_id, count in items:
print(f"Syscall {syscall_id.value}: {count.value} calls")
# Clear for next interval
syscall_count.clear()
except KeyboardInterrupt:
print("\nExiting...")
|
Run:
| sudo python3 syscall_counter.py
|
Network Packet Filter
Track TCP Connections:
| #!/usr/bin/env python3
# tcp_tracker.py
from bcc import BPF
import socket
import struct
bpf_program = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>
BPF_PERF_OUTPUT(events);
struct connection_event {
u32 pid;
char comm[16];
u32 saddr;
u32 daddr;
u16 sport;
u16 dport;
};
int trace_connect(struct pt_regs *ctx, struct sock *sk) {
struct connection_event evt = {};
evt.pid = bpf_get_current_pid_tgid() >> 32;
bpf_get_current_comm(&evt.comm, sizeof(evt.comm));
u16 family = sk->__sk_common.skc_family;
if (family == AF_INET) {
evt.saddr = sk->__sk_common.skc_rcv_saddr;
evt.daddr = sk->__sk_common.skc_daddr;
evt.sport = sk->__sk_common.skc_num;
evt.dport = sk->__sk_common.skc_dport;
evt.dport = ntohs(evt.dport);
events.perf_submit(ctx, &evt, sizeof(evt));
}
return 0;
}
"""
def print_event(cpu, data, size):
event = b["events"].event(data)
saddr = socket.inet_ntoa(struct.pack("I", event.saddr))
daddr = socket.inet_ntoa(struct.pack("I", event.daddr))
print(f"{event.comm.decode('utf-8', 'ignore'):16s} "
f"PID {event.pid:6d} "
f"{saddr}:{event.sport} -> {daddr}:{event.dport}")
# Load BPF program
b = BPF(text=bpf_program)
b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect")
print("Monitoring TCP connections... Hit Ctrl-C to stop.")
print(f"{'COMM':<16} {'PID':<6} {'SOURCE':>21} -> {'DEST'}")
# Poll events
b["events"].open_perf_buffer(print_event)
try:
while True:
b.perf_buffer_poll()
except KeyboardInterrupt:
print("\nExiting...")
|
Function Latency Tracker:
| #!/usr/bin/env python3
# function_latency.py
from bcc import BPF
import sys
if len(sys.argv) < 2:
print(f"Usage: {sys.argv[0]} <function_name>")
sys.exit(1)
function_name = sys.argv[1]
# BPF program template (with function name substitution)
bpf_program = """
#include <uapi/linux/ptrace.h>
BPF_HASH(start_time, u32);
BPF_HISTOGRAM(latency_hist);
int trace_start(struct pt_regs *ctx) {
u32 pid = bpf_get_current_pid_tgid();
u64 ts = bpf_ktime_get_ns();
start_time.update(&pid, &ts);
return 0;
}
int trace_return(struct pt_regs *ctx) {
u32 pid = bpf_get_current_pid_tgid();
u64 *tsp = start_time.lookup(&pid);
if (tsp != 0) {
u64 delta = bpf_ktime_get_ns() - *tsp;
latency_hist.increment(bpf_log2l(delta / 1000)); // microseconds
start_time.delete(&pid);
}
return 0;
}
"""
b = BPF(text=bpf_program)
b.attach_kprobe(event=function_name, fn_name="trace_start")
b.attach_kretprobe(event=function_name, fn_name="trace_return")
print(f"Tracing {function_name} latency... Hit Ctrl-C to show results.")
try:
import time
time.sleep(999999999)
except KeyboardInterrupt:
pass
print(f"\nLatency distribution for {function_name}:")
b["latency_hist"].print_log2_hist("usecs")
|
Run:
| # Profile do_sys_open() latency
sudo python3 function_latency.py do_sys_open
|
XDP allows packet processing directly in the network driver, before the kernel network stack.
Simple XDP Packet Filter
Drop ICMP Packets:
| // xdp_drop_icmp.c
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/icmp.h>
#define SEC(NAME) __attribute__((section(NAME), used))
SEC("xdp")
int xdp_drop_icmp_prog(struct xdp_md *ctx) {
void *data_end = (void *)(long)ctx->data_end;
void *data = (void *)(long)ctx->data;
// Ethernet header
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
return XDP_PASS;
// Only IPv4
if (eth->h_proto != __constant_htons(ETH_P_IP))
return XDP_PASS;
// IP header
struct iphdr *ip = (void *)(eth + 1);
if ((void *)(ip + 1) > data_end)
return XDP_PASS;
// Drop ICMP packets
if (ip->protocol == IPPROTO_ICMP)
return XDP_DROP;
return XDP_PASS;
}
char _license[] SEC("license") = "GPL";
|
Compile and Load:
| # Compile XDP program
clang -O2 -target bpf -c xdp_drop_icmp.c -o xdp_drop_icmp.o
# Load onto interface
sudo ip link set dev eth0 xdp obj xdp_drop_icmp.o sec xdp
# Verify
sudo ip link show dev eth0
# Should show: xdp
# Test (from another machine)
ping <raspberry-pi-ip>
# Should timeout (ICMP dropped)
# Unload
sudo ip link set dev eth0 xdp off
|
XDP Packet Counter
| #!/usr/bin/env python3
# xdp_packet_counter.py
from bcc import BPF
xdp_program = """
#include <uapi/linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
BPF_ARRAY(packet_count, u64, 1);
BPF_ARRAY(byte_count, u64, 1);
int xdp_count(struct xdp_md *ctx) {
void *data_end = (void *)(long)ctx->data_end;
void *data = (void *)(long)ctx->data;
u32 index = 0;
u64 *pkt_cnt = packet_count.lookup(&index);
u64 *byte_cnt = byte_count.lookup(&index);
if (pkt_cnt && byte_cnt) {
__sync_fetch_and_add(pkt_cnt, 1);
__sync_fetch_and_add(byte_cnt, data_end - data);
}
return XDP_PASS;
}
"""
# Load program
b = BPF(text=xdp_program)
fn = b.load_func("xdp_count", BPF.XDP)
# Attach to interface
interface = "eth0"
b.attach_xdp(interface, fn, 0)
print(f"Counting packets on {interface}... Hit Ctrl-C to stop.")
try:
import time
while True:
time.sleep(1)
pkt_cnt = b["packet_count"][0].value
byte_cnt = b["byte_count"][0].value
print(f"Packets: {pkt_cnt:,} Bytes: {byte_cnt:,} "
f"({byte_cnt / 1024 / 1024:.2f} MB)")
except KeyboardInterrupt:
pass
# Cleanup
b.remove_xdp(interface, 0)
print("\nDetached XDP program.")
|
Security Monitoring with eBPF
Detect Suspicious Syscalls
| #!/usr/bin/env python3
# security_monitor.py
from bcc import BPF
bpf_program = """
#include <uapi/linux/ptrace.h>
// Suspicious syscalls that might indicate malicious activity
BPF_HASH(suspicious_calls, u32);
TRACEPOINT_PROBE(syscalls, sys_enter_ptrace) {
u32 pid = bpf_get_current_pid_tgid() >> 32;
u64 count = 0;
u64 *val = suspicious_calls.lookup(&pid);
if (val) count = *val;
count++;
suspicious_calls.update(&pid, &count);
return 0;
}
TRACEPOINT_PROBE(syscalls, sys_enter_kill) {
// Log kill attempts
u32 target_pid = args->pid;
int signal = args->sig;
if (signal == 9) { // SIGKILL
bpf_trace_printk("SIGKILL to PID %d\\n", target_pid);
}
return 0;
}
TRACEPOINT_PROBE(syscalls, sys_enter_execve) {
char comm[16];
bpf_get_current_comm(&comm, sizeof(comm));
// Detect common reverse shell patterns
char filename[256];
bpf_probe_read_user_str(&filename, sizeof(filename), args->filename);
if (filename[0] == '/' && filename[1] == 'd' && filename[2] == 'e' && filename[3] == 'v') {
// Execution from /dev (suspicious)
bpf_trace_printk("Suspicious exec from /dev: %s\\n", filename);
}
return 0;
}
"""
b = BPF(text=bpf_program)
print("Security monitoring active...")
print("Watching for:")
print(" - ptrace() usage (debugger attachment)")
print(" - SIGKILL signals")
print(" - Suspicious execve() patterns")
print()
# Print trace messages
b.trace_print()
|
File Integrity Monitor
| #!/usr/bin/env python3
# file_integrity_monitor.py
from bcc import BPF
import os
# Protected directories
PROTECTED_DIRS = ["/etc", "/usr/bin", "/usr/sbin", "/boot"]
bpf_program = """
#include <uapi/linux/ptrace.h>
BPF_PERF_OUTPUT(events);
struct file_event {
u32 pid;
char comm[16];
char filename[256];
u32 flags;
};
TRACEPOINT_PROBE(syscalls, sys_enter_openat) {
struct file_event evt = {};
evt.pid = bpf_get_current_pid_tgid() >> 32;
bpf_get_current_comm(&evt.comm, sizeof(evt.comm));
bpf_probe_read_user_str(&evt.filename, sizeof(evt.filename), args->filename);
evt.flags = args->flags;
// Check for write flags (O_WRONLY, O_RDWR, O_CREAT, O_TRUNC)
if ((evt.flags & 0x01) || (evt.flags & 0x02) ||
(evt.flags & 0x40) || (evt.flags & 0x200)) {
events.perf_submit(args, &evt, sizeof(evt));
}
return 0;
}
"""
def is_protected_path(path):
"""Check if path is in protected directories"""
for protected in PROTECTED_DIRS:
if path.startswith(protected):
return True
return False
def print_event(cpu, data, size):
event = b["events"].event(data)
filename = event.filename.decode('utf-8', 'ignore')
if is_protected_path(filename):
print(f"[ALERT] PID {event.pid} ({event.comm.decode('utf-8', 'ignore')}) "
f"modifying protected file: {filename}")
b = BPF(text=bpf_program)
print(f"Monitoring write access to protected directories:")
for dir in PROTECTED_DIRS:
print(f" - {dir}")
print()
b["events"].open_perf_buffer(print_event)
try:
while True:
b.perf_buffer_poll()
except KeyboardInterrupt:
print("\nExiting...")
|
Production Monitoring Dashboard
Real-Time System Metrics Collector
| #!/usr/bin/env python3
# system_metrics.py - Comprehensive system monitoring
from bcc import BPF
import time
import json
bpf_program = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
// CPU metrics
BPF_HASH(cpu_time, u32);
// Disk I/O metrics
BPF_HASH(disk_reads, u32);
BPF_HASH(disk_writes, u32);
BPF_HASH(disk_bytes_read, u32);
BPF_HASH(disk_bytes_written, u32);
// Network metrics
BPF_HASH(net_tx_packets, u32);
BPF_HASH(net_rx_packets, u32);
BPF_HASH(net_tx_bytes, u32);
BPF_HASH(net_rx_bytes, u32);
// Process metrics
BPF_HASH(process_starts, u32);
// CPU profiling
TRACEPOINT_PROBE(sched, sched_switch) {
u32 pid = args->next_pid;
u64 ts = bpf_ktime_get_ns();
cpu_time.update(&pid, &ts);
return 0;
}
// Disk I/O
TRACEPOINT_PROBE(block, block_rq_complete) {
u32 pid = bpf_get_current_pid_tgid() >> 32;
u64 bytes = args->nr_sector * 512;
if (args->rwbs[0] == 'R') {
disk_reads.increment(pid);
u64 *val = disk_bytes_read.lookup(&pid);
u64 total = val ? *val : 0;
total += bytes;
disk_bytes_read.update(&pid, &total);
} else if (args->rwbs[0] == 'W') {
disk_writes.increment(pid);
u64 *val = disk_bytes_written.lookup(&pid);
u64 total = val ? *val : 0;
total += bytes;
disk_bytes_written.update(&pid, &total);
}
return 0;
}
// Process creation
TRACEPOINT_PROBE(sched, sched_process_fork) {
u32 zero = 0;
process_starts.increment(zero);
return 0;
}
"""
class SystemMetrics:
def __init__(self):
self.b = BPF(text=bpf_program)
self.last_time = time.time()
def collect(self):
now = time.time()
interval = now - self.last_time
self.last_time = now
metrics = {
'timestamp': now,
'interval': interval,
'disk': self._get_disk_metrics(),
'processes': self._get_process_metrics(),
}
return metrics
def _get_disk_metrics(self):
reads = sum(self.b["disk_reads"].values())
writes = sum(self.b["disk_writes"].values())
bytes_read = sum(v.value for v in self.b["disk_bytes_read"].values())
bytes_written = sum(v.value for v in self.b["disk_bytes_written"].values())
return {
'reads': reads,
'writes': writes,
'bytes_read': bytes_read,
'bytes_written': bytes_written,
'mb_read': bytes_read / 1024 / 1024,
'mb_written': bytes_written / 1024 / 1024,
}
def _get_process_metrics(self):
starts = sum(self.b["process_starts"].values())
return {'starts': starts}
def clear(self):
"""Clear counters for next interval"""
self.b["disk_reads"].clear()
self.b["disk_writes"].clear()
self.b["disk_bytes_read"].clear()
self.b["disk_bytes_written"].clear()
self.b["process_starts"].clear()
if __name__ == '__main__':
monitor = SystemMetrics()
print("Collecting system metrics... Hit Ctrl-C to stop.")
print()
try:
while True:
time.sleep(5)
metrics = monitor.collect()
print(f"=== Metrics (last {metrics['interval']:.1f}s) ===")
print(f"Disk Reads: {metrics['disk']['reads']:,} "
f"({metrics['disk']['mb_read']:.2f} MB)")
print(f"Disk Writes: {metrics['disk']['writes']:,} "
f"({metrics['disk']['mb_written']:.2f} MB)")
print(f"Process Starts: {metrics['disk']['starts']}")
print()
# Could export to Prometheus, InfluxDB, etc.
# export_to_prometheus(metrics)
monitor.clear()
except KeyboardInterrupt:
print("\nExiting...")
|
Raspberry Pi Specific Optimizations
ARM64 Considerations
| # Check eBPF JIT for ARM64
sudo sysctl net.core.bpf_jit_enable
# Should be 1 (enabled)
# View JIT compiled code
sudo sysctl net.core.bpf_jit_harden
# 0 = no hardening (faster)
# 1 = basic hardening
# 2 = full hardening (slower)
# For production, use hardening level 1
sudo sysctl -w net.core.bpf_jit_harden=1
|
Memory Optimization
| # eBPF programs use locked memory
# Check current limit
ulimit -l
# Increase if needed (in /etc/security/limits.conf)
echo "* soft memlock unlimited" | sudo tee -a /etc/security/limits.conf
echo "* hard memlock unlimited" | sudo tee -a /etc/security/limits.conf
# Or temporarily
sudo ulimit -l unlimited
|
| # Use efficient map types
# BPF_HASH (hash table) - flexible but slower
# BPF_ARRAY (array) - fast but fixed size
# BPF_LRU_HASH (LRU cache) - automatic eviction
# BPF_RINGBUF (ring buffer) - most efficient for event streaming
# Prefer ring buffers over perf buffers on Pi 4/5
BPF_RINGBUF_OUTPUT(events, 1024); # 1024 pages
# Minimize per-event overhead
# Avoid string operations in hot paths
# Use maps for aggregation, not individual events
|
Troubleshooting
Common Issues
1. "Operation not permitted"
| # Need root or CAP_BPF/CAP_PERFMON capabilities
sudo bpftrace ...
# Or add capabilities
sudo setcap cap_bpf,cap_perfmon+ep /usr/bin/bpftrace
|
2. "Failed to load program: Invalid argument"
| # Check kernel version
uname -r
# Verify eBPF support
zcat /proc/config.gz | grep BPF
# Check dmesg for verifier errors
sudo dmesg | tail -50
|
3. "Cannot allocate memory"
| # Increase memlock limit
ulimit -l unlimited
# Or in /etc/security/limits.conf
|
4. "Unknown kprobe"
| # Function doesn't exist or not exposed
# Check available kprobes
sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep <function>
# Use tracepoint instead
sudo bpftrace -l 'tracepoint:*' | grep <topic>
|
Debugging eBPF Programs
| # Enable debug output
sudo bpftrace -v -e '...'
# Check verifier log
sudo bpftool prog load program.o /sys/fs/bpf/myprog 2>&1 | less
# Trace BPF events
sudo bpftrace -e 'tracepoint:bpf:* { printf("%s\n", probe); }'
# View loaded programs
sudo bpftool prog list
# View program details
sudo bpftool prog show id <id>
# Dump JIT compiled code
sudo bpftool prog dump jited id <id>
|
Summary
This comprehensive guide covered eBPF programming on Raspberry Pi:
✅ Core Concepts
- eBPF architecture and kernel integration
- Safety guarantees through verifier
- JIT compilation to ARM64 native code
- Maps for kernel-userspace communication
- BCC: Full-featured Python/C framework with 50+ ready tools
- bpftrace: High-level tracing language for one-liners
- libbpf: Modern C library for production applications
- bpftool: Introspection and debugging utilities
✅ Ready-to-Use Monitoring
- execsnoop, opensnoop, tcpconnect, tcpretrans
- biolatency, biotop, filetop, cachestat
- cpudist, profile, memleak
- Network, disk, CPU, memory visibility
✅ Custom Development
- Python BCC programming examples
- bpftrace scripting for custom metrics
- XDP for high-performance packet processing
- Security monitoring and intrusion detection
✅ Production Use Cases
- Real-time performance monitoring
- Security threat detection
- Network traffic analysis
- Container and Kubernetes observability
- IoT device monitoring
✅ Raspberry Pi Optimization
- ARM64 JIT compilation
- Memory limit configuration
- Efficient map types for limited resources
- Ring buffers for event streaming
- Minimal overhead (~1-5% CPU)
- Kernel-level visibility without instrumentation
- Zero application code changes required
- Nanosecond-precision timing
Next Steps
Advanced Topics:
- eBPF for Kubernetes - Monitor containers and pods
- Distributed Tracing - Track requests across services
- Custom XDP Programs - DDoS protection, load balancing
- eBPF LSM (Linux Security Module) - Runtime security policies
- CO-RE (Compile Once, Run Everywhere) - Portable eBPF programs
Related Guides:
With eBPF, your Raspberry Pi gains enterprise-grade observability, security monitoring, and performance analysis capabilities - all without kernel modifications or application changes!