Skip to content

eBPF (Extended Berkeley Packet Filter) - Kernel Programming for Raspberry Pi

Introduction

eBPF (Extended Berkeley Packet Filter) is a revolutionary Linux kernel technology that allows you to run sandboxed programs directly in the kernel without changing kernel source code or loading kernel modules. Originally designed for packet filtering, eBPF has evolved into a powerful platform for observability, security, networking, and performance analysis.

With eBPF, you can:

  • Monitor System Performance: Track CPU, memory, disk I/O, and network in real-time
  • Network Packet Analysis: Capture, filter, and analyze network traffic at kernel level
  • Security Monitoring: Detect malicious behavior, syscall anomalies, and intrusions
  • Dynamic Tracing: Profile application performance without recompilation
  • Custom Networking: Implement load balancers, firewalls, and DDoS protection
  • Resource Control: Fine-grained control over process behavior and resource usage

This comprehensive guide covers:

  • eBPF Architecture: How eBPF works in the Linux kernel
  • Development Tools: BCC, bpftrace, libbpf, and modern eBPF toolchains
  • Practical Examples: Network monitoring, system tracing, security enforcement
  • Performance Analysis: CPU profiling, I/O analysis, latency tracking
  • Security Applications: Intrusion detection, syscall filtering, malware detection
  • Network Programming: Packet filtering, load balancing, XDP (eXpress Data Path)
  • Production Deployment: Monitoring Raspberry Pi clusters and IoT devices
  • Raspberry Pi Optimization: ARM64-specific considerations and performance tuning

Perfect for:

  • System Administrators: Deep visibility into system behavior
  • DevOps Engineers: Production monitoring and troubleshooting
  • Security Researchers: Runtime security monitoring and enforcement
  • Network Engineers: High-performance packet processing
  • IoT Developers: Lightweight monitoring for edge devices
  • Performance Engineers: Identifying bottlenecks and optimization opportunities
  • Kubernetes/Container Users: Container-aware observability

Understanding eBPF Architecture

Traditional Kernel Development vs eBPF

Traditional Approach:

┌──────────────────────────┐
│   User Application       │
└────────────┬─────────────┘
             │ syscalls
┌──────────────────────────┐
│     Linux Kernel         │
│                          │
│  ┌────────────────────┐  │
│  │  Custom Module     │  │ <- Requires kernel compilation
│  │  (your code)       │  │    Root access needed
│  └────────────────────┘  │    Kernel crashes possible
│                          │    Hard to debug
└──────────────────────────┘

Problems: - Requires kernel source modification or module compilation - Risk of kernel crashes from bugs - Difficult to debug and test - Security risks (full kernel access) - Distribution challenges (kernel version dependencies)

eBPF Approach:

┌──────────────────────────┐
│   User Application       │
│   (writes eBPF program)  │
└────────────┬─────────────┘
             │ bpf() syscall
┌──────────────────────────┐
│     Linux Kernel         │
│                          │
│  ┌────────────────────┐  │
│  │  eBPF Verifier     │  │ <- Safety checks
│  └──────────┬─────────┘  │
│             ▼             │
│  ┌────────────────────┐  │
│  │  JIT Compiler      │  │ <- Compile to native code
│  └──────────┬─────────┘  │
│             ▼             │
│  ┌────────────────────┐  │
│  │  eBPF VM (runs in  │  │ <- Sandboxed execution
│  │  kernel context)   │  │    Cannot crash kernel
│  └────────────────────┘  │    Verified safe
│                          │
│  Attach points:          │
│  • Kprobes/Kretprobes    │
│  • Tracepoints           │
│  • XDP (network)         │
│  • Cgroups               │
│  • Perf events           │
└──────────────────────────┘

Advantages: - Safe: Verifier ensures programs cannot crash kernel - No Compilation: No kernel rebuild required - Dynamic: Load/unload programs at runtime - Portable: Works across kernel versions (within limits) - Efficient: JIT-compiled to native machine code - Non-Intrusive: No kernel source modifications

eBPF Program Lifecycle

1. Write eBPF Program (C or bpftrace syntax)
2. Compile to eBPF bytecode
3. Load into kernel via bpf() syscall
4. Verifier checks safety:
   - No infinite loops
   - No out-of-bounds memory access
   - No unsafe pointer operations
   - Stack size limits
5. JIT Compile to ARM64 machine code (on Raspberry Pi)
6. Attach to kernel hook point:
   - kprobe (any kernel function)
   - tracepoint (stable kernel events)
   - XDP (network ingress)
   - tc (traffic control)
   - cgroup (container events)
7. Execute on events
8. Collect data via:
   - Maps (key-value store shared with userspace)
   - Perf events
   - Ring buffers
9. User program reads results

eBPF Maps: Kernel-Userspace Communication

┌─────────────────────────────────────┐
│         Kernel Space                │
│                                     │
│  eBPF Program → BPF_MAP_UPDATE     │
│                      ↓              │
│                 ┌─────────┐         │
│                 │ eBPF Map│         │
│                 │(shared) │         │
│                 └─────────┘         │
│                      ↑              │
└──────────────────────┼──────────────┘
┌──────────────────────┼──────────────┐
│         User Space   │              │
│                      │              │
│  User Program → BPF_MAP_LOOKUP     │
│                                     │
└─────────────────────────────────────┘

Map Types:
- BPF_MAP_TYPE_HASH: Hash table
- BPF_MAP_TYPE_ARRAY: Array (fixed size)
- BPF_MAP_TYPE_PERF_EVENT_ARRAY: Performance events
- BPF_MAP_TYPE_RINGBUF: Ring buffer (efficient)
- BPF_MAP_TYPE_LRU_HASH: LRU cache
- BPF_MAP_TYPE_STACK_TRACE: Stack traces

Installing eBPF Tools on Raspberry Pi

Check Kernel Support

# Check kernel version (need 4.x+ for basic eBPF, 5.x+ recommended)
uname -r
# Should show: 6.1.x or newer on Raspberry Pi OS

# Check if eBPF is enabled
cat /boot/firmware/config.txt | grep -i bpf
zcat /proc/config.gz | grep -i bpf
# Should show: CONFIG_BPF=y, CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y

# Verify BPF JIT compiler
sudo sysctl net.core.bpf_jit_enable
# Should show: net.core.bpf_jit_enable = 1

# Check available eBPF features
sudo bpftool feature probe

Install BCC (BPF Compiler Collection)

# Update system
sudo apt update
sudo apt upgrade -y

# Install dependencies
sudo apt install -y \
    build-essential \
    cmake \
    python3-dev \
    python3-pip \
    linux-headers-$(uname -r) \
    libelf-dev \
    llvm \
    clang \
    libbpf-dev \
    bpfcc-tools \
    linux-headers-generic

# Install BCC tools (may take 30-60 minutes on Raspberry Pi)
sudo apt install -y bpfcc-tools python3-bpfcc

# Verify installation
ls /usr/share/bcc/tools/
# Should show many tools: execsnoop, opensnoop, tcpconnect, etc.

Install bpftrace

# Install bpftrace (high-level tracing language)
sudo apt install -y bpftrace

# Verify
bpftrace --version
# Should show: bpftrace v0.x.x

# Test with simple one-liner
sudo bpftrace -e 'BEGIN { printf("Hello, eBPF!\n"); exit(); }'
# Should print: Hello, eBPF!

Install Modern eBPF Tools

# Install libbpf and development headers
sudo apt install -y libbpf-dev

# Install bpftool (BPF introspection and debugging)
sudo apt install -y linux-tools-common linux-tools-generic
# Or compile from source for latest version

# Verify bpftool
sudo bpftool version
sudo bpftool prog list
sudo bpftool map list

Optional: Install from Source (Latest Versions)

# For cutting-edge features, compile from source
# Warning: Takes 1-2 hours on Raspberry Pi

# Install more dependencies
sudo apt install -y \
    libdebuginfod-dev \
    libcap-dev \
    binutils-dev

# Clone BCC
git clone https://github.com/iovisor/bcc.git
cd bcc
mkdir build && cd build

# Configure and build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr
make -j$(nproc)
sudo make install

# Clone bpftrace
cd ~
git clone https://github.com/iovisor/bpftrace
cd bpftrace
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)
sudo make install

BCC Tools: Ready-to-Use Monitoring

BCC includes dozens of pre-built tools for common monitoring tasks.

Process and CPU Monitoring

execsnoop - Monitor Process Execution:

# Watch all new processes
sudo /usr/share/bcc/tools/execsnoop

# Sample output:
# PCOMM            PID    PPID   RET ARGS
# ls               12345  1234   0   /bin/ls -la
# python3          12346  1234   0   /usr/bin/python3 script.py
# grep             12347  12346  0   /bin/grep error log.txt

# Filter by specific command
sudo /usr/share/bcc/tools/execsnoop -n python3

# Include timestamps
sudo /usr/share/bcc/tools/execsnoop -t

cpudist - CPU Run Queue Latency:

# Measure time processes spend waiting for CPU
sudo /usr/share/bcc/tools/cpudist

# Sample output:
#      usecs               : count     distribution
#          0 -> 1          : 234      |*****                   |
#          2 -> 3          : 891      |********************    |
#          4 -> 7          : 1023     |************************|
#          8 -> 15         : 456      |**********              |
#         16 -> 31         : 123      |***                     |

# Monitor for 10 seconds
sudo /usr/share/bcc/tools/cpudist 10 1

profile - CPU Profiling:

# Sample CPU stack traces at 49 Hz
sudo /usr/share/bcc/tools/profile -F 49

# Profile for 30 seconds
sudo /usr/share/bcc/tools/profile -F 99 30

# Generate flame graph data
sudo /usr/share/bcc/tools/profile -F 99 -f 30 > profile.txt

# Filter specific process
sudo /usr/share/bcc/tools/profile -p $(pgrep python3)

File System Monitoring

opensnoop - Monitor File Opens:

# Watch all file open operations
sudo /usr/share/bcc/tools/opensnoop

# Sample output:
# PID    COMM               FD ERR PATH
# 12345  python3            3  0   /etc/passwd
# 12346  nginx              4  0   /var/log/nginx/access.log
# 12347  mysql              -1 2   /tmp/mysql.sock (No such file)

# Filter by process name
sudo /usr/share/bcc/tools/opensnoop -n nginx

# Filter by PID
sudo /usr/share/bcc/tools/opensnoop -p 1234

# Show failed opens only
sudo /usr/share/bcc/tools/opensnoop -x

filetop - File I/O by Process:

# Show top processes by file I/O
sudo /usr/share/bcc/tools/filetop

# Sample output:
# TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE
# 12345  python3          125    0      512     0       R /var/log/app.log
# 12346  postgres         89     234    356     1024    W /var/lib/postgres/data
# 12347  nginx            456    123    2048    512     R /var/www/html/index.html

# Update every 5 seconds
sudo /usr/share/bcc/tools/filetop 5

filelife - File Lifetime:

1
2
3
4
5
6
7
8
# Track short-lived files (temp files, etc.)
sudo /usr/share/bcc/tools/filelife

# Sample output:
# TIME     PID    COMM             AGE(s)  FILE
# 12:34:56 12345  python3          0.123   /tmp/tmpfile_abc123
# 12:34:57 12346  make             0.456   /tmp/ccXYZ.o
# 12:34:58 12347  bash             0.089   /tmp/sh-thd.xyz

Network Monitoring

tcpconnect - TCP Connection Tracing:

# Monitor all outgoing TCP connections
sudo /usr/share/bcc/tools/tcpconnect

# Sample output:
# PID    COMM         IP SADDR            DADDR            DPORT
# 12345  curl         4  192.168.1.10     93.184.216.34    80
# 12346  ssh          4  192.168.1.10     192.168.1.100    22
# 12347  python3      4  192.168.1.10     8.8.8.8          443

# Filter by port
sudo /usr/share/bcc/tools/tcpconnect -P 443

# Count connections
sudo /usr/share/bcc/tools/tcpconnect -c

tcpaccept - TCP Server Connection Acceptance:

1
2
3
4
5
6
7
8
# Monitor incoming TCP connections
sudo /usr/share/bcc/tools/tcpaccept

# Sample output:
# PID    COMM         IP RADDR            RPORT LADDR            LPORT
# 12345  nginx        4  192.168.1.50     54321 192.168.1.10     80
# 12346  sshd         4  192.168.1.60     51234 192.168.1.10     22
# 12347  python3      4  192.168.1.70     48765 192.168.1.10     8080

tcpretrans - TCP Retransmission Tracing:

# Detect network quality issues
sudo /usr/share/bcc/tools/tcpretrans

# Sample output:
# TIME     PID    IP LADDR:LPORT          T> RADDR:RPORT          STATE
# 12:34:56 12345  4  192.168.1.10:54321  R> 93.184.216.34:80     ESTABLISHED
# 12:34:57 12346  4  192.168.1.10:12345  R> 8.8.8.8:443          ESTABLISHED

# Retransmissions indicate:
# - Network congestion
# - Packet loss
# - Poor WiFi signal
# - Network misconfiguration

tcptop - TCP Traffic Summary:

# Show TCP send/receive by process
sudo /usr/share/bcc/tools/tcptop

# Sample output:
# PID    COMM         LADDR                 RADDR                 RX_KB  TX_KB
# 12345  firefox      192.168.1.10:54321    93.184.216.34:443     1234   56
# 12346  nginx        192.168.1.10:80       192.168.1.50:12345    89     5678
# 12347  rclone       192.168.1.10:44123    142.250.80.78:443     9876   543

# Update every 5 seconds
sudo /usr/share/bcc/tools/tcptop 5

Memory Monitoring

memleak - Memory Leak Detection:

# Detect memory leaks in running processes
sudo /usr/share/bcc/tools/memleak -p $(pgrep python3)

# Sample output (after 10 seconds):
# [12:34:56] Top 10 stacks with outstanding allocations:
#     addr = 7f1234567890 size = 4096
#     addr = 7f1234568000 size = 8192
#     ...
#     1234 bytes in 56 allocations from stack
#         0x00007f123456789a  malloc+0x1a
#         0x00007f12345678bc  my_function+0x2c
#         ...

# Monitor for 60 seconds
sudo /usr/share/bcc/tools/memleak -p $(pgrep python3) 60

cachestat - Page Cache Statistics:

# Monitor page cache hit/miss ratio
sudo /usr/share/bcc/tools/cachestat

# Sample output:
#     HITS   MISSES  DIRTIES HITRATIO   BUFFERS_MB  CACHED_MB
#     1234   56      89      95.65%     123         456
#     2345   67      91      97.22%     124         458
#     3456   78      92      97.79%     125         460

# Good hit ratio (>90%) = efficient caching
# Low hit ratio = consider more RAM or optimization

Disk I/O Monitoring

biolatency - Block I/O Latency:

# Measure disk I/O latency distribution
sudo /usr/share/bcc/tools/biolatency

# Sample output:
#      usecs               : count     distribution
#          0 -> 1          : 0        |                        |
#          2 -> 3          : 12       |*                       |
#          4 -> 7          : 234      |******************      |
#          8 -> 15         : 345      |************************|
#         16 -> 31         : 123      |*********               |
#         32 -> 63         : 45       |***                     |

# Monitor specific disk
sudo /usr/share/bcc/tools/biolatency -D /dev/mmcblk0

# Millisecond histograms
sudo /usr/share/bcc/tools/biolatency -m

biotop - Block I/O Top:

1
2
3
4
5
6
7
8
# Top processes by disk I/O
sudo /usr/share/bcc/tools/biotop

# Sample output:
# PID    COMM             D MAJ MIN  DISK       I/O  Kbytes  AVGms
# 12345  postgres         R 179 0    mmcblk0    123  4096    8.45
# 12346  python3          W 179 0    mmcblk0    89   2048    12.34
# 12347  rsync            R 8   0    sda        456  16384   5.67

biosnoop - Individual Block I/O Events:

# Trace every disk I/O operation
sudo /usr/share/bcc/tools/biosnoop

# Sample output:
# TIME(s)     COMM           PID    DISK    T SECTOR     BYTES   LAT(ms)
# 0.000000    postgres       12345  mmcblk0 W 12345678   4096    8.45
# 0.012345    python3        12346  mmcblk0 R 23456789   8192    6.78
# 0.023456    rsync          12347  sda     R 34567890   16384   4.56

# Filter by process
sudo /usr/share/bcc/tools/biosnoop | grep python3

bpftrace: High-Level Tracing Language

bpftrace provides a concise, awk-like syntax for writing eBPF programs.

Basic Syntax

# General format
bpftrace -e 'probe_type:probe_name /filter/ { actions }'

# BEGIN/END probes
sudo bpftrace -e 'BEGIN { printf("Starting...\n"); }
                  END { printf("Exiting...\n"); }'

# Interval probe (every second)
sudo bpftrace -e 'interval:s:1 { printf("tick\n"); }'

# Kprobe (kernel function entry)
sudo bpftrace -e 'kprobe:do_sys_open { printf("File opened\n"); }'

# Kretprobe (kernel function return)
sudo bpftrace -e 'kretprobe:do_sys_open { printf("Return value: %d\n", retval); }'

# Tracepoint
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("openat() called\n"); }'

Practical bpftrace One-Liners

System Call Counting:

1
2
3
4
5
6
7
8
9
# Count syscalls by type
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[args->id] = count(); }'

# Count syscalls by process name
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Count read() syscalls per second
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @ = count(); }
                  interval:s:1 { print(@); clear(@); }'

Process Monitoring:

# Trace new processes with arguments
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
    printf("%s: %s\n", comm, str(args->filename));
}'

# Process lifetime (creation to exit)
sudo bpftrace -e 'tracepoint:sched:sched_process_fork { @start[args->child_pid] = nsecs; }
                  tracepoint:sched:sched_process_exit /@start[pid]/ {
                      printf("%s (PID %d) lifetime: %d ms\n",
                             comm, pid, (nsecs - @start[pid]) / 1000000);
                      delete(@start[pid]);
                  }'

File I/O Tracing:

# Files opened by process
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%s opened: %s\n", comm, str(args->filename));
}'

# Read/write size distribution
sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read /args->ret > 0/ {
    @read_bytes = hist(args->ret);
}
tracepoint:syscalls:sys_exit_write /args->ret > 0/ {
    @write_bytes = hist(args->ret);
}'

# Slow file operations (>10ms)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
                  tracepoint:syscalls:sys_exit_read /@start[tid]/ {
                      $dur = nsecs - @start[tid];
                      if ($dur > 10000000) {
                          printf("Slow read: %d ms\n", $dur / 1000000);
                      }
                      delete(@start[tid]);
                  }'

Network Tracing:

# TCP connections by destination IP
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect {
    printf("%s connecting to %s\n", comm, ntop(args->addr));
}'

# TCP send/receive size
sudo bpftrace -e 'kprobe:tcp_sendmsg { @send = hist(arg2); }
                  kprobe:tcp_recvmsg { @recv = hist(arg2); }'

# Network latency (DNS queries)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_sendto /comm == "systemd-resolve"/ {
    @start[tid] = nsecs;
}
tracepoint:syscalls:sys_exit_recvfrom /@start[tid]/ {
    printf("DNS query latency: %d ms\n", (nsecs - @start[tid]) / 1000000);
    delete(@start[tid]);
}'

CPU Profiling:

# Sample on-CPU stack traces
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'

# Sample user + kernel stacks
sudo bpftrace -e 'profile:hz:99 { @[ustack, kstack] = count(); }'

# CPU time by process
sudo bpftrace -e 'profile:hz:99 { @[comm] = count(); }'

# CPU time by function (top kernel functions)
sudo bpftrace -e 'profile:hz:99 { @[kfunc] = count(); }'

Memory Allocation:

1
2
3
4
5
6
7
8
# Track memory allocations
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @bytes = sum(args->bytes_alloc); }'

# Allocation size distribution
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @alloc_size = hist(args->bytes_alloc); }'

# Top allocating processes
sudo bpftrace -e 'tracepoint:kmem:kmalloc { @[comm] = sum(args->bytes_alloc); }'

Complete bpftrace Scripts

TCP Connection Monitor:

sudo tee tcp_monitor.bt <<'EOF'
#!/usr/bin/env bpftrace

BEGIN {
    printf("Monitoring TCP connections... Hit Ctrl-C to end.\n");
    printf("%-8s %-16s %-6s %-16s %-16s %-6s\n",
           "TIME", "COMM", "PID", "SADDR", "DADDR", "DPORT");
}

tracepoint:syscalls:sys_enter_connect /args->family == AF_INET/ {
    @connect[tid] = args;
}

tracepoint:syscalls:sys_exit_connect /@connect[tid]/ {
    $args = @connect[tid];
    $sa = (struct sockaddr_in *)$args->addr;

    time("%H:%M:%S ");
    printf("%-16s %-6d %-16s %-16s %-6d\n",
           comm, pid,
           ntop(AF_INET, 0),  // Local IP (simplified)
           ntop(AF_INET, $sa->sin_addr.s_addr),
           ($sa->sin_port >> 8) | (($sa->sin_port << 8) & 0xff00));

    delete(@connect[tid]);
}

END {
    clear(@connect);
}
EOF

chmod +x tcp_monitor.bt
sudo ./tcp_monitor.bt

Slow I/O Detector:

sudo tee slow_io.bt <<'EOF'
#!/usr/bin/env bpftrace

BEGIN {
    printf("Detecting slow I/O (>50ms)...\n");
}

tracepoint:syscalls:sys_enter_read,
tracepoint:syscalls:sys_enter_write {
    @start[tid] = nsecs;
    @fd[tid] = args->fd;
}

tracepoint:syscalls:sys_exit_read,
tracepoint:syscalls:sys_exit_write
/@start[tid]/ {
    $duration_ms = (nsecs - @start[tid]) / 1000000;

    if ($duration_ms > 50) {
        time("%H:%M:%S ");
        printf("%s (PID %d) slow I/O on fd %d: %d ms\n",
               comm, pid, @fd[tid], $duration_ms);
    }

    delete(@start[tid]);
    delete(@fd[tid]);
}

END {
    clear(@start);
    clear(@fd);
}
EOF

chmod +x slow_io.bt
sudo ./slow_io.bt

Writing Custom eBPF Programs with BCC

Simple Python BCC Example

Count Syscalls:

#!/usr/bin/env python3
# syscall_counter.py

from bcc import BPF
from time import sleep

# eBPF program (C code)
bpf_program = """
#include <uapi/linux/ptrace.h>

BPF_HASH(syscall_count, u64);

TRACEPOINT_PROBE(raw_syscalls, sys_enter) {
    u64 syscall_id = args->id;
    u64 *count = syscall_count.lookup(&syscall_id);

    if (count) {
        (*count)++;
    } else {
        u64 initial = 1;
        syscall_count.update(&syscall_id, &initial);
    }

    return 0;
}
"""

# Load eBPF program
b = BPF(text=bpf_program)

print("Counting syscalls... Hit Ctrl-C to stop.")

try:
    while True:
        sleep(5)
        print("\n=== Top 10 Syscalls (last 5 seconds) ===")

        syscall_count = b["syscall_count"]

        # Get top 10
        items = sorted(syscall_count.items(), key=lambda x: x[1].value, reverse=True)[:10]

        for syscall_id, count in items:
            print(f"Syscall {syscall_id.value}: {count.value} calls")

        # Clear for next interval
        syscall_count.clear()

except KeyboardInterrupt:
    print("\nExiting...")

Run:

sudo python3 syscall_counter.py

Network Packet Filter

Track TCP Connections:

#!/usr/bin/env python3
# tcp_tracker.py

from bcc import BPF
import socket
import struct

bpf_program = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_PERF_OUTPUT(events);

struct connection_event {
    u32 pid;
    char comm[16];
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
};

int trace_connect(struct pt_regs *ctx, struct sock *sk) {
    struct connection_event evt = {};

    evt.pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&evt.comm, sizeof(evt.comm));

    u16 family = sk->__sk_common.skc_family;

    if (family == AF_INET) {
        evt.saddr = sk->__sk_common.skc_rcv_saddr;
        evt.daddr = sk->__sk_common.skc_daddr;
        evt.sport = sk->__sk_common.skc_num;
        evt.dport = sk->__sk_common.skc_dport;
        evt.dport = ntohs(evt.dport);

        events.perf_submit(ctx, &evt, sizeof(evt));
    }

    return 0;
}
"""

def print_event(cpu, data, size):
    event = b["events"].event(data)

    saddr = socket.inet_ntoa(struct.pack("I", event.saddr))
    daddr = socket.inet_ntoa(struct.pack("I", event.daddr))

    print(f"{event.comm.decode('utf-8', 'ignore'):16s} "
          f"PID {event.pid:6d} "
          f"{saddr}:{event.sport} -> {daddr}:{event.dport}")

# Load BPF program
b = BPF(text=bpf_program)
b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect")

print("Monitoring TCP connections... Hit Ctrl-C to stop.")
print(f"{'COMM':<16} {'PID':<6} {'SOURCE':>21} -> {'DEST'}")

# Poll events
b["events"].open_perf_buffer(print_event)

try:
    while True:
        b.perf_buffer_poll()
except KeyboardInterrupt:
    print("\nExiting...")

CPU Performance Profiler

Function Latency Tracker:

#!/usr/bin/env python3
# function_latency.py

from bcc import BPF
import sys

if len(sys.argv) < 2:
    print(f"Usage: {sys.argv[0]} <function_name>")
    sys.exit(1)

function_name = sys.argv[1]

# BPF program template (with function name substitution)
bpf_program = """
#include <uapi/linux/ptrace.h>

BPF_HASH(start_time, u32);
BPF_HISTOGRAM(latency_hist);

int trace_start(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    start_time.update(&pid, &ts);
    return 0;
}

int trace_return(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 *tsp = start_time.lookup(&pid);

    if (tsp != 0) {
        u64 delta = bpf_ktime_get_ns() - *tsp;
        latency_hist.increment(bpf_log2l(delta / 1000));  // microseconds
        start_time.delete(&pid);
    }

    return 0;
}
"""

b = BPF(text=bpf_program)
b.attach_kprobe(event=function_name, fn_name="trace_start")
b.attach_kretprobe(event=function_name, fn_name="trace_return")

print(f"Tracing {function_name} latency... Hit Ctrl-C to show results.")

try:
    import time
    time.sleep(999999999)
except KeyboardInterrupt:
    pass

print(f"\nLatency distribution for {function_name}:")
b["latency_hist"].print_log2_hist("usecs")

Run:

# Profile do_sys_open() latency
sudo python3 function_latency.py do_sys_open

XDP (eXpress Data Path): High-Performance Networking

XDP allows packet processing directly in the network driver, before the kernel network stack.

Simple XDP Packet Filter

Drop ICMP Packets:

// xdp_drop_icmp.c

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/icmp.h>

#define SEC(NAME) __attribute__((section(NAME), used))

SEC("xdp")
int xdp_drop_icmp_prog(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;

    // Ethernet header
    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;

    // Only IPv4
    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;

    // IP header
    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;

    // Drop ICMP packets
    if (ip->protocol == IPPROTO_ICMP)
        return XDP_DROP;

    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

Compile and Load:

# Compile XDP program
clang -O2 -target bpf -c xdp_drop_icmp.c -o xdp_drop_icmp.o

# Load onto interface
sudo ip link set dev eth0 xdp obj xdp_drop_icmp.o sec xdp

# Verify
sudo ip link show dev eth0
# Should show: xdp

# Test (from another machine)
ping <raspberry-pi-ip>
# Should timeout (ICMP dropped)

# Unload
sudo ip link set dev eth0 xdp off

XDP Packet Counter

#!/usr/bin/env python3
# xdp_packet_counter.py

from bcc import BPF

xdp_program = """
#include <uapi/linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>

BPF_ARRAY(packet_count, u64, 1);
BPF_ARRAY(byte_count, u64, 1);

int xdp_count(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;

    u32 index = 0;
    u64 *pkt_cnt = packet_count.lookup(&index);
    u64 *byte_cnt = byte_count.lookup(&index);

    if (pkt_cnt && byte_cnt) {
        __sync_fetch_and_add(pkt_cnt, 1);
        __sync_fetch_and_add(byte_cnt, data_end - data);
    }

    return XDP_PASS;
}
"""

# Load program
b = BPF(text=xdp_program)
fn = b.load_func("xdp_count", BPF.XDP)

# Attach to interface
interface = "eth0"
b.attach_xdp(interface, fn, 0)

print(f"Counting packets on {interface}... Hit Ctrl-C to stop.")

try:
    import time
    while True:
        time.sleep(1)

        pkt_cnt = b["packet_count"][0].value
        byte_cnt = b["byte_count"][0].value

        print(f"Packets: {pkt_cnt:,}  Bytes: {byte_cnt:,}  "
              f"({byte_cnt / 1024 / 1024:.2f} MB)")

except KeyboardInterrupt:
    pass

# Cleanup
b.remove_xdp(interface, 0)
print("\nDetached XDP program.")

Security Monitoring with eBPF

Detect Suspicious Syscalls

#!/usr/bin/env python3
# security_monitor.py

from bcc import BPF

bpf_program = """
#include <uapi/linux/ptrace.h>

// Suspicious syscalls that might indicate malicious activity
BPF_HASH(suspicious_calls, u32);

TRACEPOINT_PROBE(syscalls, sys_enter_ptrace) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 count = 0;

    u64 *val = suspicious_calls.lookup(&pid);
    if (val) count = *val;

    count++;
    suspicious_calls.update(&pid, &count);

    return 0;
}

TRACEPOINT_PROBE(syscalls, sys_enter_kill) {
    // Log kill attempts
    u32 target_pid = args->pid;
    int signal = args->sig;

    if (signal == 9) {  // SIGKILL
        bpf_trace_printk("SIGKILL to PID %d\\n", target_pid);
    }

    return 0;
}

TRACEPOINT_PROBE(syscalls, sys_enter_execve) {
    char comm[16];
    bpf_get_current_comm(&comm, sizeof(comm));

    // Detect common reverse shell patterns
    char filename[256];
    bpf_probe_read_user_str(&filename, sizeof(filename), args->filename);

    if (filename[0] == '/' && filename[1] == 'd' && filename[2] == 'e' && filename[3] == 'v') {
        // Execution from /dev (suspicious)
        bpf_trace_printk("Suspicious exec from /dev: %s\\n", filename);
    }

    return 0;
}
"""

b = BPF(text=bpf_program)

print("Security monitoring active...")
print("Watching for:")
print("  - ptrace() usage (debugger attachment)")
print("  - SIGKILL signals")
print("  - Suspicious execve() patterns")
print()

# Print trace messages
b.trace_print()

File Integrity Monitor

#!/usr/bin/env python3
# file_integrity_monitor.py

from bcc import BPF
import os

# Protected directories
PROTECTED_DIRS = ["/etc", "/usr/bin", "/usr/sbin", "/boot"]

bpf_program = """
#include <uapi/linux/ptrace.h>

BPF_PERF_OUTPUT(events);

struct file_event {
    u32 pid;
    char comm[16];
    char filename[256];
    u32 flags;
};

TRACEPOINT_PROBE(syscalls, sys_enter_openat) {
    struct file_event evt = {};

    evt.pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&evt.comm, sizeof(evt.comm));
    bpf_probe_read_user_str(&evt.filename, sizeof(evt.filename), args->filename);
    evt.flags = args->flags;

    // Check for write flags (O_WRONLY, O_RDWR, O_CREAT, O_TRUNC)
    if ((evt.flags & 0x01) || (evt.flags & 0x02) || 
        (evt.flags & 0x40) || (evt.flags & 0x200)) {
        events.perf_submit(args, &evt, sizeof(evt));
    }

    return 0;
}
"""

def is_protected_path(path):
    """Check if path is in protected directories"""
    for protected in PROTECTED_DIRS:
        if path.startswith(protected):
            return True
    return False

def print_event(cpu, data, size):
    event = b["events"].event(data)
    filename = event.filename.decode('utf-8', 'ignore')

    if is_protected_path(filename):
        print(f"[ALERT] PID {event.pid} ({event.comm.decode('utf-8', 'ignore')}) "
              f"modifying protected file: {filename}")

b = BPF(text=bpf_program)

print(f"Monitoring write access to protected directories:")
for dir in PROTECTED_DIRS:
    print(f"  - {dir}")
print()

b["events"].open_perf_buffer(print_event)

try:
    while True:
        b.perf_buffer_poll()
except KeyboardInterrupt:
    print("\nExiting...")

Production Monitoring Dashboard

Real-Time System Metrics Collector

#!/usr/bin/env python3
# system_metrics.py - Comprehensive system monitoring

from bcc import BPF
import time
import json

bpf_program = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

// CPU metrics
BPF_HASH(cpu_time, u32);

// Disk I/O metrics
BPF_HASH(disk_reads, u32);
BPF_HASH(disk_writes, u32);
BPF_HASH(disk_bytes_read, u32);
BPF_HASH(disk_bytes_written, u32);

// Network metrics
BPF_HASH(net_tx_packets, u32);
BPF_HASH(net_rx_packets, u32);
BPF_HASH(net_tx_bytes, u32);
BPF_HASH(net_rx_bytes, u32);

// Process metrics
BPF_HASH(process_starts, u32);

// CPU profiling
TRACEPOINT_PROBE(sched, sched_switch) {
    u32 pid = args->next_pid;
    u64 ts = bpf_ktime_get_ns();
    cpu_time.update(&pid, &ts);
    return 0;
}

// Disk I/O
TRACEPOINT_PROBE(block, block_rq_complete) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 bytes = args->nr_sector * 512;

    if (args->rwbs[0] == 'R') {
        disk_reads.increment(pid);
        u64 *val = disk_bytes_read.lookup(&pid);
        u64 total = val ? *val : 0;
        total += bytes;
        disk_bytes_read.update(&pid, &total);
    } else if (args->rwbs[0] == 'W') {
        disk_writes.increment(pid);
        u64 *val = disk_bytes_written.lookup(&pid);
        u64 total = val ? *val : 0;
        total += bytes;
        disk_bytes_written.update(&pid, &total);
    }

    return 0;
}

// Process creation
TRACEPOINT_PROBE(sched, sched_process_fork) {
    u32 zero = 0;
    process_starts.increment(zero);
    return 0;
}
"""

class SystemMetrics:
    def __init__(self):
        self.b = BPF(text=bpf_program)
        self.last_time = time.time()

    def collect(self):
        now = time.time()
        interval = now - self.last_time
        self.last_time = now

        metrics = {
            'timestamp': now,
            'interval': interval,
            'disk': self._get_disk_metrics(),
            'processes': self._get_process_metrics(),
        }

        return metrics

    def _get_disk_metrics(self):
        reads = sum(self.b["disk_reads"].values())
        writes = sum(self.b["disk_writes"].values())
        bytes_read = sum(v.value for v in self.b["disk_bytes_read"].values())
        bytes_written = sum(v.value for v in self.b["disk_bytes_written"].values())

        return {
            'reads': reads,
            'writes': writes,
            'bytes_read': bytes_read,
            'bytes_written': bytes_written,
            'mb_read': bytes_read / 1024 / 1024,
            'mb_written': bytes_written / 1024 / 1024,
        }

    def _get_process_metrics(self):
        starts = sum(self.b["process_starts"].values())
        return {'starts': starts}

    def clear(self):
        """Clear counters for next interval"""
        self.b["disk_reads"].clear()
        self.b["disk_writes"].clear()
        self.b["disk_bytes_read"].clear()
        self.b["disk_bytes_written"].clear()
        self.b["process_starts"].clear()

if __name__ == '__main__':
    monitor = SystemMetrics()

    print("Collecting system metrics... Hit Ctrl-C to stop.")
    print()

    try:
        while True:
            time.sleep(5)

            metrics = monitor.collect()

            print(f"=== Metrics (last {metrics['interval']:.1f}s) ===")
            print(f"Disk Reads:  {metrics['disk']['reads']:,} "
                  f"({metrics['disk']['mb_read']:.2f} MB)")
            print(f"Disk Writes: {metrics['disk']['writes']:,} "
                  f"({metrics['disk']['mb_written']:.2f} MB)")
            print(f"Process Starts: {metrics['disk']['starts']}")
            print()

            # Could export to Prometheus, InfluxDB, etc.
            # export_to_prometheus(metrics)

            monitor.clear()

    except KeyboardInterrupt:
        print("\nExiting...")

Raspberry Pi Specific Optimizations

ARM64 Considerations

# Check eBPF JIT for ARM64
sudo sysctl net.core.bpf_jit_enable
# Should be 1 (enabled)

# View JIT compiled code
sudo sysctl net.core.bpf_jit_harden
# 0 = no hardening (faster)
# 1 = basic hardening
# 2 = full hardening (slower)

# For production, use hardening level 1
sudo sysctl -w net.core.bpf_jit_harden=1

Memory Optimization

# eBPF programs use locked memory
# Check current limit
ulimit -l

# Increase if needed (in /etc/security/limits.conf)
echo "* soft memlock unlimited" | sudo tee -a /etc/security/limits.conf
echo "* hard memlock unlimited" | sudo tee -a /etc/security/limits.conf

# Or temporarily
sudo ulimit -l unlimited

Performance Tuning

# Use efficient map types
# BPF_HASH (hash table) - flexible but slower
# BPF_ARRAY (array) - fast but fixed size
# BPF_LRU_HASH (LRU cache) - automatic eviction
# BPF_RINGBUF (ring buffer) - most efficient for event streaming

# Prefer ring buffers over perf buffers on Pi 4/5
BPF_RINGBUF_OUTPUT(events, 1024);  # 1024 pages

# Minimize per-event overhead
# Avoid string operations in hot paths
# Use maps for aggregation, not individual events

Troubleshooting

Common Issues

1. "Operation not permitted"

1
2
3
4
5
# Need root or CAP_BPF/CAP_PERFMON capabilities
sudo bpftrace ...

# Or add capabilities
sudo setcap cap_bpf,cap_perfmon+ep /usr/bin/bpftrace

2. "Failed to load program: Invalid argument"

1
2
3
4
5
6
7
8
# Check kernel version
uname -r

# Verify eBPF support
zcat /proc/config.gz | grep BPF

# Check dmesg for verifier errors
sudo dmesg | tail -50

3. "Cannot allocate memory"

1
2
3
4
# Increase memlock limit
ulimit -l unlimited

# Or in /etc/security/limits.conf

4. "Unknown kprobe"

1
2
3
4
5
6
# Function doesn't exist or not exposed
# Check available kprobes
sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep <function>

# Use tracepoint instead
sudo bpftrace -l 'tracepoint:*' | grep <topic>

Debugging eBPF Programs

# Enable debug output
sudo bpftrace -v -e '...'

# Check verifier log
sudo bpftool prog load program.o /sys/fs/bpf/myprog 2>&1 | less

# Trace BPF events
sudo bpftrace -e 'tracepoint:bpf:* { printf("%s\n", probe); }'

# View loaded programs
sudo bpftool prog list

# View program details
sudo bpftool prog show id <id>

# Dump JIT compiled code
sudo bpftool prog dump jited id <id>

Summary

This comprehensive guide covered eBPF programming on Raspberry Pi:

✅ Core Concepts

  • eBPF architecture and kernel integration
  • Safety guarantees through verifier
  • JIT compilation to ARM64 native code
  • Maps for kernel-userspace communication

✅ Tools and Frameworks

  • BCC: Full-featured Python/C framework with 50+ ready tools
  • bpftrace: High-level tracing language for one-liners
  • libbpf: Modern C library for production applications
  • bpftool: Introspection and debugging utilities

✅ Ready-to-Use Monitoring

  • execsnoop, opensnoop, tcpconnect, tcpretrans
  • biolatency, biotop, filetop, cachestat
  • cpudist, profile, memleak
  • Network, disk, CPU, memory visibility

✅ Custom Development

  • Python BCC programming examples
  • bpftrace scripting for custom metrics
  • XDP for high-performance packet processing
  • Security monitoring and intrusion detection

✅ Production Use Cases

  • Real-time performance monitoring
  • Security threat detection
  • Network traffic analysis
  • Container and Kubernetes observability
  • IoT device monitoring

✅ Raspberry Pi Optimization

  • ARM64 JIT compilation
  • Memory limit configuration
  • Efficient map types for limited resources
  • Ring buffers for event streaming

✅ Performance

  • Minimal overhead (~1-5% CPU)
  • Kernel-level visibility without instrumentation
  • Zero application code changes required
  • Nanosecond-precision timing

Next Steps

Advanced Topics:

  1. eBPF for Kubernetes - Monitor containers and pods
  2. Distributed Tracing - Track requests across services
  3. Custom XDP Programs - DDoS protection, load balancing
  4. eBPF LSM (Linux Security Module) - Runtime security policies
  5. CO-RE (Compile Once, Run Everywhere) - Portable eBPF programs

Related Guides:

With eBPF, your Raspberry Pi gains enterprise-grade observability, security monitoring, and performance analysis capabilities - all without kernel modifications or application changes!