Skip to content

OverlayFS - Layer-based Filesystems for Raspberry Pi

Introduction

OverlayFS is a union filesystem that combines multiple directories into a single merged view, creating a powerful layered storage system. It's the technology behind Docker's storage driver and enables sophisticated file management strategies that are particularly valuable for Raspberry Pi's SD card-based storage.

This technology transforms how you manage filesystems:

  • Layered Storage: Combine read-only base with writable overlay
  • Non-Destructive Changes: Modify files without touching originals
  • System Protection: Create immutable base systems with temporary changes
  • Space Efficiency: Share common data across multiple instances
  • Fast Snapshots: Instant filesystem branching and rollback
  • A/B Updates: Safe system updates with fallback capability

Why OverlayFS matters for Raspberry Pi:

  • SD Card Longevity: Reduce write cycles by using tmpfs overlays
  • System Recovery: Always have a clean base system to fall back to
  • Development Safety: Experiment freely without breaking your system
  • Kiosk Mode: Public displays that reset on reboot
  • Container Understanding: Learn Docker's storage internals
  • Update Safety: Test updates without risking current system

Common use cases:

  • Read-Only Root: Protect SD card and system integrity
  • Temporary Workspaces: Changes disappear on reboot
  • System Snapshots: Instant rollback capability
  • A/B Partitioning: Safe over-the-air updates
  • Multi-Instance: Run same base with different overlays
  • Development: Test changes without affecting base
  • Education: Students can experiment safely
  • IoT Devices: Field devices with recovery capability

This comprehensive guide covers:

  • OverlayFS Fundamentals: Lower, upper, work, and merged directories
  • Basic Usage: Manual overlay mounting and management
  • Read-Only Root: Complete SD card protection setup
  • Tmpfs Overlays: Volatile changes for kiosk mode
  • A/B Updates: Dual partition update system
  • Docker Integration: Understanding container storage
  • Boot Configuration: Automatic overlay on startup
  • Performance: Optimization for Raspberry Pi
  • Troubleshooting: Common issues and solutions
  • Production Examples: Real-world deployments

Perfect for:

  • System Administrators: Immutable infrastructure patterns
  • IoT Developers: Field device management and updates
  • DevOps Engineers: Understanding container storage layers
  • Educators: Safe learning environments
  • Kiosk Operators: Self-resetting public displays
  • Embedded Engineers: Robust update mechanisms
  • Home Automation: Stable, recoverable systems
  • Security Professionals: Tamper-resistant deployments

Understanding OverlayFS

Architecture

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
┌─────────────────────────────────────────────────────────┐
│                   Merged Directory                       │
│                  (User's View)                          │
│                                                          │
│  /merged/file1.txt  ← Shows version from upperdir       │
│  /merged/file2.txt  ← Shows version from lowerdir       │
│  /merged/file3.txt  ← Created in upperdir               │
│                                                          │
└────────────┬────────────────────────────────────────────┘
             │ OverlayFS combines layers
             │
    ┌────────┴────────┐
    │                 │
    ▼                 ▼
┌──────────┐    ┌──────────┐
│UpperDir  │    │ LowerDir │
│(Writable)│    │(Read-Only│
│          │    │ or R/W)  │
│ Stores:  │    │          │
│ - New    │    │ Base     │
│   files  │    │ system   │
│ - Mods   │    │ files    │
│ - Deletes│    │          │
│   (whiteouts)│          │
└──────────┘    └──────────┘
                      │
                      │ Can be:
                      │ - Real partition
                      │ - Another overlay
                      │ - Multiple layers
                      │
           ┌──────────┴──────────┐
           │                     │
           ▼                     ▼
      ┌─────────┐         ┌─────────┐
      │ Layer 2 │         │ Layer 3 │
      └─────────┘         └─────────┘

┌──────────────────────────────────────────────────────────┐
│                    Work Directory                         │
│              (Internal OverlayFS Use)                    │
│                                                           │
│  Used for atomic operations and copy-up                  │
│  Must be on same filesystem as upperdir                  │
│  Not directly accessed by users                          │
└──────────────────────────────────────────────────────────┘

How OverlayFS Works

File Operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
Read Operation:
1. Check upperdir first
2. If not found, check lowerdir
3. Return first match

Write Operation (file exists in lowerdir):
1. Copy file to upperdir (copy-up)
2. Modify copy in upperdir
3. Lowerdir remains unchanged

Write Operation (new file):
1. Create directly in upperdir
2. Visible in merged view

Delete Operation:
1. Create "whiteout" file in upperdir
2. Hides file from lowerdir
3. Lowerdir file remains (hidden)

Directory Structure Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
# Example layout
/lower/          # Read-only base
├── file1.txt    # Original: "Hello"
├── file2.txt    # Original: "World"
└── dir/
    └── file3.txt

/upper/          # Writable overlay
├── file1.txt    # Modified: "Hello Modified!"
├── file4.txt    # New file
└── dir/
    └── file3.txt  # (c 0 0) - whiteout (deleted)

/work/           # OverlayFS working directory
└── (internal)

/merged/         # Combined view (what user sees)
├── file1.txt    # From upper: "Hello Modified!"
├── file2.txt    # From lower: "World"
├── file4.txt    # From upper: New file
└── dir/         # Empty (file3.txt whited out)

Basic OverlayFS Usage

Installing Required Tools

1
2
3
4
5
6
7
8
# OverlayFS is built into kernel (no installation needed)
# Verify kernel support
grep -i overlay /proc/filesystems
# Should show: overlay

# Install utilities for testing
sudo apt update
sudo apt install -y fuse-overlayfs  # Userspace overlay (optional)

Simple Overlay Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# Create directory structure
mkdir -p /tmp/overlay-test/{lower,upper,work,merged}

# Create base files in lower layer
echo "Original file 1" > /tmp/overlay-test/lower/file1.txt
echo "Original file 2" > /tmp/overlay-test/lower/file2.txt
echo "Read-only data" > /tmp/overlay-test/lower/readonly.txt

# Mount overlay
sudo mount -t overlay overlay \
    -o lowerdir=/tmp/overlay-test/lower,\
upperdir=/tmp/overlay-test/upper,\
workdir=/tmp/overlay-test/work \
    /tmp/overlay-test/merged

# Verify mount
mount | grep overlay

# View merged content
ls -la /tmp/overlay-test/merged/
# Shows all files from lower layer

# Modify a file
echo "Modified in overlay" > /tmp/overlay-test/merged/file1.txt

# Create new file
echo "New file" > /tmp/overlay-test/merged/newfile.txt

# Delete a file (creates whiteout)
rm /tmp/overlay-test/merged/readonly.txt

# Check results
echo "=== Merged view ==="
ls -la /tmp/overlay-test/merged/

echo "=== Lower layer (unchanged) ==="
ls -la /tmp/overlay-test/lower/
cat /tmp/overlay-test/lower/file1.txt  # Still "Original file 1"

echo "=== Upper layer (changes) ==="
ls -la /tmp/overlay-test/upper/
cat /tmp/overlay-test/upper/file1.txt  # "Modified in overlay"

# Unmount
sudo umount /tmp/overlay-test/merged

Multiple Lower Layers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# OverlayFS supports multiple lower layers (stacked)
mkdir -p /tmp/multi/{layer1,layer2,layer3,upper,work,merged}

# Create files in each layer
echo "From layer 1" > /tmp/multi/layer1/file1.txt
echo "From layer 2" > /tmp/multi/layer2/file2.txt
echo "From layer 3" > /tmp/multi/layer3/file3.txt
echo "Overridden" > /tmp/multi/layer2/file1.txt  # Overrides layer1

# Mount with multiple lower layers (rightmost is lowest priority)
sudo mount -t overlay overlay \
    -o lowerdir=/tmp/multi/layer1:/tmp/multi/layer2:/tmp/multi/layer3,\
upperdir=/tmp/multi/upper,\
workdir=/tmp/multi/work \
    /tmp/multi/merged

# layer1 has highest priority, layer3 lowest
# If same file exists in multiple layers, highest priority wins

ls /tmp/multi/merged/
cat /tmp/multi/merged/file1.txt  # "Overridden" (from layer2)

sudo umount /tmp/multi/merged

Read-Only Root Filesystem

Benefits for Raspberry Pi

  • SD Card Longevity: Dramatically reduces write cycles
  • System Stability: Prevents corruption from power loss
  • Security: Harder to tamper with system files
  • Easy Recovery: Reboot to clean state
  • Predictability: System always starts the same way

Creating Read-Only Root with Tmpfs Overlay

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/bin/bash
# setup-ro-root.sh - Configure read-only root with tmpfs overlay

# Step 1: Edit /boot/firmware/cmdline.txt
# Add to kernel parameters:
# boot=overlay overlayroot=tmpfs

# Step 2: Install overlayroot
sudo apt update
sudo apt install -y overlayroot

# Step 3: Configure overlayroot
sudo tee /etc/overlayroot.conf <<EOF
overlayroot="tmpfs"
overlayroot_cfgdisk="disabled"
EOF

# Step 4: Edit /etc/fstab - make root read-only
sudo cp /etc/fstab /etc/fstab.backup
sudo sed -i 's/defaults/defaults,ro/g' /etc/fstab

# Step 5: Reboot
echo "Configuration complete. System will use tmpfs overlay after reboot."
echo "All changes will be lost on reboot unless saved to lower layer."
echo "Reboot now? (y/n)"
read -r REPLY
if [[ $REPLY =~ ^[Yy]$ ]]; then
    sudo reboot
fi

Manual Read-Only Root Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/bin/bash
# manual-ro-root.sh - Manual read-only root configuration

# Create overlay directories
sudo mkdir -p /overlay/upper
sudo mkdir -p /overlay/work

# Add to /etc/initramfs-tools/scripts/init-bottom/overlay
sudo tee /etc/initramfs-tools/scripts/init-bottom/overlay <<'EOF'
#!/bin/sh

PREREQ=""
prereqs()
{
    echo "$PREREQ"
}

case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac

. /scripts/functions

# Mount tmpfs for overlay
mount -t tmpfs tmpfs /overlay
mkdir -p /overlay/upper /overlay/work

# Create overlay on root
mount -t overlay overlay \
    -o lowerdir=/root,upperdir=/overlay/upper,workdir=/overlay/work \
    /root

exit 0
EOF

# Make executable
sudo chmod +x /etc/initramfs-tools/scripts/init-bottom/overlay

# Update initramfs
sudo update-initramfs -u

# Modify /etc/fstab
sudo sed -i 's/\(.*\/.*ext4.*defaults\)/\1,ro/' /etc/fstab

echo "Read-only root configured. Reboot to activate."

Persistent Storage Areas

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Even with read-only root, some directories need to be writable

# /etc/fstab - add tmpfs mounts for writable areas
tmpfs /tmp tmpfs defaults,noatime,nosuid,size=100M 0 0
tmpfs /var/tmp tmpfs defaults,noatime,nosuid,size=50M 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,size=50M 0 0
tmpfs /var/cache tmpfs defaults,noatime,nosuid,size=50M 0 0

# For persistent data (survives reboot)
# Use separate partition or USB storage
/dev/sda1 /persistent ext4 defaults,noatime 0 2

Making Permanent Changes to Read-Only System

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# Method 1: Remount root as read-write
sudo mount -o remount,rw /

# Make changes
sudo apt update
sudo apt install -y some-package

# Remount as read-only
sudo mount -o remount,ro /

# Method 2: Use overlayroot-chroot (if using overlayroot package)
sudo overlayroot-chroot

# Now in chroot environment with read-write access
apt update && apt install some-package
exit

# Reboot to apply changes
sudo reboot

Kiosk Mode with Auto-Reset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/bin/bash
# kiosk-setup.sh - Self-resetting kiosk configuration

# Create overlay script
sudo tee /usr/local/bin/kiosk-overlay.sh <<'EOF'
#!/bin/bash

# Mount tmpfs for overlay storage
mount -t tmpfs tmpfs /overlay
mkdir -p /overlay/upper /overlay/work

# Mount overlay on home directory
mount -t overlay overlay \
    -o lowerdir=/home/kiosk-base,upperdir=/overlay/upper,workdir=/overlay/work \
    /home/kiosk

# All changes to /home/kiosk are now volatile
EOF

chmod +x /usr/local/bin/kiosk-overlay.sh

# Create systemd service
sudo tee /etc/systemd/system/kiosk-overlay.service <<EOF
[Unit]
Description=Kiosk Overlay Mount
DefaultDependencies=no
After=local-fs.target
Before=chromium.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/kiosk-overlay.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

# Enable service
sudo systemctl daemon-reload
sudo systemctl enable kiosk-overlay.service

echo "Kiosk overlay configured. Changes reset on reboot."

A/B Partition Updates

Dual Root Partition Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
Disk Layout:
┌────────────────────────────────────────────┐
│ /dev/mmcblk0                               │
├────────────────────────────────────────────┤
│ /dev/mmcblk0p1 - /boot (shared)           │
│ 256MB - FAT32                              │
├────────────────────────────────────────────┤
│ /dev/mmcblk0p2 - Root A (primary)         │
│ 4GB - ext4                                 │
├────────────────────────────────────────────┤
│ /dev/mmcblk0p3 - Root B (backup)          │
│ 4GB - ext4                                 │
├────────────────────────────────────────────┤
│ /dev/mmcblk0p4 - Data (persistent)        │
│ Remaining space - ext4                     │
└────────────────────────────────────────────┘

A/B Update Script

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
#!/bin/bash
# ab-update.sh - A/B partition update system

set -e

# Configuration
BOOT_PARTITION="/dev/mmcblk0p1"
ROOT_A="/dev/mmcblk0p2"
ROOT_B="/dev/mmcblk0p3"
MOUNT_A="/mnt/root-a"
MOUNT_B="/mnt/root-b"

# Determine current root
CURRENT_ROOT=$(findmnt -n -o SOURCE /)

if [ "$CURRENT_ROOT" = "$ROOT_A" ]; then
    CURRENT="A"
    INACTIVE_ROOT="$ROOT_B"
    INACTIVE_MOUNT="$MOUNT_B"
else
    CURRENT="B"
    INACTIVE_ROOT="$ROOT_A"
    INACTIVE_MOUNT="$MOUNT_A"
fi

echo "Current root: Partition $CURRENT ($CURRENT_ROOT)"
echo "Inactive root: $INACTIVE_ROOT"

# Function to apply update
apply_update() {
    local UPDATE_IMAGE=$1

    echo "Mounting inactive partition..."
    mkdir -p "$INACTIVE_MOUNT"
    mount "$INACTIVE_ROOT" "$INACTIVE_MOUNT"

    echo "Backing up current partition to inactive..."
    rsync -aHAXxv --delete \
        --exclude='/dev/*' \
        --exclude='/proc/*' \
        --exclude='/sys/*' \
        --exclude='/tmp/*' \
        --exclude='/run/*' \
        --exclude='/mnt/*' \
        --exclude='/media/*' \
        / "$INACTIVE_MOUNT/"

    echo "Applying update to inactive partition..."
    # Extract update or apply changes
    if [ -f "$UPDATE_IMAGE" ]; then
        tar -xzf "$UPDATE_IMAGE" -C "$INACTIVE_MOUNT/"
    fi

    echo "Update applied to inactive partition"

    # Update boot configuration
    update_boot_config

    umount "$INACTIVE_MOUNT"

    echo "Update complete. Reboot to activate."
    echo "If system fails to boot, it will automatically fall back to current partition."
}

# Update boot configuration to use new root
update_boot_config() {
    mount "$BOOT_PARTITION" /boot/firmware

    # Update cmdline.txt
    if [ "$CURRENT" = "A" ]; then
        NEW_ROOT="$ROOT_B"
    else
        NEW_ROOT="$ROOT_A"
    fi

    sed -i "s|root=[^ ]*|root=$NEW_ROOT|" /boot/firmware/cmdline.txt

    umount /boot/firmware

    echo "Boot configuration updated to use $NEW_ROOT"
}

# Rollback to previous partition
rollback() {
    echo "Rolling back to previous partition..."
    update_boot_config
    echo "Rollback configured. Reboot to activate."
}

# Main
case "${1:-}" in
    update)
        if [ -z "${2:-}" ]; then
            echo "Usage: $0 update <update-image.tar.gz>"
            exit 1
        fi
        apply_update "$2"
        ;;
    rollback)
        rollback
        ;;
    status)
        echo "Current partition: $CURRENT"
        echo "Active root: $CURRENT_ROOT"
        echo "Inactive root: $INACTIVE_ROOT"
        ;;
    *)
        echo "Usage: $0 {update|rollback|status} [update-image]"
        exit 1
        ;;
esac

Automatic Boot Fallback

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Add to /boot/firmware/config.txt for boot counting
boot_delay=1

# Create boot check script
sudo tee /usr/local/bin/boot-success.sh <<'EOF'
#!/bin/bash
# Mark current boot as successful

BOOT_COUNT_FILE="/boot/firmware/boot_count.txt"

if [ -f "$BOOT_COUNT_FILE" ]; then
    rm "$BOOT_COUNT_FILE"
    echo "Boot successful, counter reset"
fi
EOF

chmod +x /usr/local/bin/boot-success.sh

# Create systemd service to mark boot as successful
sudo tee /etc/systemd/system/boot-success.service <<EOF
[Unit]
Description=Mark Boot as Successful
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/boot-success.sh

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable boot-success.service

Docker and OverlayFS

Understanding Docker's Use of OverlayFS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Docker uses overlay2 storage driver by default

# Check Docker's storage driver
docker info | grep "Storage Driver"
# Output: Storage Driver: overlay2

# View overlay mounts for containers
docker run -d --name test-container nginx
mount | grep overlay | grep test-container

# Example output:
# overlay on /var/lib/docker/overlay2/abc123.../merged type overlay
# (lowerdir=...image-layers...,upperdir=...container-layer...,workdir=...)

# Inspect container's layers
docker inspect test-container | grep -A 10 GraphDriver

# Output shows:
# - LowerDir: Image layers (read-only)
# - UpperDir: Container layer (writable)
# - WorkDir: OverlayFS working directory
# - MergedDir: Combined view inside container

docker rm -f test-container

Docker Image Layers Visualization

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Docker Image: nginx:latest

┌─────────────────────────────────────────┐
│     Container Layer (UpperDir)          │ ← Read/Write
│   - Application logs                    │
│   - Temporary files                     │
│   - Runtime changes                     │
├─────────────────────────────────────────┤
│     nginx:latest layer                  │ ← Read-only
│   - nginx binaries                      │
│   - nginx config                        │
├─────────────────────────────────────────┤
│     debian:bullseye layer               │ ← Read-only
│   - Base OS files                       │
│   - System libraries                    │
└─────────────────────────────────────────┘

Each layer is a separate directory in:
/var/lib/docker/overlay2/

When you run a container:
- All image layers become lowerdir (read-only)
- New container layer becomes upperdir (writable)
- Changes don't affect image (can spawn unlimited containers)

Manual Container-Like Overlay

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
#!/bin/bash
# container-like-overlay.sh - Simulate Docker's overlay approach

# Create directory structure
BASE_DIR="/tmp/container-sim"
mkdir -p "$BASE_DIR"/{image-layers,containers}

# Create "image layers"
mkdir -p "$BASE_DIR/image-layers/layer1"
echo "#!/bin/bash" > "$BASE_DIR/image-layers/layer1/app.sh"
echo "echo 'Hello from app'" >> "$BASE_DIR/image-layers/layer1/app.sh"
chmod +x "$BASE_DIR/image-layers/layer1/app.sh"

mkdir -p "$BASE_DIR/image-layers/layer2"
echo "config=default" > "$BASE_DIR/image-layers/layer2/config.txt"

# Function to create container instance
create_container() {
    CONTAINER_ID="container-$$-$(date +%s)"
    CONTAINER_DIR="$BASE_DIR/containers/$CONTAINER_ID"

    mkdir -p "$CONTAINER_DIR"/{upper,work,merged}

    # Mount overlay
    sudo mount -t overlay overlay \
        -o lowerdir="$BASE_DIR/image-layers/layer2:$BASE_DIR/image-layers/layer1",\
upperdir="$CONTAINER_DIR/upper",\
workdir="$CONTAINER_DIR/work" \
        "$CONTAINER_DIR/merged"

    echo "Container created: $CONTAINER_ID"
    echo "Root directory: $CONTAINER_DIR/merged"
    echo "Run: sudo chroot $CONTAINER_DIR/merged /bin/bash"
    echo "Stop: sudo umount $CONTAINER_DIR/merged"

    return 0
}

# Create multiple containers from same base
create_container
sleep 1
create_container

# Both containers share the same image layers (lowerdir)
# But have independent writable layers (upperdir)
# Changes in one container don't affect the other

Performance Optimization

OverlayFS Performance Characteristics

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Performance comparison: Direct FS vs OverlayFS

# Test setup
mkdir -p /tmp/perf-test/{direct,overlay/{lower,upper,work,merged}}

# Direct filesystem test
time dd if=/dev/zero of=/tmp/perf-test/direct/testfile bs=1M count=100
# Baseline write performance

# OverlayFS test
sudo mount -t overlay overlay \
    -o lowerdir=/tmp/perf-test/overlay/lower,\
upperdir=/tmp/perf-test/overlay/upper,\
workdir=/tmp/perf-test/overlay/work \
    /tmp/perf-test/overlay/merged

time dd if=/dev/zero of=/tmp/perf-test/overlay/merged/testfile bs=1M count=100
# OverlayFS write performance (new file - similar to direct)

# Copy-up test (modifying file from lowerdir)
dd if=/dev/zero of=/tmp/perf-test/overlay/lower/existing bs=1M count=100
sudo umount /tmp/perf-test/overlay/merged
sudo mount -t overlay overlay \
    -o lowerdir=/tmp/perf-test/overlay/lower,\
upperdir=/tmp/perf-test/overlay/upper,\
workdir=/tmp/perf-test/overlay/work \
    /tmp/perf-test/overlay/merged

time dd if=/dev/zero of=/tmp/perf-test/overlay/merged/existing bs=1M count=10 conv=notrunc
# First write triggers copy-up (slower)

sudo umount /tmp/perf-test/overlay/merged

Optimization Tips

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 1. Keep frequently modified files in upperdir
#    - Avoid copy-up overhead
#    - Place application data in separate mount

# 2. Use tmpfs for upperdir when changes are temporary
mount -t tmpfs tmpfs /tmp/overlay-upper
mount -t overlay overlay \
    -o lowerdir=/readonly-base,upperdir=/tmp/overlay-upper,workdir=/tmp/overlay-work \
    /merged

# 3. Minimize layers
#    - Fewer layers = faster lookups
#    - Combine related files in same layer

# 4. Use redirect_dir option for better rename performance
mount -t overlay overlay \
    -o lowerdir=/lower,upperdir=/upper,workdir=/work,redirect_dir=on \
    /merged

# 5. Enable metacopy for large file handling
mount -t overlay overlay \
    -o lowerdir=/lower,upperdir=/upper,workdir=/work,metacopy=on \
    /merged
# Only metadata is copied up initially, data copied on actual modification

Troubleshooting

Common Issues

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Issue 1: "workdir and upperdir must reside under the same mount"
# Solution: Ensure upperdir and workdir are on same filesystem

# Wrong:
# upperdir=/mnt/usb/upper  (on USB)
# workdir=/tmp/work        (on tmpfs)

# Correct:
mkdir -p /mnt/usb/{upper,work}
# Both on same mount point

# Issue 2: "filesystem is read-only"
# Check if underlying filesystem is read-only
mount | grep $(df /upper | tail -1 | awk '{print $1}')

# Remount as read-write
sudo mount -o remount,rw /underlying-mount

# Issue 3: Whiteout files visible
ls -la /upper/
# Shows: file (c 0 0) - character device 0:0

# This is normal - it's how OverlayFS tracks deletions
# They won't appear in merged view

# Issue 4: "too many levels of nesting"
# OverlayFS limits nesting depth
# Don't create overlay of overlay of overlay...
# Maximum nesting: 2 levels

# Issue 5: Permission denied on files
# OverlayFS requires proper permissions on all layers
chmod -R 755 /lower /upper /work

Debugging OverlayFS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# View all overlay mounts
mount | grep overlay

# Detailed mount options
findmnt -t overlay

# Check which files are in which layer
# Lower layer only:
ls /lower/

# Upper layer only:
ls /upper/

# Merged view:
ls /merged/

# Find files that have been copied up
find /upper/ -type f

# Check for whiteouts (deleted files)
find /upper/ -type c

# Monitor overlay operations (requires kernel debug)
sudo trace-cmd record -e overlay
# Perform operations
sudo trace-cmd report

Real-World Examples

IoT Device with Recovery

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#!/bin/bash
# iot-device-setup.sh - Robust IoT device configuration

# Partition layout:
# p1: /boot
# p2: /rootfs-base (read-only system image)
# p3: /data (persistent application data)

# Create mount points
sudo mkdir -p /mnt/{rootfs-base,data,overlay/{upper,work}}

# /etc/fstab
cat <<EOF | sudo tee -a /etc/fstab
/dev/mmcblk0p2 /mnt/rootfs-base ext4 ro,noatime 0 2
/dev/mmcblk0p3 /mnt/data ext4 defaults,noatime 0 2
tmpfs /mnt/overlay/upper tmpfs size=256M 0 0
tmpfs /mnt/overlay/work tmpfs size=64M 0 0
EOF

# Create overlay mount service
sudo tee /etc/systemd/system/rootfs-overlay.service <<'EOF'
[Unit]
Description=Root Filesystem Overlay
DefaultDependencies=no
After=local-fs.target
Before=systemd-remount-fs.service

[Service]
Type=oneshot
ExecStart=/bin/mount -t overlay overlay \
    -o lowerdir=/mnt/rootfs-base,upperdir=/mnt/overlay/upper,workdir=/mnt/overlay/work \
    /
RemainAfterExit=yes

[Install]
WantedBy=local-fs.target
EOF

sudo systemctl enable rootfs-overlay.service

echo "IoT device configured with:"
echo "  - Read-only base system (recovery image)"
echo "  - Tmpfs overlay (changes lost on reboot)"
echo "  - Persistent data partition (/mnt/data)"

Development Environment with Snapshots

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
#!/bin/bash
# dev-snapshot.sh - Development environment with instant rollback

SNAPSHOT_DIR="/snapshots"
PROJECT_BASE="/projects/myapp-base"
WORK_DIR="/workspace/myapp"

mkdir -p "$SNAPSHOT_DIR" "$WORK_DIR"

# Function to create snapshot
create_snapshot() {
    TIMESTAMP=$(date +%Y%m%d-%H%M%S)
    SNAP_NAME="snapshot-$TIMESTAMP"

    mkdir -p "$SNAPSHOT_DIR/$SNAP_NAME"

    # Copy current overlay upper layer
    if [ -d "/tmp/overlay-upper" ]; then
        cp -a /tmp/overlay-upper "$SNAPSHOT_DIR/$SNAP_NAME/upper"
        echo "Snapshot created: $SNAP_NAME"
    else
        echo "No overlay mounted"
        return 1
    fi
}

# Function to restore snapshot
restore_snapshot() {
    SNAP_NAME=$1

    if [ ! -d "$SNAPSHOT_DIR/$SNAP_NAME" ]; then
        echo "Snapshot not found: $SNAP_NAME"
        return 1
    fi

    # Unmount current overlay
    sudo umount "$WORK_DIR" 2>/dev/null

    # Clear current overlay
    rm -rf /tmp/overlay-upper/*
    rm -rf /tmp/overlay-work/*

    # Restore snapshot
    cp -a "$SNAPSHOT_DIR/$SNAP_NAME/upper/"* /tmp/overlay-upper/

    # Remount overlay
    mount_workspace

    echo "Restored snapshot: $SNAP_NAME"
}

# Mount workspace with overlay
mount_workspace() {
    mkdir -p /tmp/{overlay-upper,overlay-work}

    sudo mount -t overlay overlay \
        -o lowerdir="$PROJECT_BASE",upperdir=/tmp/overlay-upper,workdir=/tmp/overlay-work \
        "$WORK_DIR"

    echo "Workspace mounted at $WORK_DIR"
}

# List snapshots
list_snapshots() {
    echo "Available snapshots:"
    ls -1 "$SNAPSHOT_DIR" | grep "^snapshot-"
}

# Main
case "${1:-}" in
    mount)
        mount_workspace
        ;;
    snapshot)
        create_snapshot
        ;;
    restore)
        if [ -z "${2:-}" ]; then
            echo "Usage: $0 restore <snapshot-name>"
            list_snapshots
            exit 1
        fi
        restore_snapshot "$2"
        ;;
    list)
        list_snapshots
        ;;
    *)
        echo "Usage: $0 {mount|snapshot|restore|list} [snapshot-name]"
        exit 1
        ;;
esac

Live System Testing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/bin/bash
# test-overlay.sh - Test system changes without risk

echo "Creating test overlay environment..."

# Create overlay directories in tmpfs
mkdir -p /tmp/test-overlay/{upper,work}

# Mount overlay on root
sudo mount -t overlay overlay \
    -o lowerdir=/,upperdir=/tmp/test-overlay/upper,workdir=/tmp/test-overlay/work \
    /mnt/test-root

echo "Test environment ready at /mnt/test-root"
echo ""
echo "You can now:"
echo "  1. Chroot into test environment: sudo chroot /mnt/test-root"
echo "  2. Make any changes (install packages, modify configs, etc.)"
echo "  3. Test your changes"
echo "  4. Exit and unmount - all changes are discarded"
echo ""
echo "Your real system remains untouched."
echo ""
echo "To enter test environment:"
echo "  sudo chroot /mnt/test-root /bin/bash"
echo ""
echo "To cleanup when done:"
echo "  exit  # (from chroot)"
echo "  sudo umount /mnt/test-root"

Advanced Techniques

Layered Configuration Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#!/bin/bash
# config-layers.sh - Manage configurations in layers

# Layer structure:
# - Base layer: Default configurations
# - Environment layer: Dev/Staging/Prod specific
# - Local layer: Machine-specific overrides

mkdir -p /etc/myapp/{base,env-prod,env-dev,local,merged}

# Base configuration
cat > /etc/myapp/base/config.ini <<EOF
[app]
name=MyApp
debug=false
port=8080
EOF

# Production overlay
cat > /etc/myapp/env-prod/config.ini <<EOF
[app]
debug=false
database=prod-db.example.com
EOF

# Development overlay
cat > /etc/myapp/env-dev/config.ini <<EOF
[app]
debug=true
database=localhost
EOF

# Mount appropriate environment
ENV=${1:-prod}

sudo mount -t overlay overlay \
    -o lowerdir="/etc/myapp/env-$ENV:/etc/myapp/base",\
upperdir=/etc/myapp/local,\
workdir=/tmp/myapp-work \
    /etc/myapp/merged

echo "Configuration merged for environment: $ENV"
echo "Merged config:"
cat /etc/myapp/merged/config.ini

Temporary Root for Updates

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!/bin/bash
# safe-update.sh - Update system in temporary overlay

echo "Creating temporary overlay for safe updates..."

# Mount tmpfs overlay on root
mkdir -p /tmp/update-overlay/{upper,work}
sudo mount -t overlay overlay \
    -o lowerdir=/,upperdir=/tmp/update-overlay/upper,workdir=/tmp/update-overlay/work \
    /mnt/update-root

echo "Entering update environment..."
sudo chroot /mnt/update-root /bin/bash -c '
    echo "You are now in temporary update environment"
    echo "Install updates, test, and exit when satisfied"
    echo ""

    # Example: Update system
    apt update
    apt list --upgradable

    echo ""
    echo "To apply updates: apt upgrade"
    echo "To test: run your applications"
    echo "If satisfied: exit and run commit-updates.sh"
    echo "If not: just exit (changes discarded)"

    bash
'

echo ""
echo "Exited update environment"
echo "Changes are in /tmp/update-overlay/upper"
echo ""
echo "To commit changes: sudo rsync -av /tmp/update-overlay/upper/ /"
echo "To discard: sudo rm -rf /tmp/update-overlay && sudo umount /mnt/update-root"

Summary

This comprehensive guide covered OverlayFS for Raspberry Pi:

✅ Core Concepts

  • Layer-based filesystem architecture
  • LowerDir, UpperDir, WorkDir, and MergedDir
  • Copy-up operations and whiteout files
  • Multiple lower layer stacking

✅ Basic Usage

  • Manual overlay mounting and management
  • Simple overlay examples
  • Multiple layer configurations
  • File operation behavior

✅ Read-Only Root

  • SD card protection through read-only base
  • Tmpfs overlays for volatile changes
  • Persistent storage areas configuration
  • Making permanent changes safely

✅ Kiosk Mode

  • Self-resetting public displays
  • Automatic reset on reboot
  • Volatile home directories
  • User session isolation

✅ A/B Updates

  • Dual partition setup
  • Safe over-the-air updates
  • Automatic fallback on boot failure
  • Rollback capability

✅ Docker Integration

  • Understanding overlay2 storage driver
  • Container image layers
  • How Docker uses OverlayFS
  • Manual container-like overlays

✅ Performance

  • Performance characteristics
  • Optimization techniques
  • Copy-up overhead mitigation
  • Best practices for Raspberry Pi

✅ Troubleshooting

  • Common issues and solutions
  • Debugging techniques
  • Permission problems
  • Mount option errors

✅ Real-World Applications

  • IoT device with recovery capability
  • Development environment snapshots
  • Live system testing without risk
  • Configuration layer management

✅ Advanced Techniques

  • Layered configuration management
  • Temporary root for safe updates
  • Multi-environment configurations
  • Update commit workflows

Next Steps

Advanced Topics:

  1. AUFS/UnionFS: Alternative union filesystems
  2. Btrfs Snapshots: Filesystem-level snapshots and rollback
  3. LVM Snapshots: Block-level snapshot management
  4. Podman/Buildah: Rootless container tools using overlay
  5. OSTree: Immutable OS deployment system

Related Guides:

With OverlayFS, you can create robust, recoverable systems that protect your Raspberry Pi's SD card, enable safe experimentation, and provide enterprise-grade update capabilities. This is the same technology powering Docker containers and modern immutable infrastructure!