No description
Find a file
aaron 87cd6d7546 Incorporate all runtime fixes into scripts and document gotchas
setup-gadget.sh now includes:
- bind-dynamic instead of bind-interfaces for dnsmasq
- usb-gadget-up.service for reliable boot activation
- NetworkManager dispatcher to restart dnsmasq on usb0 up

setup-mesh.sh cleaned up:
- Removed unused MESH_CELL and MESH_CHANNEL variables
- Added inline comments explaining brcmfmac limitations

README.md:
- Simplified Phase 1 to single script invocation (no manual steps)
- Added Lessons Learned section with 13 documented gotchas covering
  dwc2 overlay placement, dnsmasq race conditions, NM autoconnect
  reliability, brcmfmac IBSS quirks, PATH issues, and more

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-24 18:32:46 -04:00
scripts Incorporate all runtime fixes into scripts and document gotchas 2026-04-24 18:32:46 -04:00
README.md Incorporate all runtime fixes into scripts and document gotchas 2026-04-24 18:32:46 -04:00

RPi Zero 2W Mesh Network Setup

batman-adv mesh network across three Raspberry Pi Zero 2W devices, with USB Ethernet gadget mode for out-of-band management from a Mac.

Hardware

  • 3x Raspberry Pi Zero 2W running Debian 13 (trixie)
  • 3x USB data cables (micro-USB to host)
  • USB hub (optional, for connecting all three simultaneously)

Node Inventory

Node Hostname USB Gadget IP Mesh IP (bat0) USB Subnet
1 meshy01 10.55.1.1 10.99.0.1 10.55.1.0/24
2 meshy02 10.55.2.1 10.99.0.2 10.55.2.0/24
3 meshy03 10.55.3.1 10.99.0.3 10.55.3.0/24

Architecture

Mac (management host)
 ├── USB cable ── meshy01 (10.55.1.1) ──┐
 ├── USB cable ── meshy02 (10.55.2.1) ──┼── batman-adv mesh (10.99.0.0/24)
 └── USB cable ── meshy03 (10.55.3.1) ──┘    via wlan0 IBSS ad-hoc
  • USB gadget mode (dwc2 + g_ether): provides Ethernet-over-USB for SSH management
  • batman-adv: layer 2 mesh routing over wlan0 in IBSS (ad-hoc) mode
  • wlan0 is dedicated to the mesh — WiFi is not available for normal network access once mesh is configured

Setup Process

Prerequisites

  • SSH access to all three Pis (initially over WiFi or any network)
  • Passwordless sudo configured for the aaron user on each Pi:
    echo "aaron ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/010_aaron-nopasswd
    

Phase 1: USB Gadget Mode

This configures each Pi to present as a USB Ethernet device when connected to a host computer via the data micro-USB port.

  1. Copy and run the gadget setup script on each Pi:

    scp scripts/setup-gadget.sh aaron@<PI_WIFI_IP>:/tmp/
    ssh aaron@<PI_WIFI_IP> 'sudo bash /tmp/setup-gadget.sh <NODE_NUM>'
    

    Where <NODE_NUM> is 1, 2, or 3 corresponding to the node.

    This script handles everything in one shot:

    • Enables the dwc2 device tree overlay in /boot/firmware/config.txt (under [all])
    • Adds modules-load=dwc2,g_ether to /boot/firmware/cmdline.txt
    • Creates /etc/modules-load.d/usb-gadget.conf for module autoloading
    • Installs and configures dnsmasq with bind-dynamic to serve DHCP on usb0
    • Creates a NetworkManager connection profile (usb0-gadget) with a static IP
    • Installs usb-gadget-up.service to reliably activate usb0 on boot
    • Installs a NetworkManager dispatcher to restart dnsmasq when usb0 comes up
  2. Reboot all Pis, then connect them to the Mac via the data micro-USB port (the inner port, closer to the HDMI connector — not the power-only port on the edge).

  3. Verify from the Mac:

    # Check interfaces appeared
    networksetup -listallhardwareports | grep -A2 "Raspberry Pi"
    
    # Ping each node
    ping 10.55.1.1  # meshy01
    ping 10.55.2.1  # meshy02
    ping 10.55.3.1  # meshy03
    
    # Or use the helper script
    bash scripts/mac-usb-check.sh
    

    If a Pi is reachable over WiFi but not USB, activate the connection manually:

    ssh aaron@<PI_WIFI_IP> 'sudo nmcli con up usb0-gadget'
    

Phase 2: batman-adv Mesh Network

Once all three Pis are accessible over USB, deploy the mesh networking.

  1. Copy and run the mesh setup script on each Pi (over USB):

    scp scripts/setup-mesh.sh aaron@10.55.1.1:/tmp/
    ssh aaron@10.55.1.1 'sudo bash /tmp/setup-mesh.sh 1 --start'
    
    scp scripts/setup-mesh.sh aaron@10.55.2.1:/tmp/
    ssh aaron@10.55.2.1 'sudo bash /tmp/setup-mesh.sh 2 --start'
    
    scp scripts/setup-mesh.sh aaron@10.55.3.1:/tmp/
    ssh aaron@10.55.3.1 'sudo bash /tmp/setup-mesh.sh 3 --start'
    

    Or use the deploy helper:

    bash scripts/deploy-mesh.sh
    

    This script:

    • Installs batctl
    • Loads the batman-adv kernel module
    • Removes wlan0 from NetworkManager control
    • Deletes the existing WiFi connection
    • Creates /usr/local/bin/mesh-start.sh which:
      • Sets wlan0 to IBSS (ad-hoc) mode
      • Joins the rpi-mesh network on channel 1 (2412 MHz)
      • Adds wlan0 to batman-adv
      • Brings up bat0 with the node's mesh IP
    • Creates a mesh-network.service systemd unit (enabled at boot)
  2. Verify the mesh (allow ~10 seconds for neighbor discovery):

    # Check neighbors from any node
    ssh aaron@10.55.1.1 'sudo batctl n'
    
    # Check originator table (routing)
    ssh aaron@10.55.1.1 'sudo batctl o'
    
    # Ping between nodes over the mesh
    ssh aaron@10.55.1.1 'ping -c3 10.99.0.2'  # meshy01 -> meshy02
    ssh aaron@10.55.1.1 'ping -c3 10.99.0.3'  # meshy01 -> meshy03
    ssh aaron@10.55.2.1 'ping -c3 10.99.0.3'  # meshy02 -> meshy03
    

Important Notes

USB Port Selection

The Pi Zero 2W has two micro-USB ports. Use the inner port (labeled USB, closer to the HDMI connector) for data. The outer port (labeled PWR) is power-only.

WiFi After Mesh Setup

Once the mesh is configured, wlan0 is dedicated to the ad-hoc mesh network. The managed WiFi connection is deleted and NetworkManager is told to ignore wlan0. USB is your only management path. To restore WiFi:

sudo systemctl stop mesh-network
sudo nmcli con add type wifi ifname wlan0 ssid "YourSSID" wifi-security.key-mgmt wpa-psk wifi-security.psk "YourPassword"

brcmfmac Driver Limitations

The Broadcom WiFi driver (brcmfmac) on the Pi Zero 2W supports IBSS mode but does not accept HT20, fixed-freq, or explicit BSSID parameters in the iw ibss join command. The mesh scripts use the simplified form:

iw dev wlan0 ibss join rpi-mesh 2412

Services

Each Pi runs these services related to the mesh:

Service Purpose
mesh-network.service Starts/stops batman-adv mesh on boot
usb-gadget-up.service Activates usb0 NetworkManager connection on boot
dnsmasq.service Serves DHCP to the Mac on the USB interface

Troubleshooting

# Check mesh service status
sudo systemctl status mesh-network

# View mesh startup logs
sudo journalctl -u mesh-network

# Check if wlan0 is in IBSS mode
sudo /usr/sbin/iw dev wlan0 info

# Check batman-adv interfaces
sudo batctl if

# Restart the mesh
sudo systemctl restart mesh-network

# Check USB gadget status
ip addr show usb0
sudo systemctl status usb-gadget-up
sudo systemctl status dnsmasq

Performance

Tested with all three nodes sitting next to each other on 2.4 GHz channel 1 (IBSS ad-hoc mode).

Latency (ICMP ping, 10 packets)

Link Min Avg Max Packet Loss
meshy01 <-> meshy02 2.0 ms 3.9 ms 6.6 ms 0%
meshy01 <-> meshy03 1.9 ms 3.1 ms 7.3 ms 0%
meshy02 <-> meshy03 2.0 ms 6.7 ms 17.0 ms 0%

Throughput (iperf3, 10-second TCP)

Link Sender Receiver
meshy01 -> meshy02 16.9 Mbps 16.1 Mbps
meshy01 -> meshy03 17.0 Mbps 16.1 Mbps
meshy02 -> meshy03 15.9 Mbps 14.9 Mbps

The Pi Zero 2W's Broadcom WiFi chip tops out around 20 Mbps in ideal conditions on 2.4 GHz. The ~15-17 Mbps observed is typical with batman-adv overhead factored in. Sufficient for IoT, sensor data, command/control, or light file transfers.

Lessons Learned / Gotchas

These are the non-obvious issues encountered during setup that you'll want to know if adapting this for another project.

USB Gadget Mode

  1. dwc2 overlay must be in [all] section of config.txt. The default Raspberry Pi OS config.txt has dtoverlay=dwc2,dr_mode=host under [cm5], which only applies to Compute Module 5. A naive grep for dtoverlay=dwc2 will find this and skip adding it to [all], silently breaking gadget mode on the Pi Zero 2W.

  2. dwc2 is built into the kernel, not a loadable module. On this kernel (6.12.75+rpt-rpi-v8), lsmod | grep dwc2 returns nothing even when dwc2 is working. Check dmesg | grep dwc2 instead to verify the driver loaded.

  3. dnsmasq must use bind-dynamic, not bind-interfaces. With bind-interfaces, dnsmasq fails at boot if usb0 doesn't exist yet (which it won't until a USB cable is connected). bind-dynamic allows dnsmasq to start and then bind to usb0 when it appears.

  4. NetworkManager won't reliably auto-activate the usb0 connection. Even with connection.autoconnect yes, the usb0-gadget connection may not activate because usb0 appears after NM's startup scan. The usb-gadget-up.service (with BindsTo=sys-subsystem-net-devices-usb0.device) ensures activation when the device exists.

  5. dnsmasq needs a dispatcher restart. Even with bind-dynamic, if dnsmasq started before usb0 existed, it won't retroactively serve DHCP on usb0 when it appears. The NetworkManager dispatcher script (99-usb0-dnsmasq) restarts dnsmasq whenever usb0 comes up.

  6. Each node needs a different USB subnet. Using the same subnet (e.g., 10.55.0.0/24) for all nodes works if you only connect one at a time, but fails when multiple Pis are connected to the same Mac simultaneously. Each node uses 10.55.{1,2,3}.0/24.

  7. macOS may need manual DHCP renewal. If a Pi's dnsmasq wasn't ready when the Mac first connected, macOS falls back to a 169.254.x.x link-local address and may not retry DHCP. Run sudo ipconfig set enX DHCP on the Mac, or unplug/replug the USB cable after verifying dnsmasq is running on the Pi.

  8. VPNs can steal routes. If a VPN is active on the Mac, it may capture traffic destined for the 10.55.x.x subnets. Disable the VPN or add route exceptions.

batman-adv Mesh

  1. brcmfmac rejects HT20/fixed-freq/BSSID in IBSS join. The Broadcom WiFi driver supports IBSS mode (confirmed via iw list), but iw dev wlan0 ibss join rpi-mesh 2412 HT20 fixed-freq 02:12:34:56:78:9A fails with "Invalid argument (-22)". Only the simplified form works: iw dev wlan0 ibss join rpi-mesh 2412.

  2. /usr/sbin is not in PATH for systemd services. The iw binary installs to /usr/sbin/iw. When mesh-start.sh runs via systemd, /usr/sbin may not be in PATH. The script must explicitly export PATH="/usr/sbin:/usr/bin:/sbin:/bin:$PATH".

  3. wlan0 must be unmanaged by NetworkManager. Before putting wlan0 into IBSS mode, NetworkManager must be told to ignore it via /etc/NetworkManager/conf.d/mesh-unmanaged.conf. Otherwise NM will fight the manual interface configuration.

  4. No internet after mesh setup. Once wlan0 is in ad-hoc mode for the mesh, the Pi has no internet access. Install all packages (batctl, iw, wireless-tools, iperf3, etc.) before or during mesh setup. If packages are needed later, transfer .deb files via USB from the Mac.

  5. Allow ~10 seconds for neighbor discovery. batman-adv uses OGM (Originator Messages) to discover neighbors. The first batctl n check may show nothing if run immediately after starting the mesh.

Scripts

Script Description
scripts/setup-gadget.sh <node> Configures USB Ethernet gadget mode (run as root)
scripts/setup-mesh.sh <node> [--start] Installs and configures batman-adv mesh (run as root)
scripts/deploy-mesh.sh Deploys mesh setup to all nodes over USB (run from Mac)
scripts/mac-usb-check.sh Checks USB Ethernet connectivity from the Mac