Files
guix-router/plans/guix-openwrt-rpi4-plan.md
2026-10-15 16:47:55 -04:00

16 KiB
Raw Permalink Blame History

Guix + OpenWrt Userland on Raspberry Pi 4B — Router Testbed Plan

Goal: build the Guix-based router stack on a Raspberry Pi 4B first (cheap, real aarch64 hardware, no brick risk), then port the result to the GL-MT6000.

Assumed hardware: Pi 4B (2 GB+ RAM), onboard gigabit Ethernet, one USB 3 gigabit Ethernet adapter (blue USB 3 port), onboard WiFi, 32 GB+ microSD (or USB SSD).


0. Status and gaps (reviewed 2026-10-14)

Where we are. Phases 0–2 done, differently than written below: the Pi is a WISP router — wlan0 is the WAN (WiFi client of the upstream "hakodosh" 5 GHz), onboard eth0 is the LAN in br-lan; no AP (the one radio is busy as client) and no USB NIC in use. Phase 6 is done as deploy-rpi.sh (cross-build

  • guix copy + host-built extlinux.conf); guix deploy/reconfigure can't work here (PITFALLS #28). Phase 3 underway: libubox, ubus, uci, ucode, libnl-tiny, jsonfilter packaged and running (ubusd under Shepherd, UCI seeded). Decisions: README "OpenWrt on Guix" (Shepherd not procd; seed-once UCI).

Gaps this plan didn't cover — now tasks in Phases 3–5:

  1. procd's other jobs, not just starting daemons: hotplug-call (netifd → /etc/hotplug.d/iface/* → firewall/odhcpd reloads); config reload triggers (UCI change → which service reloads; LuCI depends on it); the ubus system object (board, info) and /etc/board.json.
  2. base-files shell library: /lib/functions.sh, /lib/functions/network.sh, /lib/config/uci.sh — sourced by netifd proto scripts and init scripts. Package the needed subset.
  3. FHS path rule: point programs at store paths via flags/config (netifd -p/-h/-c, ubusd -A, ucode search path); activation symlinks into /lib, /usr/share only where nothing else works.
  4. WAN under netifd needs busybox udhcpc + OpenWrt's dhcp.sh / dhcp.script. First netifd step: LAN only; WAN stays on wpa_supplicant + dhcpcd.
  5. DHCP/DNS ownership: OpenWrt's default split -- odhcpd for RA/DHCPv6, dnsmasq for DHCPv4 + DNS, both from /etc/config/dhcp (dnsmasq via OpenWrt's init script since generation 31).
  6. UCI seeds for network, dhcp, firewall, wireless (OpenWrt writes these with board_detect + uci-defaults on first boot).
  7. Recovery rule: never change LAN and WAN in the same deploy — one path in stays up (SSH on WAN or LAN), roll back with deploy-rpi.sh switch N. Get a USB-serial adapter for GPIO 14/15 (console without HDMI).
  8. UCI state backup: Guix rollbacks don't cover /etc/config; add a sysupgrade -b-like backup/restore.
  9. Hardening (unscheduled so far): root password hash is in git, password SSH on, SSH open on WAN, nothing checks LAN isolation from upstream.
  10. Updates: frozen on Guix 1.5.0 (guix pull broken upstream, PITFALLS #1), nonguix pinned — no path for security updates yet.
  11. Topology below (USB WAN, AP on wlan0) is the original idea, kept for the MT6000-like layout; the testbed runs the WISP layout.

1. Target topology

            Internet
               │
        [ wan ] USB 3 GbE adapter      ← DHCP client from ISP/modem
               │
        ┌──────┴──────┐
        │  Pi 4B      │  nftables NAT + firewall
        │  Guix       │
        └──┬───────┬──┘
   [ lan0 ] onboard GbE   [ wlan0 ] onboard WiFi (AP, hostapd)
           └─────┬──────────┘
              br-lan  192.168.10.1/24  ← DHCP/DNS (dnsmasq → later odhcpd)

Why WAN on USB: onboard bcmgenet is the most reliable NIC, so keep it on the LAN side; WAN throughput is usually lower anyway. Swap if your ISP link is faster than your LAN needs.

Interfaces get fixed names (wan, lan0) by MAC address so USB enumeration order never matters.


2. Hardware notes and constraints

Component Driver Notes
Onboard Ethernet bcmgenet Mainline, no firmware needed
USB Ethernet usually r8152 (Realtek) or ax88179_178a (ASIX) Check with lsusb; r8152 may want firmware for some chips
Onboard WiFi brcmfmac (BCM43455) Needs non-free firmware (brcmfmac43455-sdio.*, clm_blob)
  • Linux-libre blocks the WiFi firmware. Use the Nonguix channel's linux kernel and firmware packages. Ethernet works without blobs.
  • brcmfmac AP mode is limited: supports a small number of clients and one band at a time. Fine for a testbed; not production WiFi. Pick 5 GHz (ch 36/40/44/48) for less interference, or 2.4 GHz for range.
  • Bridging the AP interface into br-lan is fine (it's client-mode bridging that's problematic).
  • Power: use the official 5V/3A supply — USB NIC + WiFi under load can brown out weak supplies.
  • Pi boot firmware (start4.elf, fixup4.dat) is non-free and must be on the FAT boot partition. Guix's raspberry-pi-64.tmpl does not install it.

3. Build plan

Phase 0 — Build host setup (x86 workstation)

  • Install/updated Guix; add the Nonguix channel to ~/.config/guix/channels.scm and guix pull.
  • Enable aarch64 emulation on the host (qemu-binfmt-service-type), or plan to cross-compile with --target=aarch64-linux-gnu.
  • Pin channels for reproducibility: guix describe -f channels > channels.scm.
  • Optional: git repo for rpi4.scm, package definitions, and configs.

Phase 1 — Minimal bootable Guix on the Pi

  • Start from Guix's template: ~/.config/guix/current/share/guile/site/3.0/gnu/system/examples/raspberry-pi-64.tmpl (or from the Guix git tree).
  • Switch kernel to Nonguix linux; add Pi firmware + Broadcom WiFi firmware.
  • Ensure bcm2711-rpi-4-b.dtb and Pi boot firmware land on the boot partition.
  • Add user, SSH (openssh-service-type with your public key), serial console.
  • Build: guix system image --system=aarch64-linux rpi4.scm (or --target=aarch64-linux-gnu).
  • Flash: dd if=<image> of=/dev/sdX bs=4M status=progress conv=fsync.
  • Exit check: boots, SSH works over onboard Ethernet, ip link shows eth0 + USB NIC + wlan0, dmesg shows brcmfmac firmware loaded.

Tip: a USB-serial adapter on GPIO 14/15 (UART, add enable_uart=1 to config.txt) saves a lot of guessing when boot fails.

Phase 2 — Router with native Guix services (prove the hardware)

Get a working router before touching OpenWrt userland, so later problems are clearly software, not hardware.

  • udev rules naming NICs by MAC (wan, lan0).
  • br-lan bridge containing lan0 (hostapd adds wlan0 itself via bridge=).
  • WAN DHCP client on wan.
  • Static 192.168.10.1/24 on br-lan.
  • IPv4 forwarding via sysctl-service-type.
  • nftables-service-type with the ruleset below.
  • dnsmasq-service-type for DHCP + DNS on br-lan.
  • hostapd-service-type for the AP.
  • Exit check: wired and wireless clients get addresses and reach the internet; LAN is not reachable from WAN.

Skeleton (verify service and field names against your Guix version's manual):

(use-modules (gnu) (nongnu packages linux) (nongnu system linux-initrd))
(use-service-modules networking ssh sysctl)

(operating-system
  (host-name "pirouter")
  (kernel linux)                          ; Nonguix
  (firmware (list linux-firmware))        ; plus Pi/Broadcom firmware as needed
  ;; bootloader / file-systems: keep from raspberry-pi-64.tmpl
  (services
   (append
    (list
     (service openssh-service-type
              (openssh-configuration
               (password-authentication? #f)
               (authorized-keys `(("you" ,(local-file "id_ed25519.pub"))))))
     (service dhcp-client-service-type
              (dhcp-client-configuration (interfaces '("wan"))))
     (service nftables-service-type
              (nftables-configuration (ruleset (local-file "router.nft"))))
     (service dnsmasq-service-type
              (dnsmasq-configuration
               (extra-options '("--interface=br-lan"
                                "--bind-interfaces"
                                "--dhcp-range=192.168.10.100,192.168.10.250,12h"))))
     (service hostapd-service-type
              (hostapd-configuration
               (interface "wlan0")
               (ssid "pirouter")
               (channel 36)
               (extra-settings "bridge=br-lan
country_code=US
hw_mode=a
ieee80211n=1
ieee80211ac=1
wpa=2
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP
wpa_passphrase=CHANGE-ME"))))
     ;; + a one-shot Shepherd service creating br-lan and adding lan0,
     ;;   or static-networking with a bridge link if your Guix supports it
    (modify-services %base-services
      (sysctl-service-type config =>
        (sysctl-configuration
         (settings (append '(("net.ipv4.ip_forward" . "1"))
                           %default-sysctl-settings))))))))

router.nft:

flush ruleset

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    ct state invalid drop
    iif "lo" accept
    iifname "br-lan" accept
    meta l4proto { icmp, ipv6-icmp } accept
  }
  chain forward {
    type filter hook forward priority 0; policy drop;
    ct state established,related accept
    ct state invalid drop
    iifname "br-lan" oifname "wan" accept
  }
  chain output {
    type filter hook output priority 0; policy accept;
  }
}

table ip nat {
  chain postrouting {
    type nat hook postrouting priority 100;
    oifname "wan" masquerade
  }
}

Phase 3 — Package OpenWrt userland for Guix

Do this on the x86 workstation first (native builds are fast), then cross-build for aarch64.

  • libubox (+ jshn)
  • ubus (+ ubusd)
  • uci
  • ucode (all modules)
  • jsonfilter, libnl-tiny (netifd deps)
  • udebug (netifd dependency the plan missed)
  • netifd (+ its ucode/shell runtime files, hotplug-call) — LAN
  • base-files shell library subset (functions.sh, network.sh, …)
  • busybox udhcpc wiring for netifd's dhcp proto
  • cross glibc popen()/system() broken (PITFALLS #35): fixed by cross-libc-fix (/bin/sh)
  • odhcpd (RA + DHCPv6 on lan, ULA)
  • odhcp6c (DHCPv6/PD client on the WAN; upstream has no IPv6 yet)
  • firewall4 (ucode + nftables)
  • ubox (logd/logread, validate_data) — LuCI log page, init scripts
  • rpcd, uhttpd + ustream-ssl, LuCI (ucode), with iwinfo, lucihttp, cgi-io, rpcd-mod-luci (generation 28)
  • Goal (user, 2026-10-15): run TollGate (OpenTollGate tollgate-module-basic-go, package tollgate-wrt; TollGate OS supports the GL-MT6000). Package DEPENDS: nodogsplash, jq, ca-bundle (+ cron, Go binaries tollgate-wrt, tollgate). Its postinst runs three uci-defaults scripts (99-tollgate-setup is ~2800 lines of UCI edits incl. network, wireless, firewall, dhcp, uhttpd), then /etc/init.d/{network,firewall,dnsmasq,uhttpd, nodogsplash,tollgate-wrt} actions and wifi reload; ships an rpcd plugin + ACL, an iface hotplug script, nftables.d includes. Done so far: dnsmasq from UCI, init.d shims, rpcd/LuCI, hotplug, fw4.
  • dnsmasq configured from UCI (OpenWrt's init script; generation 31)
  • /etc/board.json (LuCI's port view; matters for the MT6000's DSA)
  • Clock at boot without RTC: sysfixtime-service-type (generation 34)
  • Trim the ~40 x86 store items cross builds leave in the closure (cross gcc via gcc-cross-*-lib, guix's and man-db's native refs; PITFALLS #39) -- dead weight in a 2.6 GB system

Keep these in a personal Guix channel (e.g. guix-openwrt) so both the Pi and the MT6000 can use them.

Phase 4 — Run OpenWrt userland under Shepherd

  • ubusd under Shepherd (ubus-service-type).
  • netifd owns the LAN (static br-lan) and the WAN (dhcp on the wpa_supplicant-managed wlan0, attached as an external device).
  • odhcpd (generation 18).
  • firewall4 (generation 19; replaces router.nft).
  • procd compatibility (generation 21): hotplug-call; procd-compat for config reload triggers (config.change → herd reload), reload_config, ubus system (board, info, reboot).
  • /etc/init.d/* shims for our Shepherd services, /etc/rc.d links (rpcd rc, LuCI Startup page, TollGate's postinst calls).
  • Run OpenWrt procd init scripts as Shepherd services (openwrt-init-script-service-type, procd-compat's openwrt-rc; generation 38; dnsmasq and uhttpd use it). Limits: first instance only, no procd user, triggers/jails ignored.
  • uhttpd from /etc/config/uhttpd (OpenWrt's init script; generation 39).
  • nodogsplash 5.0.2 + OpenWrt's files (generation 42); auth/deauth with ndsctl verified on the LAN.
  • WAN on eth0, WiFi AP on the onboard radio with OpenWrt's wireless stack (wpad, wifi-scripts, netifd wireless; generations 43-45).
  • TollGate v0.5.0 (tollgate-wrt) packaged and running (generation 46): setup applied, API on :2121 answers, tollgate status ok.
  • Pay through the portal from a client: tried from a phone 2026-10-15, the Cashu payment failed (not investigated yet).
  • TollGate on the MT6000 (private SSID needs a second AP interface).
  • Provide the scripts netifd expects (/lib/netifd/proto/*.sh, /lib/functions.sh, etc.) from the netifd package via Guix-managed files.
  • Replace Phase 2 pieces one at a time:
    1. netifd replaces udev-bridge + DHCP client + static addressing
    2. odhcpd (or keep dnsmasq) for DHCP/RA done: odhcpd IPv6, dnsmasq DHCPv4/DNS
    3. firewall4 replaces the hand-written router.nft done
    4. netifd wireless (mac80211.sh) replaces hostapd-service-type — optional
  • Exit check after each swap: same behavior as Phase 2.

Phase 5 — UCI config (decided: seed once, then UCI owns it)

  • uci-service-type seeds /etc/config/NAME only when absent.
  • Seeds for network, dhcp, firewall, wireless.
  • Backup/restore of /etc/config (Guix rollbacks don't cover it).
  • (maybe) Scheme records that render seeds, for the MT6000 variant.

Phase 6 — Deploy and upgrade workflow

  • Over SSH without reflashing: deploy-rpi.sh (not guix deploy — PITFALLS #28); switch N rolls back.
  • Rollback tested for real (U-Boot menu entry 2 on 2026-10-14).
  • Keep a spare SD card with a known-good image.
  • Hardening pass (see §0 gap 9) before anything non-lab.

Phase 7 — Port to GL-MT6000

  • Reuse the userland channel and UCI generation unchanged.
  • Swap kernel/DTS/bootloader per guix-openwrt-mt6000-plan.md.
  • Map interface names: MT6000's DSA ports replace wan/lan0; mt76 replaces brcmfmac.

4. Troubleshooting checklist

  • No WiFi: dmesg | grep brcmfmac — firmware missing or blocked by linux-libre.
  • USB NIC missing: lsusb, dmesg | grep -i r8152 — driver/firmware; try the other USB 3 port; check power supply.
  • Interfaces renamed/swapped: verify udev MAC rules.
  • Clients get no IP: dnsmasq bound to br-lan? nft list ruleset allows input on br-lan?
  • No internet from LAN: sysctl net.ipv4.ip_forward, masquerade rule on wan.
  • AP not visible: iw list (AP mode supported?), country code set, channel valid for band.

References