guix-router/docs/PITFALLS.md

Pitfalls encountered (and fixes)

Numbered for cross-reference from the README. All verified on this host with guix 1.5.0 / Debian 13, 2026-10.

1. guix pull is broken upstream (stale guile-zlib hash)

Channel tip ded4de6 pins guile-zlib-0.2.2 to sha256 04p9lb…mnx2, but Codeberg regenerated the archive/v0.2.2.tar.gz asset: the current archive hashes to 1h9q7rw8…wlv3g and the stable release asset to 011flqx…g2hz. Every guix pull of that tip fails with a download hash mismatch inside compute-guix-derivation.

Workaround: don't pull — build with the 1.5.0 CLI's bundled channel everything else keeps working (substitutes for 1.5.0 are on the CI). Re-test guix pull periodically; when it works, re-pin nonguix to match.

Diagnosis note: guix hash prints nix-base32 by default (-f base32 is lowercase RFC-4648, and the daemon's "actual hash:" error line is a different representation again) — don't chase ghosts comparing them.

2. Nonguix master requires guix newer than 1.5.0

nongnu/packages/linux.scm defines linux-7.2 → (corrupt-linux linux-libre-7.2); loading aborts at linux-libre-7.2: unbound variable and leaves a half-initialized module whose public interface is only (corrupt-linux). Symptom in configs: error: linux: unbound variable with no helpful backtrace.

Fix: pin nonguix to f5338f6 (last commit before linux-6.18/7.x were added; provides linux = 6.17).

3. Missing substitute signing keys → silent bootstrap from source

The binary tarball ships /etc/guix/acl without the CI keys. Builds then "work" but compile everything (you'll notice glibc-bootstrap-0.drv building and Python-3.5.9.tar.xz unpacking). Diagnostic: guix weather <pkg> → "substitutes from 'https://ci.guix.gnu.org' are unauthorized". Fix: authorize the .pub files in $GUIX_PROFILE/share/guix/ (see HOST-SETUP §5). aarch64 substitutes for 1.5.0 are 100% available on the CI.

4. /tmp is a 7.8 G tmpfs

Kernel build dies after ~50 min with fatal error: error writing to /tmp/guix-build-linux-…/cc*.s: No space left on device. Fix: TMPDIR=/var/guix-tmp in the daemon unit (world-writable, on the big VG), Environment= line — not the shell.

5. --system=aarch64-linux ≠ cross build

On an x86_64 host, --system means foreign: the entire dependency chain is built for aarch64 and executed under qemu-binfmt (check with ps aux | grep binfmt-P — an aarch64 cc1 under qemu-aarch64 = foreign). Foreign kernel: 3–6 h. --target=aarch64-linux-gnu cross-compiles natively: ~50 min. Always --target for images from this host.

6. Cross-built images install an x86 GRUB; bootloader chains are ignored

make-grub-efi-installer explicitly no-ops in disk-image mode, and efi-bootloader-chain files/packages are only installed by guix system init / the installer — not by guix system image. The efi-raw image ends up with host-arch EFI/BOOT/BOOTX64.EFI (verified: file says x86-64) and none of the chain's config.txt/u-boot/DTB/firmware.

Fix (this repo's design): don't use GRUB at all. rpi4-image.scm declares the FAT boot partition (firmware, U-Boot, config.txt, DTB) and the ext4 root directly, and rpi4.scm uses a u-boot-bootloader variant with no raw-disk installer that only writes extlinux.conf (see README "Boot flow"). An earlier prepare-rpi-image.sh post-processed an efi-raw image instead; it was removed once the declarative image booted on hardware.

7. make-raspi-bcm28-dtbs (gnu packages raspberry-pi, 1.5.0) is broken

Its #:install-plan gexp evaluates (search-input-directory %build-inputs …) at compile time → Apply to non-list: #<gexp …>. Define your own trivial DTB package if needed.

8. copy-build-system install-plan gotcha

("start4.elf" ".") → copy-file: Is a directory (the destination is used as the full path, not appended to). The firmware package uses an explicit trivial-build-system builder instead.

9. initrd module names must match .ko file names exactly

sdhci_arasan → not found; the module is sdhci-of-arasan (dashes!). mmc_block, sdhci_pltfm do use underscores. Symptom: kernel module not found "sdhci_arasan" during linux-modules build.

10. trivial-build-system builders: PATH is empty

tar xzf needs gzip present, not just tar+xz (set-path-environment-variable "PATH" '("bin") (list tar xz gzip)).

11. guix build -f quirks

• the file's last expression must be the package variable (hint mentions my-package);
• GUIX_PACKAGE_PATH is honored by guix build/guix system but not by guix repl (use guix repl -L DIR);
• flags are -f/--format lowercase (capital -F is a different option).

12. Image partition labels ≠ config labels

With the stock efi-raw image type, genimage names partitions Guix_image / GNU-ESP regardless of the file-system-labels in the OS config, and Guix mounts by label → boot fails to find root. Fix: declare the partitions yourself (rpi4-image.scm sets (label "RPI-BOOT") / (label "Guix")) so they match rpi4.scm's file-systems.

13. extlinux path prefix semantics (U-Boot)

U-Boot's syslinux loader resolves image paths relative to the directory containing extlinux.conf (prefix). Put the config at the partition root /extlinux/extlinux.conf so the absolute-looking /gnu/store/… paths resolve from the filesystem root.

14. 1.5.0 config API differences vs current guix

• operating-system has no password/hashed-root-password field and no (plain "…") — set root via (user-account (name "root") (group "root") (system? #t) (password "<crypt-hash>")) (generate with openssl passwd -6).
• agetty-configuration takes a single tty string (not terminals list).
• usbutils/iw live in (gnu packages linux), ethtool in (gnu packages networking), json-c in (gnu packages web); there is no (gnu packages json)/(gnu packages usb).
• substitute-keyword-arguments needs (guix utils).
• guix system image takes no -o (prints the store path); positional args are ACTION … FILE.

15. Shell footguns during automation

• pkill -f 'guix system image' matches the invoking shell's own command line → kills your session; use pgrep + explicit PID.
• The bash-tool timeout SIGKILLs the process group — nohup alone does not survive; launch long builds with setsid nohup … & disown.

16. Nonguix linux can't mount a Pi 4 SD card (or see its Ethernet)

Its arm64 config has CLK_BCM2835 off. emmc2's clock is the cprman (clocks = <&clocks BCM2711_CLOCK_EMMC2>), and so is the mini-UART's, so the SD card and serial console never probe; the SD regulators also hang off the firmware GPIO expander (GPIO_RASPBERRYPI_EXP=m). BCMGENET is off too. The SD driver is sdhci-iproc, not sdhci-of-arasan (that was a wrong guess in the old initrd list). Fix: packages/rpi4-linux.scm (customize-linux on top of nonguix linux), SD path built in, initrd-modules '().

MMC_BLOCK=y silently stays m unless RPMB=y (depends on RPMB || !RPMB); verify-config catches it. DRM_SIMPLEDRM=y is impossible with DRM=m.

17. Stock Pi U-Boot layout overlaps a distro kernel

rpi.env: kernel_addr_r=0x80000, fdt_addr_r=0x02600000, ramdisk_addr_r=0x02700000. booti relocates an Image with flags bit 3 to the 2 MB boundary (0x200000); our image_size is 36.4 MB → ends 0x2660000, on top of FDT and initrd. Fix: patches/rpi-env-layout.patch (kernel at 0x200000, everything else ≥ 0x08000000).

18. Mainline Pi 4 serial console is ttyS1, not ttyAMA0

With the upstream DT the PL011 (ttyAMA0) is wired to Bluetooth; GPIO 14/15 is the mini-UART, alias serial1 → ttyS1 (verified in QEMU: fe215040.serial: ttyS1 at MMIO 0xfe215040).

19. Root user-account needs (uid 0)

Without it Guix still prepends its own passwordless %root-account ("the following accounts appear more than once: root").

20. QEMU raspi4b (10.0) can't boot a mainline kernel on the stock DTB

Unemulated: AON block at 0x7ef00000 (L2 irqchip → external abort; DVP clock gate → abort in clk_disable_unused), firmware GPIO expander (-EIO), so emmc2 regulators never probe; mini-UART IRQ storm → RCU stall on /dev/console open. QEMU attaches the SD card to the WiFi SDHCI (7e300000). Hence U-Boot → kernel in QEMU is silent forever. run-qemu-rpi4.sh boots the image's kernel/initrd directly with a patched DTB (see its header); none of this applies to real hardware.

21. pkill -f / grep pattern | kill match your own shell (again)

Any filter over full command lines matches the shell running it, because the pattern is in its argv. Use pgrep -x <comm> (≤15 chars) or $!.

22. A plain symlink into the store is not a GC root

ln -s /gnu/store/…-guix-rpi4 foo.img doesn't stop guix gc from deleting the image. build-rpi-image.sh uses guix system image -r <link>, which registers the link as an indirect root (guix gc --list-roots shows it). Remove old images with guix gc -d <path> — it refuses live paths. Avoid a blanket guix gc: the 50-min linux-rpi4 cross build isn't rooted unless an image or system that uses it is.

23. dnsmasq (1.5.0) doesn't cross-build

Its install-dbus phase uses %output, which only exists in native builds: Unbound variable: %output. rpi4.scm defines dnsmasq/cross-fixed (same phase via outputs) and passes it as the dnsmasq service's package.

24. linux-firmware subset filter dropped the Pi 4 WiFi firmware

brcmfmac error -2 (ENOENT), no wlan0. In WHENCE the 43455 .bin and .clm_blob are Link: brcm/… -> ../cypress/…, and the Pi 4 NVRAM entry is quoted (File: "brcm/…raspberrypi,4-model-b.txt"). The old filter tested link targets and unquoted names against ^brcm/, so all three were skipped. Now: keep any entry whose own path matches ^"?(brcm|cypress|rtl_nic)/. Check with ls $(guix build … %rpi-linux-firmware)/lib/firmware/brcm | grep 43455-sdio.

25. sshd requires 'networking — don't let the WAN provide it

openssh-service-type (inetd-style under Shepherd) requires networking. With dhcpcd on wlan0 providing it and wlan0 absent, Shepherd accepted TCP on :22 but never spawned sshd -i → "timed out during banner exchange" even from the LAN. dhcpcd now provides wan; the LAN static-networking provides lan networking.

26. Onboard Ethernet name depends on probe order

GENET and USB NICs (r8152…) are both modules; plugging in a USB NIC can make it eth0. Bridge ports are matched by MAC (network-link (mac-address …)).

27. 2.4 GHz WAN: associated, but DHCP never answered

On hakodosh 2.4 GHz ch 1 (UniFi AP, -43 dBm) the Pi associated and got wpa_state=COMPLETED, but iw dev wlan0 station dump showed ~45% tx failed at MCS0 (7.2 Mbit/s) while RX ran at 72 Mbit/s; broadcasts (DHCP DISCOVER) are unacked, so they were silently lost → IPv4LL fallback. An ARP reply from the gateway proved nothing: UniFi APs can proxy-ARP. No USB 3 device was attached, so not the usual Pi 4 USB3 2.4 GHz noise. On 5 GHz (ch 44, -49 dBm): 0 tx failed, 162 Mbit/s, lease at once. Fix: freq_list= of 5 GHz channels in /boot/firmware/wpa_supplicant.conf. Diagnose with station dump (tx failed / tx bitrate) before blaming DHCP. Likely cause, found later: the router this project is meant for (GL-MT6000, on the bench and powered) was flooding the 2.4 GHz channels; switched off on 2026-10-14. 5 GHz stays pinned anyway.

28. Updating the Pi: no guix system reconfigure/switch-generation on it

On the Pi both would build natively for aarch64: reconfigure evaluates the config there (native derivations ≠ our cross-built ones → kernel from source), and switch-generation/roll-back build extlinux.conf with a native Guile the Pi doesn't have (its Guix is cross-built), and need substitutes over TLS with a 1970 clock. They also look the bootloader up by name, and u-boot-rpi4 isn't in any module. deploy-rpi.sh does it from the workstation instead: cross-build, guix copy, generation link, host-built extlinux.conf (scripts/extlinux-conf.scm), activate, reboot.

Gotchas found on the way:

• activate picks the system from GUIX_NEW_SYSTEM; run bare, it re-activates /run/current-system (the old system).
• read-boot-parameters returns the stored kernel args only; root=, gnu.system=, gnu.load= come from (private) bootable-kernel-arguments.
• guix archive --authorize on the Pi turns /etc/guix/acl into a regular file, which the next activation moves to acl.bak and replaces with the declared ACL — declare the workstation key in guix-service-type.
• The image build swaps root (file-system-label "Guix") for the image's UUID, so the first deploy's system differs from the image's even with no config change (harmless; same partition).

29. Time sync on a Pi without RTC: chrony, not ntpd/openntpd

The clock starts at 1970 every boot and the WAN arrives ~20 s after the time service starts. ntp-service-type doesn't cross-build (ntp pulls perl-net-ssleay, which links the build host's x86 OpenSSL). OpenNTPD only steps the clock in a short window at startup, then slews — hopeless for a 56-year offset. chronyd with makestep 1 -1 steps whenever it's > 1 s off, any time. 1.5.0 has no chrony service: rpi4.scm runs chronyd -n from a shepherd service, and needs a chrony system account (built-in privilege drop: "Could not get user/group ID of chrony").

30. 90% packet loss on the LAN cable: Energy-Efficient Ethernet

Pi 4 onboard port (BCM54213PE/GENET) ↔ workstation Intel e1000e: with EEE active, the Pi received ~83 of thousands of frames, with no error counters; ping RTTs of ~1 s, ARP failures, SSH "timed out during banner exchange", no DHCP. It looked like a hung Shepherd (sshd-1 disabled) or breakage from the clock step — it wasn't. Intermittent: some boots were fine. ethtool --set-eee eth0 eee off → 0% loss. rpi4.scm runs that from a one-shot service after 'lan. Two bugs on the way: br-lan inherits the port's MAC and sorts before eth0 (skip interfaces without /sys/class/net/*/device), and service PATH isn't reliable (pass sleep by store path). Diagnose with ethtool -S eth0 (rx_packets vs. what the peer sent) and ethtool --show-eee eth0. SSH on the WAN (192.168.88.222) was the way in while the cable was down.

31. Image root is only 6 GB: grow it on first boot

rpi4-image.scm makes a 6 GiB root so the image stays small; on a 119 GB card the rest was unused (3.2 GB free after a few deploys). The grow-root one-shot service (rpi4.scm) runs every boot: if root is ext4, the last partition of an MBR disk, and ≥ 1 MiB is free after it, it does what worked by hand on the live Pi — echo ,+ | sfdisk --no-reread --no-tell-kernel -N 2 /dev/mmcblk1, resizepart /dev/mmcblk1 2 <sectors> (tells the kernel about a partition on a busy disk; plain sfdisk can't), resize2fs online (the image's ext4 has resize_inode). Messages go to the kernel log as grow-root: …. Tested in QEMU (run-qemu-rpi4.sh pads the SD to 8 GiB: 6144 → 7932 MiB) and as a no-op on the already-grown Pi.

32. OpenWrt packages: CMake + pkg-config when cross-building

OpenWrt's CMakeLists use PKG_SEARCH_MODULE; in Guix cross builds only aarch64-linux-gnu-pkg-config exists → "Could NOT find PkgConfig". -DPKG_CONFIG_EXECUTABLE=<bare name> is resolved relative to the build dir; which in #:configure-flags runs before PATH is set. What works: a phase before 'configure that sets PKG_CONFIG to (which (pkg-config-for-target)). Trap: build that gexp in a procedure called from arguments — a top-level gexp evaluates pkg-config-for-target at module load (no target → native name), which returns #f, and (setenv "PKG_CONFIG" #f) silently unsets it. Also: jshn.sh calls jshn from PATH — patched to the store path. ubusd only allows root unless ACL files say otherwise (-A dir).

33. OpenWrt package notes: pins, ucode module path, jsonfilter

• Pin to the commits OpenWrt main builds (package/*/*/Makefile, PKG_SOURCE_VERSION; git.openwrt.org's gitweb redirects, the GitHub mirror raw.githubusercontent.com/openwrt/openwrt/main/... works), not each project's master, so netifd/firewall4/LuCI get the versions they're tested with. ucode lives at github.com/jow-/ucode, not git.openwrt.org.
• ucode's module search path is compiled in (LIB_SEARCH_PATH), with no environment override; scripts can't pass -L. Built with its own prefix plus /run/current-system/profile/{lib/ucode/*.so,share/ucode/*.uc} so firewall4/LuCI modules installed in the system profile are found.
• jsonfilter's CMake builds/runs contrib/lemon relative to the source tree: #:out-of-source? #f. Installed as jsonpath; symlinked to jsonfilter, the name OpenWrt scripts use.
• Testing a ubus object: don't call it from the same uloop that serves it (the reply can never be dispatched → null); call from another process.

34. netifd on Guix

• Needs libudebug (OpenWrt project/udebug), not listed in our plan; main.c uses it unconditionally.
• Its runtime files (main.uc, proto*.uc, utils.uc, ifup/ifdown, proto/dhcp.sh) live in OpenWrt's main tree (package/network/config/netifd/files), not in netifd.git: %openwrt-tree in openwrt.scm is that tree at the pinned commit, trimmed by a snippet to the directories we use.
• -p sets the script dir for shell protocols only; ucode.c loads DEFAULT_MAIN_PATH "/main.uc" (compile time). Patch DEFAULT_MAIN_PATH in netifd.h to $out/lib/netifd instead of symlinking /lib/netifd; netifd-proto.sh also hardcodes /lib/netifd and /usr/share/libubox/jshn.sh (patched).
• In-source build (make_ethtool_modes_h.sh runs relative to the tree).
• Bridge ports are by name only (no MAC match): eth0, see #26.
• Test before touching the Pi: sudo unshare -n + a dummy port + ubusd + netifd -s SOCK -c CONFDIR -h HOOK -r RESOLV shows the bridge, the network.interface.* ubus objects and the hotplug event.
• Deploy LAN changes over the WAN path (PI=pirouter-wan), never both.

35. Cross-built glibc: popen()/system() exec an x86-64 bash (fixed)

On the Pi, every popen()/system() from glibc fails silently: the cross-built glibc (glibc-cross-aarch64-linux-gnu) has _PATH_BSHELL patched to the build host's bash-static (od -j18 -N2 → 3e 00 = x86-64; running it: "Exec format error"). Guix 1.5.0 cross-base.scm marks this ;FIXME: static-bash. Symptom here: netifd registered only the built-in static protocol — its shell-handler discovery uses popen — so wan stayed proto none/NO_DEVICE. It worked on x86 tests, where that bash runs. Not affected: Guile/Shepherd (fork+exec), ucode's system() (execs /bin/sh itself).

Fixed in packages/cross-libc-fix.scm: loading it wraps cross-libc in (gnu packages cross-base) (looked up by name by standard-cross-packages), so every cross build uses a glibc whose system(), popen() and _PATH_BSHELL use /bin/sh -- always present on Guix System -- and whose ldd/sotruss/tzselect/xtrace start with #!/bin/sh. A target static-bash isn't possible: building it needs this very libc. Cost: the whole cross closure rebuilds once (~5 h here, 325 derivations). Every file that cross-builds packages for the Pi (rpi4.scm, packages/openwrt.scm) must (use-modules (cross-libc-fix)) first, or the store fills with both libcs. Verified on generation 14: netifd maps the fixed libc, awk's system()/getline pipes work.

patches/netifd-popen-bin-sh.patch (fork + /bin/sh -c) is now redundant but harmless; kept until netifd is next rebuilt anyway. Check a binary's libc with grep -ao '/gnu/store/[^/]*bash-static[^/]*/bin/sh' <libc.so.6> (no output = fixed). Unrelated leftover: cross-built man-db hardcodes the build host's glibc/bin/iconv, dragging x86 glibc + bash-static into the closure as dead weight.

36. netifd won't adopt a WiFi netdev from option device

device_create_default() refuses devices with /sys/class/net/X/phy80211 unless they're external: on OpenWrt the wireless scripts attach them. With wpa_supplicant outside netifd, wan gets no device in UCI and the netifd-attach-wlan0 one-shot (from netifd-configuration external-devices) calls ubus call network.interface.wan add_device '{"name":"wlan0","link-ext":true}' after wpa-supplicant starts (retries until netifd's object exists).

Also: OpenWrt's scripts are written for busybox ash (export -n, ...): shebangs are patched to busybox sh, not left to whatever /bin/sh is.

37. dnsmasq --bind-dynamic misses br-lan's IPv4 address at boot

dnsmasq starts before netifd has created br-lan ("interface br-lan does not currently exist"); netifd then adds 192.168.10.1 while dnsmasq is still starting up. With --bind-dynamic it binds per address on netlink events, and that IPv4 address was never bound -- not even when later IPv6 address events made it re-enumerate. Symptom: DHCP works (its own socket), DNS on 192.168.10.1 is "connection refused"; ss -lnu shows 127.0.0.1:53 and br-lan's IPv6 addresses only. ubus call network reload doesn't help; a lan down/up does. Seen on the first cold boot with odhcpd (more IPv6 address events), possibly latent before.

Fix: no --bind-dynamic. dnsmasq's default binds the wildcard address and drops queries not arriving on --interface=br-lan, so address timing doesn't matter. Nothing else uses port 53, and nftables drops WAN DNS anyway (checked: a query to the WAN address times out).

38. Small traps met wiring odhcpd

• busybox pkill -x dnsmasq matches nothing: it compares the whole /gnu/store/.../sbin/dnsmasq argv[0]. Use the Shepherd action (herd reload dnsmasq) or the pid file.
• with-environment-variables is unbound in activation snippets (the activation script's module set lacks it); the whole activation aborts there, after earlier snippets ran. Run commands via env VAR=... cmd. deploy-rpi.sh output shows the exception -- read it after each deploy.
• UCI seeds apply only to fresh systems: /etc/config/network on the running Pi was seeded long ago, so seed changes (here globals and ip6assign) must also be made live with uci set/uci commit.
• Workstation IPv6 was off: the NM profile pirouter-lan had ipv6.method ignore; now auto, never-default, ignore-auto-dns.

39. copy-build-system ignores --target: x86 store references

Guix 1.5.0's copy-build-system lowers every build with #:target #f, so in a cross-built system its inputs -- and any file-append in its phases -- are native x86_64 packages. openwrt-base-files (deployed since generation 13) had uci.sh calling an x86 uci and sourcing a jshn.sh that runs an x86 jshn: config_load, uci_* and json_* from it were broken on the Pi (netifd's dhcp path didn't hit them). firewall4 would have run an x86 nft. Also: under cross builds patch-source-shebangs points /bin/sh scripts at the native bash.

Fix: script packages that refer to other packages use gnu-build-system (cross-aware) running only copy-build-system's install phase -- cross-copy-phases / %cross-copy-modules in packages/openwrt.scm -- and set script shebangs to the target busybox explicitly.

Check a system before deploying: scripts/check-closure-arch.sh /gnu/store/...-system lists store items holding x86-64 ELF files. About 40 are expected and pre-date this (dead weight, never run): the cross gcc pulled in by gcc-cross-...-lib, the native tools the guix package and man-db (PITFALLS #35) reference. Investigate anything new in that list; trimming the old ones is a follow-up (closure is ~2.6 GB).

40. Publishing ubus service makes fw4 wait on it

fw4 asks ubus call service get_data '{"type":"firewall"}' on every start and reload (rules that procd services register). With no service object the call fails at once and fw4 carries on. Once procd-compat publishes service, fw4 waits for the answer -- and the first procd-compat ran herd reload firewall with a blocking system(), so it never answered: procd-compat, herd and fw4 all hung (no timeout).

Fix: procd-compat answers get_data (no data) and runs herd reload as a child process (uloop.process), never blocking its loop. Rule for anything that publishes a procd object: handle every method OpenWrt tools call on it, and don't block the loop while a reload you started may call back. When porting the next OpenWrt tool, grep it for "service"/"system" ubus calls.

41. ucode ubus objects reject undeclared arguments (LuCI's session id)

A method published with ucode's conn.publish() answers INVALID_ARGUMENT when a call carries an argument its args don't declare -- including a method with no args at all. Calls through uhttpd's JSON-RPC and rpcd always add ubus_rpc_session, so ubus call system board worked from the shell and failed from LuCI (result: [2]). Fix: every procd-compat method declares ubus_rpc_session. Test new ubus methods with '{"ubus_rpc_session":"x"}', not only with no message.

42. LuCI on Guix: FHS paths, and what the browser asks for

LuCI is written against OpenWrt's file system, in three places:

• Server side (ucode, rpcd's luci plugin, menu.d fs dependencies): the luci package rewrites /usr/share/..., /usr/sbin/... etc. to /run/current-system/profile/... at build time. A menu entry whose dependency doesn't exist answers 404 (admin/network/firewall needs fw3, so firewall4 now has the sbin/fw3 link OpenWrt ships).
• Browser side: views call rpcd file.exec/file.list/cgi-exec with OpenWrt paths (/sbin/ip, /www/..., /usr/libexec/syslog-wrapper), and the ACLs in share/rpcd/acl.d grant exactly those paths, so they can't be rewritten on one side only. luci-service-type creates them as symlinks into the profile (commands field, /www included). A view that execs the path file.stat returns gets the resolved store path, which no ACL names: syslog-wrapper is an exec script, not a link.
• uhttpd: -h DOCROOT goes through realpath() at startup, so it serves the profile generation that was current then -- after a deploy-rpi.sh --no-reboot, /etc/init.d/uhttpd restart (a reload only restarts it when its UCI-derived command line changed). Its CGI PATH was hardcoded to /sbin:/usr/sbin:/bin:/usr/bin (patched: profile first).

Also: busybox top output differs between OpenWrt's and Guix's builds (process list, patched), there is no logread (syslog-wrapper reads /var/log/messages), and rpcd's apply/rollback send config.change directly, bypassing reload_config's digests (procd-compat refreshes them). Check pages with a real browser: ubus calls succeeding is not enough.

43. Never make Shepherd wait on a process that talks to Shepherd

Generation 29 hung PID 1 at boot. dnsmasq's start ran the UCI renderer (then dnsmasq-uci, now procd-compat's openwrt-rc) and read its output with a blocking read-line; the renderer called herd status odhcpd. Shepherd 1.0 runs services in fibers on one thread, and a blocking read on an ordinary pipe stops all of them: no answer to herd, ever, and no services after dnsmasq (sshd, ...). Netifd, uhttpd and rpcd were up, so the Pi answered ping and HTTP but not SSH. Killing the renderer wasn't enough either: the orphaned herd had inherited the pipe's write end (the renderer's fd 3), so Shepherd never saw EOF.

It gets worse while PID 1 is blocked: Shepherd 1.0's syslog (system-log) runs inside PID 1, so anything calling logger blocks on a full /dev/log, and sshd is socket-activated by Shepherd, so new SSH connections hang at the banner. Recovery that worked without a console: log in to rpcd over uhttpd's JSON-RPC (session.login), list processes (luci getProcessList) and file exec /bin/kill -9 the renderer, its herd and logger children (all held the pipe), then deploy the fix. Queued herd reload calls from hotplug re-entered the same old code, so it took a few rounds.

Rules:

• start/actions only fork (make-forkexec-constructor, fork+exec-command); a wrapper process does slow work and then execs the daemon. No system*/open-pipe* waits on anything that may call herd, ubus services Shepherd starts, or the network.
• Nothing run by a Shepherd action or start calls herd.
• Hotplug scripts calling herd reload are fine only because of the above. (firewall-service-type's start still runs fw4 synchronously; fw4 talks to ubus/netifd/procd-compat, never to Shepherd -- keep it that way.)

44. Setting the clock from Guile: no settimeofday

The Pi has no RTC and booted at 1970 until chrony answered, so syslog, dnsmasq leases (expiry "43221") and anything time-based were wrong. sysfixtime-service-type (after OpenWrt's sysfixtime) sets the clock in activation, which runs at boot before Shepherd starts services, to the newest of a shutdown stamp (/var/lib/sysfixtime, written by its Shepherd stop), /var/log/messages and /etc/config. Traps on the way:

• Guix's Guile 3.0.9 has neither settimeofday nor clock-settime: the snippet died with "Unbound variable" -- it runs coreutils date -u -s @N.
• Activation output at boot only goes to the console, and activation order isn't the service-list order: the snippet logs to /var/log/sysfixtime.log.
• An unbalanced paren in a services module shows up in guix system build only as "unbound variable" for whatever the module exports; load it in guix repl -L packages to see the reader error. Result: the clock is off by the reboot time (~25 s) until chrony steps it.

45. Guix records: later fields see earlier ones by name

In (shepherd-action (name 'reload) (procedure #~(... #$name ...))), the name in procedure is the action's own name field, not a variable name from an enclosing let: define-record-type* lets field expressions refer to fields given before them. openwrt-rc got the symbol reload instead of the service name ("Unbound variable: reload" from herd). Don't name variables after fields of a record you build inside their scope.

46. nodogsplash: Shepherd's process attribution, idle services, iptables-nft

• Shepherd 1.0 attributes processes forked in a service's context to the service. When the firewall's start forked fw4 (to not block PID 1, #43), fw4's exit marked the service stopped and respawned it: the service's running value was #t, not a process, and Shepherd only ignores the exit of a process that isn't its running value. Fork such helpers with (parameterize ((current-service #f)) (fork+exec-command ...)). (openwrt-init-script reloads are fine: their running value is a process.)
• A UCI-disabled OpenWrt service must not exit. openwrt-rc exited 1 when start_service opened no instance (nodogsplash seeded enabled 0), and Shepherd disabled the service at its respawn limit, so enabling it in UCI later did nothing. Now openwrt-rc idles (procd keeps such services with zero instances) and a reload that finds an instance restarts it. The /etc/init.d shims' start/restart also herd enable first: OpenWrt's disable only affects boot.
• Guix's iptables-nft is built with copy-build-system: in a cross-built system its links point at an x86_64 iptables (#39, caught by check-closure-arch.sh). iptables-nft/cross in openwrt.scm.
• fw4 replaces only its inet fw4 table, so nodogsplash's iptables-nft tables (ip mangle/filter/nat) survive a firewall reload; its firewall include (restart.sh) is detached anyway, since it calls herd.
• OpenWrt's stock /etc/config/nodogsplash has enabled 1: installing it gates br-lan at once. The seed here is disabled.

47. OpenWrt's wireless stack on Guix

• netifd's main.uc loads wireless.uc from its own directory and handlers from wireless/ beside it (netifd.main_path), so wifi-scripts' lib/netifd files are copied into the netifd package; the ucode modules (wifi.*), hostap helpers and schemas live in the profile, with paths patched (wpad's HOSTAPD_UC_PATH points at the profile too).
• wifi.utils, iwinfo.uc and wifi config read /etc/board.json unconditionally. As OpenWrt: /etc/board-base.json (ours, static) + wifi-detect -> /tmp/sysinfo/board.json, /etc/board.json a link to it.
• wpad.init opens two procd instances (hostapd, wpa_supplicant): two plain Shepherd services instead of an openwrt-init-script.
• A program-file gexp using (guix build utils) needs with-imported-modules; without it the script dies at once, silently when forked outside a service (current-service #f). wifi-setup avoids it.
• brcmfmac has no ACS: set an explicit channel (36 on 5 GHz here). "Could not set MAC address" from hostapd is harmless (the BSSID ends up the locally administered address it asked for).

48. Packaging TollGate (Go) for a cross-built Guix system

• Go module dependencies come from a fixed-output derivation (computed-file with #:hash), the one kind of build with network access: go mod download with GOSUMDB off (the checksum database's tiles change over time), keeping only the module cache's download/ part. guix build --check confirms it is reproducible.
• Refer to that derivation, and inside it to Go and nss-certs, with #+: with #$, a cross build lowered Go for the target, which pulled a cross-built GCC 14 (an hour of compiling, then failing in fixincludes).
• Copied out of the store, the module cache is read-only, and Go opens its .lock files for writing: make the copy writable.
• Keep TollGate's #!/bin/sh scripts unpatched (patch-source-shebangs, patch-shebangs deleted): in a cross build they'd point at the build host's bash (#39).
• On the Pi: brcmfmac allows one AP interface, so TollGate's private SSID (br-private) doesn't come up beside the public one; the MT6000's radios can. fw4 logs "Include '/etc/config/firewall-tollgate' failed": TollGate's setup adds a UCI file as a (script) firewall include; the rule in it is also set directly (tollgate_in).

49. GL-MT6000 under GL's stock firmware: mainline DTS changes

The kernel dies or misses hardware with mainline's mt7986a.dtsi and OpenWrt's board DTS as they are; mt7986a-glinet-gl-mt6000-guix.dtsi (appended to the DTS) fixes each of these:

• secmon: GL's ATF (BL31) uses 256 KiB at 0x43000000, mainline reserves 192 KiB. The kernel hands out the rest as RAM and dies at the first allocation there -- before any console or network: a reset loop.
• wo-ilm/wo-dlm/wo-boot: mainline reserves WED's WO MCU SRAM (in MMIO space, 0x151xxxxx) as no-map reserved memory. Under GL's firmware the kernel resets within seconds with those nodes present (found by bisecting DTBs; OpenWrt maps them as syscons). Removing them means disabling WED too: its firmware loader writes wo-boot through a NULL mapping otherwise.
• eMMC partitions: OpenWrt's MT6000 DTS nests them in card@0/block/partitions; kernels since 6.13 read partitions (compatible "gpt-partitions") right under card@0, as OpenWrt's newer boards have it. Otherwise the factory partition has no NVMEM cells: WiFi runs on default calibration ("eeprom load fail") and mtk_eth_soc defers forever waiting for its MAC address -- no Ethernet, silently.
• ramoops at 0x42ff0000 (where OpenWrt has it) with a console record: the only way to see a failed boot's kernel log without a serial cable -- after a warm reset. A power cycle clears it.

50. GL-MT6000: mtk_eth_soc as a module needs clk_ignore_unused

Loaded from the root file system (~20 s), mtk_eth_soc's MDIO bus times out ("mdio: MDIO timeout", probe error -5): by then the kernel has turned off (at late_initcall) a clock the MDIO block needs and the DT doesn't list. OpenWrt builds the driver in, so it claims its clocks first. clk_ignore_unused on the command line fixes it; building it in would need BRIDGE, DSA and PHYLINK built in as well.

51. GL U-Boot's TFTP recovery, and what it does to the root partition

GL's U-Boot requests openwrt-gl-mt6000.bin from 192.168.1.2 (as 192.168.1.1) about 3 s after every power-on or reset, with a 1 s timeout. Served an OpenWrt sysupgrade tar (sysupgrade-glinet_gl-mt6000/{CONTROL, kernel,root}), it writes kernel to p6 and root to p7 and boots it; a tar without root, or a bare FIT, is fetched and ignored. It is the no-button recovery path: sudo dnsmasq --port=0 --enable-tftp ... on the workstation, then power-cycle (or let the board reset itself).

To boot OpenWrt from RAM yet keep the Guix root, use the OpenWrt initramfs FIT as kernel and the first 1 MiB of the Guix ext4 as root. That damages the file system in two ways. U-Boot writes 512 KiB of overlay marker right after root, over group 0's bitmaps and inode table. And the 1 MiB puts back the image's pre-boot superblock and group descriptors, although Guix has resized the file system since. To read it: copy p7 off, write blocks 256-383 back from the image, and use debugfs with the backup superblock (-c -s 32768 -b 4096).

52. guix system image changes the system's root device

The image's system has root=<partition UUID> where the configuration says (file-system-label "Guix"): it is a different store item from guix system build's. Boot the one inside the image (its /var/guix/profiles/system-1-link), or the initrd looks for a root that isn't the one it gets.

53. TollGate on the MT6000: restoring the OpenWrt wallet

TollGate keeps its state in /etc/tollgate: config.json, identities.json (the Nostr key: same identity), install.json, and wallet.db (BoltDB; the wallet directory is the config file's). The config versions written by OpenWrt's install matched v0.5.0's, so the files went back as they were. Order: deploy with --no-reboot (new Shepherd services don't start), copy the files in (mode 600), reboot -- otherwise TollGate's first start creates a new identity and wallet.

The restored config named the mint https://mint.minibits.cash, which answers 404 at /v1/info: TollGate then reports "no reachable mints" and never advertises. The mint is https://mint.minibits.cash/Bitcoin (v0.5.0's own default). Check tollgate status, the advertisement at http://<router>:2121/ (kind 10021 once ready) and "mint probe" lines in /var/log/messages.

54. TollGate payments fail: short v2 keyset IDs (gonuts-tollgate)

Every payment failed, on the Pi and the MT6000 alike: "PurchaseSession: Receive completed, amount=0, err=could not swap proofs: ... inputs[0].id: NUT02: ID length invalid, expected 8 bytes (short/v1) or 33 bytes (v2)". Mints such as minibits (cdk-mintd 0.17) sign with v2 keysets (33-byte IDs, prefix 01); a v4 (cashuB) token carries their short form, the first 8 bytes, and the receiving wallet must expand it from the mint's /v1/keysets (NUT-02). TollGate v0.5.0's Cashu library, gonuts-tollgate v0.7.1, swaps with the short ID. packages/patches/gonuts-tollgate-short- keyset-ids.patch expands it in Wallet.Receive and ReceiveHTLC; the package patches a local copy of the module and points the daemon's go.mod at it (go mod edit -replace), since a modified module zip would fail go.sum.

55. TollGate's private network: no DHCP, and admin isolation

TollGate's setup creates dhcp.private without option dhcpv4 'server'; OpenWrt's current dnsmasq init defaults that to disabled, so the private network (its WPA2 SSID) never handed out addresses -- "no address range available for DHCP request via br-private". files/uci-mt6000/99_zz-wired- private sets it. The same script makes the MT6000's wired ports private: TollGate's open SSID shares br-lan with the router's services, and nodogsplash's users_to_router (22, 23, 8080 by default) plus its authenticated-users rule (every port) open SSH and LuCI to anyone nearby or anyone who paid. br-lan keeps only the open SSID, an fw4 rule rejects 22, 23, 443 and 8080 from it, and lan1-lan5 join br-private (192.168.2.1).

56. Flow offloading on the MT6000: available, off

CONFIG_NFT_FLOW_OFFLOAD is built; fw4's flow_offloading works (flowtable on br-lan, br-private, eth1) but changed nothing measurable -- routing already runs at the 1 Gbit line rate; bidirectional CPU went from 19% to 15%. flow_offloading_hw falls back to software: fw4 puts every bridge member in one flowtable, and the WiFi APs can't do hardware offload without WED (eth1 and lan1-lan5 can). Both stay off because offloaded flows skip the forward chain, where nodogsplash enforces TollGate sessions: a customer's connections would outlive their paid time. Doing it right means offloading only private <-> WAN flows (fw4 picks devices per flowtable, not per zone).

57. Fresh installs: TollGate's setup ran before the WiFi config existed

uci-defaults run at activation, but /etc/config/wireless is created later, by wifi-setup (wifi config needs the radios, i.e. their drivers loaded). On a system that already had WiFi when TollGate was added this never showed; on a fresh image 99-tollgate-setup found no radios: no open TollGate SSID, the AP stayed guix-router (WPA2) -- and where it created /etc/config/wireless for its private SSID, wifi-setup skipped wifi config altogether. uci-service-type now has late-defaults: run by wifi-setup once the WiFi config exists (/etc/uci-late-defaults), followed by one reboot when any ran, so every service starts from the result, as on OpenWrt. TollGate's setup and the MT6000's 99_zz-wired-private are late defaults. Tested on the Pi by removing the WiFi config and the markers.

On a fresh install TollGate's setup also leaves uhttpd without HTTPS (it keeps listen_https only if a certificate exists yet); LuCI is on :8080.