# Pitfalls encountered (and fixes)

Numbered for cross-reference from the README.  All verified on this host with
guix 1.5.0 / Debian 13, 2026-10.

## 1. `guix pull` is broken upstream (stale guile-zlib hash)

Channel tip `ded4de6` pins `guile-zlib-0.2.2` to sha256 `04p9lb…mnx2`, but
Codeberg regenerated the `archive/v0.2.2.tar.gz` asset: the current archive
hashes to `1h9q7rw8…wlv3g` and the stable release asset to `011flqx…g2hz`.
Every `guix pull` of that tip fails with a download hash mismatch inside
`compute-guix-derivation`.

**Workaround:** don't pull — build with the 1.5.0 CLI's bundled channel
everything else keeps working (substitutes for 1.5.0 are on the CI).  Re-test
`guix pull` periodically; when it works, re-pin nonguix to match.

Diagnosis note: `guix hash` prints **nix-base32** by default (`-f base32` is
lowercase RFC-4648, and the daemon's "actual hash:" error line is a different
representation again) — don't chase ghosts comparing them.

## 2. Nonguix master requires guix newer than 1.5.0

`nongnu/packages/linux.scm` defines `linux-7.2` → `(corrupt-linux
linux-libre-7.2)`; loading aborts at `linux-libre-7.2: unbound variable` and
leaves a half-initialized module whose public interface is only
`(corrupt-linux)`.  Symptom in configs: `error: linux: unbound variable` with
no helpful backtrace.

**Fix:** pin nonguix to `f5338f6` (last commit before `linux-6.18`/`7.x` were
added; provides `linux` = 6.17).

## 3. Missing substitute signing keys → silent bootstrap from source

The binary tarball ships `/etc/guix/acl` without the CI keys.  Builds then
"work" but compile everything (you'll notice `glibc-bootstrap-0.drv` building
and `Python-3.5.9.tar.xz` unpacking).  Diagnostic:
`guix weather <pkg>` → "substitutes from 'https://ci.guix.gnu.org' are
unauthorized".  **Fix:** authorize the `.pub` files in
`$GUIX_PROFILE/share/guix/` (see HOST-SETUP §5).  aarch64 substitutes for
1.5.0 are 100% available on the CI.

## 4. `/tmp` is a 7.8 G tmpfs

Kernel build dies after ~50 min with
`fatal error: error writing to /tmp/guix-build-linux-…/cc*.s: No space left on device`.
**Fix:** `TMPDIR=/var/guix-tmp` in the daemon unit (world-writable, on the big
VG), `Environment=` line — not the shell.

## 5. `--system=aarch64-linux` ≠ cross build

On an x86_64 host, `--system` means *foreign*: the entire dependency chain is
built **for aarch64 and executed under qemu-binfmt** (check with
`ps aux | grep binfmt-P` — an aarch64 `cc1` under `qemu-aarch64` = foreign).
Foreign kernel: 3–6 h.  `--target=aarch64-linux-gnu` cross-compiles natively:
~50 min.  Always `--target` for images from this host.

## 6. Cross-built images install an x86 GRUB; bootloader chains are ignored

`make-grub-efi-installer` explicitly no-ops in disk-image mode, and
`efi-bootloader-chain` files/packages are only installed by
`guix system init` / the installer — **not** by `guix system image`.  The efi-raw
image ends up with host-arch `EFI/BOOT/BOOTX64.EFI` (verified: `file` says
x86-64) and none of the chain's `config.txt`/u-boot/DTB/firmware.

**Fix (this repo's design):** don't use GRUB at all.  `rpi4-image.scm`
declares the FAT boot partition (firmware, U-Boot, `config.txt`, DTB) and the
ext4 root directly, and `rpi4.scm` uses a `u-boot-bootloader` variant with no
raw-disk installer that only writes `extlinux.conf` (see README "Boot flow").
An earlier `prepare-rpi-image.sh` post-processed an efi-raw image instead; it
was removed once the declarative image booted on hardware.

## 7. `make-raspi-bcm28-dtbs` (gnu packages raspberry-pi, 1.5.0) is broken

Its `#:install-plan` gexp evaluates `(search-input-directory %build-inputs …)`
at compile time → `Apply to non-list: #<gexp …>`.  Define your own trivial
DTB package if needed.

## 8. copy-build-system install-plan gotcha

`("start4.elf" ".")` → `copy-file: Is a directory` (the destination is used as
the full path, not appended to).  The firmware package uses an explicit
trivial-build-system builder instead.

## 9. initrd module names must match .ko file names exactly

`sdhci_arasan` → not found; the module is **`sdhci-of-arasan`** (dashes!).
`mmc_block`, `sdhci_pltfm` do use underscores.  Symptom:
`kernel module not found "sdhci_arasan"` during `linux-modules` build.

## 10. trivial-build-system builders: PATH is empty

`tar xzf` needs `gzip` present, not just `tar`+`xz`
(`set-path-environment-variable "PATH" '("bin") (list tar xz gzip)`).

## 11. `guix build -f` quirks

- the file's **last expression** must be the package variable (hint mentions
  `my-package`);
- `GUIX_PACKAGE_PATH` is honored by `guix build`/`guix system` but **not** by
  `guix repl` (use `guix repl -L DIR`);
- flags are `-f/--format` lowercase (capital `-F` is a different option).

## 12. Image partition labels ≠ config labels

With the stock `efi-raw` image type, genimage names partitions `Guix_image` /
`GNU-ESP` regardless of the `file-system-label`s in the OS config, and Guix
mounts by label → boot fails to find root.  **Fix:** declare the partitions
yourself (`rpi4-image.scm` sets `(label "RPI-BOOT")` / `(label "Guix")`) so
they match `rpi4.scm`'s `file-systems`.

## 13. extlinux path prefix semantics (U-Boot)

U-Boot's syslinux loader resolves image paths relative to the directory
containing `extlinux.conf` (prefix).  Put the config at the **partition root**
`/extlinux/extlinux.conf` so the absolute-looking `/gnu/store/…` paths resolve
from the filesystem root.

## 14. 1.5.0 config API differences vs current guix

- `operating-system` has **no** `password`/`hashed-root-password` field and no
  `(plain "…")` — set root via
  `(user-account (name "root") (group "root") (system? #t) (password "<crypt-hash>"))`
  (generate with `openssl passwd -6`).
- `agetty-configuration` takes a single `tty` string (not `terminals` list).
- `usbutils`/`iw` live in `(gnu packages linux)`, `ethtool` in
  `(gnu packages networking)`, `json-c` in `(gnu packages web)`; there is no
  `(gnu packages json)`/`(gnu packages usb)`.
- `substitute-keyword-arguments` needs `(guix utils)`.
- `guix system image` takes no `-o` (prints the store path); positional args
  are `ACTION … FILE`.

## 15. Shell footguns during automation

- `pkill -f 'guix system image'` matches the invoking shell's own command
  line → kills your session; use `pgrep` + explicit PID.
- The bash-tool timeout SIGKILLs the process **group** — `nohup` alone does
  not survive; launch long builds with `setsid nohup … & disown`.

## 16. Nonguix `linux` can't mount a Pi 4 SD card (or see its Ethernet)

Its arm64 config has `CLK_BCM2835` **off**.  emmc2's clock is the cprman
(`clocks = <&clocks BCM2711_CLOCK_EMMC2>`), and so is the mini-UART's, so the
SD card and serial console never probe; the SD regulators also hang off the
firmware GPIO expander (`GPIO_RASPBERRYPI_EXP=m`).  `BCMGENET` is off too.
The SD driver is `sdhci-iproc`, not `sdhci-of-arasan` (that was a wrong guess
in the old initrd list).  **Fix:** `packages/rpi4-linux.scm` (customize-linux
on top of nonguix `linux`), SD path built in, initrd-modules `'()`.

`MMC_BLOCK=y` silently stays `m` unless `RPMB=y` (`depends on RPMB || !RPMB`);
`verify-config` catches it.  `DRM_SIMPLEDRM=y` is impossible with `DRM=m`.

## 17. Stock Pi U-Boot layout overlaps a distro kernel

rpi.env: `kernel_addr_r=0x80000`, `fdt_addr_r=0x02600000`,
`ramdisk_addr_r=0x02700000`.  booti relocates an `Image` with flags bit 3 to
the 2 MB boundary (0x200000); our `image_size` is 36.4 MB → ends 0x2660000,
on top of FDT and initrd.  **Fix:** `patches/rpi-env-layout.patch` (kernel at
0x200000, everything else ≥ 0x08000000).

## 18. Mainline Pi 4 serial console is `ttyS1`, not `ttyAMA0`

With the upstream DT the PL011 (`ttyAMA0`) is wired to Bluetooth; GPIO 14/15
is the mini-UART, alias `serial1` → `ttyS1` (verified in QEMU:
`fe215040.serial: ttyS1 at MMIO 0xfe215040`).

## 19. Root `user-account` needs `(uid 0)`

Without it Guix still prepends its own passwordless `%root-account`
("the following accounts appear more than once: root").

## 20. QEMU `raspi4b` (10.0) can't boot a mainline kernel on the stock DTB

Unemulated: AON block at 0x7ef00000 (L2 irqchip → external abort; DVP clock
gate → abort in `clk_disable_unused`), firmware GPIO expander (`-EIO`), so
emmc2 regulators never probe; mini-UART IRQ storm → RCU stall on
`/dev/console` open.  QEMU attaches the SD card to the *WiFi* SDHCI
(7e300000).  Hence U-Boot → kernel in QEMU is silent forever.
`run-qemu-rpi4.sh` boots the image's kernel/initrd directly with a patched
DTB (see its header); none of this applies to real hardware.

## 21. `pkill -f` / `grep pattern | kill` match your own shell (again)

Any filter over full command lines matches the shell running it, because
the pattern is in its argv.  Use `pgrep -x <comm>` (≤15 chars) or `$!`.

## 22. A plain symlink into the store is not a GC root

`ln -s /gnu/store/…-guix-rpi4 foo.img` doesn't stop `guix gc` from deleting
the image.  `build-rpi-image.sh` uses `guix system image -r <link>`, which
registers the link as an indirect root (`guix gc --list-roots` shows it).
Remove old images with `guix gc -d <path>` — it refuses live paths.  Avoid a
blanket `guix gc`: the 50-min `linux-rpi4` cross build isn't rooted unless an
image or system that uses it is.

## 23. dnsmasq (1.5.0) doesn't cross-build

Its `install-dbus` phase uses `%output`, which only exists in native builds:
`Unbound variable: %output`.  `rpi4.scm` defines `dnsmasq/cross-fixed`
(same phase via `outputs`) and passes it as the dnsmasq service's `package`.

## 24. linux-firmware subset filter dropped the Pi 4 WiFi firmware

brcmfmac `error -2` (ENOENT), no `wlan0`.  In WHENCE the 43455 `.bin` and
`.clm_blob` are `Link: brcm/… -> ../cypress/…`, and the Pi 4 NVRAM entry is
quoted (`File: "brcm/…raspberrypi,4-model-b.txt"`).  The old filter tested
link *targets* and unquoted names against `^brcm/`, so all three were
skipped.  Now: keep any entry whose own path matches `^"?(brcm|cypress|rtl_nic)/`.
Check with `ls $(guix build … %rpi-linux-firmware)/lib/firmware/brcm | grep 43455-sdio`.

## 25. sshd requires 'networking — don't let the WAN provide it

openssh-service-type (inetd-style under Shepherd) requires `networking`.
With dhcpcd on `wlan0` providing it and `wlan0` absent, Shepherd accepted
TCP on :22 but never spawned `sshd -i` → "timed out during banner exchange"
even from the LAN.  dhcpcd now provides `wan`; the LAN static-networking
provides `lan networking`.

## 26. Onboard Ethernet name depends on probe order

GENET and USB NICs (r8152…) are both modules; plugging in a USB NIC can make
it `eth0`.  Bridge ports are matched by MAC (`network-link (mac-address …)`).

## 27. 2.4 GHz WAN: associated, but DHCP never answered

On hakodosh 2.4 GHz ch 1 (UniFi AP, -43 dBm) the Pi associated and got
`wpa_state=COMPLETED`, but `iw dev wlan0 station dump` showed ~45% `tx
failed` at MCS0 (7.2 Mbit/s) while RX ran at 72 Mbit/s; broadcasts (DHCP
DISCOVER) are unacked, so they were silently lost → IPv4LL fallback.  An ARP
reply from the gateway proved nothing: UniFi APs can proxy-ARP.  No USB 3
device was attached, so not the usual Pi 4 USB3 2.4 GHz noise.  On 5 GHz
(ch 44, -49 dBm): 0 tx failed, 162 Mbit/s, lease at once.  Fix:
`freq_list=` of 5 GHz channels in `/boot/firmware/wpa_supplicant.conf`.
Diagnose with `station dump` (tx failed / tx bitrate) before blaming DHCP.
Likely cause, found later: the router this project is meant for (GL-MT6000,
on the bench and powered) was flooding the 2.4 GHz channels; switched off on
2026-10-14.  5 GHz stays pinned anyway.

## 28. Updating the Pi: no `guix system reconfigure`/`switch-generation` on it

On the Pi both would build natively for aarch64: reconfigure evaluates the
config there (native derivations ≠ our cross-built ones → kernel from
source), and switch-generation/roll-back build `extlinux.conf` with a native
Guile the Pi doesn't have (its Guix is cross-built), and need substitutes
over TLS with a 1970 clock.  They also look the bootloader up by name, and
`u-boot-rpi4` isn't in any module.  `deploy-rpi.sh` does it from the
workstation instead: cross-build, `guix copy`, generation link, host-built
extlinux.conf (`scripts/extlinux-conf.scm`), activate, reboot.

Gotchas found on the way:
- `activate` picks the system from `GUIX_NEW_SYSTEM`; run bare, it
  re-activates `/run/current-system` (the *old* system).
- `read-boot-parameters` returns the stored kernel args only; `root=`,
  `gnu.system=`, `gnu.load=` come from (private) `bootable-kernel-arguments`.
- `guix archive --authorize` on the Pi turns `/etc/guix/acl` into a regular
  file, which the next activation moves to `acl.bak` and replaces with the
  declared ACL — declare the workstation key in `guix-service-type`.
- The image build swaps root `(file-system-label "Guix")` for the image's
  UUID, so the first deploy's system differs from the image's even with no
  config change (harmless; same partition).

## 29. Time sync on a Pi without RTC: chrony, not ntpd/openntpd

The clock starts at 1970 every boot and the WAN arrives ~20 s after the
time service starts.  `ntp-service-type` doesn't cross-build (`ntp` pulls
`perl-net-ssleay`, which links the build host's x86 OpenSSL).  OpenNTPD
only steps the clock in a short window at startup, then slews — hopeless for
a 56-year offset.  chronyd with `makestep 1 -1` steps whenever it's > 1 s
off, any time.  1.5.0 has no chrony service: `rpi4.scm` runs `chronyd -n`
from a shepherd service, and needs a `chrony` system account (built-in
privilege drop: "Could not get user/group ID of chrony").

## 30. 90% packet loss on the LAN cable: Energy-Efficient Ethernet

Pi 4 onboard port (BCM54213PE/GENET) ↔ workstation Intel e1000e: with EEE
active, the Pi received ~83 of thousands of frames, with **no** error
counters; ping RTTs of ~1 s, ARP failures, SSH "timed out during banner
exchange", no DHCP.  It looked like a hung Shepherd (sshd-1 disabled) or
breakage from the clock step — it wasn't.  Intermittent: some boots were
fine.  `ethtool --set-eee eth0 eee off` → 0% loss.  `rpi4.scm` runs that
from a one-shot service after 'lan.  Two bugs on the way: br-lan inherits
the port's MAC and sorts before eth0 (skip interfaces without
`/sys/class/net/*/device`), and service `PATH` isn't reliable (pass
`sleep` by store path).  Diagnose with `ethtool -S eth0` (rx_packets vs.
what the peer sent) and `ethtool --show-eee eth0`.  SSH on the WAN
(192.168.88.222) was the way in while the cable was down.

## 31. Image root is only 6 GB: grow it on first boot

`rpi4-image.scm` makes a 6 GiB root so the image stays small; on a 119 GB
card the rest was unused (3.2 GB free after a few deploys).  The
`grow-root` one-shot service (rpi4.scm) runs every boot: if root is ext4,
the last partition of an MBR disk, and ≥ 1 MiB is free after it, it does
what worked by hand on the live Pi —
`echo ,+ | sfdisk --no-reread --no-tell-kernel -N 2 /dev/mmcblk1`,
`resizepart /dev/mmcblk1 2 <sectors>` (tells the kernel about a partition on
a busy disk; plain sfdisk can't), `resize2fs` online (the image's ext4 has
`resize_inode`).  Messages go to the kernel log as `grow-root: …`.
Tested in QEMU (`run-qemu-rpi4.sh` pads the SD to 8 GiB: 6144 → 7932 MiB)
and as a no-op on the already-grown Pi.

## 32. OpenWrt packages: CMake + pkg-config when cross-building

OpenWrt's CMakeLists use `PKG_SEARCH_MODULE`; in Guix cross builds only
`aarch64-linux-gnu-pkg-config` exists → "Could NOT find PkgConfig".
`-DPKG_CONFIG_EXECUTABLE=<bare name>` is resolved relative to the build
dir; `which` in `#:configure-flags` runs before PATH is set.  What works:
a phase before 'configure that sets `PKG_CONFIG` to `(which
(pkg-config-for-target))`.  Trap: build that gexp in a *procedure* called
from `arguments` — a top-level gexp evaluates `pkg-config-for-target` at
module load (no target → native name), `which` returns #f, and
`(setenv "PKG_CONFIG" #f)` silently *unsets* it.  Also: `jshn.sh` calls
`jshn` from PATH — patched to the store path.  ubusd only allows root
unless ACL files say otherwise (`-A dir`).

## 33. OpenWrt package notes: pins, ucode module path, jsonfilter

- Pin to the commits OpenWrt main builds (`package/*/*/Makefile`,
  `PKG_SOURCE_VERSION`; git.openwrt.org's gitweb redirects, the GitHub
  mirror `raw.githubusercontent.com/openwrt/openwrt/main/...` works), not
  each project's master, so netifd/firewall4/LuCI get the versions they're
  tested with.  ucode lives at github.com/jow-/ucode, not git.openwrt.org.
- ucode's module search path is compiled in (`LIB_SEARCH_PATH`), with no
  environment override; scripts can't pass `-L`.  Built with its own prefix
  **plus** `/run/current-system/profile/{lib/ucode/*.so,share/ucode/*.uc}`
  so firewall4/LuCI modules installed in the system profile are found.
- jsonfilter's CMake builds/runs `contrib/lemon` relative to the source
  tree: `#:out-of-source? #f`.  Installed as `jsonpath`; symlinked to
  `jsonfilter`, the name OpenWrt scripts use.
- Testing a ubus object: don't `call` it from the same uloop that serves
  it (the reply can never be dispatched → null); call from another process.

## 34. netifd on Guix

- Needs **libudebug** (OpenWrt `project/udebug`), not listed in our plan;
  `main.c` uses it unconditionally.
- Its runtime files (`main.uc`, `proto*.uc`, `utils.uc`, ifup/ifdown,
  `proto/dhcp.sh`) live in OpenWrt's *main tree*
  (`package/network/config/netifd/files`), not in netifd.git:
  `%openwrt-tree` in `openwrt.scm` is that tree at the pinned commit,
  trimmed by a snippet to the directories we use.
- `-p` sets the script dir for shell protocols only; `ucode.c` loads
  `DEFAULT_MAIN_PATH "/main.uc"` (compile time).  Patch `DEFAULT_MAIN_PATH`
  in `netifd.h` to `$out/lib/netifd` instead of symlinking `/lib/netifd`;
  `netifd-proto.sh` also hardcodes `/lib/netifd` and
  `/usr/share/libubox/jshn.sh` (patched).
- In-source build (`make_ethtool_modes_h.sh` runs relative to the tree).
- Bridge ports are by *name* only (no MAC match): `eth0`, see #26.
- Test before touching the Pi: `sudo unshare -n` + a dummy port + ubusd +
  `netifd -s SOCK -c CONFDIR -h HOOK -r RESOLV` shows the bridge, the
  `network.interface.*` ubus objects and the hotplug event.
- Deploy LAN changes over the WAN path (`PI=pirouter-wan`), never both.

## 35. Cross-built glibc: popen()/system() exec an x86-64 bash (fixed)

On the Pi, every `popen()`/`system()` from glibc fails silently: the
cross-built glibc (`glibc-cross-aarch64-linux-gnu`) has `_PATH_BSHELL`
patched to the *build host's* `bash-static` (`od -j18 -N2` → `3e 00` =
x86-64; running it: "Exec format error").  Guix 1.5.0 `cross-base.scm`
marks this `;FIXME: static-bash`.  Symptom here: netifd registered only the
built-in `static` protocol — its shell-handler discovery uses `popen` — so
`wan` stayed `proto none`/`NO_DEVICE`.  It worked on x86 tests, where that
bash runs.  Not affected: Guile/Shepherd (fork+exec), ucode's `system()`
(execs `/bin/sh` itself).

**Fixed** in `packages/cross-libc-fix.scm`: loading it wraps `cross-libc`
in `(gnu packages cross-base)` (looked up by name by
`standard-cross-packages`), so every cross build uses a glibc whose
`system()`, `popen()` and `_PATH_BSHELL` use `/bin/sh` -- always present on
Guix System -- and whose ldd/sotruss/tzselect/xtrace start with
`#!/bin/sh`.  A target `static-bash` isn't possible: building it needs this
very libc.  Cost: the whole cross closure rebuilds once (~5 h here, 325
derivations).  Every file that cross-builds packages for the Pi
(`rpi4.scm`, `packages/openwrt.scm`) must `(use-modules (cross-libc-fix))`
first, or the store fills with both libcs.  Verified on generation 14:
netifd maps the fixed libc, awk's `system()`/`getline` pipes work.

`patches/netifd-popen-bin-sh.patch` (fork + `/bin/sh -c`) is now redundant
but harmless; kept until netifd is next rebuilt anyway.  Check a binary's
libc with `grep -ao '/gnu/store/[^/]*bash-static[^/]*/bin/sh' <libc.so.6>`
(no output = fixed).  Unrelated leftover: cross-built man-db hardcodes the
build host's `glibc/bin/iconv`, dragging x86 glibc + bash-static into the
closure as dead weight.

## 36. netifd won't adopt a WiFi netdev from `option device`

`device_create_default()` refuses devices with `/sys/class/net/X/phy80211`
unless they're *external*: on OpenWrt the wireless scripts attach them.
With wpa_supplicant outside netifd, `wan` gets no `device` in UCI and the
`netifd-attach-wlan0` one-shot (from `netifd-configuration`
`external-devices`) calls
`ubus call network.interface.wan add_device '{"name":"wlan0","link-ext":true}'`
after `wpa-supplicant` starts (retries until netifd's object exists).

Also: OpenWrt's scripts are written for busybox ash (`export -n`, ...):
shebangs are patched to busybox `sh`, not left to whatever `/bin/sh` is.

## 37. dnsmasq `--bind-dynamic` misses br-lan's IPv4 address at boot

dnsmasq starts before netifd has created `br-lan` ("interface br-lan does
not currently exist"); netifd then adds 192.168.10.1 while dnsmasq is still
starting up.  With `--bind-dynamic` it binds per address on netlink events,
and that IPv4 address was never bound -- not even when later IPv6 address
events made it re-enumerate.  Symptom: DHCP works (its own socket), DNS on
192.168.10.1 is "connection refused"; `ss -lnu` shows `127.0.0.1:53` and
br-lan's IPv6 addresses only.  `ubus call network reload` doesn't help; a
lan down/up does.  Seen on the first cold boot with odhcpd (more IPv6
address events), possibly latent before.

Fix: no `--bind-dynamic`.  dnsmasq's default binds the wildcard address and
drops queries not arriving on `--interface=br-lan`, so address timing
doesn't matter.  Nothing else uses port 53, and nftables drops WAN DNS
anyway (checked: a query to the WAN address times out).

## 38. Small traps met wiring odhcpd

- **busybox `pkill -x dnsmasq` matches nothing**: it compares the whole
  `/gnu/store/.../sbin/dnsmasq` argv[0].  Use the Shepherd action
  (`herd reload dnsmasq`) or the pid file.
- **`with-environment-variables` is unbound in activation snippets** (the
  activation script's module set lacks it); the whole activation aborts
  there, after earlier snippets ran.  Run commands via `env VAR=... cmd`.
  `deploy-rpi.sh` output shows the exception -- read it after each deploy.
- **UCI seeds apply only to fresh systems**: `/etc/config/network` on the
  running Pi was seeded long ago, so seed changes (here `globals` and
  `ip6assign`) must also be made live with `uci set`/`uci commit`.
- **Workstation IPv6 was off**: the NM profile `pirouter-lan` had
  `ipv6.method ignore`; now `auto`, `never-default`, `ignore-auto-dns`.

## 39. `copy-build-system` ignores `--target`: x86 store references

Guix 1.5.0's copy-build-system lowers every build with `#:target #f`, so in
a cross-built system its inputs -- and any `file-append` in its phases --
are *native* x86_64 packages.  `openwrt-base-files` (deployed since
generation 13) had `uci.sh` calling an x86 `uci` and sourcing a `jshn.sh`
that runs an x86 `jshn`: `config_load`, `uci_*` and `json_*` from it were
broken on the Pi (netifd's dhcp path didn't hit them).  firewall4 would have
run an x86 `nft`.  Also: under cross builds `patch-source-shebangs` points
`/bin/sh` scripts at the *native* bash.

Fix: script packages that refer to other packages use gnu-build-system
(cross-aware) running only copy-build-system's install phase --
`cross-copy-phases` / `%cross-copy-modules` in `packages/openwrt.scm` -- and
set script shebangs to the target busybox explicitly.

Check a system before deploying: `scripts/check-closure-arch.sh
/gnu/store/...-system` lists store items holding x86-64 ELF files.  About
40 are expected and pre-date this (dead weight, never run): the cross gcc
pulled in by `gcc-cross-...-lib`, the native tools the `guix` package and
man-db (PITFALLS #35) reference.  Investigate anything *new* in that list;
trimming the old ones is a follow-up (closure is ~2.6 GB).


## 40. Publishing ubus `service` makes fw4 wait on it

fw4 asks `ubus call service get_data '{"type":"firewall"}'` on every start
and reload (rules that procd services register).  With no `service` object
the call fails at once and fw4 carries on.  Once procd-compat publishes
`service`, fw4 waits for the answer -- and the first procd-compat ran `herd
reload firewall` with a blocking `system()`, so it never answered:
procd-compat, herd and fw4 all hung (no timeout).

Fix: procd-compat answers `get_data` (no data) and runs `herd reload` as a
child process (`uloop.process`), never blocking its loop.  Rule for anything
that publishes a procd object: handle every method OpenWrt tools call on it,
and don't block the loop while a reload you started may call back.  When
porting the next OpenWrt tool, grep it for `"service"`/`"system"` ubus calls.

## 41. ucode ubus objects reject undeclared arguments (LuCI's session id)

A method published with ucode's `conn.publish()` answers INVALID_ARGUMENT
when a call carries an argument its `args` don't declare -- including a
method with no `args` at all.  Calls through uhttpd's JSON-RPC and rpcd
always add `ubus_rpc_session`, so `ubus call system board` worked from the
shell and failed from LuCI (`result: [2]`).  Fix: every procd-compat method
declares `ubus_rpc_session`.  Test new ubus methods with
`'{"ubus_rpc_session":"x"}'`, not only with no message.

## 42. LuCI on Guix: FHS paths, and what the browser asks for

LuCI is written against OpenWrt's file system, in three places:

- **Server side** (ucode, rpcd's `luci` plugin, menu.d `fs` dependencies):
  the `luci` package rewrites `/usr/share/...`, `/usr/sbin/...` etc. to
  `/run/current-system/profile/...` at build time.  A menu entry whose
  dependency doesn't exist answers 404 (`admin/network/firewall` needs
  `fw3`, so firewall4 now has the `sbin/fw3` link OpenWrt ships).
- **Browser side**: views call rpcd `file.exec`/`file.list`/`cgi-exec` with
  OpenWrt paths (`/sbin/ip`, `/www/...`, `/usr/libexec/syslog-wrapper`),
  and the ACLs in `share/rpcd/acl.d` grant exactly those paths, so they
  can't be rewritten on one side only.  `luci-service-type` creates them
  as symlinks into the profile (`commands` field, `/www` included).  A
  view that execs the path `file.stat` returns gets the *resolved* store
  path, which no ACL names: syslog-wrapper is an exec script, not a link.
- **uhttpd**: `-h DOCROOT` goes through `realpath()` at startup, so it
  serves the profile generation that was current then -- after a
  `deploy-rpi.sh --no-reboot`, `/etc/init.d/uhttpd restart` (a reload only
  restarts it when its UCI-derived command line changed).  Its CGI PATH was
  hardcoded to `/sbin:/usr/sbin:/bin:/usr/bin` (patched: profile first).

Also: busybox `top` output differs between OpenWrt's and Guix's builds
(process list, patched), there is no `logread` (syslog-wrapper reads
`/var/log/messages`), and rpcd's apply/rollback send `config.change`
directly, bypassing `reload_config`'s digests (procd-compat refreshes them).
Check pages with a real browser: `ubus` calls succeeding is not enough.

## 43. Never make Shepherd wait on a process that talks to Shepherd

Generation 29 hung PID 1 at boot.  dnsmasq's `start` ran the UCI renderer
(then `dnsmasq-uci`, now procd-compat's `openwrt-rc`) and read its output with a blocking `read-line`; the
renderer called `herd status odhcpd`.  Shepherd 1.0 runs services in fibers
on one thread, and a blocking read on an ordinary pipe stops all of them: no
answer to herd, ever, and no services after dnsmasq (sshd, ...).  Netifd,
uhttpd and rpcd were up, so the Pi answered ping and HTTP but not SSH.
Killing the renderer wasn't enough either: the orphaned `herd` had inherited
the pipe's write end (the renderer's fd 3), so Shepherd never saw EOF.

It gets worse while PID 1 is blocked: Shepherd 1.0's syslog (`system-log`)
runs *inside* PID 1, so anything calling `logger` blocks on a full
`/dev/log`, and sshd is socket-activated by Shepherd, so new SSH
connections hang at the banner.  Recovery that worked without a console:
log in to rpcd over uhttpd's JSON-RPC (`session.login`), list processes
(`luci getProcessList`) and `file exec /bin/kill -9` the renderer, its
`herd` and `logger` children (all held the pipe), then deploy the fix.
Queued `herd reload` calls from hotplug re-entered the same old code, so
it took a few rounds.

Rules:
- `start`/actions only fork (`make-forkexec-constructor`,
  `fork+exec-command`); a wrapper process does slow work and then execs the
  daemon.  No `system*`/`open-pipe*` waits on anything that may call herd,
  ubus services Shepherd starts, or the network.
- Nothing run *by* a Shepherd action or start calls `herd`.
- Hotplug scripts calling `herd reload` are fine only because of the above.
(`firewall-service-type`'s start still runs `fw4` synchronously; fw4 talks
to ubus/netifd/procd-compat, never to Shepherd -- keep it that way.)

## 44. Setting the clock from Guile: no `settimeofday`

The Pi has no RTC and booted at 1970 until chrony answered, so syslog,
dnsmasq leases (expiry "43221") and anything time-based were wrong.
`sysfixtime-service-type` (after OpenWrt's sysfixtime) sets the clock in
activation, which runs at boot before Shepherd starts services, to the
newest of a shutdown stamp (`/var/lib/sysfixtime`, written by its Shepherd
stop), `/var/log/messages` and `/etc/config`.  Traps on the way:
- Guix's Guile 3.0.9 has neither `settimeofday` nor `clock-settime`: the
  snippet died with "Unbound variable" -- it runs coreutils `date -u -s @N`.
- Activation output at boot only goes to the console, and activation order
  isn't the service-list order: the snippet logs to
  `/var/log/sysfixtime.log`.
- An unbalanced paren in a services module shows up in `guix system build`
  only as "unbound variable" for whatever the module exports; load it in
  `guix repl -L packages` to see the reader error.
Result: the clock is off by the reboot time (~25 s) until chrony steps it.

## 45. Guix records: later fields see earlier ones by name

In `(shepherd-action (name 'reload) (procedure #~(... #$name ...)))`, the
`name` in `procedure` is the action's own `name` field, not a variable
`name` from an enclosing `let`: `define-record-type*` lets field
expressions refer to fields given before them.  openwrt-rc got the symbol
`reload` instead of the service name ("Unbound variable: reload" from herd).
Don't name variables after fields of a record you build inside their scope.

## 46. nodogsplash: Shepherd's process attribution, idle services, iptables-nft

- **Shepherd 1.0 attributes processes forked in a service's context to the
  service.**  When the firewall's `start` forked fw4 (to not block PID 1,
  #43), fw4's exit marked the service stopped and respawned it: the service's
  running value was `#t`, not a process, and Shepherd only ignores the exit of
  a process that isn't its running value.  Fork such helpers with
  `(parameterize ((current-service #f)) (fork+exec-command ...))`.
  (openwrt-init-script reloads are fine: their running value is a process.)
- **A UCI-disabled OpenWrt service must not exit.**  openwrt-rc exited 1 when
  `start_service` opened no instance (nodogsplash seeded `enabled 0`), and
  Shepherd disabled the service at its respawn limit, so enabling it in UCI
  later did nothing.  Now openwrt-rc idles (procd keeps such services with
  zero instances) and a reload that finds an instance restarts it.  The
  `/etc/init.d` shims' start/restart also `herd enable` first: OpenWrt's
  disable only affects boot.
- **Guix's `iptables-nft` is built with copy-build-system**: in a cross-built
  system its links point at an x86_64 iptables (#39, caught by
  `check-closure-arch.sh`).  `iptables-nft/cross` in openwrt.scm.
- fw4 replaces only its `inet fw4` table, so nodogsplash's iptables-nft tables
  (`ip mangle/filter/nat`) survive a firewall reload; its firewall include
  (`restart.sh`) is detached anyway, since it calls herd.
- OpenWrt's stock `/etc/config/nodogsplash` has `enabled 1`: installing it
  gates br-lan at once.  The seed here is disabled.

## 47. OpenWrt's wireless stack on Guix

- netifd's `main.uc` loads `wireless.uc` from its own directory and handlers
  from `wireless/` beside it (`netifd.main_path`), so wifi-scripts'
  `lib/netifd` files are copied into the netifd package; the ucode modules
  (`wifi.*`), hostap helpers and schemas live in the profile, with paths
  patched (wpad's `HOSTAPD_UC_PATH` points at the profile too).
- `wifi.utils`, `iwinfo.uc` and `wifi config` read `/etc/board.json`
  unconditionally.  As OpenWrt: `/etc/board-base.json` (ours, static) +
  wifi-detect -> `/tmp/sysinfo/board.json`, `/etc/board.json` a link to it.
- wpad.init opens two procd instances (hostapd, wpa_supplicant): two plain
  Shepherd services instead of an openwrt-init-script.
- A `program-file` gexp using `(guix build utils)` needs
  `with-imported-modules`; without it the script dies at once, silently when
  forked outside a service (`current-service #f`).  wifi-setup avoids it.
- brcmfmac has no ACS: set an explicit channel (36 on 5 GHz here).
  "Could not set MAC address" from hostapd is harmless (the BSSID ends up
  the locally administered address it asked for).

## 48. Packaging TollGate (Go) for a cross-built Guix system

- Go module dependencies come from a fixed-output derivation
  (`computed-file` with `#:hash`), the one kind of build with network
  access: `go mod download` with GOSUMDB off (the checksum database's tiles
  change over time), keeping only the module cache's `download/` part.
  `guix build --check` confirms it is reproducible.
- Refer to that derivation, and inside it to Go and nss-certs, with `#+`:
  with `#$`, a cross build lowered Go for the target, which pulled a
  cross-built GCC 14 (an hour of compiling, then failing in fixincludes).
- Copied out of the store, the module cache is read-only, and Go opens its
  `.lock` files for writing: make the copy writable.
- Keep TollGate's `#!/bin/sh` scripts unpatched (`patch-source-shebangs`,
  `patch-shebangs` deleted): in a cross build they'd point at the build
  host's bash (#39).
- On the Pi: brcmfmac allows one AP interface, so TollGate's private SSID
  (br-private) doesn't come up beside the public one; the MT6000's radios
  can.  fw4 logs "Include '/etc/config/firewall-tollgate' failed": TollGate's
  setup adds a UCI file as a (script) firewall include; the rule in it is
  also set directly (`tollgate_in`).

## 49. GL-MT6000 under GL's stock firmware: mainline DTS changes

The kernel dies or misses hardware with mainline's `mt7986a.dtsi` and
OpenWrt's board DTS as they are; `mt7986a-glinet-gl-mt6000-guix.dtsi`
(appended to the DTS) fixes each of these:

- **secmon:** GL's ATF (BL31) uses 256 KiB at 0x43000000, mainline reserves
  192 KiB.  The kernel hands out the rest as RAM and dies at the first
  allocation there -- before any console or network: a reset loop.
- **wo-ilm/wo-dlm/wo-boot:** mainline reserves WED's WO MCU SRAM (in MMIO
  space, 0x151xxxxx) as no-map reserved memory.  Under GL's firmware the
  kernel resets within seconds with those nodes present (found by
  bisecting DTBs; OpenWrt maps them as syscons).  Removing them means
  disabling WED too: its firmware loader writes `wo-boot` through a NULL
  mapping otherwise.
- **eMMC partitions:** OpenWrt's MT6000 DTS nests them in
  `card@0/block/partitions`; kernels since 6.13 read `partitions`
  (compatible "gpt-partitions") right under `card@0`, as OpenWrt's newer
  boards have it.  Otherwise the `factory` partition has no NVMEM cells:
  WiFi runs on default calibration ("eeprom load fail") and mtk_eth_soc
  defers forever waiting for its MAC address -- no Ethernet, silently.
- **ramoops** at 0x42ff0000 (where OpenWrt has it) with a console record:
  the only way to see a failed boot's kernel log without a serial cable --
  after a warm reset.  A power cycle clears it.

## 50. GL-MT6000: mtk_eth_soc as a module needs clk_ignore_unused

Loaded from the root file system (~20 s), mtk_eth_soc's MDIO bus times
out ("mdio: MDIO timeout", probe error -5): by then the kernel has turned
off (at late_initcall) a clock the MDIO block needs and the DT doesn't
list.  OpenWrt builds the driver in, so it claims its clocks first.
`clk_ignore_unused` on the command line fixes it; building it in would need
BRIDGE, DSA and PHYLINK built in as well.

## 51. GL U-Boot's TFTP recovery, and what it does to the root partition

GL's U-Boot requests `openwrt-gl-mt6000.bin` from 192.168.1.2 (as
192.168.1.1) about 3 s after every power-on or reset, with a 1 s timeout.
Served an OpenWrt sysupgrade tar (`sysupgrade-glinet_gl-mt6000/{CONTROL,
kernel,root}`), it writes `kernel` to p6 and `root` to p7 and boots it; a
tar without `root`, or a bare FIT, is fetched and ignored.  It is the
no-button recovery path: `sudo dnsmasq --port=0 --enable-tftp ...` on the
workstation, then power-cycle (or let the board reset itself).

To boot OpenWrt from RAM yet keep the Guix root, use the OpenWrt
initramfs FIT as `kernel` and the first 1 MiB of the Guix ext4 as `root`.
That damages the file system in two ways.  U-Boot writes 512 KiB of
overlay marker right after `root`, over group 0's bitmaps and inode
table.  And the 1 MiB puts back the image's pre-boot superblock and group
descriptors, although Guix has resized the file system since.  To read it:
copy p7 off, write blocks 256-383 back from the image, and use debugfs
with the backup superblock (`-c -s 32768 -b 4096`).

## 52. `guix system image` changes the system's root device

The image's system has `root=<partition UUID>` where the configuration
says `(file-system-label "Guix")`: it is a different store item from
`guix system build`'s.  Boot the one inside the image (its
`/var/guix/profiles/system-1-link`), or the initrd looks for a root that
isn't the one it gets.

## 53. TollGate on the MT6000: restoring the OpenWrt wallet

TollGate keeps its state in `/etc/tollgate`: `config.json`,
`identities.json` (the Nostr key: same identity), `install.json`, and
`wallet.db` (BoltDB; the wallet directory is the config file's).  The
config versions written by OpenWrt's install matched v0.5.0's, so the
files went back as they were.  Order: deploy with `--no-reboot` (new
Shepherd services don't start), copy the files in (mode 600), reboot --
otherwise TollGate's first start creates a new identity and wallet.

The restored config named the mint `https://mint.minibits.cash`, which
answers 404 at `/v1/info`: TollGate then reports "no reachable mints" and
never advertises.  The mint is `https://mint.minibits.cash/Bitcoin`
(v0.5.0's own default).  Check `tollgate status`, the advertisement at
`http://<router>:2121/` (kind 10021 once ready) and "mint probe" lines in
/var/log/messages.

## 54. TollGate payments fail: short v2 keyset IDs (gonuts-tollgate)

Every payment failed, on the Pi and the MT6000 alike: "PurchaseSession:
Receive completed, amount=0, err=could not swap proofs: ... inputs[0].id:
NUT02: ID length invalid, expected 8 bytes (short/v1) or 33 bytes (v2)".
Mints such as minibits (cdk-mintd 0.17) sign with v2 keysets (33-byte
IDs, prefix 01); a v4 (cashuB) token carries their short form, the first
8 bytes, and the receiving wallet must expand it from the mint's
/v1/keysets (NUT-02).  TollGate v0.5.0's Cashu library, gonuts-tollgate
v0.7.1, swaps with the short ID.  packages/patches/gonuts-tollgate-short-
keyset-ids.patch expands it in Wallet.Receive and ReceiveHTLC; the package
patches a local copy of the module and points the daemon's go.mod at it
(`go mod edit -replace`), since a modified module zip would fail go.sum.

## 55. TollGate's private network: no DHCP, and admin isolation

TollGate's setup creates `dhcp.private` without `option dhcpv4 'server'`;
OpenWrt's current dnsmasq init defaults that to disabled, so the private
network (its WPA2 SSID) never handed out addresses -- "no address range
available for DHCP request via br-private".  files/uci-mt6000/99_zz-wired-
private sets it.  The same script makes the MT6000's wired ports private:
TollGate's open SSID shares br-lan with the router's services, and
nodogsplash's users_to_router (22, 23, 8080 by default) plus its
authenticated-users rule (every port) open SSH and LuCI to anyone nearby or
anyone who paid.  br-lan keeps only the open SSID, an fw4 rule rejects 22,
23, 443 and 8080 from it, and lan1-lan5 join br-private (192.168.2.1).

## 56. Flow offloading on the MT6000: available, off

`CONFIG_NFT_FLOW_OFFLOAD` is built; fw4's `flow_offloading` works
(flowtable on br-lan, br-private, eth1) but changed nothing measurable --
routing already runs at the 1 Gbit line rate; bidirectional CPU went from
19% to 15%.  `flow_offloading_hw` falls back to software: fw4 puts every
bridge member in one flowtable, and the WiFi APs can't do hardware offload
without WED (eth1 and lan1-lan5 can).  Both stay off because offloaded
flows skip the forward chain, where nodogsplash enforces TollGate sessions:
a customer's connections would outlive their paid time.  Doing it right
means offloading only private <-> WAN flows (fw4 picks devices per
flowtable, not per zone).

## 57. Fresh installs: TollGate's setup ran before the WiFi config existed

uci-defaults run at activation, but /etc/config/wireless is created later,
by wifi-setup (`wifi config` needs the radios, i.e. their drivers loaded).
On a system that already had WiFi when TollGate was added this never
showed; on a fresh image 99-tollgate-setup found no radios: no open
TollGate SSID, the AP stayed `guix-router` (WPA2) -- and where it created
/etc/config/wireless for its private SSID, wifi-setup skipped `wifi config`
altogether.  uci-service-type now has `late-defaults`: run by wifi-setup
once the WiFi config exists (/etc/uci-late-defaults), followed by one
reboot when any ran, so every service starts from the result, as on
OpenWrt.  TollGate's setup and the MT6000's 99_zz-wired-private are late
defaults.  Tested on the Pi by removing the WiFi config and the markers.

On a fresh install TollGate's setup also leaves uhttpd without HTTPS (it
keeps `listen_https` only if a certificate exists yet); LuCI is on :8080.