---
name: iso-debug
description: Diagnose and fix Archipelago ISO boot failures. Covers hybrid MBR/GPT, UEFI/BIOS boot chains, live-boot initramfs, GRUB/ISOLINUX configuration, xorriso packaging, and USB boot compatibility. Use when ISO doesn't boot, installer doesn't start, kernel panics, or USB isn't recognized by BIOS/UEFI.
allowed-tools: Bash, Read, Grep, Glob, Agent, Edit
---

# ISO Boot Debugging — Archipelago Custom Base

Systematic diagnosis of ISO boot failures for the Archipelago debootstrap-based installer.

## Architecture

The ISO boot chain has 5 stages. Failure at any stage has distinct symptoms:

| Stage | Component | Symptom if broken |
|-------|-----------|-------------------|
| 1. BIOS/UEFI recognition | Hybrid MBR + GPT | USB not in boot menu at all |
| 2. Bootloader | ISOLINUX (BIOS) or GRUB EFI (UEFI) | Black screen after selecting USB |
| 3. Kernel + initramfs | vmlinuz + initrd.img with live-boot | Kernel panic or initramfs shell |
| 4. Root filesystem | live-boot mounts filesystem.squashfs | "No root device" or blank screen |
| 5. Installer | systemd service + auto-install.sh | Boots to shell but no installer prompt |

## Stage 1: USB Not Recognized

**Most common cause**: Wrong MBR code in the ISO hybrid boot sector.

### Diagnosis
```bash
# Compare first 16 bytes of working vs broken ISO
xxd -l 16 working.iso
xxd -l 16 broken.iso

# Check for valid boot signature at offset 510
xxd -s 510 -l 2 broken.iso
# Must show: 55aa
```

### Known MBR codes
- `4552` — Debian Live MBR (extracted from Debian Live ISO). **Works on all tested hardware.**
- `33ed` — ISOLINUX package generic isohdpfx.bin. **Does NOT work on some UEFI hardware.**

### Fix
The project ships the proven MBR at `image-recipe/branding/isohdpfx.bin` (432 bytes, starts with `4552`).
Build script uses it via: `-isohybrid-mbr "$SCRIPT_DIR/branding/isohdpfx.bin"`

### xorriso flags that matter
- `-isohybrid-mbr <file>` — Embeds MBR code for USB hybrid boot
- `-isohybrid-gpt-basdat` — Adds GPT partition entry for EFI (REQUIRED for UEFI USB boot)
- `-partition_offset 16` — Reserves space for GPT table (REQUIRED — without this some UEFI firmware won't see the USB)
- `-eltorito-alt-boot -e boot/grub/efi.img -no-emul-boot` — EFI boot catalog entry

### Balena Etcher
Writes raw ISO to USB — no special formatting. If the ISO boots in QEMU but not on hardware, the MBR code is the issue, not Etcher.

## Stage 2: Bootloader Failure

### BIOS path: ISOLINUX
Required files in ISO: `isolinux/isolinux.bin`, `isolinux/ldlinux.c32`, `isolinux/boot.cat`
Config: `isolinux/isolinux.cfg`

### UEFI path: GRUB
Required files: `EFI/BOOT/BOOTX64.EFI`, `boot/grub/efi.img`, `boot/grub/grub.cfg`
The EFI image is a FAT32 filesystem containing the GRUB binary, built with:
```bash
grub-mkimage -O x86_64-efi -o BOOTX64.EFI -p /boot/grub \
    part_gpt part_msdos fat iso9660 udf normal boot linux search \
    search_fs_uuid search_fs_file search_label configfile echo cat \
    ls test true loopback gfxterm gfxmenu font png all_video video \
    video_bochs video_cirrus efi_gop efi_uga
```
**Critical**: `all_video`, `efi_gop`, `efi_uga` needed for display on real hardware.

### Diagnosis
```bash
# Mount ISO and verify files
sudo mount -o loop,ro broken.iso /mnt
ls -la /mnt/isolinux/
ls -la /mnt/EFI/BOOT/
cat /mnt/boot/grub/grub.cfg
cat /mnt/isolinux/isolinux.cfg
sudo umount /mnt
```

## Stage 3: Kernel / Initramfs

### live-boot
The initramfs must contain live-boot hooks. Without them, the kernel boots but can't find root.

**Kernel params required**: `boot=live components`
- `boot=live` — triggers live-boot's initramfs scripts
- `components` — tells live-boot to scan live/ for squashfs files

### Verify initramfs has live-boot
```bash
TMPDIR=$(mktemp -d)
unmkinitramfs /path/to/initrd.img $TMPDIR
# live-boot installs scripts/live as a FILE (not directory)
ls -la $TMPDIR/scripts/live    # or $TMPDIR/main/scripts/live
file $TMPDIR/scripts/live      # Should say "ASCII text"
```

### Common initramfs failures
1. **live-boot not installed**: debootstrap `--include` can't resolve its deps. Must install via `chroot apt-get` after debootstrap.
2. **Broken initramfs from container build**: `update-initramfs` needs `/proc`, `/sys`, `/dev` mounted in the chroot.
3. **scripts/live is a FILE not directory**: Verification code must use `[ -e ]` not `[ -d ]`.

## Stage 4: Root Filesystem

live-boot searches for squashfs files in `live/` on the boot media.
- Mounts boot media (USB/CDROM) at `/run/live/medium`
- Finds `live/filesystem.squashfs`
- Mounts it read-only, creates tmpfs overlay
- pivot_root into the combined root

### Diagnosis
If you get an initramfs shell prompt `(initramfs)`:
```bash
# Inside initramfs shell:
ls /run/live/medium/          # Is boot media mounted?
ls /run/live/medium/live/     # Is squashfs there?
cat /proc/cmdline             # Does it have boot=live?
```

## Stage 5: Installer Not Starting

The installer auto-starts via:
1. Getty auto-login on tty1 (root, no password)
2. systemd service `archipelago-installer.service`
3. Wrapper script searches for boot media at: `/run/live/medium`, `/run/archiso`, `/cdrom`

### Diagnosis
If you get a shell but no installer prompt:
```bash
systemctl status archipelago-installer.service
cat /usr/local/bin/archipelago-start-installer
ls /run/live/medium/archipelago/auto-install.sh
```

## Quick Verification Checklist

Run against any ISO before flashing:
```bash
ISO=path/to/iso
MNT=$(mktemp -d)
sudo mount -o loop,ro $ISO $MNT

echo "=== MBR ===" && xxd -l 4 $ISO
echo "=== Boot sig ===" && xxd -s 510 -l 2 $ISO
echo "=== Files ===" && for f in live/vmlinuz live/initrd.img live/filesystem.squashfs isolinux/isolinux.bin EFI/BOOT/BOOTX64.EFI boot/grub/grub.cfg archipelago/auto-install.sh; do [ -e $MNT/$f ] && echo "OK: $f" || echo "MISSING: $f"; done
echo "=== Kernel params ===" && grep "boot=live" $MNT/boot/grub/grub.cfg && echo OK || echo MISSING
echo "=== live-boot ===" && INITRD=$(mktemp -d) && unmkinitramfs $MNT/live/initrd.img $INITRD 2>/dev/null && ([ -e $INITRD/scripts/live ] && echo "OK" || echo "MISSING")

sudo umount $MNT
```

## Key Files

| File | Purpose |
|------|---------|
| `image-recipe/build-auto-installer-iso.sh` | Main build script (~2600 lines) |
| `image-recipe/branding/isohdpfx.bin` | Proven MBR code (432 bytes) |
| `image-recipe/branding/grub-theme/` | GRUB theme (theme.txt + background.png) |
| `image-recipe/branding/plymouth-theme/` | Plymouth boot splash |
| `.gitea/workflows/build-iso-dev.yml` | CI workflow with smoke test |
| `image-recipe/test-iso-qemu.sh` | QEMU testing script |
| `image-recipe/dev-branding.sh` | Quick branding iteration (patch + repackage) |

## Infrastructure

| What | Where |
|------|-------|
| CI runner | gitea-runner.service on 192.168.1.228 |
| ISO builds | FileBrowser at http://192.168.1.228:8083 → Builds/ |
| Dev branch | dev-iso (separate CI: build-iso-dev.yml) |
| Main branch | main (CI: build-iso.yml) — DO NOT break |