archy/.claude/skills/iso-debug/references/boot-chain-reference.md

# Custom Debian ISO Boot Chain — Technical Reference

Expert reference for building and debugging custom bootable Debian-based ISOs.
Covers hybrid MBR/GPT, live-boot, debootstrap, GRUB, ISOLINUX, Plymouth, and xorriso.

---

## 1. Hybrid MBR/GPT for USB Boot

### What is isohdpfx.bin?
The first 432 bytes of a hybrid-bootable ISO. Contains the Master Boot Record code that BIOS firmware executes when booting from USB. Different sources produce different MBR code:

| Source | First bytes | Compatibility |
|--------|-------------|---------------|
| Debian Live ISO (`dd if=debian-live.iso bs=1 count=432`) | `45 52` | Best — works on all tested hardware |
| `/usr/lib/ISOLINUX/isohdpfx.bin` | `33 ed` | Generic — fails on some UEFI hardware |
| Manually built with `isohybrid` | Varies | Unpredictable |

**Rule**: Always extract MBR from a known-working ISO. Never rely on the generic ISOLINUX one.

### CRITICAL: Embedded vs Appended EFI — Real Hardware Impact

Two approaches for EFI boot in xorriso. They produce DIFFERENT hybrid structures:

| Approach | xorriso flag | cyl-align | CHS geometry | Real hardware |
|----------|-------------|-----------|--------------|---------------|
| **Embedded** | `-e boot/grub/efi.img` | `cyl-align-on` | Non-zero (e.g. 244/32) | **WORKS** |
| **Appended** | `-append_partition 2 ... -e --interval:appended_partition_2:all::` | `cyl-align-off` | `0/0` | **FAILS** |

The Will Haley guide recommends appended, but on our Dell hardware only embedded works.
Use `xorriso -indev image.iso -report_system_area plain` to check which mode an ISO uses.

### Common gotcha: installer minbase missing sudo
debootstrap --variant=minbase does NOT include sudo. If the installer runs as root
(via auto-login), do NOT use sudo in scripts. `bash: sudo: command not found` is the symptom.

### xorriso flags for hybrid boot
```bash
xorriso -as mkisofs -o output.iso \
    -isohybrid-mbr isohdpfx.bin \      # Embeds MBR for BIOS USB boot
    -c isolinux/boot.cat \              # El Torito boot catalog
    -b isolinux/isolinux.bin \          # BIOS bootloader
    -no-emul-boot -boot-load-size 4 -boot-info-table \
    -eltorito-alt-boot \                # Second boot entry (EFI)
    -e boot/grub/efi.img \             # EFI boot image
    -no-emul-boot \
    -isohybrid-gpt-basdat \            # Adds GPT partition for EFI
    -partition_offset 16 \              # Space for GPT table — REQUIRED for UEFI
    /path/to/iso/contents
```

**Critical flags**:
- `-isohybrid-gpt-basdat`: Without this, UEFI firmware won't see the EFI partition
- `-partition_offset 16`: Reserves 16 sectors for GPT. Without it, some UEFI firmware ignores the USB entirely
- `-isohybrid-mbr`: Without this, the ISO won't boot from USB at all (only CD-ROM)

### Balena Etcher
Writes the ISO byte-for-byte to USB — no reformatting, no special partition creation. If the ISO works with `dd`, it works with Etcher. If BIOS doesn't see the USB, the MBR code is wrong, not Etcher.

### Verifying hybrid structure
```bash
xxd -l 4 image.iso          # MBR code (should be 45 52 for Debian Live)
xxd -s 510 -l 2 image.iso   # Boot signature (must be 55 aa)
xxd -s 512 -l 8 image.iso   # GPT signature at LBA 1 (should be "EFI PART")
file image.iso               # Should say "DOS/MBR boot sector" and "bootable"
```

---

## 2. live-boot Package

### What it does
Provides initramfs hooks that mount a squashfs file as the root filesystem using overlayfs. This is how every Debian/Ubuntu live ISO works.

Boot flow: kernel → initramfs → live-boot scripts → find squashfs → mount overlayfs → pivot_root → systemd

### Package structure
- `live-boot` (~29KB): Main package, boot scripts
- `live-boot-initramfs-tools` (~6KB): Initramfs hooks that get baked into initrd.img

**Critical**: `scripts/live` is a **FILE**, not a directory. Verification must use `[ -e ]` not `[ -d ]`.

### Kernel parameters
| Parameter | Required | Effect |
|-----------|----------|--------|
| `boot=live` | YES | Activates live-boot's initramfs hooks |
| `components` | YES | Scans live/ for additional squashfs modules |
| `toram` | No | Copies squashfs to RAM (faster, allows USB removal) |
| `persistence` | No | Enables writable overlay on a partition labeled "persistence" |
| `quiet` | No | Suppresses boot messages |
| `splash` | No | Enables Plymouth splash screen |
| `console=ttyS0,115200` | No | Serial console for QEMU debugging |

### Where live-boot mounts things
- `/run/live/medium` — The boot media (USB/CDROM) mount point
- `/run/live/rootfs/filesystem.squashfs` — The mounted squashfs
- `/run/live/overlay` — The tmpfs overlay for writes

### Verifying live-boot in initramfs
```bash
TMPDIR=$(mktemp -d)
unmkinitramfs /path/to/initrd.img $TMPDIR
# Check for live-boot scripts
file $TMPDIR/scripts/live         # Should be "ASCII text"
# OR (some initramfs have main/ prefix)
file $TMPDIR/main/scripts/live
```

### Common failures
1. **live-boot not in initrd**: Installed in rootfs but initramfs not regenerated after
2. **Missing kernel params**: `boot=live` not in GRUB/ISOLINUX config
3. **Broken initramfs**: Built without /proc /sys /dev mounted in chroot
4. **Wrong verification**: `[ -d scripts/live ]` fails because it's a file

---

## 3. debootstrap for Installer Environments

### Variants
- `--variant=minbase`: Absolute minimum (~150MB). Only essential + apt. Good for installer squashfs.
- Default (no variant): Full base system (~300MB). More packages, fewer missing deps.

### --include limitations
debootstrap's minbase resolver is simplified and **cannot resolve complex dependency chains**. Packages like `live-boot` that depend on `initramfs-tools` which depends on many other packages will silently fail or be skipped.

**Fix**: Install complex packages via `chroot apt-get` after debootstrap completes:
```bash
debootstrap --variant=minbase --include=basic,packages bookworm /installer http://deb.debian.org/debian
# Then:
mount --bind /proc /installer/proc
mount --bind /sys /installer/sys
mount --bind /dev /installer/dev
chroot /installer apt-get update
chroot /installer apt-get install -y live-boot live-boot-initramfs-tools
umount /installer/dev /installer/sys /installer/proc
```

### Initramfs generation inside containers
`update-initramfs` REQUIRES `/proc`, `/sys`, `/dev` to be mounted in the chroot. Without them:
- Module detection fails (can't read /proc/modules)
- Device nodes missing (can't detect hardware)
- The resulting initramfs boots but can't load kernel modules

### Container-in-container considerations
When running debootstrap inside a Podman/Docker container on a CI runner:
- `--privileged` flag needed for chroot to work
- The container runtime may kill the container after debootstrap exits if using `set -e`
- proc/sys/dev mounts inside the debootstrapped chroot work fine with `--privileged`

---

## 4. GRUB Theming

### theme.txt format
```
desktop-color: "#0a0a0a"           # Fallback background color
desktop-image: "background.png"    # Background image (any PNG, GRUB scales)
title-text: ""                     # Empty = hide title

+ boot_menu {
    left = 25%
    top = 40%
    width = 50%
    height = 30%
    item_color = "#aaaaaa"          # Normal menu item color
    selected_item_color = "#fb923c" # Selected item color
    item_height = 36
    item_spacing = 8
    scrollbar = false
}

+ label {
    left = 25%
    top = 20%
    width = 50%
    text = "Some Text"
    color = "#f7931a"
    align = "center"
}
```

**IMPORTANT**: Do NOT specify `font = "Name Size"` in theme elements unless you know the exact internal font name. If GRUB can't find the font, the ENTIRE theme fails to load and you get the ugly default.

### Font handling
```bash
# Generate .pf2 font file
grub-mkfont -s 16 -o dejavu_16.pf2 /usr/share/fonts/truetype/dejavu/DejaVuSans.ttf

# In grub.cfg, load fonts BEFORE setting theme:
loadfont /boot/grub/font.pf2
loadfont /boot/grub/themes/archipelago/dejavu_16.pf2
set theme=/boot/grub/themes/archipelago/theme.txt
```

### Background images
- Any PNG works, GRUB scales to screen resolution
- Smaller images (1024x768) load faster
- Large images (3000x2000+) add seconds to boot and may fail on limited GRUB heap

### grub-mkimage — essential modules for ISO boot
```bash
grub-mkimage -O x86_64-efi -o BOOTX64.EFI -p /boot/grub \
    part_gpt part_msdos fat iso9660 udf \     # Filesystem access
    normal boot linux search search_fs_uuid search_fs_file search_label \
    configfile echo cat ls test true \          # Basic commands
    loopback \                                  # Loop device support
    gfxterm gfxmenu font png \                  # Graphical display
    all_video video video_bochs video_cirrus \  # Video drivers
    efi_gop efi_uga                             # EFI display protocols
```

Missing `all_video`/`efi_gop` = black screen on real hardware (works in QEMU).

### EFI boot image creation
```bash
dd if=/dev/zero of=efi.img bs=1M count=4
mkfs.vfat efi.img
mmd -i efi.img ::/EFI ::/EFI/BOOT
mcopy -i efi.img BOOTX64.EFI ::/EFI/BOOT/BOOTX64.EFI
```

---

## 5. Plymouth Boot Splash

### Theme types
- **script**: Most flexible. Lua-like scripting with sprites, animations, callbacks.
- **two-step**: Simple logo + spinner. Less customizable but easier.
- **fade-in**: Logo fades in. Minimal.

### Script theme structure
```
/usr/share/plymouth/themes/mytheme/
  mytheme.plymouth     # Theme metadata
  mytheme.script       # Animation script
  logo.png             # Logo image (PNG with alpha)
```

### mytheme.plymouth
```ini
[Plymouth Theme]
Name=MyTheme
Description=Custom boot splash
ModuleName=script

[script]
ImageDir=/usr/share/plymouth/themes/mytheme
ScriptFile=/usr/share/plymouth/themes/mytheme/mytheme.script
```

### Script language key functions
```javascript
Window.SetBackgroundTopColor(r, g, b);    // 0.0-1.0 floats
Window.SetBackgroundBottomColor(r, g, b);
image = Image("logo.png");
sprite = Sprite(image);
sprite.SetX(x); sprite.SetY(y); sprite.SetOpacity(0.0-1.0);
Plymouth.SetRefreshFunction(fn);           // Called every frame
Plymouth.SetBootProgressFunction(fn);      // fn(duration, progress)
Plymouth.SetDisplayPasswordFunction(fn);   // fn(prompt, bullets)
Plymouth.SetQuitFunction(fn);
screen_w = Window.GetWidth();
screen_h = Window.GetHeight();
```

### Setting default theme
```bash
plymouth-set-default-theme mytheme
# OR manually:
ln -sf /usr/share/plymouth/themes/mytheme/mytheme.plymouth /etc/alternatives/default.plymouth
```

### Kernel params
- `splash` in GRUB_CMDLINE_LINUX_DEFAULT enables Plymouth
- `quiet` suppresses text that would overlay Plymouth

---

## 6. ISOLINUX/SYSLINUX

### Required files
| File | Source | Purpose |
|------|--------|---------|
| `isolinux.bin` | `/usr/lib/ISOLINUX/isolinux.bin` | BIOS bootloader |
| `ldlinux.c32` | `/usr/lib/syslinux/modules/bios/ldlinux.c32` | Core library (REQUIRED) |
| `menu.c32` | `/usr/lib/syslinux/modules/bios/menu.c32` | Text menu UI |
| `libutil.c32` | `/usr/lib/syslinux/modules/bios/libutil.c32` | Utility library |
| `boot.cat` | Auto-generated by xorriso | El Torito boot catalog |
| `isohdpfx.bin` | Extracted from working ISO | Hybrid MBR code |

### Configuration (isolinux.cfg)
```
UI menu.c32
PROMPT 0
TIMEOUT 50           # 5 seconds (units of 1/10 second)
DEFAULT install

MENU TITLE MY INSTALLER
MENU COLOR border  30;44  #40ffffff #00000000 std
MENU COLOR title   1;36;44 #ff00b7ff #00000000 std
MENU COLOR sel     7;37;40 #ffffffff #ff333333 std
MENU COLOR unsel   37;44  #ffaaaaaa #00000000 std

LABEL install
    MENU LABEL Install System
    KERNEL /live/vmlinuz
    APPEND initrd=/live/initrd.img boot=live components quiet
    MENU DEFAULT
```

### menu.c32 vs vesamenu.c32
- `menu.c32`: Text-mode menu. More compatible, no background image.
- `vesamenu.c32`: VESA graphical menu. Supports background PNG, but some hardware/VMs don't support VESA.

---

## 7. Testing Without Real Hardware

### QEMU UEFI boot
```bash
qemu-system-x86_64 \
    -machine q35 \
    -drive if=pflash,format=raw,readonly=on,file=/path/to/OVMF_CODE.fd \
    -m 4G -smp 2 \
    -boot d -cdrom image.iso \
    -drive if=virtio,format=qcow2,file=test-disk.qcow2 \
    -vga virtio -display default
```

### QEMU BIOS boot (sees ISOLINUX)
```bash
qemu-system-x86_64 \
    -machine pc \
    -m 4G -smp 2 \
    -boot d -cdrom image.iso \
    -drive if=virtio,format=qcow2,file=test-disk.qcow2 \
    -vga virtio -display default
```

### Serial console capture
Add to QEMU: `-serial file:/tmp/serial.log`
Add to kernel params: `console=ttyS0,115200 console=tty0`

### ISO structure verification (no boot required)
```bash
MNT=$(mktemp -d)
sudo mount -o loop,ro image.iso $MNT

# Check all critical files
for f in live/vmlinuz live/initrd.img live/filesystem.squashfs \
         isolinux/isolinux.bin EFI/BOOT/BOOTX64.EFI boot/grub/grub.cfg; do
    [ -e $MNT/$f ] && echo "OK: $f" || echo "MISSING: $f"
done

# Check initramfs for live-boot
INITRD=$(mktemp -d)
unmkinitramfs $MNT/live/initrd.img $INITRD
[ -e $INITRD/scripts/live ] && echo "live-boot: OK" || echo "live-boot: MISSING"

# Check kernel params
grep "boot=live" $MNT/boot/grub/grub.cfg && echo "params: OK"

sudo umount $MNT
```

---

## 8. Security Considerations for Custom ISOs

### Supply chain
- Pin the Debian mirror URL (don't use redirectors in production)
- Verify package signatures (debootstrap does this by default)
- Pin kernel and GRUB package versions for reproducibility

### Installer security
- Auto-install.sh runs as root — validate all inputs before path construction
- LUKS key generation must use CSPRNG (`/dev/urandom`, never `/dev/random` which blocks)
- Drop the LUKS key file after writing to crypttab (or store in root-only location with 0400)

### Boot security
- Secure Boot requires signed GRUB EFI binary (shim-signed package)
- Without Secure Boot, the unsigned BOOTX64.EFI works but users must disable Secure Boot in BIOS
- The MBR code (isohdpfx.bin) is not signed — Secure Boot only validates EFI path