---
description: Development workflow and deployment practices for Archipelago
alwaysApply: true
---

# Archipelago Development Workflow

## Priority: Deploy-Test-Fix Loop

**This is the primary workflow. Follow it for every change.**

1. **Make the change** the user requests
2. **SSH and build to live server** - Run `./scripts/deploy-to-target.sh --live` once done
3. **Test that it works** - Verify apps launch: iframe for most apps, new tab for BTCPay/Home Assistant

## App Launcher (iframe + new tab fallback)

Most apps launch in the iframe overlay. BTCPay (port 23000) and Home Assistant (port 8123) set `X-Frame-Options` and don't support subpath proxying—they open in a new tab instead.
4. **If broken, fix and repeat** - Debug, fix, redeploy, and test again until complete
5. **End loop** only when everything works

Do not leave deployment or testing to the user. The agent has SSH access to perform all building and work on the live server.

## Deployment Strategy

**Always deploy to live system for testing** - The target device (192.168.1.228) is a development machine, so deploy changes directly to the live system rather than using dev servers.

**When making changes, always run deploy** - After editing code (frontend, backend, scripts, or configs), run `./scripts/deploy-to-target.sh --live` to sync, build, and deploy. Do not leave the user to deploy manually.

### Backend: build on server via rsync (never on macOS)
- **Always** deploy backend by: (1) rsync `core/` to `archipelago@192.168.1.228:~/archy/core/`, then (2) SSH and run `cargo build --release` on the server, then copy binary to `/usr/local/bin/` and restart `archipelago.service`.
- Use `sshpass -p 'EwPDR8q45l0Upx@'` for non-interactive rsync/SSH. The password is stored in `scripts/deploy-config.sh` (gitignored) and sourced by the deploy script automatically.
- **Do not** build the Rust binary on macOS and copy it (causes Exec format error on Linux).

### Standard Deployment Command

```bash
./scripts/deploy-to-target.sh --live
```

This command:
1. Syncs code from local Mac to remote target
2. Builds frontend (Vue.js) and backend (Rust)
3. Deploys to live paths:
   - Frontend: `/opt/archipelago/web-ui/`
   - Backend: `/usr/local/bin/archipelago`
4. Restarts services (systemd + nginx)

### Deploy to Both Servers

```bash
./scripts/deploy-to-target.sh --both
```

Deploys to 192.168.1.228 first (builds there), then copies binary and web-ui to 192.168.1.198 (which has no rsync/cargo).

### Target Environment

- **Host**: archipelago@192.168.1.228 (primary), archipelago@192.168.1.198 (secondary)
- **OS**: Debian-based server
- **Container Runtime**: Podman (root context for system services)
- **Web Server**: Nginx
- **Backend**: Systemd service (`archipelago.service`) running as root

## SSH Access

**Current credentials**: `archipelago@192.168.1.228` with password `EwPDR8q45l0Upx@`

The deploy script sources `scripts/deploy-config.sh` (gitignored) which sets `ARCHIPELAGO_PASSWORD`. For manual SSH/rsync commands, use:
```bash
sshpass -p 'EwPDR8q45l0Upx@' ssh -o StrictHostKeyChecking=no archipelago@192.168.1.228
```

If `sshpass` hangs, SSH may be rate-limited from too many connections. Wait 10-15 seconds and retry.

## Development Paths

### Local (Mac)
- Project root: `/Users/dorian/Projects/archy`
- Frontend: `neode-ui/`
- Backend: `core/`
- Scripts: `scripts/`
- ISO Build: `image-recipe/`

### Remote (Target)
- Dev directory: `~/archy/`
- Live frontend: `/opt/archipelago/web-ui/`
- Live backend: `/usr/local/bin/archipelago`
- Data: `/var/lib/archipelago/`
- Systemd service: `/etc/systemd/system/archipelago.service`
- Nginx config: `/etc/nginx/sites-available/archipelago`

## App Icons

**Single source of truth**: `neode-ui/public/assets/img/app-icons/`

- All app icons live here. Do not duplicate icons elsewhere.
- Naming: `{app-id}.{png|webp|svg}` (e.g. `fedimint.png`, `mempool.webp`)
- References use `/assets/img/app-icons/{filename}`. Build outputs copy from this folder.
- See `neode-ui/public/assets/img/app-icons/README.md` for details.

## App Integration Standards

**When adding or fixing apps, always verify end-to-end:**

1. **Test the app UI on its port** - After getting an app working, confirm the web UI loads at its configured port (e.g. `http://192.168.1.228:4080` for Mempool).
2. **Auto-connect dependencies** - Apps must connect to their dependencies on installation:
   - **Bitcoin node**: LND, Fedimint, BTCPay Server, Mempool all need Bitcoin RPC (host.containers.internal:8332 or bitcoin-knots container).
   - **LND**: BTCPay Server and other Lightning apps need LND connection.
3. **Works out of the box** - After autoinstaller flash, apps should work without manual configuration. Ensure `get_app_config()` in `core/archipelago/src/api/rpc.rs` has correct env vars for each app.

## Testing Workflow

1. Make changes locally
2. Deploy with `--live` flag
3. Test at http://192.168.1.228
4. **Verify each modified app**: Open its UI URL and confirm it loads and connects to dependencies
5. **Test with Cursor browser MCP** (when available): After app installs or fixes, use the browser MCP to open the app URL, check for console errors (502, WebSocket failures, etc.), debug, fix, redeploy, and repeat until working.
6. Check logs if needed:
   - Backend: `ssh archipelago@192.168.1.228 'sudo journalctl -u archipelago -f'`
   - Nginx: `ssh archipelago@192.168.1.228 'sudo tail -f /var/log/nginx/error.log'`
5. **Sync changes back to ISO build** (see below)

## Running Containers

Check container status:
```bash
ssh archipelago@192.168.1.228 'sudo podman ps'
```

Common containers:
- Home Assistant (port 8123)
- Bitcoin Knots (ports 8332, 8333)
- LND (ports 9735, 10009)

## ISO Build Debug Workflow (Flash-and-Debug)

**Primary way to improve ISO builds.** After flashing a new machine from the ISO, SSH in and diagnose. Fix issues in the build, rebuild ISO, reflash, repeat.

### Debug a Fresh ISO Install

1. **Flash** the ISO to a test machine (e.g. 192.168.1.198)
2. **SSH** after first boot (default ISO password is `archipelago`, dev server uses `EwPDR8q45l0Upx@`):
   ```bash
   ssh-keygen -R 192.168.1.198   # if host key changed after reflash
   sshpass -p "archipelago" ssh -o StrictHostKeyChecking=no archipelago@192.168.1.198
   ```
3. **Run diagnostics** to find issues:
   ```bash
   # Services
   systemctl is-active archipelago nginx
   # Containers
   sudo podman ps -a
   # Tor hostname (backend needs this for peer discovery)
   sudo cat /var/lib/archipelago/tor/hidden_service_archipelago/hostname
   sudo -u archipelago cat /var/lib/archipelago/tor/hidden_service_archipelago/hostname 2>&1  # should NOT be "Permission denied"
   # Backend logs
   sudo journalctl -u archipelago -n 50
   # Nginx errors
   sudo tail -20 /var/log/nginx/error.log
   # RPC reachable?
   curl -s -X POST http://127.0.0.1:5678/rpc/v1 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1,"method":"echo","params":{}}'
   ```
4. **Fix** issues in `image-recipe/build-auto-installer-iso.sh`, scripts, or configs
5. **Rebuild** ISO, **reflash**, **re-diagnose** until clean

### Common ISO Issues to Check

| Issue | Check | Fix |
|-------|-------|-----|
| Tor hostname unreadable | `sudo -u archipelago cat .../hostname` | setup-tor.sh must chmod 711 on tor dir + hidden_service_* dirs, 644 on hostname files |
| Node not discoverable | Tor hostname + Nostr publish | Fix Tor perms so node_address is set |
| RPC timeouts | nginx error.log | Increase proxy timeouts or optimize slow RPCs |
| Missing containers | `sudo podman ps -a` | ISO is minimal; apps install from marketplace |
| bitcoin-ui 404 | Port 8334 not listening | Add bitcoin-ui to first-boot or document |

## ISO Build Integration

**CRITICAL**: After testing on the live server, always update the ISO build to include your changes.

### Building the ISO

**Recommended**: Build on the target server (has all dependencies):

```bash
# SSH to target server
ssh archipelago@192.168.1.228

# Navigate to project
cd ~/archy/image-recipe

# Run build with sudo (auto-installs missing deps like xorriso)
sudo ./build-auto-installer-iso.sh

# The ISO will be at: results/archipelago-auto-installer-*.iso

# Copy back to Mac
# On your Mac:
scp archipelago@192.168.1.228:~/archy/image-recipe/results/archipelago-auto-installer-*.iso .
```

**Alternative**: Build from Mac (requires Docker Desktop installed).

### Common ISO Build Issues

- **Missing xorriso**: Run with `sudo` to auto-install, or: `sudo apt install -y xorriso`
- **Missing podman**: Run with `sudo` to auto-install, or: `sudo apt install -y podman`
- **No Docker on Mac**: Either install Docker Desktop or build on target server (recommended)

### System Configuration Files to Sync

When you make system-level changes on the live server, capture them for the ISO build:

1. **Systemd Service** (`/etc/systemd/system/archipelago.service`)
   - Location in repo: `image-recipe/configs/archipelago.service`
   - Capture command: `ssh archipelago@192.168.1.228 'sudo cat /etc/systemd/system/archipelago.service' > image-recipe/configs/archipelago.service`

2. **Nginx Configuration** (`/etc/nginx/sites-available/archipelago`)
   - Location in repo: `image-recipe/configs/nginx-archipelago.conf`
   - Capture command: `ssh archipelago@192.168.1.228 'sudo cat /etc/nginx/sites-available/archipelago' > image-recipe/configs/nginx-archipelago.conf`

3. **Other System Files**
   - Logrotate: `image-recipe/configs/logrotate.conf`
   - Any new scripts in `/opt/archipelago/scripts/`

### Build Process Checklist

Before building a new ISO, ensure:

- [ ] Latest backend built: `cd image-recipe && ./scripts/build-backend.sh`
- [ ] Latest frontend built: `cd image-recipe && ./scripts/build-frontend.sh`
- [ ] System configs synced from live server
- [ ] Integration script updated: `./integrate-archipelago.sh`
- [ ] ISO built: `./build-debian-iso.sh`
- [ ] ISO tested in QEMU: `./test-iso-qemu.sh`

### Key Configuration Values

**Backend Service (archipelago.service)**:
- **User**: `root` (required to access root Podman containers)
- **Environment**: 
  - `ARCHIPELAGO_BIND=0.0.0.0:5678`
  - `ARCHIPELAGO_DEV_MODE=true` (for container auto-detection)

**Nginx Configuration**:
- Serves frontend from `/opt/archipelago/web-ui`
- Proxies `/rpc/` to backend at `127.0.0.1:5678`
- Proxies `/ws` for WebSocket connections

### Deployment Paths in ISO

The ISO build must install files to:
- `/usr/local/bin/archipelago` - Backend binary
- `/opt/archipelago/web-ui/` - Frontend files
- `/etc/systemd/system/archipelago.service` - Service definition
- `/etc/nginx/sites-available/archipelago` - Nginx config
- `/opt/archipelago/` - Base directory for scripts and data

## Common Issues

### Container Detection
- Containers must be in **root Podman context** (started with `sudo podman`)
- Backend must run as **root** to see root containers
- Check: `sudo podman ps` (should show containers)
- Check: `podman ps` (should be empty if using root containers)

### Service Not Starting
- Check systemd status: `sudo systemctl status archipelago`
- Check logs: `sudo journalctl -u archipelago -n 50`
- Verify binary: `ls -lh /usr/local/bin/archipelago`
- Test manually: `sudo /usr/local/bin/archipelago`