archy/.claude/skills/podman-fix/SKILL.md

---
name: podman-fix
description: >
  Fix Podman container issues on Archipelago — restart failed containers, repair port bindings,
  fix network connectivity, add missing restart policies, fix rootless UID mapping, and resolve
  config drift. Handles rootless Podman (user: archipelago, UID 1000, subuid 100000:65536).
  Use when asked to "fix container", "restart app", "fix port mapping", "container not working",
  "app won't start", "fix podman", "repair container", "container down", "permission denied",
  or after /podman-doctor identifies issues to fix.
allowed-tools: Bash Read Edit Write Glob Grep
---

# Podman Fix — Container Remediation

Targeted fix workflow for **rootless Podman** container issues on Archipelago. Given a specific problem (from /podman-doctor or user report), diagnose the root cause and fix it.

**SSH command**: `ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228`

> **ROOTLESS PODMAN**: All `podman` commands run as the `archipelago` user — NO sudo.
> Only use `sudo` for: chown on volume directories, UFW changes, systemd service edits, nginx reload.
> Container UIDs are mapped via subuid: container UID N → host UID (100000 + N).

If $ARGUMENTS is provided, fix that specific app/issue. Otherwise ask what needs fixing.

## Fix Procedures

### Fix 1: Container Not Running

```bash
# Check why it stopped
podman logs --tail 50 CONTAINER_NAME
podman inspect CONTAINER_NAME --format "{{.State.ExitCode}} {{.State.Error}}"

# If clean exit or crash — just restart
podman start CONTAINER_NAME

# If corrupt state — remove and recreate
podman rm -f CONTAINER_NAME
# Then recreate using the install flow (trigger from UI or re-run creation command)
```

**If container keeps crashing**, check logs for the actual error. Common causes:
- Missing config file → check if volume mount has the config
- Wrong permissions → fix UID mapping (see Fix 8 below)
- Dependency not ready → start dependency first, wait, then start this container
- Exit code 127 → missing binary in container image, re-pull the image

### Fix 2: Missing Restart Policy

The most common uptime killer. Fix for ALL containers at once:

```bash
# Fix a single container
podman update --restart unless-stopped CONTAINER_NAME

# Fix ALL containers that have no restart policy
for c in $(podman ps -a --format "{{.Names}}"); do
  policy=$(podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
  if [ "$policy" = "no" ] || [ -z "$policy" ]; then
    echo "Fixing restart policy for: $c"
    podman update --restart unless-stopped "$c"
  fi
done
```

**Also update the Rust source** so new installs get it right:
- Check `core/archipelago/src/api/rpc/package.rs` `get_app_config()` for the app
- Ensure `--restart` flag is in the podman run args

### Fix 3: Port Mapping Issues

#### Port conflict (address already in use)
```bash
# Find what's using the port
ss -tlnp | grep :PORT_NUMBER

# If it's another container, either change one's port or stop the conflicting one
podman stop CONFLICTING_CONTAINER

# If it's a host process (e.g., system tor vs container tor)
sudo systemctl stop tor  # Stop system service if container needs the port
sudo systemctl disable tor
```

#### Port not mapped (container running but port unreachable)
```bash
# Check current port mappings
podman port CONTAINER_NAME

# Can't add ports to running container — must recreate
podman stop CONTAINER_NAME
podman rm CONTAINER_NAME
# Recreate with correct -p flags (use the Rust install flow or manual podman run)
```

#### Nginx proxy missing or wrong
Read and fix the nginx config:
- HTTP: `image-recipe/configs/nginx-archipelago.conf`
- HTTPS: `image-recipe/configs/snippets/archipelago-https-app-proxies.conf`

Add a location block:
```nginx
location /app/APP_ID/ {
    proxy_pass http://127.0.0.1:HOST_PORT/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    # Hide X-Frame-Options so it works in our iframe
    proxy_hide_header X-Frame-Options;
    proxy_hide_header Content-Security-Policy;
}
```

After editing nginx config, deploy and reload:
```bash
# On server
sudo nginx -t && sudo systemctl reload nginx
```

#### Frontend routing missing
Edit `neode-ui/src/stores/appLauncher.ts`:
- Add entry to `PORT_TO_APP_ID` map
- If app blocks iframes, add port to the new-tab list in `resolveAppIdFromUrl()`

### Fix 4: Network Issues

#### Container not on archy-net (can't resolve other containers)
```bash
# Connect to archy-net without recreating
podman network connect archy-net CONTAINER_NAME

# Verify
podman inspect CONTAINER_NAME --format "{{.NetworkSettings.Networks}}"
```

#### archy-net doesn't exist
```bash
podman network create archy-net
# Then reconnect all containers that need it
```

#### DNS not working inside container
```bash
# Test DNS from inside container
podman exec CONTAINER_NAME nslookup bitcoin-knots 2>/dev/null || \
podman exec CONTAINER_NAME ping -c1 bitcoin-knots

# If DNS fails, check the container's resolv.conf
podman exec CONTAINER_NAME cat /etc/resolv.conf

# If DNS fails, recreate container with explicit DNS
# Add --dns 1.1.1.1 to the podman run command
```

#### Container subnet changed (rootful → rootless migration)
```bash
# Old rootful subnet: 10.88.0.0/16
# New rootless subnet: 10.89.0.0/16
# Bitcoin RPC rpcallowip must be updated if using subnet-specific allowlist

# Check current archy-net subnet
podman network inspect archy-net --format "{{range .Subnets}}{{.Subnet}}{{end}}"

# If Bitcoin RPC refuses connections from containers:
# Update bitcoin.conf rpcallowip to 0.0.0.0/0 (safe: only accessible via port mapping)
```

### Fix 5: Health Check Issues

#### Add missing health check to running container
Can't add to running container — must recreate with health check flags:
```bash
# Example for a web app
podman run ... \
  --health-cmd "curl -f http://localhost:PORT/health || exit 1" \
  --health-interval 30s \
  --health-timeout 5s \
  --health-retries 3 \
  --health-start-period 60s \
  IMAGE
```

#### Fix unhealthy container
```bash
# See what the health check is actually running
podman inspect CONTAINER_NAME --format "{{.Config.Healthcheck.Test}}"

# Run the health check manually to see the error
podman exec CONTAINER_NAME HEALTH_CHECK_COMMAND

# Common fixes:
# - curl not installed in container → use wget or nc instead
# - Wrong port in health check → fix the check command
# - App takes too long to start → increase --health-start-period
```

### Fix 6: Permission/Capability Issues

```bash
# Check what capabilities container has
podman inspect CONTAINER_NAME --format "{{.HostConfig.CapAdd}}"

# If missing required caps, must recreate with correct --cap-add flags
# Refer to the capability reference in /podman-doctor references
```

### Fix 7: Full Config Consistency Fix

When port map is inconsistent across layers, fix ALL layers:

1. **Decide the correct port** (usually what's in package.rs)
2. **Fix Podman**: recreate container with correct `-p` flags
3. **Fix Nginx**: update location block's `proxy_pass` port
4. **Fix Frontend**: update `PORT_TO_APP_ID` in appLauncher.ts
5. **Deploy**: `./scripts/deploy-to-target.sh --live`
6. **Verify**: `curl -I http://192.168.1.228/app/APP_ID/`

### Fix 8: Rootless UID Mapping (Permission Denied on Volumes)

This is the #1 rootless-specific issue. Container UIDs are remapped by user namespaces.

**Formula**: `host_uid = 100000 + container_uid`

```bash
# Fix UID 0 containers (most apps — run as root inside, mapped to 100000 on host)
sudo chown -R 100000:100000 /var/lib/archipelago/APP_NAME

# Fix Bitcoin (container UID 101 → host UID 100101)
sudo chown -R 100101:100101 /var/lib/archipelago/bitcoin

# Fix PostgreSQL (container UID 70 → host UID 100070)
sudo chown -R 100070:100070 /var/lib/archipelago/postgres-APP_NAME

# Fix Grafana (container UID 472 → host UID 100472)
sudo chown -R 100472:100472 /var/lib/archipelago/grafana

# Fix MariaDB (container UID 999 → host UID 100999)
sudo chown -R 100999:100999 /var/lib/archipelago/mysql-mempool
```

**How to find the right UID for a new container:**
```bash
# Check what user the container image runs as
podman inspect IMAGE_NAME --format "{{.Config.User}}"
# If empty = root (UID 0) → host UID 100000
# If number → host UID = 100000 + that number
# If username → run: podman run --rm IMAGE_NAME id
```

After fixing ownership, restart the container:
```bash
podman restart CONTAINER_NAME
```

### Fix 9: UFW Forward Policy (LAN Access Broken)

If containers work locally but not from other machines on the network:

```bash
# Check current policy
grep DEFAULT_FORWARD_POLICY /etc/default/ufw

# Fix: change DROP to ACCEPT
sudo sed -i 's/DEFAULT_FORWARD_POLICY="DROP"/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw
sudo ufw reload
```

### Fix 10: Systemd Sandbox Too Restrictive

If the Rust backend can't scan/manage containers after a systemd update:

```bash
# Check what's blocked
sudo journalctl -u archipelago --since "10 min ago" | grep -i "denied\|permission\|namespace\|syscall"

# The archipelago.service MUST have these for rootless podman:
# ProtectHome=no
# PrivateTmp=no (or disabled)
# RestrictNamespaces= (NOT SET — don't restrict)
# SystemCallFilter= (NOT SET — don't filter)
# ReadWritePaths=/var/lib/archipelago /etc/containers /var/lib/containers /run/containers /run/user /tmp
# Environment=XDG_RUNTIME_DIR=/run/user/1000
```

Edit the service file:
```bash
sudo systemctl edit archipelago.service
# Add overrides, then:
sudo systemctl daemon-reload
sudo systemctl restart archipelago
```

### Fix 11: Stale Podman Processes

If `podman ps` hangs or is very slow:

```bash
# Kill stuck podman processes (>10 of them = something is wrong)
stuck=$(pgrep -c -f "podman ps\|podman stats" 2>/dev/null || echo 0)
if [ "$stuck" -gt 10 ]; then
  pkill -f "podman ps\|podman stats"
  echo "Killed $stuck stuck podman processes"
fi

# Kill orphaned conmon processes holding ports
for pid in $(pgrep conmon); do
  container=$(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ' | grep -oP '(?<=--cid )\S+')
  if [ -n "$container" ] && ! podman ps -a --format "{{.ID}}" | grep -q "${container:0:12}"; then
    kill "$pid" 2>/dev/null && echo "Killed orphan conmon $pid"
  fi
done
```

## After Fixing

Always verify the fix:
```bash
# Container running?
podman ps --filter name=CONTAINER_NAME

# Port reachable?
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:PORT/

# Via nginx proxy?
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1/app/APP_ID/

# Health check passing?
podman inspect CONTAINER_NAME --format "{{.State.Health.Status}}"

# Volume permissions correct? (rootless check)
podman exec CONTAINER_NAME ls -la /data/ 2>/dev/null || echo "Check container data path"
```

Run `/podman-doctor` again to confirm all issues are resolved.