archy/.claude/skills/podman-doctor/SKILL.md

---
name: podman-doctor
description: >
  Comprehensive Podman container diagnostic for Archipelago. Audits all running containers,
  port mappings, network connectivity, health status, restart policies, and config consistency
  across all 4 layers (backend Rust, Podman runtime, Nginx proxy, frontend routing).
  Handles rootless Podman (user: archipelago, UID 1000, subuid 100000:65536).
  Use when asked to "diagnose containers", "check podman", "why is app not working",
  "container health check", "port not reachable", "audit containers", "podman status",
  or when any container/app is misbehaving.
allowed-tools: Bash Read Glob Grep
---

# Podman Doctor — Container Infrastructure Diagnostics

Systematic diagnostic for Archipelago's **rootless Podman** container stack. Catches port conflicts, network misconfigurations, health failures, missing restart policies, UID mapping issues, and config drift across all layers.

**SSH command**: `ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228`

> **ROOTLESS PODMAN**: Archipelago runs Podman as the `archipelago` user (UID 1000), NOT root.
> Never use `sudo podman` — use plain `podman` after SSH'ing in as the `archipelago` user.
> Container UIDs are mapped via subuid: container UID N → host UID (100000 + N).

If $ARGUMENTS is provided, focus diagnosis on that specific app/container. Otherwise run full audit.

## Workflow

### Step 1: Gather Runtime State

Run these on the server (as `archipelago` user — NO sudo):

```bash
# All containers with status, ports, networks
podman ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}\t{{.Networks}}"

# Check for port conflicts on known ports
ss -tlnp | grep -E ":(80|443|3000|4080|5678|8080|8081|8082|8083|8085|8096|8123|8173|8174|8175|8240|8332|8333|8334|8888|9735|10009|11434|23000|50001)\b"
```

### Step 2: Rootless Podman Health Check

Rootless Podman has specific requirements that must be verified:

```bash
# Verify running as archipelago user (NOT root)
whoami  # Must be "archipelago"
id      # Must show uid=1000(archipelago)

# Check XDG_RUNTIME_DIR is set (required for rootless podman socket)
echo "XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR"  # Must be /run/user/1000

# Verify subuid/subgid mapping exists
grep archipelago /etc/subuid  # Must show: archipelago:100000:65536
grep archipelago /etc/subgid  # Must show: archipelago:100000:65536

# Verify user lingering is enabled (keeps user services after logout)
ls /var/lib/systemd/linger/ | grep archipelago  # Must exist

# Check podman storage is accessible
podman info --format "{{.Store.GraphRoot}}"  # ~/.local/share/containers/storage
ls -la ~/.local/share/containers/storage/ 2>/dev/null || echo "ERROR: Storage not accessible"

# Check podman socket
ls -la /run/user/1000/podman/ 2>/dev/null || echo "WARNING: No podman socket directory"
```

### Step 3: Check Restart Policies

Every container MUST have `--restart unless-stopped`. This is the #1 cause of downtime after reboots.

```bash
for c in $(podman ps -a --format "{{.Names}}"); do
  echo -n "$c: "
  podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}"
done
```

**Red flag**: `no` or empty = container won't survive reboot.

### Step 4: Volume Ownership Audit (Rootless UID Mapping)

Rootless Podman maps container UIDs via subuid. Volume directories must be owned by the MAPPED UID, not the container UID. Formula: `host_uid = 100000 + container_uid`

```bash
echo "=== Volume Ownership Check ==="

# Default containers (run as root inside = UID 0 → host UID 100000)
for dir in lnd fedimint homeassistant jellyfin vaultwarden photoprism ollama filebrowser electrumx btcpay immich; do
  if [ -d "/var/lib/archipelago/$dir" ]; then
    owner=$(stat -c '%u:%g' "/var/lib/archipelago/$dir" 2>/dev/null)
    if [ "$owner" != "100000:100000" ]; then
      echo "WRONG: /var/lib/archipelago/$dir owned by $owner (should be 100000:100000)"
    else
      echo "  OK: $dir → $owner"
    fi
  fi
done

# Bitcoin Knots (container UID 101 → host UID 100101)
if [ -d "/var/lib/archipelago/bitcoin" ]; then
  owner=$(stat -c '%u:%g' "/var/lib/archipelago/bitcoin")
  [ "$owner" != "100101:100101" ] && echo "WRONG: bitcoin owned by $owner (should be 100101:100101)" || echo "  OK: bitcoin → $owner"
fi

# PostgreSQL (container UID 70 → host UID 100070)
for dir in /var/lib/archipelago/*-db /var/lib/archipelago/postgres-*; do
  if [ -d "$dir" ]; then
    owner=$(stat -c '%u:%g' "$dir")
    [ "$owner" != "100070:100070" ] && echo "WRONG: $dir owned by $owner (should be 100070:100070)" || echo "  OK: $(basename $dir) → $owner"
  fi
done

# Grafana (container UID 472 → host UID 100472)
if [ -d "/var/lib/archipelago/grafana" ]; then
  owner=$(stat -c '%u:%g' "/var/lib/archipelago/grafana")
  [ "$owner" != "100472:100472" ] && echo "WRONG: grafana owned by $owner (should be 100472:100472)" || echo "  OK: grafana → $owner"
fi

# MariaDB/MySQL (container UID 999 → host UID 100999)
if [ -d "/var/lib/archipelago/mysql-mempool" ]; then
  owner=$(stat -c '%u:%g' "/var/lib/archipelago/mysql-mempool")
  [ "$owner" != "100999:100999" ] && echo "WRONG: mysql-mempool owned by $owner (should be 100999:100999)" || echo "  OK: mysql-mempool → $owner"
fi
```

### Step 5: Verify Port Mapping Consistency

Cross-reference these 4 layers — mismatches between ANY two cause "app not loading" bugs:

**Layer 1 — Backend Config (Rust)**: Read `core/archipelago/src/api/rpc/package.rs`, look at `get_app_config()` port mappings.

**Layer 2 — Podman Runtime**: `podman ps --format "{{.Names}}: {{.Ports}}"`

**Layer 3 — Nginx Proxy**: Read these for `/app/{id}/` location blocks:
- `image-recipe/configs/nginx-archipelago.conf` (HTTP)
- `image-recipe/configs/snippets/archipelago-https-app-proxies.conf` (HTTPS)

**Layer 4 — Frontend Routing**: Read `neode-ui/src/stores/appLauncher.ts` — `PORT_TO_APP_ID` map.

| Symptom | Root Cause |
|---------|-----------|
| App iframe shows 502/504 | Nginx proxies to wrong port, or container not running |
| App loads wrong content | Port collision — two containers on same host port |
| Works on port but not /app/ path | Missing nginx location block |
| Frontend can't find app | PORT_TO_APP_ID missing in appLauncher.ts |

### Step 6: Network Connectivity Audit

```bash
# Networks and their containers
podman network ls
podman network inspect archy-net 2>/dev/null || echo "WARNING: archy-net missing!"

# Check container subnet (rootless uses 10.89.x.x, NOT 10.88.x.x)
podman network inspect archy-net --format "{{range .Subnets}}{{.Subnet}}{{end}}" 2>/dev/null
```

**Must be on archy-net**: bitcoin-knots, lnd, electrs/electrumx, mempool, btcpay-server, nbxplorer, fedimint, fedimint-gateway, nostr-rs-relay, indeedhub, ollama, open-webui

**Must NOT be on archy-net**: grafana, nextcloud, filebrowser, vaultwarden, bitcoin-ui, lnd-ui, tailscale (host network)

### Step 7: UFW Forward Policy Check

Rootless Podman requires `DEFAULT_FORWARD_POLICY="ACCEPT"` in UFW, otherwise container ports are unreachable from LAN.

```bash
grep DEFAULT_FORWARD_POLICY /etc/default/ufw
# Must be "ACCEPT", NOT "DROP"
# If DROP: containers work locally but NOT from other machines on the network
```

### Step 8: Systemd Service Sandbox Check

The `archipelago.service` must have specific settings relaxed for rootless Podman:

```bash
# Check critical settings
systemctl cat archipelago.service | grep -E "ProtectHome|PrivateTmp|RestrictNamespaces|ReadWritePaths|XDG_RUNTIME_DIR"
```

**Required settings for rootless Podman**:
- `ProtectHome=no` — podman stores images in `~/.local/share/containers/`
- `PrivateTmp=no` or disabled — podman runtime uses `/tmp/podman-run-1000/`
- `RestrictNamespaces=` must NOT be set — rootless podman needs user namespaces
- `ReadWritePaths=` must include `/var/lib/archipelago /run/user /tmp`
- `Environment=XDG_RUNTIME_DIR=/run/user/1000`

### Step 9: Health Check Status

```bash
# Containers with health checks — are they passing?
for c in $(podman ps --format "{{.Names}}"); do
  health=$(podman inspect "$c" --format "{{.State.Health.Status}}" 2>/dev/null)
  if [ -n "$health" ] && [ "$health" != "<no value>" ]; then
    echo "$c: $health"
  fi
done

# Containers WITHOUT health checks (gap in monitoring)
for c in $(podman ps --format "{{.Names}}"); do
  hc=$(podman inspect "$c" --format "{{.Config.Healthcheck}}" 2>/dev/null)
  if [ "$hc" = "<nil>" ] || [ -z "$hc" ]; then
    echo "NO HEALTHCHECK: $c"
  fi
done
```

### Step 10: Resource & Failure Analysis

```bash
# Resource usage
podman stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Recent deaths (last 24h)
podman events --filter event=died --since 24h 2>/dev/null | tail -20

# OOM kills
podman ps -a --format "{{.Names}}" | while read c; do
  oom=$(podman inspect "$c" --format "{{.State.OOMKilled}}" 2>/dev/null)
  [ "$oom" = "true" ] && echo "OOM KILLED: $c"
done

# Non-zero exits
podman ps -a --filter status=exited --format "{{.Names}}\t{{.Status}}"
```

### Step 11: Systemd Integration

```bash
systemctl is-active archipelago nginx
systemctl --user list-units --type=service 2>/dev/null | grep -i podman
systemctl list-timers --all | grep -i -E "podman|container|archipelago"
```

### Step 12: Generate Report

Produce a structured report:

```
## Container Diagnostic Report

### Rootless Podman Status
- User: archipelago (UID 1000)
- Subuid mapping: [OK/MISSING]
- XDG_RUNTIME_DIR: [OK/MISSING]
- User linger: [enabled/disabled]
- UFW forward policy: [ACCEPT/DROP]

### Summary
- Total containers: X running, Y stopped, Z unhealthy
- Port conflicts: [list or "none"]
- Missing restart policies: [list or "none"]
- Network issues: [list or "none"]
- UID mapping issues: [list or "none"]
- Health check gaps: [list]

### Critical Issues (fix immediately)
1. ...

### Warnings (fix soon)
1. ...

### Recommended Actions
1. ...
```

After diagnosis, suggest running `/podman-fix` for any issues found.

## Port Reference

See `references/port-map.md` for the canonical port assignment table across all 4 layers.

## UID Mapping Reference

See `references/uid-mapping.md` for the complete rootless UID mapping table.