archy/.claude/skills/podman-doctor/SKILL.md

---
name: podman-doctor
description: >
  Comprehensive Podman container diagnostic for Archipelago. Audits all running containers,
  port mappings, network connectivity, health status, restart policies, and config consistency
  across all 4 layers (backend Rust, Podman runtime, Nginx proxy, frontend routing).
  Use when asked to "diagnose containers", "check podman", "why is app not working",
  "container health check", "port not reachable", "audit containers", "podman status",
  or when any container/app is misbehaving.
allowed-tools: Bash Read Glob Grep
---

# Podman Doctor — Container Infrastructure Diagnostics

Systematic diagnostic for Archipelago's Podman container stack. Catches port conflicts, network misconfigurations, health failures, missing restart policies, and config drift across all layers.

**SSH command**: `ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228`

If $ARGUMENTS is provided, focus diagnosis on that specific app/container. Otherwise run full audit.

## Workflow

### Step 1: Gather Runtime State

Run these on the server:

```bash
# All containers with status, ports, networks
sudo podman ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}\t{{.Networks}}"

# Check for port conflicts on known ports
sudo ss -tlnp | grep -E ":(80|443|3000|4080|5678|8080|8081|8082|8083|8085|8096|8123|8173|8174|8175|8240|8332|8333|8334|8888|9735|10009|11434|23000|50001)\b"
```

### Step 2: Check Restart Policies

Every container MUST have `--restart unless-stopped`. This is the #1 cause of downtime after reboots.

```bash
for c in $(sudo podman ps -a --format "{{.Names}}"); do
  echo -n "$c: "
  sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}"
done
```

**Red flag**: `no` or empty = container won't survive reboot.

### Step 3: Verify Port Mapping Consistency

Cross-reference these 4 layers — mismatches between ANY two cause "app not loading" bugs:

**Layer 1 — Backend Config (Rust)**: Read `core/archipelago/src/api/rpc/package.rs`, look at `get_app_config()` port mappings.

**Layer 2 — Podman Runtime**: `sudo podman ps --format "{{.Names}}: {{.Ports}}"`

**Layer 3 — Nginx Proxy**: Read these for `/app/{id}/` location blocks:
- `image-recipe/configs/nginx-archipelago.conf` (HTTP)
- `image-recipe/configs/snippets/archipelago-https-app-proxies.conf` (HTTPS)

**Layer 4 — Frontend Routing**: Read `neode-ui/src/stores/appLauncher.ts` — `PORT_TO_APP_ID` map.

| Symptom | Root Cause |
|---------|-----------|
| App iframe shows 502/504 | Nginx proxies to wrong port, or container not running |
| App loads wrong content | Port collision — two containers on same host port |
| Works on port but not /app/ path | Missing nginx location block |
| Frontend can't find app | PORT_TO_APP_ID missing in appLauncher.ts |

### Step 4: Network Connectivity Audit

```bash
# Networks and their containers
sudo podman network ls
sudo podman network inspect archy-net 2>/dev/null || echo "WARNING: archy-net missing!"
```

**Must be on archy-net**: bitcoin-knots, lnd, electrs, mempool, btcpay-server, nbxplorer, fedimint, fedimint-gateway, nostr-rs-relay, indeedhub, ollama, open-webui

**Must NOT be on archy-net**: grafana, nextcloud, filebrowser, vaultwarden, bitcoin-ui, lnd-ui, tailscale (host network)

### Step 5: Health Check Status

```bash
# Containers with health checks — are they passing?
for c in $(sudo podman ps --format "{{.Names}}"); do
  health=$(sudo podman inspect "$c" --format "{{.State.Health.Status}}" 2>/dev/null)
  if [ -n "$health" ] && [ "$health" != "<no value>" ]; then
    echo "$c: $health"
  fi
done

# Containers WITHOUT health checks (gap in monitoring)
for c in $(sudo podman ps --format "{{.Names}}"); do
  hc=$(sudo podman inspect "$c" --format "{{.Config.Healthcheck}}" 2>/dev/null)
  if [ "$hc" = "<nil>" ] || [ -z "$hc" ]; then
    echo "NO HEALTHCHECK: $c"
  fi
done
```

### Step 6: Resource & Failure Analysis

```bash
# Resource usage
sudo podman stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Recent deaths (last 24h)
sudo podman events --filter event=died --since 24h 2>/dev/null | tail -20

# OOM kills
sudo podman ps -a --format "{{.Names}}" | while read c; do
  oom=$(sudo podman inspect "$c" --format "{{.State.OOMKilled}}" 2>/dev/null)
  [ "$oom" = "true" ] && echo "OOM KILLED: $c"
done

# Non-zero exits
sudo podman ps -a --filter status=exited --format "{{.Names}}\t{{.Status}}"
```

### Step 7: Systemd Integration

```bash
systemctl is-active archipelago nginx
systemctl list-units --type=service | grep -i podman
systemctl list-timers --all | grep -i -E "podman|container|archipelago"
```

### Step 8: Generate Report

Produce a structured report:

```
## Container Diagnostic Report

### Summary
- Total containers: X running, Y stopped, Z unhealthy
- Port conflicts: [list or "none"]
- Missing restart policies: [list or "none"]
- Network issues: [list or "none"]
- Health check gaps: [list]

### Critical Issues (fix immediately)
1. ...

### Warnings (fix soon)
1. ...

### Recommended Actions
1. ...
```

After diagnosis, suggest running `/podman-fix` for any issues found.

## Port Reference

See `references/port-map.md` for the canonical port assignment table across all 4 layers.