archy/.claude/skills/podman-doctor/SKILL.md

name, description, allowed-tools

name	description	allowed-tools
podman-doctor	Comprehensive Podman container diagnostic for Archipelago. Audits all running containers, port mappings, network connectivity, health status, restart policies, and config consistency across all 4 layers (backend Rust, Podman runtime, Nginx proxy, frontend routing). Use when asked to "diagnose containers", "check podman", "why is app not working", "container health check", "port not reachable", "audit containers", "podman status", or when any container/app is misbehaving.	Bash Read Glob Grep

Podman Doctor — Container Infrastructure Diagnostics

Systematic diagnostic for Archipelago's Podman container stack. Catches port conflicts, network misconfigurations, health failures, missing restart policies, and config drift across all layers.

SSH command: ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228

If $ARGUMENTS is provided, focus diagnosis on that specific app/container. Otherwise run full audit.

Workflow

Step 1: Gather Runtime State

Run these on the server:

# All containers with status, ports, networks
sudo podman ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}\t{{.Networks}}"

# Check for port conflicts on known ports
sudo ss -tlnp | grep -E ":(80|443|3000|4080|5678|8080|8081|8082|8083|8085|8096|8123|8173|8174|8175|8240|8332|8333|8334|8888|9735|10009|11434|23000|50001)\b"

Step 2: Check Restart Policies

Every container MUST have --restart unless-stopped. This is the #1 cause of downtime after reboots.

for c in $(sudo podman ps -a --format "{{.Names}}"); do
  echo -n "$c: "
  sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}"
done

Red flag: no or empty = container won't survive reboot.

Step 3: Verify Port Mapping Consistency

Cross-reference these 4 layers — mismatches between ANY two cause "app not loading" bugs:

Layer 1 — Backend Config (Rust): Read core/archipelago/src/api/rpc/package.rs, look at get_app_config() port mappings.

Layer 2 — Podman Runtime: sudo podman ps --format "{{.Names}}: {{.Ports}}"

Layer 3 — Nginx Proxy: Read these for /app/{id}/ location blocks:

• image-recipe/configs/nginx-archipelago.conf (HTTP)
• image-recipe/configs/snippets/archipelago-https-app-proxies.conf (HTTPS)

Layer 4 — Frontend Routing: Read neode-ui/src/stores/appLauncher.ts — PORT_TO_APP_ID map.

Symptom	Root Cause
App iframe shows 502/504	Nginx proxies to wrong port, or container not running
App loads wrong content	Port collision — two containers on same host port
Works on port but not /app/ path	Missing nginx location block
Frontend can't find app	PORT_TO_APP_ID missing in appLauncher.ts

Step 4: Network Connectivity Audit

# Networks and their containers
sudo podman network ls
sudo podman network inspect archy-net 2>/dev/null || echo "WARNING: archy-net missing!"

Must be on archy-net: bitcoin-knots, lnd, electrs, mempool, btcpay-server, nbxplorer, fedimint, fedimint-gateway, nostr-rs-relay, indeedhub, ollama, open-webui

Must NOT be on archy-net: grafana, nextcloud, filebrowser, vaultwarden, bitcoin-ui, lnd-ui, tailscale (host network)

Step 5: Health Check Status

# Containers with health checks — are they passing?
for c in $(sudo podman ps --format "{{.Names}}"); do
  health=$(sudo podman inspect "$c" --format "{{.State.Health.Status}}" 2>/dev/null)
  if [ -n "$health" ] && [ "$health" != "<no value>" ]; then
    echo "$c: $health"
  fi
done

# Containers WITHOUT health checks (gap in monitoring)
for c in $(sudo podman ps --format "{{.Names}}"); do
  hc=$(sudo podman inspect "$c" --format "{{.Config.Healthcheck}}" 2>/dev/null)
  if [ "$hc" = "<nil>" ] || [ -z "$hc" ]; then
    echo "NO HEALTHCHECK: $c"
  fi
done

Step 6: Resource & Failure Analysis

# Resource usage
sudo podman stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

# Recent deaths (last 24h)
sudo podman events --filter event=died --since 24h 2>/dev/null | tail -20

# OOM kills
sudo podman ps -a --format "{{.Names}}" | while read c; do
  oom=$(sudo podman inspect "$c" --format "{{.State.OOMKilled}}" 2>/dev/null)
  [ "$oom" = "true" ] && echo "OOM KILLED: $c"
done

# Non-zero exits
sudo podman ps -a --filter status=exited --format "{{.Names}}\t{{.Status}}"

Step 7: Systemd Integration

systemctl is-active archipelago nginx
systemctl list-units --type=service | grep -i podman
systemctl list-timers --all | grep -i -E "podman|container|archipelago"

Step 8: Generate Report

Produce a structured report:

## Container Diagnostic Report

### Summary
- Total containers: X running, Y stopped, Z unhealthy
- Port conflicts: [list or "none"]
- Missing restart policies: [list or "none"]
- Network issues: [list or "none"]
- Health check gaps: [list]

### Critical Issues (fix immediately)
1. ...

### Warnings (fix soon)
1. ...

### Recommended Actions
1. ...

After diagnosis, suggest running /podman-fix for any issues found.

Port Reference

See references/port-map.md for the canonical port assignment table across all 4 layers.