archy/.claude/skills/podman-uptime/SKILL.md

---
name: podman-uptime
description: >
  Ensure 100% container uptime on Archipelago. Sets up systemd watchdog timers, verifies
  restart policies, creates health check monitors, and configures auto-recovery for all
  containers. Use when asked to "ensure uptime", "containers keep dying", "auto-restart",
  "watchdog", "container monitoring", "uptime guarantee", "keep containers running",
  "survive reboot", or to harden container reliability.
allowed-tools: Bash Read Edit Write Glob Grep
---

# Podman Uptime — Container Reliability Guardian

Ensures every Archipelago container survives reboots, recovers from crashes, and stays healthy. Sets up the three layers of uptime defense: restart policies, systemd watchdog, and health-based auto-recovery.

**SSH command**: `ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228`

## Layer 1: Restart Policies (Survive Reboots)

Every container MUST have `--restart unless-stopped`. This is non-negotiable.

### Audit and fix all containers

```bash
# Audit
for c in $(sudo podman ps -a --format "{{.Names}}"); do
  policy=$(sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
  echo "$c: $policy"
done

# Fix any with "no" or empty policy
for c in $(sudo podman ps -a --format "{{.Names}}"); do
  policy=$(sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
  if [ "$policy" = "no" ] || [ -z "$policy" ]; then
    echo "Fixing: $c"
    sudo podman update --restart unless-stopped "$c"
  fi
done
```

### Ensure podman auto-starts containers on boot

```bash
# Enable podman-restart service (restarts containers with restart policy on boot)
sudo systemctl enable podman-restart.service 2>/dev/null || true

# If podman-restart doesn't exist, create it
cat <<'EOF' | sudo tee /etc/systemd/system/podman-restart.service
[Unit]
Description=Podman Start All Containers With Restart Policy
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/podman start --all --filter restart-policy=unless-stopped
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable podman-restart.service
```

## Layer 2: Systemd Watchdog (Detect and Recover)

Create a systemd timer that checks container health every 2 minutes and restarts unhealthy or stopped containers.

### Create the watchdog script

```bash
cat <<'SCRIPT' | sudo tee /usr/local/bin/archipelago-container-watchdog.sh
#!/bin/bash
# Archipelago Container Watchdog
# Checks all containers and restarts any that are stopped or unhealthy

LOG_TAG="container-watchdog"

# Restart any stopped containers that should be running (have restart policy)
for c in $(sudo podman ps -a --filter status=exited --filter restart-policy=unless-stopped --format "{{.Names}}"); do
  logger -t "$LOG_TAG" "Restarting stopped container: $c"
  sudo podman start "$c" 2>&1 | logger -t "$LOG_TAG"
done

# Restart unhealthy containers
for c in $(sudo podman ps --filter health=unhealthy --format "{{.Names}}"); do
  logger -t "$LOG_TAG" "Restarting unhealthy container: $c"
  sudo podman restart "$c" 2>&1 | logger -t "$LOG_TAG"
done

# Check for containers in "created" state (never started)
for c in $(sudo podman ps -a --filter status=created --format "{{.Names}}"); do
  logger -t "$LOG_TAG" "Starting created container: $c"
  sudo podman start "$c" 2>&1 | logger -t "$LOG_TAG"
done
SCRIPT

sudo chmod +x /usr/local/bin/archipelago-container-watchdog.sh
```

### Create the systemd timer

```bash
# Service unit
cat <<'EOF' | sudo tee /etc/systemd/system/archipelago-watchdog.service
[Unit]
Description=Archipelago Container Watchdog
After=podman-restart.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/archipelago-container-watchdog.sh
EOF

# Timer unit — runs every 2 minutes
cat <<'EOF' | sudo tee /etc/systemd/system/archipelago-watchdog.timer
[Unit]
Description=Run Archipelago Container Watchdog every 2 minutes

[Timer]
OnBootSec=120
OnUnitActiveSec=120
AccuracySec=30

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now archipelago-watchdog.timer
```

### Verify watchdog is running

```bash
sudo systemctl status archipelago-watchdog.timer
sudo systemctl list-timers | grep archipelago
# Check watchdog logs
sudo journalctl -t container-watchdog --since "1 hour ago" --no-pager
```

## Layer 3: Dependency-Aware Startup Order

Some containers depend on others. The watchdog handles restarts, but initial boot order matters.

### Create ordered startup script

```bash
cat <<'SCRIPT' | sudo tee /usr/local/bin/archipelago-ordered-start.sh
#!/bin/bash
# Ordered container startup for Archipelago
# Respects dependency chain: bitcoin → electrs/lnd → mempool/btcpay

LOG_TAG="ordered-start"

wait_for_container() {
  local name=$1
  local max_wait=${2:-60}
  local waited=0
  while [ $waited -lt $max_wait ]; do
    status=$(sudo podman inspect "$name" --format "{{.State.Running}}" 2>/dev/null)
    if [ "$status" = "true" ]; then
      logger -t "$LOG_TAG" "$name is running"
      return 0
    fi
    sleep 5
    waited=$((waited + 5))
  done
  logger -t "$LOG_TAG" "WARNING: $name not running after ${max_wait}s"
  return 1
}

# Tier 0: Infrastructure
logger -t "$LOG_TAG" "Starting Tier 0: Infrastructure"
sudo podman start tailscale 2>/dev/null

# Tier 1: Bitcoin (foundation)
logger -t "$LOG_TAG" "Starting Tier 1: Bitcoin"
sudo podman start bitcoin-knots 2>/dev/null
wait_for_container bitcoin-knots 120

# Tier 2: Bitcoin-dependent services
logger -t "$LOG_TAG" "Starting Tier 2: Bitcoin-dependent"
sudo podman start electrs 2>/dev/null
sudo podman start lnd 2>/dev/null
wait_for_container electrs 90
wait_for_container lnd 90

# Tier 3: Services depending on Tier 2
logger -t "$LOG_TAG" "Starting Tier 3: Second-order dependencies"
sudo podman start mempool-db 2>/dev/null
sleep 5
sudo podman start mempool 2>/dev/null
sudo podman start nbxplorer 2>/dev/null
sleep 10
sudo podman start btcpay-server 2>/dev/null
sudo podman start btcpay-postgres 2>/dev/null

# Tier 4: Independent apps (start all remaining)
logger -t "$LOG_TAG" "Starting Tier 4: Independent apps"
sudo podman start --all 2>/dev/null

# Tier 5: UI containers (need parent apps running first)
logger -t "$LOG_TAG" "Starting Tier 5: UI containers"
sudo podman start bitcoin-ui 2>/dev/null
sudo podman start lnd-ui 2>/dev/null

logger -t "$LOG_TAG" "Startup sequence complete"
SCRIPT

sudo chmod +x /usr/local/bin/archipelago-ordered-start.sh
```

### Wire into boot sequence

```bash
cat <<'EOF' | sudo tee /etc/systemd/system/archipelago-containers.service
[Unit]
Description=Archipelago Ordered Container Startup
After=network-online.target podman.service
Wants=network-online.target
Before=archipelago.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/archipelago-ordered-start.sh
RemainAfterExit=yes
TimeoutStartSec=300

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable archipelago-containers.service
```

## Verification Checklist

After setting up all 3 layers, verify:

```bash
echo "=== Layer 1: Restart Policies ==="
for c in $(sudo podman ps -a --format "{{.Names}}"); do
  policy=$(sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
  echo "  $c: $policy"
done

echo ""
echo "=== Layer 2: Watchdog Timer ==="
sudo systemctl is-active archipelago-watchdog.timer
sudo systemctl list-timers | grep archipelago

echo ""
echo "=== Layer 3: Boot Services ==="
sudo systemctl is-enabled podman-restart.service 2>/dev/null || echo "podman-restart: not found"
sudo systemctl is-enabled archipelago-containers.service 2>/dev/null || echo "ordered-start: not found"
sudo systemctl is-enabled archipelago-watchdog.timer 2>/dev/null || echo "watchdog: not found"

echo ""
echo "=== Container Health Summary ==="
total=$(sudo podman ps -a --format "{{.Names}}" | wc -l)
running=$(sudo podman ps --format "{{.Names}}" | wc -l)
stopped=$((total - running))
unhealthy=$(sudo podman ps --filter health=unhealthy --format "{{.Names}}" | wc -l)
echo "  Total: $total | Running: $running | Stopped: $stopped | Unhealthy: $unhealthy"
```

## Reboot Test

The ultimate uptime test — reboot the server and verify everything comes back:

```bash
# Before reboot: record running containers
sudo podman ps --format "{{.Names}}" | sort > /tmp/before-reboot.txt

# Reboot
sudo reboot

# After reboot (wait ~3 minutes, then SSH back in):
sudo podman ps --format "{{.Names}}" | sort > /tmp/after-reboot.txt

# Compare
diff /tmp/before-reboot.txt /tmp/after-reboot.txt
# Should show no differences
```

## Monitoring

Check uptime status anytime:
```bash
# Quick status
sudo podman ps -a --format "table {{.Names}}\t{{.Status}}" | sort

# Watchdog activity
sudo journalctl -t container-watchdog --since "24 hours ago" --no-pager

# Container events (starts, stops, deaths)
sudo podman events --since 24h --filter event=start --filter event=stop --filter event=died 2>/dev/null | tail -30
```

## Integration

- Run `/podman-doctor` first to identify issues
- Run `/podman-fix` for specific container repairs
- Run `/podman-uptime` to set up permanent reliability infrastructure
- Add to ISO build: copy watchdog scripts to `image-recipe/configs/` and enable in first-boot