archy/.claude/skills/polish-deploy/SKILL.md

# Skill: Polish Deployment Pipeline

Harden deploy-to-target.sh with rollback capability, pre-deploy checks, post-deploy health verification, and deployment locking.

## 1. Pre-Deploy Checks

Add to the beginning of deploy-to-target.sh:

```bash
pre_deploy_checks() {
  echo "Running pre-deploy checks..."

  # SSH key exists
  if [ ! -f "$SSH_KEY" ]; then
    echo "ERROR: SSH key not found at $SSH_KEY"
    exit 1
  fi

  # Target reachable
  ssh $SSH_OPTS "$TARGET_HOST" "echo ok" >/dev/null 2>&1 || {
    echo "ERROR: Cannot reach $TARGET_HOST"
    exit 1
  }

  # Disk space (need 2GB free)
  local free_kb=$(ssh $SSH_OPTS "$TARGET_HOST" "df /home | tail -1 | awk '{print \$4}'")
  if [ "$free_kb" -lt 2097152 ]; then
    echo "ERROR: Need 2GB free disk space, have $(( free_kb / 1024 ))MB"
    exit 1
  fi

  echo "Pre-deploy checks passed"
}
```

## 2. Backup Before Deploy

Before overwriting binary or frontend:

```bash
backup_current() {
  echo "Backing up current deployment..."
  ssh $SSH_OPTS "$TARGET_HOST" "
    # Backup binary
    if [ -f /usr/local/bin/archipelago ]; then
      sudo cp /usr/local/bin/archipelago /usr/local/bin/archipelago.backup
    fi
    # Backup frontend
    if [ -d /opt/archipelago/web-ui ]; then
      sudo cp -a /opt/archipelago/web-ui /opt/archipelago/web-ui.backup
    fi
    # Backup nginx config
    if [ -f /etc/nginx/sites-available/archipelago ]; then
      sudo cp /etc/nginx/sites-available/archipelago /etc/nginx/sites-available/archipelago.backup
    fi
  "
  echo "Backup complete"
}
```

## 3. Post-Deploy Health Check

After restarting services:

```bash
health_check() {
  echo "Running post-deploy health check..."
  local max_attempts=15
  local attempt=0

  while [ $attempt -lt $max_attempts ]; do
    attempt=$((attempt + 1))
    local status=$(ssh $SSH_OPTS "$TARGET_HOST" "curl -s -o /dev/null -w '%{http_code}' http://localhost:5678/health" 2>/dev/null)
    if [ "$status" = "200" ]; then
      echo "Health check passed (attempt $attempt)"
      return 0
    fi
    echo "Health check attempt $attempt/$max_attempts (status: $status)"
    sleep 2
  done

  echo "ERROR: Health check failed after $max_attempts attempts"
  return 1
}
```

## 4. Rollback on Failure

If health check fails:

```bash
rollback() {
  echo "ROLLING BACK deployment..."
  ssh $SSH_OPTS "$TARGET_HOST" "
    # Restore binary
    if [ -f /usr/local/bin/archipelago.backup ]; then
      sudo cp /usr/local/bin/archipelago.backup /usr/local/bin/archipelago
    fi
    # Restore frontend
    if [ -d /opt/archipelago/web-ui.backup ]; then
      sudo rm -rf /opt/archipelago/web-ui
      sudo mv /opt/archipelago/web-ui.backup /opt/archipelago/web-ui
    fi
    # Restore nginx
    if [ -f /etc/nginx/sites-available/archipelago.backup ]; then
      sudo cp /etc/nginx/sites-available/archipelago.backup /etc/nginx/sites-available/archipelago
      sudo nginx -t && sudo systemctl reload nginx
    fi
    # Restart with old binary
    sudo systemctl restart archipelago
  "
  echo "Rollback complete. Previous version restored."
}
```

## 5. Deployment Lock

Prevent concurrent deploys:

```bash
LOCK_FILE="/tmp/archipelago-deploy.lock"

acquire_lock() {
  exec 9>"$LOCK_FILE"
  flock -n 9 || {
    echo "ERROR: Another deployment is in progress"
    exit 1
  }
  trap "flock -u 9; rm -f $LOCK_FILE" EXIT
}
```

## 6. Nginx Config Validation

Before reloading nginx:

```bash
validate_nginx() {
  ssh $SSH_OPTS "$TARGET_HOST" "sudo nginx -t" 2>&1 || {
    echo "ERROR: Nginx config invalid. Restoring backup..."
    ssh $SSH_OPTS "$TARGET_HOST" "
      sudo cp /etc/nginx/sites-available/archipelago.backup /etc/nginx/sites-available/archipelago
      sudo nginx -t && sudo systemctl reload nginx
    "
    return 1
  }
}
```

## Integration

The deploy flow becomes:
1. `acquire_lock`
2. `pre_deploy_checks`
3. `backup_current`
4. Build + deploy (existing logic)
5. `validate_nginx`
6. Restart services
7. `health_check || rollback`

## Verification

Test the rollback:
1. Deploy a working version
2. Intentionally break the binary (e.g., truncate it)
3. Deploy the broken version
4. Verify rollback triggers and previous version is restored
5. Verify service is healthy after rollback

## Deploy

```bash
./scripts/deploy-to-target.sh --live
```

After modifying the deploy script itself, test with a known-good deploy first.