archy/.claude/skills/podman-fix/SKILL.md

---
name: podman-fix
description: >
  Fix Podman container issues on Archipelago — restart failed containers, repair port bindings,
  fix network connectivity, add missing restart policies, and resolve config drift.
  Use when asked to "fix container", "restart app", "fix port mapping", "container not working",
  "app won't start", "fix podman", "repair container", "container down", or after /podman-doctor
  identifies issues to fix.
allowed-tools: Bash Read Edit Write Glob Grep
---

# Podman Fix — Container Remediation

Targeted fix workflow for Podman container issues on Archipelago. Given a specific problem (from /podman-doctor or user report), diagnose the root cause and fix it.

**SSH command**: `ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228`

If $ARGUMENTS is provided, fix that specific app/issue. Otherwise ask what needs fixing.

## Fix Procedures

### Fix 1: Container Not Running

```bash
# Check why it stopped
sudo podman logs --tail 50 CONTAINER_NAME
sudo podman inspect CONTAINER_NAME --format "{{.State.ExitCode}} {{.State.Error}}"

# If clean exit or crash — just restart
sudo podman start CONTAINER_NAME

# If corrupt state — remove and recreate
sudo podman rm -f CONTAINER_NAME
# Then recreate using the install flow (trigger from UI or re-run creation command)
```

**If container keeps crashing**: check logs for the actual error. Common causes:
- Missing config file → check if volume mount has the config
- Wrong permissions → `chown -R` the data directory
- Dependency not ready → start dependency first, wait, then start this container

### Fix 2: Missing Restart Policy

The most common uptime killer. Fix for ALL containers at once:

```bash
# Fix a single container
sudo podman update --restart unless-stopped CONTAINER_NAME

# Fix ALL containers that have no restart policy
for c in $(sudo podman ps -a --format "{{.Names}}"); do
  policy=$(sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
  if [ "$policy" = "no" ] || [ -z "$policy" ]; then
    echo "Fixing restart policy for: $c"
    sudo podman update --restart unless-stopped "$c"
  fi
done
```

**Also update the Rust source** so new installs get it right:
- Check `core/archipelago/src/api/rpc/package.rs` `get_app_config()` for the app
- Ensure `--restart` flag is in the podman run args

### Fix 3: Port Mapping Issues

#### Port conflict (address already in use)
```bash
# Find what's using the port
sudo ss -tlnp | grep :PORT_NUMBER

# If it's another container, either change one's port or stop the conflicting one
sudo podman stop CONFLICTING_CONTAINER

# If it's a host process
sudo kill PID  # or stop the service
```

#### Port not mapped (container running but port unreachable)
```bash
# Check current port mappings
sudo podman port CONTAINER_NAME

# Can't add ports to running container — must recreate
sudo podman stop CONTAINER_NAME
sudo podman rm CONTAINER_NAME
# Recreate with correct -p flags (use the Rust install flow or manual podman run)
```

#### Nginx proxy missing or wrong
Read and fix the nginx config:
- HTTP: `image-recipe/configs/nginx-archipelago.conf`
- HTTPS: `image-recipe/configs/snippets/archipelago-https-app-proxies.conf`

Add a location block:
```nginx
location /app/APP_ID/ {
    proxy_pass http://127.0.0.1:HOST_PORT/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    # Hide X-Frame-Options so it works in our iframe
    proxy_hide_header X-Frame-Options;
    proxy_hide_header Content-Security-Policy;
}
```

After editing nginx config, deploy and reload:
```bash
# On server
sudo nginx -t && sudo systemctl reload nginx
```

#### Frontend routing missing
Edit `neode-ui/src/stores/appLauncher.ts`:
- Add entry to `PORT_TO_APP_ID` map
- If app blocks iframes, add port to the new-tab list in `resolveAppIdFromUrl()`

### Fix 4: Network Issues

#### Container not on archy-net (can't resolve other containers)
```bash
# Connect to archy-net without recreating
sudo podman network connect archy-net CONTAINER_NAME

# Verify
sudo podman inspect CONTAINER_NAME --format "{{.NetworkSettings.Networks}}"
```

#### archy-net doesn't exist
```bash
sudo podman network create archy-net
# Then reconnect all containers that need it
```

#### DNS not working inside container
```bash
# Test DNS from inside container
sudo podman exec CONTAINER_NAME nslookup bitcoin-knots 2>/dev/null || \
sudo podman exec CONTAINER_NAME ping -c1 bitcoin-knots

# If DNS fails, recreate container with explicit DNS
# Add --dns 1.1.1.1 to the podman run command
```

### Fix 5: Health Check Issues

#### Add missing health check to running container
Can't add to running container — must recreate with health check flags:
```bash
# Example for a web app
sudo podman run ... \
  --health-cmd "curl -f http://localhost:PORT/health || exit 1" \
  --health-interval 30s \
  --health-timeout 5s \
  --health-retries 3 \
  --health-start-period 60s \
  IMAGE
```

#### Fix unhealthy container
```bash
# See what the health check is actually running
sudo podman inspect CONTAINER_NAME --format "{{.Config.Healthcheck.Test}}"

# Run the health check manually to see the error
sudo podman exec CONTAINER_NAME HEALTH_CHECK_COMMAND

# Common fixes:
# - curl not installed in container → use wget or nc instead
# - Wrong port in health check → fix the check command
# - App takes too long to start → increase --health-start-period
```

### Fix 6: Permission/Capability Issues

```bash
# Check what capabilities container has
sudo podman inspect CONTAINER_NAME --format "{{.HostConfig.CapAdd}}"

# If missing required caps, must recreate with correct --cap-add flags
# Refer to the capability reference in /podman-doctor references

# Fix data directory permissions
sudo chown -R 1000:1000 /var/lib/archipelago/APP_NAME/
```

### Fix 7: Full Config Consistency Fix

When port map is inconsistent across layers, fix ALL layers:

1. **Decide the correct port** (usually what's in package.rs)
2. **Fix Podman**: recreate container with correct `-p` flags
3. **Fix Nginx**: update location block's `proxy_pass` port
4. **Fix Frontend**: update `PORT_TO_APP_ID` in appLauncher.ts
5. **Deploy**: `./scripts/deploy-to-target.sh --live`
6. **Verify**: `curl -I http://192.168.1.228/app/APP_ID/`

## After Fixing

Always verify the fix:
```bash
# Container running?
sudo podman ps --filter name=CONTAINER_NAME

# Port reachable?
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:PORT/

# Via nginx proxy?
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1/app/APP_ID/

# Health check passing?
sudo podman inspect CONTAINER_NAME --format "{{.State.Health.Status}}"
```

Run `/podman-doctor` again to confirm all issues are resolved.