archy/.claude/skills/podman-fix/SKILL.md

name, description, allowed-tools

name	description	allowed-tools
podman-fix	Fix Podman container issues on Archipelago — restart failed containers, repair port bindings, fix network connectivity, add missing restart policies, and resolve config drift. Use when asked to "fix container", "restart app", "fix port mapping", "container not working", "app won't start", "fix podman", "repair container", "container down", or after /podman-doctor identifies issues to fix.	Bash Read Edit Write Glob Grep

Podman Fix — Container Remediation

Targeted fix workflow for Podman container issues on Archipelago. Given a specific problem (from /podman-doctor or user report), diagnose the root cause and fix it.

SSH command: ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228

If $ARGUMENTS is provided, fix that specific app/issue. Otherwise ask what needs fixing.

Fix Procedures

Fix 1: Container Not Running

# Check why it stopped
sudo podman logs --tail 50 CONTAINER_NAME
sudo podman inspect CONTAINER_NAME --format "{{.State.ExitCode}} {{.State.Error}}"

# If clean exit or crash — just restart
sudo podman start CONTAINER_NAME

# If corrupt state — remove and recreate
sudo podman rm -f CONTAINER_NAME
# Then recreate using the install flow (trigger from UI or re-run creation command)

If container keeps crashing: check logs for the actual error. Common causes:

• Missing config file → check if volume mount has the config
• Wrong permissions → chown -R the data directory
• Dependency not ready → start dependency first, wait, then start this container

Fix 2: Missing Restart Policy

The most common uptime killer. Fix for ALL containers at once:

# Fix a single container
sudo podman update --restart unless-stopped CONTAINER_NAME

# Fix ALL containers that have no restart policy
for c in $(sudo podman ps -a --format "{{.Names}}"); do
  policy=$(sudo podman inspect "$c" --format "{{.HostConfig.RestartPolicy.Name}}")
  if [ "$policy" = "no" ] || [ -z "$policy" ]; then
    echo "Fixing restart policy for: $c"
    sudo podman update --restart unless-stopped "$c"
  fi
done

Also update the Rust source so new installs get it right:

• Check core/archipelago/src/api/rpc/package.rs get_app_config() for the app
• Ensure --restart flag is in the podman run args

Fix 3: Port Mapping Issues

Port conflict (address already in use)

# Find what's using the port
sudo ss -tlnp | grep :PORT_NUMBER

# If it's another container, either change one's port or stop the conflicting one
sudo podman stop CONFLICTING_CONTAINER

# If it's a host process
sudo kill PID  # or stop the service

Port not mapped (container running but port unreachable)

# Check current port mappings
sudo podman port CONTAINER_NAME

# Can't add ports to running container — must recreate
sudo podman stop CONTAINER_NAME
sudo podman rm CONTAINER_NAME
# Recreate with correct -p flags (use the Rust install flow or manual podman run)

Nginx proxy missing or wrong

Read and fix the nginx config:

• HTTP: image-recipe/configs/nginx-archipelago.conf
• HTTPS: image-recipe/configs/snippets/archipelago-https-app-proxies.conf

Add a location block:

location /app/APP_ID/ {
    proxy_pass http://127.0.0.1:HOST_PORT/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    # Hide X-Frame-Options so it works in our iframe
    proxy_hide_header X-Frame-Options;
    proxy_hide_header Content-Security-Policy;
}

After editing nginx config, deploy and reload:

# On server
sudo nginx -t && sudo systemctl reload nginx

Frontend routing missing

Edit neode-ui/src/stores/appLauncher.ts:

• Add entry to PORT_TO_APP_ID map
• If app blocks iframes, add port to the new-tab list in resolveAppIdFromUrl()

Fix 4: Network Issues

Container not on archy-net (can't resolve other containers)

# Connect to archy-net without recreating
sudo podman network connect archy-net CONTAINER_NAME

# Verify
sudo podman inspect CONTAINER_NAME --format "{{.NetworkSettings.Networks}}"

archy-net doesn't exist

sudo podman network create archy-net
# Then reconnect all containers that need it

DNS not working inside container

# Test DNS from inside container
sudo podman exec CONTAINER_NAME nslookup bitcoin-knots 2>/dev/null || \
sudo podman exec CONTAINER_NAME ping -c1 bitcoin-knots

# If DNS fails, recreate container with explicit DNS
# Add --dns 1.1.1.1 to the podman run command

Fix 5: Health Check Issues

Add missing health check to running container

Can't add to running container — must recreate with health check flags:

# Example for a web app
sudo podman run ... \
  --health-cmd "curl -f http://localhost:PORT/health || exit 1" \
  --health-interval 30s \
  --health-timeout 5s \
  --health-retries 3 \
  --health-start-period 60s \
  IMAGE

Fix unhealthy container

# See what the health check is actually running
sudo podman inspect CONTAINER_NAME --format "{{.Config.Healthcheck.Test}}"

# Run the health check manually to see the error
sudo podman exec CONTAINER_NAME HEALTH_CHECK_COMMAND

# Common fixes:
# - curl not installed in container → use wget or nc instead
# - Wrong port in health check → fix the check command
# - App takes too long to start → increase --health-start-period

Fix 6: Permission/Capability Issues

# Check what capabilities container has
sudo podman inspect CONTAINER_NAME --format "{{.HostConfig.CapAdd}}"

# If missing required caps, must recreate with correct --cap-add flags
# Refer to the capability reference in /podman-doctor references

# Fix data directory permissions
sudo chown -R 1000:1000 /var/lib/archipelago/APP_NAME/

Fix 7: Full Config Consistency Fix

When port map is inconsistent across layers, fix ALL layers:

1. Decide the correct port (usually what's in package.rs)
2. Fix Podman: recreate container with correct -p flags
3. Fix Nginx: update location block's proxy_pass port
4. Fix Frontend: update PORT_TO_APP_ID in appLauncher.ts
5. Deploy: ./scripts/deploy-to-target.sh --live
6. Verify: curl -I http://192.168.1.228/app/APP_ID/

After Fixing

Always verify the fix:

# Container running?
sudo podman ps --filter name=CONTAINER_NAME

# Port reachable?
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:PORT/

# Via nginx proxy?
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1/app/APP_ID/

# Health check passing?
sudo podman inspect CONTAINER_NAME --format "{{.State.Health.Status}}"

Run /podman-doctor again to confirm all issues are resolved.