docs: v1.2.0 changelog and operations runbook
- DOC-01: CHANGELOG.md for v1.2.0 — crash fixes, DWN sync perf, test suite, did:dht planning, DWN protocols, deploy hardening, ISO improvements - DOC-04: operations-runbook.md — 17 sections covering health checks, container management, federation, Tor, backups, updates, diagnostics, emergency recovery, and test execution Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dorian
parent
f9272650c4
commit
ec8eedca15
65
CHANGELOG.md
65
CHANGELOG.md
@ -7,6 +7,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [1.2.0] - 2026-03-14
|
||||
|
||||
### Fixed
|
||||
|
||||
#### Crash Loop Resolution
|
||||
- Identified and fixed UFW blocking Podman subnet DNS resolution on .228
|
||||
- Fixed archy-nbxplorer, btcpay-server, mempool-web, immich crash loops (3500+ restarts)
|
||||
- All 32 containers stable with zero crash loops after fix
|
||||
|
||||
#### DWN Sync Performance
|
||||
- Made `dwn.sync` endpoint non-blocking (background task with polling)
|
||||
- Added 90-second overall sync timeout to prevent indefinite blocking
|
||||
- Deduplicated peer onion addresses before syncing
|
||||
- Batched message pushes (50/batch) instead of one-at-a-time over Tor
|
||||
- Fixed HTTP handler to process all messages in batch (was only first)
|
||||
|
||||
#### Backup Reliability
|
||||
- Increased backup.create rate limit from 3/600 to 10/600 for testing
|
||||
- Increased backup.restore rate limit from 2/600 to 5/600
|
||||
|
||||
#### Deploy Script
|
||||
- Added `set -eo pipefail` for pipe error detection
|
||||
- Fixed duplicate variable initialization
|
||||
- Fail on missing binary in --both path (was silently ignored)
|
||||
- Added post-deploy health check on .198
|
||||
|
||||
### Added
|
||||
|
||||
#### Cross-Node Test Suite
|
||||
- US-08: DWN sync tests — 50/50 pass (register, write, sync, query bidirectional)
|
||||
- US-10: Backup/restore tests — 80/80 pass (create, list, verify, delete × 10 × 2 nodes)
|
||||
- US-15: Boot recovery tests — .228 9/9 pass (32/32 containers survive 3 reboots)
|
||||
- `trigger_sync_and_wait()` helper for polling async DWN sync
|
||||
|
||||
#### did:dht Integration Planning
|
||||
- Architecture document: `docs/did-dht-integration.md`
|
||||
- BEP-44 mutable DHT items, DNS packet encoding, z-base-32 identifiers
|
||||
- Publication/resolution flows, `mainline` crate selection, security notes
|
||||
|
||||
#### DWN Protocol Definitions
|
||||
- 4 Archipelago DWN protocols documented in `docs/dwn-protocols.md`
|
||||
- Node Identity Announcements (public)
|
||||
- File Sharing Catalog (public)
|
||||
- Federation State (private)
|
||||
- App Deployment Requests (private)
|
||||
- Auto-registration of all 4 protocols on backend startup
|
||||
|
||||
#### Deploy Script Improvements
|
||||
- `--dry-run` flag shows what would be deployed without executing
|
||||
- Works with all other flags (--live, --both, --frontend-only)
|
||||
|
||||
#### ISO/First-Boot Improvements
|
||||
- Auto-create swap file on first boot (50% RAM, min 2GB, max 8GB)
|
||||
- Tiered container startup ordering in first-boot script
|
||||
- Tier 1: Databases, Tier 2: Core Services (5s delay), Tier 3: Applications (5s delay)
|
||||
|
||||
### Security
|
||||
|
||||
#### Backend Hardening
|
||||
- Rate limiting on federation endpoints (join 5/60s, invite 10/300s)
|
||||
- DWN message data size limit (10MB max)
|
||||
- Container security: cap-drop ALL, no-new-privileges, per-app memory limits
|
||||
- Input validation: path traversal protection on identity/DID endpoints
|
||||
- Error sanitization: internal paths stripped from error messages
|
||||
|
||||
## [1.1.0] - 2026-03-13
|
||||
|
||||
### Added
|
||||
|
||||
364
docs/operations-runbook.md
Normal file
364
docs/operations-runbook.md
Normal file
@ -0,0 +1,364 @@
|
||||
# Archipelago Operations Runbook
|
||||
|
||||
Quick reference for common operational tasks on Archipelago nodes.
|
||||
|
||||
**Primary node**: `192.168.1.228` (Arch 1)
|
||||
**Secondary node**: `192.168.1.198` (Arch 2)
|
||||
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
|
||||
**Sudo**: `echo 'EwPDR8q45l0Upx@' | sudo -S {command}`
|
||||
|
||||
---
|
||||
|
||||
## 1. Check Node Health
|
||||
|
||||
```bash
|
||||
# Quick health check (from any machine)
|
||||
curl http://192.168.1.228/health # Should return "OK"
|
||||
curl http://192.168.1.198/health
|
||||
|
||||
# Detailed system stats via RPC
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-d '{"method":"system.stats"}' \
|
||||
http://192.168.1.228:5678/rpc/v1
|
||||
|
||||
# Check services
|
||||
ssh archipelago@192.168.1.228
|
||||
sudo systemctl status archipelago # Backend service
|
||||
sudo systemctl status nginx # Web server
|
||||
sudo systemctl status tor # Tor hidden services
|
||||
```
|
||||
|
||||
## 2. Check Container Status
|
||||
|
||||
```bash
|
||||
# List all containers
|
||||
sudo podman ps -a
|
||||
|
||||
# Running count
|
||||
sudo podman ps --format '{{.Names}}' | wc -l
|
||||
|
||||
# Find exited/crashed containers
|
||||
sudo podman ps -a --filter status=exited
|
||||
|
||||
# Container logs
|
||||
sudo podman logs {container-name} --tail 50
|
||||
|
||||
# Container resource usage
|
||||
sudo podman stats --no-stream
|
||||
```
|
||||
|
||||
## 3. Fix Crashed Containers
|
||||
|
||||
```bash
|
||||
# Restart a specific container
|
||||
sudo podman restart {container-name}
|
||||
|
||||
# If container won't start, check logs first
|
||||
sudo podman logs {container-name} --tail 100
|
||||
|
||||
# Remove and recreate (last resort)
|
||||
sudo podman rm -f {container-name}
|
||||
# Then redeploy with: ./scripts/deploy-to-target.sh --live
|
||||
|
||||
# The health monitor auto-restarts containers every 60s
|
||||
# Check its status:
|
||||
sudo journalctl -u archipelago --grep="health_monitor" --no-pager -n 20
|
||||
```
|
||||
|
||||
## 4. Add/Remove Federation Peers
|
||||
|
||||
```bash
|
||||
# Generate invite code (on inviting node)
|
||||
# Via UI: Federation page > Generate Invite
|
||||
# Via RPC:
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"federation.invite"}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Join federation (on joining node)
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"federation.join","params":{"invite_code":"{code}"}}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# List peers
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-d '{"method":"federation.list-nodes"}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Remove a peer
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"federation.remove-node","params":{"did":"{peer-did}"}}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
```
|
||||
|
||||
## 5. Rotate Tor Address
|
||||
|
||||
```bash
|
||||
# Delete current hidden service keys
|
||||
sudo rm -rf /var/lib/tor/hidden_service/
|
||||
sudo systemctl restart tor
|
||||
|
||||
# Wait for new hostname
|
||||
sleep 15
|
||||
sudo cat /var/lib/tor/hidden_service/hostname
|
||||
|
||||
# The backend picks up the new address automatically (30s refresh)
|
||||
# Federation peers need to re-discover via sync
|
||||
```
|
||||
|
||||
## 6. Create/Restore Backups
|
||||
|
||||
```bash
|
||||
# Create encrypted backup (via RPC)
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"backup.create","params":{"passphrase":"your-passphrase","description":"manual backup"}}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# List backups
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"backup.list"}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Verify backup integrity
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"backup.verify","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Restore (warning: overwrites current identity/data)
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"backup.restore","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Backup files stored at: /var/lib/archipelago/backups/
|
||||
```
|
||||
|
||||
## 7. Update the Node
|
||||
|
||||
```bash
|
||||
# From development machine:
|
||||
./scripts/deploy-to-target.sh --live # Deploy to .228
|
||||
./scripts/deploy-to-target.sh --both # Deploy to both nodes
|
||||
./scripts/deploy-to-target.sh --dry-run --live # Preview changes
|
||||
|
||||
# The deploy script:
|
||||
# 1. Syncs code to target
|
||||
# 2. Builds frontend (vue-tsc + vite)
|
||||
# 3. Builds backend (cargo build --release)
|
||||
# 4. Deploys binary, frontend, configs
|
||||
# 5. Restarts services
|
||||
# 6. Verifies health
|
||||
```
|
||||
|
||||
## 8. Diagnose High CPU
|
||||
|
||||
```bash
|
||||
# Check system load
|
||||
uptime
|
||||
|
||||
# Find CPU-heavy processes
|
||||
top -b -n 1 | head -15
|
||||
|
||||
# Check container CPU usage
|
||||
sudo podman stats --no-stream --format '{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
|
||||
|
||||
# Common causes:
|
||||
# - Bitcoin IBD (initial block download): normal, takes days
|
||||
# - Container crash loops: check `sudo podman ps -a --filter status=exited`
|
||||
# - mempool-electrs indexing: normal after Bitcoin sync
|
||||
```
|
||||
|
||||
## 9. Diagnose High Memory
|
||||
|
||||
```bash
|
||||
# Check memory
|
||||
free -h
|
||||
|
||||
# Check swap usage
|
||||
swapon --show
|
||||
|
||||
# Per-container memory
|
||||
sudo podman stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
|
||||
|
||||
# Check for OOM kills
|
||||
dmesg --level=err,crit | grep -i oom
|
||||
|
||||
# Add swap if missing
|
||||
sudo fallocate -l 4G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
|
||||
```
|
||||
|
||||
## 10. Diagnose Disk Space
|
||||
|
||||
```bash
|
||||
# Disk usage overview
|
||||
df -h /
|
||||
|
||||
# Find large directories
|
||||
sudo du -h --max-depth=2 /var/lib/archipelago/ | sort -rh | head -20
|
||||
|
||||
# Container image sizes
|
||||
sudo podman images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}'
|
||||
|
||||
# Clean unused images
|
||||
sudo podman image prune -a
|
||||
|
||||
# Clean old journal logs
|
||||
sudo journalctl --vacuum-size=500M
|
||||
```
|
||||
|
||||
## 11. Check Tor Connectivity
|
||||
|
||||
```bash
|
||||
# Tor service status
|
||||
sudo systemctl status tor
|
||||
|
||||
# Get onion address
|
||||
sudo cat /var/lib/tor/hidden_service/hostname
|
||||
|
||||
# Test self-connection via Tor
|
||||
curl --socks5-hostname 127.0.0.1:9050 http://$(sudo cat /var/lib/tor/hidden_service/hostname)/health
|
||||
|
||||
# Test cross-node Tor
|
||||
curl --socks5-hostname 127.0.0.1:9050 http://{peer-onion}/health
|
||||
```
|
||||
|
||||
## 12. Check DWN Sync
|
||||
|
||||
```bash
|
||||
# DWN status (via RPC, needs auth)
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"dwn.status"}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Trigger manual sync
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-H "Cookie: session={session}; csrf_token={csrf}" \
|
||||
-H "X-CSRF-Token: {csrf}" \
|
||||
-d '{"method":"dwn.sync"}' \
|
||||
http://localhost:5678/rpc/v1
|
||||
|
||||
# Check message count
|
||||
ls /var/lib/archipelago/dwn/messages/ | wc -l
|
||||
```
|
||||
|
||||
## 13. Restart Services
|
||||
|
||||
```bash
|
||||
# Restart backend only
|
||||
sudo systemctl restart archipelago
|
||||
|
||||
# Restart nginx
|
||||
sudo systemctl restart nginx
|
||||
|
||||
# Restart Tor
|
||||
sudo systemctl restart tor
|
||||
|
||||
# Full service restart (backend + nginx)
|
||||
sudo systemctl restart archipelago nginx
|
||||
|
||||
# Reboot (containers auto-recover via restart policy + health monitor)
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
## 14. View Logs
|
||||
|
||||
```bash
|
||||
# Backend logs
|
||||
sudo journalctl -u archipelago --no-pager -n 100
|
||||
|
||||
# Follow logs in real time
|
||||
sudo journalctl -u archipelago -f
|
||||
|
||||
# Nginx access log
|
||||
sudo tail -f /var/log/nginx/access.log
|
||||
|
||||
# Nginx error log
|
||||
sudo tail -f /var/log/nginx/error.log
|
||||
|
||||
# Container logs
|
||||
sudo podman logs {container-name} --tail 50 -f
|
||||
```
|
||||
|
||||
## 15. Network Diagnostics
|
||||
|
||||
```bash
|
||||
# Check listening ports
|
||||
sudo ss -tlnp
|
||||
|
||||
# Check firewall rules
|
||||
sudo ufw status verbose
|
||||
|
||||
# Required ports:
|
||||
# 22 - SSH
|
||||
# 80 - HTTP (nginx)
|
||||
# 443 - HTTPS (nginx)
|
||||
# 5678 - Backend API (localhost only, proxied by nginx)
|
||||
# 8332 - Bitcoin RPC (container network only)
|
||||
# 9050 - Tor SOCKS proxy (localhost only)
|
||||
|
||||
# If ports are blocked after reboot, re-add UFW rules:
|
||||
sudo ufw allow ssh
|
||||
sudo ufw allow 80/tcp
|
||||
sudo ufw allow 443/tcp
|
||||
sudo ufw allow from 10.88.0.0/16 # Podman container subnet
|
||||
sudo ufw allow from 10.89.0.0/16 # Podman container subnet
|
||||
```
|
||||
|
||||
## 16. Emergency: Node Won't Boot
|
||||
|
||||
If a node responds to ping but SSH/HTTP are down:
|
||||
|
||||
1. **Check UFW**: After reboot, UFW may block all ports
|
||||
```bash
|
||||
# If you have console access:
|
||||
sudo ufw allow ssh
|
||||
sudo ufw allow 80/tcp
|
||||
sudo ufw allow 443/tcp
|
||||
sudo ufw reload
|
||||
```
|
||||
|
||||
2. **Check services**: SSH or nginx may not have started
|
||||
```bash
|
||||
sudo systemctl start ssh
|
||||
sudo systemctl start nginx
|
||||
sudo systemctl start archipelago
|
||||
```
|
||||
|
||||
3. **Check disk**: If root filesystem is full, services won't start
|
||||
```bash
|
||||
df -h /
|
||||
sudo journalctl --vacuum-size=200M
|
||||
sudo podman image prune -a
|
||||
```
|
||||
|
||||
## 17. Run Cross-Node Tests
|
||||
|
||||
```bash
|
||||
# Full test suite (all features, 10 iterations)
|
||||
./scripts/test-cross-node.sh --iterations 10
|
||||
|
||||
# Skip reboot tests
|
||||
./scripts/test-cross-node.sh --iterations 10 --skip-reboot
|
||||
|
||||
# Reboot survival test (single node)
|
||||
./scripts/test-reboot-survival.sh --node 192.168.1.228 --iterations 3
|
||||
```
|
||||
@ -357,13 +357,13 @@ Every test must pass **10 consecutive times** from BOTH .228→.198 AND .198→.
|
||||
|
||||
### Sprint 18: Documentation Update
|
||||
|
||||
- [ ] **DOC-01** — Update CHANGELOG.md for v1.2.0. Document all changes from this hardening cycle: crash loop fixes, cross-node testing, did:dht, DWN protocols, VCs, reboot hardening, memory/swap fixes. **Acceptance**: CHANGELOG updated with all changes.
|
||||
- [x] **DOC-01** — Updated CHANGELOG.md with v1.2.0 release. Covers: crash loop fixes, DWN sync performance, backup reliability, deploy script hardening, cross-node test suite (DWN/backup/boot recovery), did:dht architecture, DWN protocol definitions, deploy --dry-run, ISO swap/tiered startup, security hardening.
|
||||
|
||||
- [ ] **DOC-02** — Update architecture.md for current state. The current doc references StartOS, Docker, macOS. Update to reflect: Debian 12, Podman, multi-node federation, did:dht, DWN protocols. **Acceptance**: Architecture doc matches actual system.
|
||||
|
||||
- [ ] **DOC-03** — Update current-state.md. Remove references to StartOS dependencies (already removed). Document actual current state: pure Archipelago backend, Podman, 33+ containers, 2-node federation. **Acceptance**: current-state.md reflects reality.
|
||||
|
||||
- [ ] **DOC-04** — Create operations runbook. `docs/operations-runbook.md` covering: how to check node health, how to fix crashed containers, how to add/remove federation peers, how to rotate Tor address, how to create/restore backups, how to update, how to diagnose high CPU/memory. **Acceptance**: Runbook covers top 20 operational scenarios.
|
||||
- [x] **DOC-04** — Created `docs/operations-runbook.md` with 17 sections: health checks, container status, fix crashes, federation peers, Tor rotation, backup/restore, updates, CPU/memory/disk diagnostics, Tor connectivity, DWN sync, service restart, log viewing, network diagnostics, emergency boot recovery, cross-node tests.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user