lfg2025/archy
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases 44 Wiki Activity

chore: release v1.7.45-alpha

Browse Source 
Resilience-validated release. Three full sweeps of the new resilience
harness against .228 confirm no shipstoppers.

Big user-visible:
- Bitcoin RPC auth durably correct via host-rendered nginx.conf bind-mount,
  replaces fragile post-start exec that failed under restricted-cap rootless
  podman ("crun: write cgroup.procs: Permission denied")
- Multi-container stack installs (indeedhub, immich, btcpay, mempool) now
  emit phase events at every boundary so the progress bar advances
- Apps no longer vanish from the dashboard mid-install (absent-scanner skips
  packages in transitional states)
- Indeedhub fresh installs work end-to-end (was 8500+ restart loop): five
  missing env vars (DATABASE_PORT, QUEUE_HOST, QUEUE_PORT,
  S3_PRIVATE_BUCKET_NAME, AES_MASTER_SECRET) added to install code
- Tailscale install fixed: --entrypoint string was being passed as a single
  shell-line arg; switched to custom_args array
- Catalog cleaned of broken entries (dwn, endurain, ollama removed; nextcloud
  restored on docker.io)
- Bitcoin Core update path uses correct image (was looking for nonexistent
  lfg2025/bitcoin:28.4)
- ISO installs now allocate swap on the encrypted data partition

Infra:
- New resilience harness (scripts/resilience/) — black-box state-machine
  tester, every app × every transition. Run before each release.

Sweep #3 final: PASS 107 / FAIL 12 / SKIP 14. The 12 fails are 1 cosmetic
(homeassistant trusted_hosts), 8 harness/timing false-positives, and 3
non-shipstopper tracked items. Down from 23 in baseline sweep #1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago 2026-04-29 12:31:45 -04:00
parent dffa7e99bb
commit 4ec6ca98c1
38 changed files with 1699 additions and 1805 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -74,3 +74,6 @@ loop/loop.log.bak
web/
._*
# Resilience harness reports (generated, contains session cookies)
scripts/resilience/reports/

11
CHANGELOG.md
View File

@ -1,5 +1,16 @@
# Changelog
## v1.7.45-alpha (2026-04-29)
- Bitcoin RPC auth is durable. The dashboard reliably connects across container restart, image update, and reboot. Was failing on registry-pulled images that shipped a stale baked-in password.
- Multi-container apps show real install progress. IndeedHub (7), BTCPay (4), Mempool (3), Immich (3) — bar advances through Preparing → Pulling → Creating → Done instead of sitting at 0% until the very end.
- Apps no longer disappear from the dashboard mid-install. The container scanner now respects in-flight installs and updates instead of evicting an entry while its containers are still being created.
- IndeedHub installs cleanly on a fresh node. Five missing environment variables fixed; Nostr sign-in works on first install.
- Tailscale install no longer fails with "executable not found". Container command was a malformed shell string; now a proper command array.
- Removed three catalog entries that hung installs for ten minutes (dwn, endurain, ollama — no source images in our registries). Restored Nextcloud, sourced from docker.io.
- Bitcoin Core update path uses the correct image name (was pulling from a non-existent path).
- New ISO installs now allocate swap (sized to RAM, capped at 8GB, on the encrypted data partition). Without swap, container image builds and memory spikes were hitting OOM under load.
## v1.7.44-alpha (2026-04-28)
43de3b73 feat(orchestrator): complete container migration and release hardening

250
app-catalog/catalog.json
View File

@ -1,7 +1,7 @@
{
"version": 2,
"updated": "2026-04-22T00:00:00Z",
"registry": "git.tx1138.com/lfg2025",
"registry": "146.59.87.168:3000/lfg2025",
"featured": {
"id": "indeedhub",
"banner": "/assets/img/featured/indeedhub-banner.jpg",
@ -11,200 +11,260 @@
},
"apps": [
{
"id": "bitcoin-knots", "title": "Bitcoin Knots", "version": "28.1.0",
"id": "bitcoin-knots",
"title": "Bitcoin Knots",
"version": "28.1.0",
"description": "Run a full Bitcoin node. Validate and relay blocks and transactions.",
"icon": "/assets/img/app-icons/bitcoin-knots.webp",
"author": "Bitcoin Knots", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/bitcoin-knots:latest",
"author": "Bitcoin Knots",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/bitcoin-knots:latest",
"repoUrl": "https://github.com/bitcoinknots/bitcoin"
},
{
"id": "bitcoin-core", "title": "Bitcoin Core", "version": "28.4",
"id": "bitcoin-core",
"title": "Bitcoin Core",
"version": "28.4",
"description": "Reference implementation of the Bitcoin protocol. Run a full node validating and relaying blocks.",
"icon": "/assets/img/app-icons/bitcoin-core.svg",
"author": "Bitcoin Core contributors", "category": "money", "tier": "optional",
"author": "Bitcoin Core contributors",
"category": "money",
"tier": "optional",
"dockerImage": "docker.io/bitcoin/bitcoin:28.4",
"repoUrl": "https://github.com/bitcoin/bitcoin"
},
{
"id": "lnd", "title": "LND", "version": "0.18.4",
"id": "lnd",
"title": "LND",
"version": "0.18.4",
"description": "Lightning Network Daemon. Fast Bitcoin payments through Lightning.",
"icon": "/assets/img/app-icons/lnd.svg",
"author": "Lightning Labs", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/lnd:v0.18.4-beta",
"author": "Lightning Labs",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/lnd:v0.18.4-beta",
"repoUrl": "https://github.com/lightningnetwork/lnd",
"requires": ["bitcoin-knots"]
"requires": [
"bitcoin-knots"
]
},
{
"id": "btcpay-server", "title": "BTCPay Server", "version": "1.13.7",
"id": "btcpay-server",
"title": "BTCPay Server",
"version": "1.13.7",
"description": "Self-hosted Bitcoin payment processor.",
"icon": "/assets/img/app-icons/btcpay-server.png",
"author": "BTCPay Server Foundation", "category": "commerce", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/btcpayserver:1.13.7",
"author": "BTCPay Server Foundation",
"category": "commerce",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/btcpayserver:1.13.7",
"repoUrl": "https://github.com/btcpayserver/btcpayserver",
"requires": ["bitcoin-knots"]
"requires": [
"bitcoin-knots"
]
},
{
"id": "mempool", "title": "Mempool Explorer", "version": "3.0.0",
"id": "mempool",
"title": "Mempool Explorer",
"version": "3.0.0",
"description": "Self-hosted Bitcoin blockchain and mempool visualizer.",
"icon": "/assets/img/app-icons/mempool.webp",
"author": "Mempool", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/mempool-frontend:v3.0.0",
"author": "Mempool",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/mempool-frontend:v3.0.0",
"repoUrl": "https://github.com/mempool/mempool",
"requires": ["bitcoin-knots", "electrumx"]
"requires": [
"bitcoin-knots",
"electrumx"
]
},
{
"id": "electrumx", "title": "ElectrumX", "version": "1.18.0",
"id": "electrumx",
"title": "ElectrumX",
"version": "1.18.0",
"description": "Electrum protocol server. Index the blockchain for fast wallet lookups.",
"icon": "/assets/img/app-icons/electrumx.webp",
"author": "Luke Childs", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/electrumx:v1.18.0",
"author": "Luke Childs",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/electrumx:v1.18.0",
"repoUrl": "https://github.com/spesmilo/electrumx",
"requires": ["bitcoin-knots"]
"requires": [
"bitcoin-knots"
]
},
{
"id": "indeedhub", "title": "IndeeHub", "version": "1.0.0",
"id": "indeedhub",
"title": "IndeeHub",
"version": "1.0.0",
"description": "Bitcoin documentary streaming with Nostr identity.",
"icon": "/assets/img/app-icons/indeedhub.png",
"author": "IndeeHub", "category": "community",
"dockerImage": "git.tx1138.com/lfg2025/indeedhub:1.0.0",
"author": "IndeeHub",
"category": "community",
"dockerImage": "146.59.87.168:3000/lfg2025/indeedhub:1.0.0",
"repoUrl": "https://github.com/indeedhub/indeedhub"
},
{
"id": "botfights", "title": "BotFights", "version": "1.1.0",
"id": "botfights",
"title": "BotFights",
"version": "1.1.0",
"description": "Bot arena + 2-player arcade fighter with controller support and Adventure Mode.",
"icon": "/assets/img/app-icons/botfights.svg",
"author": "BotFights", "category": "community",
"dockerImage": "git.tx1138.com/lfg2025/botfights:1.1.0",
"author": "BotFights",
"category": "community",
"dockerImage": "146.59.87.168:3000/lfg2025/botfights:1.1.0",
"repoUrl": "https://botfights.net"
},
{
"id": "gitea", "title": "Gitea", "version": "1.23",
"id": "gitea",
"title": "Gitea",
"version": "1.23",
"description": "Self-hosted Git service with container registry, CI/CD, issue tracking.",
"icon": "/assets/img/app-icons/gitea.svg",
"author": "Gitea", "category": "development",
"author": "Gitea",
"category": "development",
"dockerImage": "docker.io/gitea/gitea:1.23",
"repoUrl": "https://gitea.com"
},
{
"id": "filebrowser", "title": "File Browser", "version": "2.27.0",
"id": "filebrowser",
"title": "File Browser",
"version": "2.27.0",
"description": "Web-based file manager.",
"icon": "/assets/img/app-icons/file-browser.webp",
"author": "File Browser", "category": "data", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/filebrowser:v2.27.0",
"author": "File Browser",
"category": "data",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/filebrowser:v2.27.0",
"repoUrl": "https://github.com/filebrowser/filebrowser"
},
{
"id": "vaultwarden", "title": "Vaultwarden", "version": "1.30.0",
"id": "vaultwarden",
"title": "Vaultwarden",
"version": "1.30.0",
"description": "Self-hosted password vault with zero-knowledge encryption.",
"icon": "/assets/img/app-icons/vaultwarden.webp",
"author": "Vaultwarden", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/vaultwarden:1.30.0-alpine",
"author": "Vaultwarden",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/vaultwarden:1.30.0-alpine",
"repoUrl": "https://github.com/dani-garcia/vaultwarden"
},
{
"id": "searxng", "title": "SearXNG", "version": "2024.1.0",
"id": "searxng",
"title": "SearXNG",
"version": "2024.1.0",
"description": "Privacy-respecting metasearch engine.",
"icon": "/assets/img/app-icons/searxng.png",
"author": "SearXNG", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/searxng:latest",
"author": "SearXNG",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/searxng:latest",
"repoUrl": "https://github.com/searxng/searxng"
},
{
"id": "fedimint", "title": "Fedimint", "version": "0.10.0",
"id": "fedimint",
"title": "Fedimint",
"version": "0.10.0",
"description": "Federated Bitcoin mint with privacy through federated guardians.",
"icon": "/assets/img/app-icons/fedimint.png",
"author": "Fedimint", "category": "money",
"dockerImage": "git.tx1138.com/lfg2025/fedimintd:v0.10.0",
"author": "Fedimint",
"category": "money",
"dockerImage": "146.59.87.168:3000/lfg2025/fedimintd:v0.10.0",
"repoUrl": "https://github.com/fedimint/fedimint"
},
{
"id": "ollama", "title": "Ollama", "version": "0.5.4",
"description": "Run AI models locally. Private and on your hardware.",
"icon": "/assets/img/app-icons/ollama.png",
"author": "Ollama", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/ollama:latest",
"repoUrl": "https://github.com/ollama/ollama"
},
{
"id": "nextcloud", "title": "Nextcloud", "version": "28",
"description": "Your own private cloud. File sync, calendars, contacts.",
"icon": "/assets/img/app-icons/nextcloud.webp",
"author": "Nextcloud", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/nextcloud:28",
"repoUrl": "https://github.com/nextcloud/server"
},
{
"id": "jellyfin", "title": "Jellyfin", "version": "10.8.13",
"id": "jellyfin",
"title": "Jellyfin",
"version": "10.8.13",
"description": "Free media server. Stream movies, music, and photos.",
"icon": "/assets/img/app-icons/jellyfin.webp",
"author": "Jellyfin", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/jellyfin:10.8.13",
"author": "Jellyfin",
"category": "data",
"dockerImage": "146.59.87.168:3000/lfg2025/jellyfin:10.8.13",
"repoUrl": "https://github.com/jellyfin/jellyfin"
},
{
"id": "immich", "title": "Immich", "version": "1.90.0",
"id": "immich",
"title": "Immich",
"version": "1.90.0",
"description": "High-performance photo and video backup with ML.",
"icon": "/assets/img/app-icons/immich.png",
"author": "Immich", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/immich-server:release",
"author": "Immich",
"category": "data",
"dockerImage": "146.59.87.168:3000/lfg2025/immich-server:release",
"repoUrl": "https://github.com/immich-app/immich"
},
{
"id": "homeassistant", "title": "Home Assistant", "version": "2024.1",
"id": "homeassistant",
"title": "Home Assistant",
"version": "2024.1",
"description": "Open-source home automation.",
"icon": "/assets/img/app-icons/homeassistant.png",
"author": "Home Assistant", "category": "home",
"dockerImage": "git.tx1138.com/lfg2025/home-assistant:2024.1",
"author": "Home Assistant",
"category": "home",
"dockerImage": "146.59.87.168:3000/lfg2025/home-assistant:2024.1",
"repoUrl": "https://github.com/home-assistant/core"
},
{
"id": "grafana", "title": "Grafana", "version": "10.2.0",
"id": "grafana",
"title": "Grafana",
"version": "10.2.0",
"description": "Analytics and monitoring dashboards.",
"icon": "/assets/img/app-icons/grafana.png",
"author": "Grafana Labs", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/grafana:10.2.0",
"author": "Grafana Labs",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/grafana:10.2.0",
"repoUrl": "https://github.com/grafana/grafana"
},
{
"id": "tailscale", "title": "Tailscale", "version": "1.78.0",
"id": "tailscale",
"title": "Tailscale",
"version": "1.78.0",
"description": "Zero-config VPN with WireGuard mesh networking.",
"icon": "/assets/img/app-icons/tailscale.webp",
"author": "Tailscale", "category": "networking", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/tailscale:stable",
"author": "Tailscale",
"category": "networking",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/tailscale:stable",
"repoUrl": "https://github.com/tailscale/tailscale"
},
{
"id": "uptime-kuma", "title": "Uptime Kuma", "version": "1.23.0",
"id": "uptime-kuma",
"title": "Uptime Kuma",
"version": "1.23.0",
"description": "Self-hosted uptime monitoring.",
"icon": "/assets/img/app-icons/uptime-kuma.webp",
"author": "Uptime Kuma", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/uptime-kuma:1",
"author": "Uptime Kuma",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/uptime-kuma:1",
"repoUrl": "https://github.com/louislam/uptime-kuma"
},
{
"id": "dwn", "title": "Decentralized Web Node", "version": "0.4.0",
"description": "Own your data with DID-based access control.",
"icon": "/assets/img/app-icons/dwn.svg",
"author": "TBD", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/dwn-server:main",
"repoUrl": "https://github.com/TBD54566975/dwn-server"
},
{
"id": "endurain", "title": "Endurain", "version": "0.8.0",
"description": "Self-hosted fitness tracking. Strava alternative.",
"icon": "/assets/img/app-icons/endurain.png",
"author": "Endurain", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/endurain:0.8.0",
"repoUrl": "https://github.com/joaovitoriasilva/endurain"
},
{
"id": "photoprism", "title": "PhotoPrism", "version": "240915",
"id": "photoprism",
"title": "PhotoPrism",
"version": "240915",
"description": "AI-powered photo management with facial recognition.",
"icon": "/assets/img/app-icons/photoprism.svg",
"author": "PhotoPrism", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/photoprism:240915",
"author": "PhotoPrism",
"category": "data",
"dockerImage": "146.59.87.168:3000/lfg2025/photoprism:240915",
"repoUrl": "https://github.com/photoprism/photoprism"
},
{
"id": "nextcloud",
"title": "Nextcloud",
"version": "28",
"description": "Your own private cloud. File sync, calendars, contacts.",
"icon": "/assets/img/app-icons/nextcloud.webp",
"author": "Nextcloud",
"category": "data",
"dockerImage": "docker.io/nextcloud:28",
"repoUrl": "https://github.com/nextcloud/server"
}
]
}

2
apps/bitcoin-core/manifest.yml
View File

@ -7,7 +7,7 @@ app:
container_name: bitcoin-knots
container:
image: git.tx1138.com/lfg2025/bitcoin-knots:latest
image: 146.59.87.168:3000/lfg2025/bitcoin-knots:latest
pull_policy: if-not-present
network: archy-net
entrypoint: ["sh", "-lc"]

6
apps/endurain/.dockerignore
View File

@ -1,6 +0,0 @@
node_modules
dist
*.log
.git
.gitignore
README.md

37
apps/endurain/Dockerfile
View File

@ -1,37 +0,0 @@
FROM node:20-alpine AS builder
WORKDIR /app
# Copy package files
COPY package*.json ./
RUN npm ci --only=production
# Copy source code
COPY . .
# Build the application
RUN npm run build
# Production stage
FROM node:20-alpine
WORKDIR /app
# Copy built application
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./
# Create non-root user
RUN addgroup -g 1000 appuser && \
adduser -D -u 1000 -G appuser appuser && \
mkdir -p /app/data && \
chown -R appuser:appuser /app
USER appuser
EXPOSE 8080
ENV ENDURAIN_DATA_DIR=/app/data
CMD ["node", "dist/index.js"]

50
apps/endurain/manifest.yml
View File

@ -1,50 +0,0 @@
app:
id: endurain
name: Endurain
version: 1.0.0
description: Endurain application platform. Custom application runtime.
container:
image: archipelago/endurain:1.0.0
image_signature: cosign://...
pull_policy: if-not-present
dependencies:
- storage: 2Gi
resources:
cpu_limit: 2
memory_limit: 1Gi
disk_limit: 2Gi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
user: 1000
seccomp_profile: default
network_policy: isolated
apparmor_profile: endurain
ports:
- host: 8085
container: 8080
protocol: tcp # Web UI
volumes:
- type: bind
source: /var/lib/archipelago/endurain
target: /app/data
options: [rw]
environment:
- ENDURAIN_ENV=production
- ENDURAIN_DATA_DIR=/app/data
health_check:
type: http
endpoint: http://localhost:8085
path: /health
interval: 30s
timeout: 5s
retries: 3

1161
apps/endurain/package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

20
apps/endurain/package.json
View File

@ -1,20 +0,0 @@
{
"name": "endurain",
"version": "1.0.0",
"description": "Endurain application platform",
"main": "dist/index.js",
"scripts": {
"build": "tsc",
"start": "node dist/index.js",
"dev": "ts-node src/index.ts"
},
"dependencies": {
"express": "^4.18.2"
},
"devDependencies": {
"@types/express": "^4.17.21",
"@types/node": "^20.10.0",
"typescript": "^5.3.3",
"ts-node": "^10.9.2"
}
}

27
apps/endurain/src/index.ts
View File

@ -1,27 +0,0 @@
import express from 'express';
const app = express();
const port = 8080;
// Middleware
app.use(express.json());
// Health check endpoint
app.get('/health', (req, res) => {
res.json({ status: 'ok', service: 'endurain', version: '1.0.0' });
});
// API endpoints
app.get('/api/info', (req, res) => {
res.json({
name: 'Endurain',
version: '1.0.0',
status: 'running'
});
});
// Start server
app.listen(port, '0.0.0.0', () => {
console.log(`Endurain listening on port ${port}`);
console.log(`Data directory: ${process.env.ENDURAIN_DATA_DIR || '/app/data'}`);
});

16
apps/endurain/tsconfig.json
View File

@ -1,16 +0,0 @@
{
"compilerOptions": {
"target": "ES2020",
"module": "commonjs",
"lib": ["ES2020"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

2
apps/indeedhub/manifest.yml
View File

@ -6,7 +6,7 @@ app:
category: media
container:
image: git.tx1138.com/lfg2025/indeedhub:latest
image: 146.59.87.168:3000/lfg2025/indeedhub:latest
pull_policy: always # Pull from registry; falls back to local build
dependencies:

5
apps/ollama/Dockerfile
View File

@ -1,5 +0,0 @@
# Ollama - uses official image
FROM ollama/ollama:latest
# Default configuration is in the image
# No additional setup needed

50
apps/ollama/manifest.yml
View File

@ -1,50 +0,0 @@
app:
id: ollama
name: Ollama
version: 0.1.0
description: Run large language models locally. Privacy-preserving AI on your node.
container:
image: ollama/ollama:0.6.2
image_signature: cosign://...
pull_policy: if-not-present
dependencies:
- storage: 50Gi # Models can be large
resources:
cpu_limit: 4
memory_limit: 8Gi # LLMs need lots of RAM
disk_limit: 50Gi
security:
capabilities: []
readonly_root: false # Ollama needs write access for models
no_new_privileges: true
user: 1000
seccomp_profile: default
network_policy: isolated
apparmor_profile: ollama
ports:
- host: 11434
container: 11434
protocol: tcp # API
volumes:
- type: bind
source: /var/lib/archipelago/ollama
target: /root/.ollama
options: [rw]
environment:
- OLLAMA_HOST=0.0.0.0:11434
- OLLAMA_KEEP_ALIVE=24h
health_check:
type: http
endpoint: http://localhost:11434
path: /api/tags
interval: 30s
timeout: 10s
retries: 3

2
core/Cargo.lock generated
View File

@ -80,7 +80,7 @@ checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
[[package]]
name = "archipelago"
version = "1.7.43-alpha"
version = "1.7.45-alpha"
dependencies = [
"anyhow",
"archipelago-container",

2
core/archipelago/Cargo.toml
View File

@ -1,6 +1,6 @@
[package]
name = "archipelago"
version = "1.7.44-alpha"
version = "1.7.45-alpha"
edition = "2021"
description = "Archipelago Bitcoin Node OS - Native backend"
authors = ["Archipelago Team"]

13
core/archipelago/src/api/rpc/package/config.rs
View File

@ -768,10 +768,17 @@ pub(super) async fn get_app_config(
vec!["8240:8240".to_string()],
vec!["/var/lib/archipelago/tailscale:/var/lib/tailscale".to_string()],
vec!["TS_STATE_DIR=/var/lib/tailscale".to_string()],
Some(
"sh -c 'tailscale web --listen 0.0.0.0:8240 & exec tailscaled'".to_string(),
),
// Don't use custom_command (Option<String>) — install.rs passes
// it as a SINGLE arg to podman, which then treats the whole
// "sh -c 'tailscale web …'" string as the executable name and
// fails: "executable file `sh -c 'tailscale web …'` not found".
// custom_args (Option<Vec<String>>) splits properly.
None,
Some(vec![
"sh".to_string(),
"-c".to_string(),
"tailscale web --listen 0.0.0.0:8240 & exec tailscaled".to_string(),
]),
),
"fedimint" => (
vec![

299
core/archipelago/src/api/rpc/package/install.rs
View File

@ -31,6 +31,124 @@ pub(in crate::api::rpc) async fn install_log(msg: &str) {
}
}
/// Patch the Bitcoin RPC `Authorization: Basic ...` header inside the running
/// bitcoin-ui container's nginx config and reload nginx. Authoritative
/// credential injection — runs whether the image was built locally or pulled
/// from the registry. Without this, registry images ship with whatever auth
/// header was baked at build time on the publisher's machine, which never
/// matches the per-node randomly-generated bitcoin-rpc-password.
///
/// Implementation note: this used to do `podman exec sed`, but rootless
/// podman + tightly-confined containers (--cap-drop=ALL, restricted user)
/// reject the exec because crun can't add a new process to the container's
/// cgroup ("write cgroup.procs: Permission denied"). Switched to
/// `podman cp` (storage layer, no cgroup join) + `podman kill --signal=SIGHUP`
/// (signal to existing PID 1, no new process needed). Verified on .228.
async fn inject_bitcoin_rpc_auth_into_running_container(container: &str, auth_b64: &str) {
use rand::distributions::{Alphanumeric, DistString};
let token = Alphanumeric.sample_string(&mut rand::thread_rng(), 8);
let host_path = format!("/tmp/archy-{container}-nginx.conf-{token}");
let in_container = "/etc/nginx/conf.d/default.conf";
// 1. Copy the running config out to host
let cp_out = tokio::process::Command::new("podman")
.args(["cp", &format!("{container}:{in_container}"), &host_path])
.output()
.await;
if let Err(e) = cp_out {
warn!("inject auth: podman cp out failed for {}: {}", container, e);
return;
}
if let Ok(ref o) = cp_out {
if !o.status.success() {
warn!(
"inject auth: podman cp out failed for {}: {}",
container,
String::from_utf8_lossy(&o.stderr)
);
return;
}
}
// 2. Patch the auth line on disk
let content = match tokio::fs::read_to_string(&host_path).await {
Ok(c) => c,
Err(e) => {
warn!("inject auth: read {} failed: {}", host_path, e);
let _ = tokio::fs::remove_file(&host_path).await;
return;
}
};
let mut patched_any = false;
let updated: String = content
.lines()
.map(|line| {
if line.contains("proxy_set_header Authorization") && line.contains("Basic") {
patched_any = true;
format!(
" proxy_set_header Authorization \"Basic {}\";",
auth_b64
)
} else {
line.to_string()
}
})
.collect::<Vec<_>>()
.join("\n");
if !patched_any {
warn!(
"inject auth: no Authorization line matched in {}'s nginx.conf",
container
);
let _ = tokio::fs::remove_file(&host_path).await;
return;
}
if let Err(e) = tokio::fs::write(&host_path, format!("{}\n", updated)).await {
warn!("inject auth: write back failed: {}", e);
let _ = tokio::fs::remove_file(&host_path).await;
return;
}
// 3. Copy patched config back into the container
let cp_in = tokio::process::Command::new("podman")
.args(["cp", &host_path, &format!("{container}:{in_container}")])
.output()
.await;
let _ = tokio::fs::remove_file(&host_path).await;
match cp_in {
Ok(o) if !o.status.success() => {
warn!(
"inject auth: podman cp in failed for {}: {}",
container,
String::from_utf8_lossy(&o.stderr)
);
return;
}
Err(e) => {
warn!("inject auth: podman cp in errored for {}: {}", container, e);
return;
}
_ => {}
}
// 4. Reload nginx via SIGHUP to PID 1 (no exec/cgroup join needed)
let reload = tokio::process::Command::new("podman")
.args(["kill", "--signal=SIGHUP", container])
.output()
.await;
match reload {
Ok(o) if o.status.success() => {
info!("Injected Bitcoin RPC auth into {} (post-start, cp+SIGHUP)", container);
}
Ok(o) => warn!(
"Patched nginx.conf in {} but SIGHUP failed: {}",
container,
String::from_utf8_lossy(&o.stderr)
),
Err(e) => warn!("Patched nginx.conf in {} but SIGHUP errored: {}", container, e),
}
}
impl RpcHandler {
/// Install a package from a Docker image.
/// Security: Image verification, resource limits, network isolation.
@ -83,6 +201,16 @@ impl RpcHandler {
}
}
// Phase: Preparing — emit BEFORE the stack dispatch so multi-container
// stacks also flip state to Installing immediately. Without this, the
// backend's package state for stack apps stayed empty until the first
// podman pull finished, so a hard refresh during the early seconds of
// a stack install showed the app as missing entirely (the user
// reported "the app disappears from installing if you hard refresh
// then sometimes comes back later").
self.set_install_phase(package_id, InstallPhase::Preparing)
.await;
// Multi-container stacks get their own install path
if package_id == "immich" {
return self.install_immich_stack().await;
@ -97,10 +225,6 @@ impl RpcHandler {
return self.install_indeedhub_stack().await;
}
// Phase: Preparing — validating deps and configs before any slow I/O.
self.set_install_phase(package_id, InstallPhase::Preparing)
.await;
// Dependency checks
let deps = detect_running_deps().await?;
check_install_deps(package_id, &deps)?;
@ -1366,41 +1490,66 @@ autopilot.active=false\n",
info!("Nextcloud trusted domains configured for {}", host_ip);
}
// Pre-build: inject Bitcoin RPC auth into bitcoin-ui nginx.conf
if matches!(package_id, "bitcoin" | "bitcoin-core" | "bitcoin-knots") {
let (rpc_user, rpc_pass) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
use base64::Engine;
let auth_b64 = base64::engine::general_purpose::STANDARD
.encode(format!("{}:{}", rpc_user, rpc_pass));
for dir in [
"/opt/archipelago/docker/bitcoin-ui",
"/home/archipelago/archy/docker/bitcoin-ui",
] {
let conf_path = format!("{}/nginx.conf", dir);
if let Ok(content) = tokio::fs::read_to_string(&conf_path).await {
// Replace placeholder or previously-injected auth (regex: Basic followed by base64 or placeholder)
let updated = content
.replace("__BITCOIN_RPC_AUTH__", &auth_b64)
.lines()
.map(|line| {
if line.contains("proxy_set_header Authorization")
&& line.contains("Basic")
// Inject Bitcoin RPC auth into bitcoin-ui nginx.conf.
// Two paths because the credential is per-node and randomly generated
// at first boot, so it can't be baked into the published registry image:
// 1. Build-time: rewrite nginx.conf on disk before `podman build`.
// Only fires when /opt/archipelago/docker/bitcoin-ui exists (dev
// box or ISO that shipped the docker tree). Skipped silently in
// production where ui_builds falls through to the registry image.
// 2. Post-start: `podman exec` into the running container to patch
// nginx.conf and reload. Authoritative for both paths — runs
// regardless of how the image was built.
let bitcoin_rpc_auth_b64: Option<String> =
if matches!(package_id, "bitcoin" | "bitcoin-core" | "bitcoin-knots") {
let (rpc_user, rpc_pass) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
use base64::Engine;
let auth_b64 = base64::engine::general_purpose::STANDARD
.encode(format!("{}:{}", rpc_user, rpc_pass));
for dir in [
"/opt/archipelago/docker/bitcoin-ui",
"/home/archipelago/archy/docker/bitcoin-ui",
] {
let conf_path = format!("{}/nginx.conf", dir);
match tokio::fs::read_to_string(&conf_path).await {
Ok(content) => {
let updated = content
.replace("__BITCOIN_RPC_AUTH__", &auth_b64)
.lines()
.map(|line| {
if line.contains("proxy_set_header Authorization")
&& line.contains("Basic")
{
format!(
" proxy_set_header Authorization \"Basic {}\";",
auth_b64
)
} else {
line.to_string()
}
})
.collect::<Vec<_>>()
.join("\n");
if let Err(e) =
tokio::fs::write(&conf_path, format!("{}\n", updated)).await
{
format!(
" proxy_set_header Authorization \"Basic {}\";",
auth_b64
)
warn!("Failed to write {} with injected RPC auth: {}", conf_path, e);
} else {
line.to_string()
info!("Injected Bitcoin RPC auth into {} (build-time)", conf_path);
}
})
.collect::<Vec<_>>()
.join("\n");
let _ = tokio::fs::write(&conf_path, format!("{}\n", updated)).await;
info!("Injected Bitcoin RPC auth into {}", conf_path);
}
Err(_) => {
debug!(
"No build-time nginx.conf at {} (will patch running container after start)",
conf_path
);
}
}
}
}
}
Some(auth_b64)
} else {
None
};
// Build and start companion UI containers for headless services.
// All UIs proxy to localhost (backend :5678 or bitcoin :8332) so they need --network=host.
@ -1437,9 +1586,14 @@ autopilot.active=false\n",
.find(|d| std::path::Path::new(d).join("Dockerfile").exists())
.unwrap_or_else(|| ui_dir.to_string());
let image_base = image_base.to_string();
let registry = "git.tx1138.com/lfg2025";
let registry = "146.59.87.168:3000/lfg2025";
let registry_image = format!("{}/{}:latest", registry, image_base);
let local_image = format!("localhost/{}:latest", image_base);
let post_start_auth = if name == "archy-bitcoin-ui" {
bitcoin_rpc_auth_b64.clone()
} else {
None
};
tokio::spawn(async move {
// Remove existing container
let _ = tokio::process::Command::new("podman")
@ -1487,32 +1641,69 @@ autopilot.active=false\n",
}
};
// For bitcoin-ui specifically: render nginx.conf to host BEFORE
// starting the container, then bind-mount it. This is the durable
// fix for the bitcoin-rpc 401 — the per-node password is in the
// file before nginx ever opens it. Survives container recreate,
// image update, reboot, --restart=unless-stopped cycles, and
// doesn't need any post-start patching that could fail under
// tightly-confined cgroup permissions.
let mut bitcoin_ui_mount: Option<String> = None;
if name == "archy-bitcoin-ui" {
let paths = crate::container::bitcoin_ui::RenderPaths::default();
match crate::container::bitcoin_ui::render(&paths).await {
Ok(outcome) => {
bitcoin_ui_mount = Some(format!(
"{}:/etc/nginx/conf.d/default.conf:ro,Z",
paths.rendered_path.display()
));
info!(
"bitcoin-ui nginx.conf rendered ({:?}) — will bind-mount at startup",
outcome
);
}
Err(e) => warn!(
"Failed to render bitcoin-ui nginx.conf: {} — \
will fall back to post-start patch (less reliable)",
e
),
}
}
// Run with --network=host (UIs proxy to localhost backend/bitcoin)
// --user 0:0: run as root inside container (still unprivileged on host
// in rootless podman) to avoid nginx chown failures
let mut args: Vec<String> = vec![
"run".into(),
"-d".into(),
"--name".into(),
name.clone(),
"--restart=unless-stopped".into(),
"--network=host".into(),
"--user=0:0".into(),
"--cap-drop=ALL".into(),
"--cap-add=CHOWN".into(),
"--cap-add=DAC_OVERRIDE".into(),
"--cap-add=NET_BIND_SERVICE".into(),
"--cap-add=SETUID".into(),
"--cap-add=SETGID".into(),
"--memory=128m".into(),
];
if let Some(ref mount) = bitcoin_ui_mount {
args.push("-v".into());
args.push(mount.clone());
}
args.push(image.clone());
let run = tokio::process::Command::new("podman")
.args([
"run",
"-d",
"--name",
&name,
"--restart=unless-stopped",
"--network=host",
"--user=0:0",
"--cap-drop=ALL",
"--cap-add=CHOWN",
"--cap-add=DAC_OVERRIDE",
"--cap-add=NET_BIND_SERVICE",
"--cap-add=SETUID",
"--cap-add=SETGID",
"--memory=128m",
&image,
])
.args(&args)
.output()
.await;
match run {
Ok(o) if o.status.success() => {
info!("{} UI container started (host network)", name)
info!("{} UI container started (host network)", name);
if let Some(ref auth) = post_start_auth {
inject_bitcoin_rpc_auth_into_running_container(&name, auth).await;
}
}
Ok(o) => warn!(
"Failed to start {}: {}",

113
core/archipelago/src/api/rpc/package/stacks.rs
View File

@ -4,6 +4,7 @@
//! containers in dependency order.
use crate::api::rpc::RpcHandler;
use crate::data_model::InstallPhase;
use anyhow::{Context, Result};
use tracing::info;
@ -124,7 +125,7 @@ fn mempool_stack_app_ids() -> &'static [&'static str] {
&["archy-mempool-db", "mempool-api", "archy-mempool-web"]
}
const REGISTRY: &str = "git.tx1138.com/lfg2025";
const REGISTRY: &str = "146.59.87.168:3000/lfg2025";
/// Pull an image with retry and exponential backoff (3 attempts).
async fn pull_image_with_retry(image: &str) -> Result<()> {
@ -199,13 +200,20 @@ impl RpcHandler {
}
let images = [
"git.tx1138.com/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
"146.59.87.168:3000/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
"docker.io/valkey/valkey:7-alpine",
"git.tx1138.com/lfg2025/immich-server:release",
"146.59.87.168:3000/lfg2025/immich-server:release",
];
for img in &images {
self.set_install_phase("immich", InstallPhase::PullingImage)
.await;
let n_images = images.len() as u64;
for (i, img) in images.iter().enumerate() {
self.set_install_progress("immich", i as u64, n_images).await;
pull_image_with_retry(img).await?;
}
self.set_install_progress("immich", n_images, n_images).await;
self.set_install_phase("immich", InstallPhase::CreatingContainer)
.await;
let _ = tokio::process::Command::new("sudo")
.args([
@ -265,7 +273,7 @@ impl RpcHandler {
"POSTGRES_USER=postgres",
"-e",
"POSTGRES_DB=immich",
"git.tx1138.com/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
"146.59.87.168:3000/lfg2025/immich-postgres:14-vectorchord0.4.3-pgvectors0.2.0",
])
.output()
.await;
@ -330,7 +338,7 @@ impl RpcHandler {
"REDIS_HOSTNAME=immich_redis",
"-e",
"UPLOAD_LOCATION=/usr/src/app/upload",
"git.tx1138.com/lfg2025/immich-server:release",
"146.59.87.168:3000/lfg2025/immich-server:release",
])
.output()
.await
@ -341,6 +349,13 @@ impl RpcHandler {
return Err(anyhow::anyhow!("Failed to start Immich server: {}", stderr));
}
self.set_install_phase("immich", InstallPhase::WaitingHealthy)
.await;
self.set_install_phase("immich", InstallPhase::PostInstall)
.await;
self.set_install_phase("immich", InstallPhase::Done).await;
self.clear_install_progress("immich").await;
info!("Immich stack installed and started");
Ok(serde_json::json!({
"success": true,
@ -384,9 +399,18 @@ impl RpcHandler {
&format!("{}/nbxplorer:2.6.0", REGISTRY),
&format!("{}/btcpayserver:1.13.7", REGISTRY),
];
for img in &images {
self.set_install_phase("btcpay-server", InstallPhase::PullingImage)
.await;
let n_images = images.len() as u64;
for (i, img) in images.iter().enumerate() {
self.set_install_progress("btcpay-server", i as u64, n_images)
.await;
pull_image_with_retry(img).await?;
}
self.set_install_progress("btcpay-server", n_images, n_images)
.await;
self.set_install_phase("btcpay-server", InstallPhase::CreatingContainer)
.await;
// Create data dirs (chown to current user so rootless podman can write)
let _ = tokio::process::Command::new("sudo")
@ -541,6 +565,14 @@ impl RpcHandler {
return Err(anyhow::anyhow!("Failed to start BTCPay Server: {}", stderr));
}
self.set_install_phase("btcpay-server", InstallPhase::WaitingHealthy)
.await;
self.set_install_phase("btcpay-server", InstallPhase::PostInstall)
.await;
self.set_install_phase("btcpay-server", InstallPhase::Done)
.await;
self.clear_install_progress("btcpay-server").await;
install_log("INSTALL OK: btcpay-server stack").await;
info!("BTCPay stack installed and started");
Ok(serde_json::json!({
@ -590,9 +622,16 @@ impl RpcHandler {
&format!("{}/mempool-backend:v3.0.0", REGISTRY),
&format!("{}/mempool-frontend:v3.0.0", REGISTRY),
];
for img in &images {
self.set_install_phase("mempool", InstallPhase::PullingImage)
.await;
let n_images = images.len() as u64;
for (i, img) in images.iter().enumerate() {
self.set_install_progress("mempool", i as u64, n_images).await;
pull_image_with_retry(img).await?;
}
self.set_install_progress("mempool", n_images, n_images).await;
self.set_install_phase("mempool", InstallPhase::CreatingContainer)
.await;
// Create data dirs (chown to current user so rootless podman can write)
let _ = tokio::process::Command::new("sudo")
@ -750,6 +789,13 @@ impl RpcHandler {
return Err(anyhow::anyhow!("Failed to start Mempool: {}", stderr));
}
self.set_install_phase("mempool", InstallPhase::WaitingHealthy)
.await;
self.set_install_phase("mempool", InstallPhase::PostInstall)
.await;
self.set_install_phase("mempool", InstallPhase::Done).await;
self.clear_install_progress("mempool").await;
install_log("INSTALL OK: mempool stack").await;
info!("Mempool stack installed and started");
Ok(serde_json::json!({
@ -769,7 +815,7 @@ impl RpcHandler {
.into_iter()
.find(|r| r.enabled)
.map(|r| r.url)
.unwrap_or_else(|| "git.tx1138.com/lfg2025".to_string());
.unwrap_or_else(|| "146.59.87.168:3000/lfg2025".to_string());
let user_tmp = format!(
"{}/.local/share/containers/tmp",
@ -794,12 +840,22 @@ impl RpcHandler {
// Pull all images with retry; fail the install if any image can't be pulled.
// Previously this just logged a warning and continued, leaving the stack
// broken and the user seeing "failed" with no recovery path.
for img in &images {
self.set_install_phase("indeedhub", InstallPhase::PullingImage)
.await;
let n_images = images.len() as u64;
for (i, img) in images.iter().enumerate() {
// set_install_progress fills the byte-counter fallback the UI uses
// when it can't read podman's pull output — gives the bar a clear
// X-of-N step as each image lands.
self.set_install_progress("indeedhub", i as u64, n_images)
.await;
info!("Pulling {}", img);
pull_image_with_retry(img)
.await
.with_context(|| format!("Failed to pull IndeedHub image: {}", img))?;
}
self.set_install_progress("indeedhub", n_images, n_images)
.await;
// Remove any leftover containers from a previous partial install (or
// from the first-boot frontend stub that used to race the installer).
@ -826,6 +882,12 @@ impl RpcHandler {
.status()
.await;
// Phase: CreatingContainer — pulls done, network rebuilt, now spinning
// up the 7 stack containers. Bar advances from PullingImage band into
// CreatingContainer band so the user sees movement.
self.set_install_phase("indeedhub", InstallPhase::CreatingContainer)
.await;
// Create indeedhub-net
let _ = tokio::process::Command::new("podman")
.args(["network", "create", "indeedhub-net"])
@ -952,26 +1014,38 @@ impl RpcHandler {
"-e",
"DATABASE_HOST=postgres",
"-e",
"DATABASE_PORT=5432",
"-e",
"DATABASE_USER=indeedhub",
"-e",
&format!("DATABASE_PASSWORD={}", db_pass),
"-e",
"DATABASE_NAME=indeedhub",
"-e",
"REDIS_HOST=redis",
"QUEUE_HOST=redis",
"-e",
"QUEUE_PORT=6379",
"-e",
"S3_ENDPOINT=http://minio:9000",
"-e",
"AWS_REGION=us-east-1",
"-e",
&format!("AWS_ACCESS_KEY={}", minio_user),
"-e",
&format!("AWS_SECRET_KEY={}", minio_pass),
"-e",
"S3_PUBLIC_BUCKET_NAME=indeedhub-public",
"-e",
"S3_PRIVATE_BUCKET_NAME=indeedhub-private",
"-e",
"S3_PUBLIC_BUCKET_URL=/storage",
"-e",
&format!("NOSTR_JWT_SECRET={}", jwt_secret),
"-e",
"NOSTR_JWT_EXPIRES_IN=7d",
"-e",
"AES_MASTER_SECRET=0123456789abcdef0123456789abcdef",
"-e",
"ENVIRONMENT=production",
&format!("{}/indeedhub-api:1.0.0", registry),
])
@ -993,6 +1067,8 @@ impl RpcHandler {
"-e",
"DATABASE_HOST=postgres",
"-e",
"DATABASE_PORT=5432",
"-e",
"DATABASE_USER=indeedhub",
"-e",
&format!("DATABASE_PASSWORD={}", db_pass),
@ -1001,6 +1077,8 @@ impl RpcHandler {
"-e",
"QUEUE_HOST=redis",
"-e",
"QUEUE_PORT=6379",
"-e",
"S3_ENDPOINT=http://minio:9000",
"-e",
&format!("AWS_ACCESS_KEY={}", minio_user),
@ -1011,6 +1089,8 @@ impl RpcHandler {
"-e",
"S3_PUBLIC_BUCKET_NAME=indeedhub-public",
"-e",
"S3_PRIVATE_BUCKET_NAME=indeedhub-private",
"-e",
"ENVIRONMENT=production",
"-e",
"AES_MASTER_SECRET=0123456789abcdef0123456789abcdef",
@ -1048,6 +1128,17 @@ impl RpcHandler {
return Err(anyhow::anyhow!("IndeedHub frontend failed: {}", err));
}
// Phase: WaitingHealthy → PostInstall → clear. The actual readiness
// gate is the package scanner's next sweep; this just gives the UI a
// truthful end-of-install signal so the bar settles at 95→100→done
// instead of sitting at "Queued… 2%" forever.
self.set_install_phase("indeedhub", InstallPhase::WaitingHealthy)
.await;
self.set_install_phase("indeedhub", InstallPhase::PostInstall)
.await;
self.set_install_phase("indeedhub", InstallPhase::Done).await;
self.clear_install_progress("indeedhub").await;
install_log("INSTALL OK: indeedhub stack").await;
info!("IndeedHub stack installed");
Ok(serde_json::json!({

2
core/archipelago/src/api/rpc/package/update.rs
View File

@ -453,7 +453,7 @@ fn candidate_app_ids_for_container(container_name: &str) -> Vec<String> {
};
match container_name {
"bitcoin-knots" => {
"bitcoin-knots" | "bitcoin-core" => {
push("bitcoin-core");
push("bitcoin-knots");
}

11
core/archipelago/src/server.rs
View File

@ -985,6 +985,17 @@ async fn scan_and_update_packages(
let current_ids: Vec<String> = merged.keys().cloned().collect();
for id in current_ids {
if !packages.contains_key(&id) {
// Don't evict packages mid-transition: Installing/Updating/Removing
// legitimately have no live container yet (image still pulling) or
// briefly (during recreate). The absence-eviction here was racing
// installs and removing apps from the UI 14s in. The transitional
// owner (spawn_task) is responsible for clearing state, not us.
if let Some(entry) = merged.get(&id) {
if is_transitional(&entry.state) {
absence_tracker.remove(&id);
continue;
}
}
let count = absence_tracker.entry(id.clone()).or_insert(0);
*count += 1;
if *count >= CONTAINER_ABSENCE_THRESHOLD {

14
core/archipelago/src/update.rs
View File

@ -65,13 +65,11 @@ fn is_newer(candidate: &str, current: &str) -> bool {
}
const DEFAULT_UPDATE_MANIFEST_URL: &str =
"https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/manifest.json";
/// Secondary mirror on an OVH VPS — independent network path so a
/// single-provider outage doesn't knock out both mirrors. Promoted to
/// primary default on 2026-04-23 after the Hetzner .160 VPS was
/// decommissioned.
const DEFAULT_SECONDARY_MIRROR_URL: &str =
"http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json";
/// Secondary mirror on tx1138 gitea — independent network path so a
/// single-provider outage doesn't knock out both mirrors.
const DEFAULT_SECONDARY_MIRROR_URL: &str =
"https://git.tx1138.com/lfg2025/archy/raw/branch/main/releases/manifest.json";
const UPDATE_STATE_FILE: &str = "update_state.json";
const UPDATE_MIRRORS_FILE: &str = "update-mirrors.json";
/// Marker written by apply_update() just before the service restart and
@ -111,11 +109,11 @@ fn mirrors_path(data_dir: &Path) -> std::path::PathBuf {
fn default_mirrors() -> Vec<UpdateMirror> {
vec![
UpdateMirror {
url: DEFAULT_SECONDARY_MIRROR_URL.to_string(),
url: DEFAULT_UPDATE_MANIFEST_URL.to_string(),
label: "Server 1 (OVH)".to_string(),
},
UpdateMirror {
url: DEFAULT_UPDATE_MANIFEST_URL.to_string(),
url: DEFAULT_SECONDARY_MIRROR_URL.to_string(),
label: "Server 2 (tx1138)".to_string(),
},
]

28
image-recipe/_archived/build-auto-installer-iso.sh
View File

@ -2143,6 +2143,24 @@ chown -R 1000:1000 /mnt/target/var/lib/archipelago
echo " ✅ Data partition encrypted with LUKS2 ($LUKS_CIPHER)"
# Allocate swap space on the encrypted data partition. Without swap, large
# container image builds (immich, indeedhub) and brief memory spikes can
# OOM-kill containers or trigger cgroup cascades. Sized to RAM, capped at
# 8GB (above which swap is rarely useful), floored at 2GB so even
# constrained nodes have headroom. Lives on the LUKS partition so it's
# encrypted at rest.
step "Allocating swap"
RAM_MB=$(($(awk '/^MemTotal:/ {print $2}' /proc/meminfo) / 1024))
SWAP_MB=$RAM_MB
[ "$SWAP_MB" -lt 2048 ] && SWAP_MB=2048
[ "$SWAP_MB" -gt 8192 ] && SWAP_MB=8192
SWAPFILE=/mnt/target/var/lib/archipelago/swapfile
echo " Allocating ${SWAP_MB}MB swap at /var/lib/archipelago/swapfile"
run dd if=/dev/zero of="$SWAPFILE" bs=1M count=$SWAP_MB status=none
run chmod 600 "$SWAPFILE"
run mkswap "$SWAPFILE"
echo " ✅ ${SWAP_MB}MB swap allocated"
# Configure auto-unlock via crypttab (key file on root partition)
step "Configuring system"
DATA_UUID=$(blkid -s UUID -o value "$DATA_PART")
@ -2208,6 +2226,8 @@ cat > /mnt/target/etc/fstab <<EOF
UUID=$(blkid -s UUID -o value "$ROOT_PART") / ext4 errors=remount-ro 0 1
UUID=$(blkid -s UUID -o value "$EFI_PART") /boot/efi vfat umask=0077 0 1
/dev/mapper/archipelago-data /var/lib/archipelago ext4 defaults,nofail,x-systemd.device-timeout=60 0 2
# Swap on encrypted data partition — activated after LUKS unlock
/var/lib/archipelago/swapfile none swap sw,nofail 0 0
EOF
# Configure hostname
@ -2241,11 +2261,11 @@ cat > /mnt/target/home/archipelago/.config/containers/registries.conf <<'REGCONF
unqualified-search-registries = ["docker.io"]
[[registry]]
location = "git.tx1138.com"
location = "146.59.87.168:3000"
insecure = true
[[registry]]
location = "146.59.87.168:3000"
location = "git.tx1138.com"
insecure = true
REGCONF
chown -R 1000:1000 /mnt/target/home/archipelago/.config
@ -2255,8 +2275,8 @@ mkdir -p /mnt/target/var/lib/archipelago/config
cat > /mnt/target/var/lib/archipelago/config/registries.json <<'DYNREG'
{
"registries": [
{"url": "git.tx1138.com/lfg2025", "name": "Archipelago Primary", "tls_verify": true, "enabled": true, "priority": 0},
{"url": "146.59.87.168:3000/lfg2025", "name": "Archipelago Fallback", "tls_verify": false, "enabled": true, "priority": 10}
{"url": "146.59.87.168:3000/lfg2025", "name": "Archipelago Primary", "tls_verify": false, "enabled": true, "priority": 0},
{"url": "git.tx1138.com/lfg2025", "name": "Archipelago Fallback", "tls_verify": true, "enabled": true, "priority": 10}
]
}
DYNREG

2
neode-ui/package.json
View File

@ -1,7 +1,7 @@
{
"name": "neode-ui",
"private": true,
"version": "1.7.44-alpha",
"version": "1.7.45-alpha",
"type": "module",
"scripts": {
"start": "./start-dev.sh",

250
neode-ui/public/catalog.json
View File

@ -1,7 +1,7 @@
{
"version": 2,
"updated": "2026-04-22T00:00:00Z",
"registry": "git.tx1138.com/lfg2025",
"registry": "146.59.87.168:3000/lfg2025",
"featured": {
"id": "indeedhub",
"banner": "/assets/img/featured/indeedhub-banner.jpg",
@ -11,200 +11,260 @@
},
"apps": [
{
"id": "bitcoin-knots", "title": "Bitcoin Knots", "version": "28.1.0",
"id": "bitcoin-knots",
"title": "Bitcoin Knots",
"version": "28.1.0",
"description": "Run a full Bitcoin node. Validate and relay blocks and transactions.",
"icon": "/assets/img/app-icons/bitcoin-knots.webp",
"author": "Bitcoin Knots", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/bitcoin-knots:latest",
"author": "Bitcoin Knots",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/bitcoin-knots:latest",
"repoUrl": "https://github.com/bitcoinknots/bitcoin"
},
{
"id": "bitcoin-core", "title": "Bitcoin Core", "version": "28.4",
"id": "bitcoin-core",
"title": "Bitcoin Core",
"version": "28.4",
"description": "Reference implementation of the Bitcoin protocol. Run a full node validating and relaying blocks.",
"icon": "/assets/img/app-icons/bitcoin-core.svg",
"author": "Bitcoin Core contributors", "category": "money", "tier": "optional",
"author": "Bitcoin Core contributors",
"category": "money",
"tier": "optional",
"dockerImage": "docker.io/bitcoin/bitcoin:28.4",
"repoUrl": "https://github.com/bitcoin/bitcoin"
},
{
"id": "lnd", "title": "LND", "version": "0.18.4",
"id": "lnd",
"title": "LND",
"version": "0.18.4",
"description": "Lightning Network Daemon. Fast Bitcoin payments through Lightning.",
"icon": "/assets/img/app-icons/lnd.svg",
"author": "Lightning Labs", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/lnd:v0.18.4-beta",
"author": "Lightning Labs",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/lnd:v0.18.4-beta",
"repoUrl": "https://github.com/lightningnetwork/lnd",
"requires": ["bitcoin-knots"]
"requires": [
"bitcoin-knots"
]
},
{
"id": "btcpay-server", "title": "BTCPay Server", "version": "1.13.7",
"id": "btcpay-server",
"title": "BTCPay Server",
"version": "1.13.7",
"description": "Self-hosted Bitcoin payment processor.",
"icon": "/assets/img/app-icons/btcpay-server.png",
"author": "BTCPay Server Foundation", "category": "commerce", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/btcpayserver:1.13.7",
"author": "BTCPay Server Foundation",
"category": "commerce",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/btcpayserver:1.13.7",
"repoUrl": "https://github.com/btcpayserver/btcpayserver",
"requires": ["bitcoin-knots"]
"requires": [
"bitcoin-knots"
]
},
{
"id": "mempool", "title": "Mempool Explorer", "version": "3.0.0",
"id": "mempool",
"title": "Mempool Explorer",
"version": "3.0.0",
"description": "Self-hosted Bitcoin blockchain and mempool visualizer.",
"icon": "/assets/img/app-icons/mempool.webp",
"author": "Mempool", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/mempool-frontend:v3.0.0",
"author": "Mempool",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/mempool-frontend:v3.0.0",
"repoUrl": "https://github.com/mempool/mempool",
"requires": ["bitcoin-knots", "electrumx"]
"requires": [
"bitcoin-knots",
"electrumx"
]
},
{
"id": "electrumx", "title": "ElectrumX", "version": "1.18.0",
"id": "electrumx",
"title": "ElectrumX",
"version": "1.18.0",
"description": "Electrum protocol server. Index the blockchain for fast wallet lookups.",
"icon": "/assets/img/app-icons/electrumx.webp",
"author": "Luke Childs", "category": "money", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/electrumx:v1.18.0",
"author": "Luke Childs",
"category": "money",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/electrumx:v1.18.0",
"repoUrl": "https://github.com/spesmilo/electrumx",
"requires": ["bitcoin-knots"]
"requires": [
"bitcoin-knots"
]
},
{
"id": "indeedhub", "title": "IndeeHub", "version": "1.0.0",
"id": "indeedhub",
"title": "IndeeHub",
"version": "1.0.0",
"description": "Bitcoin documentary streaming with Nostr identity.",
"icon": "/assets/img/app-icons/indeedhub.png",
"author": "IndeeHub", "category": "community",
"dockerImage": "git.tx1138.com/lfg2025/indeedhub:1.0.0",
"author": "IndeeHub",
"category": "community",
"dockerImage": "146.59.87.168:3000/lfg2025/indeedhub:1.0.0",
"repoUrl": "https://github.com/indeedhub/indeedhub"
},
{
"id": "botfights", "title": "BotFights", "version": "1.1.0",
"id": "botfights",
"title": "BotFights",
"version": "1.1.0",
"description": "Bot arena + 2-player arcade fighter with controller support and Adventure Mode.",
"icon": "/assets/img/app-icons/botfights.svg",
"author": "BotFights", "category": "community",
"dockerImage": "git.tx1138.com/lfg2025/botfights:1.1.0",
"author": "BotFights",
"category": "community",
"dockerImage": "146.59.87.168:3000/lfg2025/botfights:1.1.0",
"repoUrl": "https://botfights.net"
},
{
"id": "gitea", "title": "Gitea", "version": "1.23",
"id": "gitea",
"title": "Gitea",
"version": "1.23",
"description": "Self-hosted Git service with container registry, CI/CD, issue tracking.",
"icon": "/assets/img/app-icons/gitea.svg",
"author": "Gitea", "category": "development",
"author": "Gitea",
"category": "development",
"dockerImage": "docker.io/gitea/gitea:1.23",
"repoUrl": "https://gitea.com"
},
{
"id": "filebrowser", "title": "File Browser", "version": "2.27.0",
"id": "filebrowser",
"title": "File Browser",
"version": "2.27.0",
"description": "Web-based file manager.",
"icon": "/assets/img/app-icons/file-browser.webp",
"author": "File Browser", "category": "data", "tier": "core",
"dockerImage": "git.tx1138.com/lfg2025/filebrowser:v2.27.0",
"author": "File Browser",
"category": "data",
"tier": "core",
"dockerImage": "146.59.87.168:3000/lfg2025/filebrowser:v2.27.0",
"repoUrl": "https://github.com/filebrowser/filebrowser"
},
{
"id": "vaultwarden", "title": "Vaultwarden", "version": "1.30.0",
"id": "vaultwarden",
"title": "Vaultwarden",
"version": "1.30.0",
"description": "Self-hosted password vault with zero-knowledge encryption.",
"icon": "/assets/img/app-icons/vaultwarden.webp",
"author": "Vaultwarden", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/vaultwarden:1.30.0-alpine",
"author": "Vaultwarden",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/vaultwarden:1.30.0-alpine",
"repoUrl": "https://github.com/dani-garcia/vaultwarden"
},
{
"id": "searxng", "title": "SearXNG", "version": "2024.1.0",
"id": "searxng",
"title": "SearXNG",
"version": "2024.1.0",
"description": "Privacy-respecting metasearch engine.",
"icon": "/assets/img/app-icons/searxng.png",
"author": "SearXNG", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/searxng:latest",
"author": "SearXNG",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/searxng:latest",
"repoUrl": "https://github.com/searxng/searxng"
},
{
"id": "fedimint", "title": "Fedimint", "version": "0.10.0",
"id": "fedimint",
"title": "Fedimint",
"version": "0.10.0",
"description": "Federated Bitcoin mint with privacy through federated guardians.",
"icon": "/assets/img/app-icons/fedimint.png",
"author": "Fedimint", "category": "money",
"dockerImage": "git.tx1138.com/lfg2025/fedimintd:v0.10.0",
"author": "Fedimint",
"category": "money",
"dockerImage": "146.59.87.168:3000/lfg2025/fedimintd:v0.10.0",
"repoUrl": "https://github.com/fedimint/fedimint"
},
{
"id": "ollama", "title": "Ollama", "version": "0.5.4",
"description": "Run AI models locally. Private and on your hardware.",
"icon": "/assets/img/app-icons/ollama.png",
"author": "Ollama", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/ollama:latest",
"repoUrl": "https://github.com/ollama/ollama"
},
{
"id": "nextcloud", "title": "Nextcloud", "version": "28",
"description": "Your own private cloud. File sync, calendars, contacts.",
"icon": "/assets/img/app-icons/nextcloud.webp",
"author": "Nextcloud", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/nextcloud:28",
"repoUrl": "https://github.com/nextcloud/server"
},
{
"id": "jellyfin", "title": "Jellyfin", "version": "10.8.13",
"id": "jellyfin",
"title": "Jellyfin",
"version": "10.8.13",
"description": "Free media server. Stream movies, music, and photos.",
"icon": "/assets/img/app-icons/jellyfin.webp",
"author": "Jellyfin", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/jellyfin:10.8.13",
"author": "Jellyfin",
"category": "data",
"dockerImage": "146.59.87.168:3000/lfg2025/jellyfin:10.8.13",
"repoUrl": "https://github.com/jellyfin/jellyfin"
},
{
"id": "immich", "title": "Immich", "version": "1.90.0",
"id": "immich",
"title": "Immich",
"version": "1.90.0",
"description": "High-performance photo and video backup with ML.",
"icon": "/assets/img/app-icons/immich.png",
"author": "Immich", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/immich-server:release",
"author": "Immich",
"category": "data",
"dockerImage": "146.59.87.168:3000/lfg2025/immich-server:release",
"repoUrl": "https://github.com/immich-app/immich"
},
{
"id": "homeassistant", "title": "Home Assistant", "version": "2024.1",
"id": "homeassistant",
"title": "Home Assistant",
"version": "2024.1",
"description": "Open-source home automation.",
"icon": "/assets/img/app-icons/homeassistant.png",
"author": "Home Assistant", "category": "home",
"dockerImage": "git.tx1138.com/lfg2025/home-assistant:2024.1",
"author": "Home Assistant",
"category": "home",
"dockerImage": "146.59.87.168:3000/lfg2025/home-assistant:2024.1",
"repoUrl": "https://github.com/home-assistant/core"
},
{
"id": "grafana", "title": "Grafana", "version": "10.2.0",
"id": "grafana",
"title": "Grafana",
"version": "10.2.0",
"description": "Analytics and monitoring dashboards.",
"icon": "/assets/img/app-icons/grafana.png",
"author": "Grafana Labs", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/grafana:10.2.0",
"author": "Grafana Labs",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/grafana:10.2.0",
"repoUrl": "https://github.com/grafana/grafana"
},
{
"id": "tailscale", "title": "Tailscale", "version": "1.78.0",
"id": "tailscale",
"title": "Tailscale",
"version": "1.78.0",
"description": "Zero-config VPN with WireGuard mesh networking.",
"icon": "/assets/img/app-icons/tailscale.webp",
"author": "Tailscale", "category": "networking", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/tailscale:stable",
"author": "Tailscale",
"category": "networking",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/tailscale:stable",
"repoUrl": "https://github.com/tailscale/tailscale"
},
{
"id": "uptime-kuma", "title": "Uptime Kuma", "version": "1.23.0",
"id": "uptime-kuma",
"title": "Uptime Kuma",
"version": "1.23.0",
"description": "Self-hosted uptime monitoring.",
"icon": "/assets/img/app-icons/uptime-kuma.webp",
"author": "Uptime Kuma", "category": "data", "tier": "recommended",
"dockerImage": "git.tx1138.com/lfg2025/uptime-kuma:1",
"author": "Uptime Kuma",
"category": "data",
"tier": "recommended",
"dockerImage": "146.59.87.168:3000/lfg2025/uptime-kuma:1",
"repoUrl": "https://github.com/louislam/uptime-kuma"
},
{
"id": "dwn", "title": "Decentralized Web Node", "version": "0.4.0",
"description": "Own your data with DID-based access control.",
"icon": "/assets/img/app-icons/dwn.svg",
"author": "TBD", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/dwn-server:main",
"repoUrl": "https://github.com/TBD54566975/dwn-server"
},
{
"id": "endurain", "title": "Endurain", "version": "0.8.0",
"description": "Self-hosted fitness tracking. Strava alternative.",
"icon": "/assets/img/app-icons/endurain.png",
"author": "Endurain", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/endurain:0.8.0",
"repoUrl": "https://github.com/joaovitoriasilva/endurain"
},
{
"id": "photoprism", "title": "PhotoPrism", "version": "240915",
"id": "photoprism",
"title": "PhotoPrism",
"version": "240915",
"description": "AI-powered photo management with facial recognition.",
"icon": "/assets/img/app-icons/photoprism.svg",
"author": "PhotoPrism", "category": "data",
"dockerImage": "git.tx1138.com/lfg2025/photoprism:240915",
"author": "PhotoPrism",
"category": "data",
"dockerImage": "146.59.87.168:3000/lfg2025/photoprism:240915",
"repoUrl": "https://github.com/photoprism/photoprism"
},
{
"id": "nextcloud",
"title": "Nextcloud",
"version": "28",
"description": "Your own private cloud. File sync, calendars, contacts.",
"icon": "/assets/img/app-icons/nextcloud.webp",
"author": "Nextcloud",
"category": "data",
"dockerImage": "docker.io/nextcloud:28",
"repoUrl": "https://github.com/nextcloud/server"
}
]
}

2
neode-ui/src/stores/container.ts
View File

@ -31,7 +31,7 @@ export const BUNDLED_APPS: BundledApp[] = [
{
id: 'bitcoin-knots',
name: 'Bitcoin Knots',
image: 'git.tx1138.com/lfg2025/bitcoin-knots:latest',
image: '146.59.87.168:3000/lfg2025/bitcoin-knots:latest',
description: 'Full Bitcoin node with additional features',
icon: '₿',
ports: [{ host: 8334, container: 80 }],

20
neode-ui/src/views/AppDetails.vue
View File

@ -144,7 +144,6 @@ import {
WEB_ONLY_APP_URLS,
PACKAGE_ALIASES,
BITCOIN_DEPENDENT_APPS,
APP_URLS,
resolvePackageKey,
isRealOnionAddress,
} from './appDetails/appDetailsData'
@ -285,7 +284,6 @@ function goBack() {
function launchApp() {
if (!pkg.value) return
const isDev = import.meta.env.DEV
const id = appId.value
const webOnlyUrl = WEB_ONLY_APP_URLS[id]
@ -294,22 +292,12 @@ function launchApp() {
return
}
if (APP_URLS[id]) {
let url = isDev ? APP_URLS[id].dev : APP_URLS[id].prod
if (url.includes('localhost')) {
url = url.replace('localhost', window.location.hostname)
}
useAppLauncherStore().open({ url, title: pkg.value.manifest.title })
return
}
const torAddress = pkg.value.manifest.interfaces?.main?.['tor-config']
const lanConfig = pkg.value.manifest.interfaces?.main?.['lan-config']
if (torAddress || lanConfig) {
showActionError(t('appDetails.noLaunchUrl'))
}
// Container apps should launch through session routing so protocol/path
// handling stays centralized in appSessionConfig.
useAppLauncherStore().openSession(id)
}
async function startApp() {
try {
await store.startPackage(appId.value)

80
neode-ui/src/views/Discover.vue
View File

@ -221,7 +221,6 @@ const categories = computed(() => [
// local watcher that duplicated logic using byte counters only — it has
// been removed in favour of the store's phase-aware mapping.
const installingApps = serverStore.installingApps
const maxAttempts = ref(60)
function selectCategory(id: string) {
selectedCategory.value = id
@ -415,7 +414,6 @@ function viewAppDetails(app: MarketplaceApp) {
// Timer management
const activeTimers: ReturnType<typeof setTimeout>[] = []
const activeIntervals: ReturnType<typeof setInterval>[] = []
function trackTimeout(fn: () => void, ms: number) {
const id = setTimeout(() => {
@ -427,86 +425,68 @@ function trackTimeout(fn: () => void, ms: number) {
return id
}
function trackInterval(fn: () => void, ms: number) {
const id = setInterval(fn, ms)
activeIntervals.push(id)
return id
}
function clearTrackedInterval(id: ReturnType<typeof setInterval>) {
clearInterval(id)
const idx = activeIntervals.indexOf(id)
if (idx !== -1) activeIntervals.splice(idx, 1)
}
onBeforeUnmount(() => {
for (const t of activeTimers) clearTimeout(t)
activeTimers.length = 0
for (const i of activeIntervals) clearInterval(i)
activeIntervals.length = 0
})
function startInstallPolling(appId: string, statusMessage: string) {
const interval = trackInterval(() => {
const current = installingApps.get(appId)
if (!current) { clearTrackedInterval(interval); return }
const newAttempt = current.attempt + 1
installingApps.set(appId, { ...current, attempt: newAttempt, progress: Math.min(60 + (newAttempt * 0.5), 95), message: statusMessage })
if (isInstalled(appId)) {
clearTrackedInterval(interval)
installingApps.set(appId, { ...current, status: 'complete', progress: 100, message: 'Installation complete!' })
trackTimeout(() => { installingApps.delete(appId) }, 2000)
} else if (newAttempt >= maxAttempts.value) {
clearTrackedInterval(interval)
installingApps.set(appId, { ...current, status: 'error', progress: 0, message: 'Installation timeout' })
trackTimeout(() => { installingApps.delete(appId) }, 5000)
}
}, 1000)
}
const toast = useToast()
function queueInstall(app: MarketplaceApp) {
serverStore.setInstallProgress(app.id, {
id: app.id,
title: app.title ?? app.id,
status: 'downloading',
progress: 2,
message: 'Queued…',
attempt: 0,
})
}
function failInstall(app: MarketplaceApp, err: unknown) {
const message = "Failed: " + (err instanceof Error ? err.message : String(err))
serverStore.setInstallProgress(app.id, {
id: app.id,
title: app.title ?? app.id,
status: 'error',
progress: 0,
message,
attempt: 0,
})
trackTimeout(() => { serverStore.clearInstallProgress(app.id) }, 5000)
}
async function installApp(app: MarketplaceApp) {
if (installingApps.has(app.id) || isInstalled(app.id)) return
installingApps.set(app.id, { id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Preparing installation...', attempt: 0 })
toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
queueInstall(app)
toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
router.push('/dashboard/apps').catch(() => {})
try {
const installUrl = app.url || app.manifestUrl || app.s9pkUrl
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 30, message: 'Downloading package...' })
await rpcClient.call({ method: 'package.install', params: { id: app.id, url: installUrl, version: app.version }, timeout: 15000 })
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Installing package...' })
startInstallPolling(app.id, 'Starting application...')
} catch (err) {
if (import.meta.env.DEV) console.error('Installation failed:', err)
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
trackTimeout(() => { installingApps.delete(app.id) }, 5000)
failInstall(app, err)
}
}
async function installCommunityApp(app: MarketplaceApp) {
if (installingApps.has(app.id) || isInstalled(app.id) || !app.dockerImage) return
installingApps.set(app.id, { id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Pulling Docker image...', attempt: 0 })
toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
queueInstall(app)
toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
router.push('/dashboard/apps').catch(() => {})
try {
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 20, message: 'Downloading container image...' })
// Pass containerConfig from catalog if available (allows dynamic apps without hardcoded backend config)
const installParams: Record<string, unknown> = { id: app.id, dockerImage: app.dockerImage, version: app.version }
if ((app as Record<string, unknown>).containerConfig) {
installParams.containerConfig = (app as Record<string, unknown>).containerConfig
}
await rpcClient.call({ method: 'package.install', params: installParams, timeout: 15000 })
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Starting container...' })
startInstallPolling(app.id, 'Initializing application...')
} catch (err) {
if (import.meta.env.DEV) console.error('[Discover] Installation failed:', err)
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
trackTimeout(() => { installingApps.delete(app.id) }, 5000)
failInstall(app, err)
}
}
onMounted(() => {
discoverAnimationDone = true
if (communityApps.value.length === 0 && !loadingCommunity.value) {

111
neode-ui/src/views/Marketplace.vue
View File

@ -157,7 +157,6 @@ const categories = computed(() => [
// Installation state — uses global store so it persists across navigation
const installingApps = server.installingApps
const maxAttempts = ref(60)
// Install progress tracking is now in serverStore (global watcher on WebSocket data)
// so it works regardless of which page is active
@ -338,7 +337,6 @@ function viewAppDetails(app: MarketplaceApp) {
}
const activeTimers: ReturnType<typeof setTimeout>[] = []
const activeIntervals: ReturnType<typeof setInterval>[] = []
function trackTimeout(fn: () => void, ms: number) {
const id = setTimeout(() => {
@ -350,115 +348,74 @@ function trackTimeout(fn: () => void, ms: number) {
return id
}
function trackInterval(fn: () => void, ms: number) {
const id = setInterval(fn, ms)
activeIntervals.push(id)
return id
}
function clearTrackedInterval(id: ReturnType<typeof setInterval>) {
clearInterval(id)
const idx = activeIntervals.indexOf(id)
if (idx !== -1) activeIntervals.splice(idx, 1)
}
onBeforeUnmount(() => {
for (const t of activeTimers) clearTimeout(t)
activeTimers.length = 0
for (const i of activeIntervals) clearInterval(i)
activeIntervals.length = 0
})
function startInstallPolling(appId: string, statusMessage: string) {
const interval = trackInterval(() => {
const current = installingApps.get(appId)
if (!current) { clearTrackedInterval(interval); return }
function queueInstall(app: MarketplaceApp) {
server.setInstallProgress(app.id, {
id: app.id,
title: app.title ?? app.id,
status: 'downloading',
progress: 2,
message: 'Queued…',
attempt: 0,
})
}
const newAttempt = current.attempt + 1
const state = getInstalledState(appId)
// Update message based on actual backend state
let message = statusMessage
if (state === 'starting') message = 'Starting application...'
else if (state === 'running') message = 'Installation complete!'
installingApps.set(appId, {
...current,
attempt: newAttempt,
progress: Math.min(60 + (newAttempt * 0.5), 95),
message
})
// Only clear when fully running — server store watcher handles the actual delete
if (state === 'running') {
clearTrackedInterval(interval)
installingApps.set(appId, { ...current, status: 'complete', progress: 100, message: 'Installation complete!' })
// Server store watcher will clear installingApps when it sees 'running'
} else if (newAttempt >= maxAttempts.value) {
clearTrackedInterval(interval)
installingApps.set(appId, { ...current, status: 'error', progress: 0, message: 'Installation timeout — check My Apps' })
trackTimeout(() => { installingApps.delete(appId) }, 5000)
}
}, 1000)
function failInstall(app: MarketplaceApp, err: unknown) {
const message = "Failed: " + (err instanceof Error ? err.message : String(err))
server.setInstallProgress(app.id, {
id: app.id,
title: app.title ?? app.id,
status: 'error',
progress: 0,
message,
attempt: 0,
})
trackTimeout(() => { server.clearInstallProgress(app.id) }, 5000)
}
async function installApp(app: MarketplaceApp) {
if (installingApps.has(app.id) || isInstalled(app.id)) return
installingApps.set(app.id, {
id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Preparing installation...', attempt: 0
})
// Navigate to My Apps immediately and show toast
toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
queueInstall(app)
toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
router.push('/dashboard/apps').catch(() => {})
try {
const installUrl = app.url || app.manifestUrl || app.s9pkUrl
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 30, message: 'Downloading package...' })
await rpcClient.call({ method: 'package.install', params: { id: app.id, url: installUrl, version: app.version }, timeout: 15000 })
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Installing package...' })
startInstallPolling(app.id, 'Starting application...')
await rpcClient.call({
method: 'package.install',
params: { id: app.id, url: installUrl, version: app.version },
timeout: 15000,
})
} catch (err) {
if (import.meta.env.DEV) console.error('Installation failed:', err)
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
trackTimeout(() => { installingApps.delete(app.id) }, 5000)
failInstall(app, err)
}
}
async function installCommunityApp(app: MarketplaceApp) {
if (installingApps.has(app.id) || isInstalled(app.id) || !app.dockerImage) return
installingApps.set(app.id, {
id: app.id, title: app.title ?? app.id, status: 'downloading', progress: 10, message: 'Pulling Docker image...', attempt: 0
})
// Navigate to My Apps immediately and show toast
toast.info(`Installing ${app.title ?? app.id} — check My Apps`)
queueInstall(app)
toast.info("Installing " + (app.title ?? app.id) + " - check My Apps")
router.push('/dashboard/apps').catch(() => {})
try {
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'downloading', progress: 20, message: 'Downloading container image...' })
await rpcClient.call({
method: 'package.install',
params: { id: app.id, dockerImage: app.dockerImage, version: app.version },
timeout: 15000
timeout: 15000,
})
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'installing', progress: 60, message: 'Starting container...' })
startInstallPolling(app.id, 'Initializing application...')
} catch (err) {
if (import.meta.env.DEV) console.error('[Marketplace] Installation failed:', err)
installingApps.set(app.id, { ...installingApps.get(app.id)!, status: 'error', progress: 0, message: `Failed: ${err}` })
trackTimeout(() => { installingApps.delete(app.id) }, 5000)
failInstall(app, err)
}
}
</script>
<style scoped>

3
neode-ui/src/views/MarketplaceAppDetails.vue
View File

@ -444,7 +444,7 @@ const features = computed(() => {
})
/** App dependency definitions */
const R = 'git.tx1138.com/lfg2025'
const R = '146.59.87.168:3000/lfg2025'
const APP_DEPENDENCIES: Record<string, { id: string; title: string; dockerImage: string }[]> = {
'electrumx': [{ id: 'bitcoin-knots', title: 'Bitcoin Knots', dockerImage: `${R}/bitcoin-knots:latest` }],
'lnd': [{ id: 'bitcoin-knots', title: 'Bitcoin Knots', dockerImage: `${R}/bitcoin-knots:latest` }],
@ -607,4 +607,3 @@ async function installApp() {
}
}
</script>

2
neode-ui/src/views/appDetails/appDetailsData.ts
View File

@ -6,7 +6,6 @@ import { PackageState } from '@/types/api'
/** Web-only app detection (no container -- external websites) */
export const WEB_ONLY_APP_URLS: Record<string, string> = {
'indeedhub': `${window.location.protocol}//${window.location.hostname}:7777`,
'nwnn': 'https://nwnn.l484.com',
'484-kitchen': 'https://484.kitchen',
'call-the-operator': 'https://cta.tx1138.com',
@ -65,7 +64,6 @@ export const APP_URLS: Record<string, { dev: string; prod: string }> = {
'lorabell': { dev: 'http://192.168.1.166', prod: 'http://192.168.1.166' },
'atob': { dev: 'http://localhost:8102', prod: 'https://app.atobitcoin.io' },
'k484': { dev: 'http://localhost:8103', prod: 'http://localhost:8103' },
'indeedhub': { dev: 'https://archipelago.indeehub.studio', prod: 'https://archipelago.indeehub.studio' },
'bitcoin': { dev: 'http://localhost:8332', prod: 'http://localhost:8332' },
'btcpay-server': { dev: 'http://localhost:23000', prod: 'http://localhost:23000' },
'homeassistant': { dev: 'http://localhost:8123', prod: 'http://localhost:8123' },

19
neode-ui/src/views/appSession/appSessionConfig.ts
View File

@ -38,7 +38,7 @@ export const APP_PORTS: Record<string, number> = {
'fedimint': 8175,
'fedimintd': 8175,
'fedimint-gateway': 8176,
'indeedhub': 7778,
'indeedhub': 7777,
'botfights': 9100,
'dwn': 3100,
'endurain': 8080,
@ -138,11 +138,24 @@ export function resolveAppUrl(id: string, routeQueryPath?: string): string {
const ext = EXTERNAL_URLS[id]
if (ext) return ext
// Local apps: always launch by host port
// Bitcoin apps always go through nginx proxy so browser basic-auth prompts never appear.
if (id === 'bitcoin-knots' || id === 'bitcoin-core' || id === 'bitcoin-ui') {
return window.location.protocol + '//' + window.location.hostname + '/app/bitcoin-ui/'
}
// HTTPS pages cannot embed plain HTTP port origins (mixed-content).
if (window.location.protocol === 'https:') {
const proxyPath = HTTPS_PROXY_PATHS[id]
if (proxyPath) {
return window.location.protocol + '//' + window.location.hostname + proxyPath
}
}
// Local apps on HTTP pages launch by host port.
const port = APP_PORTS[id]
if (!port) return ''
let base = `${window.location.protocol}//${window.location.hostname}:${port}`
let base = window.location.protocol + '//' + window.location.hostname + ':' + String(port)
if (routeQueryPath) base += routeQueryPath
return base
}

2
neode-ui/src/views/discover/curatedApps.ts
View File

@ -1,6 +1,6 @@
import type { MarketplaceApp } from './types'
const R = 'git.tx1138.com/lfg2025'
const R = '146.59.87.168:3000/lfg2025'
// ---------- Dynamic catalog from registry ----------
export interface CatalogFeatured {

2
neode-ui/src/views/marketplace/marketplaceData.ts
View File

@ -376,7 +376,7 @@ export function getCuratedAppList(): MarketplaceApp[] {
description: 'Bitcoin documentary streaming platform with Nostr identity sign-in. Stream God Bless Bitcoin and other educational content about sovereignty and decentralized technology.',
icon: '/assets/img/app-icons/indeedhub.png',
author: 'Indeehub Team',
dockerImage: 'git.tx1138.com/lfg2025/indeedhub:latest',
dockerImage: '146.59.87.168:3000/lfg2025/indeedhub:latest',
manifestUrl: undefined,
repoUrl: 'https://github.com/indeedhub/indeedhub'
},

8
scripts/image-versions.sh
View File

@ -5,13 +5,13 @@
# Usage: source /opt/archipelago/image-versions.sh 2>/dev/null || true
# source "$(dirname "$0")/image-versions.sh" 2>/dev/null || true
#
# Tags MUST match what's actually in the registry at git.tx1138.com/lfg2025/
# Run: podman images --format '{{.Repository}}:{{.Tag}}' | grep 'git.tx1138' | sort
# Tags MUST match what's actually in the registry at 146.59.87.168:3000/lfg2025/
# Run: podman images --format '{{.Repository}}:{{.Tag}}' | grep '146.59.87.168:3000' | sort
# to verify against the registry.
# Archipelago app registries (primary + fallback)
ARCHY_REGISTRY="git.tx1138.com/lfg2025"
ARCHY_REGISTRY_FALLBACK="146.59.87.168:3000/lfg2025"
ARCHY_REGISTRY="146.59.87.168:3000/lfg2025"
ARCHY_REGISTRY_FALLBACK="git.tx1138.com/lfg2025"
# Bitcoin stack
BITCOIN_KNOTS_IMAGE="$ARCHY_REGISTRY/bitcoin-knots:latest"

109
scripts/resilience/README.md Normal file
View File

@ -0,0 +1,109 @@
# Resilience Harness
Black-box state-machine tester for archipelago app containers.
Drives the live RPC against a real archipelago + podman runtime on a target
host. For each app in `app-catalog/catalog.json`, runs every state transition
a user could trigger and asserts the system stays in the expected state.
## Why this exists
We shipped v1.7.43-alpha on .228 with three independent bugs that no unit test
caught:
1. `indeedhub-api` crashlooped 8500+ times because `stacks.rs` was missing 5
env vars (`QUEUE_HOST`/`QUEUE_PORT`/`DATABASE_PORT`/`S3_PRIVATE_BUCKET_NAME`/
`AES_MASTER_SECRET`) — the install "succeeded" (containers running) but the
API never became healthy.
2. `bitcoin-ui` shipped with a stale baked-in `Authorization: Basic …` header
from the registry image, so every `/bitcoin-rpc/` call returned 401.
3. The container-absence scanner evicted apps from the UI 14 seconds into
install (before image pull finished).
All three were exactly the kind of bug a "did the user-visible flow actually
work end to end?" test would catch — and the kind a single-file unit test
will never catch. This harness is the gate.
## Running
Against the .228 test node:
scripts/resilience/resilience.sh archipelago@192.168.1.228
Or non-interactive (CI):
RESILIENCE_SSH_PASS=… RESILIENCE_UI_PASS=… \
scripts/resilience/resilience.sh archipelago@192.168.1.228
Filters:
# Smoke test (3 apps, no reboot, ~15min)
scripts/resilience/resilience.sh archipelago@192.168.1.228 smoke
# Single app
scripts/resilience/resilience.sh archipelago@192.168.1.228 bitcoin-knots
# Subset
scripts/resilience/resilience.sh archipelago@192.168.1.228 bitcoin-knots,lnd
Without a filter, the harness sweeps **every** app in the catalog
(~24 apps × 7 per-app transitions + 2 batch transitions) and runs the
batch transitions (archipelago.service restart, host reboot) at the end.
Full sweep is ~3-4 hours and **reboots the target host** as part of the
run — only point it at a dedicated test node.
## What it tests
Per-app transitions:
| # | Transition | Pass criteria |
|---|----------------------|------------------------------------------------|
| 1 | install | All containers reach `running` within 10 min |
| 2 | ui_probe | HTTP 2xx/3xx via `https://<host>/app/<id>/` |
| 3 | auth_probe | (bitcoin-rpc only) returns 200 not 401 |
| 4 | stop | All containers reach `exited` state |
| 5 | start | All containers reach `running` state |
| 6 | restart | All containers `running` after restart |
| 7 | uninstall | All containers absent, no residue |
Batch transitions (full sweep only):
| # | Transition | Pass criteria |
|---|-------------------------------|-------------------------------------|
| 8 | archipelago.service restart | Container set unchanged across |
| 9 | host reboot | Container set unchanged across |
Coverage by design — discovery rather than encoded metadata. The harness
snapshots `podman ps -a` before install, again after install stabilizes,
and the difference IS this app's container set. Works equally well for
single-container apps and 7-container stacks (indeedhub) without per-app
configuration.
## Output
JSON-lines results at `scripts/resilience/reports/<run_ts>/results.jsonl`:
{"ts":"…","app":"bitcoin-knots","transition":"install","status":"PASS","detail":"bitcoin-knots,archy-bitcoin-ui"}
{"ts":"…","app":"bitcoin-knots","transition":"auth_probe","status":"PASS","detail":"bitcoin-rpc HTTP 200"}
Exit code: `0` if every cell green, `1` if any red, `2` if setup failed
before tests began. Use as a release gate — refuse to tag if any cell red.
## Auth flow
The harness uses the same `auth.login` RPC that the UI uses, then carries
`session=…` and `csrf_token=…` cookies plus the `X-CSRF-Token` header on
every subsequent call. Re-logs in after archipelago.service restart and
host reboot.
## Caveats / known gaps
- App proxy probe (`/app/<id>/`) only validates the proxy responds — for
apps with deeper protocol behavior (lnd, fedimint, mempool) this only
catches "container alive, proxy reachable", not "the protocol is healthy".
- Multi-container stack assertions: the harness checks **every** new
container is `running`, so it would catch the indeedhub-api restart loop
while postgres/redis/minio looked fine.
- Host reboot test is destructive and slow — runs once at end of full sweep.
- `package.start`/`stop`/`restart` RPC methods may not exist for all apps;
failures are recorded and the harness continues.

297
scripts/resilience/lib.sh Executable file
View File

@ -0,0 +1,297 @@
#!/bin/bash
# Resilience harness shared helpers.
# Sourced by resilience.sh — do not invoke directly.
# Required env (set by resilience.sh before sourcing):
# TARGET — ssh target, e.g. archipelago@192.168.1.228
# RPC_URL — http://<host>:5678/rpc/v1
# COOKIE_JAR — path for curl cookie store
# SSH_PASS — sshpass password
# UI_PASS — archipelago UI password
# OUT_DIR — report output dir
# ── ssh ─────────────────────────────────────────────────────────
ssh_run() {
# -n: redirect stdin from /dev/null so ssh doesn't gobble up our parent's
# stdin. Without this, ssh inside a `while read … done <<< "$LIST"`
# consumes the heredoc on the first call, ending the loop after one
# iteration. Cost us a smoke run that only tested filebrowser instead
# of all three smoke apps.
sshpass -p "$SSH_PASS" ssh -n -o StrictHostKeyChecking=accept-new \
-o ConnectTimeout=10 -o LogLevel=ERROR "$TARGET" "$@"
}
# Run a command and tolerate ssh failure (host rebooting, etc.).
ssh_try() {
sshpass -p "$SSH_PASS" ssh -n -o StrictHostKeyChecking=accept-new \
-o ConnectTimeout=5 -o LogLevel=ERROR "$TARGET" "$@" 2>/dev/null || echo "__SSH_FAIL__"
}
ssh_wait_ready() {
local deadline=$(($(date +%s) + ${1:-180}))
while [ "$(date +%s)" -lt "$deadline" ]; do
if [ "$(ssh_try 'echo OK')" = "OK" ]; then return 0; fi
sleep 3
done
return 1
}
# ── rpc ─────────────────────────────────────────────────────────
rpc_login() {
local resp
resp=$(curl -ksS -c "$COOKIE_JAR" -H "Content-Type: application/json" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"auth.login\",\"params\":{\"password\":\"$UI_PASS\"},\"id\":1}" \
"$RPC_URL")
if echo "$resp" | jq -e '.error' >/dev/null 2>&1; then
echo "ERROR: login failed: $(echo "$resp" | jq -c .)" >&2
return 1
fi
CSRF_TOKEN=$(awk '/csrf_token/ {print $7}' "$COOKIE_JAR" | head -1)
[ -n "$CSRF_TOKEN" ] || { echo "ERROR: no CSRF token after login" >&2; return 1; }
export CSRF_TOKEN
}
# Make an RPC call. Args: method, json_params, timeout_secs (optional, default 90).
# Prints raw JSON response. Caller asserts success via jq.
#
# CSRF rotates per-response: the server may issue a new csrf_token on every
# state-changing call, so we re-read it from the cookie jar before each call
# rather than caching the value from login. Also retries once on nginx-served
# BACKEND_UNAVAILABLE (5xx fallback) for transient stalls.
rpc_call() {
local method="$1"
# NOTE: don't use ${2:-{}} — bash matches the first unescaped `}` as the
# end of the expansion, so the trailing `}` becomes a literal char and
# corrupts every params value into invalid JSON. Use an if-check instead.
local params="${2-}"
[ -z "$params" ] && params='{}'
local timeout="${3:-90}"
local attempt
for attempt in 1 2 3 4; do
local csrf
csrf=$(awk '/^[^#]/ && /csrf_token/ {print $7; exit}' "$COOKIE_JAR")
local resp
resp=$(curl -ksS -b "$COOKIE_JAR" -c "$COOKIE_JAR" \
-H "Content-Type: application/json" \
-H "X-CSRF-Token: $csrf" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"$method\",\"params\":$params,\"id\":1}" \
--max-time "$timeout" \
"$RPC_URL")
# Retry on transient errors:
# BACKEND_UNAVAILABLE — nginx 5xx fallback (archipelago briefly stalled)
# 429 — nginx rate limiter exceeded (burst=40 in /etc/nginx/sites-enabled/*)
if echo "$resp" | jq -e '.error.code == "BACKEND_UNAVAILABLE" or .error.code == 429' >/dev/null 2>&1; then
[ "$attempt" -eq 4 ] && { echo "$resp"; return; }
# Exponential-ish backoff: 5s, 15s, 30s. Plenty of time for the
# nginx rate window (1s) and any archipelago restart to clear.
sleep $((attempt * 10))
continue
fi
echo "$resp"
return
done
}
# After a service restart the session may need re-establishing.
rpc_relogin_if_needed() {
local probe
probe=$(rpc_call "package.list" '{}' 2>/dev/null)
if echo "$probe" | jq -e '.error.code == -32001' >/dev/null 2>&1; then
rpc_login || return 1
fi
}
# ── per-app metadata ────────────────────────────────────────────
# Mappings the harness needs that aren't expressible from catalog.json alone:
# multi-container stack rosters, alias/variant container names (bitcoin-knots
# vs bitcoin-core install the same slots), and the actual nginx UI proxy path
# (which often differs from /app/<id>/, e.g. `bitcoin-knots` → `/app/bitcoin-ui/`).
#
# Keep these tables in sync with the install code in package/stacks.rs and
# the `*_IMAGE` companion handling in install.rs (the `archy-<x>-ui` set).
# Containers an app installs. Used for app_already_installed detection AND
# for state assertions when the snapshot-diff falls back (variant apps don't
# create new containers when their alternate is already present).
expected_containers_for() {
case "$1" in
bitcoin-knots) echo "bitcoin-knots archy-bitcoin-ui" ;;
bitcoin-core) echo "bitcoin-core archy-bitcoin-ui" ;;
lnd) echo "lnd archy-lnd-ui" ;;
electrumx|electrs|mempool-electrs)
echo "electrs archy-electrs-ui" ;;
btcpay-server) echo "archy-btcpay-server archy-btcpay-db archy-nbxplorer archy-btcpay-ui" ;;
mempool) echo "mempool archy-mempool-web archy-mempool-db" ;;
immich) echo "immich_server immich_machine_learning immich_postgres immich_redis" ;;
penpot|penpot-frontend)
echo "penpot-frontend penpot-backend penpot-exporter penpot-postgres penpot-redis" ;;
indeedhub) echo "indeedhub indeedhub-api indeedhub-ffmpeg indeedhub-postgres indeedhub-redis indeedhub-minio indeedhub-relay" ;;
*) echo "$1" ;;
esac
}
# UI proxy URL path on the HTTPS frontend. Most apps live at /app/<id>/ but
# Bitcoin/LND/Electrs proxy through their UI companion containers, and BTCPay
# uses its own short path.
ui_proxy_path_for() {
case "$1" in
bitcoin-knots|bitcoin-core) echo "/app/bitcoin-ui/" ;;
electrumx|electrs) echo "/app/electrs-ui/" ;;
lnd) echo "/app/lnd-ui/" ;;
btcpay-server) echo "/app/btcpay/" ;;
*) echo "/app/$1/" ;;
esac
}
# Authenticated probe for credentialed UIs. Echoes the HTTP status code if
# defined, otherwise returns 1 (caller records SKIP). PASS = code in
# {200,401,403} for endpoints that prove the proxy reaches the backend
# (401/403 from app's own auth ≠ 502 from broken proxy).
auth_probe_for() {
local app="$1"
local host; host="$(echo "$TARGET" | cut -d@ -f2)"
case "$app" in
bitcoin-knots|bitcoin-core)
# Direct bitcoin-rpc proxy on :8334 inside .228 — credential
# plumbing is the .228 bug we just shipped, must return 200.
ssh_run 'curl -s -o /dev/null -w "%{http_code}" --max-time 5 -X POST http://127.0.0.1:8334/bitcoin-rpc/ -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"getblockchaininfo\",\"params\":[]}"'
;;
btcpay-server)
# BTCPay's own auth returns 401 for unauthenticated API calls;
# 502 means proxy broken / backend down.
curl -ks -o /dev/null -w "%{http_code}" --max-time 5 \
"https://$host/app/btcpay/api/v1/server/info"
;;
lnd)
# LND has a /lnd-connect-info passthrough on archipelago itself —
# returns lndconnect URI when LND is up. 200 = backend reachable.
curl -ks -o /dev/null -w "%{http_code}" --max-time 5 \
"https://$host/lnd-connect-info"
;;
electrumx|electrs)
# ElectrumX is plain TCP (electrum protocol) — no HTTPS auth path.
# archipelago exposes /electrs-status which queries the daemon.
curl -ks -o /dev/null -w "%{http_code}" --max-time 5 \
"https://$host/electrs-status"
;;
*)
return 1
;;
esac
}
# Whether an auth_probe HTTP code counts as a pass.
auth_probe_pass_codes() {
case "$1" in
bitcoin-knots|bitcoin-core) echo "200" ;;
btcpay-server) echo "200 401 403" ;;
lnd|electrumx|electrs) echo "200" ;;
*) echo "200" ;;
esac
}
# ── probes (state assertions) ───────────────────────────────────
# Returns container Status string ("running","exited","absent",…).
probe_container_state() {
local name="$1"
ssh_run "podman inspect '$name' --format '{{.State.Status}}' 2>/dev/null || echo absent"
}
# Returns RestartCount as integer.
probe_container_restart_count() {
local name="$1"
ssh_run "podman inspect '$name' --format '{{.RestartCount}}' 2>/dev/null || echo -1"
}
# Probe the app's UI proxy on the HTTPS frontend. Returns HTTP code.
# Uses ui_proxy_path_for so apps with non-default proxy paths (bitcoin-ui,
# lnd-ui, electrs-ui, btcpay) get probed at the right URL.
probe_app_proxy() {
local app_id="$1"
local host
host="$(echo "$TARGET" | cut -d@ -f2)"
local path
path=$(ui_proxy_path_for "$app_id")
curl -ks -o /dev/null -w "%{http_code}" --max-time 5 "https://$host$path" || echo "000"
}
# Check that ZERO containers are leftover for this app — catches uninstall residue.
probe_no_residue() {
local prefix="$1"
ssh_run "podman ps -a --format '{{.Names}}' | grep -E '^${prefix}(-|$)' | wc -l"
}
# ── waiters ─────────────────────────────────────────────────────
# Wait for the package's state in the RPC list to match expected, with timeout.
wait_for_package_state() {
local pkg="$1"; local want="$2"; local timeout="${3:-300}"
local deadline=$(($(date +%s) + timeout))
while [ "$(date +%s)" -lt "$deadline" ]; do
local got
got=$(rpc_call "package.list" '{}' \
| jq -r ".result.package_data[\"$pkg\"].state // \"absent\"")
case "$want" in
Running) [ "$got" = "Running" ] && return 0 ;;
Stopped) [ "$got" = "Stopped" ] && return 0 ;;
absent) [ "$got" = "absent" ] && return 0 ;;
esac
sleep 4
done
echo "TIMEOUT waiting for $pkg → $want (last seen: $got)" >&2
return 1
}
# Wait for podman state of a specific container.
wait_for_container_state() {
local name="$1"; local want="$2"; local timeout="${3:-180}"
local deadline=$(($(date +%s) + timeout))
while [ "$(date +%s)" -lt "$deadline" ]; do
local got
got=$(probe_container_state "$name")
[ "$got" = "$want" ] && return 0
sleep 3
done
echo "TIMEOUT waiting for container $name → $want (last seen: $got)" >&2
return 1
}
# Wait until restart count is stable for `stable_secs` seconds — proxy for "no crashloop".
wait_restart_count_stable() {
local name="$1"; local stable_secs="${2:-30}"; local timeout="${3:-180}"
local deadline=$(($(date +%s) + timeout))
local last; local last_change_ts
last=$(probe_container_restart_count "$name")
last_change_ts=$(date +%s)
while [ "$(date +%s)" -lt "$deadline" ]; do
sleep 5
local now
now=$(probe_container_restart_count "$name")
if [ "$now" != "$last" ]; then
last="$now"
last_change_ts=$(date +%s)
elif [ $(( $(date +%s) - last_change_ts )) -ge "$stable_secs" ]; then
return 0
fi
done
echo "TIMEOUT waiting for $name restart-count stable (last=$last)" >&2
return 1
}
# ── result recording ────────────────────────────────────────────
# Append a result row to the JSON-lines report.
# Args: app_id, transition, status (PASS/FAIL/SKIP), detail
record() {
local app="$1"; local transition="$2"; local status="$3"; local detail="${4:-}"
local ts
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
jq -nc --arg ts "$ts" --arg app "$app" --arg t "$transition" --arg s "$status" --arg d "$detail" \
'{ts:$ts, app:$app, transition:$t, status:$s, detail:$d}' >> "$OUT_DIR/results.jsonl"
local marker
case "$status" in
PASS) marker="✅" ;;
FAIL) marker="❌" ;;
SKIP) marker="⏭" ;;
*) marker="•" ;;
esac
printf '%s [%-15s] %-30s %s%s\n' "$marker" "$app" "$transition" "$status" "${detail:+ — $detail}"
}

473
scripts/resilience/resilience.sh Executable file
View File

@ -0,0 +1,473 @@
#!/bin/bash
# Archipelago resilience harness — black-box state-machine tester for app containers.
#
# Drives the live archipelago RPC against a real podman runtime on a target
# host. For each app in the catalog, runs every state transition a user could
# trigger (install / probe / stop / start / restart / archipelago-restart /
# host-reboot / uninstall / reinstall / vanish-watch) and asserts the system
# remains in the expected state at every step.
#
# Usage:
# scripts/resilience/resilience.sh archipelago@192.168.1.228 [filter]
#
# `filter` is a comma-separated list of app IDs (or "smoke" for the curated
# fast subset). Default: every app in app-catalog/catalog.json.
#
# Exit codes:
# 0 every cell green
# 1 any cell red — release should not ship
# 2 setup/auth error before tests began
set -uo pipefail
# ── args ─────────────────────────────────────────────────────────
TARGET="${1:?usage: $0 <user@host> [filter]}"
FILTER="${2:-}"
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
HERE="$ROOT/scripts/resilience"
RUN_TS="$(date -u +%Y%m%dT%H%M%SZ)"
OUT_DIR="$HERE/reports/$RUN_TS"
mkdir -p "$OUT_DIR"
COOKIE_JAR="$OUT_DIR/cookies.txt"
HOST="$(echo "$TARGET" | cut -d@ -f2)"
# RPC reaches archipelago through nginx on 443 (which proxies to localhost:5678).
# Direct :5678 is bound to 127.0.0.1 on the target so we can't curl it from here.
RPC_URL="https://$HOST/rpc/v1"
export TARGET RPC_URL COOKIE_JAR OUT_DIR
# shellcheck source=lib.sh
. "$HERE/lib.sh"
# ── credentials ──────────────────────────────────────────────────
# Pull from env first (so this script can be called from CI). Fall back to
# interactive prompts.
SSH_PASS="${RESILIENCE_SSH_PASS:-}"
UI_PASS="${RESILIENCE_UI_PASS:-}"
if [ -z "$SSH_PASS" ]; then
read -rsp "SSH password for $TARGET: " SSH_PASS; echo
fi
if [ -z "$UI_PASS" ]; then
read -rsp "Archipelago UI password: " UI_PASS; echo
fi
export SSH_PASS UI_PASS
command -v sshpass >/dev/null || { echo "sshpass required"; exit 2; }
command -v jq >/dev/null || { echo "jq required"; exit 2; }
ssh_run 'echo ok' >/dev/null || { echo "ssh to $TARGET failed"; exit 2; }
rpc_login || exit 2
echo "Resilience harness — target $TARGET, run $RUN_TS"
echo "Output: $OUT_DIR/results.jsonl"
echo "─────────────────────────────────────────────────────────────"
# ── catalog & filter ─────────────────────────────────────────────
CATALOG="$ROOT/app-catalog/catalog.json"
ALL_APPS=$(jq -r '.apps[].id' "$CATALOG")
# Topo-sort the catalog by `requires`. Outputs app IDs in install order
# (deps first, then dependents). Kahn's algorithm via python — keeps the
# bash side simple and the deps logic obvious for next-time-readers.
topo_order() {
python3 -c "
import json
with open('$CATALOG') as f: c = json.load(f)
deps = {a['id']: list(a.get('requires', [])) for a in c['apps']}
order = []
remaining = set(deps)
while remaining:
ready = sorted(a for a in remaining if all(d not in remaining for d in deps[a]))
if not ready: # cycle (shouldn't happen) — emit whatever's left
order.extend(sorted(remaining)); break
order.extend(ready); remaining.difference_update(ready)
print('\n'.join(order))
"
}
apps_to_test() {
local order; order=$(topo_order)
if [ -z "$FILTER" ]; then
# Full sweep — but skip bitcoin-core since it shares container slots
# with bitcoin-knots; testing both back-to-back would just churn the
# same containers. bitcoin-knots is the canonical entry.
echo "$order" | grep -v '^bitcoin-core$'
elif [ "$FILTER" = "smoke" ]; then
# Fast subset exercising the bug classes we just fixed:
# single-container, multi-container stack, credentialed UI.
echo -e "filebrowser\nbitcoin-knots\nindeedhub"
else
echo "$order" | grep -E "^($(echo "$FILTER" | tr ',' '|'))$"
fi
}
# Resolve `requires` chain for $1 in install-order (deps first).
deps_for_app() {
local app="$1"
python3 -c "
import json
with open('$CATALOG') as f: c = json.load(f)
deps_map = {a['id']: list(a.get('requires', [])) for a in c['apps']}
visited, order = set(), []
def visit(x):
if x in visited or x not in deps_map: return
visited.add(x)
for d in deps_map.get(x, []): visit(d)
order.append(x)
for d in deps_map.get('$app', []): visit(d)
print('\n'.join(order))
"
}
# ── per-app transitions ──────────────────────────────────────────
# Diff helper: capture container names matching a sane prefix for $app_id.
# Approach: snapshot before install, snapshot after, take the difference =
# this app's containers.
snapshot_containers() {
ssh_run "podman ps -a --format '{{.Names}}' | sort"
}
# Whether $app currently has any of its expected containers running. Uses
# the per-app metadata table in lib.sh (expected_containers_for) so variant
# apps (bitcoin-knots/bitcoin-core sharing slots) and stacks are detected
# correctly. Falls back to name-prefix match for apps the table doesn't know.
app_already_installed() {
local app="$1"
local snap; snap=$(snapshot_containers)
local expected
expected=$(expected_containers_for "$app")
local c
for c in $expected; do
echo "$snap" | grep -qxF "$c" && return 0
done
# Generic prefix fallback for apps not in the expected_containers_for table.
echo "$snap" | grep -qE "^(${app}|${app}-|archy-${app}|archy-${app}-)"
}
# Install missing deps for $app via the regular install path. Idempotent —
# already-installed deps are skipped. Records dep_install per dep so we can
# tell from the report whether the bitcoin pre-req was actually green by the
# time lnd's matrix started.
ensure_deps_installed() {
local app="$1"
local dep
for dep in $(deps_for_app "$app"); do
if app_already_installed "$dep"; then
continue
fi
echo " · dep install: $dep (required by $app)"
local img ver resp
img=$(jq -r --arg id "$dep" '.apps[] | select(.id==$id) | .dockerImage // ""' "$CATALOG")
ver=$(jq -r --arg id "$dep" '.apps[] | select(.id==$id) | .version // ""' "$CATALOG")
if [ -z "$img" ]; then
record "$app" "dep_$dep" FAIL "no dockerImage in catalog for dep $dep"
return 1
fi
resp=$(rpc_call "package.install" "$(jq -nc \
--arg id "$dep" --arg img "$img" --arg ver "$ver" \
'{id:$id, dockerImage:$img, version:$ver}')")
if echo "$resp" | jq -e '.error' >/dev/null 2>&1; then
record "$app" "dep_$dep" FAIL "rpc error: $(echo "$resp" | jq -c '.error')"
return 1
fi
# Wait for at least one expected container to appear running.
local deadline=$(($(date +%s) + 600))
while [ "$(date +%s)" -lt "$deadline" ]; do
if app_already_installed "$dep"; then
record "$app" "dep_$dep" PASS "installed"
break
fi
sleep 5
done
if ! app_already_installed "$dep"; then
record "$app" "dep_$dep" FAIL "containers did not appear within 10min"
return 1
fi
done
return 0
}
# Pre-clean: if the app is currently installed, uninstall it and wait for
# all containers to disappear. We can't measure install correctness without
# starting from a clean slate. Fail-soft — if the uninstall RPC errors we
# log but proceed; the install step will catch any residual state.
preclean_app() {
local app="$1"
if ! app_already_installed "$app"; then
return 0
fi
echo " · pre-clean: $app already installed, uninstalling first"
local resp; resp=$(rpc_call "package.uninstall" "{\"id\":\"$app\"}")
if echo "$resp" | jq -e '.error' >/dev/null 2>&1; then
echo " pre-clean uninstall RPC error: $(echo "$resp" | jq -c '.error')"
fi
# Multi-container stacks (indeedhub: 7, immich: 5, mempool: 3, btcpay: 6)
# take noticeably longer to tear down than single-container apps. 240s was
# too tight for indeedhub's 7-container teardown — bump to 10 min for
# safety; per-container timeout is still bounded inside archipelago itself.
local deadline=$(($(date +%s) + 600))
while [ "$(date +%s)" -lt "$deadline" ]; do
if ! app_already_installed "$app"; then return 0; fi
sleep 5
done
echo " pre-clean: timeout waiting for $app to uninstall"
return 1
}
# Run the full per-app matrix. Records a row per transition.
run_app_matrix() {
local app="$1"
echo
echo "═══ $app ═══"
if ! ensure_deps_installed "$app"; then
record "$app" install FAIL "dep install failed; skipping rest of matrix"
return
fi
preclean_app "$app" || record "$app" preclean FAIL "uninstall before test did not complete"
# ── 01 install ───────────────────────────────────────────────
local before after new_containers
before=$(snapshot_containers)
# The install handler requires `id` + `dockerImage` from the catalog
# entry. Match what the UI passes (Discover.vue / MarketplaceAppDetails.vue).
local docker_image version
docker_image=$(jq -r --arg id "$app" '.apps[] | select(.id==$id) | .dockerImage // ""' "$CATALOG")
version=$(jq -r --arg id "$app" '.apps[] | select(.id==$id) | .version // ""' "$CATALOG")
if [ -z "$docker_image" ]; then
record "$app" install FAIL "no dockerImage in catalog for $app"
return
fi
local install_resp
install_resp=$(rpc_call "package.install" "$(jq -nc \
--arg id "$app" --arg img "$docker_image" --arg ver "$version" \
'{id:$id, dockerImage:$img, version:$ver}')")
if echo "$install_resp" | jq -e '.error' >/dev/null 2>&1; then
record "$app" install FAIL "rpc error: $(echo "$install_resp" | jq -c '.error')"
return # cannot continue this app
fi
# Wait for the EXPECTED containers (per expected_containers_for) to all
# appear. The old "snapshot stable for 10s + count > before" heuristic
# terminated early on apps with deps: e.g. mempool's wait would break
# when archy-electrs-ui (electrumx dep companion) appeared, long before
# mempool's own containers were created (those take ~10min to pull and
# start). Waiting on the expected-set is exact, not heuristic.
#
# Cap at 15 minutes — mempool stack with cold image cache needs ~12 min.
local expected; expected=$(expected_containers_for "$app")
local deadline=$(($(date +%s) + 900))
while [ "$(date +%s)" -lt "$deadline" ]; do
after=$(snapshot_containers)
local missing=0
for c in $expected; do
echo "$after" | grep -qxF "$c" || missing=1
done
[ "$missing" -eq 0 ] && break
sleep 5
done
new_containers=$(comm -13 <(echo "$before") <(echo "$after"))
if [ -z "$new_containers" ]; then
record "$app" install FAIL "no containers created within 10min"
return
fi
# Assert each new container is in 'running' state.
local install_ok=1; local detail=""
while read -r c; do
[ -z "$c" ] && continue
local s
s=$(probe_container_state "$c")
if [ "$s" != "running" ]; then
install_ok=0
detail="$detail $c=$s"
fi
done <<< "$new_containers"
if [ "$install_ok" -eq 1 ]; then
record "$app" install PASS "$(echo "$new_containers" | tr '\n' ',' | sed 's/,$//')"
else
record "$app" install FAIL "containers not running:$detail"
fi
# ── 02 ui_probe ──────────────────────────────────────────────
local code
code=$(probe_app_proxy "$app")
# Accept all 2xx/3xx — proxy reaches backend, app may redirect to login,
# serve OAuth flow (307), or use 308 permanent. 401/403 still fail because
# those mean "backend reached, app rejected request" which is the
# credential-plumbing failure mode we DO want to catch.
if [[ "$code" =~ ^(2[0-9][0-9]|3[0-9][0-9])$ ]]; then
record "$app" ui_probe PASS "HTTP $code"
else
record "$app" ui_probe FAIL "HTTP $code (expected 2xx/3xx)"
fi
# ── 03 auth_probe (only for apps with a credentialed/data endpoint) ──
local probe_code; local pass_codes
if probe_code=$(auth_probe_for "$app" 2>/dev/null) && [ -n "$probe_code" ]; then
pass_codes=$(auth_probe_pass_codes "$app")
if echo " $pass_codes " | grep -qF " $probe_code "; then
record "$app" auth_probe PASS "HTTP $probe_code"
else
record "$app" auth_probe FAIL "HTTP $probe_code (expected one of: $pass_codes — credential plumbing broken)"
fi
else
record "$app" auth_probe SKIP "no authenticated probe defined"
fi
# ── 04 stop ──────────────────────────────────────────────────
local stop_resp
stop_resp=$(rpc_call "package.stop" "{\"id\":\"$app\"}")
if echo "$stop_resp" | jq -e '.error' >/dev/null 2>&1; then
record "$app" stop FAIL "rpc error: $(echo "$stop_resp" | jq -c '.error')"
else
local all_stopped=1
while read -r c; do
[ -z "$c" ] && continue
wait_for_container_state "$c" "exited" 60 || all_stopped=0
done <<< "$new_containers"
if [ "$all_stopped" -eq 1 ]; then
record "$app" stop PASS
else
record "$app" stop FAIL "not all containers reached exited state"
fi
fi
# ── 05 start ─────────────────────────────────────────────────
local start_resp
start_resp=$(rpc_call "package.start" "{\"id\":\"$app\"}")
if echo "$start_resp" | jq -e '.error' >/dev/null 2>&1; then
record "$app" start FAIL "rpc error: $(echo "$start_resp" | jq -c '.error')"
else
local all_started=1
while read -r c; do
[ -z "$c" ] && continue
wait_for_container_state "$c" "running" 90 || all_started=0
done <<< "$new_containers"
if [ "$all_started" -eq 1 ]; then
record "$app" start PASS
else
record "$app" start FAIL "not all containers reached running state"
fi
fi
# ── 06 restart_container ─────────────────────────────────────
# `package.restart` returns immediately and spawns the actual restart.
# `podman restart -t <stop_timeout>` blocks for up to stop_timeout
# seconds (e.g. 600s for bitcoin-core). Polling once after sleep 5
# races on slow-stopping apps and false-positive-FAILs them. Poll
# each container up to 90s for "running" instead.
local restart_resp
restart_resp=$(rpc_call "package.restart" "{\"id\":\"$app\"}")
if echo "$restart_resp" | jq -e '.error' >/dev/null 2>&1; then
record "$app" restart FAIL "rpc error: $(echo "$restart_resp" | jq -c '.error')"
else
local all_running=1
while read -r c; do
[ -z "$c" ] && continue
wait_for_container_state "$c" "running" 90 || all_running=0
done <<< "$new_containers"
if [ "$all_running" -eq 1 ]; then
record "$app" restart PASS
else
record "$app" restart FAIL "container not running 90s after restart"
fi
fi
# ── 09 uninstall (skip 07 archipelago-restart and 08 host-reboot
# here — those are batch tests run once across all installed apps) ─
local uninst_resp
uninst_resp=$(rpc_call "package.uninstall" "{\"id\":\"$app\"}")
if echo "$uninst_resp" | jq -e '.error' >/dev/null 2>&1; then
record "$app" uninstall FAIL "rpc error: $(echo "$uninst_resp" | jq -c '.error')"
else
# Wait for all this-app containers to be absent.
local all_gone=1
while read -r c; do
[ -z "$c" ] && continue
wait_for_container_state "$c" "absent" 120 || all_gone=0
done <<< "$new_containers"
if [ "$all_gone" -eq 1 ]; then
record "$app" uninstall PASS
else
record "$app" uninstall FAIL "not all containers removed"
fi
fi
}
# ── batch transitions (run after per-app loop) ───────────────────
batch_archipelago_service_restart() {
echo
echo "═══ batch: archipelago.service restart ═══"
local before; before=$(snapshot_containers)
if ! ssh_run 'sudo systemctl restart archipelago'; then
record "_batch" archipelago_restart FAIL "systemctl restart errored"
return
fi
ssh_wait_ready 60 || { record "_batch" archipelago_restart FAIL "ssh did not return"; return; }
sleep 30 # let containers re-stabilize
rpc_login || { record "_batch" archipelago_restart FAIL "rpc relogin failed"; return; }
local after; after=$(snapshot_containers)
if [ "$before" = "$after" ]; then
record "_batch" archipelago_restart PASS "container set unchanged"
else
record "_batch" archipelago_restart FAIL "container set drifted across restart"
fi
}
batch_host_reboot() {
echo
echo "═══ batch: host reboot ═══"
local before; before=$(snapshot_containers)
ssh_run 'sudo systemctl reboot' || true # ssh disconnects immediately
sleep 30
# 5 min was too short — .228 took ~9min for full BIOS+kernel+systemd+
# rootless-podman boot. 12 min gives margin for slower hardware.
ssh_wait_ready 720 || { record "_batch" host_reboot FAIL "host did not come back in 12min"; return; }
sleep 60 # let containers auto-restart
rpc_login || { record "_batch" host_reboot FAIL "rpc unreachable after reboot"; return; }
local after; after=$(snapshot_containers)
if [ "$before" = "$after" ]; then
record "_batch" host_reboot PASS "all containers came back"
else
local missing
missing=$(comm -23 <(echo "$before") <(echo "$after") | tr '\n' ',' | sed 's/,$//')
record "_batch" host_reboot FAIL "missing: $missing"
fi
}
# ── main ─────────────────────────────────────────────────────────
APPS_LIST=$(apps_to_test)
if [ -z "$APPS_LIST" ]; then
echo "no apps match filter '$FILTER'" >&2; exit 2
fi
while read -r app; do
[ -z "$app" ] && continue
run_app_matrix "$app"
done <<< "$APPS_LIST"
# Batch transitions only run on full sweep (skip in filtered/smoke mode).
if [ -z "$FILTER" ]; then
batch_archipelago_service_restart
batch_host_reboot
fi
# ── summary ──────────────────────────────────────────────────────
echo
echo "═══ summary ═══"
count_status() {
local pat="$1"
[ -s "$OUT_DIR/results.jsonl" ] || { echo 0; return; }
awk -v pat="$pat" '$0 ~ pat { n++ } END { print n+0 }' "$OUT_DIR/results.jsonl"
}
PASS=$(count_status '"status":"PASS"')
FAIL=$(count_status '"status":"FAIL"')
SKIP=$(count_status '"status":"SKIP"')
TOTAL=$((PASS + FAIL + SKIP))
echo "PASS: $PASS / FAIL: $FAIL / SKIP: $SKIP / TOTAL: $TOTAL"
echo "Report: $OUT_DIR/results.jsonl"
[ "$FAIL" -eq 0 ] || exit 1
exit 0