0ac673deb4
Closes failure mode adjacent to FM3 (docs/bulletproof-containers.md): on
a syncing pruned node, bitcoind's RPC thread blocks for 5-10s during block
validation. The old 10s client-side timeout was rejecting roughly 30% of
UI calls even though the node was perfectly healthy. 20x stress test on
the live .116 node (caught in IBD catch-up at block 797k) used to drop
10 of 20 calls; now drops 0 of 20.
What changed:
- core/archipelago/src/api/rpc/bitcoin.rs: bitcoin_rpc_call now retries up
to 3 times with 500ms and 1500ms backoffs between attempts. Only
transient transport errors (timeout, connect refused, send/recv IO)
trigger retry. A well-formed bitcoind error response is surfaced
immediately - real RPC bugs are never masked.
- Per-attempt hard deadline (tokio::time::timeout, 15s) layered on top
of reqwest's own timeout, so DNS starvation or TLS wedging can't
steal the entire retry budget.
- handle_bitcoin_getinfo client builder gained a 3s connect_timeout
so a dead bitcoind is fast-failed inside the first attempt instead
of eating the whole 15s.
- Retry policy extracted into a RetryConfig struct so tests can dial
down timeouts to ~100ms per attempt. Production defaults live in
RetryConfig::production().
Not changed (tracked as follow-up):
- mesh/mod.rs bitcoin_rpc_getblockcount and related helpers use the
same 10s-timeout pattern. Not migrated to the new wrapper in this
release; scheduled for v1.7.43 alongside the render_bitcoin_conf
work.
- lnd/info.rs and electrs_status have similar 10s/15s timeouts but
different failure profiles - audit first, migrate only the ones
that actually exhibit the bug.
Tests: 6 new unit tests under api::rpc::bitcoin::tests, all passing.
Uses an in-process hyper server (already a transitive dep) to simulate
bitcoind responses; no new crates required.
- happy_path_first_attempt: no retry when first attempt succeeds
- retries_on_timeout_then_succeeds: first attempt times out, second
succeeds, returns OK (uses a short-timeout RetryConfig so the test
runs in <1s instead of 15s)
- retries_exhausted_on_persistent_connect_refused: all attempts fail
against a closed port, error bubbles up, elapsed time confirms
backoffs actually ran
- does_not_retry_on_rpc_level_error: bitcoind-returned error body is
surfaced immediately, no retry
- does_not_retry_parse_errors: non-JSON response (e.g. 503 with html
body) is NOT retried - guards against the tempting "retry all
non-2xx" mistake that would mask real bitcoind misconfig
- retry_budget_invariants: asserts total wall-time ceiling stays
under 60s so a bumped constant can't silently hang a UI call
forever
Validated live on .116: 20/20 bitcoin.getinfo calls succeed during IBD
catch-up (chain at block 797419 -> 797464), vs ~40% baseline under the
old 10s timeout. Worst-case latency was 48.9s during peak validation;
happy-path latency (cached result) remains 28-77ms.
Archipelago
Self-Sovereign Bitcoin Node OS
Archipelago is a bootable personal server OS. Flash it to a USB drive, install on any x86_64 or ARM64 machine, and manage Bitcoin infrastructure, self-hosted apps, and decentralized identity through a glassmorphism web UI.
Features
Bitcoin Infrastructure
- Bitcoin Knots full node with pruning support
- LND Lightning Network daemon with channel management
- ElectrumX Electrum server for wallet connectivity
- BTCPay Server for accepting Bitcoin payments
- Mempool block explorer and fee estimator
- Fedimint federation guardian and gateway
Self-Hosted Apps (30)
Bitcoin (ThunderHub), Storage (FileBrowser, Immich, Nextcloud), Productivity (Penpot, OnlyOffice, Vaultwarden), Media (Jellyfin, PhotoPrism), Search (SearXNG), AI (Ollama), Network (Tailscale, Nginx Proxy Manager), Home (Home Assistant), Nostr (nostr-rs-relay, Nostrudel), Dev (Grafana, Portainer), and more.
Decentralized Identity
- Ed25519 node identity with DID Documents (did:key)
- Multi-identity management (Personal/Business/Anonymous)
- W3C Verifiable Credentials issuance and verification
- Decentralized Web Node (DWN) with bidirectional sync over Tor
- Nostr relay integration and NIP-07 signing for iframe apps
Multi-Node Federation
- Invite-based node joining over Tor hidden services
- Trust levels (Trusted/Verified/Untrusted) with DID-based auth
- Bidirectional DWN state sync between federated nodes
- File sharing with access controls (free/peers-only/paid)
Mesh Networking
- LoRa radio communication via Meshcore protocol
- Device discovery and mesh routing
- Off-grid Bitcoin balance checks (planned)
System Updates
- OTA updates from self-hosted Gitea (git.tx1138.com) with SHA256 verification
- Three update modes: Manual, Daily Check, Auto Apply (3 AM window)
- Rollback support with automatic backup before applying
- Full UI for update management in Settings
Security
- ChaCha20-Poly1305 encrypted secrets at rest, Argon2id password hashing
- Rootless Podman: read-only root, cap-drop ALL, non-root user, no-new-privileges
- TOTP two-factor authentication
- Per-endpoint rate limiting, CSRF protection, input validation
- AppArmor profiles for container confinement
- Tor hidden services for all inter-node communication
- All crypto and container dependencies pinned to exact versions
- Full penetration test completed (33 findings, all remediated)
Quick Start
Install from ISO
- Download the ISO for your architecture (x86_64 or ARM64)
- Flash to USB drive with Balena Etcher or
dd - Boot from USB on target hardware
- Follow the automated installer
- Access the web UI at
http://<device-ip> - Set your password and start the onboarding wizard
Supported Hardware
| Platform | Examples | Minimum |
|---|---|---|
| x86_64 | Intel NUC, mini PCs, any 64-bit PC | 4GB RAM, 32GB storage |
| ARM64 | Raspberry Pi 5, ARM64 SBCs | 4GB RAM, 32GB storage |
Recommended: 8GB+ RAM, 1TB+ NVMe SSD (for full Bitcoin node)
Development
Prerequisites
- macOS or Linux for frontend development
- Linux dev server (Debian 13) for backend builds — never build Rust on macOS for Linux
- Node.js 20+, Rust stable toolchain
Frontend Development
cd neode-ui
npm install
npm start # Dev server on http://localhost:8100 (mock backend on :5959)
npm run type-check # TypeScript validation
npm run build # Production build → web/dist/neode-ui/
Deploy to Server
./scripts/deploy-to-target.sh --live # Deploy to primary dev server
./scripts/deploy-to-target.sh --both # Deploy to both LAN servers
Build ISO
ssh archipelago@<server>
cd ~/archy/image-recipe
sudo ./build-auto-installer-iso.sh
Architecture
Debian 13 (Trixie)
├── Rootless Podman (30 containers, archy-net DNS)
├── Nginx (reverse proxy, security headers, rate limiting)
├── Rust Backend (JSON-RPC API on 127.0.0.1:5678)
│ ├── core/archipelago/ — RPC endpoints, auth, identity, federation, mesh
│ ├── core/container/ — PodmanClient (REST API socket), manifests, health
│ ├── core/security/ — AppArmor, secrets, Cosign image verification
│ └── 6 more crates — models, helpers, js-engine, performance, etc.
├── Vue 3 Frontend (Composition API + TypeScript strict + Pinia + Tailwind)
└── System Tor (hidden services, SOCKS5 proxy)
~49,000 lines of Rust | ~47,000 lines of TypeScript/Vue | 78 shell scripts | 30 container apps
Documentation
| Doc | Purpose |
|---|---|
| Architecture | System design, codebase stats, data paths |
| Architecture Review (HTML) | Interactive guide with diagrams and learning path |
| Developer Guide | Dev setup, workflow, code conventions |
| API Reference | Complete RPC endpoint reference |
| App Developer Guide | Building and publishing apps |
| User Walkthrough | End-user installation and usage guide |
| Troubleshooting | Diagnostic scenarios and solutions |
| Operations Runbook | Ops commands and emergency recovery |
| Security Audit | Penetration test findings |
| Master Plan | Phased roadmap and task tracking |
Contributing
- Fork the repository
- Create a feature branch (
feature/description) - Follow the coding standards in CLAUDE.md
- Submit a pull request