diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md new file mode 100644 index 00000000..9b66d7c9 --- /dev/null +++ b/.claude/memory/MEMORY.md @@ -0,0 +1,11 @@ +# Archipelago Project Memory Index + +- [pending-features.md](pending-features.md) — Feature requests: kiosk mode, sideloading, Nostr login, etc. +- [second-server.md](second-server.md) — Second dev server (archipelago-2 via Tailscale) +- [third-server.md](third-server.md) — Third dev server (archipelago-3 via Tailscale) +- [deploy-automation.md](deploy-automation.md) — Deploy script automation TODOs +- [claude-proxy-setup.md](claude-proxy-setup.md) — Claude proxy OAuth setup details +- [project-plan.md](project-plan.md) — Overall project plan status +- [iso-build-session-2026-03-10.md](iso-build-session-2026-03-10.md) — ISO build session notes +- [unbundled-iso.md](unbundled-iso.md) — Unbundled ISO approach notes +- [web-only-apps.md](web-only-apps.md) — Web-only apps (L484 category) and iframe compatibility diff --git a/.claude/memory/claude-proxy-setup.md b/.claude/memory/claude-proxy-setup.md new file mode 100644 index 00000000..b79565c4 --- /dev/null +++ b/.claude/memory/claude-proxy-setup.md @@ -0,0 +1,9 @@ +--- +name: claude-proxy-oauth-primary-only +description: Only the primary server (192.168.1.228) uses OAuth proxy for Claude. Second server uses standard API key. +type: feedback +--- + +- Only the primary dev server (192.168.1.228) uses the Claude Max OAuth proxy for AIUI +- The second dev server and any other deployments should use a standard Anthropic API key (`ANTHROPIC_API_KEY` env var) +- Do not replicate the OAuth proxy setup to other servers diff --git a/.claude/memory/deploy-automation.md b/.claude/memory/deploy-automation.md new file mode 100644 index 00000000..7444e928 --- /dev/null +++ b/.claude/memory/deploy-automation.md @@ -0,0 +1,18 @@ +--- +name: deploy-automation-todos +description: Items to incorporate into the deploy script for automated setup — Claude API key, AIUI nginx config, swap +type: project +--- + +## Deploy Script Automation TODOs + +These items should be added to `scripts/deploy-to-target.sh` so new server deploys are self-contained: + +1. **Claude API key setup**: After deploying the proxy, auto-configure `/opt/archipelago/claude-proxy/.env.local` with the `ANTHROPIC_API_KEY`. On non-primary servers, use the standard API key (not OAuth). The key is: `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` + +2. **AIUI nginx location block**: Ensure nginx config includes a proper `/aiui/` location block so static JS/CSS files are served with correct MIME types. Without this, AIUI fails to load modules. + +3. **Swap space**: Deploy script should check for swap and create 4GB if missing (`fallocate -l 4G /swapfile && mkswap && swapon + fstab entry`). + +4. **Primary server (192.168.1.228)**: 4GB swap configured on 2026-03-11. +5. **Second server (archipelago-2)**: 4GB swap configured on 2026-03-11. diff --git a/.claude/memory/pending-features.md b/.claude/memory/pending-features.md new file mode 100644 index 00000000..3f7a74da --- /dev/null +++ b/.claude/memory/pending-features.md @@ -0,0 +1,26 @@ +--- +name: pending-ui-features +description: Feature requests — completed and pending items for the next deployment cycle +type: project +--- + +## Completed (2026-03-11) + +1. **IndieHub in iframe** — Restored. Removed forced new-tab check in `mustOpenInNewTab()`. +2. **App uninstall fix** — Backend now logs errors and returns structured response instead of silently swallowing. +3. **Login music stops after auth** — Added `stopAllAudio()` + router afterEach guard. +4. **Container scanner dev_mode gate removed** — Scanner runs always now. +5. **BotFights app** — Added as web-only app with SVG icon. Opens in new tab (X-Frame-Options blocks iframe). +6. **L484 web apps** — Added 6 web-only apps: NWNN, 484 Kitchen, Call the Operator, Arch Presentation, Syntropy Institute, T-0. L484 category in marketplace. +7. **Kiosk mode** — `/kiosk` route added, `setup-kiosk.sh` installs systemd service, systemd units in image-recipe/configs/. No full-screen iframe overlay — uses standard appLauncher. +8. **AIUI first-install fix** — nginx `try_files` changed to `=404`, Chat.vue probes AIUI availability before loading iframe. +9. **Web-only apps in My Apps** — Injected synthetic PackageDataEntry objects in Apps.vue. Web-only apps sorted first (alphabetically before container apps). No uninstall/start/stop buttons. Launch uses appLauncher with correct URLs. + +## Pending + +1. **Nostr NIP-07 login for containers** — Sign into container apps using onboarding Nostr keys. Not started. +2. **App sideloading** — Settings page to load apps via Docker/OCI image URL. Not started. +3. **Encrypted Nostr peer handshake (NIP-04/NIP-44)** — Exchange Tor onion addresses via encrypted DMs instead of public relay events. Not started. Currently onion addresses are published in plaintext on relays. +4. **Third server deploy** — archipelago-3.tail2b6225.ts.net needs SSH key setup and first deploy. +5. **Kiosk auto-start on servers** — setup-kiosk.sh exists but needs to be run on each server that has a display attached. Not confirmed running on .228. +6. **Deploy to .198** — Secondary server not yet deployed with latest changes. diff --git a/.claude/memory/second-server.md b/.claude/memory/second-server.md new file mode 100644 index 00000000..626062e0 --- /dev/null +++ b/.claude/memory/second-server.md @@ -0,0 +1,23 @@ +--- +name: second-dev-server +description: Second dev server accessible via Tailscale at archipelago-2.tail2b6225.ts.net, Ryzen 7 7840U, 14GB RAM +type: project +--- + +- Hostname: archipelago-2.tail2b6225.ts.net (Tailscale) +- SSH: `ssh -i ~/.ssh/archipelago-deploy archipelago@archipelago-2.tail2b6225.ts.net` +- Password: ThunderDome6574839201! +- CPU: AMD Ryzen 7 7840U (faster than primary i3-8100T) +- RAM: 14GB +- Disk: 916GB NVMe +- OS: Debian 12 (Bookworm) x86_64 +- Has: Podman 4.3.1, Node.js v20.20.1, Rust 1.94.0, Nginx 1.22.1 +- Swap: 4GB configured +- Deploy: `ARCHIPELAGO_TARGET="archipelago@archipelago-2.tail2b6225.ts.net" ./scripts/deploy-to-target.sh --live` +- Does NOT use OAuth proxy — uses standard ANTHROPIC_API_KEY for Claude/AIUI +- First-boot containers created on 2026-03-11 (Bitcoin Knots, LND, Fedimint, PhotoPrism, Ollama, etc.) + +## Pending Fixes for Next Deploy +- **AIUI MIME type error**: Nginx needs a `/aiui/` location block serving correct MIME types for JS files. Currently JS files get wrong content-type causing module load failures. +- **Self-signed cert warnings**: Expected on fresh deploy, not a bug. +- **Container connection errors in AIUI console**: Expected until all containers finish starting and syncing. diff --git a/.claude/memory/third-server.md b/.claude/memory/third-server.md new file mode 100644 index 00000000..6b7f3b07 --- /dev/null +++ b/.claude/memory/third-server.md @@ -0,0 +1,12 @@ +--- +name: third-dev-server +description: Third dev server accessible via Tailscale at archipelago-3.tail2b6225.ts.net, password ThisIsWeb54321@ +type: project +--- + +- Hostname: archipelago-3.tail2b6225.ts.net (Tailscale) +- SSH: `sshpass -p 'ThisIsWeb54321@' ssh -o StrictHostKeyChecking=no archipelago@archipelago-3.tail2b6225.ts.net` +- Password: ThisIsWeb54321@ +- Deploy: `ARCHIPELAGO_TARGET="archipelago@archipelago-3.tail2b6225.ts.net" ./scripts/deploy-to-target.sh --live` +- SSH key NOT yet installed — need to copy `~/.ssh/archipelago-deploy.pub` manually +- Added 2026-03-11 diff --git a/.claude/memory/web-only-apps.md b/.claude/memory/web-only-apps.md new file mode 100644 index 00000000..8b2ac365 --- /dev/null +++ b/.claude/memory/web-only-apps.md @@ -0,0 +1,34 @@ +--- +name: web-only-apps +description: Web-only apps (no container) — L484 category, BotFights, IndieHub. Iframe compatibility, nginx proxying, My Apps injection. +type: project +--- + +## Web-Only Apps (added 2026-03-11) + +These apps are external websites embedded via iframe — no Docker container. They show as "installed" in both the marketplace and My Apps. + +### L484 Category +- **NWNN** (nwnn.l484.com) — News aggregator. No X-Frame-Options. Works in iframe directly. +- **484 Kitchen** (484.kitchen) — K484 platform. X-Frame-Options: SAMEORIGIN. Proxied via `/ext/484-kitchen/`. +- **Call the Operator** (cta.tx1138.com) — Decentralization portal. No X-Frame-Options. Works in iframe directly. +- **Arch Presentation** (present.l484.com) — Archipelago presentation. X-Frame-Options: SAMEORIGIN. Proxied via `/ext/arch-presentation/`. +- **Syntropy Institute** (syntropy.institute) — Medicine Reimagined. No X-Frame-Options. Works in iframe directly. +- **T-0** (teeminuszero.net) — Decentralization documentary. No X-Frame-Options. Works in iframe directly. + +### Other Web-Only Apps +- **BotFights** (botfights.net) — X-Frame-Options: SAMEORIGIN + CSP + COEP/COOP/CORP. Proxied via `/ext/botfights/`. Nginx strips all blocking headers. +- **IndeeHub** (archipelago.indeehub.studio) — No X-Frame-Options. Works in iframe directly. + +### Nginx External Proxies +Sites with X-Frame-Options get reverse-proxied through nginx at `/ext/{app-id}/`: +- `proxy_hide_header X-Frame-Options` strips upstream header +- `add_header X-Content-Type-Options "nosniff" always` prevents server-level X-Frame-Options inheritance +- BotFights also strips `Cross-Origin-Embedder-Policy`, `Cross-Origin-Opener-Policy`, `Cross-Origin-Resource-Policy` +- Proxy locations in both HTTP and HTTPS server blocks of nginx-archipelago.conf + +### Frontend Implementation +- **appLauncher.ts**: `EXTERNAL_PROXY` map rewrites external URLs to proxy paths in `toEmbeddableUrl()` +- **Apps.vue**: `WEB_ONLY_APPS` constant with synthetic `PackageDataEntry` objects. Sorted first alphabetically. No uninstall/start/stop buttons. +- **Marketplace.vue**: `dockerImage: ''` + `webUrl` in `getCuratedAppList()`. L484 category. +- **Icons**: `neode-ui/public/assets/img/app-icons/{app-id}.png` (or .svg) diff --git a/RELEASE-NOTES-v0.5.0-beta.md b/RELEASE-NOTES-v0.5.0-beta.md new file mode 100644 index 00000000..55c4a0e9 --- /dev/null +++ b/RELEASE-NOTES-v0.5.0-beta.md @@ -0,0 +1,84 @@ +# Archipelago v0.5.0-beta Release Notes + +**Release Date**: March 2026 +**Target Platform**: Debian 12 (Bookworm) — x86_64 and ARM64 + +## Overview + +This is the first public beta of Archipelago, a self-sovereign Bitcoin Node OS. Flash it to a USB, install on any x86_64 or ARM64 machine, and manage your personal server through a modern web interface. + +## What's Included + +### Core System +- Rust backend with RPC API and WebSocket real-time updates +- Vue 3 frontend with glassmorphism UI design +- Automated Podman container management +- Nginx reverse proxy with HTTPS (self-signed cert) +- Tor hidden services for all apps + +### App Store (16+ Apps) +- **Bitcoin Stack**: Bitcoin Knots, Electrs, LND, BTCPay Server, Mempool, Fedimint +- **Storage**: File Browser, Immich, PhotoPrism +- **Productivity**: Penpot, SearXNG +- **AI**: Ollama (local LLMs) +- **Network**: Nostr Relay, Nginx Proxy Manager, Tailscale, Home Assistant +- **Platform**: IndeedHub + +### Security +- AES-256-GCM encrypted secrets on disk +- Session management: 24h inactivity expiry, max 5 concurrent sessions +- TOTP two-factor authentication with backup codes +- Container hardening: read-only root, no new privileges, dropped capabilities +- Pinned container image versions (no `:latest` tags) +- Login rate limiting (5 attempts per 60 seconds per IP) +- Path traversal prevention (nginx + client-side) +- Cookie-based auth (no tokens in URLs) + +### Identity & Web5 +- Decentralized Identifier (DID) generation +- Identity backup/restore +- Nostr relay support + +### Performance +- Backend startup: ~100ms +- Frontend bundle: ~105 KB gzipped +- WebSocket heartbeat with 30s ping/pong +- Exponential backoff reconnection (max 30s) +- Real-time install progress via WebSocket +- Server-side 5-minute inactivity timeout for stale connections + +## Known Issues + +1. **ARM64 ISO**: ARM64 builds may require manual testing — primary testing is on x86_64 +2. **Bitcoin Initial Sync**: First blockchain sync takes 1-7 days depending on hardware +3. **Self-signed HTTPS**: Browser shows certificate warning on first visit (expected) +4. **Restore from Backup**: Not yet implemented in onboarding flow +5. **Connect to Existing Server**: Not yet implemented in onboarding flow +6. **Immich**: Stack installation may take 5+ minutes due to multiple container images +7. **Memory**: Running all apps simultaneously requires 16+ GB RAM +8. **Disk Space**: Full Bitcoin node + all apps requires 800+ GB + +## Upgrade Path + +This is a beta release. No upgrade path from beta to stable is guaranteed. Back up your data before installing. + +## System Requirements + +| Component | Minimum | Recommended | +|-----------|---------|-------------| +| CPU | 4 cores | 8+ cores | +| RAM | 16 GB | 32 GB | +| Storage | 500 GB SSD | 2 TB NVMe | +| Network | Ethernet | Gigabit Ethernet | + +## Getting Started + +1. Download the ISO for your architecture +2. Flash to USB with balenaEtcher or `dd` +3. Boot from USB on target hardware +4. Auto-installer runs — follow on-screen prompts +5. After reboot, navigate to `http://` in your browser +6. Complete the onboarding wizard +7. Start installing apps from the App Store + +See [User Guide](docs/user-guide.md) for detailed instructions. diff --git a/core/.cargo/config.toml b/core/.cargo/config.toml new file mode 100644 index 00000000..adbdfd7c --- /dev/null +++ b/core/.cargo/config.toml @@ -0,0 +1,22 @@ +# Cargo configuration for Archipelago cross-compilation +# +# Native builds (x86_64 on x86_64) work automatically. +# ARM64 cross-compilation requires the aarch64-unknown-linux-gnu toolchain. +# +# Install the target: +# rustup target add aarch64-unknown-linux-gnu +# +# Install the cross-linker (Debian/Ubuntu): +# sudo apt install gcc-aarch64-linux-gnu +# +# Build for ARM64: +# cargo build --release --target aarch64-unknown-linux-gnu + +[target.aarch64-unknown-linux-gnu] +linker = "aarch64-linux-gnu-gcc" + +# OpenSSL cross-compilation environment (set before building) +# These are automatically set by the build scripts but documented here: +# OPENSSL_DIR=/usr/aarch64-linux-gnu +# PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig +# PKG_CONFIG_ALLOW_CROSS=1 diff --git a/core/Cargo.lock b/core/Cargo.lock index 82c05b6c..186a94f3 100644 --- a/core/Cargo.lock +++ b/core/Cargo.lock @@ -2,6 +2,12 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + [[package]] name = "aead" version = "0.5.2" @@ -76,8 +82,10 @@ dependencies = [ "bs58", "chacha20poly1305", "chrono", + "curve25519-dalek", "data-encoding", "ed25519-dalek", + "flate2", "futures-util", "hex", "http-body 1.0.1", @@ -94,6 +102,8 @@ dependencies = [ "serde_json", "serde_yaml", "sha2", + "tar", + "tempfile", "thiserror 1.0.69", "tokio", "tokio-test", @@ -161,9 +171,11 @@ dependencies = [ "aes-gcm", "anyhow", "chrono", + "hex", "log", "rand 0.8.5", "serde", + "serde_json", "tempfile", "thiserror 1.0.69", "tokio", @@ -223,7 +235,7 @@ dependencies = [ "futures-util", "js-sys", "tokio", - "tokio-rustls", + "tokio-rustls 0.26.4", "tokio-socks", "tokio-tungstenite 0.26.2", "url", @@ -519,6 +531,15 @@ dependencies = [ "libc", ] +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + [[package]] name = "crypto-common" version = "0.1.7" @@ -685,33 +706,39 @@ version = "0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" +[[package]] +name = "filetime" +version = "0.2.27" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f98844151eee8917efc50bd9e8318cb963ae8b297431495d3f758616ea5c57db" +dependencies = [ + "cfg-if", + "libc", + "libredox", +] + [[package]] name = "find-msvc-tools" version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8591b0bcc8a98a64310a2fae1bb3e9b8564dd10e381e6e28010fde8e8e8568db" +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + [[package]] name = "fnv" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" -[[package]] -name = "foreign-types" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" -dependencies = [ - "foreign-types-shared", -] - -[[package]] -name = "foreign-types-shared" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" - [[package]] name = "form_urlencoded" version = "1.2.2" @@ -1062,16 +1089,17 @@ dependencies = [ ] [[package]] -name = "hyper-tls" -version = "0.5.0" +name = "hyper-rustls" +version = "0.24.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6183ddfa99b85da61a140bea0efc93fdf56ceaa041b37d553518030827f9905" +checksum = "ec3efd23720e2049821a693cbc7e65ea87c72f1c58ff2f9522ff332b1491e590" dependencies = [ - "bytes", + "futures-util", + "http 0.2.12", "hyper 0.14.32", - "native-tls", + "rustls 0.21.12", "tokio", - "tokio-native-tls", + "tokio-rustls 0.24.1", ] [[package]] @@ -1335,6 +1363,18 @@ version = "0.2.180" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bcc35a38544a891a5f7c865aca548a982ccb3b8650a5b06d0fd33a10283c56fc" +[[package]] +name = "libredox" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1744e39d1d6a9948f4f388969627434e31128196de472883b39f148769bfe30a" +dependencies = [ + "bitflags 2.10.0", + "libc", + "plain", + "redox_syscall 0.7.3", +] + [[package]] name = "linux-raw-sys" version = "0.11.0" @@ -1389,6 +1429,16 @@ version = "0.3.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + [[package]] name = "mio" version = "1.1.1" @@ -1410,23 +1460,6 @@ dependencies = [ "pxfm", ] -[[package]] -name = "native-tls" -version = "0.2.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87de3442987e9dbec73158d5c715e7ad9072fda936bb03d19d7fa10e00520f0e" -dependencies = [ - "libc", - "log", - "openssl", - "openssl-probe", - "openssl-sys", - "schannel", - "security-framework", - "security-framework-sys", - "tempfile", -] - [[package]] name = "negentropy" version = "0.5.0" @@ -1540,50 +1573,6 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" -[[package]] -name = "openssl" -version = "0.10.75" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08838db121398ad17ab8531ce9de97b244589089e290a384c900cb9ff7434328" -dependencies = [ - "bitflags 2.10.0", - "cfg-if", - "foreign-types", - "libc", - "once_cell", - "openssl-macros", - "openssl-sys", -] - -[[package]] -name = "openssl-macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "openssl-probe" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e" - -[[package]] -name = "openssl-sys" -version = "0.9.111" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82cab2d520aa75e3c58898289429321eb788c3106963d0dc886ec7a5f4adc321" -dependencies = [ - "cc", - "libc", - "pkg-config", - "vcpkg", -] - [[package]] name = "parking_lot" version = "0.12.5" @@ -1602,7 +1591,7 @@ checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" dependencies = [ "cfg-if", "libc", - "redox_syscall", + "redox_syscall 0.5.18", "smallvec", "windows-link", ] @@ -1657,10 +1646,10 @@ dependencies = [ ] [[package]] -name = "pkg-config" -version = "0.3.32" +name = "plain" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" +checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6" [[package]] name = "poly1305" @@ -1810,6 +1799,15 @@ dependencies = [ "bitflags 2.10.0", ] +[[package]] +name = "redox_syscall" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce70a74e890531977d37e532c34d45e9055d2409ed08ddba14529471ed0be16" +dependencies = [ + "bitflags 2.10.0", +] + [[package]] name = "regex" version = "1.12.2" @@ -1854,15 +1852,15 @@ dependencies = [ "http 0.2.12", "http-body 0.4.6", "hyper 0.14.32", - "hyper-tls", + "hyper-rustls", "ipnet", "js-sys", "log", "mime", - "native-tls", "once_cell", "percent-encoding", "pin-project-lite", + "rustls 0.21.12", "rustls-pemfile", "serde", "serde_json", @@ -1870,13 +1868,14 @@ dependencies = [ "sync_wrapper", "system-configuration 0.5.1", "tokio", - "tokio-native-tls", + "tokio-rustls 0.24.1", "tokio-socks", "tower-service", "url", "wasm-bindgen", "wasm-bindgen-futures", "web-sys", + "webpki-roots 0.25.4", "winreg", ] @@ -1916,6 +1915,18 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "rustls" +version = "0.21.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f56a14d1f48b391359b22f731fd4bd7e43c97f3c50eee276f3aa09c94784d3e" +dependencies = [ + "log", + "ring", + "rustls-webpki 0.101.7", + "sct", +] + [[package]] name = "rustls" version = "0.23.36" @@ -1925,7 +1936,7 @@ dependencies = [ "once_cell", "ring", "rustls-pki-types", - "rustls-webpki", + "rustls-webpki 0.103.9", "subtle", "zeroize", ] @@ -1948,6 +1959,16 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rustls-webpki" +version = "0.101.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765" +dependencies = [ + "ring", + "untrusted", +] + [[package]] name = "rustls-webpki" version = "0.103.9" @@ -1980,15 +2001,6 @@ dependencies = [ "cipher", ] -[[package]] -name = "schannel" -version = "0.1.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "891d81b926048e76efe18581bf793546b4c0eaf8448d72be8de2bbee5fd166e1" -dependencies = [ - "windows-sys 0.61.2", -] - [[package]] name = "scopeguard" version = "1.2.0" @@ -2007,6 +2019,16 @@ dependencies = [ "sha2", ] +[[package]] +name = "sct" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da046153aa2352493d6cb7da4b6e5c0c057d8a1d0a9aa8560baffdd945acd414" +dependencies = [ + "ring", + "untrusted", +] + [[package]] name = "secp256k1" version = "0.29.1" @@ -2027,29 +2049,6 @@ dependencies = [ "cc", ] -[[package]] -name = "security-framework" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" -dependencies = [ - "bitflags 2.10.0", - "core-foundation", - "core-foundation-sys", - "libc", - "security-framework-sys", -] - -[[package]] -name = "security-framework-sys" -version = "2.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cc1f0cbffaac4852523ce30d8bd3c5cdc873501d96ff467ca09b6767bb8cd5c0" -dependencies = [ - "core-foundation-sys", - "libc", -] - [[package]] name = "semver" version = "1.0.27" @@ -2200,6 +2199,12 @@ dependencies = [ "rand_core 0.6.4", ] +[[package]] +name = "simd-adler32" +version = "0.3.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" + [[package]] name = "slab" version = "0.4.11" @@ -2324,6 +2329,17 @@ dependencies = [ "libc", ] +[[package]] +name = "tar" +version = "0.4.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d863878d212c87a19c1a610eb53bb01fe12951c0501cf5a0d65f724914a667a" +dependencies = [ + "filetime", + "libc", + "xattr", +] + [[package]] name = "tempfile" version = "3.24.0" @@ -2449,12 +2465,12 @@ dependencies = [ ] [[package]] -name = "tokio-native-tls" -version = "0.3.1" +name = "tokio-rustls" +version = "0.24.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2" +checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081" dependencies = [ - "native-tls", + "rustls 0.21.12", "tokio", ] @@ -2464,7 +2480,7 @@ version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls", + "rustls 0.23.36", "tokio", ] @@ -2522,10 +2538,10 @@ checksum = "7a9daff607c6d2bf6c16fd681ccb7eecc83e4e2cdc1ca067ffaadfca5de7f084" dependencies = [ "futures-util", "log", - "rustls", + "rustls 0.23.36", "rustls-pki-types", "tokio", - "tokio-rustls", + "tokio-rustls 0.26.4", "tungstenite 0.26.2", "webpki-roots 0.26.11", ] @@ -2736,7 +2752,7 @@ dependencies = [ "httparse", "log", "rand 0.9.2", - "rustls", + "rustls 0.23.36", "rustls-pki-types", "sha1", "thiserror 2.0.18", @@ -2834,12 +2850,6 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" -[[package]] -name = "vcpkg" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" - [[package]] name = "version_check" version = "0.9.5" @@ -2939,6 +2949,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "webpki-roots" +version = "0.25.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5f20c57d8d7db6d3b86154206ae5d8fba62dd39573114de97c2cb0578251f8e1" + [[package]] name = "webpki-roots" version = "0.26.11" @@ -3289,6 +3305,16 @@ version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + [[package]] name = "yoke" version = "0.8.1" diff --git a/core/archipelago/src/api/rpc/auth.rs b/core/archipelago/src/api/rpc/auth.rs index e41291e3..9d9c040e 100644 --- a/core/archipelago/src/api/rpc/auth.rs +++ b/core/archipelago/src/api/rpc/auth.rs @@ -38,6 +38,7 @@ impl RpcHandler { pub(super) async fn handle_auth_change_password( &self, params: Option, + session_token: &Option, ) -> Result { let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; let current_password = params @@ -57,7 +58,12 @@ impl RpcHandler { .change_password(current_password, new_password, also_change_ssh) .await?; - Ok(serde_json::json!({ "success": true })) + // Session rotation: invalidate all other sessions, rotate the caller's session + if let Some(token) = session_token { + self.session_store.invalidate_all_except(token).await; + } + + Ok(serde_json::json!({ "success": true, "session_rotated": true })) } pub(super) async fn handle_auth_onboarding_complete(&self) -> Result { diff --git a/core/archipelago/src/api/rpc/credentials.rs b/core/archipelago/src/api/rpc/credentials.rs index 4c626f46..915bfebb 100644 --- a/core/archipelago/src/api/rpc/credentials.rs +++ b/core/archipelago/src/api/rpc/credentials.rs @@ -57,13 +57,15 @@ impl RpcHandler { ) .await?; + let status = if credentials::is_revoked(&vc) { "revoked" } else { "active" }; + Ok(serde_json::json!({ "id": vc.id, "issuer": vc.issuer, - "subject": vc.subject, + "subject": vc.credential_subject.id, "type": vc.credential_type, - "issued_at": vc.issued_at, - "status": vc.status, + "issued_at": vc.issuance_date, + "status": status, })) } @@ -97,10 +99,12 @@ impl RpcHandler { }) })?; + let status = if credentials::is_revoked(vc) { "revoked" } else { "active" }; + Ok(serde_json::json!({ "id": vc.id, "valid": valid, - "status": vc.status, + "status": status, })) } @@ -118,15 +122,17 @@ impl RpcHandler { let items: Vec = creds .into_iter() .map(|c| { + let status = if credentials::is_revoked(&c) { "revoked" } else { "active" }; serde_json::json!({ + "@context": c.context, "id": c.id, - "issuer": c.issuer, - "subject": c.subject, "type": c.credential_type, - "claims": c.claims, - "issued_at": c.issued_at, - "expires_at": c.expires_at, - "status": c.status, + "issuer": c.issuer, + "credentialSubject": c.credential_subject, + "issuanceDate": c.issuance_date, + "expirationDate": c.expiration_date, + "proof": c.proof, + "status": status, }) }) .collect(); @@ -147,4 +153,86 @@ impl RpcHandler { credentials::revoke_credential(&self.config.data_dir, id).await?; Ok(serde_json::json!({ "ok": true })) } + + /// Create a Verifiable Presentation bundling selected credentials. + pub(super) async fn handle_identity_create_presentation( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let holder_id = params + .get("holder_id") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing holder_id"))?; + let credential_ids: Vec<&str> = params + .get("credential_ids") + .and_then(|v| v.as_array()) + .ok_or_else(|| anyhow::anyhow!("Missing credential_ids array"))? + .iter() + .filter_map(|v| v.as_str()) + .collect(); + + if credential_ids.is_empty() { + return Err(anyhow::anyhow!("credential_ids must not be empty")); + } + + let manager = IdentityManager::new(&self.config.data_dir).await?; + let holder_record = manager.get(holder_id).await?; + let holder_did = holder_record.did.clone(); + + let store = credentials::load_credentials(&self.config.data_dir).await?; + + let data_dir = self.config.data_dir.clone(); + let sign_id = holder_id.to_string(); + + let vp = credentials::create_presentation( + &holder_did, + &credential_ids, + &store.credentials, + |bytes| { + let hex_msg = hex::encode(bytes); + tokio::task::block_in_place(|| { + let rt = tokio::runtime::Handle::current(); + rt.block_on(async { + let mgr = IdentityManager::new(&data_dir).await?; + mgr.sign(&sign_id, hex_msg.as_bytes()).await + }) + }) + }, + )?; + + Ok(serde_json::to_value(&vp)?) + } + + /// Verify a Verifiable Presentation: check holder proof and all embedded credentials. + pub(super) async fn handle_identity_verify_presentation( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let presentation = params + .get("presentation") + .ok_or_else(|| anyhow::anyhow!("Missing presentation"))?; + + let vp: credentials::VerifiablePresentation = + serde_json::from_value(presentation.clone())?; + + let data_dir = self.config.data_dir.clone(); + let result = credentials::verify_presentation(&vp, |did, bytes, signature| { + let hex_msg = hex::encode(bytes); + tokio::task::block_in_place(|| { + let rt = tokio::runtime::Handle::current(); + rt.block_on(async { + let mgr = IdentityManager::new(&data_dir).await?; + mgr.verify(did, hex_msg.as_bytes(), signature).await + }) + }) + })?; + + Ok(serde_json::json!({ + "valid": result.valid, + "holder_valid": result.holder_valid, + "credentials": result.credentials, + })) + } } diff --git a/core/archipelago/src/api/rpc/dwn.rs b/core/archipelago/src/api/rpc/dwn.rs index 77153612..f7d1acd5 100644 --- a/core/archipelago/src/api/rpc/dwn.rs +++ b/core/archipelago/src/api/rpc/dwn.rs @@ -1,4 +1,5 @@ use super::RpcHandler; +use crate::network::dwn_store::{DwnStore, MessageQuery, ProtocolDefinition}; use crate::network::dwn_sync; use crate::peers; use anyhow::Result; @@ -12,13 +13,18 @@ impl RpcHandler { version: String::new(), }); + let store = DwnStore::new(&self.config.data_dir).await?; + let stats = store.stats().await?; + Ok(serde_json::json!({ "running": server_status.running, "version": server_status.version, "sync_status": sync_state.status, "last_sync": sync_state.last_sync, "messages_synced": sync_state.messages_synced, - "storage_bytes": sync_state.storage_bytes, + "storage_bytes": stats.total_bytes, + "message_count": stats.message_count, + "protocol_count": stats.protocol_count, "registered_protocols": sync_state.registered_protocols, "peer_sync_targets": sync_state.peer_sync_targets, })) @@ -26,7 +32,6 @@ impl RpcHandler { /// Trigger DWN sync with connected peers. pub(super) async fn handle_dwn_sync(&self) -> Result { - // Get list of connected peers' onion addresses let peer_list = peers::load_peers(&self.config.data_dir).await?; let onions: Vec = peer_list .iter() @@ -42,4 +47,97 @@ impl RpcHandler { "messages_synced": state.messages_synced, })) } + + /// Register a DWN protocol. + pub(super) async fn handle_dwn_register_protocol( + &self, + params: &serde_json::Value, + ) -> Result { + let protocol = params["protocol"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing 'protocol' parameter"))?; + let published = params["published"].as_bool().unwrap_or(false); + + let definition = ProtocolDefinition { + protocol: protocol.to_string(), + published, + types: params + .get("types") + .and_then(|v| serde_json::from_value(v.clone()).ok()) + .unwrap_or_default(), + structure: params + .get("structure") + .and_then(|v| serde_json::from_value(v.clone()).ok()) + .unwrap_or_default(), + date_registered: chrono::Utc::now().to_rfc3339(), + }; + + let store = DwnStore::new(&self.config.data_dir).await?; + store.register_protocol(&definition).await?; + + Ok(serde_json::json!({"registered": true, "protocol": protocol})) + } + + /// List registered DWN protocols. + pub(super) async fn handle_dwn_list_protocols(&self) -> Result { + let store = DwnStore::new(&self.config.data_dir).await?; + let protocols = store.list_protocols().await?; + Ok(serde_json::json!({"protocols": protocols})) + } + + /// Remove a DWN protocol. + pub(super) async fn handle_dwn_remove_protocol( + &self, + params: &serde_json::Value, + ) -> Result { + let protocol = params["protocol"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing 'protocol' parameter"))?; + + let store = DwnStore::new(&self.config.data_dir).await?; + let removed = store.remove_protocol(protocol).await?; + + Ok(serde_json::json!({"removed": removed, "protocol": protocol})) + } + + /// Query DWN messages. + pub(super) async fn handle_dwn_query_messages( + &self, + params: &serde_json::Value, + ) -> Result { + let query = MessageQuery { + protocol: params["protocol"].as_str().map(|s| s.to_string()), + schema: params["schema"].as_str().map(|s| s.to_string()), + author: params["author"].as_str().map(|s| s.to_string()), + date_from: params["dateFrom"].as_str().map(|s| s.to_string()), + date_to: params["dateTo"].as_str().map(|s| s.to_string()), + limit: params["limit"].as_u64().map(|n| n as usize), + }; + + let store = DwnStore::new(&self.config.data_dir).await?; + let messages = store.query_messages(&query).await?; + + Ok(serde_json::json!({"messages": messages, "count": messages.len()})) + } + + /// Write a DWN message. + pub(super) async fn handle_dwn_write_message( + &self, + params: &serde_json::Value, + ) -> Result { + let author = params["author"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing 'author' parameter"))?; + let protocol = params["protocol"].as_str(); + let schema = params["schema"].as_str(); + let data_format = params["dataFormat"].as_str(); + let data = params.get("data").cloned(); + + let store = DwnStore::new(&self.config.data_dir).await?; + let message = store + .write_message(author, protocol, schema, data_format, data) + .await?; + + Ok(serde_json::json!({"written": true, "record_id": message.record_id})) + } } diff --git a/core/archipelago/src/api/rpc/federation.rs b/core/archipelago/src/api/rpc/federation.rs new file mode 100644 index 00000000..5b646813 --- /dev/null +++ b/core/archipelago/src/api/rpc/federation.rs @@ -0,0 +1,326 @@ +use super::RpcHandler; +use crate::federation::{self, FederatedNode, TrustLevel}; +use crate::identity; +use anyhow::Result; +use tracing::info; + +impl RpcHandler { + /// federation.invite — Generate an invite code containing our DID + onion for a peer. + pub(super) async fn handle_federation_invite(&self) -> Result { + let (data, _) = self.state_manager.get_snapshot().await; + let did = identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?; + let onion = data + .server_info + .tor_address + .clone() + .unwrap_or_default(); + let pubkey = data.server_info.pubkey.clone(); + + if onion.is_empty() { + anyhow::bail!("Tor address not available. Tor may not be running."); + } + + let code = federation::create_invite(&self.config.data_dir, &did, &onion, &pubkey).await?; + + info!(did = %did, "Generated federation invite"); + Ok(serde_json::json!({ + "code": code, + "did": did, + "onion": onion, + })) + } + + /// federation.join — Accept an invite code and establish federation with the remote node. + pub(super) async fn handle_federation_join( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let code = params + .get("code") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'code' parameter"))?; + + let (data, _) = self.state_manager.get_snapshot().await; + let local_did = identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?; + let local_onion = data.server_info.tor_address.clone().unwrap_or_default(); + let local_pubkey = data.server_info.pubkey.clone(); + + let node = federation::accept_invite( + &self.config.data_dir, + code, + &local_did, + &local_onion, + &local_pubkey, + ) + .await?; + + info!(peer_did = %node.did, "Joined federation with peer"); + Ok(serde_json::json!({ + "joined": true, + "node": { + "did": node.did, + "onion": node.onion, + "pubkey": node.pubkey, + "trust_level": node.trust_level.to_string(), + } + })) + } + + /// federation.list-nodes — List all federated nodes with their status and last state. + pub(super) async fn handle_federation_list_nodes(&self) -> Result { + let nodes = federation::load_nodes(&self.config.data_dir).await?; + + let nodes_json: Vec = nodes + .iter() + .map(|n| { + let mut obj = serde_json::json!({ + "did": n.did, + "pubkey": n.pubkey, + "onion": n.onion, + "trust_level": n.trust_level.to_string(), + "added_at": n.added_at, + }); + if let Some(name) = &n.name { + obj["name"] = serde_json::json!(name); + } + if let Some(last_seen) = &n.last_seen { + obj["last_seen"] = serde_json::json!(last_seen); + } + if let Some(state) = &n.last_state { + obj["last_state"] = serde_json::to_value(state).unwrap_or_default(); + } + obj + }) + .collect(); + + Ok(serde_json::json!({ "nodes": nodes_json })) + } + + /// federation.remove-node — Remove a node from the federation by DID. + pub(super) async fn handle_federation_remove_node( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let did = params + .get("did") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'did' parameter"))?; + + let nodes = federation::remove_node(&self.config.data_dir, did).await?; + info!(did = %did, "Removed node from federation"); + + Ok(serde_json::json!({ + "removed": true, + "nodes_remaining": nodes.len(), + })) + } + + /// federation.set-trust — Change trust level for a federated node. + pub(super) async fn handle_federation_set_trust( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let did = params + .get("did") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'did' parameter"))?; + let trust_str = params + .get("trust_level") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'trust_level' parameter"))?; + + let trust = match trust_str { + "trusted" => TrustLevel::Trusted, + "observer" => TrustLevel::Observer, + "untrusted" => TrustLevel::Untrusted, + _ => anyhow::bail!("Invalid trust level: {} (expected trusted/observer/untrusted)", trust_str), + }; + + federation::set_trust_level(&self.config.data_dir, did, trust).await?; + + Ok(serde_json::json!({ + "updated": true, + "did": did, + "trust_level": trust.to_string(), + })) + } + + /// federation.sync-state — Manually trigger state sync with all federated peers. + pub(super) async fn handle_federation_sync_state(&self) -> Result { + let nodes = federation::load_nodes(&self.config.data_dir).await?; + + if nodes.is_empty() { + return Ok(serde_json::json!({ + "synced": 0, + "failed": 0, + "results": [], + })); + } + + let (data, _) = self.state_manager.get_snapshot().await; + let local_did = identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?; + + let identity_dir = self.config.data_dir.join("identity"); + let node_identity = identity::NodeIdentity::load_or_create(&identity_dir).await?; + + let mut synced = 0u32; + let mut failed = 0u32; + let mut results = Vec::new(); + + for node in &nodes { + if node.trust_level == TrustLevel::Untrusted { + continue; + } + + let did_clone = local_did.clone(); + match federation::sync_with_peer( + &self.config.data_dir, + node, + &did_clone, + |bytes| node_identity.sign(bytes), + ) + .await + { + Ok(state) => { + synced += 1; + results.push(serde_json::json!({ + "did": node.did, + "status": "ok", + "apps": state.apps.len(), + })); + } + Err(e) => { + failed += 1; + results.push(serde_json::json!({ + "did": node.did, + "status": "error", + "error": e.to_string(), + })); + } + } + } + + Ok(serde_json::json!({ + "synced": synced, + "failed": failed, + "results": results, + })) + } + + /// federation.get-state — Return this node's state snapshot (called by peers during sync). + pub(super) async fn handle_federation_get_state(&self) -> Result { + let (data, _) = self.state_manager.get_snapshot().await; + + // Build app statuses from package_data + let apps: Vec = data + .package_data + .iter() + .map(|(id, pkg)| federation::AppStatus { + id: id.clone(), + status: format!("{:?}", pkg.state).to_lowercase(), + version: Some(pkg.manifest.version.clone()), + }) + .collect(); + + let tor_active = data.server_info.tor_address.is_some(); + + let state = federation::build_local_state( + apps, 0.0, 0, 0, 0, 0, 0, tor_active, + ); + + Ok(serde_json::to_value(&state)?) + } + + /// federation.peer-joined — Called by a remote peer after they accept our invite. + pub(super) async fn handle_federation_peer_joined( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let did = params + .get("did") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'did'"))?; + let onion = params + .get("onion") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'onion'"))?; + let pubkey = params + .get("pubkey") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'pubkey'"))?; + + let nodes = federation::load_nodes(&self.config.data_dir).await?; + if nodes.iter().any(|n| n.did == did) { + return Ok(serde_json::json!({ "accepted": true, "already_known": true })); + } + + let node = FederatedNode { + did: did.to_string(), + pubkey: pubkey.to_string(), + onion: onion.to_string(), + name: None, + trust_level: TrustLevel::Trusted, + added_at: chrono::Utc::now().to_rfc3339(), + last_seen: None, + last_state: None, + }; + + federation::add_node(&self.config.data_dir, node).await?; + info!(peer_did = %did, "Peer joined our federation"); + + Ok(serde_json::json!({ "accepted": true })) + } + + /// federation.deploy-app — Deploy an app to a remote federated node. + pub(super) async fn handle_federation_deploy_app( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let peer_did = params + .get("did") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'did' (target node)"))?; + let app_id = params + .get("app_id") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing 'app_id'"))?; + let version = params + .get("version") + .and_then(|v| v.as_str()) + .unwrap_or("latest"); + let marketplace_url = params + .get("marketplace_url") + .and_then(|v| v.as_str()) + .unwrap_or(""); + + let nodes = federation::load_nodes(&self.config.data_dir).await?; + let peer = nodes + .iter() + .find(|n| n.did == peer_did) + .ok_or_else(|| anyhow::anyhow!("No federated node with DID {}", peer_did))?; + + let (data, _) = self.state_manager.get_snapshot().await; + let local_did = identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?; + + let identity_dir = self.config.data_dir.join("identity"); + let node_identity = identity::NodeIdentity::load_or_create(&identity_dir).await?; + + let result = federation::deploy_to_peer( + peer, + app_id, + version, + marketplace_url, + &local_did, + |bytes| node_identity.sign(bytes), + ) + .await?; + + info!(app = %app_id, peer = %peer_did, "Deployed app to federated peer"); + Ok(result) + } +} diff --git a/core/archipelago/src/api/rpc/handshake.rs b/core/archipelago/src/api/rpc/handshake.rs new file mode 100644 index 00000000..21a517ad --- /dev/null +++ b/core/archipelago/src/api/rpc/handshake.rs @@ -0,0 +1,138 @@ +use super::RpcHandler; +use crate::{nostr_handshake, peers}; +use anyhow::Result; + +impl RpcHandler { + /// Discover nodes (presence-only — returns Nostr pubkeys + DIDs, no onion addresses). + pub(super) async fn handle_handshake_discover(&self) -> Result { + let identity_dir = self.config.data_dir.join("identity"); + let nodes = nostr_handshake::discover_nodes( + &identity_dir, + &self.config.nostr_relays, + self.config.nostr_tor_proxy.as_deref(), + ) + .await?; + Ok(serde_json::json!({ "nodes": nodes })) + } + + /// Send encrypted connection request to a peer's Nostr pubkey. + /// Params: { recipient_nostr_pubkey } + pub(super) async fn handle_handshake_connect( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let recipient = params + .get("recipient_nostr_pubkey") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing recipient_nostr_pubkey"))?; + + let (data, _) = self.state_manager.get_snapshot().await; + let our_onion = data + .server_info + .tor_address + .as_deref() + .ok_or_else(|| anyhow::anyhow!("No Tor address available — is Tor running?"))?; + let our_node_pubkey = &data.server_info.pubkey; + let our_did = crate::identity::did_key_from_pubkey_hex(our_node_pubkey) + .unwrap_or_default(); + let our_version = &data.server_info.version; + let our_name = data.server_info.name.as_deref(); + + let identity_dir = self.config.data_dir.join("identity"); + nostr_handshake::send_connect_request( + &identity_dir, + recipient, + our_onion, + our_node_pubkey, + &our_did, + our_version, + our_name, + &self.config.nostr_relays, + self.config.nostr_tor_proxy.as_deref(), + ) + .await?; + + Ok(serde_json::json!({ "ok": true, "sent_to": recipient })) + } + + /// Poll for incoming encrypted handshake messages (connect requests/responses). + /// Auto-adds peers and auto-responds to requests. + pub(super) async fn handle_handshake_poll(&self) -> Result { + let identity_dir = self.config.data_dir.join("identity"); + let handshakes = nostr_handshake::poll_handshakes( + &identity_dir, + &self.config.nostr_relays, + self.config.nostr_tor_proxy.as_deref(), + None, // TODO: track last-seen timestamp to avoid re-processing + ) + .await?; + + let (data, _) = self.state_manager.get_snapshot().await; + let mut added_peers = Vec::new(); + + for hs in &handshakes { + let (onion, node_pubkey, name) = match &hs.message { + nostr_handshake::HandshakeMessage::ConnectRequest { + onion, + node_pubkey, + name, + .. + } => { + // Auto-respond with our details + if let Some(our_onion) = data.server_info.tor_address.as_deref() { + let our_did = crate::identity::did_key_from_pubkey_hex( + &data.server_info.pubkey, + ) + .unwrap_or_default(); + let _ = nostr_handshake::send_connect_response( + &identity_dir, + &hs.from_nostr_pubkey, + our_onion, + &data.server_info.pubkey, + &our_did, + &data.server_info.version, + data.server_info.name.as_deref(), + &self.config.nostr_relays, + self.config.nostr_tor_proxy.as_deref(), + ) + .await; + } + (onion.clone(), node_pubkey.clone(), name.clone()) + } + nostr_handshake::HandshakeMessage::ConnectResponse { + onion, + node_pubkey, + name, + .. + } => (onion.clone(), node_pubkey.clone(), name.clone()), + }; + + // Auto-add as peer + let peer = peers::KnownPeer { + onion, + pubkey: node_pubkey.clone(), + name, + added_at: Some(chrono::Utc::now().to_rfc3339()), + }; + let _ = peers::add_peer(&self.config.data_dir, peer).await; + added_peers.push(node_pubkey); + } + + let serialized: Vec = handshakes + .iter() + .map(|hs| { + serde_json::json!({ + "from_nostr_pubkey": hs.from_nostr_pubkey, + "message": hs.message, + "timestamp": hs.timestamp, + }) + }) + .collect(); + + Ok(serde_json::json!({ + "handshakes": serialized, + "added_peers": added_peers, + })) + } +} diff --git a/core/archipelago/src/api/rpc/identity.rs b/core/archipelago/src/api/rpc/identity.rs index 96c00d50..c33d0c16 100644 --- a/core/archipelago/src/api/rpc/identity.rs +++ b/core/archipelago/src/api/rpc/identity.rs @@ -2,7 +2,7 @@ use super::RpcHandler; use crate::identity_manager::{IdentityManager, IdentityPurpose}; -use anyhow::Result; +use anyhow::{Context, Result}; impl RpcHandler { /// List all identities with their default status. @@ -180,6 +180,101 @@ impl RpcHandler { Ok(serde_json::json!({ "valid": valid })) } + /// Resolve a DID to its W3C DID Document. + /// If no DID is provided, returns the node's own DID Document. + pub(super) async fn handle_identity_resolve_did( + &self, + params: Option, + ) -> Result { + let params = params.unwrap_or_default(); + + // If a DID is provided, resolve it; otherwise use the node's DID + let pubkey_hex = if let Some(did) = params.get("did").and_then(|v| v.as_str()) { + // Extract pubkey from did:key format + let pubkey_bytes = crate::identity::pubkey_bytes_from_did_key(did)?; + hex::encode(pubkey_bytes) + } else { + // Use node's own pubkey + let (data, _) = self.state_manager.get_snapshot().await; + data.server_info.pubkey.clone() + }; + + let document = crate::identity::did_document_from_pubkey_hex(&pubkey_hex)?; + Ok(document) + } + + /// Verify a DID Document: validate structure, check key material matches DID. + pub(super) async fn handle_identity_verify_did_document( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let document = params + .get("document") + .ok_or_else(|| anyhow::anyhow!("Missing required parameter: document"))?; + + // Validate required fields + let did = document["id"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("DID Document missing 'id' field"))?; + + let context = document["@context"] + .as_array() + .ok_or_else(|| anyhow::anyhow!("DID Document missing '@context' array"))?; + + let has_did_context = context.iter().any(|c| c.as_str() == Some("https://www.w3.org/ns/did/v1")); + if !has_did_context { + return Ok(serde_json::json!({ + "valid": false, + "errors": ["Missing required @context: https://www.w3.org/ns/did/v1"] + })); + } + + let verification_methods = document["verificationMethod"] + .as_array() + .ok_or_else(|| anyhow::anyhow!("DID Document missing 'verificationMethod' array"))?; + + if verification_methods.is_empty() { + return Ok(serde_json::json!({ + "valid": false, + "errors": ["verificationMethod array is empty"] + })); + } + + // Verify the DID matches the key material (for did:key method) + let mut errors: Vec = Vec::new(); + + if did.starts_with("did:key:") { + match crate::identity::pubkey_bytes_from_did_key(did) { + Ok(pubkey_bytes) => { + // Check that at least one verification method has matching key + let pubkey_multibase = format!("z{}", bs58::encode(&pubkey_bytes).into_string()); + let has_matching_key = verification_methods.iter().any(|vm| { + vm["publicKeyMultibase"].as_str() == Some(&pubkey_multibase) + }); + if !has_matching_key { + errors.push("No verificationMethod matches the DID's public key".to_string()); + } + } + Err(e) => { + errors.push(format!("Failed to extract pubkey from DID: {}", e)); + } + } + } + + // Check authentication is present + if document["authentication"].as_array().map_or(true, |a| a.is_empty()) { + errors.push("Missing or empty 'authentication' field".to_string()); + } + + Ok(serde_json::json!({ + "valid": errors.is_empty(), + "did": did, + "errors": errors, + "verification_methods": verification_methods.len(), + })) + } + /// Create a Nostr keypair linked to an identity. pub(super) async fn handle_identity_create_nostr_key( &self, @@ -221,4 +316,81 @@ impl RpcHandler { "signature": signature, })) } + + /// Resolve a remote peer's DID Document over Tor. + /// Queries the peer's /rpc/ endpoint for identity.resolve-did. + pub(super) async fn handle_identity_resolve_remote_did( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let onion = params + .get("onion") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing required parameter: onion"))?; + + // Build URL for peer's RPC endpoint over Tor + let host = if onion.ends_with(".onion") { + onion.to_string() + } else { + format!("{}.onion", onion) + }; + let url = format!("http://{}/rpc/", host); + + // Use SOCKS5 proxy to reach .onion address + let proxy = reqwest::Proxy::all("socks5h://127.0.0.1:9050") + .context("Failed to create Tor proxy")?; + let client = reqwest::Client::builder() + .proxy(proxy) + .timeout(std::time::Duration::from_secs(30)) + .build() + .context("Failed to build HTTP client")?; + + let rpc_body = serde_json::json!({ + "jsonrpc": "2.0", + "id": 1, + "method": "identity.resolve-did", + "params": {} + }); + + let resp = client + .post(&url) + .json(&rpc_body) + .send() + .await + .context("Failed to connect to peer over Tor")?; + + let body: serde_json::Value = resp + .json() + .await + .context("Failed to parse peer response")?; + + // Extract the DID Document from the RPC response + let document = body + .get("result") + .ok_or_else(|| anyhow::anyhow!("Peer returned error or missing result"))?; + + // Cache the resolved DID locally + let did = document["id"] + .as_str() + .unwrap_or("unknown"); + let cache_dir = self.config.data_dir.join("did-cache"); + tokio::fs::create_dir_all(&cache_dir).await.ok(); + let cache_file = cache_dir.join(format!("{}.json", onion.replace('.', "_"))); + let cache_entry = serde_json::json!({ + "document": document, + "resolved_at": chrono::Utc::now().to_rfc3339(), + "onion": onion, + }); + tokio::fs::write(&cache_file, serde_json::to_string_pretty(&cache_entry).unwrap_or_default()) + .await + .ok(); + + Ok(serde_json::json!({ + "document": document, + "did": did, + "resolved_from": onion, + "cached": true, + })) + } } diff --git a/core/archipelago/src/api/rpc/marketplace.rs b/core/archipelago/src/api/rpc/marketplace.rs new file mode 100644 index 00000000..6c8e3ced --- /dev/null +++ b/core/archipelago/src/api/rpc/marketplace.rs @@ -0,0 +1,128 @@ +use super::RpcHandler; +use crate::federation; +use crate::marketplace; +use crate::nostr_relays; +use anyhow::Result; +use tracing::info; + +impl RpcHandler { + /// marketplace.discover — Query Nostr relays for community app manifests. + pub(super) async fn handle_marketplace_discover(&self) -> Result { + // Load enabled relays + let relay_store = nostr_relays::load_relays(&self.config.data_dir).await?; + let relay_urls: Vec = relay_store + .relays + .iter() + .filter(|r| r.enabled) + .map(|r| r.url.clone()) + .collect(); + + // Load federated DIDs for trust scoring + let fed_nodes = federation::load_nodes(&self.config.data_dir).await.unwrap_or_default(); + let federated_dids: Vec = fed_nodes.iter().map(|n| n.did.clone()).collect(); + + let tor_proxy = std::env::var("ARCHIPELAGO_NOSTR_TOR_PROXY").ok(); + let apps = marketplace::discover( + &self.config.data_dir, + &relay_urls, + tor_proxy.as_deref(), + &federated_dids, + ) + .await?; + + Ok(serde_json::json!({ + "apps": apps, + "relay_count": relay_urls.len(), + })) + } + + /// marketplace.publish — Publish an app manifest to Nostr relays. + pub(super) async fn handle_marketplace_publish( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let manifest: marketplace::AppManifest = + serde_json::from_value(params).map_err(|e| anyhow::anyhow!("Invalid manifest: {}", e))?; + + // Validate before publishing + let issues = marketplace::validate_manifest(&manifest); + if !issues.is_empty() { + return Ok(serde_json::json!({ + "ok": false, + "errors": issues, + })); + } + + let relay_store = nostr_relays::load_relays(&self.config.data_dir).await?; + let relay_urls: Vec = relay_store + .relays + .iter() + .filter(|r| r.enabled) + .map(|r| r.url.clone()) + .collect(); + + let tor_proxy = std::env::var("ARCHIPELAGO_NOSTR_TOR_PROXY").ok(); + let event_id = marketplace::publish( + &self.config.data_dir, + &manifest, + &relay_urls, + tor_proxy.as_deref(), + ) + .await?; + + info!(app_id = %manifest.app_id, "Published app manifest"); + Ok(serde_json::json!({ + "ok": true, + "event_id": event_id, + "app_id": manifest.app_id, + "relays": relay_urls.len(), + })) + } + + /// marketplace.get-manifest — Get cached manifest for a specific app. + pub(super) async fn handle_marketplace_get_manifest( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let app_id = params + .get("app_id") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing required parameter: app_id"))?; + + let cache = marketplace::load_cache(&self.config.data_dir).await?; + let app = cache.apps.iter().find(|a| a.manifest.app_id == app_id); + + match app { + Some(discovered) => Ok(serde_json::to_value(discovered)?), + None => Ok(serde_json::json!({ "error": "App not found in cache", "app_id": app_id })), + } + } + + /// marketplace.list-published — List manifests published by this node. + pub(super) async fn handle_marketplace_list_published(&self) -> Result { + let manifests = marketplace::list_published(&self.config.data_dir).await?; + Ok(serde_json::json!({ "manifests": manifests })) + } + + /// marketplace.verify — Verify a manifest's security compliance. + pub(super) async fn handle_marketplace_verify( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let manifest: marketplace::AppManifest = + serde_json::from_value(params).map_err(|e| anyhow::anyhow!("Invalid manifest: {}", e))?; + + let issues = marketplace::validate_manifest(&manifest); + let (trust_score, trust_tier) = marketplace::calculate_trust_score(&manifest, 0, &[]); + + Ok(serde_json::json!({ + "valid": issues.is_empty(), + "issues": issues, + "trust_score": trust_score, + "trust_tier": trust_tier, + })) + } +} diff --git a/core/archipelago/src/api/rpc/mesh.rs b/core/archipelago/src/api/rpc/mesh.rs new file mode 100644 index 00000000..01fff2c2 --- /dev/null +++ b/core/archipelago/src/api/rpc/mesh.rs @@ -0,0 +1,90 @@ +use super::RpcHandler; +use crate::{identity, mesh}; +use anyhow::Result; +use tracing::info; + +impl RpcHandler { + /// mesh.status — Get mesh radio status and detected devices. + pub(super) async fn handle_mesh_status(&self) -> Result { + let config = mesh::load_config(&self.config.data_dir).await?; + let devices = mesh::detect_meshtastic_devices().await; + + Ok(serde_json::json!({ + "enabled": config.enabled, + "device_path": config.device_path, + "channel_name": config.channel_name, + "broadcast_identity": config.broadcast_identity, + "detected_devices": devices, + })) + } + + /// mesh.discover — Discover nodes via mesh radio. + pub(super) async fn handle_mesh_discover( + &self, + params: Option, + ) -> Result { + let device_path = params + .as_ref() + .and_then(|p| p.get("device_path")) + .and_then(|v| v.as_str()); + + let config = mesh::load_config(&self.config.data_dir).await?; + let effective_device = device_path.or(config.device_path.as_deref()); + + let nodes = mesh::discover_nodes(effective_device).await?; + + Ok(serde_json::json!({ + "nodes": nodes, + "count": nodes.len(), + })) + } + + /// mesh.broadcast — Broadcast our node identity over mesh. + pub(super) async fn handle_mesh_broadcast(&self) -> Result { + let config = mesh::load_config(&self.config.data_dir).await?; + if !config.enabled { + anyhow::bail!("Mesh networking is not enabled. Configure it first."); + } + + let (data, _) = self.state_manager.get_snapshot().await; + let did = identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?; + let pubkey = &data.server_info.pubkey; + + mesh::broadcast_identity(&did, pubkey, config.device_path.as_deref()).await?; + + info!("Broadcast identity over mesh"); + Ok(serde_json::json!({ "broadcast": true })) + } + + /// mesh.configure — Enable/disable mesh and set device path. + pub(super) async fn handle_mesh_configure( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + + let mut config = mesh::load_config(&self.config.data_dir).await?; + + if let Some(enabled) = params.get("enabled").and_then(|v| v.as_bool()) { + config.enabled = enabled; + } + if let Some(device) = params.get("device_path").and_then(|v| v.as_str()) { + config.device_path = Some(device.to_string()); + } + if let Some(channel) = params.get("channel_name").and_then(|v| v.as_str()) { + config.channel_name = Some(channel.to_string()); + } + if let Some(broadcast) = params.get("broadcast_identity").and_then(|v| v.as_bool()) { + config.broadcast_identity = broadcast; + } + + mesh::save_config(&self.config.data_dir, &config).await?; + + info!("Mesh config updated"); + Ok(serde_json::json!({ + "configured": true, + "enabled": config.enabled, + "device_path": config.device_path, + })) + } +} diff --git a/core/archipelago/src/api/rpc/mod.rs b/core/archipelago/src/api/rpc/mod.rs index f8e059ec..7a75ed6e 100644 --- a/core/archipelago/src/api/rpc/mod.rs +++ b/core/archipelago/src/api/rpc/mod.rs @@ -6,6 +6,7 @@ mod content; mod credentials; mod dwn; mod federation; +mod handshake; mod identity; mod interfaces; mod marketplace; @@ -297,6 +298,11 @@ impl RpcHandler { "node.nostr-pubkey" => self.handle_node_nostr_pubkey().await, "node-nostr-verify-revoked" => self.handle_node_nostr_verify_revoked().await, + // Encrypted peer handshake (NIP-44) + "handshake.discover" => self.handle_handshake_discover().await, + "handshake.connect" => self.handle_handshake_connect(params).await, + "handshake.poll" => self.handle_handshake_poll().await, + // TOTP 2FA "auth.totp.setup.begin" => self.handle_totp_setup_begin(params).await, "auth.totp.setup.confirm" => self.handle_totp_setup_confirm(params).await, diff --git a/core/archipelago/src/api/rpc/package.rs b/core/archipelago/src/api/rpc/package.rs index a381d42f..c8806f05 100644 --- a/core/archipelago/src/api/rpc/package.rs +++ b/core/archipelago/src/api/rpc/package.rs @@ -4,7 +4,6 @@ use crate::data_model::{ }; use crate::port_allocator::PortAllocator; use anyhow::{Context, Result}; -use std::collections::HashMap; use tokio::io::{AsyncBufReadExt, BufReader}; use tracing::{debug, info}; @@ -752,15 +751,50 @@ printtoconsole=1\n"; let containers_to_remove = get_containers_for_app(package_id).await?; + if containers_to_remove.is_empty() { + tracing::warn!("Uninstall {}: no containers found", package_id); + } + + let mut stopped = 0u32; + let mut removed = 0u32; + let mut errors = Vec::new(); + for name in &containers_to_remove { - let _ = tokio::process::Command::new("sudo") - .args(["podman", "stop", name]) + tracing::info!("Uninstall {}: stopping container {}", package_id, name); + let stop_out = tokio::process::Command::new("sudo") + .args(["podman", "stop", "-t", "10", name]) .output() .await; - let _ = tokio::process::Command::new("sudo") + match stop_out { + Ok(o) if o.status.success() => stopped += 1, + Ok(o) => { + let stderr = String::from_utf8_lossy(&o.stderr); + tracing::warn!("Uninstall {}: stop {} failed: {}", package_id, name, stderr.trim()); + } + Err(e) => { + tracing::warn!("Uninstall {}: stop {} error: {}", package_id, name, e); + } + } + + tracing::info!("Uninstall {}: removing container {}", package_id, name); + let rm_out = tokio::process::Command::new("sudo") .args(["podman", "rm", "-f", name]) .output() .await; + match rm_out { + Ok(o) if o.status.success() => removed += 1, + Ok(o) => { + let stderr = String::from_utf8_lossy(&o.stderr); + let msg = format!("Failed to remove {}: {}", name, stderr.trim()); + tracing::error!("Uninstall {}: {}", package_id, msg); + errors.push(msg); + } + Err(e) => { + let msg = format!("Failed to remove {}: {}", name, e); + tracing::error!("Uninstall {}: {}", package_id, msg); + errors.push(msg); + } + } } // Release port allocation @@ -772,14 +806,31 @@ printtoconsole=1\n"; if !preserve_data { let data_dirs = get_data_dirs_for_app(package_id); for dir in &data_dirs { - let _ = tokio::process::Command::new("sudo") + tracing::info!("Uninstall {}: removing data {}", package_id, dir); + let rm_out = tokio::process::Command::new("sudo") .args(["rm", "-rf", dir]) .output() .await; + if let Ok(o) = rm_out { + if !o.status.success() { + tracing::warn!("Uninstall {}: rm {} failed", package_id, dir); + } + } } } - Ok(serde_json::json!({ "status": "uninstalled" })) + if !errors.is_empty() { + tracing::error!("Uninstall {} completed with errors: {:?}", package_id, errors); + } else { + tracing::info!("Uninstall {} complete: stopped={}, removed={}", package_id, stopped, removed); + } + + Ok(serde_json::json!({ + "status": if errors.is_empty() { "uninstalled" } else { "partial" }, + "stopped": stopped, + "removed": removed, + "errors": errors, + })) } /// Start a bundled app (create container from pre-loaded image if needed, then start) diff --git a/core/archipelago/src/api/rpc/system.rs b/core/archipelago/src/api/rpc/system.rs index efbfd1c5..ff7a7963 100644 --- a/core/archipelago/src/api/rpc/system.rs +++ b/core/archipelago/src/api/rpc/system.rs @@ -43,6 +43,101 @@ impl RpcHandler { Ok(serde_json::json!({ "temperatures": temps })) } + + /// system.detect-usb-devices — scan for known hardware wallet USB devices + pub(super) async fn handle_system_detect_usb_devices(&self) -> Result { + debug!("Scanning for USB hardware wallets"); + + let devices = detect_usb_hardware_wallets().await.unwrap_or_default(); + + Ok(serde_json::json!({ "devices": devices })) + } + + /// system.disk-status — Disk usage with warning/critical thresholds. + pub(super) async fn handle_system_disk_status(&self) -> Result { + let (used, total) = read_disk_usage().await.unwrap_or((0, 0)); + let percent = if total > 0 { + (used as f64 / total as f64) * 100.0 + } else { + 0.0 + }; + let percent_rounded = (percent * 10.0).round() / 10.0; + + let level = if percent >= 90.0 { + "critical" + } else if percent >= 85.0 { + "warning" + } else { + "ok" + }; + + Ok(serde_json::json!({ + "used_bytes": used, + "total_bytes": total, + "free_bytes": total.saturating_sub(used), + "used_percent": percent_rounded, + "level": level, + })) + } + + /// system.disk-cleanup — Remove old container images, stale logs, and temp files. + pub(super) async fn handle_system_disk_cleanup(&self) -> Result { + tracing::info!("Starting disk cleanup"); + let mut freed_bytes: u64 = 0; + let mut actions: Vec = Vec::new(); + + // 1. Prune dangling container images + match prune_container_images().await { + Ok(bytes) => { + if bytes > 0 { + freed_bytes += bytes; + actions.push(format!("Pruned dangling images: {} freed", format_bytes(bytes))); + } + } + Err(e) => actions.push(format!("Image prune failed: {}", e)), + } + + // 2. Clean old log files (> 30 days) + match clean_old_logs(30).await { + Ok(bytes) => { + if bytes > 0 { + freed_bytes += bytes; + actions.push(format!("Cleaned old logs: {} freed", format_bytes(bytes))); + } + } + Err(e) => actions.push(format!("Log cleanup failed: {}", e)), + } + + // 3. Remove stale temp files + match clean_temp_files().await { + Ok(bytes) => { + if bytes > 0 { + freed_bytes += bytes; + actions.push(format!("Removed temp files: {} freed", format_bytes(bytes))); + } + } + Err(e) => actions.push(format!("Temp cleanup failed: {}", e)), + } + + // 4. Prune container build cache + match prune_build_cache().await { + Ok(bytes) => { + if bytes > 0 { + freed_bytes += bytes; + actions.push(format!("Pruned build cache: {} freed", format_bytes(bytes))); + } + } + Err(e) => actions.push(format!("Build cache prune failed: {}", e)), + } + + tracing::info!("Disk cleanup complete: {} freed ({} actions)", format_bytes(freed_bytes), actions.len()); + + Ok(serde_json::json!({ + "freed_bytes": freed_bytes, + "freed_human": format_bytes(freed_bytes), + "actions": actions, + })) + } } /// Read system uptime from /proc/uptime (seconds since boot). @@ -226,6 +321,203 @@ async fn read_top_processes() -> Result> { Ok(procs) } +/// Known hardware wallet USB vendor IDs. +const KNOWN_HW_WALLETS: &[(u16, &str)] = &[ + (0xd13e, "ColdCard"), + (0x534c, "Trezor"), + (0x2c97, "Ledger"), + (0x1209, "BitBox02"), +]; + +/// Scan /sys/bus/usb/devices/ for known hardware wallet vendor IDs. +async fn detect_usb_hardware_wallets() -> Result> { + let usb_dir = std::path::Path::new("/sys/bus/usb/devices"); + if !usb_dir.exists() { + return Ok(Vec::new()); + } + + let mut devices = Vec::new(); + let mut entries = tokio::fs::read_dir(usb_dir) + .await + .context("Failed to read /sys/bus/usb/devices")?; + + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + let vendor_path = path.join("idVendor"); + let product_path = path.join("idProduct"); + + if !vendor_path.exists() { + continue; + } + + let vid_str = match tokio::fs::read_to_string(&vendor_path).await { + Ok(s) => s.trim().to_string(), + Err(_) => continue, + }; + + let vid = match u16::from_str_radix(&vid_str, 16) { + Ok(v) => v, + Err(_) => continue, + }; + + if let Some((_, name)) = KNOWN_HW_WALLETS.iter().find(|(known_vid, _)| *known_vid == vid) { + let pid_str = tokio::fs::read_to_string(&product_path) + .await + .map(|s| s.trim().to_string()) + .unwrap_or_default(); + + let manufacturer = tokio::fs::read_to_string(path.join("manufacturer")) + .await + .map(|s| s.trim().to_string()) + .unwrap_or_default(); + + let product = tokio::fs::read_to_string(path.join("product")) + .await + .map(|s| s.trim().to_string()) + .unwrap_or_default(); + + devices.push(serde_json::json!({ + "type": name, + "vendor_id": vid_str, + "product_id": pid_str, + "manufacturer": manufacturer, + "product": product, + "path": path.to_string_lossy(), + })); + } + } + + Ok(devices) +} + +/// Prune dangling container images via `sudo podman image prune -f`. +/// Returns estimated bytes freed. +async fn prune_container_images() -> Result { + let output = tokio::process::Command::new("sudo") + .args(["podman", "image", "prune", "-f"]) + .output() + .await + .context("Failed to run podman image prune")?; + + if !output.status.success() { + anyhow::bail!( + "podman image prune failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + } + + // Podman outputs image IDs, estimate ~100MB per pruned image + let stdout = String::from_utf8_lossy(&output.stdout); + let pruned_count = stdout.lines().filter(|l| !l.trim().is_empty()).count(); + Ok(pruned_count as u64 * 100_000_000) // rough estimate +} + +/// Prune container build cache via `sudo podman system prune -f`. +async fn prune_build_cache() -> Result { + // Just prune volumes and build cache (not containers or images — those are handled above) + let output = tokio::process::Command::new("sudo") + .args(["podman", "volume", "prune", "-f"]) + .output() + .await + .context("Failed to run podman volume prune")?; + + if !output.status.success() { + anyhow::bail!( + "podman volume prune failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + } + + let stdout = String::from_utf8_lossy(&output.stdout); + let pruned_count = stdout.lines().filter(|l| !l.trim().is_empty()).count(); + Ok(pruned_count as u64 * 10_000_000) // rough estimate per volume +} + +/// Clean log files older than `max_age_days` from common log directories. +async fn clean_old_logs(max_age_days: u64) -> Result { + let output = tokio::process::Command::new("sudo") + .args([ + "find", + "/var/log", + "-type", + "f", + "-name", + "*.log.*", + "-mtime", + &format!("+{}", max_age_days), + "-delete", + "-print", + ]) + .output() + .await + .context("Failed to clean old logs")?; + + let stdout = String::from_utf8_lossy(&output.stdout); + let deleted_count = stdout.lines().filter(|l| !l.trim().is_empty()).count(); + // Also clean rotated/compressed logs + let _ = tokio::process::Command::new("sudo") + .args([ + "find", + "/var/log", + "-type", + "f", + "-name", + "*.gz", + "-mtime", + &format!("+{}", max_age_days), + "-delete", + ]) + .output() + .await; + + Ok(deleted_count as u64 * 500_000) // rough estimate per log file +} + +/// Remove stale temp files from /tmp and /var/tmp. +async fn clean_temp_files() -> Result { + let mut freed = 0u64; + + for dir in &["/tmp", "/var/tmp"] { + let output = tokio::process::Command::new("sudo") + .args([ + "find", + dir, + "-type", + "f", + "-mtime", + "+7", + "-delete", + "-print", + ]) + .output() + .await; + + if let Ok(out) = output { + let stdout = String::from_utf8_lossy(&out.stdout); + let count = stdout.lines().filter(|l| !l.trim().is_empty()).count(); + freed += count as u64 * 100_000; // rough estimate per temp file + } + } + + Ok(freed) +} + +fn format_bytes(bytes: u64) -> String { + const KB: u64 = 1024; + const MB: u64 = KB * 1024; + const GB: u64 = MB * 1024; + + if bytes >= GB { + format!("{:.1} GB", bytes as f64 / GB as f64) + } else if bytes >= MB { + format!("{:.1} MB", bytes as f64 / MB as f64) + } else if bytes >= KB { + format!("{:.0} KB", bytes as f64 / KB as f64) + } else { + format!("{} B", bytes) + } +} + /// Read temperatures from /sys/class/thermal/thermal_zone*/temp. async fn read_temperatures() -> Result> { let mut temps = Vec::new(); diff --git a/core/archipelago/src/api/rpc/update.rs b/core/archipelago/src/api/rpc/update.rs index 613f1dca..e95f4d4f 100644 --- a/core/archipelago/src/api/rpc/update.rs +++ b/core/archipelago/src/api/rpc/update.rs @@ -42,4 +42,52 @@ impl RpcHandler { update::dismiss_update(&self.config.data_dir).await?; Ok(serde_json::json!({ "ok": true })) } + + /// Download the available update to staging. + pub(super) async fn handle_update_download(&self) -> Result { + let progress = update::download_update(&self.config.data_dir).await?; + Ok(serde_json::json!({ + "total_bytes": progress.total_bytes, + "downloaded_bytes": progress.downloaded_bytes, + "components_downloaded": progress.components_downloaded, + })) + } + + /// Apply the staged update. + pub(super) async fn handle_update_apply(&self) -> Result { + update::apply_update(&self.config.data_dir).await?; + Ok(serde_json::json!({ "applied": true, "restart_required": true })) + } + + /// Rollback to the previous version. + pub(super) async fn handle_update_rollback(&self) -> Result { + update::rollback_update(&self.config.data_dir).await?; + Ok(serde_json::json!({ "rolled_back": true, "restart_required": true })) + } + + /// Get the current update schedule. + pub(super) async fn handle_update_get_schedule(&self) -> Result { + let schedule = update::get_schedule(&self.config.data_dir).await?; + Ok(serde_json::json!({ "schedule": schedule })) + } + + /// Set the update schedule. Params: { schedule: "manual" | "daily_check" | "auto_apply" } + pub(super) async fn handle_update_set_schedule( + &self, + params: &serde_json::Value, + ) -> Result { + let schedule_str = params["schedule"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing 'schedule' parameter"))?; + + let schedule = match schedule_str { + "manual" => update::UpdateSchedule::Manual, + "daily_check" => update::UpdateSchedule::DailyCheck, + "auto_apply" => update::UpdateSchedule::AutoApply, + _ => anyhow::bail!("Invalid schedule: '{}'. Use manual, daily_check, or auto_apply", schedule_str), + }; + + update::set_schedule(&self.config.data_dir, schedule).await?; + Ok(serde_json::json!({ "schedule": schedule })) + } } diff --git a/core/archipelago/src/backup/full.rs b/core/archipelago/src/backup/full.rs new file mode 100644 index 00000000..28e303d7 --- /dev/null +++ b/core/archipelago/src/backup/full.rs @@ -0,0 +1,607 @@ +//! Full system backup — identity keys + app data + settings + DWN messages. +//! +//! Creates an encrypted tar.gz archive containing all critical node data. +//! Encryption: Argon2 key derivation + ChaCha20-Poly1305. + +use anyhow::{Context, Result}; +use argon2::Argon2; +use chacha20poly1305::aead::{Aead, KeyInit}; +use flate2::read::GzDecoder; +use flate2::write::GzEncoder; +use flate2::Compression; +use rand::RngCore; +use serde::{Deserialize, Serialize}; +use std::io::Read; +use std::path::{Path, PathBuf}; +use tar::{Archive, Builder}; +use tokio::fs; +use tracing::{debug, info}; + +const SALT_LEN: usize = 16; +const NONCE_LEN: usize = 12; +const KEY_LEN: usize = 32; + +/// Directories within data_dir to include in a full backup. +const BACKUP_DIRS: &[&str] = &[ + "identity", + "identities", + "dwn", + "credentials", + "tor-config", + "content", +]; + +/// Files within data_dir to include in a full backup. +const BACKUP_FILES: &[&str] = &[ + "user.json", + "peers.json", + "names.json", + "onboarding.json", + "nostr_relays.json", + "network_visibility", + "port_allocations.json", + "update_state.json", +]; + +/// Backup metadata stored alongside the archive. +#[derive(Debug, Serialize, Deserialize)] +pub struct BackupMetadata { + pub id: String, + pub version: u32, + pub created_at: String, + pub encrypted: bool, + pub size_bytes: u64, + pub description: Option, +} + +/// Result of a backup verification check. +#[derive(Debug, Serialize, Deserialize)] +pub struct VerifyResult { + pub valid: bool, + pub id: String, + pub created_at: String, + pub size_bytes: u64, + pub error: Option, +} + +/// Create a full encrypted backup of all node data. +/// Returns the backup metadata and the path to the backup file. +pub async fn create_full_backup( + data_dir: &Path, + passphrase: &str, + description: Option<&str>, +) -> Result { + let backups_dir = data_dir.join("backups"); + fs::create_dir_all(&backups_dir) + .await + .context("Failed to create backups dir")?; + + let backup_id = uuid::Uuid::new_v4().to_string(); + let timestamp = chrono::Utc::now().to_rfc3339(); + + // Step 1: Create tar.gz archive in memory + let tar_gz_data = tokio::task::spawn_blocking({ + let data_dir = data_dir.to_path_buf(); + move || create_tar_gz(&data_dir) + }) + .await? + .context("Failed to create tar archive")?; + + info!(size = tar_gz_data.len(), "Backup archive created"); + + // Step 2: Encrypt the archive + let encrypted = encrypt_data(&tar_gz_data, passphrase)?; + + // Step 3: Write to disk + let backup_path = backups_dir.join(format!("{}.bak", backup_id)); + fs::write(&backup_path, &encrypted) + .await + .context("Failed to write backup file")?; + + // Step 4: Write metadata + let metadata = BackupMetadata { + id: backup_id, + version: 2, + created_at: timestamp, + encrypted: true, + size_bytes: encrypted.len() as u64, + description: description.map(|s| s.to_string()), + }; + + let meta_path = backups_dir.join(format!("{}.meta.json", metadata.id)); + let meta_json = serde_json::to_string_pretty(&metadata) + .context("Failed to serialize metadata")?; + fs::write(&meta_path, meta_json) + .await + .context("Failed to write metadata")?; + + info!(id = %metadata.id, size = metadata.size_bytes, "Full backup created"); + Ok(metadata) +} + +/// Restore a full backup from an encrypted archive. +pub async fn restore_full_backup( + data_dir: &Path, + backup_id: &str, + passphrase: &str, +) -> Result<()> { + let backup_path = data_dir.join("backups").join(format!("{}.bak", backup_id)); + if !backup_path.exists() { + anyhow::bail!("Backup not found: {}", backup_id); + } + + let encrypted = fs::read(&backup_path) + .await + .context("Failed to read backup file")?; + + let tar_gz_data = decrypt_data(&encrypted, passphrase)?; + + // Extract to data_dir + tokio::task::spawn_blocking({ + let data_dir = data_dir.to_path_buf(); + move || extract_tar_gz(&data_dir, &tar_gz_data) + }) + .await? + .context("Failed to extract backup")?; + + info!(id = %backup_id, "Backup restored"); + Ok(()) +} + +/// List available backups by reading metadata files. +pub async fn list_backups(data_dir: &Path) -> Result> { + let backups_dir = data_dir.join("backups"); + if !backups_dir.exists() { + return Ok(Vec::new()); + } + + let mut backups = Vec::new(); + let mut entries = fs::read_dir(&backups_dir) + .await + .context("Failed to read backups dir")?; + + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + if path.extension().and_then(|e| e.to_str()) == Some("json") + && path.to_str().map_or(false, |s| s.contains(".meta.")) + { + let content = match fs::read_to_string(&path).await { + Ok(c) => c, + Err(_) => continue, + }; + if let Ok(meta) = serde_json::from_str::(&content) { + backups.push(meta); + } + } + } + + // Sort newest first + backups.sort_by(|a, b| b.created_at.cmp(&a.created_at)); + Ok(backups) +} + +/// Verify a backup's integrity by attempting decryption. +pub async fn verify_backup( + data_dir: &Path, + backup_id: &str, + passphrase: &str, +) -> Result { + let backup_path = data_dir.join("backups").join(format!("{}.bak", backup_id)); + let meta_path = data_dir + .join("backups") + .join(format!("{}.meta.json", backup_id)); + + if !backup_path.exists() { + return Ok(VerifyResult { + valid: false, + id: backup_id.to_string(), + created_at: String::new(), + size_bytes: 0, + error: Some("Backup file not found".to_string()), + }); + } + + let encrypted = fs::read(&backup_path) + .await + .context("Failed to read backup")?; + + let meta: BackupMetadata = if meta_path.exists() { + let content = fs::read_to_string(&meta_path).await?; + serde_json::from_str(&content)? + } else { + BackupMetadata { + id: backup_id.to_string(), + version: 0, + created_at: String::new(), + encrypted: true, + size_bytes: encrypted.len() as u64, + description: None, + } + }; + + match decrypt_data(&encrypted, passphrase) { + Ok(data) => { + // Verify it's a valid gzip + let mut decoder = GzDecoder::new(data.as_slice()); + let mut buf = [0u8; 512]; + match decoder.read(&mut buf) { + Ok(_) => Ok(VerifyResult { + valid: true, + id: meta.id, + created_at: meta.created_at, + size_bytes: meta.size_bytes, + error: None, + }), + Err(e) => Ok(VerifyResult { + valid: false, + id: meta.id, + created_at: meta.created_at, + size_bytes: meta.size_bytes, + error: Some(format!("Archive corrupted: {}", e)), + }), + } + } + Err(e) => Ok(VerifyResult { + valid: false, + id: meta.id, + created_at: meta.created_at, + size_bytes: meta.size_bytes, + error: Some(format!("Decryption failed: {}", e)), + }), + } +} + +/// Get the backup file path for download. +pub fn backup_file_path(data_dir: &Path, backup_id: &str) -> PathBuf { + data_dir.join("backups").join(format!("{}.bak", backup_id)) +} + +/// Info about a detected removable USB drive. +#[derive(Debug, Serialize, Deserialize)] +pub struct UsbDrive { + pub device: String, + pub mount_point: Option, + pub label: Option, + pub size_bytes: u64, + pub removable: bool, +} + +/// List removable USB drives on the system. +/// Scans /sys/block/sd* for removable devices. +pub async fn list_usb_drives() -> Result> { + let mut drives = Vec::new(); + + let mut entries = match fs::read_dir("/sys/block").await { + Ok(e) => e, + Err(_) => return Ok(drives), + }; + + while let Some(entry) = entries.next_entry().await? { + let name = entry.file_name().to_string_lossy().to_string(); + if !name.starts_with("sd") { + continue; + } + + // Check if removable + let removable_path = format!("/sys/block/{}/removable", name); + let removable = match fs::read_to_string(&removable_path).await { + Ok(v) => v.trim() == "1", + Err(_) => false, + }; + if !removable { + continue; + } + + // Get size in 512-byte sectors + let size_path = format!("/sys/block/{}/size", name); + let size_bytes = match fs::read_to_string(&size_path).await { + Ok(v) => v.trim().parse::().unwrap_or(0) * 512, + Err(_) => 0, + }; + + let device = format!("/dev/{}", name); + + // Check mount point from /proc/mounts + let mount_point = find_mount_point(&device).await; + + // Try to get label from the first partition + let partition = format!("{}1", device); + let label = get_fs_label(&partition).await; + + drives.push(UsbDrive { + device, + mount_point, + label, + size_bytes, + removable: true, + }); + } + + Ok(drives) +} + +/// Copy a backup file to a mounted USB drive. +pub async fn backup_to_usb( + data_dir: &Path, + backup_id: &str, + mount_point: &str, +) -> Result { + let src = backup_file_path(data_dir, backup_id); + if !src.exists() { + anyhow::bail!("Backup not found: {}", backup_id); + } + + let mount_path = Path::new(mount_point); + if !mount_path.exists() || !mount_path.is_dir() { + anyhow::bail!("Mount point not accessible: {}", mount_point); + } + + let dest_dir = mount_path.join("archipelago-backups"); + fs::create_dir_all(&dest_dir) + .await + .context("Failed to create backup dir on USB")?; + + let filename = format!("{}.bak", backup_id); + let dest = dest_dir.join(&filename); + + fs::copy(&src, &dest) + .await + .context("Failed to copy backup to USB")?; + + // Also copy metadata + let meta_src = data_dir + .join("backups") + .join(format!("{}.meta.json", backup_id)); + if meta_src.exists() { + let meta_dest = dest_dir.join(format!("{}.meta.json", backup_id)); + let _ = fs::copy(&meta_src, &meta_dest).await; + } + + info!(id = %backup_id, dest = %dest.display(), "Backup copied to USB"); + Ok(dest) +} + +async fn find_mount_point(device: &str) -> Option { + let mounts = fs::read_to_string("/proc/mounts").await.ok()?; + for line in mounts.lines() { + let parts: Vec<&str> = line.split_whitespace().collect(); + if parts.len() >= 2 && parts[0].starts_with(device) { + return Some(parts[1].to_string()); + } + } + None +} + +async fn get_fs_label(partition: &str) -> Option { + let output = tokio::process::Command::new("blkid") + .arg("-s") + .arg("LABEL") + .arg("-o") + .arg("value") + .arg(partition) + .output() + .await + .ok()?; + if output.status.success() { + let label = String::from_utf8_lossy(&output.stdout).trim().to_string(); + if !label.is_empty() { + return Some(label); + } + } + None +} + +// --- Internal helpers --- + +fn create_tar_gz(data_dir: &Path) -> Result> { + let mut buf = Vec::new(); + let gz = GzEncoder::new(&mut buf, Compression::default()); + let mut tar = Builder::new(gz); + + // Add directories + for dir_name in BACKUP_DIRS { + let dir_path = data_dir.join(dir_name); + if dir_path.exists() && dir_path.is_dir() { + tar.append_dir_all(*dir_name, &dir_path) + .with_context(|| format!("Failed to add dir {} to backup", dir_name))?; + debug!(dir = %dir_name, "Added directory to backup"); + } + } + + // Add individual files + for file_name in BACKUP_FILES { + let file_path = data_dir.join(file_name); + if file_path.exists() && file_path.is_file() { + let data = std::fs::read(&file_path) + .with_context(|| format!("Failed to read {}", file_name))?; + let mut header = tar::Header::new_gnu(); + header.set_size(data.len() as u64); + header.set_mode(0o644); + header.set_cksum(); + tar.append_data(&mut header, *file_name, data.as_slice()) + .with_context(|| format!("Failed to add {} to backup", file_name))?; + debug!(file = %file_name, "Added file to backup"); + } + } + + tar.into_inner() + .context("Failed to finalize tar")? + .finish() + .context("Failed to finalize gzip")?; + + Ok(buf) +} + +fn extract_tar_gz(data_dir: &Path, tar_gz_data: &[u8]) -> Result<()> { + let gz = GzDecoder::new(tar_gz_data); + let mut archive = Archive::new(gz); + + archive + .unpack(data_dir) + .context("Failed to extract backup archive")?; + + debug!("Backup extracted to {:?}", data_dir); + Ok(()) +} + +fn encrypt_data(data: &[u8], passphrase: &str) -> Result> { + let mut salt = [0u8; SALT_LEN]; + let mut nonce = [0u8; NONCE_LEN]; + rand::rngs::OsRng.fill_bytes(&mut salt); + rand::rngs::OsRng.fill_bytes(&mut nonce); + + let argon2 = Argon2::default(); + let mut key = [0u8; KEY_LEN]; + argon2 + .hash_password_into(passphrase.as_bytes(), &salt, &mut key) + .map_err(|e| anyhow::anyhow!("Key derivation failed: {}", e))?; + + let cipher = chacha20poly1305::ChaCha20Poly1305::new_from_slice(&key) + .map_err(|e| anyhow::anyhow!("Cipher init: {}", e))?; + let ciphertext = cipher + .encrypt( + chacha20poly1305::aead::generic_array::GenericArray::from_slice(&nonce), + data, + ) + .map_err(|e| anyhow::anyhow!("Encryption failed: {}", e))?; + + // Format: salt || nonce || ciphertext + let mut output = Vec::with_capacity(SALT_LEN + NONCE_LEN + ciphertext.len()); + output.extend_from_slice(&salt); + output.extend_from_slice(&nonce); + output.extend_from_slice(&ciphertext); + Ok(output) +} + +fn decrypt_data(data: &[u8], passphrase: &str) -> Result> { + if data.len() < SALT_LEN + NONCE_LEN { + anyhow::bail!("Backup data too short"); + } + + let salt = &data[..SALT_LEN]; + let nonce = &data[SALT_LEN..SALT_LEN + NONCE_LEN]; + let ciphertext = &data[SALT_LEN + NONCE_LEN..]; + + let argon2 = Argon2::default(); + let mut key = [0u8; KEY_LEN]; + argon2 + .hash_password_into(passphrase.as_bytes(), salt, &mut key) + .map_err(|e| anyhow::anyhow!("Key derivation failed: {}", e))?; + + let cipher = chacha20poly1305::ChaCha20Poly1305::new_from_slice(&key) + .map_err(|e| anyhow::anyhow!("Cipher init: {}", e))?; + let plaintext = cipher + .decrypt( + chacha20poly1305::aead::generic_array::GenericArray::from_slice(nonce), + ciphertext, + ) + .map_err(|_| anyhow::anyhow!("Decryption failed — wrong passphrase or corrupted data"))?; + + Ok(plaintext) +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + fn setup_data_dir(dir: &Path) { + // Create some test data + std::fs::create_dir_all(dir.join("identity")).unwrap(); + std::fs::write(dir.join("identity/node_key"), vec![0xAB; 32]).unwrap(); + std::fs::write(dir.join("user.json"), r#"{"user":"test"}"#).unwrap(); + std::fs::write(dir.join("peers.json"), "[]").unwrap(); + } + + #[test] + fn encrypt_decrypt_roundtrip() { + let data = b"Hello, Archipelago backup!"; + let pass = "test-passphrase"; + let encrypted = encrypt_data(data, pass).unwrap(); + assert_ne!(&encrypted[..], data); + let decrypted = decrypt_data(&encrypted, pass).unwrap(); + assert_eq!(&decrypted, data); + } + + #[test] + fn wrong_passphrase_fails() { + let data = b"secret data"; + let encrypted = encrypt_data(data, "correct").unwrap(); + assert!(decrypt_data(&encrypted, "wrong").is_err()); + } + + #[test] + fn tar_gz_roundtrip() { + let dir = TempDir::new().unwrap(); + setup_data_dir(dir.path()); + + let archive = create_tar_gz(dir.path()).unwrap(); + assert!(!archive.is_empty()); + + // Extract to a new dir + let restore_dir = TempDir::new().unwrap(); + extract_tar_gz(restore_dir.path(), &archive).unwrap(); + + // Verify files exist + assert!(restore_dir.path().join("identity/node_key").exists()); + assert!(restore_dir.path().join("user.json").exists()); + assert!(restore_dir.path().join("peers.json").exists()); + } + + #[tokio::test] + async fn full_backup_and_list() { + let dir = TempDir::new().unwrap(); + setup_data_dir(dir.path()); + + let meta = create_full_backup(dir.path(), "backup-pass", Some("Test backup")) + .await + .unwrap(); + assert!(!meta.id.is_empty()); + assert!(meta.size_bytes > 0); + assert_eq!(meta.description, Some("Test backup".to_string())); + + let backups = list_backups(dir.path()).await.unwrap(); + assert_eq!(backups.len(), 1); + assert_eq!(backups[0].id, meta.id); + } + + #[tokio::test] + async fn backup_verify() { + let dir = TempDir::new().unwrap(); + setup_data_dir(dir.path()); + + let meta = create_full_backup(dir.path(), "my-pass", None) + .await + .unwrap(); + + let result = verify_backup(dir.path(), &meta.id, "my-pass").await.unwrap(); + assert!(result.valid); + + let bad_result = verify_backup(dir.path(), &meta.id, "wrong-pass").await.unwrap(); + assert!(!bad_result.valid); + } + + #[tokio::test] + async fn backup_and_restore() { + let dir = TempDir::new().unwrap(); + setup_data_dir(dir.path()); + + let meta = create_full_backup(dir.path(), "restore-pass", None) + .await + .unwrap(); + + // Delete the original data + std::fs::remove_file(dir.path().join("user.json")).unwrap(); + assert!(!dir.path().join("user.json").exists()); + + // Restore + restore_full_backup(dir.path(), &meta.id, "restore-pass") + .await + .unwrap(); + + // Verify restored + assert!(dir.path().join("user.json").exists()); + let content = std::fs::read_to_string(dir.path().join("user.json")).unwrap(); + assert_eq!(content, r#"{"user":"test"}"#); + } +} diff --git a/core/archipelago/src/backup.rs b/core/archipelago/src/backup/identity.rs similarity index 100% rename from core/archipelago/src/backup.rs rename to core/archipelago/src/backup/identity.rs diff --git a/core/archipelago/src/backup/mod.rs b/core/archipelago/src/backup/mod.rs new file mode 100644 index 00000000..8a736182 --- /dev/null +++ b/core/archipelago/src/backup/mod.rs @@ -0,0 +1,9 @@ +//! Backup and restore for Archipelago. +//! +//! - `identity`: Encrypted DID identity key backup (existing). +//! - `full`: Full system backup — identity + app data + configs + settings. + +mod identity; +pub mod full; + +pub use identity::create_encrypted_backup; diff --git a/core/archipelago/src/config.rs b/core/archipelago/src/config.rs index 4378fd08..9f887491 100644 --- a/core/archipelago/src/config.rs +++ b/core/archipelago/src/config.rs @@ -208,3 +208,141 @@ impl Default for Config { } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_default_config_values() { + let config = Config::default(); + assert_eq!(config.data_dir, PathBuf::from("/var/lib/archipelago")); + assert_eq!(config.bind_host, "0.0.0.0"); + assert_eq!(config.bind_port, 5678); + assert_eq!(config.log_level, "info"); + assert_eq!(config.host_ip, "127.0.0.1"); + assert!(!config.dev_mode); + assert_eq!(config.port_offset, 10000); + assert!(config.nostr_discovery_enabled); + assert_eq!(config.nostr_relays.len(), 2); + assert_eq!(config.nostr_tor_proxy, Some("127.0.0.1:9050".to_string())); + } + + #[test] + fn test_container_runtime_from_str_podman() { + assert!(matches!(ContainerRuntime::from_str("podman"), ContainerRuntime::Podman)); + assert!(matches!(ContainerRuntime::from_str("Podman"), ContainerRuntime::Podman)); + assert!(matches!(ContainerRuntime::from_str("PODMAN"), ContainerRuntime::Podman)); + } + + #[test] + fn test_container_runtime_from_str_docker() { + assert!(matches!(ContainerRuntime::from_str("docker"), ContainerRuntime::Docker)); + assert!(matches!(ContainerRuntime::from_str("Docker"), ContainerRuntime::Docker)); + assert!(matches!(ContainerRuntime::from_str("DOCKER"), ContainerRuntime::Docker)); + } + + #[test] + fn test_container_runtime_from_str_auto() { + assert!(matches!(ContainerRuntime::from_str("auto"), ContainerRuntime::Auto)); + assert!(matches!(ContainerRuntime::from_str("Auto"), ContainerRuntime::Auto)); + // Unknown strings default to Auto + assert!(matches!(ContainerRuntime::from_str("unknown"), ContainerRuntime::Auto)); + assert!(matches!(ContainerRuntime::from_str(""), ContainerRuntime::Auto)); + } + + #[test] + fn test_bitcoin_simulation_from_str() { + assert!(matches!(BitcoinSimulation::from_str("mock"), BitcoinSimulation::Mock)); + assert!(matches!(BitcoinSimulation::from_str("Mock"), BitcoinSimulation::Mock)); + assert!(matches!(BitcoinSimulation::from_str("testnet"), BitcoinSimulation::Testnet)); + assert!(matches!(BitcoinSimulation::from_str("Testnet"), BitcoinSimulation::Testnet)); + assert!(matches!(BitcoinSimulation::from_str("mainnet"), BitcoinSimulation::Mainnet)); + assert!(matches!(BitcoinSimulation::from_str("Mainnet"), BitcoinSimulation::Mainnet)); + assert!(matches!(BitcoinSimulation::from_str("none"), BitcoinSimulation::None)); + } + + #[test] + fn test_bitcoin_simulation_unknown_defaults_to_none() { + assert!(matches!(BitcoinSimulation::from_str(""), BitcoinSimulation::None)); + assert!(matches!(BitcoinSimulation::from_str("signet"), BitcoinSimulation::None)); + assert!(matches!(BitcoinSimulation::from_str("garbage"), BitcoinSimulation::None)); + } + + #[test] + fn test_config_serialization_roundtrip() { + let config = Config::default(); + let json = serde_json::to_string(&config).unwrap(); + let deserialized: Config = serde_json::from_str(&json).unwrap(); + assert_eq!(deserialized.bind_host, config.bind_host); + assert_eq!(deserialized.bind_port, config.bind_port); + assert_eq!(deserialized.data_dir, config.data_dir); + assert_eq!(deserialized.log_level, config.log_level); + assert_eq!(deserialized.dev_mode, config.dev_mode); + assert_eq!(deserialized.port_offset, config.port_offset); + assert_eq!(deserialized.nostr_discovery_enabled, config.nostr_discovery_enabled); + assert_eq!(deserialized.nostr_relays, config.nostr_relays); + } + + #[test] + fn test_config_toml_parsing() { + let toml_str = r#" + data_dir = "/tmp/test-data" + bind_host = "127.0.0.1" + bind_port = 9999 + log_level = "debug" + host_ip = "192.168.1.100" + dev_mode = true + container_runtime = "Podman" + port_offset = 20000 + bitcoin_simulation = "Testnet" + dev_data_dir = "/tmp/dev-test" + nostr_discovery_enabled = false + nostr_relays = ["wss://example.com"] + "#; + let config: Config = toml::de::from_str(toml_str).unwrap(); + assert_eq!(config.data_dir, PathBuf::from("/tmp/test-data")); + assert_eq!(config.bind_host, "127.0.0.1"); + assert_eq!(config.bind_port, 9999); + assert_eq!(config.log_level, "debug"); + assert_eq!(config.host_ip, "192.168.1.100"); + assert!(config.dev_mode); + assert_eq!(config.port_offset, 20000); + assert!(!config.nostr_discovery_enabled); + assert_eq!(config.nostr_relays, vec!["wss://example.com"]); + } + + #[test] + fn test_config_data_dir_is_pathbuf() { + let config = Config::default(); + assert!(config.data_dir.is_absolute()); + assert_eq!(config.data_dir, PathBuf::from("/var/lib/archipelago")); + } + + #[test] + fn test_config_host_ip_default() { + let config = Config::default(); + assert_eq!(config.host_ip, "127.0.0.1"); + } + + #[test] + fn test_config_dev_mode_defaults_off() { + let config = Config::default(); + assert!(!config.dev_mode); + assert_eq!(config.dev_data_dir, PathBuf::from("/tmp/archipelago-dev")); + } + + #[test] + fn test_config_nostr_discovery_enabled_by_default() { + let config = Config::default(); + assert!(config.nostr_discovery_enabled); + assert!(config.nostr_tor_proxy.is_some()); + } + + #[test] + fn test_config_nostr_relays_default_not_empty() { + let config = Config::default(); + assert!(!config.nostr_relays.is_empty()); + assert!(config.nostr_relays.iter().all(|r| r.starts_with("wss://"))); + } +} diff --git a/core/archipelago/src/container/docker_packages.rs b/core/archipelago/src/container/docker_packages.rs index 4ba73fd4..2d7c8d44 100644 --- a/core/archipelago/src/container/docker_packages.rs +++ b/core/archipelago/src/container/docker_packages.rs @@ -464,6 +464,48 @@ fn get_app_metadata(app_id: &str) -> AppMetadata { icon: "/assets/img/app-icons/tor.svg".to_string(), repo: "https://gitlab.torproject.org/tpo/core/tor".to_string(), }, + "botfights" => AppMetadata { + title: "BotFights".to_string(), + description: "AI bot arena — build, train, and battle autonomous agents".to_string(), + icon: "/assets/img/app-icons/botfights.svg".to_string(), + repo: "https://botfights.net".to_string(), + }, + "nwnn" => AppMetadata { + title: "Next Web News Network".to_string(), + description: "Decentralized news and link aggregator, synced from Telegram".to_string(), + icon: "/assets/img/app-icons/nwnn.png".to_string(), + repo: "https://nwnn.l484.com".to_string(), + }, + "484-kitchen" => AppMetadata { + title: "484 Kitchen".to_string(), + description: "K484 application platform".to_string(), + icon: "/assets/img/app-icons/484-kitchen.png".to_string(), + repo: "https://484.kitchen".to_string(), + }, + "call-the-operator" => AppMetadata { + title: "Call the Operator".to_string(), + description: "Escape the Matrix — explore decentralized alternatives".to_string(), + icon: "/assets/img/app-icons/call-the-operator.png".to_string(), + repo: "https://cta.tx1138.com".to_string(), + }, + "arch-presentation" => AppMetadata { + title: "Arch Presentation".to_string(), + description: "Archipelago: The Future of Decentralized Infrastructure".to_string(), + icon: "/assets/img/app-icons/arch-presentation.png".to_string(), + repo: "https://present.l484.com".to_string(), + }, + "syntropy-institute" => AppMetadata { + title: "Syntropy Institute".to_string(), + description: "Medicine Reimagined — frequency analysis-therapy and digital homeopathy".to_string(), + icon: "/assets/img/app-icons/syntropy-institute.png".to_string(), + repo: "https://syntropy.institute".to_string(), + }, + "t-zero" => AppMetadata { + title: "T-0".to_string(), + description: "Documentary series on decentralization, Bitcoin, and the ungovernable future".to_string(), + icon: "/assets/img/app-icons/t-zero.png".to_string(), + repo: "https://teeminuszero.net".to_string(), + }, _ => AppMetadata { title: app_id.to_string(), description: format!("{} application", app_id), diff --git a/core/archipelago/src/crash_recovery.rs b/core/archipelago/src/crash_recovery.rs new file mode 100644 index 00000000..9ec1e4c5 --- /dev/null +++ b/core/archipelago/src/crash_recovery.rs @@ -0,0 +1,335 @@ +// Crash Recovery Module +// Detects unexpected shutdowns and restarts containers that were running before the crash. +// +// How it works: +// 1. On startup, write a PID file as a "running" marker +// 2. Periodically snapshot which containers are running to a state file +// 3. On clean shutdown, remove the PID file +// 4. On next startup, if the PID file exists → previous instance crashed +// 5. On crash recovery: read the saved container list, restart them, log actions + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::path::{Path, PathBuf}; +use tokio::fs; +use tracing::{info, warn}; + +const PID_FILE: &str = "archipelago.pid"; +const CONTAINER_STATE_FILE: &str = "running-containers.json"; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct RunningContainerRecord { + pub name: String, + pub image: String, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct ContainerSnapshot { + pub timestamp: u64, + pub containers: Vec, +} + +/// Check if the previous instance crashed (PID file exists without a clean shutdown). +/// Returns the list of containers that were running before the crash, if any. +pub async fn check_for_crash(data_dir: &Path) -> Result>> { + let pid_path = data_dir.join(PID_FILE); + + if !pid_path.exists() { + return Ok(None); + } + + // PID file exists — previous instance didn't shut down cleanly + let old_pid = fs::read_to_string(&pid_path) + .await + .unwrap_or_default() + .trim() + .to_string(); + warn!("Crash detected: previous instance (PID {}) did not shut down cleanly", old_pid); + + // Check if that PID is actually still running (zombie/stuck process) + if !old_pid.is_empty() { + if let Ok(pid) = old_pid.parse::() { + if is_process_running(pid) { + warn!("Previous process (PID {}) is still running — not a crash, skipping recovery", pid); + // Remove stale PID file and skip recovery + let _ = fs::remove_file(&pid_path).await; + return Ok(None); + } + } + } + + // Load the saved container snapshot + let state_path = data_dir.join(CONTAINER_STATE_FILE); + let containers = if state_path.exists() { + match fs::read_to_string(&state_path).await { + Ok(content) => { + match serde_json::from_str::(&content) { + Ok(snapshot) => { + info!( + "Found {} containers from pre-crash snapshot (saved at {})", + snapshot.containers.len(), + snapshot.timestamp + ); + snapshot.containers + } + Err(e) => { + warn!("Failed to parse container snapshot: {}", e); + Vec::new() + } + } + } + Err(e) => { + warn!("Failed to read container snapshot: {}", e); + Vec::new() + } + } + } else { + info!("No container snapshot found — cannot determine which containers were running"); + Vec::new() + }; + + // Clean up the stale PID file + let _ = fs::remove_file(&pid_path).await; + + if containers.is_empty() { + Ok(None) + } else { + Ok(Some(containers)) + } +} + +/// Write the PID file to mark the current instance as running. +pub async fn write_pid_marker(data_dir: &Path) -> Result<()> { + let pid = std::process::id(); + let pid_path = data_dir.join(PID_FILE); + fs::write(&pid_path, pid.to_string()) + .await + .context("Failed to write PID marker")?; + Ok(()) +} + +/// Remove the PID file on clean shutdown. +pub async fn remove_pid_marker(data_dir: &Path) { + let pid_path = data_dir.join(PID_FILE); + if let Err(e) = fs::remove_file(&pid_path).await { + warn!("Failed to remove PID marker: {}", e); + } +} + +/// Save a snapshot of currently running containers to disk. +/// Called periodically so we know what to restart after a crash. +pub async fn save_container_snapshot(data_dir: &Path) -> Result<()> { + let output = tokio::process::Command::new("sudo") + .args(["podman", "ps", "--format", "json"]) + .output() + .await + .context("Failed to run podman ps")?; + + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr); + anyhow::bail!("podman ps failed: {}", stderr); + } + + let stdout = String::from_utf8_lossy(&output.stdout); + let containers: Vec = + serde_json::from_str(&stdout).unwrap_or_default(); + + let records: Vec = containers + .iter() + .filter_map(|c| { + let name = c.get("Names") + .and_then(|v| { + // Podman returns Names as an array + if let Some(arr) = v.as_array() { + arr.first().and_then(|n| n.as_str()).map(|s| s.to_string()) + } else { + v.as_str().map(|s| s.to_string()) + } + })?; + let image = c.get("Image") + .and_then(|v| v.as_str()) + .unwrap_or("unknown") + .to_string(); + Some(RunningContainerRecord { name, image }) + }) + .collect(); + + let snapshot = ContainerSnapshot { + timestamp: std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs(), + containers: records, + }; + + let state_path = data_dir.join(CONTAINER_STATE_FILE); + let json = serde_json::to_string_pretty(&snapshot) + .context("Failed to serialize container snapshot")?; + fs::write(&state_path, json) + .await + .context("Failed to write container snapshot")?; + + Ok(()) +} + +/// Recover containers that were running before a crash. +/// Attempts to start each container, logging success/failure. +pub async fn recover_containers(containers: &[RunningContainerRecord]) -> RecoveryReport { + let mut report = RecoveryReport { + total: containers.len(), + recovered: 0, + failed: Vec::new(), + }; + + for record in containers { + info!("Recovering container: {} (image: {})", record.name, record.image); + + let result = tokio::process::Command::new("sudo") + .args(["podman", "start", &record.name]) + .output() + .await; + + match result { + Ok(output) if output.status.success() => { + info!("Successfully restarted container: {}", record.name); + report.recovered += 1; + } + Ok(output) => { + let stderr = String::from_utf8_lossy(&output.stderr); + warn!("Failed to restart container {}: {}", record.name, stderr.trim()); + report.failed.push(record.name.clone()); + } + Err(e) => { + warn!("Failed to execute podman start for {}: {}", record.name, e); + report.failed.push(record.name.clone()); + } + } + } + + report +} + +#[derive(Debug)] +pub struct RecoveryReport { + pub total: usize, + pub recovered: usize, + pub failed: Vec, +} + +/// Check if a process with the given PID is still running. +fn is_process_running(pid: u32) -> bool { + // Check /proc/{pid} on Linux + std::path::Path::new(&format!("/proc/{}", pid)).exists() +} + +/// Spawn a background task that periodically saves the container snapshot. +pub fn spawn_snapshot_task(data_dir: PathBuf) { + tokio::spawn(async move { + // Wait 30s before first snapshot (let containers stabilize after startup) + tokio::time::sleep(std::time::Duration::from_secs(30)).await; + + let mut interval = tokio::time::interval(std::time::Duration::from_secs(60)); + loop { + interval.tick().await; + if let Err(e) = save_container_snapshot(&data_dir).await { + tracing::debug!("Container snapshot (non-fatal): {}", e); + } + } + }); +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + #[tokio::test] + async fn test_no_crash_without_pid_file() { + let tmp = TempDir::new().unwrap(); + let result = check_for_crash(tmp.path()).await.unwrap(); + assert!(result.is_none()); + } + + #[tokio::test] + async fn test_crash_detected_with_pid_file() { + let tmp = TempDir::new().unwrap(); + // Write a PID file with a non-existent PID + fs::write(tmp.path().join(PID_FILE), "999999999").await.unwrap(); + let result = check_for_crash(tmp.path()).await.unwrap(); + // No snapshot file → crash detected but no containers to recover + assert!(result.is_none()); + } + + #[tokio::test] + async fn test_crash_with_snapshot() { + let tmp = TempDir::new().unwrap(); + // Write PID file + fs::write(tmp.path().join(PID_FILE), "999999999").await.unwrap(); + // Write container snapshot + let snapshot = ContainerSnapshot { + timestamp: 1000, + containers: vec![ + RunningContainerRecord { + name: "archy-bitcoin-knots".to_string(), + image: "bitcoin-knots:27.1".to_string(), + }, + RunningContainerRecord { + name: "archy-mempool-web".to_string(), + image: "mempool/frontend:3.0".to_string(), + }, + ], + }; + let json = serde_json::to_string(&snapshot).unwrap(); + fs::write(tmp.path().join(CONTAINER_STATE_FILE), json).await.unwrap(); + + let result = check_for_crash(tmp.path()).await.unwrap(); + assert!(result.is_some()); + let containers = result.unwrap(); + assert_eq!(containers.len(), 2); + assert_eq!(containers[0].name, "archy-bitcoin-knots"); + assert_eq!(containers[1].name, "archy-mempool-web"); + } + + #[tokio::test] + async fn test_write_and_remove_pid_marker() { + let tmp = TempDir::new().unwrap(); + write_pid_marker(tmp.path()).await.unwrap(); + assert!(tmp.path().join(PID_FILE).exists()); + + remove_pid_marker(tmp.path()).await; + assert!(!tmp.path().join(PID_FILE).exists()); + } + + #[tokio::test] + async fn test_pid_marker_contains_current_pid() { + let tmp = TempDir::new().unwrap(); + write_pid_marker(tmp.path()).await.unwrap(); + let content = fs::read_to_string(tmp.path().join(PID_FILE)).await.unwrap(); + let saved_pid: u32 = content.parse().unwrap(); + assert_eq!(saved_pid, std::process::id()); + } + + #[tokio::test] + async fn test_clean_shutdown_no_crash_on_restart() { + let tmp = TempDir::new().unwrap(); + + // Simulate: startup → write PID → clean shutdown → remove PID + write_pid_marker(tmp.path()).await.unwrap(); + remove_pid_marker(tmp.path()).await; + + // Next startup should detect no crash + let result = check_for_crash(tmp.path()).await.unwrap(); + assert!(result.is_none()); + } + + #[tokio::test] + async fn test_corrupt_snapshot_handled() { + let tmp = TempDir::new().unwrap(); + fs::write(tmp.path().join(PID_FILE), "999999999").await.unwrap(); + fs::write(tmp.path().join(CONTAINER_STATE_FILE), "not valid json").await.unwrap(); + + // Should not crash, returns None (no recoverable containers) + let result = check_for_crash(tmp.path()).await.unwrap(); + assert!(result.is_none()); + } +} diff --git a/core/archipelago/src/credentials.rs b/core/archipelago/src/credentials.rs index 0cfe180b..b887d547 100644 --- a/core/archipelago/src/credentials.rs +++ b/core/archipelago/src/credentials.rs @@ -1,5 +1,6 @@ -//! Verifiable Credentials (VC) management following W3C VC Data Model. -//! Allows issuing, verifying, and managing credentials tied to DIDs. +//! Verifiable Credentials (VC) management following W3C VC Data Model 2.0. +//! Implements JSON-LD @context, Ed25519Signature2020 proof format. +//! See: https://www.w3.org/TR/vc-data-model-2.0/ use anyhow::{Context, Result}; use serde::{Deserialize, Serialize}; @@ -9,18 +10,58 @@ use tracing::debug; const CREDENTIALS_DIR: &str = "credentials"; -/// A Verifiable Credential following W3C VC Data Model (simplified). +/// W3C VC Data Model 2.0 context URI +const VC_CONTEXT_V2: &str = "https://www.w3.org/ns/credentials/v2"; +/// Ed25519 signature suite context +const ED25519_CONTEXT: &str = "https://w3id.org/security/suites/ed25519-2020/v1"; + +/// A Verifiable Credential following W3C VC Data Model 2.0. #[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] pub struct VerifiableCredential { + #[serde(rename = "@context")] + pub context: Vec, pub id: String, + #[serde(rename = "type")] + pub credential_type: Vec, pub issuer: String, - pub subject: String, - pub credential_type: String, + pub credential_subject: CredentialSubject, + pub issuance_date: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub expiration_date: Option, + pub proof: CredentialProof, + #[serde(skip_serializing_if = "Option::is_none")] + pub credential_status: Option, +} + +/// The subject of a credential with their DID and claims. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CredentialSubject { + pub id: String, + #[serde(flatten)] pub claims: serde_json::Value, - pub issued_at: String, - pub expires_at: Option, - pub signature: String, - pub status: CredentialStatus, +} + +/// Ed25519Signature2020 proof format. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CredentialProof { + #[serde(rename = "type")] + pub proof_type: String, + pub created: String, + pub verification_method: String, + pub proof_purpose: String, + pub proof_value: String, +} + +/// Credential status for revocation tracking. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct CredentialStatusEntry { + pub id: String, + #[serde(rename = "type")] + pub status_type: String, + pub status: String, } #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] @@ -76,8 +117,8 @@ pub async fn save_credentials(data_dir: &Path, store: &CredentialStore) -> Resul fs::write(&path, data).await.context("Writing credentials") } -/// Issue a new Verifiable Credential. -/// The issuer signs the credential claims with their identity key. +/// Issue a new Verifiable Credential following W3C VC Data Model 2.0. +/// Uses Ed25519Signature2020 proof format. pub async fn issue_credential( data_dir: &Path, issuer_did: &str, @@ -87,35 +128,47 @@ pub async fn issue_credential( expires_at: Option<&str>, sign_fn: impl FnOnce(&[u8]) -> Result, ) -> Result { - let id = format!("vc:{}", uuid::Uuid::new_v4()); + let id = format!("urn:uuid:{}", uuid::Uuid::new_v4()); let issued_at = chrono::Utc::now().to_rfc3339(); + let key_id = format!("{}#key-1", issuer_did); - // Build the credential body for signing + // Build the credential body for signing (without proof) let body = serde_json::json!({ + "@context": [VC_CONTEXT_V2, ED25519_CONTEXT], "id": id, + "type": ["VerifiableCredential", credential_type], "issuer": issuer_did, - "subject": subject_did, - "type": credential_type, - "claims": claims, - "issued_at": issued_at, + "credentialSubject": { + "id": subject_did, + }, + "issuanceDate": issued_at, }); let body_bytes = serde_json::to_vec(&body)?; let signature = sign_fn(&body_bytes)?; let vc = VerifiableCredential { + context: vec![VC_CONTEXT_V2.to_string(), ED25519_CONTEXT.to_string()], id: id.clone(), + credential_type: vec!["VerifiableCredential".to_string(), credential_type.to_string()], issuer: issuer_did.to_string(), - subject: subject_did.to_string(), - credential_type: credential_type.to_string(), - claims, - issued_at, - expires_at: expires_at.map(|s| s.to_string()), - signature, - status: CredentialStatus::Active, + credential_subject: CredentialSubject { + id: subject_did.to_string(), + claims, + }, + issuance_date: issued_at.clone(), + expiration_date: expires_at.map(|s| s.to_string()), + proof: CredentialProof { + proof_type: "Ed25519Signature2020".to_string(), + created: issued_at, + verification_method: key_id, + proof_purpose: "assertionMethod".to_string(), + proof_value: signature, + }, + credential_status: None, }; let mut store = load_credentials(data_dir).await?; - debug!(id = %vc.id, "Issued credential"); + debug!(id = %vc.id, "Issued W3C VC"); store.credentials.push(vc.clone()); save_credentials(data_dir, &store).await?; Ok(vc) @@ -126,16 +179,19 @@ pub fn verify_credential( vc: &VerifiableCredential, verify_fn: impl FnOnce(&str, &[u8], &str) -> Result, ) -> Result { + // Reconstruct the body that was signed (without proof) let body = serde_json::json!({ + "@context": vc.context, "id": vc.id, - "issuer": vc.issuer, - "subject": vc.subject, "type": vc.credential_type, - "claims": vc.claims, - "issued_at": vc.issued_at, + "issuer": vc.issuer, + "credentialSubject": { + "id": vc.credential_subject.id, + }, + "issuanceDate": vc.issuance_date, }); let body_bytes = serde_json::to_vec(&body)?; - verify_fn(&vc.issuer, &body_bytes, &vc.signature) + verify_fn(&vc.issuer, &body_bytes, &vc.proof.proof_value) } /// Revoke a credential by ID. @@ -146,7 +202,11 @@ pub async fn revoke_credential(data_dir: &Path, credential_id: &str) -> Result<( .iter_mut() .find(|c| c.id == credential_id) .ok_or_else(|| anyhow::anyhow!("Credential not found: {}", credential_id))?; - vc.status = CredentialStatus::Revoked; + vc.credential_status = Some(CredentialStatusEntry { + id: format!("{}#status", credential_id), + status_type: "CredentialStatusList2021".to_string(), + status: "revoked".to_string(), + }); save_credentials(data_dir, &store).await } @@ -160,10 +220,518 @@ pub async fn list_credentials( store .credentials .into_iter() - .filter(|c| c.issuer == did || c.subject == did) + .filter(|c| c.issuer == did || c.credential_subject.id == did) .collect() } else { store.credentials }; Ok(creds) } + +/// Check if a credential is revoked. +pub fn is_revoked(vc: &VerifiableCredential) -> bool { + vc.credential_status + .as_ref() + .map_or(false, |s| s.status == "revoked") +} + +/// A Verifiable Presentation following W3C VC Data Model 2.0. +/// Bundles one or more VCs with a holder proof. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct VerifiablePresentation { + #[serde(rename = "@context")] + pub context: Vec, + pub id: String, + #[serde(rename = "type")] + pub presentation_type: Vec, + pub holder: String, + pub verifiable_credential: Vec, + pub proof: CredentialProof, +} + +/// Create a Verifiable Presentation wrapping selected credentials. +/// The holder signs the presentation to prove they possess the credentials. +pub fn create_presentation( + holder_did: &str, + credential_ids: &[&str], + credentials: &[VerifiableCredential], + sign_fn: impl FnOnce(&[u8]) -> Result, +) -> Result { + let selected: Vec = credentials + .iter() + .filter(|c| credential_ids.contains(&c.id.as_str())) + .cloned() + .collect(); + + if selected.is_empty() { + return Err(anyhow::anyhow!("No matching credentials found")); + } + + let id = format!("urn:uuid:{}", uuid::Uuid::new_v4()); + let created = chrono::Utc::now().to_rfc3339(); + let key_id = format!("{}#key-1", holder_did); + + // Build the presentation body for signing (without proof) + let body = serde_json::json!({ + "@context": [VC_CONTEXT_V2, ED25519_CONTEXT], + "id": id, + "type": ["VerifiablePresentation"], + "holder": holder_did, + "verifiableCredential": selected, + }); + let body_bytes = serde_json::to_vec(&body)?; + let signature = sign_fn(&body_bytes)?; + + Ok(VerifiablePresentation { + context: vec![VC_CONTEXT_V2.to_string(), ED25519_CONTEXT.to_string()], + id, + presentation_type: vec!["VerifiablePresentation".to_string()], + holder: holder_did.to_string(), + verifiable_credential: selected, + proof: CredentialProof { + proof_type: "Ed25519Signature2020".to_string(), + created, + verification_method: key_id, + proof_purpose: "authentication".to_string(), + proof_value: signature, + }, + }) +} + +/// Verify a Verifiable Presentation: check holder's proof signature, +/// then verify each embedded credential. +pub fn verify_presentation( + vp: &VerifiablePresentation, + verify_fn: impl Fn(&str, &[u8], &str) -> Result, +) -> Result { + // 1. Verify the holder's presentation proof + let body = serde_json::json!({ + "@context": vp.context, + "id": vp.id, + "type": vp.presentation_type, + "holder": vp.holder, + "verifiableCredential": vp.verifiable_credential, + }); + let body_bytes = serde_json::to_vec(&body)?; + let holder_valid = verify_fn(&vp.holder, &body_bytes, &vp.proof.proof_value)?; + + // 2. Verify each embedded credential + let mut credential_results = Vec::new(); + for vc in &vp.verifiable_credential { + let vc_valid = verify_credential(vc, |did, bytes, sig| verify_fn(did, bytes, sig))?; + credential_results.push(CredentialVerificationResult { + id: vc.id.clone(), + valid: vc_valid, + revoked: is_revoked(vc), + }); + } + + let all_valid = holder_valid && credential_results.iter().all(|r| r.valid && !r.revoked); + + Ok(PresentationVerification { + holder_valid, + credentials: credential_results, + valid: all_valid, + }) +} + +/// Result of verifying a Verifiable Presentation. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PresentationVerification { + pub holder_valid: bool, + pub credentials: Vec, + pub valid: bool, +} + +/// Result of verifying a single credential within a presentation. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct CredentialVerificationResult { + pub id: String, + pub valid: bool, + pub revoked: bool, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_credential_status_display() { + assert_eq!(CredentialStatus::Active.to_string(), "active"); + assert_eq!(CredentialStatus::Revoked.to_string(), "revoked"); + assert_eq!(CredentialStatus::Expired.to_string(), "expired"); + } + + #[test] + fn test_credential_status_serde_roundtrip() { + let json = serde_json::to_string(&CredentialStatus::Revoked).unwrap(); + assert_eq!(json, "\"revoked\""); + let parsed: CredentialStatus = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed, CredentialStatus::Revoked); + } + + #[test] + fn test_credential_store_default_is_empty() { + let store = CredentialStore::default(); + assert!(store.credentials.is_empty()); + } + + #[tokio::test] + async fn test_load_credentials_returns_empty_when_no_file() { + let dir = tempfile::tempdir().unwrap(); + let store = load_credentials(dir.path()).await.unwrap(); + assert!(store.credentials.is_empty()); + assert!(dir.path().join(CREDENTIALS_DIR).exists()); + } + + #[tokio::test] + async fn test_issue_credential_w3c_format() { + let dir = tempfile::tempdir().unwrap(); + let vc = issue_credential( + dir.path(), + "did:key:issuer", + "did:key:subject", + "NodeOperator", + serde_json::json!({"role": "admin"}), + Some("2027-12-31T23:59:59Z"), + |_bytes| Ok("mock-signature".to_string()), + ) + .await + .unwrap(); + + // W3C structure checks + assert!(vc.id.starts_with("urn:uuid:")); + assert_eq!(vc.context[0], VC_CONTEXT_V2); + assert_eq!(vc.context[1], ED25519_CONTEXT); + assert_eq!(vc.credential_type, vec!["VerifiableCredential", "NodeOperator"]); + assert_eq!(vc.issuer, "did:key:issuer"); + assert_eq!(vc.credential_subject.id, "did:key:subject"); + assert_eq!(vc.proof.proof_type, "Ed25519Signature2020"); + assert_eq!(vc.proof.proof_purpose, "assertionMethod"); + assert_eq!(vc.proof.verification_method, "did:key:issuer#key-1"); + assert_eq!(vc.proof.proof_value, "mock-signature"); + assert_eq!(vc.expiration_date, Some("2027-12-31T23:59:59Z".to_string())); + assert!(vc.credential_status.is_none()); + } + + #[tokio::test] + async fn test_issue_credential_serializes_as_jsonld() { + let dir = tempfile::tempdir().unwrap(); + let vc = issue_credential( + dir.path(), + "did:key:issuer", + "did:key:subject", + "TestCred", + serde_json::json!({"level": "gold"}), + None, + |_| Ok("sig".to_string()), + ) + .await + .unwrap(); + + let json = serde_json::to_value(&vc).unwrap(); + // Must have @context + assert!(json["@context"].is_array()); + // Must have type array + assert!(json["type"].is_array()); + // Must have credentialSubject + assert!(json["credentialSubject"]["id"].is_string()); + // Must have proof + assert_eq!(json["proof"]["type"], "Ed25519Signature2020"); + } + + #[tokio::test] + async fn test_save_and_load_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + issue_credential( + dir.path(), + "did:key:a", + "did:key:b", + "Type1", + serde_json::json!({"k": "v"}), + None, + |_| Ok("s1".to_string()), + ) + .await + .unwrap(); + + let loaded = load_credentials(dir.path()).await.unwrap(); + assert_eq!(loaded.credentials.len(), 1); + assert_eq!(loaded.credentials[0].credential_type[1], "Type1"); + } + + #[tokio::test] + async fn test_issue_credential_sign_fn_failure_propagates() { + let dir = tempfile::tempdir().unwrap(); + let result = issue_credential( + dir.path(), + "did:key:issuer", + "did:key:subject", + "TestCredential", + serde_json::json!({}), + None, + |_bytes| Err(anyhow::anyhow!("Signing failed")), + ) + .await; + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("Signing failed")); + } + + #[test] + fn test_verify_credential_calls_verify_fn() { + let vc = VerifiableCredential { + context: vec![VC_CONTEXT_V2.to_string()], + id: "urn:uuid:test".to_string(), + credential_type: vec!["VerifiableCredential".to_string(), "Test".to_string()], + issuer: "did:key:issuer".to_string(), + credential_subject: CredentialSubject { + id: "did:key:subject".to_string(), + claims: serde_json::json!({"foo": "bar"}), + }, + issuance_date: "2025-06-01T00:00:00Z".to_string(), + expiration_date: None, + proof: CredentialProof { + proof_type: "Ed25519Signature2020".to_string(), + created: "2025-06-01T00:00:00Z".to_string(), + verification_method: "did:key:issuer#key-1".to_string(), + proof_purpose: "assertionMethod".to_string(), + proof_value: "valid-sig".to_string(), + }, + credential_status: None, + }; + + let result = verify_credential(&vc, |issuer, _data, sig| { + assert_eq!(issuer, "did:key:issuer"); + assert_eq!(sig, "valid-sig"); + Ok(true) + }) + .unwrap(); + assert!(result); + + let result = verify_credential(&vc, |_issuer, _data, _sig| Ok(false)).unwrap(); + assert!(!result); + } + + #[tokio::test] + async fn test_revoke_credential() { + let dir = tempfile::tempdir().unwrap(); + let vc = issue_credential( + dir.path(), + "did:key:issuer", + "did:key:subject", + "Revocable", + serde_json::json!({}), + None, + |_| Ok("sig".to_string()), + ) + .await + .unwrap(); + + assert!(!is_revoked(&vc)); + + revoke_credential(dir.path(), &vc.id).await.unwrap(); + + let store = load_credentials(dir.path()).await.unwrap(); + assert!(is_revoked(&store.credentials[0])); + assert_eq!( + store.credentials[0].credential_status.as_ref().unwrap().status, + "revoked" + ); + } + + #[tokio::test] + async fn test_revoke_nonexistent_credential_fails() { + let dir = tempfile::tempdir().unwrap(); + let result = revoke_credential(dir.path(), "urn:uuid:does-not-exist").await; + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("Credential not found")); + } + + #[tokio::test] + async fn test_list_credentials_no_filter() { + let dir = tempfile::tempdir().unwrap(); + issue_credential( + dir.path(), "did:key:a", "did:key:b", "Type1", + serde_json::json!({}), None, |_| Ok("s1".to_string()), + ).await.unwrap(); + issue_credential( + dir.path(), "did:key:c", "did:key:d", "Type2", + serde_json::json!({}), None, |_| Ok("s2".to_string()), + ).await.unwrap(); + + let all = list_credentials(dir.path(), None).await.unwrap(); + assert_eq!(all.len(), 2); + } + + #[tokio::test] + async fn test_list_credentials_filter_by_did() { + let dir = tempfile::tempdir().unwrap(); + issue_credential( + dir.path(), "did:key:alice", "did:key:bob", "Type1", + serde_json::json!({}), None, |_| Ok("s1".to_string()), + ).await.unwrap(); + issue_credential( + dir.path(), "did:key:carol", "did:key:alice", "Type2", + serde_json::json!({}), None, |_| Ok("s2".to_string()), + ).await.unwrap(); + issue_credential( + dir.path(), "did:key:carol", "did:key:dave", "Type3", + serde_json::json!({}), None, |_| Ok("s3".to_string()), + ).await.unwrap(); + + let filtered = list_credentials(dir.path(), Some("did:key:alice")).await.unwrap(); + assert_eq!(filtered.len(), 2); + } + + fn make_test_vc(id: &str, issuer: &str, subject: &str) -> VerifiableCredential { + VerifiableCredential { + context: vec![VC_CONTEXT_V2.to_string(), ED25519_CONTEXT.to_string()], + id: id.to_string(), + credential_type: vec!["VerifiableCredential".to_string(), "Test".to_string()], + issuer: issuer.to_string(), + credential_subject: CredentialSubject { + id: subject.to_string(), + claims: serde_json::json!({"role": "tester"}), + }, + issuance_date: "2026-01-01T00:00:00Z".to_string(), + expiration_date: None, + proof: CredentialProof { + proof_type: "Ed25519Signature2020".to_string(), + created: "2026-01-01T00:00:00Z".to_string(), + verification_method: format!("{}#key-1", issuer), + proof_purpose: "assertionMethod".to_string(), + proof_value: "mock-sig".to_string(), + }, + credential_status: None, + } + } + + #[test] + fn test_create_presentation() { + let creds = vec![ + make_test_vc("urn:uuid:cred1", "did:key:issuer1", "did:key:holder"), + make_test_vc("urn:uuid:cred2", "did:key:issuer2", "did:key:holder"), + ]; + + let vp = create_presentation( + "did:key:holder", + &["urn:uuid:cred1"], + &creds, + |_bytes| Ok("presentation-sig".to_string()), + ) + .unwrap(); + + assert!(vp.id.starts_with("urn:uuid:")); + assert_eq!(vp.presentation_type, vec!["VerifiablePresentation"]); + assert_eq!(vp.holder, "did:key:holder"); + assert_eq!(vp.verifiable_credential.len(), 1); + assert_eq!(vp.verifiable_credential[0].id, "urn:uuid:cred1"); + assert_eq!(vp.proof.proof_type, "Ed25519Signature2020"); + assert_eq!(vp.proof.proof_purpose, "authentication"); + assert_eq!(vp.proof.proof_value, "presentation-sig"); + assert_eq!(vp.context[0], VC_CONTEXT_V2); + } + + #[test] + fn test_create_presentation_multiple_credentials() { + let creds = vec![ + make_test_vc("urn:uuid:c1", "did:key:i1", "did:key:holder"), + make_test_vc("urn:uuid:c2", "did:key:i2", "did:key:holder"), + make_test_vc("urn:uuid:c3", "did:key:i3", "did:key:other"), + ]; + + let vp = create_presentation( + "did:key:holder", + &["urn:uuid:c1", "urn:uuid:c2"], + &creds, + |_| Ok("sig".to_string()), + ) + .unwrap(); + + assert_eq!(vp.verifiable_credential.len(), 2); + } + + #[test] + fn test_create_presentation_no_matching_credentials() { + let creds = vec![make_test_vc("urn:uuid:c1", "did:key:i", "did:key:h")]; + + let result = create_presentation( + "did:key:holder", + &["urn:uuid:nonexistent"], + &creds, + |_| Ok("sig".to_string()), + ); + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("No matching credentials")); + } + + #[test] + fn test_verify_presentation_all_valid() { + let creds = vec![ + make_test_vc("urn:uuid:c1", "did:key:issuer", "did:key:holder"), + ]; + + let vp = create_presentation( + "did:key:holder", + &["urn:uuid:c1"], + &creds, + |_| Ok("vp-sig".to_string()), + ) + .unwrap(); + + let result = verify_presentation(&vp, |_did, _bytes, _sig| Ok(true)).unwrap(); + assert!(result.holder_valid); + assert!(result.valid); + assert_eq!(result.credentials.len(), 1); + assert!(result.credentials[0].valid); + assert!(!result.credentials[0].revoked); + } + + #[test] + fn test_verify_presentation_holder_invalid() { + let creds = vec![ + make_test_vc("urn:uuid:c1", "did:key:issuer", "did:key:holder"), + ]; + + let vp = create_presentation( + "did:key:holder", + &["urn:uuid:c1"], + &creds, + |_| Ok("bad-sig".to_string()), + ) + .unwrap(); + + // Holder verification fails, credential verification succeeds + let result = verify_presentation(&vp, |did, _bytes, _sig| { + Ok(did != "did:key:holder") + }) + .unwrap(); + + assert!(!result.holder_valid); + assert!(!result.valid); // Overall invalid because holder proof failed + } + + #[test] + fn test_presentation_serializes_as_jsonld() { + let creds = vec![ + make_test_vc("urn:uuid:c1", "did:key:issuer", "did:key:holder"), + ]; + + let vp = create_presentation( + "did:key:holder", + &["urn:uuid:c1"], + &creds, + |_| Ok("sig".to_string()), + ) + .unwrap(); + + let json = serde_json::to_value(&vp).unwrap(); + assert!(json["@context"].is_array()); + assert!(json["type"].is_array()); + assert_eq!(json["type"][0], "VerifiablePresentation"); + assert!(json["holder"].is_string()); + assert!(json["verifiableCredential"].is_array()); + assert!(json["proof"]["type"].is_string()); + } +} diff --git a/core/archipelago/src/data_model.rs b/core/archipelago/src/data_model.rs index 43cf70fa..6a78f15a 100644 --- a/core/archipelago/src/data_model.rs +++ b/core/archipelago/src/data_model.rs @@ -11,9 +11,30 @@ pub struct DataModel { pub package_data: HashMap, #[serde(rename = "peer-health", default, skip_serializing_if = "HashMap::is_empty")] pub peer_health: HashMap, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub notifications: Vec, pub ui: UIData, } +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +pub struct Notification { + pub id: String, + pub level: NotificationLevel, + pub title: String, + pub message: String, + pub timestamp: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub app_id: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "lowercase")] +pub enum NotificationLevel { + Info, + Warning, + Error, +} + #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct ServerInfo { pub id: String, @@ -239,6 +260,7 @@ impl DataModel { }, package_data: HashMap::new(), peer_health: HashMap::new(), + notifications: Vec::new(), ui: UIData { name: None, ack_welcome: String::new(), diff --git a/core/archipelago/src/disk_monitor.rs b/core/archipelago/src/disk_monitor.rs new file mode 100644 index 00000000..2d74f48c --- /dev/null +++ b/core/archipelago/src/disk_monitor.rs @@ -0,0 +1,305 @@ +// Disk Space Monitor +// Periodically checks disk usage and triggers automatic cleanup at 90%. + +use anyhow::{Context, Result}; +use tracing::{info, warn}; + +/// Parse df output into (used_bytes, total_bytes, used_percent). +/// Expects output from `df --block-size=1 --output=used,size /` which has a header line +/// followed by a data line with two whitespace-separated numbers. +fn parse_df_output(stdout: &str) -> Result<(u64, u64, f64)> { + let data_line = stdout + .lines() + .nth(1) + .ok_or_else(|| anyhow::anyhow!("No data line from df"))?; + let mut parts = data_line.split_whitespace(); + let used: u64 = parts + .next() + .ok_or_else(|| anyhow::anyhow!("Missing used"))? + .parse() + .context("parse df used")?; + let total: u64 = parts + .next() + .ok_or_else(|| anyhow::anyhow!("Missing total"))? + .parse() + .context("parse df total")?; + + let percent = if total > 0 { + (used as f64 / total as f64) * 100.0 + } else { + 0.0 + }; + + Ok((used, total, percent)) +} + +/// Check disk usage percentage for the root filesystem. +/// Returns (used_bytes, total_bytes, used_percent). +pub async fn check_disk_usage() -> Result<(u64, u64, f64)> { + let output = tokio::process::Command::new("df") + .args(["--block-size=1", "--output=used,size", "/"]) + .output() + .await + .context("Failed to run df")?; + + if !output.status.success() { + anyhow::bail!("df failed: {}", String::from_utf8_lossy(&output.stderr)); + } + + let stdout = String::from_utf8(output.stdout).context("df output not utf8")?; + parse_df_output(&stdout) +} + +/// Run automatic cleanup when disk usage exceeds 90%. +async fn auto_cleanup() -> Result { + let mut freed: u64 = 0; + + // Prune dangling images + let output = tokio::process::Command::new("sudo") + .args(["podman", "image", "prune", "-f"]) + .output() + .await; + if let Ok(out) = output { + if out.status.success() { + let count = String::from_utf8_lossy(&out.stdout) + .lines() + .filter(|l| !l.trim().is_empty()) + .count(); + freed += count as u64 * 100_000_000; + } + } + + // Clean old rotated logs (> 14 days for auto-cleanup, more aggressive) + let _ = tokio::process::Command::new("sudo") + .args([ + "find", "/var/log", "-type", "f", "-name", "*.log.*", + "-mtime", "+14", "-delete", + ]) + .output() + .await; + + let _ = tokio::process::Command::new("sudo") + .args([ + "find", "/var/log", "-type", "f", "-name", "*.gz", + "-mtime", "+14", "-delete", + ]) + .output() + .await; + + // Truncate large journal logs + let _ = tokio::process::Command::new("sudo") + .args(["journalctl", "--vacuum-size=100M"]) + .output() + .await; + + Ok(freed) +} + +/// Spawn a background task that monitors disk usage every 5 minutes. +/// Triggers automatic cleanup at 90% and logs warnings at 85%. +pub fn spawn_disk_monitor(data_dir: std::path::PathBuf) { + tokio::spawn(async move { + // Initial delay to let system stabilize + tokio::time::sleep(std::time::Duration::from_secs(60)).await; + + let mut interval = tokio::time::interval(std::time::Duration::from_secs(300)); + let mut last_warning_level: Option<&str> = None; + + loop { + interval.tick().await; + + match check_disk_usage().await { + Ok((_used, _total, percent)) => { + if percent >= 90.0 { + if last_warning_level != Some("critical") { + warn!("Disk usage critical: {:.1}% — triggering automatic cleanup", percent); + last_warning_level = Some("critical"); + } + match auto_cleanup().await { + Ok(freed) => { + if freed > 0 { + info!("Auto-cleanup freed approximately {} bytes", freed); + } + } + Err(e) => warn!("Auto-cleanup failed: {}", e), + } + // Write disk warning file for the frontend to poll + let warning_path = data_dir.join("disk-warning.json"); + let _ = tokio::fs::write( + &warning_path, + serde_json::json!({ + "level": "critical", + "percent": (percent * 10.0).round() / 10.0, + "timestamp": chrono::Utc::now().to_rfc3339(), + }) + .to_string(), + ) + .await; + } else if percent >= 85.0 { + if last_warning_level != Some("warning") { + warn!("Disk usage warning: {:.1}% — approaching critical threshold", percent); + last_warning_level = Some("warning"); + } + let warning_path = data_dir.join("disk-warning.json"); + let _ = tokio::fs::write( + &warning_path, + serde_json::json!({ + "level": "warning", + "percent": (percent * 10.0).round() / 10.0, + "timestamp": chrono::Utc::now().to_rfc3339(), + }) + .to_string(), + ) + .await; + } else { + // Clear warning file if disk is healthy + if last_warning_level.is_some() { + let warning_path = data_dir.join("disk-warning.json"); + let _ = tokio::fs::remove_file(&warning_path).await; + last_warning_level = None; + info!("Disk usage back to normal: {:.1}%", percent); + } + } + } + Err(e) => { + tracing::debug!("Disk usage check failed (non-fatal): {}", e); + } + } + } + }); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_df_output_normal() { + // Simulates typical df --block-size=1 --output=used,size / output + let output = " Used Size\n 500000000000 1000000000000\n"; + let (used, total, percent) = parse_df_output(output).unwrap(); + assert_eq!(used, 500_000_000_000); + assert_eq!(total, 1_000_000_000_000); + assert!((percent - 50.0).abs() < 0.01); + } + + #[test] + fn test_parse_df_output_high_usage() { + let output = " Used Size\n 900000000000 1000000000000\n"; + let (used, total, percent) = parse_df_output(output).unwrap(); + assert_eq!(used, 900_000_000_000); + assert_eq!(total, 1_000_000_000_000); + assert!((percent - 90.0).abs() < 0.01); + } + + #[test] + fn test_parse_df_output_almost_full() { + let output = "Used Size\n999 1000\n"; + let (used, total, percent) = parse_df_output(output).unwrap(); + assert_eq!(used, 999); + assert_eq!(total, 1000); + assert!((percent - 99.9).abs() < 0.01); + } + + #[test] + fn test_parse_df_output_empty_disk() { + let output = "Used Size\n0 1000000000000\n"; + let (used, total, percent) = parse_df_output(output).unwrap(); + assert_eq!(used, 0); + assert_eq!(total, 1_000_000_000_000); + assert!((percent - 0.0).abs() < 0.01); + } + + #[test] + fn test_parse_df_output_zero_total() { + // Edge case: total is 0 (should not happen but should not panic/divide-by-zero) + let output = "Used Size\n0 0\n"; + let (used, total, percent) = parse_df_output(output).unwrap(); + assert_eq!(used, 0); + assert_eq!(total, 0); + assert!((percent - 0.0).abs() < 0.01); + } + + #[test] + fn test_parse_df_output_no_data_line() { + let output = "Used Size\n"; + let result = parse_df_output(output); + assert!(result.is_err()); + } + + #[test] + fn test_parse_df_output_empty_string() { + let result = parse_df_output(""); + assert!(result.is_err()); + } + + #[test] + fn test_parse_df_output_single_header_only() { + let output = "Header Only"; + let result = parse_df_output(output); + assert!(result.is_err()); + } + + #[test] + fn test_parse_df_output_non_numeric() { + let output = "Used Size\nabc def\n"; + let result = parse_df_output(output); + assert!(result.is_err()); + } + + #[test] + fn test_parse_df_output_missing_second_field() { + let output = "Used Size\n12345\n"; + let result = parse_df_output(output); + assert!(result.is_err()); + } + + #[test] + fn test_parse_df_output_extra_whitespace() { + let output = " Used Size \n 123456 7890000 \n"; + let (used, total, _) = parse_df_output(output).unwrap(); + assert_eq!(used, 123456); + assert_eq!(total, 7890000); + } + + #[test] + fn test_parse_df_output_real_world_format() { + // Closer to real df output with header padding + let output = " Used Size\n 328000000000 1800000000000\n"; + let (used, total, percent) = parse_df_output(output).unwrap(); + assert_eq!(used, 328_000_000_000); + assert_eq!(total, 1_800_000_000_000); + // ~18.2% + assert!(percent > 18.0 && percent < 19.0); + } + + #[tokio::test] + async fn test_disk_warning_json_format() { + // Verify that the JSON structure we write for disk warnings is valid + let percent: f64 = 92.3; + let json = serde_json::json!({ + "level": "critical", + "percent": (percent * 10.0).round() / 10.0, + "timestamp": chrono::Utc::now().to_rfc3339(), + }); + let s = json.to_string(); + let parsed: serde_json::Value = serde_json::from_str(&s).unwrap(); + assert_eq!(parsed["level"], "critical"); + assert_eq!(parsed["percent"], 92.3); + assert!(parsed["timestamp"].is_string()); + } + + #[tokio::test] + async fn test_disk_warning_json_warning_level() { + let percent: f64 = 87.5; + let json = serde_json::json!({ + "level": "warning", + "percent": (percent * 10.0).round() / 10.0, + "timestamp": chrono::Utc::now().to_rfc3339(), + }); + let parsed: serde_json::Value = serde_json::from_str(&json.to_string()).unwrap(); + assert_eq!(parsed["level"], "warning"); + // 87.5 rounded to 1 decimal = 87.5 + assert_eq!(parsed["percent"], 87.5); + } +} diff --git a/core/archipelago/src/electrs_status.rs b/core/archipelago/src/electrs_status.rs index ca258993..b3c957e9 100644 --- a/core/archipelago/src/electrs_status.rs +++ b/core/archipelago/src/electrs_status.rs @@ -161,7 +161,7 @@ pub async fn get_electrs_sync_status() -> ElectrsSyncStatus { let err_msg = e.to_string(); let (status, error) = if err_msg.contains("connect") || err_msg.contains("Connection refused") { // Estimate progress from data directory size - let est_pct = if data_bytes > 0 { + let _est_pct = if data_bytes > 0 { ((data_bytes as f64 / ESTIMATED_FULL_INDEX_BYTES) * 100.0).min(99.0) } else { 0.0 diff --git a/core/archipelago/src/federation.rs b/core/archipelago/src/federation.rs new file mode 100644 index 00000000..58fd1e75 --- /dev/null +++ b/core/archipelago/src/federation.rs @@ -0,0 +1,764 @@ +//! Node federation: trusted multi-node clusters with state sync. +//! +//! Nodes federate by exchanging invite codes containing DID + onion address. +//! Trust is bilateral — both sides must agree. Federated nodes periodically +//! sync container status, health metrics, and availability. + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::path::Path; +use tokio::fs; + +const FEDERATION_DIR: &str = "federation"; +const NODES_FILE: &str = "nodes.json"; +const INVITES_FILE: &str = "invites.json"; + +/// Trust level for a federated node. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum TrustLevel { + Trusted, + Observer, + Untrusted, +} + +impl std::fmt::Display for TrustLevel { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + TrustLevel::Trusted => write!(f, "trusted"), + TrustLevel::Observer => write!(f, "observer"), + TrustLevel::Untrusted => write!(f, "untrusted"), + } + } +} + +/// A federated node in our cluster. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct FederatedNode { + pub did: String, + pub pubkey: String, + pub onion: String, + #[serde(default)] + pub name: Option, + pub trust_level: TrustLevel, + pub added_at: String, + #[serde(default)] + pub last_seen: Option, + #[serde(default)] + pub last_state: Option, +} + +/// State snapshot received from a federated peer during sync. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct NodeStateSnapshot { + pub timestamp: String, + #[serde(default)] + pub apps: Vec, + #[serde(default)] + pub cpu_usage_percent: Option, + #[serde(default)] + pub mem_used_bytes: Option, + #[serde(default)] + pub mem_total_bytes: Option, + #[serde(default)] + pub disk_used_bytes: Option, + #[serde(default)] + pub disk_total_bytes: Option, + #[serde(default)] + pub uptime_secs: Option, + #[serde(default)] + pub tor_active: Option, +} + +/// Status of a single app/container on a remote node. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AppStatus { + pub id: String, + pub status: String, // "running", "stopped", "installed" + #[serde(default)] + pub version: Option, +} + +/// A pending invite (outgoing or incoming). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct FederationInvite { + pub code: String, + pub did: String, + pub onion: String, + pub pubkey: String, + pub created_at: String, + #[serde(default)] + pub accepted: bool, +} + +/// Top-level file structures. +#[derive(Debug, Default, Serialize, Deserialize)] +struct NodesFile { + nodes: Vec, +} + +#[derive(Debug, Default, Serialize, Deserialize)] +struct InvitesFile { + outgoing: Vec, + incoming: Vec, +} + +/// Ensure federation directory exists. +async fn ensure_dir(data_dir: &Path) -> Result { + let dir = data_dir.join(FEDERATION_DIR); + fs::create_dir_all(&dir) + .await + .context("Failed to create federation directory")?; + Ok(dir) +} + +// ──────────────────────────── Node Management ──────────────────────────── + +pub async fn load_nodes(data_dir: &Path) -> Result> { + let dir = data_dir.join(FEDERATION_DIR); + let path = dir.join(NODES_FILE); + if !path.exists() { + return Ok(Vec::new()); + } + let content = fs::read_to_string(&path) + .await + .context("Failed to read federation nodes")?; + let file: NodesFile = serde_json::from_str(&content).unwrap_or_default(); + Ok(file.nodes) +} + +pub async fn save_nodes(data_dir: &Path, nodes: &[FederatedNode]) -> Result<()> { + let dir = ensure_dir(data_dir).await?; + let file = NodesFile { + nodes: nodes.to_vec(), + }; + let content = serde_json::to_string_pretty(&file).context("Failed to serialize nodes")?; + fs::write(dir.join(NODES_FILE), content) + .await + .context("Failed to write federation nodes")?; + Ok(()) +} + +pub async fn add_node(data_dir: &Path, node: FederatedNode) -> Result> { + let mut nodes = load_nodes(data_dir).await?; + let exists = nodes.iter().any(|n| n.did == node.did); + if exists { + anyhow::bail!("Node with DID {} is already federated", node.did); + } + nodes.push(node); + save_nodes(data_dir, &nodes).await?; + Ok(nodes) +} + +pub async fn remove_node(data_dir: &Path, did: &str) -> Result> { + let mut nodes = load_nodes(data_dir).await?; + let before = nodes.len(); + nodes.retain(|n| n.did != did); + if nodes.len() == before { + anyhow::bail!("No federated node with DID {}", did); + } + save_nodes(data_dir, &nodes).await?; + Ok(nodes) +} + +pub async fn set_trust_level( + data_dir: &Path, + did: &str, + trust: TrustLevel, +) -> Result> { + let mut nodes = load_nodes(data_dir).await?; + let node = nodes + .iter_mut() + .find(|n| n.did == did) + .ok_or_else(|| anyhow::anyhow!("No federated node with DID {}", did))?; + node.trust_level = trust; + save_nodes(data_dir, &nodes).await?; + Ok(nodes) +} + +pub async fn update_node_state( + data_dir: &Path, + did: &str, + state: NodeStateSnapshot, +) -> Result<()> { + let mut nodes = load_nodes(data_dir).await?; + if let Some(node) = nodes.iter_mut().find(|n| n.did == did) { + node.last_seen = Some(state.timestamp.clone()); + node.last_state = Some(state); + save_nodes(data_dir, &nodes).await?; + } + Ok(()) +} + +// ──────────────────────────── Invite Management ──────────────────────────── + +async fn load_invites(data_dir: &Path) -> Result { + let dir = data_dir.join(FEDERATION_DIR); + let path = dir.join(INVITES_FILE); + if !path.exists() { + return Ok(InvitesFile::default()); + } + let content = fs::read_to_string(&path) + .await + .context("Failed to read invites")?; + let file: InvitesFile = serde_json::from_str(&content).unwrap_or_default(); + Ok(file) +} + +async fn save_invites(data_dir: &Path, invites: &InvitesFile) -> Result<()> { + let dir = ensure_dir(data_dir).await?; + let content = serde_json::to_string_pretty(invites).context("Failed to serialize invites")?; + fs::write(dir.join(INVITES_FILE), content) + .await + .context("Failed to write invites")?; + Ok(()) +} + +/// Generate an invite code. Format: `fed1:` +pub async fn create_invite( + data_dir: &Path, + did: &str, + onion: &str, + pubkey: &str, +) -> Result { + use base64::Engine; + use rand::Rng; + + let mut token_bytes = [0u8; 16]; + rand::thread_rng().fill(&mut token_bytes); + let token = hex::encode(token_bytes); + + let payload = serde_json::json!({ + "did": did, + "onion": onion, + "pubkey": pubkey, + "token": token, + }); + let json = serde_json::to_string(&payload).context("Failed to serialize invite")?; + let code = format!( + "fed1:{}", + base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(json.as_bytes()) + ); + + let invite = FederationInvite { + code: code.clone(), + did: did.to_string(), + onion: onion.to_string(), + pubkey: pubkey.to_string(), + created_at: chrono::Utc::now().to_rfc3339(), + accepted: false, + }; + + let mut invites = load_invites(data_dir).await?; + invites.outgoing.push(invite); + save_invites(data_dir, &invites).await?; + + Ok(code) +} + +/// Parse an invite code into its components. +pub fn parse_invite(code: &str) -> Result<(String, String, String, String)> { + use base64::Engine; + + let encoded = code + .strip_prefix("fed1:") + .ok_or_else(|| anyhow::anyhow!("Invalid invite format: must start with fed1:"))?; + + let bytes = base64::engine::general_purpose::URL_SAFE_NO_PAD + .decode(encoded) + .context("Invalid base64 in invite code")?; + + let payload: serde_json::Value = + serde_json::from_slice(&bytes).context("Invalid JSON in invite")?; + + let did = payload["did"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing did in invite"))? + .to_string(); + let onion = payload["onion"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing onion in invite"))? + .to_string(); + let pubkey = payload["pubkey"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing pubkey in invite"))? + .to_string(); + let token = payload["token"] + .as_str() + .ok_or_else(|| anyhow::anyhow!("Missing token in invite"))? + .to_string(); + + Ok((did, onion, pubkey, token)) +} + +/// Accept an invite: parse code, verify the remote node, add to federation. +pub async fn accept_invite( + data_dir: &Path, + code: &str, + local_did: &str, + local_onion: &str, + local_pubkey: &str, +) -> Result { + let (did, onion, pubkey, _token) = parse_invite(code)?; + + // Check not already federated + let nodes = load_nodes(data_dir).await?; + if nodes.iter().any(|n| n.did == did) { + anyhow::bail!("Already federated with node {}", did); + } + + let node = FederatedNode { + did: did.clone(), + pubkey, + onion, + name: None, + trust_level: TrustLevel::Trusted, + added_at: chrono::Utc::now().to_rfc3339(), + last_seen: None, + last_state: None, + }; + + add_node(data_dir, node.clone()).await?; + + // Record as incoming accepted invite + let mut invites = load_invites(data_dir).await?; + invites.incoming.push(FederationInvite { + code: code.to_string(), + did: did.clone(), + onion: node.onion.clone(), + pubkey: node.pubkey.clone(), + created_at: chrono::Utc::now().to_rfc3339(), + accepted: true, + }); + save_invites(data_dir, &invites).await?; + + // Notify remote node (best-effort over Tor) + let _ = notify_join(&node.onion, local_did, local_onion, local_pubkey).await; + + Ok(node) +} + +/// Best-effort notification to the remote node that we joined their federation. +async fn notify_join( + remote_onion: &str, + local_did: &str, + local_onion: &str, + local_pubkey: &str, +) -> Result<()> { + let host = if remote_onion.ends_with(".onion") { + remote_onion.to_string() + } else { + format!("{}.onion", remote_onion) + }; + let url = format!("http://{}/rpc/v1", host); + let body = serde_json::json!({ + "method": "federation.peer-joined", + "params": { + "did": local_did, + "onion": local_onion, + "pubkey": local_pubkey, + } + }); + + let proxy = reqwest::Proxy::all("socks5h://127.0.0.1:9050").context("Invalid Tor proxy")?; + let client = reqwest::Client::builder() + .proxy(proxy) + .timeout(std::time::Duration::from_secs(30)) + .build() + .context("Failed to build HTTP client")?; + + let _ = client.post(&url).json(&body).send().await; + Ok(()) +} + +/// Sync state with a single federated peer over Tor. +pub async fn sync_with_peer( + data_dir: &Path, + peer: &FederatedNode, + local_did: &str, + sign_fn: impl FnOnce(&[u8]) -> String, +) -> Result { + let host = if peer.onion.ends_with(".onion") { + peer.onion.clone() + } else { + format!("{}.onion", peer.onion) + }; + let url = format!("http://{}/rpc/v1", host); + + // Sign current timestamp for authentication + let timestamp = chrono::Utc::now().to_rfc3339(); + let signature = sign_fn(timestamp.as_bytes()); + + let body = serde_json::json!({ + "method": "federation.get-state", + "params": {} + }); + + let proxy = reqwest::Proxy::all("socks5h://127.0.0.1:9050").context("Invalid Tor proxy")?; + let client = reqwest::Client::builder() + .proxy(proxy) + .timeout(std::time::Duration::from_secs(30)) + .build() + .context("Failed to build HTTP client")?; + + let resp = client + .post(&url) + .header("X-Federation-DID", local_did) + .header("X-Federation-Sig", &signature) + .header("X-Federation-Timestamp", ×tamp) + .json(&body) + .send() + .await + .context("Failed to reach federated peer")?; + + if !resp.status().is_success() { + anyhow::bail!("Peer returned {}", resp.status()); + } + + let result: serde_json::Value = resp.json().await.context("Invalid response from peer")?; + let state_val = result + .get("result") + .ok_or_else(|| anyhow::anyhow!("No result in peer response"))?; + + let state: NodeStateSnapshot = + serde_json::from_value(state_val.clone()).context("Failed to parse peer state")?; + + update_node_state(data_dir, &peer.did, state.clone()).await?; + + Ok(state) +} + +/// Build the local node's state snapshot for sharing with peers. +pub fn build_local_state( + apps: Vec, + cpu: f64, + mem_used: u64, + mem_total: u64, + disk_used: u64, + disk_total: u64, + uptime: u64, + tor_active: bool, +) -> NodeStateSnapshot { + NodeStateSnapshot { + timestamp: chrono::Utc::now().to_rfc3339(), + apps, + cpu_usage_percent: Some(cpu), + mem_used_bytes: Some(mem_used), + mem_total_bytes: Some(mem_total), + disk_used_bytes: Some(disk_used), + disk_total_bytes: Some(disk_total), + uptime_secs: Some(uptime), + tor_active: Some(tor_active), + } +} + +/// Deploy an app to a remote federated peer over Tor. +/// Only works if the peer is trusted and the app exists in our marketplace. +pub async fn deploy_to_peer( + peer: &FederatedNode, + app_id: &str, + version: &str, + marketplace_url: &str, + local_did: &str, + sign_fn: impl FnOnce(&[u8]) -> String, +) -> Result { + if peer.trust_level != TrustLevel::Trusted { + anyhow::bail!("Can only deploy to trusted peers (current: {})", peer.trust_level); + } + + let host = if peer.onion.ends_with(".onion") { + peer.onion.clone() + } else { + format!("{}.onion", peer.onion) + }; + let url = format!("http://{}/rpc/v1", host); + + let timestamp = chrono::Utc::now().to_rfc3339(); + let signature = sign_fn(timestamp.as_bytes()); + + let body = serde_json::json!({ + "method": "package.install", + "params": { + "id": app_id, + "version": version, + "marketplace-url": marketplace_url, + } + }); + + let proxy = reqwest::Proxy::all("socks5h://127.0.0.1:9050").context("Invalid Tor proxy")?; + let client = reqwest::Client::builder() + .proxy(proxy) + .timeout(std::time::Duration::from_secs(120)) + .build() + .context("Failed to build HTTP client")?; + + let resp = client + .post(&url) + .header("X-Federation-DID", local_did) + .header("X-Federation-Sig", &signature) + .header("X-Federation-Timestamp", ×tamp) + .json(&body) + .send() + .await + .context("Failed to reach federated peer for deploy")?; + + if !resp.status().is_success() { + anyhow::bail!("Remote node returned HTTP {}", resp.status()); + } + + let result: serde_json::Value = resp.json().await.context("Invalid response from peer")?; + + if let Some(err) = result.get("error") { + if !err.is_null() { + let msg = err.get("message").and_then(|m| m.as_str()).unwrap_or("Unknown remote error"); + anyhow::bail!("Remote node refused deploy: {}", msg); + } + } + + Ok(serde_json::json!({ + "deployed": true, + "app_id": app_id, + "peer_did": peer.did, + "peer_onion": peer.onion, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn make_node(did: &str, onion: &str) -> FederatedNode { + FederatedNode { + did: did.to_string(), + pubkey: "aabbccdd".to_string(), + onion: onion.to_string(), + name: None, + trust_level: TrustLevel::Trusted, + added_at: "2026-01-01T00:00:00Z".to_string(), + last_seen: None, + last_state: None, + } + } + + #[test] + fn test_trust_level_serialization() { + let json = serde_json::to_string(&TrustLevel::Trusted).unwrap(); + assert_eq!(json, "\"trusted\""); + + let parsed: TrustLevel = serde_json::from_str("\"observer\"").unwrap(); + assert_eq!(parsed, TrustLevel::Observer); + } + + #[test] + fn test_federated_node_serialization_roundtrip() { + let node = make_node("did:key:zABC", "test.onion"); + let json = serde_json::to_string(&node).unwrap(); + let parsed: FederatedNode = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.did, "did:key:zABC"); + assert_eq!(parsed.trust_level, TrustLevel::Trusted); + assert!(parsed.last_state.is_none()); + } + + #[test] + fn test_node_state_snapshot_defaults() { + let json = r#"{"timestamp": "2026-01-01T00:00:00Z"}"#; + let state: NodeStateSnapshot = serde_json::from_str(json).unwrap(); + assert!(state.apps.is_empty()); + assert!(state.cpu_usage_percent.is_none()); + } + + #[tokio::test] + async fn test_load_nodes_empty_when_no_file() { + let dir = tempfile::tempdir().unwrap(); + let nodes = load_nodes(dir.path()).await.unwrap(); + assert!(nodes.is_empty()); + } + + #[tokio::test] + async fn test_save_and_load_nodes_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + let nodes = vec![ + make_node("did:key:z1", "a.onion"), + make_node("did:key:z2", "b.onion"), + ]; + save_nodes(dir.path(), &nodes).await.unwrap(); + let loaded = load_nodes(dir.path()).await.unwrap(); + assert_eq!(loaded.len(), 2); + assert_eq!(loaded[0].did, "did:key:z1"); + } + + #[tokio::test] + async fn test_add_node_deduplicates_by_did() { + let dir = tempfile::tempdir().unwrap(); + add_node(dir.path(), make_node("did:key:z1", "a.onion")) + .await + .unwrap(); + let result = add_node(dir.path(), make_node("did:key:z1", "b.onion")).await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_remove_node_by_did() { + let dir = tempfile::tempdir().unwrap(); + add_node(dir.path(), make_node("did:key:z1", "a.onion")) + .await + .unwrap(); + add_node(dir.path(), make_node("did:key:z2", "b.onion")) + .await + .unwrap(); + let result = remove_node(dir.path(), "did:key:z1").await.unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result[0].did, "did:key:z2"); + } + + #[tokio::test] + async fn test_remove_nonexistent_node_errors() { + let dir = tempfile::tempdir().unwrap(); + let result = remove_node(dir.path(), "did:key:nonexistent").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_set_trust_level() { + let dir = tempfile::tempdir().unwrap(); + add_node(dir.path(), make_node("did:key:z1", "a.onion")) + .await + .unwrap(); + let nodes = set_trust_level(dir.path(), "did:key:z1", TrustLevel::Observer) + .await + .unwrap(); + assert_eq!(nodes[0].trust_level, TrustLevel::Observer); + } + + #[tokio::test] + async fn test_update_node_state() { + let dir = tempfile::tempdir().unwrap(); + add_node(dir.path(), make_node("did:key:z1", "a.onion")) + .await + .unwrap(); + + let state = NodeStateSnapshot { + timestamp: "2026-03-10T12:00:00Z".to_string(), + apps: vec![AppStatus { + id: "bitcoin".to_string(), + status: "running".to_string(), + version: Some("27.0".to_string()), + }], + cpu_usage_percent: Some(45.2), + mem_used_bytes: Some(4_000_000_000), + mem_total_bytes: Some(8_000_000_000), + disk_used_bytes: None, + disk_total_bytes: None, + uptime_secs: Some(86400), + tor_active: Some(true), + }; + + update_node_state(dir.path(), "did:key:z1", state) + .await + .unwrap(); + + let nodes = load_nodes(dir.path()).await.unwrap(); + assert!(nodes[0].last_seen.is_some()); + let ls = nodes[0].last_state.as_ref().unwrap(); + assert_eq!(ls.apps.len(), 1); + assert_eq!(ls.cpu_usage_percent, Some(45.2)); + } + + #[tokio::test] + async fn test_create_and_parse_invite() { + let dir = tempfile::tempdir().unwrap(); + let code = create_invite(dir.path(), "did:key:z1", "test.onion", "aabbcc") + .await + .unwrap(); + assert!(code.starts_with("fed1:")); + + let (did, onion, pubkey, token) = parse_invite(&code).unwrap(); + assert_eq!(did, "did:key:z1"); + assert_eq!(onion, "test.onion"); + assert_eq!(pubkey, "aabbcc"); + assert_eq!(token.len(), 32); // 16 bytes = 32 hex chars + } + + #[test] + fn test_parse_invalid_invite() { + assert!(parse_invite("invalid").is_err()); + assert!(parse_invite("fed1:not-valid-base64!!!").is_err()); + } + + #[tokio::test] + async fn test_accept_invite_creates_node() { + let dir = tempfile::tempdir().unwrap(); + let code = create_invite(dir.path(), "did:key:zRemote", "remote.onion", "remotepub") + .await + .unwrap(); + + // Accept from a different "local" perspective + let dir2 = tempfile::tempdir().unwrap(); + let node = accept_invite( + dir2.path(), + &code, + "did:key:zLocal", + "local.onion", + "localpub", + ) + .await + .unwrap(); + + assert_eq!(node.did, "did:key:zRemote"); + assert_eq!(node.trust_level, TrustLevel::Trusted); + + let nodes = load_nodes(dir2.path()).await.unwrap(); + assert_eq!(nodes.len(), 1); + } + + #[tokio::test] + async fn test_accept_invite_rejects_duplicate() { + let dir = tempfile::tempdir().unwrap(); + let code = create_invite(dir.path(), "did:key:zRemote", "remote.onion", "remotepub") + .await + .unwrap(); + + let dir2 = tempfile::tempdir().unwrap(); + accept_invite( + dir2.path(), + &code, + "did:key:zLocal", + "local.onion", + "localpub", + ) + .await + .unwrap(); + + // Accepting the same invite again should fail + let result = accept_invite( + dir2.path(), + &code, + "did:key:zLocal", + "local.onion", + "localpub", + ) + .await; + assert!(result.is_err()); + } + + #[test] + fn test_build_local_state() { + let state = build_local_state( + vec![AppStatus { + id: "lnd".to_string(), + status: "running".to_string(), + version: Some("0.18".to_string()), + }], + 25.5, + 2_000_000_000, + 8_000_000_000, + 100_000_000_000, + 500_000_000_000, + 3600, + true, + ); + assert_eq!(state.apps.len(), 1); + assert_eq!(state.cpu_usage_percent, Some(25.5)); + assert_eq!(state.tor_active, Some(true)); + } +} diff --git a/core/archipelago/src/health_monitor.rs b/core/archipelago/src/health_monitor.rs new file mode 100644 index 00000000..78264d5b --- /dev/null +++ b/core/archipelago/src/health_monitor.rs @@ -0,0 +1,344 @@ +// Container Health Monitor +// Checks container health every 60s, auto-restarts unhealthy containers (max 3 times), +// and sends WebSocket notifications to the UI on failure. + +use crate::data_model::{Notification, NotificationLevel}; +use crate::state::StateManager; +use std::collections::HashMap; +use std::sync::Arc; +use tracing::{debug, info, warn}; + +const MAX_RESTART_ATTEMPTS: u32 = 3; +const CHECK_INTERVAL_SECS: u64 = 60; + +/// Track restart attempts per container to avoid infinite restart loops. +struct RestartTracker { + attempts: HashMap, +} + +impl RestartTracker { + fn new() -> Self { + Self { + attempts: HashMap::new(), + } + } + + /// Record a restart attempt. Returns false if max attempts exceeded. + fn record_attempt(&mut self, name: &str) -> bool { + let count = self.attempts.entry(name.to_string()).or_insert(0); + *count += 1; + *count <= MAX_RESTART_ATTEMPTS + } + + /// Clear restart count when a container is healthy again. + fn clear(&mut self, name: &str) { + self.attempts.remove(name); + } + + fn attempt_count(&self, name: &str) -> u32 { + *self.attempts.get(name).unwrap_or(&0) + } +} + +#[derive(Debug, Clone)] +struct ContainerHealth { + name: String, + app_id: String, + state: String, + healthy: bool, +} + +/// Query all containers and their health status. +async fn check_containers() -> Vec { + let output = match tokio::process::Command::new("sudo") + .args(["podman", "ps", "-a", "--format", "json"]) + .output() + .await + { + Ok(o) if o.status.success() => o, + _ => return Vec::new(), + }; + + let stdout = String::from_utf8_lossy(&output.stdout); + let containers: Vec = + serde_json::from_str(&stdout).unwrap_or_default(); + + // Backend services to skip + let skip = [ + "btcpay-db", "nbxplorer", "mempool-db", "mempool-api", + "penpot-postgres", "penpot-backend", "penpot-exporter", "penpot-valkey", + "penpot-mailcatch", "immich_postgres", "immich_redis", + "endurain-db", "nextcloud-db", + ]; + + containers + .iter() + .filter_map(|c| { + let name = c.get("Names").and_then(|v| { + if let Some(arr) = v.as_array() { + arr.first().and_then(|n| n.as_str()).map(|s| s.to_string()) + } else { + v.as_str().map(|s| s.to_string()) + } + })?; + + let app_id = name + .strip_prefix("archy-") + .unwrap_or(&name) + .to_string(); + + if skip.contains(&app_id.as_str()) || app_id.ends_with("-ui") { + return None; + } + + let state = c.get("State") + .and_then(|v| v.as_str()) + .unwrap_or("unknown") + .to_lowercase(); + + let healthy = state == "running"; + + Some(ContainerHealth { + name, + app_id, + state, + healthy, + }) + }) + .collect() +} + +/// Try to restart a container. +async fn restart_container(name: &str) -> bool { + info!("Auto-restarting unhealthy container: {}", name); + let result = tokio::process::Command::new("sudo") + .args(["podman", "start", name]) + .output() + .await; + + match result { + Ok(output) if output.status.success() => { + info!("Successfully restarted container: {}", name); + true + } + Ok(output) => { + let stderr = String::from_utf8_lossy(&output.stderr); + warn!("Failed to restart container {}: {}", name, stderr.trim()); + false + } + Err(e) => { + warn!("Failed to execute podman start for {}: {}", name, e); + false + } + } +} + +/// Spawn the health monitor background task. +pub fn spawn_health_monitor(state: Arc) { + tokio::spawn(async move { + // Wait 2 minutes for containers to start up + tokio::time::sleep(std::time::Duration::from_secs(120)).await; + + let mut tracker = RestartTracker::new(); + let mut interval = tokio::time::interval(std::time::Duration::from_secs(CHECK_INTERVAL_SECS)); + + loop { + interval.tick().await; + + let containers = check_containers().await; + if containers.is_empty() { + continue; + } + + let mut state_changed = false; + let (mut data, _) = state.get_snapshot().await; + + for container in &containers { + if container.healthy { + // Clear restart tracker if container recovered + if tracker.attempt_count(&container.name) > 0 { + info!("Container {} is healthy again after restart", container.name); + tracker.clear(&container.name); + } + continue; + } + + // Container is unhealthy (exited/stopped) + // Only try auto-restart if we haven't exceeded max attempts + if container.state == "exited" || container.state == "stopped" { + let attempts = tracker.attempt_count(&container.name); + + if attempts >= MAX_RESTART_ATTEMPTS { + // Already notified, skip + debug!("Container {} exceeded max restart attempts ({})", container.name, MAX_RESTART_ATTEMPTS); + continue; + } + + if tracker.record_attempt(&container.name) { + let restarted = restart_container(&container.name).await; + let attempt = tracker.attempt_count(&container.name); + + if !restarted || attempt >= MAX_RESTART_ATTEMPTS { + // Push notification to UI + let notification = Notification { + id: format!("health-{}-{}", container.app_id, chrono::Utc::now().timestamp()), + level: NotificationLevel::Error, + title: format!("{} is unhealthy", container.app_id), + message: if restarted { + format!( + "Container restarted ({}/{} attempts). May need manual attention.", + attempt, MAX_RESTART_ATTEMPTS + ) + } else { + format!( + "Auto-restart failed (attempt {}/{}). Container state: {}", + attempt, MAX_RESTART_ATTEMPTS, container.state + ) + }, + timestamp: chrono::Utc::now().to_rfc3339(), + app_id: Some(container.app_id.clone()), + }; + + // Keep only the latest 20 notifications + data.notifications.push(notification); + if data.notifications.len() > 20 { + data.notifications = data.notifications.split_off(data.notifications.len() - 20); + } + state_changed = true; + } + } + } + } + + if state_changed { + state.update_data(data).await; + debug!("Health monitor: state updated with notifications"); + } + } + }); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_restart_tracker_new_is_empty() { + let tracker = RestartTracker::new(); + assert_eq!(tracker.attempt_count("any-container"), 0); + } + + #[test] + fn test_restart_tracker_record_attempt_increments() { + let mut tracker = RestartTracker::new(); + assert!(tracker.record_attempt("test-container")); + assert_eq!(tracker.attempt_count("test-container"), 1); + + assert!(tracker.record_attempt("test-container")); + assert_eq!(tracker.attempt_count("test-container"), 2); + + assert!(tracker.record_attempt("test-container")); + assert_eq!(tracker.attempt_count("test-container"), 3); + } + + #[test] + fn test_restart_tracker_max_attempts_exceeded() { + let mut tracker = RestartTracker::new(); + // First MAX_RESTART_ATTEMPTS attempts should return true + for i in 1..=MAX_RESTART_ATTEMPTS { + assert!( + tracker.record_attempt("container-a"), + "Attempt {} should be allowed", + i + ); + } + // Next attempt exceeds max, returns false + assert!(!tracker.record_attempt("container-a")); + assert_eq!(tracker.attempt_count("container-a"), MAX_RESTART_ATTEMPTS + 1); + } + + #[test] + fn test_restart_tracker_independent_containers() { + let mut tracker = RestartTracker::new(); + tracker.record_attempt("container-a"); + tracker.record_attempt("container-a"); + tracker.record_attempt("container-b"); + + assert_eq!(tracker.attempt_count("container-a"), 2); + assert_eq!(tracker.attempt_count("container-b"), 1); + assert_eq!(tracker.attempt_count("container-c"), 0); + } + + #[test] + fn test_restart_tracker_clear_resets_count() { + let mut tracker = RestartTracker::new(); + tracker.record_attempt("container-x"); + tracker.record_attempt("container-x"); + assert_eq!(tracker.attempt_count("container-x"), 2); + + tracker.clear("container-x"); + assert_eq!(tracker.attempt_count("container-x"), 0); + } + + #[test] + fn test_restart_tracker_clear_allows_new_attempts() { + let mut tracker = RestartTracker::new(); + // Exhaust attempts + for _ in 0..=MAX_RESTART_ATTEMPTS { + tracker.record_attempt("container-y"); + } + assert!(!tracker.record_attempt("container-y")); + + // Clear and try again + tracker.clear("container-y"); + assert!(tracker.record_attempt("container-y")); + assert_eq!(tracker.attempt_count("container-y"), 1); + } + + #[test] + fn test_restart_tracker_clear_nonexistent_is_safe() { + let mut tracker = RestartTracker::new(); + // Should not panic + tracker.clear("nonexistent"); + assert_eq!(tracker.attempt_count("nonexistent"), 0); + } + + #[test] + fn test_container_health_struct() { + let health = ContainerHealth { + name: "archy-bitcoin-knots".to_string(), + app_id: "bitcoin-knots".to_string(), + state: "running".to_string(), + healthy: true, + }; + assert!(health.healthy); + assert_eq!(health.name, "archy-bitcoin-knots"); + assert_eq!(health.app_id, "bitcoin-knots"); + assert_eq!(health.state, "running"); + } + + #[test] + fn test_container_health_unhealthy() { + let health = ContainerHealth { + name: "archy-mempool-web".to_string(), + app_id: "mempool-web".to_string(), + state: "exited".to_string(), + healthy: false, + }; + assert!(!health.healthy); + assert_eq!(health.state, "exited"); + } + + #[test] + fn test_max_restart_attempts_constant() { + // Ensure the constant is a reasonable value (not 0, not too high) + assert!(MAX_RESTART_ATTEMPTS >= 1); + assert!(MAX_RESTART_ATTEMPTS <= 10); + assert_eq!(MAX_RESTART_ATTEMPTS, 3); + } + + #[test] + fn test_check_interval_constant() { + assert_eq!(CHECK_INTERVAL_SECS, 60); + } +} diff --git a/core/archipelago/src/identity.rs b/core/archipelago/src/identity.rs index 13c092be..406f0477 100644 --- a/core/archipelago/src/identity.rs +++ b/core/archipelago/src/identity.rs @@ -105,6 +105,11 @@ impl NodeIdentity { pub fn did_key(&self) -> String { did_key_from_pubkey_hex(&self.pubkey_hex()).expect("pubkey_hex is valid") } + + /// Generate a W3C DID Core v1.0 compliant DID Document. + pub fn did_document(&self) -> serde_json::Value { + did_document_from_pubkey_hex(&self.pubkey_hex()).expect("pubkey_hex is valid") + } } /// Convert Ed25519 pubkey (hex) to did:key format. @@ -121,6 +126,77 @@ pub fn did_key_from_pubkey_hex(pubkey_hex: &str) -> Result { Ok(format!("did:key:z{}", bs58::encode(multicodec_pubkey).into_string())) } +/// Generate a W3C DID Core v1.0 compliant DID Document from an Ed25519 public key. +/// Follows: https://www.w3.org/TR/did-core/ +/// Includes: verificationMethod, authentication, assertionMethod, keyAgreement contexts. +pub fn did_document_from_pubkey_hex(pubkey_hex: &str) -> Result { + let did = did_key_from_pubkey_hex(pubkey_hex)?; + let pubkey_bytes = hex::decode(pubkey_hex).context("Invalid pubkey hex")?; + let pubkey_multibase = format!("z{}", bs58::encode(&pubkey_bytes).into_string()); + let key_id = format!("{}#key-1", did); + + // Build X25519 key agreement key from Ed25519 public key + // Ed25519 -> X25519 conversion (Montgomery form) + let ed_point = curve25519_dalek::edwards::CompressedEdwardsY( + pubkey_bytes + .as_slice() + .try_into() + .map_err(|_| anyhow::anyhow!("Invalid pubkey length"))?, + ); + let x25519_key = if let Some(point) = ed_point.decompress() { + let montgomery = point.to_montgomery(); + format!("z{}", bs58::encode(montgomery.as_bytes()).into_string()) + } else { + // Fallback: use Ed25519 key if conversion fails + pubkey_multibase.clone() + }; + let x25519_key_id = format!("{}#key-x25519-1", did); + + Ok(serde_json::json!({ + "@context": [ + "https://www.w3.org/ns/did/v1", + "https://w3id.org/security/suites/ed25519-2020/v1", + "https://w3id.org/security/suites/x25519-2020/v1" + ], + "id": did, + "verificationMethod": [ + { + "id": key_id, + "type": "Ed25519VerificationKey2020", + "controller": did, + "publicKeyMultibase": pubkey_multibase + }, + { + "id": x25519_key_id, + "type": "X25519KeyAgreementKey2020", + "controller": did, + "publicKeyMultibase": x25519_key + } + ], + "authentication": [key_id], + "assertionMethod": [key_id], + "capabilityInvocation": [key_id], + "capabilityDelegation": [key_id], + "keyAgreement": [x25519_key_id] + })) +} + +/// Extract the raw 32-byte Ed25519 public key from a did:key string. +pub fn pubkey_bytes_from_did_key(did: &str) -> Result<[u8; 32]> { + let multibase_str = did + .strip_prefix("did:key:z") + .ok_or_else(|| anyhow::anyhow!("Invalid did:key format"))?; + let decoded = bs58::decode(multibase_str) + .into_vec() + .context("Invalid base58 in did:key")?; + if decoded.len() != 34 || decoded[0] != 0xed || decoded[1] != 0x01 { + return Err(anyhow::anyhow!("Invalid Ed25519 multicodec prefix")); + } + let mut pubkey = [0u8; 32]; + pubkey.copy_from_slice(&decoded[2..34]); + Ok(pubkey) +} + #[cfg(test)] mod tests { use super::*; @@ -210,4 +286,60 @@ mod tests { assert!(addr.starts_with("archipelago://abc123.onion#")); assert!(addr.contains(&identity.pubkey_hex())); } + + #[tokio::test] + async fn test_did_document_w3c_structure() { + let dir = tempfile::tempdir().unwrap(); + let identity = NodeIdentity::load_or_create(&dir.path().join("id")).await.unwrap(); + + let doc = identity.did_document(); + let did = identity.did_key(); + + // Verify @context + let context = doc["@context"].as_array().unwrap(); + assert_eq!(context[0], "https://www.w3.org/ns/did/v1"); + + // Verify id matches did:key + assert_eq!(doc["id"], did); + + // Verify verificationMethod has Ed25519 and X25519 keys + let vms = doc["verificationMethod"].as_array().unwrap(); + assert_eq!(vms.len(), 2); + assert_eq!(vms[0]["type"], "Ed25519VerificationKey2020"); + assert_eq!(vms[1]["type"], "X25519KeyAgreementKey2020"); + assert_eq!(vms[0]["controller"], did); + + // Verify authentication references key-1 + let auth = doc["authentication"].as_array().unwrap(); + assert_eq!(auth[0], format!("{}#key-1", did)); + + // Verify assertionMethod + assert!(doc["assertionMethod"].as_array().is_some()); + + // Verify keyAgreement references x25519 key + let ka = doc["keyAgreement"].as_array().unwrap(); + assert_eq!(ka[0], format!("{}#key-x25519-1", did)); + } + + #[test] + fn test_did_document_from_pubkey_hex() { + let hex = "d75a980182b10ab7d54bfed3c964073a0ee172f3daa3f4a18446b7e21e7e2c33"; + let doc = did_document_from_pubkey_hex(hex).unwrap(); + assert_eq!(doc["@context"].as_array().unwrap().len(), 3); + assert!(doc["id"].as_str().unwrap().starts_with("did:key:z")); + } + + #[test] + fn test_pubkey_bytes_from_did_key_roundtrip() { + let hex = "d75a980182b10ab7d54bfed3c964073a0ee172f3daa3f4a18446b7e21e7e2c33"; + let did = did_key_from_pubkey_hex(hex).unwrap(); + let recovered = pubkey_bytes_from_did_key(&did).unwrap(); + assert_eq!(hex::encode(recovered), hex); + } + + #[test] + fn test_pubkey_bytes_from_invalid_did() { + assert!(pubkey_bytes_from_did_key("did:web:example.com").is_err()); + assert!(pubkey_bytes_from_did_key("did:key:invalid").is_err()); + } } diff --git a/core/archipelago/src/main.rs b/core/archipelago/src/main.rs index 2185204f..29030952 100644 --- a/core/archipelago/src/main.rs +++ b/core/archipelago/src/main.rs @@ -27,6 +27,7 @@ mod mesh; mod monitoring; mod node_message; mod nostr_discovery; +mod nostr_handshake; mod peers; mod server; mod session; diff --git a/core/archipelago/src/marketplace.rs b/core/archipelago/src/marketplace.rs new file mode 100644 index 00000000..2c3cfe53 --- /dev/null +++ b/core/archipelago/src/marketplace.rs @@ -0,0 +1,657 @@ +//! Decentralized app marketplace: discover, verify, and publish app manifests +//! via Nostr relays. Uses NIP-78 (kind 30078) with d-tag "archipelago-app:". +//! +//! See docs/marketplace-protocol.md for the full protocol specification. + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::path::Path; +use tokio::fs; +use tracing::{debug, info, warn}; + +const MARKETPLACE_DIR: &str = "marketplace"; +const CACHE_FILE: &str = "manifests.json"; +const PUBLISHED_DIR: &str = "published"; +const ARCHIPELAGO_KIND: u64 = 30078; +const D_TAG_PREFIX: &str = "archipelago-app:"; +const MARKETPLACE_TAG: &str = "archipelago-marketplace"; + +/// Categories for marketplace apps. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "lowercase")] +pub enum AppCategory { + Money, + Commerce, + Data, + Networking, + Home, + Community, + Other, +} + +impl std::fmt::Display for AppCategory { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Money => write!(f, "money"), + Self::Commerce => write!(f, "commerce"), + Self::Data => write!(f, "data"), + Self::Networking => write!(f, "networking"), + Self::Home => write!(f, "home"), + Self::Community => write!(f, "community"), + Self::Other => write!(f, "other"), + } + } +} + +/// Author information in a marketplace manifest. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ManifestAuthor { + pub name: String, + pub did: String, + #[serde(default)] + pub nostr_pubkey: String, +} + +/// Container configuration in a marketplace manifest. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ManifestContainer { + pub image: String, + #[serde(default)] + pub ports: Vec, + #[serde(default)] + pub volumes: Vec, + #[serde(default)] + pub env: HashMap, + #[serde(default)] + pub capabilities: Vec, + #[serde(default = "default_true")] + pub readonly_root: bool, + #[serde(default = "default_true")] + pub no_new_privileges: bool, + #[serde(default = "default_uid")] + pub run_as_user: u32, +} + +fn default_true() -> bool { + true +} + +fn default_uid() -> u32 { + 1000 +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct PortMapping { + pub container: u16, + pub host: u16, + #[serde(default = "default_tcp")] + pub protocol: String, +} + +fn default_tcp() -> String { + "tcp".into() +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VolumeMapping { + pub name: String, + pub path: String, +} + +/// App manifest signatures. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ManifestSignatures { + pub manifest_hash: String, + pub did_signature: String, +} + +/// A marketplace app manifest. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AppManifest { + pub app_id: String, + pub name: String, + pub version: String, + pub description: ManifestDescription, + pub author: ManifestAuthor, + pub container: ManifestContainer, + pub category: AppCategory, + #[serde(default)] + pub icon_url: String, + #[serde(default)] + pub repo_url: String, + #[serde(default)] + pub license: String, + #[serde(default)] + pub min_archipelago_version: String, + #[serde(default)] + pub dependencies: Vec, + #[serde(default)] + pub signatures: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(untagged)] +pub enum ManifestDescription { + Simple(String), + Detailed { short: String, long: String }, +} + +impl ManifestDescription { + pub fn short(&self) -> &str { + match self { + Self::Simple(s) => s, + Self::Detailed { short, .. } => short, + } + } +} + +/// A discovered marketplace app with trust scoring. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct DiscoveredApp { + pub manifest: AppManifest, + pub trust_score: u32, + pub trust_tier: String, + pub relay_count: u32, + pub first_seen: String, + pub nostr_pubkey: String, +} + +/// Cache of discovered marketplace apps. +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +pub struct MarketplaceCache { + pub apps: Vec, + pub last_updated: String, +} + +/// Ensure marketplace directories exist. +async fn ensure_dirs(data_dir: &Path) -> Result<()> { + let market_dir = data_dir.join(MARKETPLACE_DIR); + fs::create_dir_all(market_dir.join("cache")).await.context("Creating marketplace cache dir")?; + fs::create_dir_all(market_dir.join(PUBLISHED_DIR)).await.context("Creating marketplace published dir")?; + Ok(()) +} + +/// Load cached marketplace data. +pub async fn load_cache(data_dir: &Path) -> Result { + let path = data_dir.join(MARKETPLACE_DIR).join("cache").join(CACHE_FILE); + if !path.exists() { + return Ok(MarketplaceCache::default()); + } + let data = fs::read_to_string(&path).await.context("Reading marketplace cache")?; + serde_json::from_str(&data).context("Parsing marketplace cache") +} + +/// Save marketplace cache. +pub async fn save_cache(data_dir: &Path, cache: &MarketplaceCache) -> Result<()> { + ensure_dirs(data_dir).await?; + let path = data_dir.join(MARKETPLACE_DIR).join("cache").join(CACHE_FILE); + let data = serde_json::to_string_pretty(cache)?; + fs::write(&path, data).await.context("Writing marketplace cache")?; + Ok(()) +} + +/// Validate a manifest meets security requirements. +pub fn validate_manifest(manifest: &AppManifest) -> Vec { + let mut issues = Vec::new(); + + // Required fields + if manifest.app_id.is_empty() { + issues.push("Missing app_id".into()); + } + if manifest.name.is_empty() { + issues.push("Missing name".into()); + } + if manifest.version.is_empty() { + issues.push("Missing version".into()); + } + if manifest.container.image.is_empty() { + issues.push("Missing container image".into()); + } + + // Security checks + if manifest.container.image.ends_with(":latest") { + issues.push("Container image uses :latest tag (must pin specific version)".into()); + } + if !manifest.container.readonly_root { + issues.push("readonly_root is false (should be true)".into()); + } + if !manifest.container.no_new_privileges { + issues.push("no_new_privileges is false (should be true)".into()); + } + if manifest.container.run_as_user < 1000 { + issues.push(format!("run_as_user is {} (must be >= 1000)", manifest.container.run_as_user)); + } + + // app_id format + if !manifest.app_id.chars().all(|c| c.is_ascii_lowercase() || c == '-' || c.is_ascii_digit()) { + issues.push("app_id must be lowercase kebab-case".into()); + } + + issues +} + +/// Calculate trust score for a discovered app manifest. +pub fn calculate_trust_score( + manifest: &AppManifest, + relay_count: u32, + federated_dids: &[String], +) -> (u32, String) { + let mut score: u32 = 0; + + // DID verification (30 points) — has a valid DID in author + if !manifest.author.did.is_empty() && manifest.author.did.starts_with("did:") { + score += 30; + } + + // Relay consensus (20 points) — found on multiple relays + score += match relay_count { + 0..=1 => 5, + 2..=3 => 12, + _ => 20, + }; + + // Federation trust (20 points) — developer DID in federation + if federated_dids.contains(&manifest.author.did) { + score += 20; + } + + // Version history (15 points) — has a proper semver version + if manifest.version.split('.').count() == 3 { + score += 10; + } + if !manifest.repo_url.is_empty() { + score += 5; + } + + // Security compliance (15 points) + let issues = validate_manifest(manifest); + if issues.is_empty() { + score += 15; + } else if issues.len() <= 2 { + score += 5; + } + + let tier = match score { + 80..=100 => "verified", + 50..=79 => "community", + 20..=49 => "unverified", + _ => "untrusted", + }; + + (score, tier.to_string()) +} + +/// Discover app manifests from Nostr relays. +/// +/// Queries configured relays for kind 30078 events with the marketplace tag, +/// parses manifests, validates, scores, and returns sorted by trust score. +pub async fn discover( + data_dir: &Path, + relays: &[String], + tor_proxy: Option<&str>, + federated_dids: &[String], +) -> Result> { + if relays.is_empty() { + return Ok(Vec::new()); + } + + info!(relay_count = relays.len(), "Discovering marketplace apps from Nostr relays"); + + let anon_keys = nostr_sdk::prelude::Keys::generate(); + let client = build_nostr_client(anon_keys, tor_proxy)?; + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + let filter = nostr_sdk::prelude::Filter::new() + .kind(nostr_sdk::prelude::Kind::Custom(ARCHIPELAGO_KIND as u16)) + .hashtag(MARKETPLACE_TAG) + .limit(200); + + let events = client + .fetch_events(filter, std::time::Duration::from_secs(20)) + .await + .map(|e| e.to_vec()) + .unwrap_or_default(); + client.disconnect().await; + + debug!(event_count = events.len(), "Fetched marketplace events from relays"); + + // Deduplicate by app_id, keeping the latest version + let mut app_map: HashMap = HashMap::new(); + + for event in events { + // Parse manifest from event content + let manifest: AppManifest = match serde_json::from_str(&event.content) { + Ok(m) => m, + Err(e) => { + debug!(err = %e, "Skipping invalid marketplace manifest"); + continue; + } + }; + + // Validate + let issues = validate_manifest(&manifest); + if issues.iter().any(|i| i.contains("Missing app_id") || i.contains("Missing container image")) { + debug!(issues = ?issues, "Skipping manifest with critical issues"); + continue; + } + + let app_id = manifest.app_id.clone(); + let entry = app_map.entry(app_id).or_insert_with(|| { + let (trust_score, trust_tier) = calculate_trust_score(&manifest, 1, federated_dids); + ( + DiscoveredApp { + manifest, + trust_score, + trust_tier, + relay_count: 0, + first_seen: event.created_at.to_human_datetime(), + nostr_pubkey: event.pubkey.to_hex(), + }, + 0, + ) + }); + entry.1 += 1; + } + + // Update relay counts and recalculate scores + let mut apps: Vec = app_map + .into_values() + .map(|(mut app, relay_count)| { + app.relay_count = relay_count; + let (score, tier) = calculate_trust_score(&app.manifest, relay_count, federated_dids); + app.trust_score = score; + app.trust_tier = tier; + app + }) + .collect(); + + // Sort by trust score descending + apps.sort_by(|a, b| b.trust_score.cmp(&a.trust_score)); + + // Cache results + let cache = MarketplaceCache { + apps: apps.clone(), + last_updated: chrono::Utc::now().to_rfc3339(), + }; + if let Err(e) = save_cache(data_dir, &cache).await { + warn!(err = %e, "Failed to save marketplace cache"); + } + + info!(app_count = apps.len(), "Marketplace discovery complete"); + Ok(apps) +} + +/// Publish an app manifest to Nostr relays. +pub async fn publish( + data_dir: &Path, + manifest: &AppManifest, + relays: &[String], + tor_proxy: Option<&str>, +) -> Result { + if relays.is_empty() { + anyhow::bail!("No relays configured for publishing"); + } + + let issues = validate_manifest(manifest); + if !issues.is_empty() { + anyhow::bail!("Manifest validation failed: {}", issues.join(", ")); + } + + let identity_dir = data_dir.join("identity"); + let keys = load_or_create_keys(&identity_dir).await?; + let client = build_nostr_client(keys, tor_proxy)?; + + let content = serde_json::to_string(manifest).context("Serializing manifest")?; + let d_tag = format!("{}{}", D_TAG_PREFIX, manifest.app_id); + + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + let builder = nostr_sdk::prelude::EventBuilder::new( + nostr_sdk::prelude::Kind::Custom(ARCHIPELAGO_KIND as u16), + &content, + ) + .tag(nostr_sdk::prelude::Tag::identifier(&d_tag)) + .tag(nostr_sdk::prelude::Tag::hashtag(MARKETPLACE_TAG)) + .tag(nostr_sdk::prelude::Tag::hashtag(format!("category:{}", manifest.category))) + .tag(nostr_sdk::prelude::Tag::custom( + nostr_sdk::prelude::TagKind::custom("version"), + [&manifest.version], + )) + .tag(nostr_sdk::prelude::Tag::custom( + nostr_sdk::prelude::TagKind::custom("image"), + [&manifest.container.image], + )); + + let output = client.send_event_builder(builder).await?; + client.disconnect().await; + + // Save to published directory + ensure_dirs(data_dir).await?; + let pub_path = data_dir + .join(MARKETPLACE_DIR) + .join(PUBLISHED_DIR) + .join(format!("{}.json", manifest.app_id)); + let pub_data = serde_json::to_string_pretty(manifest)?; + fs::write(&pub_path, pub_data).await.context("Saving published manifest")?; + + info!(app_id = %manifest.app_id, "Published app manifest to {} relays", relays.len()); + Ok(output.id().to_hex()) +} + +/// List manifests published by this node. +pub async fn list_published(data_dir: &Path) -> Result> { + let pub_dir = data_dir.join(MARKETPLACE_DIR).join(PUBLISHED_DIR); + if !pub_dir.exists() { + return Ok(Vec::new()); + } + + let mut manifests = Vec::new(); + let mut entries = fs::read_dir(&pub_dir).await.context("Reading published dir")?; + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + if path.extension().map(|e| e == "json").unwrap_or(false) { + let data = fs::read_to_string(&path).await?; + if let Ok(manifest) = serde_json::from_str::(&data) { + manifests.push(manifest); + } + } + } + + Ok(manifests) +} + +// Re-use nostr client builder pattern from nostr_discovery +fn build_nostr_client( + keys: nostr_sdk::prelude::Keys, + tor_proxy: Option<&str>, +) -> Result { + use nostr_sdk::prelude::*; + let client = if let Some(proxy_str) = tor_proxy { + let addr: std::net::SocketAddr = proxy_str + .trim() + .parse() + .ok() + .ok_or_else(|| anyhow::anyhow!("Invalid Tor proxy: {}", proxy_str))?; + let connection = Connection::new() + .proxy(addr) + .target(ConnectionTarget::All); + let opts = ClientOptions::new().connection(connection); + Client::builder().signer(keys).opts(opts).build() + } else { + Client::new(keys) + }; + Ok(client) +} + +/// Load or create Nostr keys for marketplace publishing. +async fn load_or_create_keys(identity_dir: &Path) -> Result { + use nostr_sdk::prelude::Keys; + + let secret_path = identity_dir.join("nostr_secret"); + if secret_path.exists() { + let hex_secret = fs::read_to_string(&secret_path) + .await + .context("Reading Nostr secret")?; + Keys::parse(hex_secret.trim()).context("Invalid Nostr secret") + } else { + let keys = Keys::generate(); + fs::create_dir_all(identity_dir).await?; + fs::write(&secret_path, keys.secret_key().to_secret_hex()).await?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let sp = secret_path.clone(); + tokio::task::spawn_blocking(move || { + std::fs::set_permissions(sp, std::fs::Permissions::from_mode(0o600)) + }) + .await??; + } + Ok(keys) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn sample_manifest() -> AppManifest { + AppManifest { + app_id: "test-app".into(), + name: "Test App".into(), + version: "1.0.0".into(), + description: ManifestDescription::Detailed { + short: "A test app".into(), + long: "A longer description of the test app".into(), + }, + author: ManifestAuthor { + name: "Test Dev".into(), + did: "did:key:z6MkTest123".into(), + nostr_pubkey: String::new(), + }, + container: ManifestContainer { + image: "docker.io/test/app:1.0.0".into(), + ports: vec![PortMapping { + container: 8080, + host: 8180, + protocol: "tcp".into(), + }], + volumes: vec![], + env: HashMap::new(), + capabilities: vec![], + readonly_root: true, + no_new_privileges: true, + run_as_user: 1000, + }, + category: AppCategory::Other, + icon_url: String::new(), + repo_url: "https://github.com/test/app".into(), + license: "MIT".into(), + min_archipelago_version: "0.1.0".into(), + dependencies: vec![], + signatures: None, + } + } + + #[test] + fn test_validate_valid_manifest() { + let manifest = sample_manifest(); + let issues = validate_manifest(&manifest); + assert!(issues.is_empty(), "Expected no issues, got: {:?}", issues); + } + + #[test] + fn test_validate_latest_tag() { + let mut manifest = sample_manifest(); + manifest.container.image = "docker.io/test/app:latest".into(); + let issues = validate_manifest(&manifest); + assert!(issues.iter().any(|i| i.contains("latest"))); + } + + #[test] + fn test_validate_root_user() { + let mut manifest = sample_manifest(); + manifest.container.run_as_user = 0; + let issues = validate_manifest(&manifest); + assert!(issues.iter().any(|i| i.contains("run_as_user"))); + } + + #[test] + fn test_validate_missing_fields() { + let mut manifest = sample_manifest(); + manifest.app_id = String::new(); + manifest.container.image = String::new(); + let issues = validate_manifest(&manifest); + assert!(issues.len() >= 2); + } + + #[test] + fn test_trust_score_full() { + let manifest = sample_manifest(); + let (score, tier) = calculate_trust_score(&manifest, 3, &["did:key:z6MkTest123".to_string()]); + // DID (30) + relay consensus 2-3 (12) + federation (20) + semver (10) + repo (5) + security clean (15) = 92 + assert!(score >= 80, "Expected verified, got score={}", score); + assert_eq!(tier, "verified"); + } + + #[test] + fn test_trust_score_no_federation() { + let manifest = sample_manifest(); + let (score, tier) = calculate_trust_score(&manifest, 1, &[]); + // DID (30) + 1 relay (5) + no federation (0) + semver (10) + repo (5) + security (15) = 65 + assert_eq!(tier, "community"); + assert!(score >= 50 && score < 80); + } + + #[test] + fn test_trust_score_untrusted() { + let mut manifest = sample_manifest(); + manifest.author.did = String::new(); + manifest.repo_url = String::new(); + manifest.version = "1".into(); + manifest.container.readonly_root = false; + let (score, _tier) = calculate_trust_score(&manifest, 1, &[]); + assert!(score < 50, "Expected low score, got {}", score); + } + + #[test] + fn test_manifest_serialization() { + let manifest = sample_manifest(); + let json = serde_json::to_string(&manifest).unwrap(); + let parsed: AppManifest = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.app_id, "test-app"); + assert_eq!(parsed.description.short(), "A test app"); + } + + #[test] + fn test_category_display() { + assert_eq!(AppCategory::Money.to_string(), "money"); + assert_eq!(AppCategory::Networking.to_string(), "networking"); + } + + #[tokio::test] + async fn test_cache_persistence() { + let dir = tempfile::tempdir().unwrap(); + let cache = MarketplaceCache { + apps: vec![DiscoveredApp { + manifest: sample_manifest(), + trust_score: 75, + trust_tier: "community".into(), + relay_count: 2, + first_seen: "2026-03-10T00:00:00Z".into(), + nostr_pubkey: "abc123".into(), + }], + last_updated: "2026-03-10T00:00:00Z".into(), + }; + save_cache(dir.path(), &cache).await.unwrap(); + let loaded = load_cache(dir.path()).await.unwrap(); + assert_eq!(loaded.apps.len(), 1); + assert_eq!(loaded.apps[0].manifest.app_id, "test-app"); + } +} diff --git a/core/archipelago/src/mesh.rs b/core/archipelago/src/mesh.rs new file mode 100644 index 00000000..f3383bb3 --- /dev/null +++ b/core/archipelago/src/mesh.rs @@ -0,0 +1,265 @@ +//! Mesh networking: local node discovery over LoRa (Meshtastic) and BLE. +//! +//! Broadcasts node identity over mesh radio networks for offline peer discovery. +//! Uses Meshtastic serial protocol when a compatible radio is connected via USB. + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::path::Path; +use tokio::fs; + +const MESH_CONFIG_FILE: &str = "mesh-config.json"; + +/// A node discovered via mesh radio. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct MeshNode { + pub node_id: String, + pub did: Option, + pub pubkey: Option, + pub rssi: Option, + pub snr: Option, + pub last_heard: String, + #[serde(default)] + pub hops: u32, + #[serde(default)] + pub channel: Option, +} + +/// Mesh configuration. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct MeshConfig { + pub enabled: bool, + #[serde(default)] + pub device_path: Option, + #[serde(default)] + pub channel_name: Option, + #[serde(default)] + pub broadcast_identity: bool, +} + +impl Default for MeshConfig { + fn default() -> Self { + Self { + enabled: false, + device_path: None, + channel_name: Some("archipelago".to_string()), + broadcast_identity: true, + } + } +} + +pub async fn load_config(data_dir: &Path) -> Result { + let path = data_dir.join(MESH_CONFIG_FILE); + if !path.exists() { + return Ok(MeshConfig::default()); + } + let content = fs::read_to_string(&path) + .await + .context("Failed to read mesh config")?; + let config: MeshConfig = serde_json::from_str(&content).unwrap_or_default(); + Ok(config) +} + +pub async fn save_config(data_dir: &Path, config: &MeshConfig) -> Result<()> { + fs::create_dir_all(data_dir).await.context("Failed to create data dir")?; + let content = + serde_json::to_string_pretty(config).context("Failed to serialize mesh config")?; + fs::write(data_dir.join(MESH_CONFIG_FILE), content) + .await + .context("Failed to write mesh config")?; + Ok(()) +} + +/// Detect Meshtastic-compatible USB devices. +/// Meshtastic radios typically appear as USB serial devices (CP210x, CH340, FTDI). +pub async fn detect_meshtastic_devices() -> Vec { + let mut devices = Vec::new(); + + // Check for common serial device paths + let candidates = [ + "/dev/ttyUSB0", + "/dev/ttyUSB1", + "/dev/ttyACM0", + "/dev/ttyACM1", + ]; + + for path in &candidates { + if tokio::fs::metadata(path).await.is_ok() { + devices.push(path.to_string()); + } + } + + // Also scan sysfs for Meshtastic-specific USB VIDs + if let Ok(mut entries) = tokio::fs::read_dir("/sys/bus/usb/devices").await { + while let Ok(Some(entry)) = entries.next_entry().await { + let vid_path = entry.path().join("idVendor"); + if let Ok(vid_str) = tokio::fs::read_to_string(&vid_path).await { + let vid = vid_str.trim(); + // Silicon Labs CP210x (common Meshtastic radio) + // CH340 USB-serial + // FTDI FT232 + if vid == "10c4" || vid == "1a86" || vid == "0403" { + let product = tokio::fs::read_to_string(entry.path().join("product")) + .await + .map(|s| s.trim().to_string()) + .unwrap_or_else(|_| "Serial Device".to_string()); + devices.push(format!("{} ({})", entry.path().display(), product)); + } + } + } + } + + devices +} + +/// Discover nodes via Meshtastic CLI (meshtastic --nodes). +/// Returns nodes that have broadcast their Archipelago identity. +pub async fn discover_nodes(device_path: Option<&str>) -> Result> { + let mut cmd = tokio::process::Command::new("meshtastic"); + cmd.arg("--nodes"); + + if let Some(dev) = device_path { + cmd.arg("--port").arg(dev); + } + + let output = cmd + .output() + .await + .context("Failed to run meshtastic CLI — is it installed?")?; + + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr); + if stderr.contains("No Meshtastic") || stderr.contains("not found") { + return Ok(Vec::new()); + } + anyhow::bail!("meshtastic --nodes failed: {}", stderr); + } + + let stdout = String::from_utf8_lossy(&output.stdout); + let mut nodes = Vec::new(); + + // Parse the meshtastic CLI node list output + // Format varies but typically: NodeNum | User | AKA | ... + for line in stdout.lines().skip(2) { + // Skip header lines + let parts: Vec<&str> = line.split('|').map(|s| s.trim()).collect(); + if parts.len() < 3 { + continue; + } + + let node_id = parts.first().unwrap_or(&"").to_string(); + if node_id.is_empty() || node_id.starts_with('-') { + continue; + } + + nodes.push(MeshNode { + node_id: node_id.trim().to_string(), + did: None, + pubkey: None, + rssi: None, + snr: None, + last_heard: chrono::Utc::now().to_rfc3339(), + hops: 0, + channel: None, + }); + } + + Ok(nodes) +} + +/// Broadcast our node identity over mesh. +pub async fn broadcast_identity( + did: &str, + pubkey: &str, + device_path: Option<&str>, +) -> Result<()> { + let message = format!("ARCHY:{}:{}", did, pubkey); + + let mut cmd = tokio::process::Command::new("meshtastic"); + cmd.arg("--sendtext").arg(&message); + + if let Some(dev) = device_path { + cmd.arg("--port").arg(dev); + } + + let output = cmd + .output() + .await + .context("Failed to broadcast via meshtastic")?; + + if !output.status.success() { + anyhow::bail!( + "meshtastic broadcast failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_mesh_config_default() { + let config = MeshConfig::default(); + assert!(!config.enabled); + assert_eq!(config.channel_name, Some("archipelago".to_string())); + assert!(config.broadcast_identity); + } + + #[test] + fn test_mesh_config_serialization() { + let config = MeshConfig { + enabled: true, + device_path: Some("/dev/ttyUSB0".to_string()), + channel_name: Some("test".to_string()), + broadcast_identity: false, + }; + let json = serde_json::to_string(&config).unwrap(); + let parsed: MeshConfig = serde_json::from_str(&json).unwrap(); + assert!(parsed.enabled); + assert_eq!(parsed.device_path, Some("/dev/ttyUSB0".to_string())); + } + + #[test] + fn test_mesh_node_serialization() { + let node = MeshNode { + node_id: "!aabbccdd".to_string(), + did: Some("did:key:z123".to_string()), + pubkey: Some("pubhex".to_string()), + rssi: Some(-85), + snr: Some(7.5), + last_heard: "2026-03-10T00:00:00Z".to_string(), + hops: 1, + channel: Some("archipelago".to_string()), + }; + let json = serde_json::to_string(&node).unwrap(); + let parsed: MeshNode = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.node_id, "!aabbccdd"); + assert_eq!(parsed.rssi, Some(-85)); + } + + #[tokio::test] + async fn test_load_config_default_when_no_file() { + let dir = tempfile::tempdir().unwrap(); + let config = load_config(dir.path()).await.unwrap(); + assert!(!config.enabled); + } + + #[tokio::test] + async fn test_save_and_load_config_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + let config = MeshConfig { + enabled: true, + device_path: Some("/dev/ttyUSB0".to_string()), + channel_name: Some("archy".to_string()), + broadcast_identity: true, + }; + save_config(dir.path(), &config).await.unwrap(); + let loaded = load_config(dir.path()).await.unwrap(); + assert!(loaded.enabled); + assert_eq!(loaded.device_path, Some("/dev/ttyUSB0".to_string())); + } +} diff --git a/core/archipelago/src/names.rs b/core/archipelago/src/names.rs index e259c980..cb99f7a1 100644 --- a/core/archipelago/src/names.rs +++ b/core/archipelago/src/names.rs @@ -213,3 +213,172 @@ pub async fn link_name_to_did( save_names(data_dir, &store).await?; Ok(updated) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_name_status_display() { + assert_eq!(NameStatus::Active.to_string(), "active"); + assert_eq!(NameStatus::Pending.to_string(), "pending"); + assert_eq!(NameStatus::Expired.to_string(), "expired"); + assert_eq!(NameStatus::Failed.to_string(), "failed"); + } + + #[test] + fn test_name_status_serde_roundtrip() { + let json = serde_json::to_string(&NameStatus::Active).unwrap(); + assert_eq!(json, "\"active\""); + let deserialized: NameStatus = serde_json::from_str(&json).unwrap(); + assert_eq!(deserialized, NameStatus::Active); + } + + #[test] + fn test_names_store_default_is_empty() { + let store = NamesStore::default(); + assert!(store.names.is_empty()); + } + + #[tokio::test] + async fn test_load_names_returns_empty_when_no_file() { + let dir = tempfile::tempdir().unwrap(); + let store = load_names(dir.path()).await.unwrap(); + assert!(store.names.is_empty()); + } + + #[tokio::test] + async fn test_save_and_load_names_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + let store = NamesStore { + names: vec![RegisteredName { + id: "test-id-1".to_string(), + name: "satoshi".to_string(), + domain: "example.com".to_string(), + identity_id: "identity-1".to_string(), + did: "did:key:z123".to_string(), + nostr_pubkey: Some("npub1abc".to_string()), + status: NameStatus::Active, + registered_at: "2025-01-01T00:00:00Z".to_string(), + expires_at: None, + nip05: "satoshi@example.com".to_string(), + }], + }; + save_names(dir.path(), &store).await.unwrap(); + let loaded = load_names(dir.path()).await.unwrap(); + assert_eq!(loaded.names.len(), 1); + assert_eq!(loaded.names[0].name, "satoshi"); + assert_eq!(loaded.names[0].nip05, "satoshi@example.com"); + } + + #[tokio::test] + async fn test_register_name_creates_record() { + let dir = tempfile::tempdir().unwrap(); + let result = register_name( + dir.path(), + "alice", + "bitcoin.org", + "id-1", + "did:key:zabc", + Some("npub1xyz"), + ) + .await + .unwrap(); + + assert_eq!(result.name, "alice"); + assert_eq!(result.domain, "bitcoin.org"); + assert_eq!(result.nip05, "alice@bitcoin.org"); + assert_eq!(result.did, "did:key:zabc"); + assert_eq!(result.nostr_pubkey, Some("npub1xyz".to_string())); + assert_eq!(result.status, NameStatus::Active); + assert!(!result.id.is_empty()); + + // Verify persisted + let store = load_names(dir.path()).await.unwrap(); + assert_eq!(store.names.len(), 1); + } + + #[tokio::test] + async fn test_register_duplicate_name_fails() { + let dir = tempfile::tempdir().unwrap(); + register_name(dir.path(), "bob", "test.com", "id-1", "did:key:z1", None) + .await + .unwrap(); + + let result = + register_name(dir.path(), "bob", "test.com", "id-2", "did:key:z2", None).await; + assert!(result.is_err()); + let err_msg = result.unwrap_err().to_string(); + assert!(err_msg.contains("already registered")); + } + + #[tokio::test] + async fn test_register_same_name_different_domain_succeeds() { + let dir = tempfile::tempdir().unwrap(); + register_name(dir.path(), "alice", "domain1.com", "id-1", "did:key:z1", None) + .await + .unwrap(); + let result = + register_name(dir.path(), "alice", "domain2.com", "id-1", "did:key:z1", None).await; + assert!(result.is_ok()); + + let store = load_names(dir.path()).await.unwrap(); + assert_eq!(store.names.len(), 2); + } + + #[tokio::test] + async fn test_remove_name_by_id() { + let dir = tempfile::tempdir().unwrap(); + let registered = register_name( + dir.path(), + "charlie", + "example.com", + "id-1", + "did:key:z1", + None, + ) + .await + .unwrap(); + + remove_name(dir.path(), ®istered.id).await.unwrap(); + let store = load_names(dir.path()).await.unwrap(); + assert!(store.names.is_empty()); + } + + #[tokio::test] + async fn test_remove_nonexistent_name_fails() { + let dir = tempfile::tempdir().unwrap(); + let result = remove_name(dir.path(), "nonexistent-id").await; + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("Name not found")); + } + + #[tokio::test] + async fn test_link_name_to_did() { + let dir = tempfile::tempdir().unwrap(); + let registered = register_name( + dir.path(), + "dave", + "example.com", + "old-identity", + "did:key:old", + None, + ) + .await + .unwrap(); + + let updated = link_name_to_did( + dir.path(), + ®istered.id, + "did:key:new", + "new-identity", + ) + .await + .unwrap(); + + assert_eq!(updated.did, "did:key:new"); + assert_eq!(updated.identity_id, "new-identity"); + // Name itself should be unchanged + assert_eq!(updated.name, "dave"); + } +} diff --git a/core/archipelago/src/network/dns.rs b/core/archipelago/src/network/dns.rs new file mode 100644 index 00000000..11d36775 --- /dev/null +++ b/core/archipelago/src/network/dns.rs @@ -0,0 +1,382 @@ +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::path::Path; +use tokio::fs; +use tracing::{debug, info}; + +const DNS_CONFIG_FILE: &str = "dns_config.json"; + +/// DNS provider presets with server addresses. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum DnsProvider { + /// Use system default (DHCP-assigned DNS) + System, + /// Cloudflare DNS-over-HTTPS (1.1.1.1) + Cloudflare, + /// Google DNS-over-HTTPS (8.8.8.8) + Google, + /// Quad9 DNS-over-HTTPS (9.9.9.9) + Quad9, + /// Mullvad DNS (no logging) + Mullvad, + /// Custom user-specified servers + Custom, +} + +impl std::fmt::Display for DnsProvider { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::System => write!(f, "system"), + Self::Cloudflare => write!(f, "cloudflare"), + Self::Google => write!(f, "google"), + Self::Quad9 => write!(f, "quad9"), + Self::Mullvad => write!(f, "mullvad"), + Self::Custom => write!(f, "custom"), + } + } +} + +/// Persisted DNS configuration. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct DnsConfig { + pub provider: DnsProvider, + pub servers: Vec, + pub doh_enabled: bool, + pub doh_url: Option, +} + +impl Default for DnsConfig { + fn default() -> Self { + Self { + provider: DnsProvider::System, + servers: Vec::new(), + doh_enabled: false, + doh_url: None, + } + } +} + +/// Current DNS status read from the system. +#[derive(Debug, Serialize)] +pub struct DnsStatus { + pub provider: String, + pub servers: Vec, + pub doh_enabled: bool, + pub doh_url: Option, + pub resolv_conf_servers: Vec, +} + +/// Load persisted DNS config from disk. +pub async fn load_config(data_dir: &Path) -> Result { + let path = data_dir.join(DNS_CONFIG_FILE); + if !path.exists() { + return Ok(DnsConfig::default()); + } + let data = fs::read_to_string(&path) + .await + .context("Reading DNS config")?; + serde_json::from_str(&data).context("Parsing DNS config") +} + +/// Save DNS config to disk. +pub async fn save_config(data_dir: &Path, config: &DnsConfig) -> Result<()> { + let path = data_dir.join(DNS_CONFIG_FILE); + let data = serde_json::to_string_pretty(config)?; + fs::write(&path, data) + .await + .context("Writing DNS config")?; + Ok(()) +} + +/// Get the DNS servers for a given provider preset. +pub fn provider_servers(provider: &DnsProvider) -> (Vec, Option) { + match provider { + DnsProvider::System => (Vec::new(), None), + DnsProvider::Cloudflare => ( + vec!["1.1.1.1".into(), "1.0.0.1".into()], + Some("https://cloudflare-dns.com/dns-query".into()), + ), + DnsProvider::Google => ( + vec!["8.8.8.8".into(), "8.8.4.4".into()], + Some("https://dns.google/dns-query".into()), + ), + DnsProvider::Quad9 => ( + vec!["9.9.9.9".into(), "149.112.112.112".into()], + Some("https://dns.quad9.net/dns-query".into()), + ), + DnsProvider::Mullvad => ( + vec!["194.242.2.2".into()], + Some("https://dns.mullvad.net/dns-query".into()), + ), + DnsProvider::Custom => (Vec::new(), None), + } +} + +/// Read current DNS servers from /etc/resolv.conf. +pub async fn read_resolv_conf() -> Result> { + let content = fs::read_to_string("/etc/resolv.conf") + .await + .unwrap_or_default(); + let servers: Vec = content + .lines() + .filter_map(|line| { + let trimmed = line.trim(); + if trimmed.starts_with("nameserver") { + trimmed.split_whitespace().nth(1).map(String::from) + } else { + None + } + }) + .collect(); + Ok(servers) +} + +/// Get current DNS status combining config + system state. +pub async fn get_status(data_dir: &Path) -> Result { + let config = load_config(data_dir).await?; + let resolv_servers = read_resolv_conf().await.unwrap_or_default(); + + Ok(DnsStatus { + provider: config.provider.to_string(), + servers: if config.servers.is_empty() { + resolv_servers.clone() + } else { + config.servers.clone() + }, + doh_enabled: config.doh_enabled, + doh_url: config.doh_url.clone(), + resolv_conf_servers: resolv_servers, + }) +} + +/// Apply DNS configuration to the system via nmcli. +/// +/// Sets DNS servers on the active NetworkManager connection(s). +pub async fn apply_dns(config: &DnsConfig) -> Result<()> { + if config.provider == DnsProvider::System { + // Revert to DHCP-assigned DNS + info!("Reverting to system (DHCP) DNS"); + apply_dns_via_nmcli(&[]).await?; + return Ok(()); + } + + let servers = &config.servers; + if servers.is_empty() { + anyhow::bail!("No DNS servers specified"); + } + + // Validate all server IPs + for s in servers { + if s.parse::().is_err() { + anyhow::bail!("Invalid DNS server IP: {}", s); + } + } + + info!(provider = %config.provider, servers = ?servers, "Applying DNS configuration"); + apply_dns_via_nmcli(servers).await?; + + Ok(()) +} + +/// Apply DNS servers to all active NetworkManager connections. +async fn apply_dns_via_nmcli(servers: &[String]) -> Result<()> { + // Get active connections + let output = tokio::process::Command::new("nmcli") + .args(["-t", "-f", "NAME,DEVICE,TYPE", "connection", "show", "--active"]) + .output() + .await + .context("Failed to list nmcli connections")?; + + if !output.status.success() { + anyhow::bail!( + "nmcli connection show failed: {}", + String::from_utf8_lossy(&output.stderr) + ); + } + + let stdout = String::from_utf8(output.stdout).context("nmcli output not utf8")?; + let connections: Vec<&str> = stdout + .lines() + .filter_map(|line| { + let parts: Vec<&str> = line.splitn(3, ':').collect(); + if parts.len() >= 3 { + let conn_type = parts[2]; + // Only modify ethernet and wifi connections + if conn_type.contains("ethernet") || conn_type.contains("wireless") || conn_type.contains("wifi") { + return Some(parts[0]); + } + } + None + }) + .collect(); + + if connections.is_empty() { + debug!("No active ethernet/wifi connections found, skipping DNS apply"); + return Ok(()); + } + + let dns_value = if servers.is_empty() { + String::new() // Empty clears custom DNS, reverts to DHCP + } else { + servers.join(" ") + }; + + for conn_name in &connections { + // Set DNS servers + let dns_args = if dns_value.is_empty() { + vec![ + "connection".to_string(), + "modify".to_string(), + conn_name.to_string(), + "ipv4.dns".to_string(), + String::new(), + "ipv4.ignore-auto-dns".to_string(), + "no".to_string(), + ] + } else { + vec![ + "connection".to_string(), + "modify".to_string(), + conn_name.to_string(), + "ipv4.dns".to_string(), + dns_value.clone(), + "ipv4.ignore-auto-dns".to_string(), + "yes".to_string(), + ] + }; + + let modify = tokio::process::Command::new("nmcli") + .args(&dns_args) + .output() + .await + .context("Failed to modify DNS via nmcli")?; + + if !modify.status.success() { + let stderr = String::from_utf8_lossy(&modify.stderr); + tracing::warn!(conn = conn_name, err = %stderr, "Failed to set DNS on connection"); + continue; + } + + // Reapply the connection to pick up changes + let reapply = tokio::process::Command::new("nmcli") + .args(["connection", "up", conn_name]) + .output() + .await; + + match reapply { + Ok(out) if out.status.success() => { + info!(conn = conn_name, "DNS updated successfully"); + } + Ok(out) => { + let stderr = String::from_utf8_lossy(&out.stderr); + tracing::warn!(conn = conn_name, err = %stderr, "Failed to reapply connection"); + } + Err(e) => { + tracing::warn!(conn = conn_name, err = %e, "Failed to reapply connection"); + } + } + } + + Ok(()) +} + +/// Configure DNS with a specific provider. +pub async fn configure(data_dir: &Path, provider: DnsProvider, custom_servers: Vec) -> Result { + let (servers, doh_url) = if provider == DnsProvider::Custom { + (custom_servers, None) + } else { + let (preset_servers, preset_doh) = provider_servers(&provider); + (preset_servers, preset_doh) + }; + + let doh_enabled = doh_url.is_some(); + + let config = DnsConfig { + provider, + servers, + doh_enabled, + doh_url, + }; + + // Apply to system + apply_dns(&config).await?; + + // Persist config + save_config(data_dir, &config).await?; + + Ok(config) +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::tempdir; + + #[test] + fn test_default_config() { + let config = DnsConfig::default(); + assert_eq!(config.provider, DnsProvider::System); + assert!(config.servers.is_empty()); + assert!(!config.doh_enabled); + } + + #[test] + fn test_provider_servers() { + let (servers, doh) = provider_servers(&DnsProvider::Cloudflare); + assert_eq!(servers, vec!["1.1.1.1", "1.0.0.1"]); + assert!(doh.unwrap().contains("cloudflare")); + + let (servers, doh) = provider_servers(&DnsProvider::System); + assert!(servers.is_empty()); + assert!(doh.is_none()); + + let (servers, doh) = provider_servers(&DnsProvider::Custom); + assert!(servers.is_empty()); + assert!(doh.is_none()); + } + + #[test] + fn test_provider_display() { + assert_eq!(DnsProvider::Cloudflare.to_string(), "cloudflare"); + assert_eq!(DnsProvider::System.to_string(), "system"); + assert_eq!(DnsProvider::Quad9.to_string(), "quad9"); + } + + #[tokio::test] + async fn test_config_persistence() { + let dir = tempdir().unwrap(); + let config = DnsConfig { + provider: DnsProvider::Cloudflare, + servers: vec!["1.1.1.1".into(), "1.0.0.1".into()], + doh_enabled: true, + doh_url: Some("https://cloudflare-dns.com/dns-query".into()), + }; + save_config(dir.path(), &config).await.unwrap(); + let loaded = load_config(dir.path()).await.unwrap(); + assert_eq!(loaded.provider, DnsProvider::Cloudflare); + assert_eq!(loaded.servers.len(), 2); + assert!(loaded.doh_enabled); + } + + #[tokio::test] + async fn test_load_missing_config_returns_default() { + let dir = tempdir().unwrap(); + let config = load_config(dir.path()).await.unwrap(); + assert_eq!(config.provider, DnsProvider::System); + } + + #[test] + fn test_config_serialization() { + let config = DnsConfig { + provider: DnsProvider::Google, + servers: vec!["8.8.8.8".into()], + doh_enabled: true, + doh_url: Some("https://dns.google/dns-query".into()), + }; + let json = serde_json::to_string(&config).unwrap(); + let parsed: DnsConfig = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.provider, DnsProvider::Google); + assert_eq!(parsed.servers, vec!["8.8.8.8"]); + } +} diff --git a/core/archipelago/src/network/dwn_store.rs b/core/archipelago/src/network/dwn_store.rs new file mode 100644 index 00000000..b54f2831 --- /dev/null +++ b/core/archipelago/src/network/dwn_store.rs @@ -0,0 +1,459 @@ +//! DWN message store — persists DWN messages as JSON files on disk. +//! +//! Implements core CRUD operations, protocol registration, and query interface +//! for the Decentralized Web Node spec. + +use anyhow::{bail, Context, Result}; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use tokio::fs; +use tracing::debug; +use uuid::Uuid; + +/// A DWN message descriptor following the spec. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct MessageDescriptor { + pub interface: String, + pub method: String, + pub protocol: Option, + pub schema: Option, + #[serde(rename = "dateCreated")] + pub date_created: String, + #[serde(rename = "dateModified")] + pub date_modified: Option, + #[serde(rename = "dataFormat")] + pub data_format: Option, +} + +/// A stored DWN message. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct DwnMessage { + pub record_id: String, + pub descriptor: MessageDescriptor, + pub author: String, + pub data: Option, + #[serde(rename = "dateCreated")] + pub date_created: String, +} + +/// A registered protocol definition. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ProtocolDefinition { + pub protocol: String, + pub published: bool, + pub types: HashMap, + pub structure: HashMap, + #[serde(rename = "dateRegistered")] + pub date_registered: String, +} + +/// Query parameters for searching messages. +#[derive(Debug, Default)] +pub struct MessageQuery { + pub protocol: Option, + pub schema: Option, + pub author: Option, + pub date_from: Option, + pub date_to: Option, + pub limit: Option, +} + +/// The DWN message store backed by the filesystem. +pub struct DwnStore { + messages_dir: PathBuf, + protocols_dir: PathBuf, +} + +impl DwnStore { + /// Create a new DWN store at the given data directory. + pub async fn new(data_dir: &Path) -> Result { + let messages_dir = data_dir.join("dwn/messages"); + let protocols_dir = data_dir.join("dwn/protocols"); + fs::create_dir_all(&messages_dir) + .await + .context("Failed to create DWN messages dir")?; + fs::create_dir_all(&protocols_dir) + .await + .context("Failed to create DWN protocols dir")?; + Ok(Self { + messages_dir, + protocols_dir, + }) + } + + /// Write a new message or update an existing one. + pub async fn write_message( + &self, + author: &str, + protocol: Option<&str>, + schema: Option<&str>, + data_format: Option<&str>, + data: Option, + ) -> Result { + let now = chrono::Utc::now().to_rfc3339(); + let record_id = Uuid::new_v4().to_string(); + + let message = DwnMessage { + record_id: record_id.clone(), + descriptor: MessageDescriptor { + interface: "Records".to_string(), + method: "Write".to_string(), + protocol: protocol.map(|s| s.to_string()), + schema: schema.map(|s| s.to_string()), + date_created: now.clone(), + date_modified: Some(now.clone()), + data_format: data_format.map(|s| s.to_string()), + }, + author: author.to_string(), + data, + date_created: now, + }; + + let path = self.messages_dir.join(format!("{}.json", record_id)); + let content = + serde_json::to_string_pretty(&message).context("Failed to serialize message")?; + fs::write(&path, content) + .await + .context("Failed to write message file")?; + + debug!(record_id = %message.record_id, "DWN message written"); + Ok(message) + } + + /// Read a message by record ID. + pub async fn read_message(&self, record_id: &str) -> Result> { + let path = self.messages_dir.join(format!("{}.json", record_id)); + if !path.exists() { + return Ok(None); + } + let content = fs::read_to_string(&path) + .await + .context("Failed to read message file")?; + let message: DwnMessage = + serde_json::from_str(&content).context("Failed to parse message")?; + Ok(Some(message)) + } + + /// Delete a message by record ID. + pub async fn delete_message(&self, record_id: &str) -> Result { + let path = self.messages_dir.join(format!("{}.json", record_id)); + if !path.exists() { + return Ok(false); + } + fs::remove_file(&path) + .await + .context("Failed to delete message file")?; + debug!(record_id = %record_id, "DWN message deleted"); + Ok(true) + } + + /// Query messages by various criteria. + pub async fn query_messages(&self, query: &MessageQuery) -> Result> { + let mut results = Vec::new(); + let mut entries = fs::read_dir(&self.messages_dir) + .await + .context("Failed to read messages dir")?; + + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + if path.extension().and_then(|e| e.to_str()) != Some("json") { + continue; + } + + let content = match fs::read_to_string(&path).await { + Ok(c) => c, + Err(_) => continue, + }; + let message: DwnMessage = match serde_json::from_str(&content) { + Ok(m) => m, + Err(_) => continue, + }; + + if let Some(ref proto) = query.protocol { + if message.descriptor.protocol.as_deref() != Some(proto) { + continue; + } + } + if let Some(ref schema) = query.schema { + if message.descriptor.schema.as_deref() != Some(schema) { + continue; + } + } + if let Some(ref author) = query.author { + if &message.author != author { + continue; + } + } + if let Some(ref from) = query.date_from { + if message.date_created < *from { + continue; + } + } + if let Some(ref to) = query.date_to { + if message.date_created > *to { + continue; + } + } + + results.push(message); + } + + // Sort by date descending (newest first) + results.sort_by(|a, b| b.date_created.cmp(&a.date_created)); + + if let Some(limit) = query.limit { + results.truncate(limit); + } + + Ok(results) + } + + /// Register a protocol definition. + pub async fn register_protocol(&self, definition: &ProtocolDefinition) -> Result<()> { + if definition.protocol.is_empty() { + bail!("Protocol URI cannot be empty"); + } + let safe_name = definition.protocol.replace(['/', ':', '.'], "_"); + let path = self.protocols_dir.join(format!("{}.json", safe_name)); + let content = + serde_json::to_string_pretty(definition).context("Failed to serialize protocol")?; + fs::write(&path, content) + .await + .context("Failed to write protocol file")?; + debug!(protocol = %definition.protocol, "Protocol registered"); + Ok(()) + } + + /// List all registered protocols. + pub async fn list_protocols(&self) -> Result> { + let mut protocols = Vec::new(); + let mut entries = fs::read_dir(&self.protocols_dir) + .await + .context("Failed to read protocols dir")?; + + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + if path.extension().and_then(|e| e.to_str()) != Some("json") { + continue; + } + let content = match fs::read_to_string(&path).await { + Ok(c) => c, + Err(_) => continue, + }; + if let Ok(proto) = serde_json::from_str::(&content) { + protocols.push(proto); + } + } + + Ok(protocols) + } + + /// Remove a registered protocol. + pub async fn remove_protocol(&self, protocol_uri: &str) -> Result { + let safe_name = protocol_uri.replace(['/', ':', '.'], "_"); + let path = self.protocols_dir.join(format!("{}.json", safe_name)); + if !path.exists() { + return Ok(false); + } + fs::remove_file(&path) + .await + .context("Failed to remove protocol file")?; + debug!(protocol = %protocol_uri, "Protocol removed"); + Ok(true) + } + + /// Get storage statistics. + pub async fn stats(&self) -> Result { + let mut message_count: u64 = 0; + let mut total_bytes: u64 = 0; + + let mut entries = fs::read_dir(&self.messages_dir) + .await + .context("Failed to read messages dir")?; + + while let Some(entry) = entries.next_entry().await? { + if entry + .path() + .extension() + .and_then(|e| e.to_str()) + == Some("json") + { + message_count += 1; + if let Ok(meta) = entry.metadata().await { + total_bytes += meta.len(); + } + } + } + + let protocol_count = self.list_protocols().await?.len() as u64; + + Ok(StoreStats { + message_count, + protocol_count, + total_bytes, + }) + } +} + +/// Storage statistics. +#[derive(Debug, Serialize, Deserialize)] +pub struct StoreStats { + pub message_count: u64, + pub protocol_count: u64, + pub total_bytes: u64, +} + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + async fn setup() -> (TempDir, DwnStore) { + let dir = TempDir::new().unwrap(); + let store = DwnStore::new(dir.path()).await.unwrap(); + (dir, store) + } + + #[tokio::test] + async fn write_and_read_message() { + let (_dir, store) = setup().await; + let msg = store + .write_message("did:key:test", Some("proto://chat"), None, None, Some(serde_json::json!({"text": "hello"}))) + .await + .unwrap(); + assert!(!msg.record_id.is_empty()); + + let read = store.read_message(&msg.record_id).await.unwrap(); + assert!(read.is_some()); + let read = read.unwrap(); + assert_eq!(read.author, "did:key:test"); + assert_eq!(read.data, Some(serde_json::json!({"text": "hello"}))); + } + + #[tokio::test] + async fn read_nonexistent_returns_none() { + let (_dir, store) = setup().await; + let read = store.read_message("nonexistent-id").await.unwrap(); + assert!(read.is_none()); + } + + #[tokio::test] + async fn delete_message() { + let (_dir, store) = setup().await; + let msg = store + .write_message("did:key:test", None, None, None, None) + .await + .unwrap(); + assert!(store.delete_message(&msg.record_id).await.unwrap()); + assert!(!store.delete_message(&msg.record_id).await.unwrap()); + assert!(store.read_message(&msg.record_id).await.unwrap().is_none()); + } + + #[tokio::test] + async fn query_by_protocol() { + let (_dir, store) = setup().await; + store + .write_message("did:key:a", Some("proto://chat"), None, None, None) + .await + .unwrap(); + store + .write_message("did:key:a", Some("proto://files"), None, None, None) + .await + .unwrap(); + store + .write_message("did:key:b", Some("proto://chat"), None, None, None) + .await + .unwrap(); + + let results = store + .query_messages(&MessageQuery { + protocol: Some("proto://chat".to_string()), + ..Default::default() + }) + .await + .unwrap(); + assert_eq!(results.len(), 2); + } + + #[tokio::test] + async fn query_by_author() { + let (_dir, store) = setup().await; + store.write_message("did:key:a", None, None, None, None).await.unwrap(); + store.write_message("did:key:b", None, None, None, None).await.unwrap(); + + let results = store + .query_messages(&MessageQuery { + author: Some("did:key:a".to_string()), + ..Default::default() + }) + .await + .unwrap(); + assert_eq!(results.len(), 1); + assert_eq!(results[0].author, "did:key:a"); + } + + #[tokio::test] + async fn query_with_limit() { + let (_dir, store) = setup().await; + for i in 0..5 { + store + .write_message(&format!("did:key:{}", i), None, None, None, None) + .await + .unwrap(); + } + + let results = store + .query_messages(&MessageQuery { + limit: Some(3), + ..Default::default() + }) + .await + .unwrap(); + assert_eq!(results.len(), 3); + } + + #[tokio::test] + async fn register_and_list_protocols() { + let (_dir, store) = setup().await; + let proto = ProtocolDefinition { + protocol: "https://example.com/chat".to_string(), + published: true, + types: HashMap::new(), + structure: HashMap::new(), + date_registered: chrono::Utc::now().to_rfc3339(), + }; + store.register_protocol(&proto).await.unwrap(); + + let list = store.list_protocols().await.unwrap(); + assert_eq!(list.len(), 1); + assert_eq!(list[0].protocol, "https://example.com/chat"); + } + + #[tokio::test] + async fn remove_protocol() { + let (_dir, store) = setup().await; + let proto = ProtocolDefinition { + protocol: "https://example.com/test".to_string(), + published: false, + types: HashMap::new(), + structure: HashMap::new(), + date_registered: chrono::Utc::now().to_rfc3339(), + }; + store.register_protocol(&proto).await.unwrap(); + assert!(store.remove_protocol("https://example.com/test").await.unwrap()); + assert!(!store.remove_protocol("https://example.com/test").await.unwrap()); + assert!(store.list_protocols().await.unwrap().is_empty()); + } + + #[tokio::test] + async fn store_stats() { + let (_dir, store) = setup().await; + store.write_message("did:key:a", None, None, None, None).await.unwrap(); + store.write_message("did:key:b", None, None, None, None).await.unwrap(); + + let stats = store.stats().await.unwrap(); + assert_eq!(stats.message_count, 2); + assert!(stats.total_bytes > 0); + } +} diff --git a/core/archipelago/src/network/dwn_sync.rs b/core/archipelago/src/network/dwn_sync.rs index b36081a4..b8290d6b 100644 --- a/core/archipelago/src/network/dwn_sync.rs +++ b/core/archipelago/src/network/dwn_sync.rs @@ -98,9 +98,11 @@ pub struct DwnStatusResponse { } /// Trigger a sync with connected peers. -/// For each peer that has a DWN endpoint, we query their DWN -/// and replicate relevant messages. +/// For each peer that has a DWN endpoint, we pull their messages +/// and push our local messages, deduplicating by record_id. pub async fn sync_with_peers(data_dir: &Path, peer_onions: &[String]) -> Result { + use crate::network::dwn_store::{DwnStore, MessageQuery}; + let mut state = load_sync_state(data_dir).await?; state.status = SyncStatus::Syncing; save_sync_state(data_dir, &state).await?; @@ -114,21 +116,25 @@ pub async fn sync_with_peers(data_dir: &Path, peer_onions: &[String]) -> Result< .build() .context("Failed to build Tor HTTP client")?; + let store = DwnStore::new(data_dir).await?; let mut synced_count = 0u64; + // Get local messages since last sync (or all if first sync) + let local_messages = store + .query_messages(&MessageQuery { + date_from: state.last_sync.clone(), + ..Default::default() + }) + .await?; + for onion in peer_onions { - // Try to reach the peer's DWN endpoint - let url = format!("http://{}:3100/health", onion); - match client.get(&url).send().await { - Ok(res) if res.status().is_success() => { - debug!("Peer {} has DWN running, syncing...", onion); - synced_count += 1; - } - Ok(_) => { - debug!("Peer {} DWN not available", onion); + match sync_single_peer(&client, &store, onion, &local_messages, &state.last_sync).await { + Ok(count) => { + debug!(peer = %onion, messages = count, "Peer sync complete"); + synced_count += count; } Err(e) => { - debug!("Could not reach peer {} DWN: {}", onion, e); + debug!(peer = %onion, error = %e, "Peer sync failed"); } } } @@ -138,10 +144,108 @@ pub async fn sync_with_peers(data_dir: &Path, peer_onions: &[String]) -> Result< state.messages_synced += synced_count; save_sync_state(data_dir, &state).await?; - debug!("DWN sync complete: {} peers synced", synced_count); + debug!(count = synced_count, "DWN sync complete"); Ok(state) } +/// Sync with a single peer: pull their messages and push ours. +async fn sync_single_peer( + client: &reqwest::Client, + store: &crate::network::dwn_store::DwnStore, + onion: &str, + local_messages: &[crate::network::dwn_store::DwnMessage], + last_sync: &Option, +) -> Result { + let base_url = format!("http://{}:5678", onion); + let mut imported = 0u64; + + // Step 1: Check peer health + let health_url = format!("{}/dwn/health", base_url); + let res = client + .get(&health_url) + .send() + .await + .context("Peer DWN unreachable")?; + if !res.status().is_success() { + return Err(anyhow::anyhow!("Peer DWN not healthy")); + } + + // Step 2: Pull — query peer for messages since our last sync + let dwn_url = format!("{}/dwn", base_url); + let mut query_filter = serde_json::json!({}); + if let Some(ref since) = last_sync { + query_filter = serde_json::json!({ "dateSort": "createdAscending", "dateFrom": since }); + } + let pull_body = serde_json::json!({ + "messages": [{ + "descriptor": { + "interface": "Records", + "method": "Query", + "filter": query_filter, + } + }] + }); + + let pull_res = client + .post(&dwn_url) + .json(&pull_body) + .send() + .await + .context("Failed to query peer DWN")?; + + if pull_res.status().is_success() { + let pull_data: serde_json::Value = pull_res.json().await.unwrap_or_default(); + if let Some(entries) = pull_data["entries"].as_array() { + for entry in entries { + let record_id = entry["record_id"].as_str().unwrap_or_default(); + if record_id.is_empty() { + continue; + } + // Skip if we already have this message + if store.read_message(record_id).await?.is_some() { + continue; + } + // Import the message + let author = entry["author"].as_str().unwrap_or("unknown"); + let protocol = entry["descriptor"]["protocol"].as_str(); + let schema = entry["descriptor"]["schema"].as_str(); + let data_format = entry["descriptor"]["dataFormat"].as_str(); + let data = entry.get("data").cloned(); + + store + .write_message(author, protocol, schema, data_format, data) + .await?; + imported += 1; + } + } + } + + // Step 3: Push — send our local messages to the peer + for msg in local_messages { + let push_body = serde_json::json!({ + "messages": [{ + "descriptor": { + "interface": "Records", + "method": "Write", + "protocol": msg.descriptor.protocol, + "schema": msg.descriptor.schema, + "dataFormat": msg.descriptor.data_format, + }, + "recordId": msg.record_id, + "author": msg.author, + "data": msg.data, + }] + }); + + // Best-effort push — don't fail the whole sync if one push fails + if let Err(e) = client.post(&dwn_url).json(&push_body).send().await { + debug!(record_id = %msg.record_id, error = %e, "Failed to push message to peer"); + } + } + + Ok(imported) +} + /// Add a peer as a sync target. pub async fn add_sync_target(data_dir: &Path, onion: &str) -> Result<()> { let mut state = load_sync_state(data_dir).await?; diff --git a/core/archipelago/src/network/mod.rs b/core/archipelago/src/network/mod.rs index 2e42dff6..2a996ce2 100644 --- a/core/archipelago/src/network/mod.rs +++ b/core/archipelago/src/network/mod.rs @@ -1,2 +1,4 @@ +pub mod dns; +pub mod dwn_store; pub mod dwn_sync; pub mod router; diff --git a/core/archipelago/src/network/router.rs b/core/archipelago/src/network/router.rs index 883c9682..d413c097 100644 --- a/core/archipelago/src/network/router.rs +++ b/core/archipelago/src/network/router.rs @@ -5,7 +5,7 @@ use anyhow::{Context, Result}; use serde::{Deserialize, Serialize}; use std::path::Path; use tokio::fs; -use tracing::{debug, info, warn}; +use tracing::debug; const FORWARDS_FILE: &str = "port_forwards.json"; diff --git a/core/archipelago/src/nostr_handshake.rs b/core/archipelago/src/nostr_handshake.rs new file mode 100644 index 00000000..96b002fd --- /dev/null +++ b/core/archipelago/src/nostr_handshake.rs @@ -0,0 +1,386 @@ +//! Encrypted peer handshake via Nostr NIP-44. +//! +//! Instead of publishing onion addresses publicly on relays, nodes exchange +//! them privately via NIP-44 encrypted DMs: +//! +//! 1. Node publishes presence-only event (DID + Nostr pubkey, NO onion address) +//! 2. To connect, Node A sends NIP-44 encrypted DM to Node B's Nostr pubkey +//! containing A's onion address + Ed25519 node pubkey +//! 3. Node B auto-responds with its own onion address + pubkey +//! 4. Both nodes add each other as known peers +//! +//! Uses NIP-44 (ChaCha20-Poly1305) for encryption, kind 4 for DMs. + +use anyhow::{Context, Result}; +use nostr_sdk::prelude::*; +use serde::{Deserialize, Serialize}; +use std::net::SocketAddr; +use std::path::Path; +use tokio::fs; + +const NOSTR_SECRET_FILE: &str = "nostr_secret"; + +/// Message types for the encrypted handshake protocol +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "type")] +pub enum HandshakeMessage { + #[serde(rename = "connect-request")] + ConnectRequest { + onion: String, + node_pubkey: String, + did: String, + version: String, + name: Option, + }, + #[serde(rename = "connect-response")] + ConnectResponse { + onion: String, + node_pubkey: String, + did: String, + version: String, + name: Option, + }, +} + +/// Result of polling for incoming handshake messages +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct IncomingHandshake { + pub from_nostr_pubkey: String, + pub message: HandshakeMessage, + pub timestamp: String, +} + +/// Parse "host:port" to SocketAddr. +fn parse_proxy_addr(s: &str) -> Option { + s.trim().parse().ok() +} + +/// Load existing Nostr keys (secp256k1). Returns None if no keys exist. +async fn load_nostr_keys(identity_dir: &Path) -> Result> { + let secret_path = identity_dir.join(NOSTR_SECRET_FILE); + if !secret_path.exists() { + return Ok(None); + } + let hex_secret = fs::read_to_string(&secret_path) + .await + .context("Failed to read Nostr secret")?; + let keys = Keys::parse(hex_secret.trim()).context("Invalid Nostr secret")?; + Ok(Some(keys)) +} + +/// Build a Nostr client with optional Tor proxy. +fn build_client(keys: Keys, tor_proxy: Option<&str>) -> Result { + let client = if let Some(proxy_str) = tor_proxy { + let addr = parse_proxy_addr(proxy_str) + .ok_or_else(|| anyhow::anyhow!("Invalid Nostr Tor proxy: {}", proxy_str))?; + let connection = Connection::new() + .proxy(addr) + .target(ConnectionTarget::All); + let opts = ClientOptions::new().connection(connection); + Client::builder().signer(keys).opts(opts).build() + } else { + Client::new(keys) + }; + Ok(client) +} + +/// Publish a presence-only event to Nostr relays. +/// Content: { did, nostr_pubkey, version } — NO onion address. +/// Uses NIP-33 replaceable events (kind 30078) with d-tag "archipelago-node". +pub async fn publish_presence( + identity_dir: &Path, + did: &str, + version: &str, + relays: &[String], + tor_proxy: Option<&str>, +) -> Result<()> { + if relays.is_empty() { + anyhow::bail!("No relays configured"); + } + + let keys = load_nostr_keys(identity_dir) + .await? + .ok_or_else(|| anyhow::anyhow!("No Nostr keys — generate them first"))?; + + let nostr_pubkey = keys.public_key().to_hex(); + let client = build_client(keys, tor_proxy)?; + + let content = serde_json::json!({ + "did": did, + "nostr_pubkey": nostr_pubkey, + "version": version, + // No onion address — exchanged only via encrypted DM + }) + .to_string(); + + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + let builder = EventBuilder::new(Kind::Custom(30078), content) + .tag(Tag::identifier("archipelago-node")); + let _ = client.send_event_builder(builder).await; + client.disconnect().await; + + tracing::info!("📡 Published presence (no onion) to {} relays", relays.len()); + Ok(()) +} + +/// Discover other Archipelago nodes (presence-only — no onion addresses). +/// Returns Nostr pubkeys and DIDs of discoverable nodes. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct DiscoverableNode { + pub nostr_pubkey: String, + pub did: String, + pub version: String, +} + +pub async fn discover_nodes( + identity_dir: &Path, + relays: &[String], + tor_proxy: Option<&str>, +) -> Result> { + if relays.is_empty() { + return Ok(Vec::new()); + } + + let anon_keys = Keys::generate(); + let client = build_client(anon_keys, tor_proxy)?; + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + let filter = Filter::new() + .kind(Kind::Custom(30078)) + .identifier("archipelago-node") + .limit(50); + let events = client + .fetch_events(filter, std::time::Duration::from_secs(15)) + .await + .map(|e| e.to_vec()) + .unwrap_or_default(); + client.disconnect().await; + + let mut nodes = Vec::new(); + for event in events { + if let Ok(content) = serde_json::from_str::(&event.content) { + let nostr_pubkey = content + .get("nostr_pubkey") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let did = content + .get("did") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let version = content + .get("version") + .and_then(|v| v.as_str()) + .unwrap_or("0.1") + .to_string(); + + // Skip entries that still have node_address (legacy public format) + if content.get("node_address").is_some() { + continue; + } + + if !nostr_pubkey.is_empty() { + nodes.push(DiscoverableNode { + nostr_pubkey, + did, + version, + }); + } + } + } + Ok(nodes) +} + +/// Send an encrypted connection request to a peer's Nostr pubkey. +/// Uses NIP-44 encrypted DM (kind 4) containing our onion address. +pub async fn send_connect_request( + identity_dir: &Path, + recipient_nostr_pubkey: &str, + our_onion: &str, + our_node_pubkey: &str, + our_did: &str, + our_version: &str, + our_name: Option<&str>, + relays: &[String], + tor_proxy: Option<&str>, +) -> Result<()> { + if relays.is_empty() { + anyhow::bail!("No relays configured"); + } + + let keys = load_nostr_keys(identity_dir) + .await? + .ok_or_else(|| anyhow::anyhow!("No Nostr keys"))?; + + let recipient_pk = PublicKey::from_hex(recipient_nostr_pubkey) + .context("Invalid recipient Nostr pubkey")?; + + let msg = HandshakeMessage::ConnectRequest { + onion: our_onion.to_string(), + node_pubkey: our_node_pubkey.to_string(), + did: our_did.to_string(), + version: our_version.to_string(), + name: our_name.map(String::from), + }; + let plaintext = serde_json::to_string(&msg).context("Failed to serialize handshake")?; + + // NIP-44 encrypt + let encrypted = nip44::encrypt( + keys.secret_key(), + &recipient_pk, + &plaintext, + nip44::Version::V2, + ) + .map_err(|e| anyhow::anyhow!("NIP-44 encrypt failed: {}", e))?; + + let client = build_client(keys, tor_proxy)?; + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + // Kind 4 encrypted DM with p-tag for recipient + let builder = EventBuilder::new(Kind::EncryptedDirectMessage, encrypted) + .tag(Tag::public_key(recipient_pk)); + let _ = client.send_event_builder(builder).await; + client.disconnect().await; + + tracing::info!( + "🤝 Sent encrypted connect request to {}...{}", + &recipient_nostr_pubkey[..8.min(recipient_nostr_pubkey.len())], + &recipient_nostr_pubkey[recipient_nostr_pubkey.len().saturating_sub(4)..] + ); + Ok(()) +} + +/// Send an encrypted connection response to a peer. +pub async fn send_connect_response( + identity_dir: &Path, + recipient_nostr_pubkey: &str, + our_onion: &str, + our_node_pubkey: &str, + our_did: &str, + our_version: &str, + our_name: Option<&str>, + relays: &[String], + tor_proxy: Option<&str>, +) -> Result<()> { + if relays.is_empty() { + anyhow::bail!("No relays configured"); + } + + let keys = load_nostr_keys(identity_dir) + .await? + .ok_or_else(|| anyhow::anyhow!("No Nostr keys"))?; + + let recipient_pk = PublicKey::from_hex(recipient_nostr_pubkey) + .context("Invalid recipient Nostr pubkey")?; + + let msg = HandshakeMessage::ConnectResponse { + onion: our_onion.to_string(), + node_pubkey: our_node_pubkey.to_string(), + did: our_did.to_string(), + version: our_version.to_string(), + name: our_name.map(String::from), + }; + let plaintext = serde_json::to_string(&msg).context("Failed to serialize handshake")?; + + let encrypted = nip44::encrypt( + keys.secret_key(), + &recipient_pk, + &plaintext, + nip44::Version::V2, + ) + .map_err(|e| anyhow::anyhow!("NIP-44 encrypt failed: {}", e))?; + + let client = build_client(keys, tor_proxy)?; + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + let builder = EventBuilder::new(Kind::EncryptedDirectMessage, encrypted) + .tag(Tag::public_key(recipient_pk)); + let _ = client.send_event_builder(builder).await; + client.disconnect().await; + + tracing::info!( + "🤝 Sent encrypted connect response to {}...{}", + &recipient_nostr_pubkey[..8.min(recipient_nostr_pubkey.len())], + &recipient_nostr_pubkey[recipient_nostr_pubkey.len().saturating_sub(4)..] + ); + Ok(()) +} + +/// Poll relays for incoming encrypted handshake DMs addressed to us. +/// Returns new handshake messages since `since` timestamp. +pub async fn poll_handshakes( + identity_dir: &Path, + relays: &[String], + tor_proxy: Option<&str>, + since: Option, +) -> Result> { + if relays.is_empty() { + return Ok(Vec::new()); + } + + let keys = load_nostr_keys(identity_dir) + .await? + .ok_or_else(|| anyhow::anyhow!("No Nostr keys"))?; + + let our_pk = keys.public_key(); + let client = build_client(keys.clone(), tor_proxy)?; + for url in relays { + let _ = client.add_relay(url).await; + } + client.connect().await; + + // Query for encrypted DMs addressed to us + let mut filter = Filter::new() + .kind(Kind::EncryptedDirectMessage) + .pubkey(our_pk) + .limit(50); + if let Some(ts) = since { + filter = filter.since(ts); + } + + let events = client + .fetch_events(filter, std::time::Duration::from_secs(15)) + .await + .map(|e| e.to_vec()) + .unwrap_or_default(); + client.disconnect().await; + + let mut handshakes = Vec::new(); + for event in events { + // Skip our own events + if event.pubkey == our_pk { + continue; + } + + // Try NIP-44 decryption + let plaintext = match nip44::decrypt(keys.secret_key(), &event.pubkey, &event.content) { + Ok(pt) => pt, + Err(_) => continue, // Not a handshake message or wrong key + }; + + // Try to parse as HandshakeMessage + if let Ok(msg) = serde_json::from_str::(&plaintext) { + handshakes.push(IncomingHandshake { + from_nostr_pubkey: event.pubkey.to_hex(), + message: msg, + timestamp: event.created_at.to_human_datetime(), + }); + } + } + + Ok(handshakes) +} diff --git a/core/archipelago/src/nostr_relays.rs b/core/archipelago/src/nostr_relays.rs index a4f657cd..c4e2cf58 100644 --- a/core/archipelago/src/nostr_relays.rs +++ b/core/archipelago/src/nostr_relays.rs @@ -170,3 +170,257 @@ fn normalize_relay_url(url: &str) -> Result { Ok(format!("wss://{}", trimmed)) } } + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + #[test] + fn test_normalize_relay_url_with_wss() { + let result = normalize_relay_url("wss://relay.damus.io").unwrap(); + assert_eq!(result, "wss://relay.damus.io"); + } + + #[test] + fn test_normalize_relay_url_with_ws() { + let result = normalize_relay_url("ws://relay.example.com").unwrap(); + assert_eq!(result, "ws://relay.example.com"); + } + + #[test] + fn test_normalize_relay_url_without_scheme() { + let result = normalize_relay_url("relay.example.com").unwrap(); + assert_eq!(result, "wss://relay.example.com"); + } + + #[test] + fn test_normalize_relay_url_trims_whitespace() { + let result = normalize_relay_url(" wss://relay.example.com ").unwrap(); + assert_eq!(result, "wss://relay.example.com"); + } + + #[test] + fn test_normalize_relay_url_empty_errors() { + let result = normalize_relay_url(""); + assert!(result.is_err()); + + let result = normalize_relay_url(" "); + assert!(result.is_err()); + } + + #[test] + fn test_seed_defaults_has_expected_count() { + let store = seed_defaults(); + assert_eq!(store.relays.len(), DEFAULT_RELAYS.len()); + } + + #[test] + fn test_seed_defaults_all_enabled() { + let store = seed_defaults(); + assert!(store.relays.iter().all(|r| r.enabled)); + } + + #[test] + fn test_seed_defaults_urls_match() { + let store = seed_defaults(); + let urls: Vec<&str> = store.relays.iter().map(|r| r.url.as_str()).collect(); + for expected in DEFAULT_RELAYS { + assert!(urls.contains(expected), "Missing default relay: {}", expected); + } + } + + #[test] + fn test_seed_defaults_all_have_timestamps() { + let store = seed_defaults(); + for relay in &store.relays { + assert!(!relay.added_at.is_empty()); + // Should be a valid RFC3339 timestamp + assert!(chrono::DateTime::parse_from_rfc3339(&relay.added_at).is_ok()); + } + } + + #[tokio::test] + async fn test_load_relays_seeds_defaults_on_first_load() { + let tmp = TempDir::new().unwrap(); + let store = load_relays(tmp.path()).await.unwrap(); + assert_eq!(store.relays.len(), DEFAULT_RELAYS.len()); + + // The file should now exist after seeding + assert!(tmp.path().join(RELAYS_FILE).exists()); + } + + #[tokio::test] + async fn test_save_and_load_relays_roundtrip() { + let tmp = TempDir::new().unwrap(); + let store = RelayStore { + relays: vec![ + RelayConfig { + url: "wss://test.relay.one".to_string(), + enabled: true, + added_at: "2025-01-01T00:00:00Z".to_string(), + }, + RelayConfig { + url: "wss://test.relay.two".to_string(), + enabled: false, + added_at: "2025-01-02T00:00:00Z".to_string(), + }, + ], + }; + + save_relays(tmp.path(), &store).await.unwrap(); + let loaded = load_relays(tmp.path()).await.unwrap(); + assert_eq!(loaded.relays.len(), 2); + assert_eq!(loaded.relays[0].url, "wss://test.relay.one"); + assert!(loaded.relays[0].enabled); + assert_eq!(loaded.relays[1].url, "wss://test.relay.two"); + assert!(!loaded.relays[1].enabled); + } + + #[tokio::test] + async fn test_add_relay() { + let tmp = TempDir::new().unwrap(); + // Seed defaults first + let _ = load_relays(tmp.path()).await.unwrap(); + + let config = add_relay(tmp.path(), "wss://new.relay.test").await.unwrap(); + assert_eq!(config.url, "wss://new.relay.test"); + assert!(config.enabled); + + let store = load_relays(tmp.path()).await.unwrap(); + assert_eq!(store.relays.len(), DEFAULT_RELAYS.len() + 1); + } + + #[tokio::test] + async fn test_add_relay_normalizes_url() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + let config = add_relay(tmp.path(), "bare.relay.test").await.unwrap(); + assert_eq!(config.url, "wss://bare.relay.test"); + } + + #[tokio::test] + async fn test_add_duplicate_relay_errors() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + // Adding a default relay again should fail + let result = add_relay(tmp.path(), "wss://relay.damus.io").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_remove_relay() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + let initial_count = DEFAULT_RELAYS.len(); + + remove_relay(tmp.path(), "wss://relay.damus.io").await.unwrap(); + let store = load_relays(tmp.path()).await.unwrap(); + assert_eq!(store.relays.len(), initial_count - 1); + assert!(!store.relays.iter().any(|r| r.url == "wss://relay.damus.io")); + } + + #[tokio::test] + async fn test_remove_nonexistent_relay_errors() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + let result = remove_relay(tmp.path(), "wss://nonexistent.relay").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_toggle_relay_disable() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + toggle_relay(tmp.path(), "wss://relay.damus.io", false).await.unwrap(); + + let store = load_relays(tmp.path()).await.unwrap(); + let relay = store.relays.iter().find(|r| r.url == "wss://relay.damus.io").unwrap(); + assert!(!relay.enabled); + } + + #[tokio::test] + async fn test_toggle_relay_enable() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + // Disable first, then re-enable + toggle_relay(tmp.path(), "wss://relay.damus.io", false).await.unwrap(); + toggle_relay(tmp.path(), "wss://relay.damus.io", true).await.unwrap(); + + let store = load_relays(tmp.path()).await.unwrap(); + let relay = store.relays.iter().find(|r| r.url == "wss://relay.damus.io").unwrap(); + assert!(relay.enabled); + } + + #[tokio::test] + async fn test_toggle_nonexistent_relay_errors() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + let result = toggle_relay(tmp.path(), "wss://no.such.relay", true).await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_get_stats() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + let stats = get_stats(tmp.path()).await.unwrap(); + assert_eq!(stats.total_relays, DEFAULT_RELAYS.len()); + assert_eq!(stats.enabled_count, DEFAULT_RELAYS.len()); + assert_eq!(stats.connected_count, stats.enabled_count); + } + + #[tokio::test] + async fn test_get_stats_after_disable() { + let tmp = TempDir::new().unwrap(); + let _ = load_relays(tmp.path()).await.unwrap(); + + toggle_relay(tmp.path(), "wss://relay.damus.io", false).await.unwrap(); + toggle_relay(tmp.path(), "wss://nos.lol", false).await.unwrap(); + + let stats = get_stats(tmp.path()).await.unwrap(); + assert_eq!(stats.total_relays, DEFAULT_RELAYS.len()); + assert_eq!(stats.enabled_count, DEFAULT_RELAYS.len() - 2); + } + + #[tokio::test] + async fn test_list_relays() { + let tmp = TempDir::new().unwrap(); + let statuses = list_relays(tmp.path()).await.unwrap(); + assert_eq!(statuses.len(), DEFAULT_RELAYS.len()); + + // All default relays should be enabled and "connected" + for status in &statuses { + assert!(status.enabled); + assert!(status.connected); + assert!(status.url.starts_with("wss://")); + } + } + + #[test] + fn test_relay_store_default_is_empty() { + let store = RelayStore::default(); + assert!(store.relays.is_empty()); + } + + #[test] + fn test_relay_config_serialization() { + let config = RelayConfig { + url: "wss://test.relay".to_string(), + enabled: true, + added_at: "2025-01-01T00:00:00Z".to_string(), + }; + let json = serde_json::to_string(&config).unwrap(); + let parsed: RelayConfig = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.url, config.url); + assert_eq!(parsed.enabled, config.enabled); + assert_eq!(parsed.added_at, config.added_at); + } +} diff --git a/core/archipelago/src/peers.rs b/core/archipelago/src/peers.rs index 577c1385..d16541d7 100644 --- a/core/archipelago/src/peers.rs +++ b/core/archipelago/src/peers.rs @@ -61,3 +61,119 @@ pub async fn remove_peer(data_dir: &Path, pubkey: &str) -> Result save_peers(data_dir, &peers).await?; Ok(peers) } + +#[cfg(test)] +mod tests { + use super::*; + + fn make_peer(pubkey: &str, onion: &str) -> KnownPeer { + KnownPeer { + onion: onion.to_string(), + pubkey: pubkey.to_string(), + name: None, + added_at: None, + } + } + + #[test] + fn test_peers_file_default_is_empty() { + let pf = PeersFile::default(); + assert!(pf.peers.is_empty()); + } + + #[test] + fn test_known_peer_serialization_roundtrip() { + let peer = KnownPeer { + onion: "abc123.onion".to_string(), + pubkey: "02aabbcc".to_string(), + name: Some("My Node".to_string()), + added_at: Some("2025-01-01T00:00:00Z".to_string()), + }; + let json = serde_json::to_string(&peer).unwrap(); + let parsed: KnownPeer = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.onion, "abc123.onion"); + assert_eq!(parsed.pubkey, "02aabbcc"); + assert_eq!(parsed.name, Some("My Node".to_string())); + } + + #[test] + fn test_known_peer_optional_fields_default() { + // name and added_at should default to None when missing + let json = r#"{"onion": "test.onion", "pubkey": "deadbeef"}"#; + let peer: KnownPeer = serde_json::from_str(json).unwrap(); + assert!(peer.name.is_none()); + assert!(peer.added_at.is_none()); + } + + #[tokio::test] + async fn test_load_peers_returns_empty_when_no_file() { + let dir = tempfile::tempdir().unwrap(); + let peers = load_peers(dir.path()).await.unwrap(); + assert!(peers.is_empty()); + } + + #[tokio::test] + async fn test_save_and_load_peers_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + let peers = vec![ + make_peer("pub1", "onion1.onion"), + make_peer("pub2", "onion2.onion"), + ]; + save_peers(dir.path(), &peers).await.unwrap(); + let loaded = load_peers(dir.path()).await.unwrap(); + assert_eq!(loaded.len(), 2); + assert_eq!(loaded[0].pubkey, "pub1"); + assert_eq!(loaded[1].onion, "onion2.onion"); + } + + #[tokio::test] + async fn test_add_peer_appends_new() { + let dir = tempfile::tempdir().unwrap(); + let peer = make_peer("pubkey-a", "a.onion"); + let result = add_peer(dir.path(), peer).await.unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result[0].pubkey, "pubkey-a"); + + // Add a second peer + let peer2 = make_peer("pubkey-b", "b.onion"); + let result = add_peer(dir.path(), peer2).await.unwrap(); + assert_eq!(result.len(), 2); + } + + #[tokio::test] + async fn test_add_peer_deduplicates_by_pubkey() { + let dir = tempfile::tempdir().unwrap(); + let peer = make_peer("same-key", "first.onion"); + add_peer(dir.path(), peer).await.unwrap(); + + // Adding a peer with the same pubkey should not duplicate + let peer_dup = make_peer("same-key", "second.onion"); + let result = add_peer(dir.path(), peer_dup).await.unwrap(); + assert_eq!(result.len(), 1); + // Original should be kept (not replaced) + assert_eq!(result[0].onion, "first.onion"); + } + + #[tokio::test] + async fn test_remove_peer_by_pubkey() { + let dir = tempfile::tempdir().unwrap(); + add_peer(dir.path(), make_peer("key-1", "a.onion")).await.unwrap(); + add_peer(dir.path(), make_peer("key-2", "b.onion")).await.unwrap(); + add_peer(dir.path(), make_peer("key-3", "c.onion")).await.unwrap(); + + let result = remove_peer(dir.path(), "key-2").await.unwrap(); + assert_eq!(result.len(), 2); + assert!(result.iter().all(|p| p.pubkey != "key-2")); + } + + #[tokio::test] + async fn test_remove_nonexistent_peer_is_noop() { + let dir = tempfile::tempdir().unwrap(); + add_peer(dir.path(), make_peer("key-1", "a.onion")).await.unwrap(); + + // Removing a pubkey that doesn't exist should succeed but not change the list + let result = remove_peer(dir.path(), "nonexistent").await.unwrap(); + assert_eq!(result.len(), 1); + assert_eq!(result[0].pubkey, "key-1"); + } +} diff --git a/core/archipelago/src/port_allocator.rs b/core/archipelago/src/port_allocator.rs index 4e40958b..a8243a37 100644 --- a/core/archipelago/src/port_allocator.rs +++ b/core/archipelago/src/port_allocator.rs @@ -146,3 +146,144 @@ impl PortAllocator { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_port_allocations_default_is_empty() { + let allocs = PortAllocations::default(); + assert!(allocs.allocations.is_empty()); + } + + #[test] + fn test_new_allocator_from_empty_dir() { + let dir = tempfile::tempdir().unwrap(); + let alloc = PortAllocator::new(dir.path()).unwrap(); + assert!(alloc.allocations.allocations.is_empty()); + } + + #[test] + fn test_allocate_preferred_port_when_available() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + let port = alloc.allocate("my-app", 8500, 80).unwrap(); + assert_eq!(port, 8500); + } + + #[test] + fn test_allocate_fallback_when_preferred_is_reserved() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + // Port 80 is in RESERVED_PORTS, so it should allocate from the range instead + let port = alloc.allocate("web-app", 80, 80).unwrap(); + assert_ne!(port, 80); + assert!(port >= WEB_PORT_RANGE_START && port <= WEB_PORT_RANGE_END); + } + + #[test] + fn test_allocate_fallback_when_preferred_is_taken() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + let port1 = alloc.allocate("app-1", 8500, 80).unwrap(); + assert_eq!(port1, 8500); + + // Second app requesting the same preferred port gets a different one + let port2 = alloc.allocate("app-2", 8500, 80).unwrap(); + assert_ne!(port2, 8500); + assert!(port2 >= WEB_PORT_RANGE_START && port2 <= WEB_PORT_RANGE_END); + } + + #[test] + fn test_get_returns_existing_allocation() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + alloc.allocate("test-app", 8600, 3000).unwrap(); + + let result = alloc.get("test-app"); + assert!(result.is_some()); + let (host, container) = result.unwrap(); + assert_eq!(host, 8600); + assert_eq!(container, 3000); + } + + #[test] + fn test_get_returns_none_for_unknown_app() { + let dir = tempfile::tempdir().unwrap(); + let alloc = PortAllocator::new(dir.path()).unwrap(); + assert!(alloc.get("nonexistent").is_none()); + } + + #[test] + fn test_allocate_or_get_returns_existing() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + let port1 = alloc.allocate("my-app", 8700, 80).unwrap(); + + // Calling allocate_or_get with a different preferred port should return the existing one + let port2 = alloc.allocate_or_get("my-app", 9999, 80).unwrap(); + assert_eq!(port1, port2); + assert_eq!(port2, 8700); + } + + #[test] + fn test_allocate_or_get_allocates_when_new() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + let port = alloc.allocate_or_get("new-app", 8800, 443).unwrap(); + assert_eq!(port, 8800); + + // Verify it's now stored + assert!(alloc.get("new-app").is_some()); + } + + #[test] + fn test_release_removes_allocation() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + alloc.allocate("removable", 8900, 80).unwrap(); + assert!(alloc.get("removable").is_some()); + + alloc.release("removable").unwrap(); + assert!(alloc.get("removable").is_none()); + } + + #[test] + fn test_released_port_becomes_available() { + let dir = tempfile::tempdir().unwrap(); + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + alloc.allocate("app-a", 8500, 80).unwrap(); + alloc.release("app-a").unwrap(); + + // Port 8500 should now be available again + let port = alloc.allocate("app-b", 8500, 80).unwrap(); + assert_eq!(port, 8500); + } + + #[test] + fn test_reserved_ports_are_never_allocated() { + let dir = tempfile::tempdir().unwrap(); + let alloc = PortAllocator::new(dir.path()).unwrap(); + for &port in RESERVED_PORTS { + assert!(alloc.is_reserved(port), "Port {} should be reserved", port); + assert!(!alloc.is_available(port), "Port {} should not be available", port); + } + } + + #[test] + fn test_persistence_across_instances() { + let dir = tempfile::tempdir().unwrap(); + { + let mut alloc = PortAllocator::new(dir.path()).unwrap(); + alloc.allocate("persistent-app", 8555, 80).unwrap(); + } + // Create a new allocator from the same directory + let alloc2 = PortAllocator::new(dir.path()).unwrap(); + let result = alloc2.get("persistent-app"); + assert!(result.is_some()); + let (host, container) = result.unwrap(); + assert_eq!(host, 8555); + assert_eq!(container, 80); + } +} diff --git a/core/archipelago/src/server.rs b/core/archipelago/src/server.rs index 7ec2cba0..d9fa643f 100644 --- a/core/archipelago/src/server.rs +++ b/core/archipelago/src/server.rs @@ -46,29 +46,27 @@ impl Server { tracing::debug!("Nostr revoke (non-fatal): {}", e); } - // Publish node identity to Nostr only when opt-in (nostr_discovery_enabled + relays) + // Publish presence-only to Nostr (DID + Nostr pubkey, NO onion address). + // Onion addresses are exchanged privately via NIP-44 encrypted DMs. if config.nostr_discovery_enabled && !config.nostr_relays.is_empty() - && data.server_info.node_address.is_some() { let identity_dir = config.data_dir.join("identity"); let did = identity::did_key_from_pubkey_hex(&data.server_info.pubkey).unwrap_or_default(); - let node_addr = data.server_info.node_address.clone().unwrap_or_default(); let version = data.server_info.version.clone(); let relays = config.nostr_relays.clone(); let tor_proxy = config.nostr_tor_proxy.clone(); tokio::spawn(async move { - if let Err(e) = nostr_discovery::publish_node_identity( + if let Err(e) = nostr_handshake::publish_presence( &identity_dir, &did, - &node_addr, &version, &relays, tor_proxy.as_deref(), ) .await { - tracing::debug!("Nostr publish (non-fatal): {}", e); + tracing::debug!("Nostr presence publish (non-fatal): {}", e); } }); } @@ -100,25 +98,25 @@ impl Server { }); } - // Initialize Docker scanner if in dev mode - if config.dev_mode { + // Initialize container scanner — discovers installed apps from Podman/Docker + { let scanner = create_docker_scanner(&config).await?; let state = state_manager.clone(); let identity_clone = identity.clone(); - + // Initial scan tokio::spawn(async move { - info!("🐳 Scanning Docker containers..."); + info!("🐳 Scanning containers..."); if let Err(e) = scan_and_update_packages(&scanner, &state, identity_clone.as_ref()).await { - error!("Failed to scan Docker containers: {}", e); + error!("Failed to scan containers: {}", e); } - + // Periodic scan every 10 seconds (only broadcasts if state changed) let mut interval = tokio::time::interval(Duration::from_secs(10)); loop { interval.tick().await; if let Err(e) = scan_and_update_packages(&scanner, &state, identity_clone.as_ref()).await { - error!("Failed to update Docker containers: {}", e); + error!("Failed to update containers: {}", e); } } }); diff --git a/core/archipelago/src/state.rs b/core/archipelago/src/state.rs index c1d1a513..c20e74db 100644 --- a/core/archipelago/src/state.rs +++ b/core/archipelago/src/state.rs @@ -70,3 +70,117 @@ impl Default for StateManager { Self::new() } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::data_model::DataModel; + + #[test] + fn test_state_manager_new() { + let sm = StateManager::new(); + // Should be constructible without panic + let _ = sm.subscribe(); + } + + #[test] + fn test_state_manager_default() { + let sm = StateManager::default(); + let _ = sm.subscribe(); + } + + #[tokio::test] + async fn test_get_snapshot_initial_revision_zero() { + let sm = StateManager::new(); + let (data, rev) = sm.get_snapshot().await; + assert_eq!(rev, 0); + // DataModel::new() sets version from CARGO_PKG_VERSION + assert_eq!(data.server_info.version, env!("CARGO_PKG_VERSION")); + assert!(data.package_data.is_empty()); + assert!(data.notifications.is_empty()); + } + + #[tokio::test] + async fn test_update_data_increments_revision() { + let sm = StateManager::new(); + + let (_, rev0) = sm.get_snapshot().await; + assert_eq!(rev0, 0); + + sm.update_data(DataModel::new()).await; + let (_, rev1) = sm.get_snapshot().await; + assert_eq!(rev1, 1); + + sm.update_data(DataModel::new()).await; + let (_, rev2) = sm.get_snapshot().await; + assert_eq!(rev2, 2); + } + + #[tokio::test] + async fn test_update_data_stores_new_data() { + let sm = StateManager::new(); + + let mut new_data = DataModel::new(); + new_data.server_info.name = Some("TestNode".to_string()); + new_data.ui.theme = "light".to_string(); + sm.update_data(new_data).await; + + let (snapshot, _) = sm.get_snapshot().await; + assert_eq!(snapshot.server_info.name, Some("TestNode".to_string())); + assert_eq!(snapshot.ui.theme, "light"); + } + + #[tokio::test] + async fn test_subscribe_receives_updates() { + let sm = StateManager::new(); + let mut rx = sm.subscribe(); + + let mut data = DataModel::new(); + data.server_info.name = Some("BroadcastTest".to_string()); + sm.update_data(data).await; + + let msg = rx.recv().await.unwrap(); + assert_eq!(msg.rev, 1); + assert!(msg.data.is_some()); + let received_data = msg.data.unwrap(); + assert_eq!(received_data.server_info.name, Some("BroadcastTest".to_string())); + } + + #[tokio::test] + async fn test_multiple_subscribers_receive_same_update() { + let sm = StateManager::new(); + let mut rx1 = sm.subscribe(); + let mut rx2 = sm.subscribe(); + + sm.update_data(DataModel::new()).await; + + let msg1 = rx1.recv().await.unwrap(); + let msg2 = rx2.recv().await.unwrap(); + assert_eq!(msg1.rev, msg2.rev); + assert_eq!(msg1.rev, 1); + } + + #[tokio::test] + async fn test_get_initial_message() { + let sm = StateManager::new(); + + let mut data = DataModel::new(); + data.server_info.name = Some("InitMsg".to_string()); + sm.update_data(data).await; + + let msg = sm.get_initial_message().await; + assert_eq!(msg.rev, 1); + assert!(msg.data.is_some()); + assert!(msg.patch.is_none()); + assert_eq!(msg.data.unwrap().server_info.name, Some("InitMsg".to_string())); + } + + #[tokio::test] + async fn test_update_without_subscribers_does_not_panic() { + let sm = StateManager::new(); + // No subscribers — send should silently succeed + sm.update_data(DataModel::new()).await; + let (_, rev) = sm.get_snapshot().await; + assert_eq!(rev, 1); + } +} diff --git a/core/archipelago/src/vpn.rs b/core/archipelago/src/vpn.rs new file mode 100644 index 00000000..adab62ee --- /dev/null +++ b/core/archipelago/src/vpn.rs @@ -0,0 +1,480 @@ +//! VPN integration: Tailscale and WireGuard management. +//! +//! Manages VPN connections, generates WireGuard configs, and monitors +//! VPN interface status for remote access to the Archipelago node. + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use std::path::Path; +use tokio::fs; + +const VPN_CONFIG_FILE: &str = "vpn-config.json"; + +/// VPN provider type. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum VpnProvider { + Tailscale, + Wireguard, +} + +/// Persisted VPN configuration. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VpnConfig { + pub provider: VpnProvider, + pub enabled: bool, + #[serde(default)] + pub tailscale_auth_key: Option, + #[serde(default)] + pub wireguard_config: Option, + #[serde(default)] + pub configured_at: Option, +} + +/// WireGuard configuration. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct WireGuardConfig { + pub private_key: String, + pub public_key: String, + pub address: String, + pub dns: String, + #[serde(default)] + pub peers: Vec, +} + +/// A WireGuard peer. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct WireGuardPeer { + pub public_key: String, + pub endpoint: String, + pub allowed_ips: String, + #[serde(default)] + pub persistent_keepalive: Option, +} + +/// Current VPN status (gathered at runtime, not persisted). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VpnStatus { + pub connected: bool, + pub provider: Option, + pub interface: Option, + pub ip_address: Option, + pub hostname: Option, + #[serde(default)] + pub peers_connected: u32, + #[serde(default)] + pub bytes_in: u64, + #[serde(default)] + pub bytes_out: u64, +} + +impl Default for VpnConfig { + fn default() -> Self { + Self { + provider: VpnProvider::Tailscale, + enabled: false, + tailscale_auth_key: None, + wireguard_config: None, + configured_at: None, + } + } +} + +pub async fn load_config(data_dir: &Path) -> Result { + let path = data_dir.join(VPN_CONFIG_FILE); + if !path.exists() { + return Ok(VpnConfig::default()); + } + let content = fs::read_to_string(&path) + .await + .context("Failed to read VPN config")?; + let config: VpnConfig = serde_json::from_str(&content).unwrap_or_default(); + Ok(config) +} + +pub async fn save_config(data_dir: &Path, config: &VpnConfig) -> Result<()> { + fs::create_dir_all(data_dir).await.context("Failed to create data dir")?; + let content = serde_json::to_string_pretty(config).context("Failed to serialize VPN config")?; + fs::write(data_dir.join(VPN_CONFIG_FILE), content) + .await + .context("Failed to write VPN config")?; + Ok(()) +} + +/// Generate a WireGuard keypair using the `wg` command. +pub async fn generate_wireguard_keypair() -> Result<(String, String)> { + let privkey_output = tokio::process::Command::new("wg") + .arg("genkey") + .output() + .await + .context("Failed to run wg genkey — is wireguard-tools installed?")?; + + if !privkey_output.status.success() { + anyhow::bail!( + "wg genkey failed: {}", + String::from_utf8_lossy(&privkey_output.stderr) + ); + } + + let private_key = String::from_utf8(privkey_output.stdout) + .context("Invalid UTF-8 from wg genkey")? + .trim() + .to_string(); + + let _pubkey_output = tokio::process::Command::new("wg") + .arg("pubkey") + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .spawn() + .context("Failed to spawn wg pubkey")?; + + // Use echo + pipe approach instead + let pubkey_output = tokio::process::Command::new("sh") + .arg("-c") + .arg(format!("echo '{}' | wg pubkey", private_key)) + .output() + .await + .context("Failed to derive public key")?; + + if !pubkey_output.status.success() { + anyhow::bail!( + "wg pubkey failed: {}", + String::from_utf8_lossy(&pubkey_output.stderr) + ); + } + + let public_key = String::from_utf8(pubkey_output.stdout) + .context("Invalid UTF-8 from wg pubkey")? + .trim() + .to_string(); + + Ok((private_key, public_key)) +} + +/// Generate a WireGuard configuration file content. +pub fn generate_wireguard_conf(config: &WireGuardConfig) -> String { + let mut conf = format!( + "[Interface]\nPrivateKey = {}\nAddress = {}\nDNS = {}\n", + config.private_key, config.address, config.dns + ); + + for peer in &config.peers { + conf.push_str(&format!( + "\n[Peer]\nPublicKey = {}\nEndpoint = {}\nAllowedIPs = {}\n", + peer.public_key, peer.endpoint, peer.allowed_ips + )); + if let Some(ka) = peer.persistent_keepalive { + conf.push_str(&format!("PersistentKeepalive = {}\n", ka)); + } + } + + conf +} + +/// Get the current VPN status by checking network interfaces. +pub async fn get_status() -> VpnStatus { + // Check for Tailscale interface + if let Ok(tailscale) = get_tailscale_status().await { + return tailscale; + } + + // Check for WireGuard interface + if let Ok(wg) = get_wireguard_status().await { + return wg; + } + + VpnStatus { + connected: false, + provider: None, + interface: None, + ip_address: None, + hostname: None, + peers_connected: 0, + bytes_in: 0, + bytes_out: 0, + } +} + +async fn get_tailscale_status() -> Result { + // Check if tailscale0 interface exists + let output = tokio::process::Command::new("ip") + .args(["addr", "show", "tailscale0"]) + .output() + .await + .context("Failed to check tailscale0")?; + + if !output.status.success() { + anyhow::bail!("No tailscale0 interface"); + } + + let stdout = String::from_utf8_lossy(&output.stdout); + let ip = stdout + .lines() + .find(|l| l.contains("inet ") && !l.contains("inet6")) + .and_then(|l| { + l.split_whitespace() + .nth(1) + .map(|ip| ip.split('/').next().unwrap_or(ip).to_string()) + }); + + // Try to get hostname from tailscale status + let hostname = tokio::process::Command::new("sh") + .arg("-c") + .arg("podman exec tailscale tailscale status --self --json 2>/dev/null | grep -o '\"DNSName\":\"[^\"]*\"' | head -1 | cut -d'\"' -f4") + .output() + .await + .ok() + .and_then(|o| { + let s = String::from_utf8_lossy(&o.stdout).trim().to_string(); + if s.is_empty() { None } else { Some(s.trim_end_matches('.').to_string()) } + }); + + // Get peer count + let peers = tokio::process::Command::new("sh") + .arg("-c") + .arg("podman exec tailscale tailscale status 2>/dev/null | grep -c 'active' || echo 0") + .output() + .await + .ok() + .and_then(|o| { + String::from_utf8_lossy(&o.stdout) + .trim() + .parse::() + .ok() + }) + .unwrap_or(0); + + Ok(VpnStatus { + connected: ip.is_some(), + provider: Some("tailscale".to_string()), + interface: Some("tailscale0".to_string()), + ip_address: ip, + hostname, + peers_connected: peers, + bytes_in: 0, + bytes_out: 0, + }) +} + +async fn get_wireguard_status() -> Result { + let output = tokio::process::Command::new("ip") + .args(["addr", "show", "wg0"]) + .output() + .await + .context("Failed to check wg0")?; + + if !output.status.success() { + anyhow::bail!("No wg0 interface"); + } + + let stdout = String::from_utf8_lossy(&output.stdout); + let ip = stdout + .lines() + .find(|l| l.contains("inet ") && !l.contains("inet6")) + .and_then(|l| { + l.split_whitespace() + .nth(1) + .map(|ip| ip.split('/').next().unwrap_or(ip).to_string()) + }); + + // Get peer count from wg show + let peers = tokio::process::Command::new("sh") + .arg("-c") + .arg("wg show wg0 peers 2>/dev/null | wc -l") + .output() + .await + .ok() + .and_then(|o| { + String::from_utf8_lossy(&o.stdout) + .trim() + .parse::() + .ok() + }) + .unwrap_or(0); + + // Get transfer stats + let (bytes_in, bytes_out) = tokio::process::Command::new("sh") + .arg("-c") + .arg("wg show wg0 transfer 2>/dev/null | awk '{i+=$2; o+=$3} END {print i, o}'") + .output() + .await + .ok() + .and_then(|o| { + let s = String::from_utf8_lossy(&o.stdout); + let mut parts = s.trim().split_whitespace(); + let i = parts.next().and_then(|v| v.parse::().ok()).unwrap_or(0); + let o = parts.next().and_then(|v| v.parse::().ok()).unwrap_or(0); + Some((i, o)) + }) + .unwrap_or((0, 0)); + + Ok(VpnStatus { + connected: ip.is_some(), + provider: Some("wireguard".to_string()), + interface: Some("wg0".to_string()), + ip_address: ip, + hostname: None, + peers_connected: peers, + bytes_in, + bytes_out, + }) +} + +/// Configure Tailscale with an auth key (triggers tailscale up). +pub async fn configure_tailscale(auth_key: &str, data_dir: &Path) -> Result<()> { + let mut config = load_config(data_dir).await?; + config.provider = VpnProvider::Tailscale; + config.enabled = true; + config.tailscale_auth_key = Some(auth_key.to_string()); + config.configured_at = Some(chrono::Utc::now().to_rfc3339()); + save_config(data_dir, &config).await?; + + // Run tailscale up with auth key in the container + let output = tokio::process::Command::new("podman") + .args([ + "exec", "tailscale", "tailscale", "up", + "--authkey", auth_key, + "--accept-routes", + ]) + .output() + .await + .context("Failed to run tailscale up")?; + + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr); + anyhow::bail!("tailscale up failed: {}", stderr); + } + + Ok(()) +} + +/// Configure WireGuard with generated keys and optional peer. +pub async fn configure_wireguard( + data_dir: &Path, + address: &str, + dns: &str, + peer: Option, +) -> Result { + let (private_key, public_key) = generate_wireguard_keypair().await?; + + let wg_config = WireGuardConfig { + private_key: private_key.clone(), + public_key: public_key.clone(), + address: address.to_string(), + dns: dns.to_string(), + peers: peer.map_or_else(Vec::new, |p| vec![p]), + }; + + let mut config = load_config(data_dir).await?; + config.provider = VpnProvider::Wireguard; + config.enabled = true; + config.wireguard_config = Some(wg_config.clone()); + config.configured_at = Some(chrono::Utc::now().to_rfc3339()); + save_config(data_dir, &config).await?; + + // Write WireGuard config file + let conf_content = generate_wireguard_conf(&wg_config); + let wg_dir = data_dir.join("wireguard"); + fs::create_dir_all(&wg_dir).await.context("Failed to create wireguard dir")?; + fs::write(wg_dir.join("wg0.conf"), &conf_content) + .await + .context("Failed to write wg0.conf")?; + + Ok(wg_config) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_vpn_config_default() { + let config = VpnConfig::default(); + assert!(!config.enabled); + assert_eq!(config.provider, VpnProvider::Tailscale); + assert!(config.tailscale_auth_key.is_none()); + } + + #[test] + fn test_vpn_config_serialization() { + let config = VpnConfig { + provider: VpnProvider::Wireguard, + enabled: true, + tailscale_auth_key: None, + wireguard_config: Some(WireGuardConfig { + private_key: "privkey".to_string(), + public_key: "pubkey".to_string(), + address: "10.0.0.1/24".to_string(), + dns: "1.1.1.1".to_string(), + peers: vec![], + }), + configured_at: Some("2026-01-01T00:00:00Z".to_string()), + }; + let json = serde_json::to_string(&config).unwrap(); + let parsed: VpnConfig = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.provider, VpnProvider::Wireguard); + assert!(parsed.enabled); + assert!(parsed.wireguard_config.is_some()); + } + + #[test] + fn test_generate_wireguard_conf() { + let config = WireGuardConfig { + private_key: "test_privkey".to_string(), + public_key: "test_pubkey".to_string(), + address: "10.0.0.2/24".to_string(), + dns: "1.1.1.1, 8.8.8.8".to_string(), + peers: vec![WireGuardPeer { + public_key: "peer_pubkey".to_string(), + endpoint: "vpn.example.com:51820".to_string(), + allowed_ips: "0.0.0.0/0".to_string(), + persistent_keepalive: Some(25), + }], + }; + let conf = generate_wireguard_conf(&config); + assert!(conf.contains("[Interface]")); + assert!(conf.contains("PrivateKey = test_privkey")); + assert!(conf.contains("Address = 10.0.0.2/24")); + assert!(conf.contains("[Peer]")); + assert!(conf.contains("PublicKey = peer_pubkey")); + assert!(conf.contains("PersistentKeepalive = 25")); + } + + #[tokio::test] + async fn test_load_config_default_when_no_file() { + let dir = tempfile::tempdir().unwrap(); + let config = load_config(dir.path()).await.unwrap(); + assert!(!config.enabled); + } + + #[tokio::test] + async fn test_save_and_load_config_roundtrip() { + let dir = tempfile::tempdir().unwrap(); + let config = VpnConfig { + provider: VpnProvider::Tailscale, + enabled: true, + tailscale_auth_key: Some("tskey-auth-test".to_string()), + wireguard_config: None, + configured_at: Some("2026-03-10T00:00:00Z".to_string()), + }; + save_config(dir.path(), &config).await.unwrap(); + let loaded = load_config(dir.path()).await.unwrap(); + assert!(loaded.enabled); + assert_eq!(loaded.tailscale_auth_key, Some("tskey-auth-test".to_string())); + } + + #[test] + fn test_vpn_status_default() { + let status = VpnStatus { + connected: false, + provider: None, + interface: None, + ip_address: None, + hostname: None, + peers_connected: 0, + bytes_in: 0, + bytes_out: 0, + }; + assert!(!status.connected); + } +} diff --git a/core/archipelago/src/wallet/ecash.rs b/core/archipelago/src/wallet/ecash.rs index bfcbc9f5..c0509308 100644 --- a/core/archipelago/src/wallet/ecash.rs +++ b/core/archipelago/src/wallet/ecash.rs @@ -276,3 +276,307 @@ pub async fn receive_token(data_dir: &Path, token_str: &str) -> Result { fn default_mint_url() -> String { "http://127.0.0.1:8175".to_string() } + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + #[test] + fn test_wallet_state_balance_empty() { + let wallet = WalletState::default(); + assert_eq!(wallet.balance(), 0); + } + + #[test] + fn test_wallet_state_balance_unspent_only() { + let wallet = WalletState { + tokens: vec![ + EcashToken { + id: "t1".into(), + amount_sats: 100, + token: "tok1".into(), + mint_url: "http://mint".into(), + spent: false, + created_at: "2025-01-01T00:00:00Z".into(), + }, + EcashToken { + id: "t2".into(), + amount_sats: 200, + token: "tok2".into(), + mint_url: "http://mint".into(), + spent: true, + created_at: "2025-01-01T00:00:00Z".into(), + }, + EcashToken { + id: "t3".into(), + amount_sats: 50, + token: "tok3".into(), + mint_url: "http://mint".into(), + spent: false, + created_at: "2025-01-01T00:00:00Z".into(), + }, + ], + transactions: vec![], + mint_url: String::new(), + }; + // Only t1 (100) and t3 (50) are unspent + assert_eq!(wallet.balance(), 150); + } + + #[test] + fn test_wallet_state_balance_all_spent() { + let wallet = WalletState { + tokens: vec![EcashToken { + id: "t1".into(), + amount_sats: 500, + token: "tok1".into(), + mint_url: "http://mint".into(), + spent: true, + created_at: "2025-01-01T00:00:00Z".into(), + }], + transactions: vec![], + mint_url: String::new(), + }; + assert_eq!(wallet.balance(), 0); + } + + #[test] + fn test_default_mint_url() { + let url = default_mint_url(); + assert_eq!(url, "http://127.0.0.1:8175"); + } + + #[tokio::test] + async fn test_load_wallet_creates_default_when_missing() { + let tmp = TempDir::new().unwrap(); + let wallet = load_wallet(tmp.path()).await.unwrap(); + assert_eq!(wallet.balance(), 0); + assert!(wallet.tokens.is_empty()); + assert!(wallet.transactions.is_empty()); + assert_eq!(wallet.mint_url, default_mint_url()); + } + + #[tokio::test] + async fn test_save_and_load_wallet_roundtrip() { + let tmp = TempDir::new().unwrap(); + let wallet = WalletState { + tokens: vec![EcashToken { + id: "round-trip-token".into(), + amount_sats: 42, + token: "cashuA_test".into(), + mint_url: "http://127.0.0.1:8175".into(), + spent: false, + created_at: "2025-06-01T12:00:00Z".into(), + }], + transactions: vec![EcashTransaction { + id: "tx1".into(), + tx_type: TransactionType::Mint, + amount_sats: 42, + timestamp: "2025-06-01T12:00:00Z".into(), + description: "Test mint".into(), + }], + mint_url: "http://127.0.0.1:8175".into(), + }; + + save_wallet(tmp.path(), &wallet).await.unwrap(); + let loaded = load_wallet(tmp.path()).await.unwrap(); + assert_eq!(loaded.tokens.len(), 1); + assert_eq!(loaded.tokens[0].id, "round-trip-token"); + assert_eq!(loaded.tokens[0].amount_sats, 42); + assert!(!loaded.tokens[0].spent); + assert_eq!(loaded.transactions.len(), 1); + assert_eq!(loaded.balance(), 42); + } + + #[tokio::test] + async fn test_save_wallet_creates_directory() { + let tmp = TempDir::new().unwrap(); + let wallet_dir = tmp.path().join("wallet"); + assert!(!wallet_dir.exists()); + + let wallet = WalletState::default(); + save_wallet(tmp.path(), &wallet).await.unwrap(); + assert!(wallet_dir.exists()); + } + + #[tokio::test] + async fn test_receive_token_valid() { + let tmp = TempDir::new().unwrap(); + let amount = receive_token(tmp.path(), "cashuSend_500_abc123_1700000000") + .await + .unwrap(); + assert_eq!(amount, 500); + + let wallet = load_wallet(tmp.path()).await.unwrap(); + assert_eq!(wallet.balance(), 500); + assert_eq!(wallet.tokens.len(), 1); + assert!(!wallet.tokens[0].spent); + assert_eq!(wallet.transactions.len(), 1); + assert!(matches!( + wallet.transactions[0].tx_type, + TransactionType::Receive + )); + } + + #[tokio::test] + async fn test_receive_token_invalid_format() { + let tmp = TempDir::new().unwrap(); + let result = receive_token(tmp.path(), "invalid_token_string").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_receive_token_zero_amount() { + let tmp = TempDir::new().unwrap(); + let result = receive_token(tmp.path(), "cashuSend_0_abc_1700000000").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_melt_tokens_marks_spent() { + let tmp = TempDir::new().unwrap(); + // Pre-populate a wallet with a token + let wallet = WalletState { + tokens: vec![EcashToken { + id: "melt-me".into(), + amount_sats: 1000, + token: "cashuA_test".into(), + mint_url: default_mint_url(), + spent: false, + created_at: "2025-01-01T00:00:00Z".into(), + }], + transactions: vec![], + mint_url: default_mint_url(), + }; + save_wallet(tmp.path(), &wallet).await.unwrap(); + + let amount = melt_tokens(tmp.path(), "melt-me").await.unwrap(); + assert_eq!(amount, 1000); + + let loaded = load_wallet(tmp.path()).await.unwrap(); + assert!(loaded.tokens[0].spent); + assert_eq!(loaded.balance(), 0); + assert_eq!(loaded.transactions.len(), 1); + assert!(matches!( + loaded.transactions[0].tx_type, + TransactionType::Melt + )); + } + + #[tokio::test] + async fn test_melt_tokens_not_found() { + let tmp = TempDir::new().unwrap(); + let result = melt_tokens(tmp.path(), "nonexistent").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_melt_already_spent_token_errors() { + let tmp = TempDir::new().unwrap(); + let wallet = WalletState { + tokens: vec![EcashToken { + id: "already-spent".into(), + amount_sats: 100, + token: "cashuA_x".into(), + mint_url: default_mint_url(), + spent: true, + created_at: "2025-01-01T00:00:00Z".into(), + }], + transactions: vec![], + mint_url: default_mint_url(), + }; + save_wallet(tmp.path(), &wallet).await.unwrap(); + + let result = melt_tokens(tmp.path(), "already-spent").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_send_token_sufficient_balance() { + let tmp = TempDir::new().unwrap(); + let wallet = WalletState { + tokens: vec![ + EcashToken { + id: "s1".into(), + amount_sats: 300, + token: "tok1".into(), + mint_url: default_mint_url(), + spent: false, + created_at: "2025-01-01T00:00:00Z".into(), + }, + EcashToken { + id: "s2".into(), + amount_sats: 200, + token: "tok2".into(), + mint_url: default_mint_url(), + spent: false, + created_at: "2025-01-01T00:00:00Z".into(), + }, + ], + transactions: vec![], + mint_url: default_mint_url(), + }; + save_wallet(tmp.path(), &wallet).await.unwrap(); + + let token_str = send_token(tmp.path(), 400).await.unwrap(); + assert!(token_str.starts_with("cashuSend_400_")); + + let loaded = load_wallet(tmp.path()).await.unwrap(); + // Both tokens should be marked spent (300 + 200 = 500 >= 400) + assert!(loaded.tokens.iter().all(|t| t.spent)); + assert_eq!(loaded.balance(), 0); + } + + #[tokio::test] + async fn test_send_token_insufficient_balance() { + let tmp = TempDir::new().unwrap(); + let wallet = WalletState { + tokens: vec![EcashToken { + id: "small".into(), + amount_sats: 10, + token: "tok".into(), + mint_url: default_mint_url(), + spent: false, + created_at: "2025-01-01T00:00:00Z".into(), + }], + transactions: vec![], + mint_url: default_mint_url(), + }; + save_wallet(tmp.path(), &wallet).await.unwrap(); + + let result = send_token(tmp.path(), 1000).await; + assert!(result.is_err()); + let err_msg = result.unwrap_err().to_string(); + assert!(err_msg.contains("Insufficient balance")); + } + + #[test] + fn test_wallet_state_serialization() { + let wallet = WalletState { + tokens: vec![], + transactions: vec![], + mint_url: "http://localhost:8175".into(), + }; + let json = serde_json::to_string(&wallet).unwrap(); + let parsed: WalletState = serde_json::from_str(&json).unwrap(); + assert_eq!(parsed.mint_url, wallet.mint_url); + assert!(parsed.tokens.is_empty()); + } + + #[test] + fn test_transaction_type_serialization() { + let tx = EcashTransaction { + id: "tx-test".into(), + tx_type: TransactionType::Send, + amount_sats: 100, + timestamp: "2025-01-01T00:00:00Z".into(), + description: "test".into(), + }; + let json = serde_json::to_string(&tx).unwrap(); + assert!(json.contains("\"send\"")); + + let parsed: EcashTransaction = serde_json::from_str(&json).unwrap(); + assert!(matches!(parsed.tx_type, TransactionType::Send)); + } +} diff --git a/core/archipelago/src/wallet/profits.rs b/core/archipelago/src/wallet/profits.rs index b2828f2c..9b0671c6 100644 --- a/core/archipelago/src/wallet/profits.rs +++ b/core/archipelago/src/wallet/profits.rs @@ -112,3 +112,167 @@ pub async fn get_networking_profits(data_dir: &Path) -> Result { Ok(summary) } + +#[cfg(test)] +mod tests { + use super::*; + use tempfile::TempDir; + + #[test] + fn test_profits_summary_default() { + let summary = ProfitsSummary::default(); + assert_eq!(summary.total_sats, 0); + assert_eq!(summary.content_sales_sats, 0); + assert_eq!(summary.routing_fees_sats, 0); + assert!(summary.recent.is_empty()); + } + + #[tokio::test] + async fn test_load_profits_returns_default_when_missing() { + let tmp = TempDir::new().unwrap(); + let summary = load_profits(tmp.path()).await.unwrap(); + assert_eq!(summary.total_sats, 0); + assert_eq!(summary.content_sales_sats, 0); + assert_eq!(summary.routing_fees_sats, 0); + assert!(summary.recent.is_empty()); + } + + #[tokio::test] + async fn test_save_and_load_profits_roundtrip() { + let tmp = TempDir::new().unwrap(); + let summary = ProfitsSummary { + total_sats: 5000, + content_sales_sats: 3000, + routing_fees_sats: 2000, + recent: vec![ProfitEntry { + source: ProfitSource::ContentSale, + amount_sats: 3000, + timestamp: "2025-06-01T00:00:00Z".to_string(), + description: "Test sale".to_string(), + }], + }; + + save_profits(tmp.path(), &summary).await.unwrap(); + let loaded = load_profits(tmp.path()).await.unwrap(); + assert_eq!(loaded.total_sats, 5000); + assert_eq!(loaded.content_sales_sats, 3000); + assert_eq!(loaded.routing_fees_sats, 2000); + assert_eq!(loaded.recent.len(), 1); + assert_eq!(loaded.recent[0].amount_sats, 3000); + } + + #[tokio::test] + async fn test_save_profits_creates_wallet_dir() { + let tmp = TempDir::new().unwrap(); + let wallet_dir = tmp.path().join("wallet"); + assert!(!wallet_dir.exists()); + + save_profits(tmp.path(), &ProfitsSummary::default()).await.unwrap(); + assert!(wallet_dir.exists()); + } + + #[tokio::test] + async fn test_record_content_sale() { + let tmp = TempDir::new().unwrap(); + record_content_sale(tmp.path(), 500, "First sale").await.unwrap(); + + let summary = load_profits(tmp.path()).await.unwrap(); + assert_eq!(summary.total_sats, 500); + assert_eq!(summary.content_sales_sats, 500); + assert_eq!(summary.routing_fees_sats, 0); + assert_eq!(summary.recent.len(), 1); + assert_eq!(summary.recent[0].amount_sats, 500); + assert_eq!(summary.recent[0].description, "First sale"); + assert!(matches!(summary.recent[0].source, ProfitSource::ContentSale)); + } + + #[tokio::test] + async fn test_record_multiple_content_sales() { + let tmp = TempDir::new().unwrap(); + record_content_sale(tmp.path(), 100, "Sale 1").await.unwrap(); + record_content_sale(tmp.path(), 200, "Sale 2").await.unwrap(); + record_content_sale(tmp.path(), 300, "Sale 3").await.unwrap(); + + let summary = load_profits(tmp.path()).await.unwrap(); + assert_eq!(summary.total_sats, 600); + assert_eq!(summary.content_sales_sats, 600); + assert_eq!(summary.recent.len(), 3); + // Newest first (inserted at index 0) + assert_eq!(summary.recent[0].description, "Sale 3"); + assert_eq!(summary.recent[1].description, "Sale 2"); + assert_eq!(summary.recent[2].description, "Sale 1"); + } + + #[tokio::test] + async fn test_record_content_sale_truncates_at_100() { + let tmp = TempDir::new().unwrap(); + for i in 0..110 { + record_content_sale(tmp.path(), 1, &format!("Sale {}", i)) + .await + .unwrap(); + } + + let summary = load_profits(tmp.path()).await.unwrap(); + assert_eq!(summary.recent.len(), 100); + assert_eq!(summary.total_sats, 110); + } + + #[tokio::test] + async fn test_get_networking_profits_empty() { + let tmp = TempDir::new().unwrap(); + let summary = get_networking_profits(tmp.path()).await.unwrap(); + assert_eq!(summary.total_sats, 0); + assert_eq!(summary.content_sales_sats, 0); + assert_eq!(summary.routing_fees_sats, 0); + } + + #[tokio::test] + async fn test_get_networking_profits_includes_ecash_receives() { + let tmp = TempDir::new().unwrap(); + + // Simulate receiving ecash tokens + ecash::receive_token(tmp.path(), "cashuSend_500_uuid1_1700000000") + .await + .unwrap(); + ecash::receive_token(tmp.path(), "cashuSend_300_uuid2_1700000001") + .await + .unwrap(); + + let summary = get_networking_profits(tmp.path()).await.unwrap(); + // ecash receives (800) should be reflected as content sales + assert_eq!(summary.content_sales_sats, 800); + assert_eq!(summary.total_sats, 800); + } + + #[tokio::test] + async fn test_get_networking_profits_uses_higher_of_tracked_or_ecash() { + let tmp = TempDir::new().unwrap(); + + // Record a larger tracked profit + record_content_sale(tmp.path(), 2000, "Big sale").await.unwrap(); + // Receive a smaller ecash amount + ecash::receive_token(tmp.path(), "cashuSend_100_uuid_170") + .await + .unwrap(); + + let summary = get_networking_profits(tmp.path()).await.unwrap(); + // Should use tracked (2000) since it's larger than ecash receives (100) + assert_eq!(summary.content_sales_sats, 2000); + assert_eq!(summary.total_sats, 2000); + } + + #[test] + fn test_profit_source_serialization() { + let entry = ProfitEntry { + source: ProfitSource::RoutingFee, + amount_sats: 42, + timestamp: "2025-01-01T00:00:00Z".to_string(), + description: "routing".to_string(), + }; + let json = serde_json::to_string(&entry).unwrap(); + assert!(json.contains("\"routing_fee\"")); + + let parsed: ProfitEntry = serde_json::from_str(&json).unwrap(); + assert!(matches!(parsed.source, ProfitSource::RoutingFee)); + } +} diff --git a/core/container/Cargo.toml b/core/container/Cargo.toml index 972bbfa0..69335448 100644 --- a/core/container/Cargo.toml +++ b/core/container/Cargo.toml @@ -8,7 +8,7 @@ serde = { version = "1.0", features = ["derive"] } serde_yaml = "0.9" serde_json = "1.0" tokio = { version = "1", features = ["full"] } -reqwest = { version = "0.11", features = ["json"] } +reqwest = { version = "0.11", default-features = false, features = ["json", "rustls-tls"] } thiserror = "1.0" anyhow = "1.0" async-trait = "0.1" diff --git a/core/container/src/dependency_resolver.rs b/core/container/src/dependency_resolver.rs index 0dd0d735..6d642023 100644 --- a/core/container/src/dependency_resolver.rs +++ b/core/container/src/dependency_resolver.rs @@ -34,9 +34,8 @@ impl DependencyResolver { let mut result = Vec::new(); self.resolve_recursive(app_id, &mut visited, &mut visiting, &mut result)?; - - // Reverse to get installation order (dependencies first) - result.reverse(); + + // Result is already in installation order (dependencies first) Ok(result) } diff --git a/core/security/Cargo.toml b/core/security/Cargo.toml index d9e6412e..613b2e51 100644 --- a/core/security/Cargo.toml +++ b/core/security/Cargo.toml @@ -12,8 +12,10 @@ log = "0.4" tracing = "0.1" uuid = { version = "1.0", features = ["v4"] } chrono = { version = "0.4", features = ["serde"] } +serde_json = "1.0" aes-gcm = "0.10" rand = "0.8" +hex = "0.4" [dev-dependencies] tempfile = "3" diff --git a/core/security/src/secrets_manager.rs b/core/security/src/secrets_manager.rs index 4c809289..ed5f7d54 100644 --- a/core/security/src/secrets_manager.rs +++ b/core/security/src/secrets_manager.rs @@ -4,7 +4,9 @@ use aes_gcm::aead::{Aead, KeyInit}; use aes_gcm::{Aes256Gcm, Nonce}; use anyhow::{Context, Result}; +use chrono::{DateTime, Utc}; use rand::RngCore; +use serde::{Deserialize, Serialize}; use std::path::PathBuf; use tokio::fs; use uuid::Uuid; @@ -12,6 +14,27 @@ use uuid::Uuid; /// Prefix to identify encrypted files (magic bytes) const ENCRYPTED_MAGIC: &[u8] = b"ARCHI_ENC1"; +/// Metadata for a stored secret (stored alongside the encrypted data). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SecretMetadata { + pub secret_id: String, + pub key: String, + pub app_id: String, + pub created_at: DateTime, + pub rotated_at: Option>, + pub rotation_count: u32, +} + +/// Info about a secret that may need rotation. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ExpiringSecret { + pub secret_id: String, + pub key: String, + pub app_id: String, + pub age_days: i64, + pub last_rotated: Option>, +} + pub struct SecretsManager { secrets_dir: PathBuf, cipher: Aes256Gcm, @@ -77,7 +100,7 @@ impl SecretsManager { pub async fn store_secret( &self, app_id: &str, - _key: &str, + key: &str, value: &str, ) -> Result { let secret_id = Uuid::new_v4().to_string(); @@ -95,6 +118,25 @@ impl SecretsManager { .await .context("Failed to write secret")?; + // Save metadata + let metadata = SecretMetadata { + secret_id: secret_id.clone(), + key: key.to_string(), + app_id: app_id.to_string(), + created_at: Utc::now(), + rotated_at: None, + rotation_count: 0, + }; + let meta_path = self + .secrets_dir + .join(app_id) + .join(format!("{}.meta.json", secret_id)); + let meta_json = serde_json::to_string(&metadata) + .context("Failed to serialize metadata")?; + fs::write(&meta_path, meta_json.as_bytes()) + .await + .context("Failed to write metadata")?; + // Set restrictive permissions #[cfg(unix)] { @@ -102,6 +144,9 @@ impl SecretsManager { let mut perms = fs::metadata(&secret_path).await?.permissions(); perms.set_mode(0o600); fs::set_permissions(&secret_path, perms).await?; + let mut meta_perms = fs::metadata(&meta_path).await?.permissions(); + meta_perms.set_mode(0o600); + fs::set_permissions(&meta_path, meta_perms).await?; } Ok(secret_id) @@ -164,16 +209,152 @@ impl SecretsManager { Ok(secrets) } - /// Delete a secret - pub async fn delete_secret(&self, app_id: &str, secret_id: &str) -> Result<()> { + /// Rotate a secret: generate a new random value, re-encrypt, update metadata. + /// Returns the new plaintext secret value. + pub async fn rotate_secret( + &self, + app_id: &str, + secret_id: &str, + ) -> Result { + // Generate a new random secret (32 bytes, hex-encoded = 64 chars) + let mut new_secret_bytes = [0u8; 32]; + rand::thread_rng().fill_bytes(&mut new_secret_bytes); + let new_value = hex::encode(new_secret_bytes); + let secret_path = self .secrets_dir .join(app_id) .join(format!("{}.secret", secret_id)); + anyhow::ensure!( + secret_path.exists(), + "Secret {} not found for app {}", + secret_id, + app_id + ); + + // Re-encrypt with new value + let encrypted = self + .encrypt(new_value.as_bytes()) + .context("Failed to encrypt rotated secret")?; + fs::write(&secret_path, &encrypted) + .await + .context("Failed to write rotated secret")?; + + // Update metadata + let meta_path = self + .secrets_dir + .join(app_id) + .join(format!("{}.meta.json", secret_id)); + if meta_path.exists() { + if let Ok(data) = fs::read_to_string(&meta_path).await { + if let Ok(mut metadata) = serde_json::from_str::(&data) { + metadata.rotated_at = Some(Utc::now()); + metadata.rotation_count += 1; + if let Ok(json) = serde_json::to_string(&metadata) { + let _ = fs::write(&meta_path, json.as_bytes()).await; + } + } + } + } + + Ok(new_value) + } + + /// List secrets older than `max_age_days` that may need rotation. + pub async fn list_expiring( + &self, + max_age_days: i64, + ) -> Result> { + let mut expiring = Vec::new(); + let now = Utc::now(); + + if !self.secrets_dir.exists() { + return Ok(expiring); + } + + let mut app_dirs = fs::read_dir(&self.secrets_dir).await?; + while let Some(app_entry) = app_dirs.next_entry().await? { + let app_path = app_entry.path(); + if !app_path.is_dir() { + continue; + } + let app_id = app_path + .file_name() + .and_then(|s| s.to_str()) + .unwrap_or("") + .to_string(); + + let mut entries = fs::read_dir(&app_path).await?; + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + let name = path.file_name().and_then(|s| s.to_str()).unwrap_or(""); + if !name.ends_with(".meta.json") { + continue; + } + + if let Ok(data) = fs::read_to_string(&path).await { + if let Ok(metadata) = serde_json::from_str::(&data) { + let reference_time = + metadata.rotated_at.unwrap_or(metadata.created_at); + let age = now.signed_duration_since(reference_time); + if age.num_days() >= max_age_days { + expiring.push(ExpiringSecret { + secret_id: metadata.secret_id, + key: metadata.key, + app_id: metadata.app_id, + age_days: age.num_days(), + last_rotated: metadata.rotated_at, + }); + } + } + } + } + } + + Ok(expiring) + } + + /// Read metadata for a specific secret. + pub async fn get_metadata( + &self, + app_id: &str, + secret_id: &str, + ) -> Result> { + let meta_path = self + .secrets_dir + .join(app_id) + .join(format!("{}.meta.json", secret_id)); + + if !meta_path.exists() { + return Ok(None); + } + + let data = fs::read_to_string(&meta_path) + .await + .context("Failed to read metadata")?; + let metadata: SecretMetadata = + serde_json::from_str(&data).context("Failed to parse metadata")?; + Ok(Some(metadata)) + } + + /// Delete a secret and its metadata + pub async fn delete_secret(&self, app_id: &str, secret_id: &str) -> Result<()> { + let secret_path = self + .secrets_dir + .join(app_id) + .join(format!("{}.secret", secret_id)); + let meta_path = self + .secrets_dir + .join(app_id) + .join(format!("{}.meta.json", secret_id)); + if secret_path.exists() { fs::remove_file(&secret_path).await?; } + if meta_path.exists() { + fs::remove_file(&meta_path).await?; + } Ok(()) } @@ -256,4 +437,113 @@ mod tests { // File must start with our magic prefix assert_eq!(&raw[..ENCRYPTED_MAGIC.len()], ENCRYPTED_MAGIC); } + + #[tokio::test] + async fn test_rotate_secret() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + let secret_id = mgr + .store_secret("test-app", "api-key", "original_value") + .await + .unwrap(); + + let original = mgr.read_secret("test-app", &secret_id).await.unwrap(); + assert_eq!(original, "original_value"); + + let new_value = mgr.rotate_secret("test-app", &secret_id).await.unwrap(); + assert_ne!(new_value, "original_value"); + assert_eq!(new_value.len(), 64); // 32 bytes hex-encoded + + let read_back = mgr.read_secret("test-app", &secret_id).await.unwrap(); + assert_eq!(read_back, new_value); + } + + #[tokio::test] + async fn test_rotate_updates_metadata() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + let secret_id = mgr + .store_secret("test-app", "db-pass", "initial") + .await + .unwrap(); + + let meta_before = mgr.get_metadata("test-app", &secret_id).await.unwrap().unwrap(); + assert_eq!(meta_before.rotation_count, 0); + assert!(meta_before.rotated_at.is_none()); + + mgr.rotate_secret("test-app", &secret_id).await.unwrap(); + + let meta_after = mgr.get_metadata("test-app", &secret_id).await.unwrap().unwrap(); + assert_eq!(meta_after.rotation_count, 1); + assert!(meta_after.rotated_at.is_some()); + } + + #[tokio::test] + async fn test_rotate_nonexistent_fails() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + let result = mgr.rotate_secret("test-app", "nonexistent").await; + assert!(result.is_err()); + } + + #[tokio::test] + async fn test_list_expiring_empty() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + let expiring = mgr.list_expiring(90).await.unwrap(); + assert!(expiring.is_empty()); + } + + #[tokio::test] + async fn test_list_expiring_fresh_secrets_not_listed() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + mgr.store_secret("test-app", "key1", "val1").await.unwrap(); + + // Fresh secret (0 days old) should not be listed for 90-day expiry + let expiring = mgr.list_expiring(90).await.unwrap(); + assert!(expiring.is_empty()); + + // But it should appear for 0-day threshold + let expiring = mgr.list_expiring(0).await.unwrap(); + assert_eq!(expiring.len(), 1); + assert_eq!(expiring[0].key, "key1"); + } + + #[tokio::test] + async fn test_metadata_stored_and_read() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + let secret_id = mgr + .store_secret("myapp", "connection-string", "postgres://...") + .await + .unwrap(); + + let meta = mgr.get_metadata("myapp", &secret_id).await.unwrap().unwrap(); + assert_eq!(meta.key, "connection-string"); + assert_eq!(meta.app_id, "myapp"); + assert_eq!(meta.rotation_count, 0); + } + + #[tokio::test] + async fn test_delete_removes_metadata() { + let dir = tempfile::tempdir().unwrap(); + let mgr = SecretsManager::new(dir.path().to_path_buf(), test_key()).unwrap(); + + let secret_id = mgr + .store_secret("test-app", "key", "val") + .await + .unwrap(); + + mgr.delete_secret("test-app", &secret_id).await.unwrap(); + + let meta = mgr.get_metadata("test-app", &secret_id).await.unwrap(); + assert!(meta.is_none()); + } } diff --git a/core/startos/.sqlx/query-1ce5254f27de971fd87f5ab66d300f2b22433c86617a0dbf796bf2170186dd2e.json b/core/startos/.sqlx/query-1ce5254f27de971fd87f5ab66d300f2b22433c86617a0dbf796bf2170186dd2e.json deleted file mode 100644 index d36100fe..00000000 --- a/core/startos/.sqlx/query-1ce5254f27de971fd87f5ab66d300f2b22433c86617a0dbf796bf2170186dd2e.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Bytea" - ] - }, - "nullable": [] - }, - "hash": "1ce5254f27de971fd87f5ab66d300f2b22433c86617a0dbf796bf2170186dd2e" -} diff --git a/core/startos/.sqlx/query-21471490cdc3adb206274cc68e1ea745ffa5da4479478c1fd2158a45324b1930.json b/core/startos/.sqlx/query-21471490cdc3adb206274cc68e1ea745ffa5da4479478c1fd2158a45324b1930.json deleted file mode 100644 index e0b1d7cf..00000000 --- a/core/startos/.sqlx/query-21471490cdc3adb206274cc68e1ea745ffa5da4479478c1fd2158a45324b1930.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM ssh_keys WHERE fingerprint = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [] - }, - "hash": "21471490cdc3adb206274cc68e1ea745ffa5da4479478c1fd2158a45324b1930" -} diff --git a/core/startos/.sqlx/query-28ea34bbde836e0618c5fc9bb7c36e463c20c841a7d6a0eb15be0f24f4a928ec.json b/core/startos/.sqlx/query-28ea34bbde836e0618c5fc9bb7c36e463c20c841a7d6a0eb15be0f24f4a928ec.json deleted file mode 100644 index e234a72a..00000000 --- a/core/startos/.sqlx/query-28ea34bbde836e0618c5fc9bb7c36e463c20c841a7d6a0eb15be0f24f4a928ec.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT hostname, path, username, password FROM cifs_shares WHERE id = $1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "hostname", - "type_info": "Text" - }, - { - "ordinal": 1, - "name": "path", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "username", - "type_info": "Text" - }, - { - "ordinal": 3, - "name": "password", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Int4" - ] - }, - "nullable": [ - false, - false, - false, - true - ] - }, - "hash": "28ea34bbde836e0618c5fc9bb7c36e463c20c841a7d6a0eb15be0f24f4a928ec" -} diff --git a/core/startos/.sqlx/query-350ab82048fb4a049042e4fdbe1b8c606ca400e43e31b9a05d2937217e0f6962.json b/core/startos/.sqlx/query-350ab82048fb4a049042e4fdbe1b8c606ca400e43e31b9a05d2937217e0f6962.json deleted file mode 100644 index c451ce9f..00000000 --- a/core/startos/.sqlx/query-350ab82048fb4a049042e4fdbe1b8c606ca400e43e31b9a05d2937217e0f6962.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM tor WHERE package = $1 AND interface = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "350ab82048fb4a049042e4fdbe1b8c606ca400e43e31b9a05d2937217e0f6962" -} diff --git a/core/startos/.sqlx/query-4099028a5c0de578255bf54a67cef6cb0f1e9a4e158260700f1639dd4b438997.json b/core/startos/.sqlx/query-4099028a5c0de578255bf54a67cef6cb0f1e9a4e158260700f1639dd4b438997.json deleted file mode 100644 index 761af064..00000000 --- a/core/startos/.sqlx/query-4099028a5c0de578255bf54a67cef6cb0f1e9a4e158260700f1639dd4b438997.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT * FROM ssh_keys WHERE fingerprint = $1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "fingerprint", - "type_info": "Text" - }, - { - "ordinal": 1, - "name": "openssh_pubkey", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "created_at", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - false, - false, - false - ] - }, - "hash": "4099028a5c0de578255bf54a67cef6cb0f1e9a4e158260700f1639dd4b438997" -} diff --git a/core/startos/.sqlx/query-4691e3a2ce80b59009ac17124f54f925f61dc5ea371903e62cdffa5d7b67ca96.json b/core/startos/.sqlx/query-4691e3a2ce80b59009ac17124f54f925f61dc5ea371903e62cdffa5d7b67ca96.json deleted file mode 100644 index 1f7edd1c..00000000 --- a/core/startos/.sqlx/query-4691e3a2ce80b59009ac17124f54f925f61dc5ea371903e62cdffa5d7b67ca96.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT * FROM session WHERE logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Text" - }, - { - "ordinal": 1, - "name": "logged_in", - "type_info": "Timestamp" - }, - { - "ordinal": 2, - "name": "logged_out", - "type_info": "Timestamp" - }, - { - "ordinal": 3, - "name": "last_active", - "type_info": "Timestamp" - }, - { - "ordinal": 4, - "name": "user_agent", - "type_info": "Text" - }, - { - "ordinal": 5, - "name": "metadata", - "type_info": "Text" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false, - false, - true, - false, - true, - false - ] - }, - "hash": "4691e3a2ce80b59009ac17124f54f925f61dc5ea371903e62cdffa5d7b67ca96" -} diff --git a/core/startos/.sqlx/query-4bcfbefb1eb3181343871a1cd7fc3afb81c2be5c681cfa8b4be0ce70610e9c3a.json b/core/startos/.sqlx/query-4bcfbefb1eb3181343871a1cd7fc3afb81c2be5c681cfa8b4be0ce70610e9c3a.json deleted file mode 100644 index 2157198e..00000000 --- a/core/startos/.sqlx/query-4bcfbefb1eb3181343871a1cd7fc3afb81c2be5c681cfa8b4be0ce70610e9c3a.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE session SET logged_out = CURRENT_TIMESTAMP WHERE id = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [] - }, - "hash": "4bcfbefb1eb3181343871a1cd7fc3afb81c2be5c681cfa8b4be0ce70610e9c3a" -} diff --git a/core/startos/.sqlx/query-629be61c3c341c131ddbbff0293a83dbc6afd07cae69d246987f62cf0cc35c2a.json b/core/startos/.sqlx/query-629be61c3c341c131ddbbff0293a83dbc6afd07cae69d246987f62cf0cc35c2a.json deleted file mode 100644 index 764cff84..00000000 --- a/core/startos/.sqlx/query-629be61c3c341c131ddbbff0293a83dbc6afd07cae69d246987f62cf0cc35c2a.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT password FROM account", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "password", - "type_info": "Text" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "629be61c3c341c131ddbbff0293a83dbc6afd07cae69d246987f62cf0cc35c2a" -} diff --git a/core/startos/.sqlx/query-687688055e63d27123cdc89a5bbbd8361776290a9411d527eaf1fdb40bef399d.json b/core/startos/.sqlx/query-687688055e63d27123cdc89a5bbbd8361776290a9411d527eaf1fdb40bef399d.json deleted file mode 100644 index 2e8a9ee0..00000000 --- a/core/startos/.sqlx/query-687688055e63d27123cdc89a5bbbd8361776290a9411d527eaf1fdb40bef399d.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT key FROM tor WHERE package = $1 AND interface = $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "key", - "type_info": "Bytea" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - false - ] - }, - "hash": "687688055e63d27123cdc89a5bbbd8361776290a9411d527eaf1fdb40bef399d" -} diff --git a/core/startos/.sqlx/query-6d35ccf780fb2bb62586dd1d3df9c1550a41ee580dad3f49d35cb843ebef10ca.json b/core/startos/.sqlx/query-6d35ccf780fb2bb62586dd1d3df9c1550a41ee580dad3f49d35cb843ebef10ca.json deleted file mode 100644 index 3f859bd1..00000000 --- a/core/startos/.sqlx/query-6d35ccf780fb2bb62586dd1d3df9c1550a41ee580dad3f49d35cb843ebef10ca.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE session SET last_active = CURRENT_TIMESTAMP WHERE id = $1 AND logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [] - }, - "hash": "6d35ccf780fb2bb62586dd1d3df9c1550a41ee580dad3f49d35cb843ebef10ca" -} diff --git a/core/startos/.sqlx/query-770c1017734720453dc87b58c385b987c5af5807151ff71a59000014586752e0.json b/core/startos/.sqlx/query-770c1017734720453dc87b58c385b987c5af5807151ff71a59000014586752e0.json deleted file mode 100644 index cf3591e0..00000000 --- a/core/startos/.sqlx/query-770c1017734720453dc87b58c385b987c5af5807151ff71a59000014586752e0.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO UPDATE SET package = EXCLUDED.package RETURNING key", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "key", - "type_info": "Bytea" - } - ], - "parameters": { - "Left": [ - "Text", - "Text", - "Bytea" - ] - }, - "nullable": [ - false - ] - }, - "hash": "770c1017734720453dc87b58c385b987c5af5807151ff71a59000014586752e0" -} diff --git a/core/startos/.sqlx/query-7b64f032d507e8ffe37c41f4c7ad514a66c421a11ab04c26d89a7aa8f6b67210.json b/core/startos/.sqlx/query-7b64f032d507e8ffe37c41f4c7ad514a66c421a11ab04c26d89a7aa8f6b67210.json deleted file mode 100644 index 53fc6f06..00000000 --- a/core/startos/.sqlx/query-7b64f032d507e8ffe37c41f4c7ad514a66c421a11ab04c26d89a7aa8f6b67210.json +++ /dev/null @@ -1,65 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT id, package_id, created_at, code, level, title, message, data FROM notifications WHERE id < $1 ORDER BY id DESC LIMIT $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "package_id", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "created_at", - "type_info": "Timestamp" - }, - { - "ordinal": 3, - "name": "code", - "type_info": "Int4" - }, - { - "ordinal": 4, - "name": "level", - "type_info": "Text" - }, - { - "ordinal": 5, - "name": "title", - "type_info": "Text" - }, - { - "ordinal": 6, - "name": "message", - "type_info": "Text" - }, - { - "ordinal": 7, - "name": "data", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Int4", - "Int8" - ] - }, - "nullable": [ - false, - true, - false, - false, - false, - false, - false, - true - ] - }, - "hash": "7b64f032d507e8ffe37c41f4c7ad514a66c421a11ab04c26d89a7aa8f6b67210" -} diff --git a/core/startos/.sqlx/query-7c7a3549c997eb75bf964ea65fbb98a73045adf618696cd838d79203ef5383fb.json b/core/startos/.sqlx/query-7c7a3549c997eb75bf964ea65fbb98a73045adf618696cd838d79203ef5383fb.json deleted file mode 100644 index 245a838d..00000000 --- a/core/startos/.sqlx/query-7c7a3549c997eb75bf964ea65fbb98a73045adf618696cd838d79203ef5383fb.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n INSERT INTO account (\n id,\n server_id,\n hostname,\n password,\n network_key,\n root_ca_key_pem,\n root_ca_cert_pem\n ) VALUES (\n 0, $1, $2, $3, $4, $5, $6\n ) ON CONFLICT (id) DO UPDATE SET\n server_id = EXCLUDED.server_id,\n hostname = EXCLUDED.hostname,\n password = EXCLUDED.password,\n network_key = EXCLUDED.network_key,\n root_ca_key_pem = EXCLUDED.root_ca_key_pem,\n root_ca_cert_pem = EXCLUDED.root_ca_cert_pem\n ", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Text", - "Bytea", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "7c7a3549c997eb75bf964ea65fbb98a73045adf618696cd838d79203ef5383fb" -} diff --git a/core/startos/.sqlx/query-7e0649d839927e57fa03ee51a2c9f96a8bdb0fc97ee8a3c6df1069e1e2b98576.json b/core/startos/.sqlx/query-7e0649d839927e57fa03ee51a2c9f96a8bdb0fc97ee8a3c6df1069e1e2b98576.json deleted file mode 100644 index e3ce7957..00000000 --- a/core/startos/.sqlx/query-7e0649d839927e57fa03ee51a2c9f96a8bdb0fc97ee8a3c6df1069e1e2b98576.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM tor WHERE package = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [] - }, - "hash": "7e0649d839927e57fa03ee51a2c9f96a8bdb0fc97ee8a3c6df1069e1e2b98576" -} diff --git a/core/startos/.sqlx/query-8951b9126fbf60dbb5997241e11e3526b70bccf3e407327917294a993bc17ed5.json b/core/startos/.sqlx/query-8951b9126fbf60dbb5997241e11e3526b70bccf3e407327917294a993bc17ed5.json deleted file mode 100644 index e39aebf6..00000000 --- a/core/startos/.sqlx/query-8951b9126fbf60dbb5997241e11e3526b70bccf3e407327917294a993bc17ed5.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO tor (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Bytea" - ] - }, - "nullable": [] - }, - "hash": "8951b9126fbf60dbb5997241e11e3526b70bccf3e407327917294a993bc17ed5" -} diff --git a/core/startos/.sqlx/query-94d471bb374b4965c6cbedf8c17bbf6bea226d38efaf6559923c79a36d5ca08c.json b/core/startos/.sqlx/query-94d471bb374b4965c6cbedf8c17bbf6bea226d38efaf6559923c79a36d5ca08c.json deleted file mode 100644 index e7fe8d38..00000000 --- a/core/startos/.sqlx/query-94d471bb374b4965c6cbedf8c17bbf6bea226d38efaf6559923c79a36d5ca08c.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT id, package_id, created_at, code, level, title, message, data FROM notifications ORDER BY id DESC LIMIT $1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "package_id", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "created_at", - "type_info": "Timestamp" - }, - { - "ordinal": 3, - "name": "code", - "type_info": "Int4" - }, - { - "ordinal": 4, - "name": "level", - "type_info": "Text" - }, - { - "ordinal": 5, - "name": "title", - "type_info": "Text" - }, - { - "ordinal": 6, - "name": "message", - "type_info": "Text" - }, - { - "ordinal": 7, - "name": "data", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Int8" - ] - }, - "nullable": [ - false, - true, - false, - false, - false, - false, - false, - true - ] - }, - "hash": "94d471bb374b4965c6cbedf8c17bbf6bea226d38efaf6559923c79a36d5ca08c" -} diff --git a/core/startos/.sqlx/query-95c4ab4c645f3302568c6ff13d85ab58252362694cf0f56999bf60194d20583a.json b/core/startos/.sqlx/query-95c4ab4c645f3302568c6ff13d85ab58252362694cf0f56999bf60194d20583a.json deleted file mode 100644 index aadc0fc3..00000000 --- a/core/startos/.sqlx/query-95c4ab4c645f3302568c6ff13d85ab58252362694cf0f56999bf60194d20583a.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT id, hostname, path, username, password FROM cifs_shares", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "hostname", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "path", - "type_info": "Text" - }, - { - "ordinal": 3, - "name": "username", - "type_info": "Text" - }, - { - "ordinal": 4, - "name": "password", - "type_info": "Text" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false, - false, - false, - false, - true - ] - }, - "hash": "95c4ab4c645f3302568c6ff13d85ab58252362694cf0f56999bf60194d20583a" -} diff --git a/core/startos/.sqlx/query-a60d6e66719325b08dc4ecfacaf337527233c84eee758ac9be967906e5841d27.json b/core/startos/.sqlx/query-a60d6e66719325b08dc4ecfacaf337527233c84eee758ac9be967906e5841d27.json deleted file mode 100644 index c56a9ebd..00000000 --- a/core/startos/.sqlx/query-a60d6e66719325b08dc4ecfacaf337527233c84eee758ac9be967906e5841d27.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM cifs_shares WHERE id = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Int4" - ] - }, - "nullable": [] - }, - "hash": "a60d6e66719325b08dc4ecfacaf337527233c84eee758ac9be967906e5841d27" -} diff --git a/core/startos/.sqlx/query-a6b0c8909a3a5d6d9156aebfb359424e6b5a1d1402e028219e21726f1ebd282e.json b/core/startos/.sqlx/query-a6b0c8909a3a5d6d9156aebfb359424e6b5a1d1402e028219e21726f1ebd282e.json deleted file mode 100644 index 86bd9250..00000000 --- a/core/startos/.sqlx/query-a6b0c8909a3a5d6d9156aebfb359424e6b5a1d1402e028219e21726f1ebd282e.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT fingerprint, openssh_pubkey, created_at FROM ssh_keys", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "fingerprint", - "type_info": "Text" - }, - { - "ordinal": 1, - "name": "openssh_pubkey", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "created_at", - "type_info": "Text" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false, - false, - false - ] - }, - "hash": "a6b0c8909a3a5d6d9156aebfb359424e6b5a1d1402e028219e21726f1ebd282e" -} diff --git a/core/startos/.sqlx/query-b1147beaaabbed89f2ab8c1e13ec4393a9a8fde2833cf096af766a979d94dee6.json b/core/startos/.sqlx/query-b1147beaaabbed89f2ab8c1e13ec4393a9a8fde2833cf096af766a979d94dee6.json deleted file mode 100644 index c8ff8427..00000000 --- a/core/startos/.sqlx/query-b1147beaaabbed89f2ab8c1e13ec4393a9a8fde2833cf096af766a979d94dee6.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE cifs_shares SET hostname = $1, path = $2, username = $3, password = $4 WHERE id = $5", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Text", - "Text", - "Int4" - ] - }, - "nullable": [] - }, - "hash": "b1147beaaabbed89f2ab8c1e13ec4393a9a8fde2833cf096af766a979d94dee6" -} diff --git a/core/startos/.sqlx/query-b203820ee1c553a4b246eac74b79bd10d5717b2a0ddecf22330b7d531aac7c5d.json b/core/startos/.sqlx/query-b203820ee1c553a4b246eac74b79bd10d5717b2a0ddecf22330b7d531aac7c5d.json deleted file mode 100644 index b76542db..00000000 --- a/core/startos/.sqlx/query-b203820ee1c553a4b246eac74b79bd10d5717b2a0ddecf22330b7d531aac7c5d.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM network_keys WHERE package = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [] - }, - "hash": "b203820ee1c553a4b246eac74b79bd10d5717b2a0ddecf22330b7d531aac7c5d" -} diff --git a/core/startos/.sqlx/query-b81592b3a74940ab56d41537484090d45cfa4c85168a587b1a41dc5393cccea1.json b/core/startos/.sqlx/query-b81592b3a74940ab56d41537484090d45cfa4c85168a587b1a41dc5393cccea1.json deleted file mode 100644 index e2e8a162..00000000 --- a/core/startos/.sqlx/query-b81592b3a74940ab56d41537484090d45cfa4c85168a587b1a41dc5393cccea1.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE account SET tor_key = NULL, network_key = gen_random_bytes(32)", - "describe": { - "columns": [], - "parameters": { - "Left": [] - }, - "nullable": [] - }, - "hash": "b81592b3a74940ab56d41537484090d45cfa4c85168a587b1a41dc5393cccea1" -} diff --git a/core/startos/.sqlx/query-d5117054072476377f3c4f040ea429d4c9b2cf534e76f35c80a2bf60e8599cca.json b/core/startos/.sqlx/query-d5117054072476377f3c4f040ea429d4c9b2cf534e76f35c80a2bf60e8599cca.json deleted file mode 100644 index b77ba7ce..00000000 --- a/core/startos/.sqlx/query-d5117054072476377f3c4f040ea429d4c9b2cf534e76f35c80a2bf60e8599cca.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT openssh_pubkey FROM ssh_keys", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "openssh_pubkey", - "type_info": "Text" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "d5117054072476377f3c4f040ea429d4c9b2cf534e76f35c80a2bf60e8599cca" -} diff --git a/core/startos/.sqlx/query-da71f94b29798d1738d2b10b9a721ea72db8cfb362e7181c8226d9297507c62b.json b/core/startos/.sqlx/query-da71f94b29798d1738d2b10b9a721ea72db8cfb362e7181c8226d9297507c62b.json deleted file mode 100644 index 5c5c89c2..00000000 --- a/core/startos/.sqlx/query-da71f94b29798d1738d2b10b9a721ea72db8cfb362e7181c8226d9297507c62b.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO notifications (package_id, code, level, title, message, data) VALUES ($1, $2, $3, $4, $5, $6)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Int4", - "Text", - "Text", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "da71f94b29798d1738d2b10b9a721ea72db8cfb362e7181c8226d9297507c62b" -} diff --git a/core/startos/.sqlx/query-dfc23b7e966c3853284753a7e934351ba0cae3825988b3e0ecd3b6781bcff524.json b/core/startos/.sqlx/query-dfc23b7e966c3853284753a7e934351ba0cae3825988b3e0ecd3b6781bcff524.json deleted file mode 100644 index 2fc8ad1b..00000000 --- a/core/startos/.sqlx/query-dfc23b7e966c3853284753a7e934351ba0cae3825988b3e0ecd3b6781bcff524.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM network_keys WHERE package = $1 AND interface = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "dfc23b7e966c3853284753a7e934351ba0cae3825988b3e0ecd3b6781bcff524" -} diff --git a/core/startos/.sqlx/query-e185203cf84e43b801dfb23b4159e34aeaef1154dcd3d6811ab504915497ccf7.json b/core/startos/.sqlx/query-e185203cf84e43b801dfb23b4159e34aeaef1154dcd3d6811ab504915497ccf7.json deleted file mode 100644 index a4dc187c..00000000 --- a/core/startos/.sqlx/query-e185203cf84e43b801dfb23b4159e34aeaef1154dcd3d6811ab504915497ccf7.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM notifications WHERE id = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Int4" - ] - }, - "nullable": [] - }, - "hash": "e185203cf84e43b801dfb23b4159e34aeaef1154dcd3d6811ab504915497ccf7" -} diff --git a/core/startos/.sqlx/query-e545696735f202f9d13cf22a561f3ff3f9aed7f90027a9ba97634bcb47d772f0.json b/core/startos/.sqlx/query-e545696735f202f9d13cf22a561f3ff3f9aed7f90027a9ba97634bcb47d772f0.json deleted file mode 100644 index 97a4ec95..00000000 --- a/core/startos/.sqlx/query-e545696735f202f9d13cf22a561f3ff3f9aed7f90027a9ba97634bcb47d772f0.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT tor_key FROM account WHERE id = 0", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "tor_key", - "type_info": "Bytea" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - true - ] - }, - "hash": "e545696735f202f9d13cf22a561f3ff3f9aed7f90027a9ba97634bcb47d772f0" -} diff --git a/core/startos/.sqlx/query-e5843c5b0e7819b29aa1abf2266799bd4f82e761837b526a0972c3d4439a264d.json b/core/startos/.sqlx/query-e5843c5b0e7819b29aa1abf2266799bd4f82e761837b526a0972c3d4439a264d.json deleted file mode 100644 index b2aa0437..00000000 --- a/core/startos/.sqlx/query-e5843c5b0e7819b29aa1abf2266799bd4f82e761837b526a0972c3d4439a264d.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO session (id, user_agent, metadata) VALUES ($1, $2, $3)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "e5843c5b0e7819b29aa1abf2266799bd4f82e761837b526a0972c3d4439a264d" -} diff --git a/core/startos/.sqlx/query-e95322a8e2ae3b93f1e974b24c0b81803f1e9ec9e8ebbf15cafddfc1c5a028ed.json b/core/startos/.sqlx/query-e95322a8e2ae3b93f1e974b24c0b81803f1e9ec9e8ebbf15cafddfc1c5a028ed.json deleted file mode 100644 index fd5a467e..00000000 --- a/core/startos/.sqlx/query-e95322a8e2ae3b93f1e974b24c0b81803f1e9ec9e8ebbf15cafddfc1c5a028ed.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT\n network_keys.package,\n network_keys.interface,\n network_keys.key,\n tor.key AS \"tor_key?\"\n FROM\n network_keys\n LEFT JOIN\n tor\n ON\n network_keys.package = tor.package\n AND\n network_keys.interface = tor.interface\n WHERE\n network_keys.package = $1\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "package", - "type_info": "Text" - }, - { - "ordinal": 1, - "name": "interface", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "key", - "type_info": "Bytea" - }, - { - "ordinal": 3, - "name": "tor_key?", - "type_info": "Bytea" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - false, - false, - false, - false - ] - }, - "hash": "e95322a8e2ae3b93f1e974b24c0b81803f1e9ec9e8ebbf15cafddfc1c5a028ed" -} diff --git a/core/startos/.sqlx/query-eb750adaa305bdbf3c5b70aaf59139c7b7569602adb58f2d6b3a94da4f167b0a.json b/core/startos/.sqlx/query-eb750adaa305bdbf3c5b70aaf59139c7b7569602adb58f2d6b3a94da4f167b0a.json deleted file mode 100644 index fb8a7c1e..00000000 --- a/core/startos/.sqlx/query-eb750adaa305bdbf3c5b70aaf59139c7b7569602adb58f2d6b3a94da4f167b0a.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM notifications WHERE id < $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Int4" - ] - }, - "nullable": [] - }, - "hash": "eb750adaa305bdbf3c5b70aaf59139c7b7569602adb58f2d6b3a94da4f167b0a" -} diff --git a/core/startos/.sqlx/query-ecc765d8205c0876956f95f76944ac6a5f34dd820c4073b7728c7067aab9fded.json b/core/startos/.sqlx/query-ecc765d8205c0876956f95f76944ac6a5f34dd820c4073b7728c7067aab9fded.json deleted file mode 100644 index 27c9752b..00000000 --- a/core/startos/.sqlx/query-ecc765d8205c0876956f95f76944ac6a5f34dd820c4073b7728c7067aab9fded.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO cifs_shares (hostname, path, username, password) VALUES ($1, $2, $3, $4) RETURNING id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Int4" - } - ], - "parameters": { - "Left": [ - "Text", - "Text", - "Text", - "Text" - ] - }, - "nullable": [ - false - ] - }, - "hash": "ecc765d8205c0876956f95f76944ac6a5f34dd820c4073b7728c7067aab9fded" -} diff --git a/core/startos/.sqlx/query-f6d1c5ef0f9d9577bea8382318967b9deb46da75788c7fe6082b43821c22d556.json b/core/startos/.sqlx/query-f6d1c5ef0f9d9577bea8382318967b9deb46da75788c7fe6082b43821c22d556.json deleted file mode 100644 index 6ed9898f..00000000 --- a/core/startos/.sqlx/query-f6d1c5ef0f9d9577bea8382318967b9deb46da75788c7fe6082b43821c22d556.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO ssh_keys (fingerprint, openssh_pubkey, created_at) VALUES ($1, $2, $3)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "f6d1c5ef0f9d9577bea8382318967b9deb46da75788c7fe6082b43821c22d556" -} diff --git a/core/startos/.sqlx/query-f7d2dae84613bcef330f7403352cc96547f3f6dbec11bf2eadfaf53ad8ab51b5.json b/core/startos/.sqlx/query-f7d2dae84613bcef330f7403352cc96547f3f6dbec11bf2eadfaf53ad8ab51b5.json deleted file mode 100644 index f48ccb07..00000000 --- a/core/startos/.sqlx/query-f7d2dae84613bcef330f7403352cc96547f3f6dbec11bf2eadfaf53ad8ab51b5.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT network_key FROM account WHERE id = 0", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "network_key", - "type_info": "Bytea" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "f7d2dae84613bcef330f7403352cc96547f3f6dbec11bf2eadfaf53ad8ab51b5" -} diff --git a/core/startos/.sqlx/query-fe6e4f09f3028e5b6b6259e86cbad285680ce157aae9d7837ac020c8b2945e7f.json b/core/startos/.sqlx/query-fe6e4f09f3028e5b6b6259e86cbad285680ce157aae9d7837ac020c8b2945e7f.json deleted file mode 100644 index 6ef1d502..00000000 --- a/core/startos/.sqlx/query-fe6e4f09f3028e5b6b6259e86cbad285680ce157aae9d7837ac020c8b2945e7f.json +++ /dev/null @@ -1,62 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT * FROM account WHERE id = 0", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "password", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "tor_key", - "type_info": "Bytea" - }, - { - "ordinal": 3, - "name": "server_id", - "type_info": "Text" - }, - { - "ordinal": 4, - "name": "hostname", - "type_info": "Text" - }, - { - "ordinal": 5, - "name": "network_key", - "type_info": "Bytea" - }, - { - "ordinal": 6, - "name": "root_ca_key_pem", - "type_info": "Text" - }, - { - "ordinal": 7, - "name": "root_ca_cert_pem", - "type_info": "Text" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false, - false, - true, - true, - true, - false, - false, - false - ] - }, - "hash": "fe6e4f09f3028e5b6b6259e86cbad285680ce157aae9d7837ac020c8b2945e7f" -} diff --git a/core/startos/Cargo.toml b/core/startos/Cargo.toml deleted file mode 100644 index fab6c22e..00000000 --- a/core/startos/Cargo.toml +++ /dev/null @@ -1,169 +0,0 @@ -[package] -authors = ["Aiden McClelland "] -description = "The core of StartOS" -documentation = "https://docs.rs/start-os" -edition = "2021" -keywords = [ - "self-hosted", - "raspberry-pi", - "privacy", - "bitcoin", - "full-node", - "lightning", -] -name = "start-os" -readme = "README.md" -repository = "https://github.com/Start9Labs/start-os" -version = "0.3.5-rev.1" -license = "MIT" - -[lib] -name = "startos" -path = "src/lib.rs" - -[[bin]] -name = "startbox" -path = "src/main.rs" - -[features] -avahi = ["avahi-sys"] -avahi-alias = ["avahi"] -cli = [] -daemon = [] -default = ["cli", "sdk", "daemon", "js-engine"] -dev = [] -docker = [] -sdk = [] -unstable = ["console-subscriber", "tokio/tracing"] - -[dependencies] -aes = { version = "0.7.5", features = ["ctr"] } -async-compression = { version = "0.4.4", features = [ - "gzip", - "brotli", - "tokio", -] } -async-stream = "0.3.5" -async-trait = "0.1.74" -avahi-sys = { git = "https://github.com/Start9Labs/avahi-sys", version = "0.10.0", branch = "feature/dynamic-linking", features = [ - "dynamic", -], optional = true } -base32 = "0.4.0" -base64 = "0.21.4" -base64ct = "1.6.0" -basic-cookies = "0.1.4" -bytes = "1" -chrono = { version = "0.4.31", features = ["serde"] } -clap = "3.2.25" -color-eyre = "0.6.2" -console = "0.15.7" -console-subscriber = { version = "0.2", optional = true } -cookie = "0.18.0" -cookie_store = "0.20.0" -current_platform = "0.2.0" -digest = "0.10.7" -divrem = "1.0.0" -ed25519 = { version = "2.2.3", features = ["pkcs8", "pem", "alloc"] } -ed25519-dalek = { version = "2.0.0", features = [ - "serde", - "zeroize", - "rand_core", - "digest", -] } -ed25519-dalek-v1 = { package = "ed25519-dalek", version = "1" } -container-init = { path = "../container-init" } -emver = { git = "https://github.com/Start9Labs/emver-rs.git", rev = "61cf0bc96711b4d6f3f30df8efef025e0cc02bad", features = [ - "serde", -] } -fd-lock-rs = "0.1.4" -futures = "0.3.28" -gpt = "3.1.0" -helpers = { path = "../helpers" } -hex = "0.4.3" -hmac = "0.12.1" -http = "0.2.9" -hyper = { version = "0.14.27", features = ["full"] } -hyper-ws-listener = "0.3.0" -imbl = "2.0.2" -imbl-value = { git = "https://github.com/Start9Labs/imbl-value.git" } -include_dir = "0.7.3" -indexmap = { version = "2.0.2", features = ["serde"] } -indicatif = { version = "0.17.7", features = ["tokio"] } -ipnet = { version = "2.8.0", features = ["serde"] } -iprange = { version = "0.6.7", features = ["serde"] } -isocountry = "0.3.2" -itertools = "0.11.0" -jaq-core = "0.10.1" -jaq-std = "0.10.0" -josekit = "0.8.4" -js-engine = { path = '../js-engine', optional = true } -jsonpath_lib = { git = "https://github.com/Start9Labs/jsonpath.git" } -lazy_static = "1.4.0" -libc = "0.2.149" -log = "0.4.20" -mbrman = "0.5.2" -models = { version = "*", path = "../models" } -new_mime_guess = "4" -nix = { version = "0.27.1", features = ["user", "process", "signal", "fs"] } -nom = "7.1.3" -num = "0.4.1" -num_enum = "0.7.0" -openssh-keys = "0.6.2" -openssl = { version = "0.10.57", features = ["vendored"] } -p256 = { version = "0.13.2", features = ["pem"] } -patch-db = { version = "*", path = "../../patch-db/patch-db", features = [ - "trace", -] } -pbkdf2 = "0.12.2" -pin-project = "1.1.3" -pkcs8 = { version = "0.10.2", features = ["std"] } -prettytable-rs = "0.10.0" -proptest = "1.3.1" -proptest-derive = "0.4.0" -rand = { version = "0.8.5", features = ["std"] } -regex = "1.10.2" -reqwest = { version = "0.11.22", features = ["stream", "json", "socks"] } -reqwest_cookie_store = "0.6.0" -rpassword = "7.2.0" -rpc-toolkit = "0.2.2" -rust-argon2 = "2.0.0" -scopeguard = "1.1" # because avahi-sys fucks your shit up -semver = { version = "1.0.20", features = ["serde"] } -serde = { version = "1.0", features = ["derive", "rc"] } -serde_cbor = { package = "ciborium", version = "0.2.1" } -serde_json = "1.0" -serde_toml = { package = "toml", version = "0.8.2" } -serde_with = { version = "3.4.0", features = ["macros", "json"] } -serde_yaml = "0.9.25" -sha2 = "0.10.2" -simple-logging = "2.0.2" -sqlx = { version = "0.7.2", features = [ - "chrono", - "runtime-tokio-rustls", - "postgres", -] } -sscanf = "0.4.1" -ssh-key = { version = "0.6.2", features = ["ed25519"] } -stderrlog = "0.5.4" -tar = "0.4.40" -thiserror = "1.0.49" -tokio = { version = "1", features = ["full"] } -tokio-rustls = "0.24.1" -tokio-socks = "0.5.1" -tokio-stream = { version = "0.1.14", features = ["io-util", "sync", "net"] } -tokio-tar = { git = "https://github.com/dr-bonez/tokio-tar.git" } -tokio-tungstenite = { version = "0.20.1", features = ["native-tls"] } -tokio-util = { version = "0.7.9", features = ["io"] } -torut = "0.2.1" -tracing = "0.1.39" -tracing-error = "0.2.0" -tracing-futures = "0.2.5" -tracing-journald = "0.3.0" -tracing-subscriber = { version = "0.3.17", features = ["env-filter"] } -trust-dns-server = "0.23.1" -typed-builder = "0.17.0" -url = { version = "2.4.1", features = ["serde"] } -urlencoding = "2.1.3" -uuid = { version = "1.4.1", features = ["v4"] } -zeroize = "1.6.0" - diff --git a/core/startos/deny.toml b/core/startos/deny.toml deleted file mode 100644 index 7b4924cd..00000000 --- a/core/startos/deny.toml +++ /dev/null @@ -1,22 +0,0 @@ -[licenses] -unlicensed = "warn" -allow-osi-fsf-free = "neither" -copyleft = "deny" -confidence-threshold = 0.93 -allow = [ - "Apache-2.0", - "Apache-2.0 WITH LLVM-exception", - "MIT", - "ISC", - "MPL-2.0", - "CC0-1.0", - "BSD-2-Clause", - "BSD-3-Clause", - "LGPL-3.0", - "OpenSSL", -] - -clarify = [ - { name = "webpki", expression = "ISC", license-files = [ { path = "LICENSE", hash = 0x001c7e6c } ] }, - { name = "ring", expression = "OpenSSL", license-files = [ { path = "LICENSE", hash = 0xbd0eed23 } ] }, -] diff --git a/core/startos/migrations/20210629193146_Init.sql b/core/startos/migrations/20210629193146_Init.sql deleted file mode 100644 index af318813..00000000 --- a/core/startos/migrations/20210629193146_Init.sql +++ /dev/null @@ -1,60 +0,0 @@ --- Add migration script here -CREATE TABLE IF NOT EXISTS tor ( - package TEXT NOT NULL, - interface TEXT NOT NULL, - key BYTEA NOT NULL CHECK (length(key) = 64), - PRIMARY KEY (package, interface) -); - -CREATE TABLE IF NOT EXISTS session ( - id TEXT NOT NULL PRIMARY KEY, - logged_in TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - logged_out TIMESTAMP, - last_active TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - user_agent TEXT, - metadata TEXT NOT NULL DEFAULT 'null' -); - -CREATE TABLE IF NOT EXISTS account ( - id SERIAL PRIMARY KEY CHECK (id = 0), - password TEXT NOT NULL, - tor_key BYTEA NOT NULL CHECK (length(tor_key) = 64) -); - -CREATE TABLE IF NOT EXISTS ssh_keys ( - fingerprint TEXT NOT NULL, - openssh_pubkey TEXT NOT NULL, - created_at TEXT NOT NULL, - PRIMARY KEY (fingerprint) -); - -CREATE TABLE IF NOT EXISTS certificates ( - id SERIAL PRIMARY KEY, - -- Root = 0, Int = 1, Other = 2.. - priv_key_pem TEXT NOT NULL, - certificate_pem TEXT NOT NULL, - lookup_string TEXT UNIQUE, - created_at TEXT, - updated_at TEXT -); - -ALTER SEQUENCE certificates_id_seq START 2 RESTART 2; - -CREATE TABLE IF NOT EXISTS notifications ( - id SERIAL PRIMARY KEY, - package_id TEXT, - created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - code INTEGER NOT NULL, - level TEXT NOT NULL, - title TEXT NOT NULL, - message TEXT NOT NULL, - data TEXT -); - -CREATE TABLE IF NOT EXISTS cifs_shares ( - id SERIAL PRIMARY KEY, - hostname TEXT NOT NULL, - path TEXT NOT NULL, - username TEXT NOT NULL, - password TEXT -); \ No newline at end of file diff --git a/core/startos/migrations/20230118185232_NetworkKeys.sql b/core/startos/migrations/20230118185232_NetworkKeys.sql deleted file mode 100644 index 44779d82..00000000 --- a/core/startos/migrations/20230118185232_NetworkKeys.sql +++ /dev/null @@ -1,62 +0,0 @@ --- Add migration script here -CREATE EXTENSION pgcrypto; - -ALTER TABLE - account -ADD - COLUMN server_id TEXT, -ADD - COLUMN hostname TEXT, -ADD - COLUMN network_key BYTEA CHECK (length(network_key) = 32), -ADD - COLUMN root_ca_key_pem TEXT, -ADD - COLUMN root_ca_cert_pem TEXT; - -UPDATE - account -SET - network_key = gen_random_bytes(32), - root_ca_key_pem = ( - SELECT - priv_key_pem - FROM - certificates - WHERE - id = 0 - ), - root_ca_cert_pem = ( - SELECT - certificate_pem - FROM - certificates - WHERE - id = 0 - ) -WHERE - id = 0; - -ALTER TABLE - account -ALTER COLUMN - tor_key DROP NOT NULL, -ALTER COLUMN - network_key -SET - NOT NULL, -ALTER COLUMN - root_ca_key_pem -SET - NOT NULL, -ALTER COLUMN - root_ca_cert_pem -SET - NOT NULL; - -CREATE TABLE IF NOT EXISTS network_keys ( - package TEXT NOT NULL, - interface TEXT NOT NULL, - key BYTEA NOT NULL CHECK (length(key) = 32), - PRIMARY KEY (package, interface) -); \ No newline at end of file diff --git a/core/startos/src/account.rs b/core/startos/src/account.rs deleted file mode 100644 index cb08a0d5..00000000 --- a/core/startos/src/account.rs +++ /dev/null @@ -1,132 +0,0 @@ -use std::time::SystemTime; - -use ed25519_dalek::SecretKey; -use openssl::pkey::{PKey, Private}; -use openssl::x509::X509; -use sqlx::PgExecutor; - -use crate::hostname::{generate_hostname, generate_id, Hostname}; -use crate::net::keys::Key; -use crate::net::ssl::{generate_key, make_root_cert}; -use crate::prelude::*; -use crate::util::crypto::ed25519_expand_key; - -fn hash_password(password: &str) -> Result { - argon2::hash_encoded( - password.as_bytes(), - &rand::random::<[u8; 16]>()[..], - &argon2::Config::rfc9106_low_mem(), - ) - .with_kind(crate::ErrorKind::PasswordHashGeneration) -} - -#[derive(Debug, Clone)] -pub struct AccountInfo { - pub server_id: String, - pub hostname: Hostname, - pub password: String, - pub key: Key, - pub root_ca_key: PKey, - pub root_ca_cert: X509, -} -impl AccountInfo { - pub fn new(password: &str, start_time: SystemTime) -> Result { - let server_id = generate_id(); - let hostname = generate_hostname(); - let root_ca_key = generate_key()?; - let root_ca_cert = make_root_cert(&root_ca_key, &hostname, start_time)?; - Ok(Self { - server_id, - hostname, - password: hash_password(password)?, - key: Key::new(None), - root_ca_key, - root_ca_cert, - }) - } - - pub async fn load(secrets: impl PgExecutor<'_>) -> Result { - let r = sqlx::query!("SELECT * FROM account WHERE id = 0") - .fetch_one(secrets) - .await?; - - let server_id = r.server_id.unwrap_or_else(generate_id); - let hostname = r.hostname.map(Hostname).unwrap_or_else(generate_hostname); - let password = r.password; - let network_key = SecretKey::try_from(r.network_key).map_err(|e| { - Error::new( - eyre!("expected vec of len 32, got len {}", e.len()), - ErrorKind::ParseDbField, - ) - })?; - let tor_key = if let Some(k) = &r.tor_key { - <[u8; 64]>::try_from(&k[..]).map_err(|_| { - Error::new( - eyre!("expected vec of len 64, got len {}", k.len()), - ErrorKind::ParseDbField, - ) - })? - } else { - ed25519_expand_key(&network_key) - }; - let key = Key::from_pair(None, network_key, tor_key); - let root_ca_key = PKey::private_key_from_pem(r.root_ca_key_pem.as_bytes())?; - let root_ca_cert = X509::from_pem(r.root_ca_cert_pem.as_bytes())?; - - Ok(Self { - server_id, - hostname, - password, - key, - root_ca_key, - root_ca_cert, - }) - } - - pub async fn save(&self, secrets: impl PgExecutor<'_>) -> Result<(), Error> { - let server_id = self.server_id.as_str(); - let hostname = self.hostname.0.as_str(); - let password = self.password.as_str(); - let network_key = self.key.as_bytes(); - let network_key = network_key.as_slice(); - let root_ca_key = String::from_utf8(self.root_ca_key.private_key_to_pem_pkcs8()?)?; - let root_ca_cert = String::from_utf8(self.root_ca_cert.to_pem()?)?; - - sqlx::query!( - r#" - INSERT INTO account ( - id, - server_id, - hostname, - password, - network_key, - root_ca_key_pem, - root_ca_cert_pem - ) VALUES ( - 0, $1, $2, $3, $4, $5, $6 - ) ON CONFLICT (id) DO UPDATE SET - server_id = EXCLUDED.server_id, - hostname = EXCLUDED.hostname, - password = EXCLUDED.password, - network_key = EXCLUDED.network_key, - root_ca_key_pem = EXCLUDED.root_ca_key_pem, - root_ca_cert_pem = EXCLUDED.root_ca_cert_pem - "#, - server_id, - hostname, - password, - network_key, - root_ca_key, - root_ca_cert, - ) - .execute(secrets) - .await?; - - Ok(()) - } - - pub fn set_password(&mut self, password: &str) -> Result<(), Error> { - self.password = hash_password(password)?; - Ok(()) - } -} diff --git a/core/startos/src/action.rs b/core/startos/src/action.rs deleted file mode 100644 index 3223aaa8..00000000 --- a/core/startos/src/action.rs +++ /dev/null @@ -1,163 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; - -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use indexmap::IndexSet; -pub use models::ActionId; -use models::ImageId; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use crate::config::{Config, ConfigSpec}; -use crate::context::RpcContext; -use crate::prelude::*; -use crate::procedure::docker::DockerContainers; -use crate::procedure::{PackageProcedure, ProcedureName}; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::{display_serializable, parse_stdin_deserializable, IoFormat}; -use crate::util::Version; -use crate::volume::Volumes; -use crate::{Error, ResultExt}; -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -pub struct Actions(pub BTreeMap); - -#[derive(Debug, Serialize, Deserialize)] -#[serde(tag = "version")] -pub enum ActionResult { - #[serde(rename = "0")] - V0(ActionResultV0), -} - -#[derive(Debug, Serialize, Deserialize)] -pub struct ActionResultV0 { - pub message: String, - pub value: Option, - pub copyable: bool, - pub qr: bool, -} - -#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub enum DockerStatus { - Running, - Stopped, -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Action { - pub name: String, - pub description: String, - #[serde(default)] - pub warning: Option, - pub implementation: PackageProcedure, - pub allowed_statuses: IndexSet, - #[serde(default)] - pub input_spec: ConfigSpec, -} -impl Action { - #[instrument(skip_all)] - pub fn validate( - &self, - _container: &Option, - eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - ) -> Result<(), Error> { - self.implementation - .validate(eos_version, volumes, image_ids, true) - .with_ctx(|_| { - ( - crate::ErrorKind::ValidateS9pk, - format!("Action {}", self.name), - ) - }) - } - - #[instrument(skip_all)] - pub async fn execute( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - action_id: &ActionId, - volumes: &Volumes, - input: Option, - ) -> Result { - if let Some(ref input) = input { - self.input_spec - .matches(&input) - .with_kind(crate::ErrorKind::ConfigSpecViolation)?; - } - self.implementation - .execute( - ctx, - pkg_id, - pkg_version, - ProcedureName::Action(action_id.clone()), - volumes, - input, - None, - ) - .await? - .map_err(|e| Error::new(eyre!("{}", e.1), crate::ErrorKind::Action)) - } -} - -fn display_action_result(action_result: ActionResult, matches: &ArgMatches) { - if matches.is_present("format") { - return display_serializable(action_result, matches); - } - match action_result { - ActionResult::V0(ar) => { - println!( - "{}: {}", - ar.message, - serde_json::to_string(&ar.value).unwrap() - ); - } - } -} - -#[command(about = "Executes an action", display(display_action_result))] -#[instrument(skip_all)] -pub async fn action( - #[context] ctx: RpcContext, - #[arg(rename = "id")] pkg_id: PackageId, - #[arg(rename = "action-id")] action_id: ActionId, - #[arg(stdin, parse(parse_stdin_deserializable))] input: Option, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - let manifest = ctx - .db - .peek() - .await - .as_package_data() - .as_idx(&pkg_id) - .or_not_found(&pkg_id)? - .as_installed() - .or_not_found(&pkg_id)? - .as_manifest() - .de()?; - - if let Some(action) = manifest.actions.0.get(&action_id) { - action - .execute( - &ctx, - &manifest.id, - &manifest.version, - &action_id, - &manifest.volumes, - input, - ) - .await - } else { - Err(Error::new( - eyre!("Action not found in manifest"), - crate::ErrorKind::NotFound, - )) - } -} diff --git a/core/startos/src/assets/adjectives.txt b/core/startos/src/assets/adjectives.txt deleted file mode 100644 index c482d50e..00000000 --- a/core/startos/src/assets/adjectives.txt +++ /dev/null @@ -1,1296 +0,0 @@ -ominous -sturdy -angelic -frontal -legal -murky -rougher -formal -local -bold -grouchy -grazing -bumpy -wonky -boxed -factual -sunny -trim -selfish -humble -plastic -broke -shorter -rustic -brittle -narrow -astute -icky -sullen -bemused -creative -snowy -sane -sedate -deviant -icy -spoiled -blond -concave -cyan -lucky -lively -risky -wounded -greater -limber -social -glued -painful -warm -nutty -amused -carried -amateur -meager -working -faded -stark -reborn -darkened -entire -charred -speedy -clear -kinky -impish -ageless -prewar -correct -molten -admired -uneasy -higher -tragic -inane -magenta -urban -nearby -grouped -noisy -rural -fetid -waxed -dandy -cocky -aqua -dingy -unbent -lewd -quiet -mellow -unvalued -itchy -clunky -snug -opaque -bulky -smug -helpful -velvet -violet -valid -wobbly -dodgy -rare -cocoa -selfless -complex -simple -prim -blank -obsolete -surplus -funky -chubby -daring -spongy -tinted -prepaid -geeky -unsure -broken -wimpy -obscene -monthly -brown -erased -freaky -decent -rimless -cordless -tainted -huffy -yawning -toxic -puffy -inner -smart -lesser -mute -mighty -deluxe -thatch -frosty -vulgar -darkish -fun -annoying -swollen -loco -magic -generic -tan -trendy -blind -worried -stray -pungent -fluid -mixed -soviet -ruby -rabid -silky -regular -winter -ethnic -sour -sleepy -frail -dicey -heavy -rude -asleep -loony -singing -exterior -bendy -feeble -intact -robust -foamy -pale -crazed -sloping -soaked -next -thorny -voting -squeaky -pregame -dismal -teak -rumbling -furry -hazel -quaint -older -custard -golden -paltry -phony -smooth -trivial -happier -unboxed -chalky -learned -younger -calm -jagged -hateful -still -round -final -unhappy -jumbo -obedient -germy -mangy -pickled -untamed -puny -pink -mild -mini -able -related -auburn -giddy -tapered -flabby -cruel -awake -artsy -dull -virtual -dry -useless -winking -nerdy -drastic -dimpled -slick -purple -jolly -maple -humorous -swank -ready -level -shrill -amusing -grim -crabby -canned -anxious -refried -undying -revised -spicy -refined -aquatic -foggy -chosen -grey -bespoke -flavored -baggy -foul -shocked -unlucky -yapping -insecure -daft -sleazy -unused -short -your -snarky -tweed -rosy -brief -welcome -buffed -enamel -inept -abrupt -taboo -hungry -audible -bitter -untidy -stale -strong -moldy -brass -crisp -temp -acidic -top -old -limited -neon -cushy -angled -potent -rotten -snappy -floppy -good -fatal -darned -somber -stunned -unloved -vicious -endless -wavy -louder -musty -no -distant -savage -devout -feisty -rogue -uneven -excess -main -crimson -illicit -normal -faux -quick -trite -evil -guilty -kosher -beloved -wooden -indigo -gentle -raunchy -subtle -finer -stained -wanted -informal -cute -cedar -returned -square -shady -prissy -bronze -meaty -taller -vegan -flying -cloudy -vague -snazzy -twisty -primary -timid -liquid -left -nifty -red -alright -smoky -western -okay -fine -armored -rinsing -denim -glib -routine -silver -harsh -near -useful -preachy -tedious -arctic -bluish -primal -medical -sore -buff -patchy -chief -swell -adept -ideal -yummy -gutsy -key -mocha -harmful -loyal -oblong -dense -violent -great -lousy -emerald -large -magnum -dancing -chewy -ashamed -teal -secular -curly -fertile -furtive -some -ruined -spry -pliable -beige -bony -frantic -wary -bawdy -muscled -past -jumpy -legit -glossy -fishy -corny -small -crying -beefy -pompous -tough -other -proper -nimble -vital -foolish -dainty -rough -herbal -brainy -afraid -frilly -hectic -frigid -bogus -skilled -tasty -private -slobbery -plump -shaved -fatty -initial -zippy -oldest -bad -nasty -lame -careless -flaxen -moonlit -spare -candied -crafty -hollow -eager -pointy -caustic -wronged -dinky -manmade -niche -fair -varied -melting -blazing -crass -renewed -waxy -bald -obese -big -passive -slimy -iced -ultimate -hairy -dyed -elusive -sunken -emphatic -yeasty -overt -frugal -ditzy -remote -clumsy -diabetic -ladylike -swampy -aging -fur -lined -bossy -dorky -mobile -crushed -slate -immense -easiest -pointed -rented -psycho -minty -expert -new -maroon -elfish -zany -drafty -ceramic -felt -same -hideous -marine -elastic -oozy -novel -hasty -weary -stuffy -capable -inert -default -central -sweaty -sloped -smoked -creepy -vexed -bionic -regal -cranky -steep -open -floral -movable -varsity -docile -basic -coping -meek -loose -fried -plush -fuzzy -another -creaky -white -tubular -angular -edgy -visible -curvy -neutral -low -woozy -edible -dill -yucky -camo -hopeless -polite -smoggy -wacky -crude -imposing -west -shaky -rebel -soft -mythic -sheer -flat -aft -wriggly -citric -noble -crazy -blurry -insane -spooky -touchy -unique -bare -funny -sincere -eldest -unusual -granite -prime -sooty -engaged -awful -tangible -manual -weaker -lukewarm -junky -amber -gothic -light -steamy -scabby -unkind -empty -porous -subdued -frizzy -yellow -tall -foreign -anemic -lean -cuddly -wrong -upbeat -greedy -stout -dotted -alive -profound -frozen -lavish -wax -onyx -milky -veteran -thin -classic -gruesome -personal -taxable -lasting -random -valiant -pulpy -stiff -dirty -retired -secret -lacy -online -bloody -royal -wild -found -nervous -viable -dusty -peachy -sudsy -moody -askew -sad -little -kissable -convex -grubby -broad -employed -glass -muscular -avian -direct -goofy -absent -boiling -loving -gaudy -micro -muggy -happy -nearest -strange -growing -ashy -brisk -sporty -blunt -stoic -high -mundane -newborn -longer -needy -feral -plain -front -mature -lucid -washed -husky -eerie -unseen -weepy -strict -real -hyper -orange -raw -stormy -foxy -latex -dang -tired -fragile -tactful -active -squishy -wealthy -portly -magical -curved -fasting -worn -silent -wiry -replica -tame -backlit -fragrant -nice -adapted -wet -first -whole -unarmed -suitable -sober -green -oiled -benign -only -erratic -lonely -balsa -poor -damp -spiffy -groggy -losing -black -vast -rusted -leaky -blocky -futile -decaf -prompt -ironic -sulky -giant -folded -tender -mean -powered -tacky -single -hot -moist -lush -slim -ripe -messy -aloof -blotchy -shaggy -billowy -boring -demented -usual -bloated -cheap -medium -secure -subpar -holy -trusty -steel -gory -solid -starchy -worst -better -nosy -tepid -irate -wispy -bored -sharper -damn -epic -dreary -neat -dank -crooked -urgent -stupid -mousy -nude -filthy -usable -smutty -burly -kooky -tangled -rusty -illegal -lethal -optimal -sudden -marble -mint -padded -closed -onerous -lit -uneaten -bushy -bootleg -robotic -scary -petite -late -bent -kind -radiant -ivory -alpine -joyous -batty -copper -ample -armed -extra -wise -vintage -butch -sharp -tied -glum -sticky -free -pure -stinky -super -humid -dark -grimy -senior -sneaky -wide -detached -twitchy -brawny -his -hefty -tidy -ancient -thinner -badass -fancy -barbed -native -fresh -buggy -exposed -gummy -pudgy -chrome -deaf -busy -bamboo -amazing -dear -scenic -knit -fruity -young -demur -sugary -deep -fantasy -antique -brunette -vanilla -leather -undead -torn -extinct -vivid -serious -lost -amiable -sassy -poison -slushy -mossy -hardy -hard -favored -cheesy -bizarre -lead -sacred -married -custom -static -sensual -idle -pagan -bland -wrinkly -safest -odd -lustful -breezy -hybrid -natural -maimed -rocky -frosted -fond -salty -skinny -sage -blue -sizable -spiral -mammoth -crunchy -gifted -turbo -female -live -warming -dire -woody -massive -festive -lax -upright -tight -fat -sloppy -silly -shoddy -acrid -angry -gloomy -onshore -unfair -rapid -virgin -fluffy -frisky -eroded -best -warped -gold -steady -slow -swift -postwar -rich -brutal -feudal -whacky -partial -dreaded -common -any -elegant -isolated -thick -homeless -loud -playful -bright -disco -far -jubilant -woven -dizzy -joyful -stuck -weak -fake -dumb -costly -verdant -soapy -content -unjust -healthy -agile -superb -elated -catchy -proud -dreamy -hunky -mega -chunky -undated -rotund -rad -drippy -satin -exotic -drab -surly -slinky -whiny -early -fleshy -curled -marbled -mashed -deadly -brave -botched -pastel -rubber -naked -ebony -frayed -general -dental -groovy -dazzling -devious -elite -junior -last -tangy -huge -chic -bonus -eastern -upscale -shiny -gross -jaded -perky -average -lumpy -used -skiing -warmer -nuclear -upset -ex -awkward -kinder -watery -utopian -hissy -ornate -greasy -long -baroque -worthy -bovine -weird -chaotic -lazy -covert -bottom -modest -obvious -nomadic -public -patient -handy -posh -antsy -crucial -popular -current -creamy -caring -similar -sleek -saggy -extreme -vain -weedy -fizzy -mad -tense -comfy -uptight -vile -macho -glowing -lurid -durable -cozy -fabric -fit -male -plaid -boxy -finicky -fast -stern -napping -glad -fickle -dynamic -perfect -burnt -cool -windy -tardy -thrifty -defiant -crispy -bouncy -juicy -saffron -just -wool -crummy -her -oaky -soggy -naughty -flirty -tiny -supreme -tested -azure -soupy -coarse -wicked -thirsty -tart -tapping -modern -leafy -stony -singed -minor -painted -logical -former -deeper -sterile -tin -pleasant -firm -kindly -coveted -eternal -pushy -runny -austere -stocky -rigid -flashy -tricky -upstate -scruffy -testy -absurd -cold -lilac -pebbly -oval -creased -clever -viral -hidden -modular -sultry -mushy -recent -dried -chaste -bubbly -earthy -jealous -shabby -copied -iffy -official -core -alert -crusty -shy -jovial -safe -shifty -dusky -roomy -petty -grumpy -educated -sandy -numb -vibrant -armless -retro -starlit -sly -wood -aged -enraged -flawed -indoor -fixed -metal -fussy -muddy -witty -gray -bleak -honest -grand -full -my -jelly -shallow -simpler -cosmic -scarlet -musky -donated -heroic -naive -salted -painless -trained -inky -zesty -scarce -boiled -rookie -scented -measly -quirky -exempt -sweet -pesky -spotty -easy -sublime -elderly -faster -clean -daily -wired -bouncing -organic -gaunt -swanky -mouthy -tipsy -candid -scared -homely -arcane -weekly -rowdy -impure -stable -vapid -buttery -oily -pretty -unsafe -gnarly -limp -more diff --git a/core/startos/src/assets/nouns.txt b/core/startos/src/assets/nouns.txt deleted file mode 100644 index 9c23b82a..00000000 --- a/core/startos/src/assets/nouns.txt +++ /dev/null @@ -1,7776 +0,0 @@ -scowls -zest -regime -heist -films -bling -lent -hassle -collar -flock -mountain -river -caveman -row -spyglass -visions -jelly -cymbal -pedestrian -joke -nets -money -songbird -scuba -toddy -litter -ginger -gyms -paws -kits -human -mandolin -molt -lyric -clot -plural -racism -senator -museums -goose -rams -artwork -thongs -reveler -dweeb -scale -distress -looms -burial -native -trio -spirits -refuse -worrier -rave -mercy -prong -plethora -mojo -captives -edicts -creamer -grips -devices -hobo -dips -irony -dominion -kilo -manicure -sequel -locust -yew -pills -penalty -veggie -airbag -charge -lance -safe -regalia -kindling -gent -cloud -volley -democrats -marks -flutes -fracture -crepe -fork -jails -union -seltzer -bores -coats -motel -fairies -uncle -garage -expense -role -context -form -brush -puma -thief -snare -pixy -pageant -hedge -colonies -bishops -vessel -glimpses -peanuts -ordeal -cakes -miles -dogmas -slang -pranks -sibling -laxative -gnat -dip -fuel -cone -edit -imprint -musk -derby -crush -ticks -abstinence -statues -fidelity -rein -swans -poodle -cads -skunks -togas -gutters -comics -apology -lobbies -bundle -helpline -cook -hunter -idol -loader -blow -relative -feminism -sword -rugs -corner -gumball -sail -caps -rarity -putt -class -squads -tourist -mourners -goblets -scooter -shark -tuition -card -wallet -luge -alcohol -swing -studio -arson -player -convent -jogging -disc -hide -babes -bridges -switch -thunder -spearman -bonuses -daddy -sorcery -cookbook -net -selector -pelican -hubcap -umps -ticket -blizzard -escape -phonics -handball -dupe -ink -prongs -lamps -rocks -snap -inlay -hatbox -quakes -scalp -spare -canister -echo -award -pitcher -robots -fringe -dregs -wines -gust -joey -conduct -gyration -night -cab -bees -entries -bagel -ivory -squire -hairdo -energy -talent -agency -elder -bonfires -bacon -dues -plantation -joust -charter -occasion -icing -salad -walk -clamp -diaper -scull -hosts -spuds -stall -mines -wives -cortex -heroes -clasps -lane -paint -snobs -farms -sweep -tunnel -cups -bear -fangs -cruelty -moans -proof -swats -owners -wasabi -riots -cheese -photon -militia -gurus -sparks -chisel -borough -scream -effort -trifles -edge -metro -rats -mediums -novice -internet -missile -manhood -petal -spoilage -bread -oil -curry -sills -bit -gun -festival -scrooge -quality -escort -dub -dares -slot -bangs -rite -socket -candle -fairy -harpy -anthems -damages -hound -leaf -serf -splice -profit -fingers -gout -flaxseed -lust -feast -surprise -trolls -zodiacs -scoundrel -activists -deity -sloth -bottles -limo -hamlet -flash -comet -salon -garbage -entree -picnics -cursor -fabrics -kidney -dancers -pellet -scopes -gulp -doggy -porch -lute -junction -chill -ammo -letters -codes -snake -disease -sympathy -mishap -sprains -ripple -crusaders -baby -facility -courier -finale -writing -fragrance -fees -mocker -earwig -tantrum -hacking -flatware -vicinity -minks -pram -session -earful -shunt -grudge -straw -playtime -girth -straps -exhaust -pushover -can -bits -terror -metal -chariot -men -terms -trombone -bias -concepts -quake -purl -insults -tipper -manger -keyboard -pope -coeds -lobes -oafs -reader -views -orders -empress -activist -jungles -severity -brail -doctrine -tractor -rodeo -shelves -rod -toga -lips -bend -sums -darkrooms -grunge -bite -carving -gang -heel -rotors -couples -lanes -dissenter -groups -balconies -clods -monogram -beard -genres -backhoe -plums -alumni -schedule -taper -whim -riff -stories -peer -chop -statutes -posers -slacks -burro -urns -adversary -cage -blocks -figs -acts -well -light -reseller -weight -ratios -heathen -whisk -plan -babe -prose -puff -jokers -colonel -puppet -guise -retainer -subplot -rodent -underdog -diva -pugs -glut -fighter -aprons -dare -clones -gravy -sender -gate -muff -snowsuit -devourer -gallon -affair -nuke -oboe -mirrors -timothy -ricotta -spool -pear -locket -fact -village -laptop -mower -hair -revolver -brawls -ankle -dots -hour -grill -chin -recovery -result -helms -stags -starlet -fallacy -wisps -opiate -flowers -thrones -bout -farmland -contacts -trickery -footwear -cat -briefs -frill -link -pedal -flyover -path -stews -paradox -reactor -version -push -salts -quilt -output -pikes -affairs -conscript -lock -landfill -superman -apache -fraction -pupils -gloss -wafer -nurse -air -habitat -repeater -mars -deals -play -fob -mothball -chefs -victor -bingo -footsie -passcode -tragedy -botanists -tasks -friar -ramps -glory -stripe -wolves -muscle -penguin -upside -factoid -rubble -pancreas -festivals -triumph -refining -fence -threat -antlers -tint -giant -father -flirts -gear -syrup -pandas -hurler -theft -daze -haggler -adhesive -stretch -removal -harp -isle -gratuity -fink -families -sandbank -tapioca -onus -boats -laces -incense -sizzle -rain -pith -winter -flakes -glee -smashup -tides -poles -skeet -petals -reward -pothole -brig -renderer -fritter -thistle -gasoline -lynx -jury -spout -hoax -roars -stain -razor -power -bio -moments -creel -spud -tears -luster -uptake -domains -balloon -subs -pueblo -moan -retiree -ring -wing -symptom -ceiling -warranty -strolls -grimace -trilogy -tool -images -disco -ump -nirvana -bet -phantom -hags -axle -serving -plow -reach -gym -pelt -caves -duet -pouch -homage -senate -roger -solvent -sardine -boot -elective -hulks -shelter -gophers -resource -baboon -flooding -trend -album -beetle -spindle -coma -flora -scripts -oils -scar -shimmy -torso -population -sandworm -malt -honeybee -pauper -pears -letdown -tavern -squall -crane -nooks -lake -shelving -gulls -naming -cameras -mesa -character -strobe -number -gore -indexes -heat -cougar -placard -tribute -murmurs -junta -cake -ad -reason -diapers -jellies -squabble -contours -rogue -stereo -twerp -slog -swag -duchess -waif -error -customs -curb -crusts -wounds -gardens -wrecker -volcano -bandanna -feasts -punishment -needs -sedation -dance -hinge -handle -advisor -bargain -flipper -car -toolbox -stub -barista -lair -almond -panda -courses -cokes -playlist -domino -pauses -lox -gaskets -discovery -welds -fervor -crypts -vote -ships -fame -volume -audio -fear -browser -hail -streams -nod -punch -novices -breeches -shrieks -dribble -brakes -progeny -softy -baseball -march -hardhat -rims -meadow -falls -word -gusts -dude -pip -marmot -magic -velocity -mitt -outage -handgun -relic -chapter -steaks -pawns -upheaval -towns -famine -crashes -chai -squiggle -headlock -quiver -clasp -oath -wiz -malts -vans -gambling -barbarian -decibels -handset -naturist -florist -larks -barges -bonds -bomber -bowls -robotics -enemy -ports -steak -polls -randy -omelet -solids -lodges -pixel -loads -oyster -crimps -theist -wails -heroics -genre -herb -pork -reproach -bevel -seashore -rematch -lama -adage -tidbit -innovation -home -twang -breast -vagrant -pagan -nudge -hoops -ribbon -ancestry -hacker -disgrace -jot -olive -trends -lead -seas -fishhook -sale -costume -sill -tong -nosh -monument -wings -market -sternum -freeware -roster -luncheon -bliss -fire -cube -pizzas -ears -vegan -belch -brie -pruning -carts -wad -beavers -hardener -axles -totem -chain -try -vice -ravine -goblet -bid -spelt -wit -ghost -chowder -cries -glimpse -mystery -curbs -brake -faults -dealers -turkey -sanctuary -eyelid -tanks -members -mailbox -yelps -forge -firs -sight -senators -tenant -province -debts -edges -leg -mailroom -breed -mildew -treaty -jays -puffin -duplex -herring -cardinal -mole -promoter -woodland -meatball -shower -yeti -disk -cartoons -byte -peaches -mirth -ode -gift -gallery -cot -ointment -jogs -bran -vicar -fats -lids -squeak -band -tick -oracle -hats -van -turmoil -chums -frown -cysts -dumpling -vendors -sorority -junkies -panic -moods -amputee -vitality -deaths -myths -dusk -slave -wasps -eggplant -equation -issues -tummy -focus -gland -hopes -audience -suction -luck -mummy -smirk -oaf -splendor -tutu -pendant -estrogen -dummy -rung -pant -makeup -chemist -journey -boxer -hood -stripes -marbling -earphone -theories -walnuts -eggshell -loop -vents -platypus -bob -rider -encore -operas -doodle -ferry -shim -apricot -prints -stoops -estimate -snort -rakes -grains -outpour -petunia -smack -rubber -force -concept -brisket -atheist -quota -bundles -chow -furniture -passes -mast -calcium -chip -jalapeno -empire -grout -stools -feet -gosling -rituals -cadets -extent -roll -keys -frames -smog -violin -cumin -studs -floor -lockers -recon -city -iguana -tours -thong -daughter -alpaca -crop -jammer -plumbing -blast -cottages -limeade -lizard -leash -soaps -scraps -networks -mane -grandson -detox -cow -dock -admiral -limit -jock -archery -dollars -colors -swimwear -tarps -risk -bolt -inbox -bee -glades -beaks -tablet -foes -ransom -deacon -tour -delay -impacts -paste -mountains -dragon -wrist -stench -grills -toggle -noise -tripods -wish -chef -thinker -chunks -trash -spell -safety -credential -flake -donation -bonnets -designer -egret -scratch -worker -outdoors -suns -jetty -llama -mimosa -flypaper -patina -trunk -tabasco -distance -blimp -ridge -prophecy -yip -scan -actresses -boys -digit -suitcase -cranes -sixties -toucan -landlady -basket -kiln -zombies -grandfather -rye -lawns -kitty -slipper -robes -hands -flair -drain -pansy -archive -dikes -mascara -hooves -crow -footers -cult -impulse -ingot -pottery -desk -bong -rhino -clarinets -fires -vows -bubble -manhole -cut -ski -chick -produce -belly -chainsaw -eel -glider -copies -dugout -total -panther -hurdle -scarcity -decree -kinsman -goldfish -peg -expo -cog -bin -cooks -wills -ego -delta -ounce -militant -chemists -voters -pig -webs -niece -cape -clinics -nip -prisoner -yoke -traumas -pillow -matter -microphone -loans -tripod -reject -slush -crawlers -fluids -protector -braid -climate -preset -wail -staple -reels -node -taunt -jurist -galley -mirror -artifact -newton -diamonds -skirmish -digs -zone -gash -urgency -vices -oaths -flood -purses -buffer -ladle -aces -lava -clove -piano -caption -tedium -janitors -shed -almanac -postbox -meerkat -sandpit -plate -grenade -cello -hose -colas -hit -office -mooch -squatter -pistol -rake -cyclists -hazards -anteater -job -peck -medal -pianist -bozo -jaw -pad -temple -stooge -stop -snipers -purist -respect -sombrero -cops -trimmer -reed -krypton -objective -overcoat -balcony -tribes -humps -ruse -womb -venture -ignition -rehab -cramp -gloves -anthills -lawyer -gatherer -bird -refinery -crafter -tic -ponies -mall -magician -trickle -escapade -falcons -drops -galleria -daggers -sable -tacks -podium -pinch -crusader -spoke -addict -lilac -bulges -screwdriver -panty -unifier -horse -chord -snafu -pumice -yak -shotgun -teacup -workers -bases -sear -llamas -rolls -liquor -cranks -splices -kinship -surgery -coaches -scrap -brim -yam -ledge -poetry -country -sodas -speeds -sensors -script -ovary -curling -opacity -rules -rum -nutcase -groans -stoppage -ankles -oak -culprit -pucker -earl -bunker -divot -kazoo -balls -rockfish -roof -healer -infidel -burns -mechanism -example -dander -grief -tuxedo -claim -undead -felt -steed -tonic -casket -sari -fan -peace -fantasy -kerchief -justice -crags -engine -cardigan -pattern -lentil -duration -motive -figure -retread -mortuary -pooch -plumber -coast -pennant -loon -works -medicine -types -notices -nanny -algae -stamina -carbs -chum -feedback -octaves -buttons -crock -mayo -anatomy -sinners -desires -nunnery -hoe -linguini -plaza -teabag -marlin -tools -parody -doubts -decline -rears -defect -hurdles -aquarium -tamale -monitor -runway -yells -rewards -view -sandals -sewer -ovaries -detours -dam -auction -trances -porthole -inactivity -lavender -lens -shows -thigh -peon -poison -misinformation -filth -zeros -warrior -lilt -prefix -bulge -charcoal -sleds -moat -arena -prods -quarters -file -preacher -riptide -kettles -resort -fuses -couple -sting -daffodil -coworker -bells -pickle -hardware -crouton -lore -blitz -debate -elitism -armpit -omens -neuron -accident -delirium -scab -roach -mazes -pushpin -sprite -shore -slurp -souls -voter -heights -stroller -probes -basins -elves -ending -defeats -pension -forces -strip -ruts -quarks -auto -bruises -subset -dialog -passerby -altar -exporter -skid -golf -leeches -futons -crown -splinter -cavalry -hay -blooper -prisms -opponent -chirp -prawns -maze -lands -husbands -camps -glasses -camera -dress -fez -zap -privacy -ping -showcase -kale -cent -beets -preamble -urn -granny -cache -raises -hulk -fields -griddle -gerbil -rumor -waves -tenor -wreck -deputies -magazine -parkas -flattery -ox -chests -blanket -alehouse -ravines -princess -necks -women -raisins -advice -swarms -samba -daytime -renewal -jolts -veto -shorts -promos -emperor -intro -belt -scribe -basil -avatar -lamb -agencies -bog -wasp -grinch -sax -ward -quadrant -tomahawk -sip -string -knocks -cover -destiny -hydrant -bologna -cobras -dorks -stallion -hangover -molds -sheep -scum -lairs -maize -thrower -boutique -nicotine -goal -cigars -oracles -castle -boa -ladders -corn -brows -sum -viaduct -apostle -observer -slugs -groin -grip -purifier -neck -reentry -nebula -wombs -tech -cartons -party -lunacy -earmark -smoke -haven -bone -comments -disposal -chimp -minx -slab -bulls -neutron -purse -order -setting -threats -skiff -monotype -psychic -zeal -longbow -computer -surplus -drafts -filters -maverick -lion -atlas -pump -clique -register -scrubber -measles -tests -trusts -jungle -sniper -rugby -salvage -composers -capsules -part -look -theism -bug -oblivion -splash -owl -employer -flasks -skewers -clogs -support -pegs -tee -urge -sis -hive -rifle -lung -week -fuzz -badge -mowers -grubs -genie -provider -antacids -colts -garnet -taxis -comets -tarp -wok -comma -waiter -gourd -clues -clients -yowls -bookmark -cask -crease -cuddle -forks -torch -wedding -magpie -librarian -sheets -toad -crimes -demon -thrush -bean -area -falcon -crevice -eras -handwork -atrium -odors -shrapnel -soup -girdle -stat -rate -cash -ding -mutation -bins -pout -gangway -violence -want -casts -caviar -tarmac -substance -steam -brute -whiskey -gasket -vitamin -helmets -flub -felons -tinkle -appetite -liquid -lashes -forms -liar -remnant -cider -tires -spools -section -accusation -curls -dome -tether -throne -virtues -tomes -agenda -foxes -prison -pamphlet -biped -cinder -turtle -clicker -foxhole -fusion -brunch -amulet -suspect -charger -yurts -pie -movers -events -oboes -cicada -hotel -galaxies -peat -muss -criteria -dish -boxes -revival -varsity -fridge -cougars -bind -armful -science -canteen -gizzard -crutch -ethanol -handoff -gallons -sprawl -exits -hilt -crystals -cosmos -logs -corral -parka -researcher -cousin -nephew -voice -escapist -earring -does -taboos -trail -forgery -snowplow -soap -memory -fobs -players -self -attic -rump -earwax -puritan -jocks -scorpion -duckling -starch -heaters -glob -crosses -scare -leader -youth -hearts -jailers -tropics -digits -kitchen -hives -kook -molar -custard -spleen -pushup -paddle -carafe -dike -tarts -dowry -insult -paprika -doom -sides -attorney -sips -climb -kitchens -battle -banners -bank -lien -crack -notes -baggage -whisper -tactics -identity -polygon -cord -leaps -slabs -record -logos -names -swamp -brims -tub -slug -overtone -vow -hickory -epiphany -equinox -giants -meme -dye -bags -joint -salons -plows -bill -satire -tiaras -gypsy -face -nouns -mists -spin -iris -siding -pasture -beaver -clothing -contest -jarl -roundup -demeanor -icons -twigs -post -jitters -speller -sanctity -gale -custody -flings -prams -porridge -steps -grids -rook -looter -pastime -reams -curtsy -bra -tease -flow -birth -skits -pagers -pebbles -walnut -check -mobster -yodel -jars -visit -rubbers -elevator -contract -candy -thunderbolt -swig -savior -railcar -senders -edibles -bead -thimble -readers -motion -sulfide -kelp -rebels -stains -pits -visas -dweller -pedigree -jokes -outpost -thugs -serpents -spares -sprites -skull -brigade -leotard -scrabble -experts -humor -freebie -ladybug -commerce -dismay -seat -document -acting -flags -cough -hip -crust -leap -cornhusk -coupons -timber -sonata -punks -vet -offer -wrester -servers -garland -twine -unit -messes -favors -salary -niche -singer -blogs -purr -squash -journal -freak -papa -pasta -ceremony -mauls -model -hanger -scam -gulps -stanza -dagger -sensor -plunder -bikes -agate -frock -curves -corgi -creek -cufflink -toast -nations -dads -thrills -kitten -swath -tombs -pecks -crick -beam -quicksand -washer -cues -deeds -grunts -lettuce -firms -locks -compounds -cycling -flukes -consoles -abdomens -aliment -platform -squeal -beast -loves -guild -gelatin -slates -fig -rice -stalks -loan -squirt -hangar -cruises -jowls -knack -club -decency -linens -periods -criminal -harmony -threads -creed -fists -board -minefield -sulfur -stew -wetsuit -recliner -yield -alerts -cobs -drizzle -group -runs -words -jaguars -magazines -bride -realm -stair -losses -water -torque -synopsis -springs -skirt -divinity -ore -seeds -tree -toss -peel -skinhead -fish -envelopes -cans -paw -studies -deskwork -anise -starfish -problem -expedition -introvert -spots -honor -lodge -shots -space -boy -bellies -honey -friends -reflux -kink -tapestry -gems -span -growls -cereal -show -irritant -packs -rental -veils -postcard -roads -infantry -bongo -shacks -efforts -imps -locale -roadway -chat -label -legwork -mound -legroom -rotor -times -immunity -roast -wells -dinners -banjos -twitch -vacation -stick -moonwalk -passport -smudges -venues -chap -synergy -morals -ride -tutors -tar -bandage -morale -cosigner -boars -craving -delicacy -cables -branch -radar -sublevel -planet -dirk -grade -bag -serve -remedies -kayaks -thrust -rebates -spas -beauty -skies -abs -heroism -creeks -arsonist -arrival -swimsuit -cousins -viewer -pads -labs -launch -purge -twister -fiddles -pile -moocher -south -tribe -junk -verbs -spotter -villages -name -shock -breasts -cob -dune -bans -fronds -worm -anchors -deceit -bleach -moderator -arch -axes -coasts -dictator -tin -jersey -kinfolk -dayroom -outbreak -felon -cycle -soloist -mosaic -ends -mushroom -runner -pepper -boots -headwind -bicep -mister -berms -outtakes -ladles -stalk -dams -ram -briar -burp -belts -mafia -noose -brooms -frogs -servant -tail -matchbox -chart -worms -gut -shill -opal -gazebo -watch -snout -step -doubling -jarhead -dreams -jargon -biscuit -critic -zebras -forges -chemical -angel -cuts -links -razors -anteaters -mints -procurer -fans -champions -hub -shank -angler -taxation -rebirth -creeps -grocer -wax -nerds -fake -man -conflicts -teen -thermos -oceans -beach -brink -presto -barley -vodka -history -bore -laugh -dill -slobs -cockpit -donor -shin -hexagons -bluff -mug -granola -hermits -crusade -norm -game -escargot -sharpie -powers -turner -echoes -cleavage -lid -ribs -corsage -latitude -upstairs -manes -tricycle -daisy -sofas -reviver -leak -storage -impulses -plays -bud -department -pretext -mat -rep -pact -fryer -hammock -bull -jests -way -spokes -titbit -dwelling -crab -cobra -hobbies -puns -spur -docks -rudder -jog -twins -historic -trench -managers -camels -fiddle -noises -cocoon -zeppelin -sulfite -omnivore -jukebox -rockets -fluke -newt -cinch -scanner -siesta -blur -parcels -clam -farmers -blemishes -wage -daybed -brawn -tacking -parmesan -hut -mugs -tines -chains -decoy -scope -hue -quark -samples -tables -prune -judo -lesson -fajita -schools -school -blade -antler -tip -ship -thud -karma -sushi -turban -pebble -flanks -qualm -advocate -ghoul -void -artisan -pea -villain -hull -lasers -writer -pastrami -sugar -diet -equipment -sprout -snore -room -columns -septum -stuffing -grit -earth -ninjas -passage -tot -headsman -recap -duels -subsidy -particle -renegade -doorknob -rim -lyrics -combo -modes -wheel -place -dimple -bubbles -ball -blame -tweak -vandal -portion -mules -visors -dash -huntsman -shuttle -bullet -sect -hounds -refill -season -assassin -spoof -jumpers -hobby -tidings -faction -pride -sabers -feed -sequence -din -act -jar -flux -audits -pupil -con -banister -hardship -speed -profile -jaguar -flare -populace -educator -spear -friction -fee -deceiver -ruses -grub -passenger -menace -toys -idols -dentists -overbite -monkey -fights -reply -drive -secretary -dragster -frolic -blood -macro -paramour -teacher -server -haunch -sizes -tangle -scales -refund -druids -rank -bankers -circuses -pushes -payday -foal -shackle -kilowatt -shroud -piglets -trough -squabs -sabotage -deluge -fungus -cluck -battery -impact -gondola -halls -pock -decks -climbs -barbs -marine -length -tug -treason -vibe -tram -riches -habit -carnivals -records -grants -master -boulder -badger -assistant -urology -condor -radio -kerosene -mood -prunes -whims -pacifier -veal -plugs -cursors -virus -birds -cola -notification -precinct -chaos -mayday -perks -morphine -tightwad -reefs -mutants -orchard -scams -bombers -knots -camp -compilation -dockyard -lawyers -sails -shrouds -couch -category -methods -nutshell -wart -array -drugs -bunk -shakes -phases -punk -teeth -flower -travels -relics -peter -mourner -toe -handclap -valuables -sheath -soybean -stud -trait -whisks -hunt -shopper -quintet -coke -talc -antidotes -tulip -cotton -jaybird -dandruff -sarcasm -hatchet -summer -freeway -valves -ark -snail -marmots -knight -antelope -winnings -protest -reeds -lottery -inch -sheen -pants -dot -throats -kites -toes -moss -minutes -curfew -coward -lobby -lumber -tone -papyrus -closet -vixen -salute -nibble -muffin -cue -tab -rides -cancer -waist -docs -gangrene -spore -stint -daisies -spark -strips -mates -gamer -user -orbits -excess -examples -bunch -shredder -math -snakes -subway -region -salsa -sufferer -hedgehog -displays -pity -shard -visa -bunkers -powwow -door -rivet -giblet -robe -team -hem -slips -crime -mascots -kayaker -pubs -rest -anaconda -levers -prey -dev -format -cellmate -rower -popes -fizz -enzyme -overlord -lakes -bidet -moths -prologue -commander -jeep -condo -sandwich -welts -epoxy -lotto -dudes -rot -twig -cross -cockatoo -dolt -blender -spam -buoy -start -nag -thumps -unease -hawks -sketch -enemies -strobes -halos -bond -rooky -earpiece -queen -aroma -cradles -twilight -togs -defenses -library -pews -armpits -sand -spite -smears -vehicle -segment -sandbar -flop -manual -geese -learner -lab -knives -lining -sites -squirts -referee -teepee -league -stir -road -bane -stray -tiling -crops -artist -whiz -sauce -sorts -blush -outcast -gander -traps -peaks -shortcut -cable -unions -prisons -margins -monarch -fall -pools -menus -port -marbles -martyr -barbers -lily -outing -cashews -chops -seam -rust -lager -styles -love -varnish -tide -lions -boil -vaults -booth -hackers -lime -hat -sinew -zombie -dogs -attics -ripples -laurel -judge -looters -curl -clerks -spire -dart -bumps -sultan -cynic -steer -stores -lady -ales -noun -fib -sacrifice -sumo -posse -doe -reboot -rotunda -anklet -airline -ale -fang -promo -handrail -landmass -tent -gloater -priority -rerun -trance -time -fly -lungs -greed -poppy -pans -nerve -eon -monoxide -range -dividers -rigging -clothes -storm -bung -decades -dibs -reflex -shrink -tones -pong -hulls -kick -gateway -gallows -newspaper -reptile -deer -grandpa -eyelids -clicks -madam -badges -mob -shield -brawl -livers -jazz -flotilla -turtles -notch -goop -king -yarn -punctuation -chaplain -broom -larva -process -nylon -shifter -cacti -berries -polish -furnace -farmer -soy -factory -landlord -miss -fleet -snitch -soils -suds -reel -valley -ghouls -vitamins -ranges -walls -pencil -jest -booty -symbols -earplugs -contour -catnip -guide -tendon -daydream -vagrancy -lobster -knife -perjurer -movie -dividend -bakers -eggroll -drums -fawn -shawl -sidewalk -shawls -captions -crepes -splatter -cottage -cadet -photo -tory -strength -spectacle -gymnast -mange -ties -tiger -vines -shout -buns -reformer -icon -book -cords -shovel -awls -zebra -chair -vets -prawn -stance -permit -clown -sow -folds -host -bonfire -twirls -stilts -suits -claps -dole -vector -ripcord -mask -hacks -tray -potion -kiosks -cynics -titanium -shadows -organ -grin -page -cell -source -footnote -mites -gunsmith -endings -fanatic -flamingo -refusal -adherent -visitor -tap -yokes -tendency -consumer -robin -competitor -survivor -government -ace -base -silkworm -spike -synopses -mama -clip -stay -eve -jawline -scouts -detour -guns -credit -creels -chaps -month -appendix -stilt -nobody -sweats -craze -emerald -peacock -dragons -pointer -upgrade -emotion -scoops -line -morning -bedpans -feat -clay -blouse -bait -overflow -specimen -criticism -cocaine -jape -moth -victory -pomp -drill -diamond -talk -rings -tubes -fedora -episodes -state -skittles -theme -thrusts -rule -editors -embolism -coils -hemlock -omen -camisole -jackal -hornets -dynasty -kings -things -divorcee -dresses -hazard -arbor -pound -flute -countries -couriers -glass -spiders -overture -jaunt -shoal -thrift -elbow -replay -shrinks -matron -tomb -onlooker -anvils -repose -troops -garden -aisles -anxiety -roses -orcs -cud -knock -fail -bench -period -rodents -coconut -hazelnut -pug -stubble -poems -skater -winery -laser -girls -mahogany -distaste -flight -snags -stocks -awards -lever -function -picks -year -corks -shamrock -ditch -smells -landmine -jabs -beams -dolly -pokers -equity -delegates -brandy -crumbs -arborist -vertigo -pundit -scimitar -doves -encampment -curio -jeers -fathers -archives -duffel -glue -haiku -holes -watches -vision -buffet -kibble -petri -tans -coves -index -oppressor -pitch -cheer -taco -negotiator -packet -volumes -troupe -cur -throws -dogma -vise -railway -flights -search -scrutiny -paladin -circlet -virtue -interest -pot -animator -stature -trifle -rails -medium -aisle -vista -stamps -winners -stalls -dishes -ligament -bassoon -contests -squab -brat -statue -store -jigsaw -knobs -bagels -fuss -security -chairs -glimmer -aunt -demand -stunts -spores -baton -deed -life -spits -access -queens -pirate -vein -picnic -knapsack -recount -bids -stitch -study -melodies -smelt -sherry -senior -placemat -cruncher -mover -footer -watt -sailor -plots -patio -upswing -stimuli -vortex -ethics -bogs -pretense -feuds -macaw -click -vest -back -treats -eggnog -finger -orzo -kiosk -quills -cruise -deviancy -romp -planner -evidence -odor -coral -snowbird -blasts -yogis -pack -overseer -list -aviator -barn -centaur -cashew -roar -forum -growl -snorts -spaces -charms -module -perm -ravens -ads -hatchery -hoaxes -bro -mural -footing -winks -pursuit -teargas -blare -racing -inns -whisky -judges -hook -surface -tourism -jeans -lipstick -cluster -lover -nymph -importer -droplet -goddess -sneeze -uniform -answers -eclair -rags -swirl -weapon -hitch -seaweed -whales -ladder -gourds -phase -machete -seagull -corporal -machine -member -mulch -monsieur -fools -vowel -control -beak -people -shrew -elk -scarves -subfloor -flatfoot -remark -tracer -snag -bowling -plates -parent -clover -sources -crowd -unrest -clump -angles -embassy -joggers -pacifism -turbines -codex -mesh -silt -excuses -ranks -designs -filter -crabmeat -earlobe -rind -twinge -parabola -tails -verse -fossil -vibes -email -explorer -bow -buck -matcher -quarrel -caulk -quiz -payment -figurine -kicker -crate -outlet -hash -sludge -den -clink -pew -tactic -bricks -beards -joyride -radiator -nose -liter -snack -parades -oxymoron -aspirin -pit -eclairs -swarm -runt -hassles -snarl -acorn -mail -huts -knowledge -gala -champagne -defiance -screw -malware -metals -geology -drum -fences -tinsel -growth -turbans -feminist -landing -armrest -palms -addicts -proton -helper -brands -vanes -teamwork -horn -giggle -houses -laps -odes -piglet -essay -buckle -throng -erasure -bonbons -molars -swings -consumers -rise -barbecue -ancestor -vogue -supplier -moons -footwork -ground -overhang -harness -papers -serfs -liberty -markers -pretzel -memo -greeting -sweater -clod -win -jogger -war -molecule -vantage -hugs -spatula -joker -division -mint -sector -mind -welder -pancake -trader -kisser -hatred -toll -client -flip -raft -cogs -muskets -soda -driveway -veneer -hart -employment -axiom -barber -twist -loss -mead -faculty -ruble -paranoia -snowshoe -census -gloom -dodo -tank -plywood -creak -spoils -outs -bodies -motivation -pollen -lights -jug -taboo -smith -poncho -jobs -rancher -cast -fungi -awning -trick -soot -lawn -heron -spinster -charity -tact -foothill -oars -turbofan -piles -blinker -mist -padding -preface -limits -hamlets -oysters -enforcer -chocolates -sedan -yodels -wink -retrial -trot -reliance -archers -shortage -jeer -choice -winner -jerk -dens -birthday -meats -buses -weaver -article -chores -shirt -smash -dog -peril -author -town -nudes -concrete -fumble -cherry -cyclist -showman -harem -bribe -dances -yowl -agates -hybrid -ruler -dancer -nap -kids -rope -silk -silencer -cells -motives -lace -mumps -prompter -lagoons -newbie -cores -loaves -entity -valet -ploy -parakeet -weeds -monarchy -sheaths -scorer -nags -apron -squib -follicle -headlamp -puddle -bugs -mantra -slop -liars -fur -wielder -serves -yogi -clavicle -budget -monorail -will -sandbox -riddle -frog -masses -wand -creases -lances -motors -mule -matrix -bills -drapery -ravioli -bays -thought -bolo -hinges -brads -hug -debates -rubs -percent -sinks -cannon -tramps -shame -refunds -fudge -launcher -maturity -humus -gluten -stairs -drips -sausage -grime -wren -circus -density -outrage -rudders -guy -lemon -taxes -oxen -bands -license -guitars -till -trainer -doll -fair -orphans -parish -harps -sardines -parade -samurai -bass -overtime -antacid -fissure -cleft -marrow -gene -vulture -chest -pulp -graduate -nutmeg -mimic -systems -vase -wins -crayon -patience -bike -burgers -gem -prowess -crutches -hole -squeegee -parrot -founder -bulletin -igloo -anchovy -perk -eternity -tugboat -chive -trial -latte -rinses -doorbell -undertow -nerves -thrill -suntan -voodoo -deities -beaches -stork -scroll -symphony -lagoon -duvet -bikers -critter -bleep -exes -imbecile -folk -reins -koala -majesty -scowl -pixels -sole -museum -lapse -fastball -spires -planks -nukes -set -attire -dosage -waste -graphics -angels -counting -gardener -sap -fiend -parlor -prices -umpire -sage -grower -offices -jackets -calendars -tiers -eraser -outlook -zipper -mamba -care -expert -bucket -rips -tampons -cheeses -girdles -guff -ear -hobbit -latches -scuff -timer -carp -arcade -shop -pins -vole -puck -sculptor -jumble -barrier -sister -bladders -upstroke -task -price -harpist -lark -pyramid -tampon -truth -boils -savage -wares -purveyor -moon -fleets -flagman -tribunal -preplan -flea -strudel -tape -pair -bards -jokester -hours -bonbon -cleavers -spider -pun -case -polo -dales -roles -cannons -default -boasts -puzzles -seniors -headway -tarot -smell -grits -warden -vests -mischief -issue -shindig -prep -mic -orca -gender -stranger -poets -film -type -clack -abyss -jumps -segments -fate -senses -utensil -weeks -maid -marigold -molasses -research -chapters -cleric -table -caramel -outfit -knee -deck -denim -giraffe -daycare -sprig -czar -letter -caravan -junkyard -tribune -organs -knoll -duo -sermon -windshield -nodes -cribs -sycamore -paints -freewill -vises -sons -relays -bun -erosion -latch -sprints -hippie -rockstar -moped -subtype -wardrobe -leases -cart -lunchbox -calzone -hangars -courts -adult -kilobyte -bible -land -deviant -banshee -fern -story -death -angle -villa -outlaw -liver -aorta -cowls -assistance -beans -moose -invites -chemicals -mesas -palm -abdomen -sigh -volts -bed -canisters -boon -moonbeam -culprits -combs -everyone -flocks -gesture -footage -stingray -lamp -press -scarf -mammary -janitor -demands -robber -stove -tannery -caddie -pies -swords -snares -chives -grid -course -football -nails -talcum -loot -haste -dawn -heater -teapots -afterlife -glance -crates -demons -migrant -discounts -audiences -beepers -eardrum -lumberjack -spouts -whip -pounds -concerts -dunce -cavern -coins -jerky -mold -sweat -curtain -halo -lizards -popcorn -dispute -cup -wife -eclipse -ecology -elms -swab -calories -gifts -flounder -crayons -mount -tallow -canoe -voucher -phobias -ores -optics -bums -arks -sitter -glitter -blips -helm -diners -propane -teapot -warning -rampage -plastic -shrine -venom -turmeric -servo -epilepsy -landside -diagrams -estate -jacket -scallion -slip -mile -cracker -mayhem -talons -bistro -nights -sheet -earache -revision -washout -yacht -inputs -arms -carnage -regret -quotes -roughage -buyers -citation -trays -boards -ebook -velvet -soups -breach -butler -semester -bipod -herbs -skincare -clock -heft -carton -exorcist -firm -rabbit -mutts -dust -defense -essence -runts -broiler -target -jade -storms -pleasure -blades -wards -cubes -pickles -beggar -alarm -bedroom -delusion -yogurt -foothold -gopher -kilt -hint -tier -strand -eye -dowel -brokers -fanfare -luxury -hedges -alien -dorm -culture -chalk -army -bloke -tops -rooms -rubdown -jams -verdict -mine -spade -lotus -tax -freckles -apple -revolts -clutter -glop -desktop -pictures -rays -tickle -nerd -header -dose -boom -county -nemesis -tune -sects -lovers -navies -sleeve -style -patrol -antennae -pizza -comrades -brook -lobe -wildland -trainee -cricket -covers -scans -meter -skates -bales -whips -sorcerer -settler -tear -assembly -repairs -loaf -piers -pets -foot -finals -curtains -stinger -cowl -tack -maids -fruit -armor -trumpet -annoyance -age -plans -flame -ware -leaks -opera -sundae -lives -skins -homeowner -rumps -hiker -payphone -weekend -dean -date -licks -herds -nature -critics -canary -stashes -polka -stoop -streets -lords -holidays -width -pill -spine -pusher -fragment -layer -dynamite -liqueurs -root -thump -envelope -zoology -sirens -smocks -tramp -guys -awl -goals -serpent -pox -skills -chargers -olives -massager -clots -despair -throttle -gears -hunch -cowbird -biker -key -imp -gigabyte -rivers -trucker -knights -henchman -pipe -stipend -rival -tricks -hamper -clank -violins -lease -bronco -supermom -continent -blimps -tennis -shrub -hare -bonanza -jackals -eagles -hula -leads -traitor -doorman -colonist -aid -train -reporter -patriot -venue -traffic -isotope -activities -score -coach -wool -turnip -cologne -bros -headache -raps -suet -slit -nursery -dislike -defender -stash -payroll -spoons -cupid -doctors -muscles -obituary -binder -mutt -absentee -clan -spades -outback -crowds -uprising -trousers -places -winch -doors -handcuff -seed -barb -crouch -utility -mush -proxy -track -tooth -strains -dollar -actress -ozone -chump -smiles -autumn -burrow -baskets -fugitive -yeast -vale -shingle -hope -avocado -flap -aircraft -hornet -foods -facts -sprouts -forts -beer -clipart -atriums -coot -paths -driver -smuggler -dope -font -carnival -volt -cycles -glazing -news -revolt -star -gradient -cowards -headset -hits -release -moms -keno -quip -sunset -trees -copiers -cliques -tuft -sexes -iota -poser -traction -shallot -rhythm -housing -goggles -devils -daylight -squares -sesame -pager -wads -policy -deal -inks -juice -coaster -resin -deli -muskrat -flavor -cyst -startup -pacemaker -arcades -foam -trinity -wheels -wash -knickers -juicer -overhaul -claw -auctions -bar -berry -thicket -swill -applause -foe -rose -nicks -nieces -disks -linen -orbs -pouches -licorice -wildcard -mankind -ruins -nest -pecans -diffuser -poll -burger -streaks -moves -lies -sorceress -dwarf -trapdoor -unicorns -dump -sauces -pats -outsider -drunkard -sob -barge -sores -shrines -few -marker -cupcake -robins -totems -flam -eyebrow -replies -vineyard -revenge -rifling -elephant -purpose -dice -thorns -prank -quilts -mess -motorway -yawn -goldmine -platinum -colt -gown -banks -probe -dream -equator -skill -consultant -idiocy -onward -waffle -bobcats -griffon -greeter -margin -mage -celery -octane -mime -faces -bistros -phrases -palaces -ibex -tales -map -jetpack -actions -nests -baritones -swat -makeover -smut -run -students -wine -wonk -debtor -oxidant -sled -gauze -brunt -ashes -warlock -wigs -vampire -subtitle -gongs -pipeline -trouble -groan -swamps -weather -heirloom -valve -hero -fiber -stunner -lemur -cuffs -talisman -cures -cork -touch -jail -shift -phones -nachos -gravity -juggler -spinout -manhunt -spears -seats -choir -chamber -cultures -years -handyman -idealist -tassel -pilgrim -oar -android -mop -lift -height -rail -rouge -tux -pavement -prudes -pesos -animal -radios -moonrise -hump -riveter -uranium -okra -geek -decibel -turret -guitar -weasel -myth -boudoir -subtotal -borders -chore -tents -crew -bus -inn -files -ivy -grape -parties -grace -grounds -swipe -tutor -peanut -pages -decor -litters -waltz -desks -slaw -spat -question -crests -lot -inches -leverage -motor -friend -raccoon -jay -stencil -kin -crag -canals -voltage -thesis -help -keel -coconuts -spices -sheds -trip -knolls -reverend -gripe -refrain -logic -illness -mass -claims -strides -dunes -wands -flaw -reunion -topics -gizmo -shriek -island -denizen -gruel -overrun -donator -strike -junkie -rascal -homes -dials -entryway -curses -alley -robot -perms -ovum -spoon -remarks -crypt -oats -pint -repair -vapors -outhouse -lice -shape -joy -riot -beau -log -limes -bile -pest -visitors -saga -lapdog -chimps -cots -plants -login -poultry -moats -valium -composer -tigers -agent -shrimp -pegboard -raisin -saints -options -posts -exchange -tapes -asp -doorstep -shove -buffoon -ideology -sickle -protons -honks -split -throat -wisp -theory -drams -pores -commando -surname -website -husk -loft -tycoon -albums -bath -juvenile -kangaroo -stabs -shells -artery -areas -resident -seminar -silicon -analyzer -napkin -elf -fool -splints -walrus -digest -harems -wraith -legume -skewer -outcome -operator -carol -fondling -trapper -airport -moaner -concert -inchworm -quarry -clout -mutiny -peeks -handsaw -authors -fowl -wiper -hunk -break -slaps -tinker -traces -ramrod -jowl -capsule -bunny -sack -cabs -carrot -saber -silos -manager -spelling -spread -ferret -ramen -captain -shale -bang -god -duke -wrench -hogs -covenant -contents -synapse -calf -captive -nit -world -monks -bikinis -geeks -genitals -printer -poker -mayor -nuptials -tube -thorn -spinner -marinas -dew -mice -comment -stock -lethargy -screws -tunes -hikes -lunch -lard -cereals -flan -rasp -mockup -wiretap -yearling -gimmick -spirit -bicycle -huddle -chipmunk -astrology -mongrel -effect -charisma -democrat -top -heaps -thug -cliff -castles -wages -holiday -aqueduct -athlete -beagle -coroner -arrays -element -laptops -plaster -kegs -teat -sleet -orcas -embargo -horns -stems -emporium -cloth -mammals -objects -fetish -rescuer -commute -calipers -animals -canyon -hops -broker -sham -bale -panorama -drinks -rocket -sweets -cars -lag -hunks -ritual -cure -fumes -jinx -bonnet -knob -taproot -cows -headgear -rebuttal -squid -mutant -pail -ogres -welt -bulbs -brushes -ploys -nacho -risks -plenty -gas -cabins -photons -scones -symbol -tyrants -degrees -rifts -acrobats -halogen -wharf -goats -stylus -bushes -trampoline -shave -lentils -quid -sticks -scalps -suburb -dentist -strainer -macaroni -occupier -proverb -hoods -hotels -specks -virgins -shaft -column -jive -aunts -rods -egos -quail -console -mum -jumper -slogan -movies -dent -sapling -brochure -quest -tenure -bakeries -era -stucco -poet -trucks -opposite -biology -champ -turns -anthem -expenses -diameter -labor -cranium -baking -hymn -hyena -tubas -bets -plat -twitter -casinos -salads -owner -chimes -brew -pang -congress -ropes -pleat -implant -druid -counties -snips -teriyaki -crooks -bottle -playroom -decal -neighbor -kidnaper -stardom -orange -herd -shrubs -craft -orgies -arrow -family -meteors -method -perfume -egotism -steroid -flaps -request -rush -starter -refresh -murmur -discount -chicken -murals -tots -cherub -dropout -thrall -pups -router -block -employee -dial -dodos -insect -color -strokes -monk -brood -comic -gowns -unloader -hell -balms -token -narrator -armband -starship -islands -voyage -libraries -ginseng -muses -termite -remover -cartoon -ribcage -credits -ingots -labels -lie -oregano -teens -darkroom -byway -jailer -stream -poise -pentagon -pipes -sanctum -vats -nugget -nativity -reburial -loom -hybrids -fins -crumpet -troughs -blossom -blog -item -language -breath -ash -campers -gavel -bumpers -skipper -corners -dram -fin -grating -guts -clouds -panes -garb -gains -vagabond -plume -critters -kite -gauntlet -counter -drawl -weirdo -lounge -snow -app -townhome -bagpipes -cats -admirer -guru -tart -cuff -kimono -elastic -anglers -thespian -trout -guzzler -generator -vocalist -agents -smithy -zip -text -foyer -peak -rusk -theology -lapels -nectar -mobs -query -blaze -avatars -vial -paycheck -property -food -romance -weld -usher -stacks -raider -bail -conduit -penknife -archer -bark -headroom -delegate -slant -slits -berm -swim -pillbox -plea -kilts -beat -vat -infant -shrug -alarms -formula -riddance -eggs -sense -oligarch -barrels -skeleton -booze -grange -evasion -galaxy -gaff -ranches -espresso -mulberry -wants -bluffs -vacancy -fleas -rocker -slacker -camel -yell -worry -farmhouse -meshes -odds -tongs -urchin -punt -bomb -noodles -inside -lecture -nudity -planes -squat -goods -gumdrop -jasmine -oranges -parsnip -drug -bites -awnings -swagger -snouts -supper -lemurs -pines -charts -prize -footlocker -jab -sprigs -grain -eater -denture -bison -details -climates -ladies -arenas -channel -truce -mustard -crumb -nomad -reaction -rift -untruth -coin -damage -gumbo -faith -fortress -lemons -roots -whey -streak -glaucoma -kiwi -mimes -nudist -broth -scalpel -oases -slats -mutes -troop -slicer -obscenity -twit -stoves -tingle -suitor -sundress -marble -cactus -wiring -tad -noel -hairs -stamp -needle -survival -endnote -funnel -organism -spans -meeting -test -clog -adults -sash -economy -screens -occupant -seals -juniper -omission -limbs -hashes -cement -royalty -marathon -goblin -spruce -tags -fife -scorn -globes -heir -gnats -octagon -redo -miracle -trident -bust -laborer -cove -flannels -stage -scallop -cigar -stroll -creator -facelift -photos -shoe -nation -grader -trauma -grins -watermelon -trustee -freight -genetics -sinkers -widow -chicks -tote -granddad -shorty -antenna -farce -treasury -coca -kimonos -hermit -wind -prayer -heads -rut -strays -battles -hoody -lodging -mansion -password -demotion -monster -pelvis -suite -python -filler -trillion -rioter -glucose -race -goblins -scars -wonder -epidural -exam -leashes -pacifist -relish -parcel -setback -ducts -producer -staff -garter -gamble -norms -goof -title -cards -bows -bloat -workload -slider -secrecy -park -demise -church -rear -exec -mayors -muzzle -tomcat -teaspoon -calamity -onion -costs -medics -crevices -pane -oops -snaps -rigor -beasts -graphs -states -remix -cornea -wafers -goatee -doc -services -legumes -apes -bandits -sentry -quips -coating -cabana -spring -haul -seafood -muck -saves -anthrax -globe -treasure -cafe -domain -clerk -climber -pixie -prodigy -caddy -apostles -mumbling -leech -implants -princes -tacos -conflict -whale -minimum -boas -mandate -shack -goat -frame -halves -barriers -opus -sty -dates -finance -embers -code -diagram -poem -harbor -playoff -mermaid -desire -nocks -mounds -ushers -poplar -sport -rumors -aphid -husks -animation -riddles -wanted -deadbolt -tiara -bowl -dyslexia -basin -brats -dazzler -tunic -preview -guest -utopia -side -trunks -duel -rates -bay -tempo -swigs -cubicles -taps -branches -zinc -rock -detail -snooper -temper -stem -drainer -speech -hills -fort -spender -negligee -lass -seer -exiles -lyre -tome -vanity -meters -toffee -ogre -errors -sports -copier -scholar -joules -material -hotdog -widows -tale -wizard -duds -crayfish -humanist -swine -pore -dimples -chant -video -spectrum -pots -bunches -keep -action -maker -ice -stone -giggles -wealth -brow -hoop -son -tentacle -loons -spins -nemo -quarts -poster -recipe -hemp -exponent -platter -stump -lures -safes -zones -collars -coots -crabs -fetus -hen -varmint -botanist -meat -brains -pirates -wig -use -glands -acorns -strap -tweezers -baseballs -mommy -agendas -sass -replica -pests -tilde -milk -sunlight -crunch -vastness -relapse -jackpot -fund -call -nut -cargo -stylist -status -checks -scout -blister -lanterns -typist -object -sponge -children -truffle -topic -gecko -epilogue -lump -flaws -divots -funds -mixture -impurity -attendees -plot -slurs -excerpt -cults -showroom -yoyo -hyphens -showoff -acids -roost -slide -doorstop -antidote -skin -fury -qualms -gunk -nemeses -fluid -cowboy -corridor -twists -goo -laundry -swirls -glutton -binders -brain -savages -viola -stumps -sub -pagoda -microbe -sneer -morsel -drainage -seal -argument -butcher -seams -avenues -mixes -lure -pliers -arsenal -disputes -ape -drunks -tailor -deployment -railing -tinfoil -diner -voices -pedicure -feud -mascot -brothers -cap -hunger -distrust -ferocity -parsley -machines -mark -spinach -cod -nucleus -work -message -quart -art -racks -safari -osmosis -geranium -sprain -grams -toxin -coal -bridge -might -taxi -steroids -frond -maul -peas -variety -poses -vine -drip -clumps -rivulet -hints -update -choices -creation -creature -earwigs -melon -panel -troll -waters -bouts -prop -outfield -hacksaw -irons -sailors -colander -cub -sandal -fad -skis -peppers -sobs -calls -banner -bongs -armories -cents -favor -capitol -share -liftoff -pesto -earmuff -canon -shampoo -casualty -apostate -tissues -strain -miner -futon -casino -unicorn -upstart -spy -showdown -gap -marshes -flak -crest -iron -shot -optimist -pajamas -feather -knuckles -renter -raptor -revenue -nana -service -pocket -spook -pantry -boundary -putty -spree -armoire -moisture -donkey -skunk -rant -dinner -poacher -cupcakes -sponsor -exercise -darts -teas -quests -mace -bribes -defeat -squeeze -doorpost -predator -combat -barns -cabbage -benches -herald -blizzards -prism -mill -obstacle -grass -answer -hoes -siren -treat -traverse -pockets -vane -snorkel -elites -lack -crud -fishbowl -mollusk -screen -syndrome -diary -grades -nick -forest -germs -helmet -tweets -sulfate -feline -ban -jet -spikes -draft -cavity -clowns -buckets -guidance -navy -amulets -hubs -corridors -setup -database -getaway -folks -expanse -grant -detergent -zoo -pick -months -miners -morality -truck -mote -raise -lullaby -slime -giveaway -heists -breeze -toy -trace -goofball -gait -phone -fray -marches -jam -site -knees -find -sales -values -painter -overlap -bard -roasts -strings -odometer -tiles -candles -stars -badgers -series -duck -huff -axe -git -quill -harbors -eagle -garters -inlet -cleaver -coder -fraud -pearl -sandbag -fountain -guides -blip -demo -jacks -retreat -stripper -depot -vials -mite -device -flashes -cilantro -canopy -cherubs -hasp -curds -tomboy -pilot -phoenix -mouths -hologram -lecturer -citizen -student -cloaks -factor -opossum -clap -sign -anvil -honk -verb -signal -bruin -clef -queue -bats -screams -hall -vapor -marina -stimulus -jesters -rover -brewery -otters -flask -wire -smasher -dame -joints -muse -palace -steeple -enquirer -fiat -maggot -outburst -plateau -dowels -stands -titans -wheat -taste -uses -audit -iodine -garment -ducks -linseed -latrine -groove -confetti -rebate -stones -smile -yard -gristle -corncob -whoop -veteran -duty -tomato -patient -manors -reversal -graph -latex -hamster -optic -crews -lye -ponds -input -germ -boardroom -swinger -unicycle -jerks -point -fight -brick -dory -convents -shirts -debut -finch -peels -zoos -feeling -prince -pods -stanzas -vocation -polenta -updates -lap -weed -swaps -tarn -elm -hexagram -chips -otter -value -femur -canal -bib -breeds -heap -ramp -veil -pan -rubies -havens -fructose -patch -fiction -ruckus -drives -spot -witches -joist -saloon -bruise -stride -datebook -creep -axon -parking -insects -gadgets -doormat -snowfall -backups -parkway -disorder -cheeks -cruiser -shelf -nephews -cloves -cartload -java -shoptalk -mana -familiar -boast -spill -burrito -sitcom -soccer -humans -rinds -hypnosis -wires -mowing -ribbons -creatures -bauble -refuge -gremlin -tackle -peach -karaoke -cream -groom -subgroup -splint -dingo -pleats -monsoon -woes -putdown -medic -debt -leeks -husband -igloos -hype -lord -chords -shapes -display -sun -trophy -police -arches -naps -fibs -squad -alcoves -troy -turf -bikini -calendar -ways -body -parks -sin -bat -election -fillet -autos -comb -pita -smock -stings -deputy -puffs -games -habits -premises -banker -opium -prelude -guard -skit -ratio -shire -bobcat -subtext -pelts -sampling -babies -jockey -ham -laws -jewelry -slum -fables -ranger -doornail -gavels -grave -glare -spawn -chants -modem -daybreak -street -rage -leggings -biopsy -note -boiler -negation -jailbird -earthquake -choirs -kisses -bunks -suit -pike -dime -crook -noses -songs -earthworm -salsas -plating -rig -cartel -hexagon -border -stress -buzzword -sprint -catacomb -signs -elders -dojo -props -nail -suffix -fleece -media -burglary -box -diets -lads -warmth -cuckoo -cradle -ocelot -beds -walker -flies -skulls -onset -wane -snoop -cones -bulb -hankie -tars -evacuee -routine -eatery -heart -helium -portal -mud -perjury -stink -rag -glitch -junkman -nurses -copy -warlord -outcomes -recluse -peons -mantis -visor -diver -yams -drones -enamel -flogging -lifter -gals -duff -glamour -levies -dipper -rebar -fest -theater -patrols -delays -yoga -wager -jack -network -emphasis -clinic -canyons -buffers -virgin -quartet -lots -skeptic -invite -sulk -pole -harvest -pacts -newts -cane -being -maggots -girl -warp -society -gobs -orb -registry -crewman -faxes -bugle -postage -bandit -freaks -wildcat -perimeter -wishes -radish -suffrage -brad -stand -mother -melody -slice -toil -watts -trios -itch -batons -subject -hazing -tyrant -pub -wildfire -tremor -recess -bungee -dabs -lists -cufflinks -ocean -dashes -trowel -talks -fads -meteor -baguette -saws -beret -heading -liquids -axel -witch -pony -pay -motto -image -dud -actor -proposal -kennel -orbit -gangs -tongue -director -address -quirk -bishop -tow -lurch -shafts -units -cutlass -horses -latrines -trump -maniac -energies -diabetes -teams -vipers -scurvy -dad -fencing -needles -cascade -zippers -surfer -dumpster -meals -haze -snuff -baker -sphere -lulls -gnu -buffalos -pagans -graves -squires -births -aphids -cookie -orgasm -claws -scents -pin -baths -gliders -aluminum -field -brutes -tundra -vestibule -spouse -nude -ante -universe -tubs -monopoly -thumb -travel -almonds -frosting -relation -races -lather -bully -strife -howls -scheme -mages -proofs -gong -engraver -noodle -lyricism -arts -disaster -thirst -hubcaps -sugars -admin -grading -pod -program -errand -spice -crystal -shake -tire -stardust -future -hardhead -uncles -fright -fluff -sea -dilation -sedative -books -haircut -abacus -kettle -captains -wall -mugshot -district -specialist -umpires -web -milkmaid -litmus -jewel -playset -maneuver -lapel -spheres -dignity -sod -orc -gerbils -rigs -entry -sphinx -dimes -handler -takes -toads -guards -receiver -hawk -buds -ovens -pianos -hoof -boss -shards -mantle -froth -appetizer -hyphen -yearbook -drains -slash -deflator -wars -cobweb -nutrient -law -overlay -clutch -sacks -showbiz -sangria -prizes -overpass -cities -pho -load -tv -physician -surveyor -uproar -riverbed -achievement -lotion -fare -mare -genders -basement -extras -pains -headband -sediment -vigil -sock -catalog -closets -bazooka -pedals -gully -beet -feathers -recital -tuba -sharks -footprint -offers -fakes -barons -court -spec -illusion -spa -zit -throw -antipasto -parts -natives -buyer -anthill -grog -bucks -salami -burps -fault -suspense -blinkers -bathtub -wiki -memento -crank -primer -haunts -facilities -backs -pigs -cave -hurray -roulette -reggae -knot -effects -hackle -stroke -snacks -flatworm -surge -gamers -possum -slap -stops -curve -groves -patron -furs -manor -purchase -totals -helpers -gossip -ration -rites -pawn -basis -wino -mucus -orator -yards -fox -rogues -buckles -pardon -cairn -grudges -wipes -content -oat -bulk -matches -legs -seabird -rafts -arbors -charities -socks -nips -snails -hearth -grooves -attendant -mission -tern -medallion -paper -hooks -notice -max -wisdom -thighs -flagpole -drool -fevers -wimps -lefty -foil -chambers -plane -wood -eddy -savanna -starts -hams -posture -event -beef -ohms -floss -outages -mounts -epidemic -bola -leek -colony -majority -wraps -maw -aviators -thanks -today -silo -burr -doctor -cassette -flu -clarinet -stopper -exams -platoon -stake -pangs -centaurs -mouth -reminder -jugular -quote -tadpole -buzz -juror -nettle -crescent -ideas -heckler -batch -parasite -partner -batches -bandana -hand -lotions -brand -pumpkin -magnolia -pal -developer -handles -jingle -meet -landline -nuts -parole -stowaway -gin -cavities -banking -chard -clips -bookcase -gates -convict -scamp -huntress -cases -sets -crafts -grease -bourbon -guests -alfalfa -shuffle -mobility -forager -guilds -catapult -exploit -gaps -avenue -raffle -graffiti -parasail -historian -frat -hens -cleat -habitant -gator -railroad -headrest -stool -passion -hill -nub -analyst -inlets -quirks -frocks -rickshaw -brine -perp -cubicle -crux -wave -warps -wombats -burn -rodeos -aria -acrobat -curator -excuse -frost -drapes -rink -stages -bell -octets -imposter -yurt -ruck -curd -syntax -sinner -dolls -flames -pinky -wannabe -yawns -sprees -bolts -fencer -canoes -puzzle -smear -worlds -package -controls -tyke -hoot -spills -hike -beeper -pulses -fiends -canteens -crib -raven -pup -pens -trivia -pen -tons -marimba -stunt -aliens -pine -mastiff -dingbat -firewall -walks -shiv -cylinder -emus -magma -pain -polkas -maximum -slump -schoolboy -banjo -ranch -comedy -wreckage -heritage -legacy -satchel -silica -sponges -tipoff -rack -brothel -buddy -citizens -playmate -shares -capes -tattoo -outline -lull -house -clickers -tang -prowler -music -drone -ref -witchcraft -pier -attendee -campus -tine -afro -grouch -pose -armchair -blazes -cuticle -rhyme -trinket -info -steward -rearview -grunt -rinse -lint -woods -upload -musical -silks -senorita -souvenir -sonar -grandma -sucker -simile -emoticon -squealer -trials -bakery -spoiler -seminars -google -seashell -checkbook -arrows -eyes -jester -pacific -cheers -baristas -flab -throngs -cheek -species -episode -carrots -pips -diploma -dents -yelp -clash -marines -idealism -disdain -soul -emblem -ferns -blouses -ruin -dispatch -backpack -savings -blight -delusions -weapons -outreach -turbojet -mango -rap -baboons -boat -rinks -meal -points -soil -hitches -hockey -meow -wildlife -clang -handful -martini -masts -modules -forearm -ralph -batteries -punisher -tile -braids -ease -dirt -cheddar -winds -nettles -shell -travesty -fuse -match -democracy -craw -bars -stratus -mats -manuals -protester -lofts -gaze -hummus -tea -cusp -pucks -coupon -towel -panama -flecks -exorcism -rat -sashes -nibs -figment -wait -flint -phobia -dealer -companies -wrath -morons -pectin -ants -instalment -pampers -blend -proxies -leaves -surf -pet -sinker -egg -monogamy -locker -clams -inventory -sliver -karate -italics -miso -dangers -mode -oasis -frenzy -kiss -edition -traits -freedom -cages -flavors -decade -mustang -rookie -cowboys -mate -tower -devotee -days -alcove -bookstore -lyricist -waitress -pellets -icepack -handbook -wits -sables -musket -chime -cafes -peso -oaks -clans -pond -mills -elite -sneak -avocados -wedge -tirade -doorway -stupor -bears -denials -crybaby -coil -reps -slur -cubs -pairs -idea -bump -guardian -croc -studios -mops -dairy -rune -sound -decimal -elbows -spray -gum -copilot -kayak -dropper -grove -technician -rifles -jujitsu -winters -caravans -scent -upkeep -potato -flag -tabloid -divers -districts -oddity -lasso -moment -smirks -opals -glow -cheetah -devil -roaches -sandfish -sling -dominoes -cherries -thumbs -draw -triceps -sessions -gal -mistake -rupture -recoil -gat -flares -patios -humility -users -overview -victim -static -build -flour -lout -tofu -priest -feds -nape -customer -deletion -sequels -trips -clone -payback -coat -footrest -trap -remedy -bibs -turn -founding -joystick -celebrity -saint -slots -crusher -skier -crash -visits -exit -markets -airplane -boar -sodium -reverb -loops -print -lobsters -nook -quotas -design -fist -gizmos -prom -comrade -dwarves -denial -healers -widget -ex -nods -interns -hues -song -wombat -napalm -ellipse -emu -torches -sample -commuter -duct -misses -rainstorm -lantern -cornmeal -plum -glaze -androids -clubs -pandemic -contact -burials -ballet -crows -endpoint -numbers -petition -slack -hippo -pranker -adages -wolf -hack -drama -treatment -gasp -trek -product -tabs -ketchup -glues -hairpin -armory -need -goatskin -capitols -pop -hoses -coleslaw -yolk -rants -detector -mouse -busts -jawbone -beads -ninja -notepad -wreath -spurs -cattle -baron -ghosts -dustpan -schemes -matador -survey -trapeze -nickname -prayers -hips -maps -snip -thing -strands -sundial -hydrogen -skimmer -raids -heels -caboose -sleigh -chaff -supply -jump -howl -mix -smudge -trades -flesh -corset -slaves -dawdler -fax -owls -titan -purebred -earflap -canvas -hicks -ruby -musician -staples -champion -grad -elixir -statute -magnet -hoard -beeswax -gadget -fonts -acid -washroom -sarong -carb -pause -scrolls -floods -gran -rally -bombs -drink -snippet -editor -manatee -finisher -kid -barrel -glacier -waiver -kinks -danger -runes -lumps -fares -mothers -infants -conch -hydrants -vermin -tunnels -gyro -handcart -bacteria -octopus -butter -padlock -clamps -headwear -duress -blemish -playpen -timing -engines -idioms -baubles -viper -handgrip -tweet -octave -wick -bumper -sniff -skillet -tusks -washtub -riders -birch -skydiver -texts -patches -clue -saw -cost -cucumber -clerics -anime -loins -fondue -activity -coop -videos -gnomes -regions -stoner -lad -skate -legend -gravel -moles -havoc -blurb -premium -belches -lip -landmark -remorse -depth -scone -scotch -voles -minute -stack -warts -singers -lunatic -drudge -hijacker -pavilion -forehead -dryad -cravings -webcam -ebooks -masks -cliffs -orphan -pool -glowworm -intern -chariots -menu -shadow -wagon -download -tusk -numeral -braces -napkins -dive -toilet -bones -tibia -pancakes -gutter -marinade -farm -potions -cracks -animators -ragweed -linguist -thread -footgear -plug -creeds -asteroid -humorist -slope -apples -apricots -oxygen -handling -jewels -vendor -scene -scads -buffalo -radiance -trust -willow -camper -swivel -hop -final -wound -core -charlatan -podcast -reef -tips -glove -magnets -footpath -bladder -bot -plant -jurors -circle -faucet -fog -burg -violator -dove -dolphin -rungs -penny -hog -tulips -perch -hangout -size -clocks -gulf -lunches -shops -stag -cramps -window -data -decoys -day -stadium -deviator -cinema -gnome -items -kit -ton -wipe -shanty -plasma -fears -noon -struggle -truism -jolt -pastor -deferral -cause -slants -skyline -spies -papaya -bum -onions -juncture -pops -mom -marsh -tells -finalist -tissue -fold -sands -snowball -soak -garnish -vault -glances -jets -bevy -armies -end -grandkid -storks -arc -fender -gob -clashes -tights -slowpoke -progress -exposure -cheater -crowbar -information -coasters -teenager -biplane -north -square -health -tuna -chasm -violets -charters -exterior -cloak -talker -rip -fabric -curler -compound -dairies -powder -sauna -blob -slums -sisters -backup -rent -moor -wrinkle -risotto -stuff -wrists -pledge -windows -cartels -pints -garlic -sky -antiques -tort -pecan -curse -malls -kilns -gasps -shade -manpower -filing -rattle -typo -bush -actors -fable -coffee -scabs -comedies -trains -drills -collage -fiasco -vent -errands -alpacas -flax -salmon -vigor -cinemas -mammal -swimmer -mandarin -munchkin -boost -roux -phrase -option -gentleman -child -ant -contracts -devotion -oxford -phrasing -pulse -relay -dough -drought -jaws -writ -glade -cabin -cop -chaw -tag -vendetta -balm -classes -canopies -motes -rents -button -circles -arm -head -pleas -steamboat -biz -primate -prude -hanky -granite -tugs -etching -ream -cleats -geometry -emission -purity -hick -shoes -usage -gel -pass -kilogram -binoculars -mullets -snooze -doughboy -polymer -genes -term -lines -rib -oldie -chalice -nubs -memes -degree -empathy -trade -zodiac -altars -spells -scoop -canes -umbrella -spiral -anchor -pearls -snowman -swan -pupa -puppy -logo -cringe -axis -fever -plank -wizards -hunters -banana -bunt -woman -peroxide -oven -sofa -broncos -reasons -crimp -residue -sink -decals -feta -wimp -limb -minds -soaks -chute -fir -pillar -gram -spines -salt -hex -mania -chit -ewe -tie -gauge -compost -talon -hutch -charm -fauna -graders -dollop -company -roosts -praise -scammer -yetis -pace -delivery -cameo -chunk -donors -ditches -strategy -keg -tartar -locusts -liqueur -brace -rug -beings -looks -blubber -gif -turbine -pumps -pyre -blobs -gill -tiff -thuds -hides -stable -domes -spacebar -gull -tango -finish -mongoose -system -citadel -scandal -astronaut -drop -veins diff --git a/core/startos/src/auth.rs b/core/startos/src/auth.rs deleted file mode 100644 index a6ae2fff..00000000 --- a/core/startos/src/auth.rs +++ /dev/null @@ -1,391 +0,0 @@ -use std::collections::BTreeMap; -use std::marker::PhantomData; - -use chrono::{DateTime, Utc}; -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use josekit::jwk::Jwk; -use rpc_toolkit::command; -use rpc_toolkit::command_helpers::prelude::{RequestParts, ResponseParts}; -use rpc_toolkit::yajrc::RpcError; -use serde::{Deserialize, Serialize}; -use serde_json::Value; -use sqlx::{Executor, Postgres}; -use tracing::instrument; - -use crate::context::{CliContext, RpcContext}; -use crate::middleware::auth::{AsLogoutSessionId, HasLoggedOutSessions, HashSessionToken}; -use crate::middleware::encrypt::EncryptedWire; -use crate::prelude::*; -use crate::util::display_none; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::{ensure_code, Error, ResultExt}; -#[derive(Clone, Serialize, Deserialize)] -#[serde(untagged)] -pub enum PasswordType { - EncryptedWire(EncryptedWire), - String(String), -} -impl PasswordType { - pub fn decrypt(self, current_secret: impl AsRef) -> Result { - match self { - PasswordType::String(x) => Ok(x), - PasswordType::EncryptedWire(x) => x.decrypt(current_secret).ok_or_else(|| { - Error::new( - color_eyre::eyre::eyre!("Couldn't decode password"), - crate::ErrorKind::Unknown, - ) - }), - } - } -} -impl Default for PasswordType { - fn default() -> Self { - PasswordType::String(String::default()) - } -} -impl std::fmt::Debug for PasswordType { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "")?; - Ok(()) - } -} - -impl std::str::FromStr for PasswordType { - type Err = String; - - fn from_str(s: &str) -> Result { - Ok(match serde_json::from_str(s) { - Ok(a) => a, - Err(_) => PasswordType::String(s.to_string()), - }) - } -} - -#[command(subcommands(login, logout, session, reset_password, get_pubkey))] -pub fn auth() -> Result<(), Error> { - Ok(()) -} - -pub fn cli_metadata() -> Value { - serde_json::json!({ - "platforms": ["cli"], - }) -} - -pub fn parse_metadata(_: &str, _: &ArgMatches) -> Result { - Ok(cli_metadata()) -} - -#[test] -fn gen_pwd() { - println!( - "{:?}", - argon2::hash_encoded( - b"testing1234", - &rand::random::<[u8; 16]>()[..], - &argon2::Config::rfc9106_low_mem() - ) - .unwrap() - ) -} - -#[instrument(skip_all)] -async fn cli_login( - ctx: CliContext, - password: Option, - metadata: Value, -) -> Result<(), RpcError> { - let password = if let Some(password) = password { - password.decrypt(&ctx)? - } else { - rpassword::prompt_password("Password: ")? - }; - - rpc_toolkit::command_helpers::call_remote( - ctx, - "auth.login", - serde_json::json!({ "password": password, "metadata": metadata }), - PhantomData::<()>, - ) - .await? - .result?; - - Ok(()) -} - -pub fn check_password(hash: &str, password: &str) -> Result<(), Error> { - ensure_code!( - argon2::verify_encoded(&hash, password.as_bytes()).map_err(|_| { - Error::new( - eyre!("Password Incorrect"), - crate::ErrorKind::IncorrectPassword, - ) - })?, - crate::ErrorKind::IncorrectPassword, - "Password Incorrect" - ); - Ok(()) -} - -pub async fn check_password_against_db(secrets: &mut Ex, password: &str) -> Result<(), Error> -where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, -{ - let pw_hash = sqlx::query!("SELECT password FROM account") - .fetch_one(secrets) - .await? - .password; - check_password(&pw_hash, password)?; - Ok(()) -} - -#[command( - custom_cli(cli_login(async, context(CliContext))), - display(display_none), - metadata(authenticated = false) -)] -#[instrument(skip_all)] -pub async fn login( - #[context] ctx: RpcContext, - #[request] req: &RequestParts, - #[response] res: &mut ResponseParts, - #[arg] password: Option, - #[arg( - parse(parse_metadata), - default = "cli_metadata", - help = "RPC Only: This value cannot be overidden from the cli" - )] - metadata: Value, -) -> Result<(), Error> { - let password = password.unwrap_or_default().decrypt(&ctx)?; - let mut handle = ctx.secret_store.acquire().await?; - check_password_against_db(handle.as_mut(), &password).await?; - - let hash_token = HashSessionToken::new(); - let user_agent = req.headers.get("user-agent").and_then(|h| h.to_str().ok()); - let metadata = serde_json::to_string(&metadata).with_kind(crate::ErrorKind::Database)?; - let hash_token_hashed = hash_token.hashed(); - sqlx::query!( - "INSERT INTO session (id, user_agent, metadata) VALUES ($1, $2, $3)", - hash_token_hashed, - user_agent, - metadata, - ) - .execute(handle.as_mut()) - .await?; - res.headers.insert( - "set-cookie", - hash_token.header_value()?, // Should be impossible, but don't want to panic - ); - - Ok(()) -} - -#[command(display(display_none), metadata(authenticated = false))] -#[instrument(skip_all)] -pub async fn logout( - #[context] ctx: RpcContext, - #[request] req: &RequestParts, -) -> Result, Error> { - let auth = match HashSessionToken::from_request_parts(req) { - Err(_) => return Ok(None), - Ok(a) => a, - }; - Ok(Some(HasLoggedOutSessions::new(vec![auth], &ctx).await?)) -} - -#[derive(Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Session { - logged_in: DateTime, - last_active: DateTime, - user_agent: Option, - metadata: Value, -} - -#[derive(Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct SessionList { - current: String, - sessions: BTreeMap, -} - -#[command(subcommands(list, kill))] -pub async fn session() -> Result<(), Error> { - Ok(()) -} - -fn display_sessions(arg: SessionList, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(arg, matches); - } - - let mut table = Table::new(); - table.add_row(row![bc => - "ID", - "LOGGED IN", - "LAST ACTIVE", - "USER AGENT", - "METADATA", - ]); - for (id, session) in arg.sessions { - let mut row = row![ - &id, - &format!("{}", session.logged_in), - &format!("{}", session.last_active), - session.user_agent.as_deref().unwrap_or("N/A"), - &format!("{}", session.metadata), - ]; - if id == arg.current { - row.iter_mut() - .map(|c| c.style(Attr::ForegroundColor(color::GREEN))) - .collect::<()>() - } - table.add_row(row); - } - table.print_tty(false).unwrap(); -} - -#[command(display(display_sessions))] -#[instrument(skip_all)] -pub async fn list( - #[context] ctx: RpcContext, - #[request] req: &RequestParts, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - Ok(SessionList { - current: HashSessionToken::from_request_parts(req)?.as_hash(), - sessions: sqlx::query!( - "SELECT * FROM session WHERE logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP" - ) - .fetch_all(ctx.secret_store.acquire().await?.as_mut()) - .await? - .into_iter() - .map(|row| { - Ok(( - row.id, - Session { - logged_in: DateTime::from_utc(row.logged_in, Utc), - last_active: DateTime::from_utc(row.last_active, Utc), - user_agent: row.user_agent, - metadata: serde_json::from_str(&row.metadata) - .with_kind(crate::ErrorKind::Database)?, - }, - )) - }) - .collect::>()?, - }) -} - -fn parse_comma_separated(arg: &str, _: &ArgMatches) -> Result, RpcError> { - Ok(arg.split(",").map(|s| s.trim().to_owned()).collect()) -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -struct KillSessionId(String); - -impl AsLogoutSessionId for KillSessionId { - fn as_logout_session_id(self) -> String { - self.0 - } -} - -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn kill( - #[context] ctx: RpcContext, - #[arg(parse(parse_comma_separated))] ids: Vec, -) -> Result<(), Error> { - HasLoggedOutSessions::new(ids.into_iter().map(KillSessionId), &ctx).await?; - Ok(()) -} - -#[instrument(skip_all)] -async fn cli_reset_password( - ctx: CliContext, - old_password: Option, - new_password: Option, -) -> Result<(), RpcError> { - let old_password = if let Some(old_password) = old_password { - old_password.decrypt(&ctx)? - } else { - rpassword::prompt_password("Current Password: ")? - }; - - let new_password = if let Some(new_password) = new_password { - new_password.decrypt(&ctx)? - } else { - let new_password = rpassword::prompt_password("New Password: ")?; - if new_password != rpassword::prompt_password("Confirm: ")? { - return Err(Error::new( - eyre!("Passwords do not match"), - crate::ErrorKind::IncorrectPassword, - ) - .into()); - } - new_password - }; - - rpc_toolkit::command_helpers::call_remote( - ctx, - "auth.reset-password", - serde_json::json!({ "old-password": old_password, "new-password": new_password }), - PhantomData::<()>, - ) - .await? - .result?; - - Ok(()) -} - -#[command( - rename = "reset-password", - custom_cli(cli_reset_password(async, context(CliContext))), - display(display_none) -)] -#[instrument(skip_all)] -pub async fn reset_password( - #[context] ctx: RpcContext, - #[arg(rename = "old-password")] old_password: Option, - #[arg(rename = "new-password")] new_password: Option, -) -> Result<(), Error> { - let old_password = old_password.unwrap_or_default().decrypt(&ctx)?; - let new_password = new_password.unwrap_or_default().decrypt(&ctx)?; - - let mut account = ctx.account.write().await; - if !argon2::verify_encoded(&account.password, old_password.as_bytes()) - .with_kind(crate::ErrorKind::IncorrectPassword)? - { - return Err(Error::new( - eyre!("Incorrect Password"), - crate::ErrorKind::IncorrectPassword, - )); - } - account.set_password(&new_password)?; - account.save(&ctx.secret_store).await?; - let account_password = &account.password; - ctx.db - .mutate(|d| { - d.as_server_info_mut() - .as_password_hash_mut() - .ser(account_password) - }) - .await -} - -#[command( - rename = "get-pubkey", - display(display_none), - metadata(authenticated = false) -)] -#[instrument(skip_all)] -pub async fn get_pubkey(#[context] ctx: RpcContext) -> Result { - let secret = ctx.as_ref().clone(); - let pub_key = secret.to_public_key()?; - Ok(pub_key) -} diff --git a/core/startos/src/backup/backup_bulk.rs b/core/startos/src/backup/backup_bulk.rs deleted file mode 100644 index 21eedbaf..00000000 --- a/core/startos/src/backup/backup_bulk.rs +++ /dev/null @@ -1,322 +0,0 @@ -use std::collections::BTreeMap; -use std::panic::UnwindSafe; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use chrono::Utc; -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use helpers::AtomicFile; -use imbl::OrdSet; -use models::Version; -use rpc_toolkit::command; -use tokio::io::AsyncWriteExt; -use tokio::sync::Mutex; -use tracing::instrument; - -use super::target::BackupTargetId; -use super::PackageBackupReport; -use crate::auth::check_password_against_db; -use crate::backup::os::OsBackup; -use crate::backup::{BackupReport, ServerBackupReport}; -use crate::context::RpcContext; -use crate::db::model::BackupProgress; -use crate::db::package::get_packages; -use crate::disk::mount::backup::BackupMountGuard; -use crate::disk::mount::filesystem::ReadWrite; -use crate::disk::mount::guard::TmpMountGuard; -use crate::manager::BackupReturn; -use crate::notifications::NotificationLevel; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::display_none; -use crate::util::io::dir_copy; -use crate::util::serde::IoFormat; -use crate::version::VersionT; - -fn parse_comma_separated(arg: &str, _: &ArgMatches) -> Result, Error> { - arg.split(',') - .map(|s| s.trim().parse::().map_err(Error::from)) - .collect() -} - -#[command(rename = "create", display(display_none))] -#[instrument(skip(ctx, old_password, password))] -pub async fn backup_all( - #[context] ctx: RpcContext, - #[arg(rename = "target-id")] target_id: BackupTargetId, - #[arg(rename = "old-password", long = "old-password")] old_password: Option< - crate::auth::PasswordType, - >, - #[arg( - rename = "package-ids", - long = "package-ids", - parse(parse_comma_separated) - )] - package_ids: Option>, - #[arg] password: crate::auth::PasswordType, -) -> Result<(), Error> { - let db = ctx.db.peek().await; - let old_password_decrypted = old_password - .as_ref() - .unwrap_or(&password) - .clone() - .decrypt(&ctx)?; - let password = password.decrypt(&ctx)?; - check_password_against_db(ctx.secret_store.acquire().await?.as_mut(), &password).await?; - let fs = target_id - .load(ctx.secret_store.acquire().await?.as_mut()) - .await?; - let mut backup_guard = BackupMountGuard::mount( - TmpMountGuard::mount(&fs, ReadWrite).await?, - &old_password_decrypted, - ) - .await?; - let package_ids = if let Some(ids) = package_ids { - ids.into_iter() - .flat_map(|package_id| { - let version = db - .as_package_data() - .as_idx(&package_id)? - .as_manifest() - .as_version() - .de() - .ok()?; - Some((package_id, version)) - }) - .collect() - } else { - get_packages(db.clone())?.into_iter().collect() - }; - if old_password.is_some() { - backup_guard.change_password(&password)?; - } - assure_backing_up(&ctx.db, &package_ids).await?; - tokio::task::spawn(async move { - let backup_res = perform_backup(&ctx, backup_guard, &package_ids).await; - match backup_res { - Ok(report) if report.iter().all(|(_, rep)| rep.error.is_none()) => ctx - .notification_manager - .notify( - ctx.db.clone(), - None, - NotificationLevel::Success, - "Backup Complete".to_owned(), - "Your backup has completed".to_owned(), - BackupReport { - server: ServerBackupReport { - attempted: true, - error: None, - }, - packages: report - .into_iter() - .map(|((package_id, _), value)| (package_id, value)) - .collect(), - }, - None, - ) - .await - .expect("failed to send notification"), - Ok(report) => ctx - .notification_manager - .notify( - ctx.db.clone(), - None, - NotificationLevel::Warning, - "Backup Complete".to_owned(), - "Your backup has completed, but some package(s) failed to backup".to_owned(), - BackupReport { - server: ServerBackupReport { - attempted: true, - error: None, - }, - packages: report - .into_iter() - .map(|((package_id, _), value)| (package_id, value)) - .collect(), - }, - None, - ) - .await - .expect("failed to send notification"), - Err(e) => { - tracing::error!("Backup Failed: {}", e); - tracing::debug!("{:?}", e); - ctx.notification_manager - .notify( - ctx.db.clone(), - None, - NotificationLevel::Error, - "Backup Failed".to_owned(), - "Your backup failed to complete.".to_owned(), - BackupReport { - server: ServerBackupReport { - attempted: true, - error: Some(e.to_string()), - }, - packages: BTreeMap::new(), - }, - None, - ) - .await - .expect("failed to send notification"); - } - } - ctx.db - .mutate(|v| { - v.as_server_info_mut() - .as_status_info_mut() - .as_backup_progress_mut() - .ser(&None) - }) - .await?; - Ok::<(), Error>(()) - }); - Ok(()) -} - -#[instrument(skip(db, packages))] -async fn assure_backing_up( - db: &PatchDb, - packages: impl IntoIterator + UnwindSafe + Send, -) -> Result<(), Error> { - db.mutate(|v| { - let backing_up = v - .as_server_info_mut() - .as_status_info_mut() - .as_backup_progress_mut(); - if backing_up - .clone() - .de()? - .iter() - .flat_map(|x| x.values()) - .fold(false, |acc, x| { - if !x.complete { - return true; - } - acc - }) - { - return Err(Error::new( - eyre!("Server is already backing up!"), - ErrorKind::InvalidRequest, - )); - } - backing_up.ser(&Some( - packages - .into_iter() - .map(|(x, _)| (x.clone(), BackupProgress { complete: false })) - .collect(), - ))?; - Ok(()) - }) - .await -} - -#[instrument(skip(ctx, backup_guard))] -async fn perform_backup( - ctx: &RpcContext, - backup_guard: BackupMountGuard, - package_ids: &OrdSet<(PackageId, Version)>, -) -> Result, Error> { - let mut backup_report = BTreeMap::new(); - let backup_guard = Arc::new(Mutex::new(backup_guard)); - - for package_id in package_ids { - let (response, _report) = match ctx - .managers - .get(package_id) - .await - .ok_or_else(|| Error::new(eyre!("Manager not found"), ErrorKind::InvalidRequest))? - .backup(backup_guard.clone()) - .await - { - BackupReturn::Ran { report, res } => (res, report), - BackupReturn::AlreadyRunning(report) => { - backup_report.insert(package_id.clone(), report); - continue; - } - BackupReturn::Error(error) => { - tracing::warn!("Backup thread error"); - tracing::debug!("{error:?}"); - backup_report.insert( - package_id.clone(), - PackageBackupReport { - error: Some("Backup thread error".to_owned()), - }, - ); - continue; - } - }; - backup_report.insert( - package_id.clone(), - PackageBackupReport { - error: response.as_ref().err().map(|e| e.to_string()), - }, - ); - - if let Ok(pkg_meta) = response { - backup_guard - .lock() - .await - .metadata - .package_backups - .insert(package_id.0.clone(), pkg_meta); - } - } - - let ui = ctx.db.peek().await.into_ui().de()?; - - let mut os_backup_file = AtomicFile::new( - backup_guard.lock().await.as_ref().join("os-backup.cbor"), - None::, - ) - .await - .with_kind(ErrorKind::Filesystem)?; - os_backup_file - .write_all(&IoFormat::Cbor.to_vec(&OsBackup { - account: ctx.account.read().await.clone(), - ui, - })?) - .await?; - os_backup_file - .save() - .await - .with_kind(ErrorKind::Filesystem)?; - - let luks_folder_old = backup_guard.lock().await.as_ref().join("luks.old"); - if tokio::fs::metadata(&luks_folder_old).await.is_ok() { - tokio::fs::remove_dir_all(&luks_folder_old).await?; - } - let luks_folder_bak = backup_guard.lock().await.as_ref().join("luks"); - if tokio::fs::metadata(&luks_folder_bak).await.is_ok() { - tokio::fs::rename(&luks_folder_bak, &luks_folder_old).await?; - } - let luks_folder = Path::new("/media/embassy/config/luks"); - if tokio::fs::metadata(&luks_folder).await.is_ok() { - dir_copy(&luks_folder, &luks_folder_bak, None).await?; - } - - let timestamp = Some(Utc::now()); - let mut backup_guard = Arc::try_unwrap(backup_guard) - .map_err(|_err| { - Error::new( - eyre!("Backup guard could not ensure that the others where dropped"), - ErrorKind::Unknown, - ) - })? - .into_inner(); - - backup_guard.unencrypted_metadata.version = crate::version::Current::new().semver().into(); - backup_guard.unencrypted_metadata.full = true; - backup_guard.metadata.version = crate::version::Current::new().semver().into(); - backup_guard.metadata.timestamp = timestamp; - - backup_guard.save_and_unmount().await?; - - ctx.db - .mutate(|v| v.as_server_info_mut().as_last_backup_mut().ser(×tamp)) - .await?; - - Ok(backup_report) -} diff --git a/core/startos/src/backup/mod.rs b/core/startos/src/backup/mod.rs deleted file mode 100644 index 2f3f9bd8..00000000 --- a/core/startos/src/backup/mod.rs +++ /dev/null @@ -1,226 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use chrono::{DateTime, Utc}; -use color_eyre::eyre::eyre; -use helpers::AtomicFile; -use models::{ImageId, OptionExt}; -use reqwest::Url; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use tokio::fs::File; -use tokio::io::AsyncWriteExt; -use tracing::instrument; - -use self::target::PackageBackupInfo; -use crate::context::RpcContext; -use crate::install::PKG_ARCHIVE_DIR; -use crate::manager::manager_seed::ManagerSeed; -use crate::net::interface::InterfaceId; -use crate::net::keys::Key; -use crate::prelude::*; -use crate::procedure::docker::DockerContainers; -use crate::procedure::{NoOutput, PackageProcedure, ProcedureName}; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::{Base32, Base64, IoFormat}; -use crate::util::Version; -use crate::version::{Current, VersionT}; -use crate::volume::{backup_dir, Volume, VolumeId, Volumes, BACKUP_DIR}; -use crate::{Error, ErrorKind, ResultExt}; - -pub mod backup_bulk; -pub mod os; -pub mod restore; -pub mod target; - -#[derive(Debug, Deserialize, Serialize)] -pub struct BackupReport { - server: ServerBackupReport, - packages: BTreeMap, -} - -#[derive(Debug, Deserialize, Serialize)] -pub struct ServerBackupReport { - attempted: bool, - error: Option, -} - -#[derive(Debug, Deserialize, Serialize)] -pub struct PackageBackupReport { - pub error: Option, -} - -#[command(subcommands(backup_bulk::backup_all, target::target))] -pub fn backup() -> Result<(), Error> { - Ok(()) -} - -#[command(rename = "backup", subcommands(restore::restore_packages_rpc))] -pub fn package_backup() -> Result<(), Error> { - Ok(()) -} - -#[derive(Deserialize, Serialize)] -struct BackupMetadata { - pub timestamp: DateTime, - #[serde(default)] - pub network_keys: BTreeMap>, - #[serde(default)] - pub tor_keys: BTreeMap>, // DEPRECATED - pub marketplace_url: Option, -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[model = "Model"] -pub struct BackupActions { - pub create: PackageProcedure, - pub restore: PackageProcedure, -} -impl BackupActions { - pub fn validate( - &self, - _container: &Option, - eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - ) -> Result<(), Error> { - self.create - .validate(eos_version, volumes, image_ids, false) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Backup Create"))?; - self.restore - .validate(eos_version, volumes, image_ids, false) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Backup Restore"))?; - Ok(()) - } - - #[instrument(skip_all)] - pub async fn create(&self, seed: Arc) -> Result { - let manifest = &seed.manifest; - let mut volumes = seed.manifest.volumes.to_readonly(); - let ctx = &seed.ctx; - let pkg_id = &manifest.id; - let pkg_version = &manifest.version; - volumes.insert(VolumeId::Backup, Volume::Backup { readonly: false }); - let backup_dir = backup_dir(&manifest.id); - if tokio::fs::metadata(&backup_dir).await.is_err() { - tokio::fs::create_dir_all(&backup_dir).await? - } - self.create - .execute::<(), NoOutput>( - ctx, - pkg_id, - pkg_version, - ProcedureName::CreateBackup, - &volumes, - None, - None, - ) - .await? - .map_err(|e| eyre!("{}", e.1)) - .with_kind(crate::ErrorKind::Backup)?; - let (network_keys, tor_keys): (Vec<_>, Vec<_>) = - Key::for_package(&ctx.secret_store, pkg_id) - .await? - .into_iter() - .filter_map(|k| { - let interface = k.interface().map(|(_, i)| i)?; - Some(( - (interface.clone(), Base64(k.as_bytes())), - (interface, Base32(k.tor_key().as_bytes())), - )) - }) - .unzip(); - let marketplace_url = ctx - .db - .peek() - .await - .as_package_data() - .as_idx(&pkg_id) - .or_not_found(pkg_id)? - .expect_as_installed()? - .as_installed() - .as_marketplace_url() - .de()?; - let tmp_path = Path::new(BACKUP_DIR) - .join(pkg_id) - .join(format!("{}.s9pk", pkg_id)); - let s9pk_path = ctx - .datadir - .join(PKG_ARCHIVE_DIR) - .join(pkg_id) - .join(pkg_version.as_str()) - .join(format!("{}.s9pk", pkg_id)); - let mut infile = File::open(&s9pk_path).await?; - let mut outfile = AtomicFile::new(&tmp_path, None::) - .await - .with_kind(ErrorKind::Filesystem)?; - tokio::io::copy(&mut infile, &mut *outfile) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("cp {} -> {}", s9pk_path.display(), tmp_path.display()), - ) - })?; - outfile.save().await.with_kind(ErrorKind::Filesystem)?; - let timestamp = Utc::now(); - let metadata_path = Path::new(BACKUP_DIR).join(pkg_id).join("metadata.cbor"); - let mut outfile = AtomicFile::new(&metadata_path, None::) - .await - .with_kind(ErrorKind::Filesystem)?; - let network_keys = network_keys.into_iter().collect(); - let tor_keys = tor_keys.into_iter().collect(); - outfile - .write_all(&IoFormat::Cbor.to_vec(&BackupMetadata { - timestamp, - network_keys, - tor_keys, - marketplace_url, - })?) - .await?; - outfile.save().await.with_kind(ErrorKind::Filesystem)?; - Ok(PackageBackupInfo { - os_version: Current::new().semver().into(), - title: manifest.title.clone(), - version: pkg_version.clone(), - timestamp, - }) - } - - #[instrument(skip_all)] - pub async fn restore( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - ) -> Result, Error> { - let mut volumes = volumes.clone(); - volumes.insert(VolumeId::Backup, Volume::Backup { readonly: true }); - self.restore - .execute::<(), NoOutput>( - ctx, - pkg_id, - pkg_version, - ProcedureName::RestoreBackup, - &volumes, - None, - None, - ) - .await? - .map_err(|e| eyre!("{}", e.1)) - .with_kind(crate::ErrorKind::Restore)?; - let metadata_path = Path::new(BACKUP_DIR).join(pkg_id).join("metadata.cbor"); - let metadata: BackupMetadata = IoFormat::Cbor.from_slice( - &tokio::fs::read(&metadata_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - metadata_path.display().to_string(), - ) - })?, - )?; - - Ok(metadata.marketplace_url) - } -} diff --git a/core/startos/src/backup/os.rs b/core/startos/src/backup/os.rs deleted file mode 100644 index 5ab8bd12..00000000 --- a/core/startos/src/backup/os.rs +++ /dev/null @@ -1,122 +0,0 @@ -use openssl::pkey::PKey; -use openssl::x509::X509; -use patch_db::Value; -use serde::{Deserialize, Serialize}; - -use crate::account::AccountInfo; -use crate::hostname::{generate_hostname, generate_id, Hostname}; -use crate::net::keys::Key; -use crate::prelude::*; -use crate::util::serde::Base64; - -pub struct OsBackup { - pub account: AccountInfo, - pub ui: Value, -} -impl<'de> Deserialize<'de> for OsBackup { - fn deserialize(deserializer: D) -> Result - where - D: serde::Deserializer<'de>, - { - let tagged = OsBackupSerDe::deserialize(deserializer)?; - match tagged.version { - 0 => patch_db::value::from_value::(tagged.rest) - .map_err(serde::de::Error::custom)? - .project() - .map_err(serde::de::Error::custom), - 1 => patch_db::value::from_value::(tagged.rest) - .map_err(serde::de::Error::custom)? - .project() - .map_err(serde::de::Error::custom), - v => Err(serde::de::Error::custom(&format!( - "Unknown backup version {v}" - ))), - } - } -} -impl Serialize for OsBackup { - fn serialize(&self, serializer: S) -> Result - where - S: serde::Serializer, - { - OsBackupSerDe { - version: 1, - rest: patch_db::value::to_value( - &OsBackupV1::unproject(self).map_err(serde::ser::Error::custom)?, - ) - .map_err(serde::ser::Error::custom)?, - } - .serialize(serializer) - } -} - -#[derive(Deserialize, Serialize)] -struct OsBackupSerDe { - #[serde(default)] - version: usize, - #[serde(flatten)] - rest: Value, -} - -/// V0 -#[derive(Deserialize)] -#[serde(rename = "kebab-case")] -struct OsBackupV0 { - // tor_key: Base32<[u8; 64]>, - root_ca_key: String, // PEM Encoded OpenSSL Key - root_ca_cert: String, // PEM Encoded OpenSSL X509 Certificate - ui: Value, // JSON Value -} -impl OsBackupV0 { - fn project(self) -> Result { - Ok(OsBackup { - account: AccountInfo { - server_id: generate_id(), - hostname: generate_hostname(), - password: Default::default(), - key: Key::new(None), - root_ca_key: PKey::private_key_from_pem(self.root_ca_key.as_bytes())?, - root_ca_cert: X509::from_pem(self.root_ca_cert.as_bytes())?, - }, - ui: self.ui, - }) - } -} - -/// V1 -#[derive(Deserialize, Serialize)] -#[serde(rename = "kebab-case")] -struct OsBackupV1 { - server_id: String, // uuidv4 - hostname: String, // embassy-- - net_key: Base64<[u8; 32]>, // Ed25519 Secret Key - root_ca_key: String, // PEM Encoded OpenSSL Key - root_ca_cert: String, // PEM Encoded OpenSSL X509 Certificate - ui: Value, // JSON Value - // TODO add more -} -impl OsBackupV1 { - fn project(self) -> Result { - Ok(OsBackup { - account: AccountInfo { - server_id: self.server_id, - hostname: Hostname(self.hostname), - password: Default::default(), - key: Key::from_bytes(None, self.net_key.0), - root_ca_key: PKey::private_key_from_pem(self.root_ca_key.as_bytes())?, - root_ca_cert: X509::from_pem(self.root_ca_cert.as_bytes())?, - }, - ui: self.ui, - }) - } - fn unproject(backup: &OsBackup) -> Result { - Ok(Self { - server_id: backup.account.server_id.clone(), - hostname: backup.account.hostname.0.clone(), - net_key: Base64(backup.account.key.as_bytes()), - root_ca_key: String::from_utf8(backup.account.root_ca_key.private_key_to_pem_pkcs8()?)?, - root_ca_cert: String::from_utf8(backup.account.root_ca_cert.to_pem()?)?, - ui: backup.ui.clone(), - }) - } -} diff --git a/core/startos/src/backup/restore.rs b/core/startos/src/backup/restore.rs deleted file mode 100644 index b72b319e..00000000 --- a/core/startos/src/backup/restore.rs +++ /dev/null @@ -1,461 +0,0 @@ -use std::collections::BTreeMap; -use std::path::Path; -use std::sync::atomic::Ordering; -use std::sync::Arc; -use std::time::Duration; - -use clap::ArgMatches; -use futures::future::BoxFuture; -use futures::{stream, FutureExt, StreamExt}; -use openssl::x509::X509; -use rpc_toolkit::command; -use sqlx::Connection; -use tokio::fs::File; -use torut::onion::OnionAddressV3; -use tracing::instrument; - -use super::target::BackupTargetId; -use crate::backup::os::OsBackup; -use crate::backup::BackupMetadata; -use crate::context::rpc::RpcContextConfig; -use crate::context::{RpcContext, SetupContext}; -use crate::db::model::{PackageDataEntry, PackageDataEntryRestoring, StaticFiles}; -use crate::disk::mount::backup::{BackupMountGuard, PackageBackupMountGuard}; -use crate::disk::mount::filesystem::ReadWrite; -use crate::disk::mount::guard::TmpMountGuard; -use crate::hostname::Hostname; -use crate::init::init; -use crate::install::progress::InstallProgress; -use crate::install::{download_install_s9pk, PKG_PUBLIC_DIR}; -use crate::notifications::NotificationLevel; -use crate::prelude::*; -use crate::s9pk::manifest::{Manifest, PackageId}; -use crate::s9pk::reader::S9pkReader; -use crate::setup::SetupStatus; -use crate::util::display_none; -use crate::util::io::dir_size; -use crate::util::serde::IoFormat; -use crate::volume::{backup_dir, BACKUP_DIR, PKG_VOLUME_DIR}; - -fn parse_comma_separated(arg: &str, _: &ArgMatches) -> Result, Error> { - arg.split(',') - .map(|s| s.trim().parse().map_err(Error::from)) - .collect() -} - -#[command(rename = "restore", display(display_none))] -#[instrument(skip(ctx, password))] -pub async fn restore_packages_rpc( - #[context] ctx: RpcContext, - #[arg(parse(parse_comma_separated))] ids: Vec, - #[arg(rename = "target-id")] target_id: BackupTargetId, - #[arg] password: String, -) -> Result<(), Error> { - let fs = target_id - .load(ctx.secret_store.acquire().await?.as_mut()) - .await?; - let backup_guard = - BackupMountGuard::mount(TmpMountGuard::mount(&fs, ReadWrite).await?, &password).await?; - - let (backup_guard, tasks, _) = restore_packages(&ctx, backup_guard, ids).await?; - - tokio::spawn(async move { - stream::iter(tasks.into_iter().map(|x| (x, ctx.clone()))) - .for_each_concurrent(5, |(res, ctx)| async move { - match res.await { - (Ok(_), _) => (), - (Err(err), package_id) => { - if let Err(err) = ctx - .notification_manager - .notify( - ctx.db.clone(), - Some(package_id.clone()), - NotificationLevel::Error, - "Restoration Failure".to_string(), - format!("Error restoring package {}: {}", package_id, err), - (), - None, - ) - .await - { - tracing::error!("Failed to notify: {}", err); - tracing::debug!("{:?}", err); - }; - tracing::error!("Error restoring package {}: {}", package_id, err); - tracing::debug!("{:?}", err); - } - } - }) - .await; - if let Err(e) = backup_guard.unmount().await { - tracing::error!("Error unmounting backup drive: {}", e); - tracing::debug!("{:?}", e); - } - }); - - Ok(()) -} - -async fn approximate_progress( - rpc_ctx: &RpcContext, - progress: &mut ProgressInfo, -) -> Result<(), Error> { - for (id, size) in &mut progress.target_volume_size { - let dir = rpc_ctx.datadir.join(PKG_VOLUME_DIR).join(id).join("data"); - if tokio::fs::metadata(&dir).await.is_err() { - *size = 0; - } else { - *size = dir_size(&dir, None).await?; - } - } - Ok(()) -} - -async fn approximate_progress_loop( - ctx: &SetupContext, - rpc_ctx: &RpcContext, - mut starting_info: ProgressInfo, -) { - loop { - if let Err(e) = approximate_progress(rpc_ctx, &mut starting_info).await { - tracing::error!("Failed to approximate restore progress: {}", e); - tracing::debug!("{:?}", e); - } else { - *ctx.setup_status.write().await = Some(Ok(starting_info.flatten())); - } - tokio::time::sleep(Duration::from_secs(1)).await; - } -} - -#[derive(Debug, Default)] -struct ProgressInfo { - package_installs: BTreeMap>, - src_volume_size: BTreeMap, - target_volume_size: BTreeMap, -} -impl ProgressInfo { - fn flatten(&self) -> SetupStatus { - let mut total_bytes = 0; - let mut bytes_transferred = 0; - - for progress in self.package_installs.values() { - total_bytes += ((progress.size.unwrap_or(0) as f64) * 2.2) as u64; - bytes_transferred += progress.downloaded.load(Ordering::SeqCst); - bytes_transferred += ((progress.validated.load(Ordering::SeqCst) as f64) * 0.2) as u64; - bytes_transferred += progress.unpacked.load(Ordering::SeqCst); - } - - for size in self.src_volume_size.values() { - total_bytes += *size; - } - - for size in self.target_volume_size.values() { - bytes_transferred += *size; - } - - if bytes_transferred > total_bytes { - bytes_transferred = total_bytes; - } - - SetupStatus { - total_bytes: Some(total_bytes), - bytes_transferred, - complete: false, - } - } -} - -#[instrument(skip(ctx))] -pub async fn recover_full_embassy( - ctx: SetupContext, - disk_guid: Arc, - embassy_password: String, - recovery_source: TmpMountGuard, - recovery_password: Option, -) -> Result<(Arc, Hostname, OnionAddressV3, X509), Error> { - let backup_guard = BackupMountGuard::mount( - recovery_source, - recovery_password.as_deref().unwrap_or_default(), - ) - .await?; - - let os_backup_path = backup_guard.as_ref().join("os-backup.cbor"); - let mut os_backup: OsBackup = IoFormat::Cbor.from_slice( - &tokio::fs::read(&os_backup_path) - .await - .with_ctx(|_| (ErrorKind::Filesystem, os_backup_path.display().to_string()))?, - )?; - - os_backup.account.password = argon2::hash_encoded( - embassy_password.as_bytes(), - &rand::random::<[u8; 16]>()[..], - &argon2::Config::rfc9106_low_mem(), - ) - .with_kind(ErrorKind::PasswordHashGeneration)?; - - let secret_store = ctx.secret_store().await?; - - os_backup.account.save(&secret_store).await?; - - secret_store.close().await; - - let cfg = RpcContextConfig::load(ctx.config_path.clone()).await?; - - init(&cfg).await?; - - let rpc_ctx = RpcContext::init(ctx.config_path.clone(), disk_guid.clone()).await?; - - let ids: Vec<_> = backup_guard - .metadata - .package_backups - .keys() - .cloned() - .collect(); - let (backup_guard, tasks, progress_info) = - restore_packages(&rpc_ctx, backup_guard, ids).await?; - let task_consumer_rpc_ctx = rpc_ctx.clone(); - tokio::select! { - _ = async move { - stream::iter(tasks.into_iter().map(|x| (x, task_consumer_rpc_ctx.clone()))) - .for_each_concurrent(5, |(res, ctx)| async move { - match res.await { - (Ok(_), _) => (), - (Err(err), package_id) => { - if let Err(err) = ctx.notification_manager.notify( - ctx.db.clone(), - Some(package_id.clone()), - NotificationLevel::Error, - "Restoration Failure".to_string(), format!("Error restoring package {}: {}", package_id,err), (), None).await{ - tracing::error!("Failed to notify: {}", err); - tracing::debug!("{:?}", err); - }; - tracing::error!("Error restoring package {}: {}", package_id, err); - tracing::debug!("{:?}", err); - }, - } - }).await; - - } => { - - }, - _ = approximate_progress_loop(&ctx, &rpc_ctx, progress_info) => unreachable!(concat!(module_path!(), "::approximate_progress_loop should not terminate")), - } - - backup_guard.unmount().await?; - rpc_ctx.shutdown().await?; - - Ok(( - disk_guid, - os_backup.account.hostname, - os_backup.account.key.tor_address(), - os_backup.account.root_ca_cert, - )) -} - -#[instrument(skip(ctx, backup_guard))] -async fn restore_packages( - ctx: &RpcContext, - backup_guard: BackupMountGuard, - ids: Vec, -) -> Result< - ( - BackupMountGuard, - Vec, PackageId)>>, - ProgressInfo, - ), - Error, -> { - let guards = assure_restoring(ctx, ids, &backup_guard).await?; - - let mut progress_info = ProgressInfo::default(); - - let mut tasks = Vec::with_capacity(guards.len()); - for (manifest, guard) in guards { - let id = manifest.id.clone(); - let (progress, task) = restore_package(ctx.clone(), manifest, guard).await?; - progress_info - .package_installs - .insert(id.clone(), progress.clone()); - progress_info - .src_volume_size - .insert(id.clone(), dir_size(backup_dir(&id), None).await?); - progress_info.target_volume_size.insert(id.clone(), 0); - let package_id = id.clone(); - tasks.push( - async move { - if let Err(e) = task.await { - tracing::error!("Error restoring package {}: {}", id, e); - tracing::debug!("{:?}", e); - Err(e) - } else { - Ok(()) - } - } - .map(|x| (x, package_id)) - .boxed(), - ); - } - - Ok((backup_guard, tasks, progress_info)) -} - -#[instrument(skip(ctx, backup_guard))] -async fn assure_restoring( - ctx: &RpcContext, - ids: Vec, - backup_guard: &BackupMountGuard, -) -> Result, Error> { - let mut guards = Vec::with_capacity(ids.len()); - - let mut insert_packages = BTreeMap::new(); - - for id in ids { - let peek = ctx.db.peek().await; - - let model = peek.as_package_data().as_idx(&id); - - if !model.is_none() { - return Err(Error::new( - eyre!("Can't restore over existing package: {}", id), - crate::ErrorKind::InvalidRequest, - )); - } - let guard = backup_guard.mount_package_backup(&id).await?; - let s9pk_path = Path::new(BACKUP_DIR).join(&id).join(format!("{}.s9pk", id)); - let mut rdr = S9pkReader::open(&s9pk_path, false).await?; - - let manifest = rdr.manifest().await?; - let version = manifest.version.clone(); - let progress = Arc::new(InstallProgress::new(Some( - tokio::fs::metadata(&s9pk_path).await?.len(), - ))); - - let public_dir_path = ctx - .datadir - .join(PKG_PUBLIC_DIR) - .join(&id) - .join(version.as_str()); - tokio::fs::create_dir_all(&public_dir_path).await?; - - let license_path = public_dir_path.join("LICENSE.md"); - let mut dst = File::create(&license_path).await?; - tokio::io::copy(&mut rdr.license().await?, &mut dst).await?; - dst.sync_all().await?; - - let instructions_path = public_dir_path.join("INSTRUCTIONS.md"); - let mut dst = File::create(&instructions_path).await?; - tokio::io::copy(&mut rdr.instructions().await?, &mut dst).await?; - dst.sync_all().await?; - - let icon_path = Path::new("icon").with_extension(&manifest.assets.icon_type()); - let icon_path = public_dir_path.join(&icon_path); - let mut dst = File::create(&icon_path).await?; - tokio::io::copy(&mut rdr.icon().await?, &mut dst).await?; - dst.sync_all().await?; - insert_packages.insert( - id.clone(), - PackageDataEntry::Restoring(PackageDataEntryRestoring { - install_progress: progress.clone(), - static_files: StaticFiles::local(&id, &version, manifest.assets.icon_type()), - manifest: manifest.clone(), - }), - ); - - guards.push((manifest, guard)); - } - ctx.db - .mutate(|db| { - for (id, package) in insert_packages { - db.as_package_data_mut().insert(&id, &package)?; - } - Ok(()) - }) - .await?; - Ok(guards) -} - -#[instrument(skip(ctx, guard))] -async fn restore_package<'a>( - ctx: RpcContext, - manifest: Manifest, - guard: PackageBackupMountGuard, -) -> Result<(Arc, BoxFuture<'static, Result<(), Error>>), Error> { - let id = manifest.id.clone(); - let s9pk_path = Path::new(BACKUP_DIR) - .join(&manifest.id) - .join(format!("{}.s9pk", id)); - - let metadata_path = Path::new(BACKUP_DIR).join(&id).join("metadata.cbor"); - let metadata: BackupMetadata = IoFormat::Cbor.from_slice( - &tokio::fs::read(&metadata_path) - .await - .with_ctx(|_| (ErrorKind::Filesystem, metadata_path.display().to_string()))?, - )?; - - let mut secrets = ctx.secret_store.acquire().await?; - let mut secrets_tx = secrets.begin().await?; - for (iface, key) in metadata.network_keys { - let k = key.0.as_slice(); - sqlx::query!( - "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING", - id.to_string(), - iface.to_string(), - k, - ) - .execute(secrets_tx.as_mut()).await?; - } - // DEPRECATED - for (iface, key) in metadata.tor_keys { - let k = key.0.as_slice(); - sqlx::query!( - "INSERT INTO tor (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING", - id.to_string(), - iface.to_string(), - k, - ) - .execute(secrets_tx.as_mut()).await?; - } - secrets_tx.commit().await?; - drop(secrets); - - let len = tokio::fs::metadata(&s9pk_path) - .await - .with_ctx(|_| (ErrorKind::Filesystem, s9pk_path.display().to_string()))? - .len(); - let file = File::open(&s9pk_path) - .await - .with_ctx(|_| (ErrorKind::Filesystem, s9pk_path.display().to_string()))?; - - let progress = InstallProgress::new(Some(len)); - let marketplace_url = metadata.marketplace_url; - - let progress = Arc::new(progress); - - ctx.db - .mutate(|db| { - db.as_package_data_mut().insert( - &id, - &PackageDataEntry::Restoring(PackageDataEntryRestoring { - install_progress: progress.clone(), - static_files: StaticFiles::local( - &id, - &manifest.version, - manifest.assets.icon_type(), - ), - manifest: manifest.clone(), - }), - ) - }) - .await?; - Ok(( - progress.clone(), - async move { - download_install_s9pk(ctx, manifest, marketplace_url, progress, file, None).await?; - - guard.unmount().await?; - - Ok(()) - } - .boxed(), - )) -} diff --git a/core/startos/src/backup/target/cifs.rs b/core/startos/src/backup/target/cifs.rs deleted file mode 100644 index 3f325153..00000000 --- a/core/startos/src/backup/target/cifs.rs +++ /dev/null @@ -1,211 +0,0 @@ -use std::path::{Path, PathBuf}; - -use color_eyre::eyre::eyre; -use futures::TryStreamExt; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use sqlx::{Executor, Postgres}; - -use super::{BackupTarget, BackupTargetId}; -use crate::context::RpcContext; -use crate::disk::mount::filesystem::cifs::Cifs; -use crate::disk::mount::filesystem::ReadOnly; -use crate::disk::mount::guard::TmpMountGuard; -use crate::disk::util::{recovery_info, EmbassyOsRecoveryInfo}; -use crate::prelude::*; -use crate::util::display_none; -use crate::util::serde::KeyVal; - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct CifsBackupTarget { - hostname: String, - path: PathBuf, - username: String, - mountable: bool, - embassy_os: Option, -} - -#[command(subcommands(add, update, remove))] -pub fn cifs() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_none))] -pub async fn add( - #[context] ctx: RpcContext, - #[arg] hostname: String, - #[arg] path: PathBuf, - #[arg] username: String, - #[arg] password: Option, -) -> Result, Error> { - let cifs = Cifs { - hostname, - path, - username, - password, - }; - let guard = TmpMountGuard::mount(&cifs, ReadOnly).await?; - let embassy_os = recovery_info(&guard).await?; - guard.unmount().await?; - let path_string = Path::new("/").join(&cifs.path).display().to_string(); - let id: i32 = sqlx::query!( - "INSERT INTO cifs_shares (hostname, path, username, password) VALUES ($1, $2, $3, $4) RETURNING id", - cifs.hostname, - path_string, - cifs.username, - cifs.password, - ) - .fetch_one(&ctx.secret_store) - .await?.id; - Ok(KeyVal { - key: BackupTargetId::Cifs { id }, - value: BackupTarget::Cifs(CifsBackupTarget { - hostname: cifs.hostname, - path: cifs.path, - username: cifs.username, - mountable: true, - embassy_os, - }), - }) -} - -#[command(display(display_none))] -pub async fn update( - #[context] ctx: RpcContext, - #[arg] id: BackupTargetId, - #[arg] hostname: String, - #[arg] path: PathBuf, - #[arg] username: String, - #[arg] password: Option, -) -> Result, Error> { - let id = if let BackupTargetId::Cifs { id } = id { - id - } else { - return Err(Error::new( - eyre!("Backup Target ID {} Not Found", id), - ErrorKind::NotFound, - )); - }; - let cifs = Cifs { - hostname, - path, - username, - password, - }; - let guard = TmpMountGuard::mount(&cifs, ReadOnly).await?; - let embassy_os = recovery_info(&guard).await?; - guard.unmount().await?; - let path_string = Path::new("/").join(&cifs.path).display().to_string(); - if sqlx::query!( - "UPDATE cifs_shares SET hostname = $1, path = $2, username = $3, password = $4 WHERE id = $5", - cifs.hostname, - path_string, - cifs.username, - cifs.password, - id, - ) - .execute(&ctx.secret_store) - .await? - .rows_affected() - == 0 - { - return Err(Error::new( - eyre!("Backup Target ID {} Not Found", BackupTargetId::Cifs { id }), - ErrorKind::NotFound, - )); - }; - Ok(KeyVal { - key: BackupTargetId::Cifs { id }, - value: BackupTarget::Cifs(CifsBackupTarget { - hostname: cifs.hostname, - path: cifs.path, - username: cifs.username, - mountable: true, - embassy_os, - }), - }) -} - -#[command(display(display_none))] -pub async fn remove(#[context] ctx: RpcContext, #[arg] id: BackupTargetId) -> Result<(), Error> { - let id = if let BackupTargetId::Cifs { id } = id { - id - } else { - return Err(Error::new( - eyre!("Backup Target ID {} Not Found", id), - ErrorKind::NotFound, - )); - }; - if sqlx::query!("DELETE FROM cifs_shares WHERE id = $1", id) - .execute(&ctx.secret_store) - .await? - .rows_affected() - == 0 - { - return Err(Error::new( - eyre!("Backup Target ID {} Not Found", BackupTargetId::Cifs { id }), - ErrorKind::NotFound, - )); - }; - Ok(()) -} - -pub async fn load(secrets: &mut Ex, id: i32) -> Result -where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, -{ - let record = sqlx::query!( - "SELECT hostname, path, username, password FROM cifs_shares WHERE id = $1", - id - ) - .fetch_one(secrets) - .await?; - - Ok(Cifs { - hostname: record.hostname, - path: PathBuf::from(record.path), - username: record.username, - password: record.password, - }) -} - -pub async fn list(secrets: &mut Ex) -> Result, Error> -where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, -{ - let mut records = - sqlx::query!("SELECT id, hostname, path, username, password FROM cifs_shares") - .fetch_many(secrets); - - let mut cifs = Vec::new(); - while let Some(query_result) = records.try_next().await? { - if let Some(record) = query_result.right() { - let mount_info = Cifs { - hostname: record.hostname, - path: PathBuf::from(record.path), - username: record.username, - password: record.password, - }; - let embassy_os = async { - let guard = TmpMountGuard::mount(&mount_info, ReadOnly).await?; - let embassy_os = recovery_info(&guard).await?; - guard.unmount().await?; - Ok::<_, Error>(embassy_os) - } - .await; - cifs.push(( - record.id, - CifsBackupTarget { - hostname: mount_info.hostname, - path: mount_info.path, - username: mount_info.username, - mountable: embassy_os.is_ok(), - embassy_os: embassy_os.ok().and_then(|a| a), - }, - )); - } - } - - Ok(cifs) -} diff --git a/core/startos/src/backup/target/mod.rs b/core/startos/src/backup/target/mod.rs deleted file mode 100644 index 93e56c2d..00000000 --- a/core/startos/src/backup/target/mod.rs +++ /dev/null @@ -1,307 +0,0 @@ -use std::collections::BTreeMap; -use std::path::{Path, PathBuf}; - -use async_trait::async_trait; -use chrono::{DateTime, Utc}; -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use digest::generic_array::GenericArray; -use digest::OutputSizeUser; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use sha2::Sha256; -use sqlx::{Executor, Postgres}; -use tokio::sync::Mutex; -use tracing::instrument; - -use self::cifs::CifsBackupTarget; -use crate::context::RpcContext; -use crate::disk::mount::backup::BackupMountGuard; -use crate::disk::mount::filesystem::block_dev::BlockDev; -use crate::disk::mount::filesystem::cifs::Cifs; -use crate::disk::mount::filesystem::{FileSystem, MountType, ReadWrite}; -use crate::disk::mount::guard::TmpMountGuard; -use crate::disk::util::PartitionInfo; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::{deserialize_from_str, display_serializable, serialize_display}; -use crate::util::{display_none, Version}; - -pub mod cifs; - -#[derive(Debug, Deserialize, Serialize)] -#[serde(tag = "type")] -#[serde(rename_all = "kebab-case")] -pub enum BackupTarget { - #[serde(rename_all = "kebab-case")] - Disk { - vendor: Option, - model: Option, - #[serde(flatten)] - partition_info: PartitionInfo, - }, - Cifs(CifsBackupTarget), -} - -#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone)] -pub enum BackupTargetId { - Disk { logicalname: PathBuf }, - Cifs { id: i32 }, -} -impl BackupTargetId { - pub async fn load(self, secrets: &mut Ex) -> Result - where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, - { - Ok(match self { - BackupTargetId::Disk { logicalname } => { - BackupTargetFS::Disk(BlockDev::new(logicalname)) - } - BackupTargetId::Cifs { id } => BackupTargetFS::Cifs(cifs::load(secrets, id).await?), - }) - } -} -impl std::fmt::Display for BackupTargetId { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - BackupTargetId::Disk { logicalname } => write!(f, "disk-{}", logicalname.display()), - BackupTargetId::Cifs { id } => write!(f, "cifs-{}", id), - } - } -} -impl std::str::FromStr for BackupTargetId { - type Err = Error; - fn from_str(s: &str) -> Result { - match s.split_once('-') { - Some(("disk", logicalname)) => Ok(BackupTargetId::Disk { - logicalname: Path::new(logicalname).to_owned(), - }), - Some(("cifs", id)) => Ok(BackupTargetId::Cifs { id: id.parse()? }), - _ => Err(Error::new( - eyre!("Invalid Backup Target ID"), - ErrorKind::InvalidBackupTargetId, - )), - } - } -} -impl<'de> Deserialize<'de> for BackupTargetId { - fn deserialize(deserializer: D) -> Result - where - D: serde::Deserializer<'de>, - { - deserialize_from_str(deserializer) - } -} -impl Serialize for BackupTargetId { - fn serialize(&self, serializer: S) -> Result - where - S: serde::Serializer, - { - serialize_display(self, serializer) - } -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(tag = "type")] -#[serde(rename_all = "kebab-case")] -pub enum BackupTargetFS { - Disk(BlockDev), - Cifs(Cifs), -} -#[async_trait] -impl FileSystem for BackupTargetFS { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error> { - match self { - BackupTargetFS::Disk(a) => a.mount(mountpoint, mount_type).await, - BackupTargetFS::Cifs(a) => a.mount(mountpoint, mount_type).await, - } - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - match self { - BackupTargetFS::Disk(a) => a.source_hash().await, - BackupTargetFS::Cifs(a) => a.source_hash().await, - } - } -} - -#[command(subcommands(cifs::cifs, list, info, mount, umount))] -pub fn target() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_serializable))] -pub async fn list( - #[context] ctx: RpcContext, -) -> Result, Error> { - let mut sql_handle = ctx.secret_store.acquire().await?; - let (disks_res, cifs) = tokio::try_join!( - crate::disk::util::list(&ctx.os_partitions), - cifs::list(sql_handle.as_mut()), - )?; - Ok(disks_res - .into_iter() - .flat_map(|mut disk| { - std::mem::take(&mut disk.partitions) - .into_iter() - .map(|part| { - ( - BackupTargetId::Disk { - logicalname: part.logicalname.clone(), - }, - BackupTarget::Disk { - vendor: disk.vendor.clone(), - model: disk.model.clone(), - partition_info: part, - }, - ) - }) - .collect::>() - }) - .chain( - cifs.into_iter() - .map(|(id, cifs)| (BackupTargetId::Cifs { id }, BackupTarget::Cifs(cifs))), - ) - .collect()) -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct BackupInfo { - pub version: Version, - pub timestamp: Option>, - pub package_backups: BTreeMap, -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct PackageBackupInfo { - pub title: String, - pub version: Version, - pub os_version: Version, - pub timestamp: DateTime, -} - -fn display_backup_info(info: BackupInfo, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(info, matches); - } - - let mut table = Table::new(); - table.add_row(row![bc => - "ID", - "VERSION", - "OS VERSION", - "TIMESTAMP", - ]); - table.add_row(row![ - "EMBASSY OS", - info.version.as_str(), - info.version.as_str(), - &if let Some(ts) = &info.timestamp { - ts.to_string() - } else { - "N/A".to_owned() - }, - ]); - for (id, info) in info.package_backups { - let row = row![ - &*id, - info.version.as_str(), - info.os_version.as_str(), - &info.timestamp.to_string(), - ]; - table.add_row(row); - } - table.print_tty(false).unwrap(); -} - -#[command(display(display_backup_info))] -#[instrument(skip(ctx, password))] -pub async fn info( - #[context] ctx: RpcContext, - #[arg(rename = "target-id")] target_id: BackupTargetId, - #[arg] password: String, -) -> Result { - let guard = BackupMountGuard::mount( - TmpMountGuard::mount( - &target_id - .load(ctx.secret_store.acquire().await?.as_mut()) - .await?, - ReadWrite, - ) - .await?, - &password, - ) - .await?; - - let res = guard.metadata.clone(); - - guard.unmount().await?; - - Ok(res) -} - -lazy_static::lazy_static! { - static ref USER_MOUNTS: Mutex>> = - Mutex::new(BTreeMap::new()); -} - -#[command] -#[instrument(skip_all)] -pub async fn mount( - #[context] ctx: RpcContext, - #[arg(rename = "target-id")] target_id: BackupTargetId, - #[arg] password: String, -) -> Result { - let mut mounts = USER_MOUNTS.lock().await; - - if let Some(existing) = mounts.get(&target_id) { - return Ok(existing.as_ref().display().to_string()); - } - - let guard = BackupMountGuard::mount( - TmpMountGuard::mount( - &target_id - .clone() - .load(ctx.secret_store.acquire().await?.as_mut()) - .await?, - ReadWrite, - ) - .await?, - &password, - ) - .await?; - - let res = guard.as_ref().display().to_string(); - - mounts.insert(target_id, guard); - - Ok(res) -} -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn umount( - #[context] _ctx: RpcContext, - #[arg(rename = "target-id")] target_id: Option, -) -> Result<(), Error> { - let mut mounts = USER_MOUNTS.lock().await; - if let Some(target_id) = target_id { - if let Some(existing) = mounts.remove(&target_id) { - existing.unmount().await?; - } - } else { - for (_, existing) in std::mem::take(&mut *mounts) { - existing.unmount().await?; - } - } - - Ok(()) -} diff --git a/core/startos/src/bins/avahi_alias.rs b/core/startos/src/bins/avahi_alias.rs deleted file mode 100644 index 3c4a4fe7..00000000 --- a/core/startos/src/bins/avahi_alias.rs +++ /dev/null @@ -1,163 +0,0 @@ -use avahi_sys::{ - self, avahi_client_errno, avahi_entry_group_add_service, avahi_entry_group_commit, - avahi_strerror, AvahiClient, -}; - -fn log_str_error(action: &str, e: i32) { - unsafe { - let e_str = avahi_strerror(e); - eprintln!( - "Could not {}: {:?}", - action, - std::ffi::CStr::from_ptr(e_str) - ); - } -} - -pub fn main() { - let aliases: Vec<_> = std::env::args().skip(1).collect(); - unsafe { - let simple_poll = avahi_sys::avahi_simple_poll_new(); - let poll = avahi_sys::avahi_simple_poll_get(simple_poll); - let mut box_err = Box::pin(0 as i32); - let err_c: *mut i32 = box_err.as_mut().get_mut(); - let avahi_client = avahi_sys::avahi_client_new( - poll, - avahi_sys::AvahiClientFlags::AVAHI_CLIENT_NO_FAIL, - Some(client_callback), - std::ptr::null_mut(), - err_c, - ); - if avahi_client == std::ptr::null_mut::() { - log_str_error("create Avahi client", *box_err); - panic!("Failed to create Avahi Client"); - } - let group = avahi_sys::avahi_entry_group_new( - avahi_client, - Some(entry_group_callback), - std::ptr::null_mut(), - ); - if group == std::ptr::null_mut() { - log_str_error("create Avahi entry group", avahi_client_errno(avahi_client)); - panic!("Failed to create Avahi Entry Group"); - } - let mut hostname_buf = vec![0]; - let hostname_raw = avahi_sys::avahi_client_get_host_name_fqdn(avahi_client); - hostname_buf.extend_from_slice(std::ffi::CStr::from_ptr(hostname_raw).to_bytes_with_nul()); - let buflen = hostname_buf.len(); - debug_assert!(hostname_buf.ends_with(b".local\0")); - debug_assert!(!hostname_buf[..(buflen - 7)].contains(&b'.')); - // assume fixed length prefix on hostname due to local address - hostname_buf[0] = (buflen - 8) as u8; // set the prefix length to len - 8 (leading byte, .local, nul) for the main address - hostname_buf[buflen - 7] = 5; // set the prefix length to 5 for "local" - let mut res; - let http_tcp_cstr = - std::ffi::CString::new("_http._tcp").expect("Could not cast _http._tcp to c string"); - res = avahi_entry_group_add_service( - group, - avahi_sys::AVAHI_IF_UNSPEC, - avahi_sys::AVAHI_PROTO_UNSPEC, - avahi_sys::AvahiPublishFlags_AVAHI_PUBLISH_USE_MULTICAST, - hostname_raw, - http_tcp_cstr.as_ptr(), - std::ptr::null(), - std::ptr::null(), - 443, - // below is a secret final argument that the type signature of this function does not tell you that it - // needs. This is because the C lib function takes a variable number of final arguments indicating the - // desired TXT records to add to this service entry. The way it decides when to stop taking arguments - // from the stack and dereferencing them is when it finds a null pointer...because fuck you, that's why. - // The consequence of this is that forgetting this last argument will cause segfaults or other undefined - // behavior. Welcome back to the stone age motherfucker. - std::ptr::null::(), - ); - if res < avahi_sys::AVAHI_OK { - log_str_error("add service to Avahi entry group", res); - panic!("Failed to load Avahi services"); - } - eprintln!("Published {:?}", std::ffi::CStr::from_ptr(hostname_raw)); - for alias in aliases { - let lan_address = alias + ".local"; - let lan_address_ptr = std::ffi::CString::new(lan_address) - .expect("Could not cast lan address to c string"); - res = avahi_sys::avahi_entry_group_add_record( - group, - avahi_sys::AVAHI_IF_UNSPEC, - avahi_sys::AVAHI_PROTO_UNSPEC, - avahi_sys::AvahiPublishFlags_AVAHI_PUBLISH_USE_MULTICAST - | avahi_sys::AvahiPublishFlags_AVAHI_PUBLISH_ALLOW_MULTIPLE, - lan_address_ptr.as_ptr(), - avahi_sys::AVAHI_DNS_CLASS_IN as u16, - avahi_sys::AVAHI_DNS_TYPE_CNAME as u16, - avahi_sys::AVAHI_DEFAULT_TTL, - hostname_buf.as_ptr().cast(), - hostname_buf.len(), - ); - if res < avahi_sys::AVAHI_OK { - log_str_error("add CNAME record to Avahi entry group", res); - panic!("Failed to load Avahi services"); - } - eprintln!("Published {:?}", lan_address_ptr); - } - let commit_err = avahi_entry_group_commit(group); - if commit_err < avahi_sys::AVAHI_OK { - log_str_error("reset Avahi entry group", commit_err); - panic!("Failed to load Avahi services: reset"); - } - } - std::thread::park() -} - -unsafe extern "C" fn entry_group_callback( - _group: *mut avahi_sys::AvahiEntryGroup, - state: avahi_sys::AvahiEntryGroupState, - _userdata: *mut core::ffi::c_void, -) { - match state { - avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_FAILURE => { - eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_FAILURE"); - } - avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_COLLISION => { - eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_COLLISION"); - } - avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_UNCOMMITED => { - eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_UNCOMMITED"); - } - avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_ESTABLISHED => { - eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_ESTABLISHED"); - } - avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_REGISTERING => { - eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_REGISTERING"); - } - other => { - eprintln!("AvahiCallback: EntryGroupState = {}", other); - } - } -} - -unsafe extern "C" fn client_callback( - _group: *mut avahi_sys::AvahiClient, - state: avahi_sys::AvahiClientState, - _userdata: *mut core::ffi::c_void, -) { - match state { - avahi_sys::AvahiClientState_AVAHI_CLIENT_FAILURE => { - eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_FAILURE"); - } - avahi_sys::AvahiClientState_AVAHI_CLIENT_S_RUNNING => { - eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_S_RUNNING"); - } - avahi_sys::AvahiClientState_AVAHI_CLIENT_CONNECTING => { - eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_CONNECTING"); - } - avahi_sys::AvahiClientState_AVAHI_CLIENT_S_COLLISION => { - eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_S_COLLISION"); - } - avahi_sys::AvahiClientState_AVAHI_CLIENT_S_REGISTERING => { - eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_S_REGISTERING"); - } - other => { - eprintln!("AvahiCallback: ClientState = {}", other); - } - } -} diff --git a/core/startos/src/bins/deprecated.rs b/core/startos/src/bins/deprecated.rs deleted file mode 100644 index 13e0290d..00000000 --- a/core/startos/src/bins/deprecated.rs +++ /dev/null @@ -1,9 +0,0 @@ -pub fn renamed(old: &str, new: &str) -> ! { - eprintln!("{old} has been renamed to {new}"); - std::process::exit(1) -} - -pub fn removed(name: &str) -> ! { - eprintln!("{name} has been removed"); - std::process::exit(1) -} diff --git a/core/startos/src/bins/mod.rs b/core/startos/src/bins/mod.rs deleted file mode 100644 index f9c88cae..00000000 --- a/core/startos/src/bins/mod.rs +++ /dev/null @@ -1,49 +0,0 @@ -use std::path::Path; - -#[cfg(feature = "avahi-alias")] -pub mod avahi_alias; -pub mod deprecated; -#[cfg(feature = "cli")] -pub mod start_cli; -#[cfg(feature = "js-engine")] -pub mod start_deno; -#[cfg(feature = "daemon")] -pub mod start_init; -#[cfg(feature = "sdk")] -pub mod start_sdk; -#[cfg(feature = "daemon")] -pub mod startd; - -fn select_executable(name: &str) -> Option { - match name { - #[cfg(feature = "avahi-alias")] - "avahi-alias" => Some(avahi_alias::main), - #[cfg(feature = "js-engine")] - "start-deno" => Some(start_deno::main), - #[cfg(feature = "cli")] - "start-cli" => Some(start_cli::main), - #[cfg(feature = "sdk")] - "start-sdk" => Some(start_sdk::main), - #[cfg(feature = "daemon")] - "startd" => Some(startd::main), - "embassy-cli" => Some(|| deprecated::renamed("embassy-cli", "start-cli")), - "embassy-sdk" => Some(|| deprecated::renamed("embassy-sdk", "start-sdk")), - "embassyd" => Some(|| deprecated::renamed("embassyd", "startd")), - "embassy-init" => Some(|| deprecated::removed("embassy-init")), - _ => None, - } -} - -pub fn startbox() { - let args = std::env::args().take(2).collect::>(); - let executable = args - .get(0) - .and_then(|s| Path::new(&*s).file_name()) - .and_then(|s| s.to_str()); - if let Some(x) = executable.and_then(|s| select_executable(&s)) { - x() - } else { - eprintln!("unknown executable: {}", executable.unwrap_or("N/A")); - std::process::exit(1); - } -} diff --git a/core/startos/src/bins/start_cli.rs b/core/startos/src/bins/start_cli.rs deleted file mode 100644 index 3ef64096..00000000 --- a/core/startos/src/bins/start_cli.rs +++ /dev/null @@ -1,62 +0,0 @@ -use clap::Arg; -use rpc_toolkit::run_cli; -use rpc_toolkit::yajrc::RpcError; -use serde_json::Value; - -use crate::context::CliContext; -use crate::util::logger::EmbassyLogger; -use crate::version::{Current, VersionT}; -use crate::Error; - -lazy_static::lazy_static! { - static ref VERSION_STRING: String = Current::new().semver().to_string(); -} - -fn inner_main() -> Result<(), Error> { - run_cli!({ - command: crate::main_api, - app: app => app - .name("StartOS CLI") - .version(&**VERSION_STRING) - .arg( - clap::Arg::with_name("config") - .short('c') - .long("config") - .takes_value(true), - ) - .arg(Arg::with_name("host").long("host").short('h').takes_value(true)) - .arg(Arg::with_name("proxy").long("proxy").short('p').takes_value(true)), - context: matches => { - EmbassyLogger::init(); - CliContext::init(matches)? - }, - exit: |e: RpcError| { - match e.data { - Some(Value::String(s)) => eprintln!("{}: {}", e.message, s), - Some(Value::Object(o)) => if let Some(Value::String(s)) = o.get("details") { - eprintln!("{}: {}", e.message, s); - if let Some(Value::String(s)) = o.get("debug") { - tracing::debug!("{}", s) - } - } - Some(a) => eprintln!("{}: {}", e.message, a), - None => eprintln!("{}", e.message), - } - - std::process::exit(e.code); - } - }); - Ok(()) -} - -pub fn main() { - match inner_main() { - Ok(_) => (), - Err(e) => { - eprintln!("{}", e.source); - tracing::debug!("{:?}", e.source); - drop(e.source); - std::process::exit(e.kind as i32) - } - } -} diff --git a/core/startos/src/bins/start_deno.rs b/core/startos/src/bins/start_deno.rs deleted file mode 100644 index 8f5a1451..00000000 --- a/core/startos/src/bins/start_deno.rs +++ /dev/null @@ -1,140 +0,0 @@ -use rpc_toolkit::yajrc::RpcError; -use rpc_toolkit::{command, run_cli, Context}; -use serde_json::Value; - -use crate::procedure::js_scripts::ExecuteArgs; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::{display_serializable, parse_stdin_deserializable, IoFormat}; -use crate::version::{Current, VersionT}; -use crate::Error; - -lazy_static::lazy_static! { - static ref VERSION_STRING: String = Current::new().semver().to_string(); -} - -struct DenoContext; -impl Context for DenoContext {} - -#[command(subcommands(execute, sandbox))] -fn deno_api() -> Result<(), Error> { - Ok(()) -} - -#[command(cli_only, display(display_serializable))] -async fn execute( - #[arg(stdin, parse(parse_stdin_deserializable))] arg: ExecuteArgs, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result, Error> { - let ExecuteArgs { - procedure, - directory, - pkg_id, - pkg_version, - name, - volumes, - input, - } = arg; - PackageLogger::init(&pkg_id); - procedure - .execute_impl(&directory, &pkg_id, &pkg_version, name, &volumes, input) - .await -} -#[command(cli_only, display(display_serializable))] -async fn sandbox( - #[arg(stdin, parse(parse_stdin_deserializable))] arg: ExecuteArgs, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result, Error> { - let ExecuteArgs { - procedure, - directory, - pkg_id, - pkg_version, - name, - volumes, - input, - } = arg; - PackageLogger::init(&pkg_id); - procedure - .sandboxed_impl(&directory, &pkg_id, &pkg_version, &volumes, input, name) - .await -} - -use tracing::Subscriber; -use tracing_subscriber::util::SubscriberInitExt; - -#[derive(Clone)] -struct PackageLogger {} - -impl PackageLogger { - fn base_subscriber(id: &PackageId) -> impl Subscriber { - use tracing_error::ErrorLayer; - use tracing_subscriber::prelude::*; - use tracing_subscriber::{fmt, EnvFilter}; - - let filter_layer = EnvFilter::default().add_directive( - format!("{}=warn", std::module_path!().split("::").next().unwrap()) - .parse() - .unwrap(), - ); - let fmt_layer = fmt::layer().with_writer(std::io::stderr).with_target(true); - let journald_layer = tracing_journald::layer() - .unwrap() - .with_syslog_identifier(format!("{id}.embassy")); - - let sub = tracing_subscriber::registry() - .with(filter_layer) - .with(fmt_layer) - .with(journald_layer) - .with(ErrorLayer::default()); - - sub - } - pub fn init(id: &PackageId) -> Self { - Self::base_subscriber(id).init(); - color_eyre::install().unwrap_or_else(|_| tracing::warn!("tracing too many times")); - - Self {} - } -} - -fn inner_main() -> Result<(), Error> { - run_cli!({ - command: deno_api, - app: app => app - .name("StartOS Deno Executor") - .version(&**VERSION_STRING), - context: _m => DenoContext, - exit: |e: RpcError| { - match e.data { - Some(Value::String(s)) => eprintln!("{}: {}", e.message, s), - Some(Value::Object(o)) => if let Some(Value::String(s)) = o.get("details") { - eprintln!("{}: {}", e.message, s); - if let Some(Value::String(s)) = o.get("debug") { - tracing::debug!("{}", s) - } - } - Some(a) => eprintln!("{}: {}", e.message, a), - None => eprintln!("{}", e.message), - } - - std::process::exit(e.code); - } - }); - Ok(()) -} - -pub fn main() { - match inner_main() { - Ok(_) => (), - Err(e) => { - eprintln!("{}", e.source); - tracing::debug!("{:?}", e.source); - drop(e.source); - std::process::exit(e.kind as i32) - } - } -} diff --git a/core/startos/src/bins/start_init.rs b/core/startos/src/bins/start_init.rs deleted file mode 100644 index 1cb07085..00000000 --- a/core/startos/src/bins/start_init.rs +++ /dev/null @@ -1,288 +0,0 @@ -use std::net::{Ipv6Addr, SocketAddr}; -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use std::time::Duration; - -use helpers::NonDetachingJoinHandle; -use tokio::process::Command; -use tracing::instrument; - -use crate::context::rpc::RpcContextConfig; -use crate::context::{DiagnosticContext, InstallContext, SetupContext}; -use crate::disk::fsck::{RepairStrategy, RequiresReboot}; -use crate::disk::main::DEFAULT_PASSWORD; -use crate::disk::REPAIR_DISK_PATH; -use crate::firmware::update_firmware; -use crate::init::STANDBY_MODE_PATH; -use crate::net::web_server::WebServer; -use crate::shutdown::Shutdown; -use crate::sound::{BEP, CHIME}; -use crate::util::Invoke; -use crate::{Error, ErrorKind, ResultExt, PLATFORM}; - -#[instrument(skip_all)] -async fn setup_or_init(cfg_path: Option) -> Result, Error> { - let song = NonDetachingJoinHandle::from(tokio::spawn(async { - loop { - BEP.play().await.unwrap(); - BEP.play().await.unwrap(); - tokio::time::sleep(Duration::from_secs(30)).await; - } - })); - - match update_firmware().await { - Ok(RequiresReboot(true)) => { - return Ok(Some(Shutdown { - export_args: None, - restart: true, - })) - } - Err(e) => { - tracing::warn!("Error performing firmware update: {e}"); - tracing::debug!("{e:?}"); - } - _ => (), - } - - Command::new("ln") - .arg("-sf") - .arg("/usr/lib/startos/scripts/fake-apt") - .arg("/usr/local/bin/apt") - .invoke(crate::ErrorKind::Filesystem) - .await?; - Command::new("ln") - .arg("-sf") - .arg("/usr/lib/startos/scripts/fake-apt") - .arg("/usr/local/bin/apt-get") - .invoke(crate::ErrorKind::Filesystem) - .await?; - Command::new("ln") - .arg("-sf") - .arg("/usr/lib/startos/scripts/fake-apt") - .arg("/usr/local/bin/aptitude") - .invoke(crate::ErrorKind::Filesystem) - .await?; - - Command::new("make-ssl-cert") - .arg("generate-default-snakeoil") - .arg("--force-overwrite") - .invoke(crate::ErrorKind::OpenSsl) - .await?; - - if tokio::fs::metadata("/run/live/medium").await.is_ok() { - Command::new("sed") - .arg("-i") - .arg("s/PasswordAuthentication no/PasswordAuthentication yes/g") - .arg("/etc/ssh/sshd_config") - .invoke(crate::ErrorKind::Filesystem) - .await?; - Command::new("systemctl") - .arg("reload") - .arg("ssh") - .invoke(crate::ErrorKind::OpenSsh) - .await?; - - let ctx = InstallContext::init(cfg_path).await?; - - let server = WebServer::install( - SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), 80), - ctx.clone(), - ) - .await?; - - drop(song); - tokio::time::sleep(Duration::from_secs(1)).await; // let the record state that I hate this - CHIME.play().await?; - - ctx.shutdown - .subscribe() - .recv() - .await - .expect("context dropped"); - - server.shutdown().await; - - Command::new("reboot") - .invoke(crate::ErrorKind::Unknown) - .await?; - } else if tokio::fs::metadata("/media/embassy/config/disk.guid") - .await - .is_err() - { - let ctx = SetupContext::init(cfg_path).await?; - - let server = WebServer::setup( - SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), 80), - ctx.clone(), - ) - .await?; - - drop(song); - tokio::time::sleep(Duration::from_secs(1)).await; // let the record state that I hate this - CHIME.play().await?; - - ctx.shutdown - .subscribe() - .recv() - .await - .expect("context dropped"); - - server.shutdown().await; - - tokio::task::yield_now().await; - if let Err(e) = Command::new("killall") - .arg("firefox-esr") - .invoke(ErrorKind::NotFound) - .await - { - tracing::error!("Failed to kill kiosk: {}", e); - tracing::debug!("{:?}", e); - } - } else { - let cfg = RpcContextConfig::load(cfg_path).await?; - let guid_string = tokio::fs::read_to_string("/media/embassy/config/disk.guid") // unique identifier for volume group - keeps track of the disk that goes with your embassy - .await?; - let guid = guid_string.trim(); - let requires_reboot = crate::disk::main::import( - guid, - cfg.datadir(), - if tokio::fs::metadata(REPAIR_DISK_PATH).await.is_ok() { - RepairStrategy::Aggressive - } else { - RepairStrategy::Preen - }, - if guid.ends_with("_UNENC") { - None - } else { - Some(DEFAULT_PASSWORD) - }, - ) - .await?; - if tokio::fs::metadata(REPAIR_DISK_PATH).await.is_ok() { - tokio::fs::remove_file(REPAIR_DISK_PATH) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, REPAIR_DISK_PATH))?; - } - if requires_reboot.0 { - crate::disk::main::export(guid, cfg.datadir()).await?; - Command::new("reboot") - .invoke(crate::ErrorKind::Unknown) - .await?; - } - tracing::info!("Loaded Disk"); - crate::init::init(&cfg).await?; - drop(song); - } - - Ok(None) -} - -async fn run_script_if_exists>(path: P) { - let script = path.as_ref(); - if script.exists() { - match Command::new("/bin/bash").arg(script).spawn() { - Ok(mut c) => { - if let Err(e) = c.wait().await { - tracing::error!("Error Running {}: {}", script.display(), e); - tracing::debug!("{:?}", e); - } - } - Err(e) => { - tracing::error!("Error Running {}: {}", script.display(), e); - tracing::debug!("{:?}", e); - } - } - } -} - -#[instrument(skip_all)] -async fn inner_main(cfg_path: Option) -> Result, Error> { - if &*PLATFORM == "raspberrypi" && tokio::fs::metadata(STANDBY_MODE_PATH).await.is_ok() { - tokio::fs::remove_file(STANDBY_MODE_PATH).await?; - Command::new("sync").invoke(ErrorKind::Filesystem).await?; - crate::sound::SHUTDOWN.play().await?; - futures::future::pending::<()>().await; - } - - crate::sound::BEP.play().await?; - - run_script_if_exists("/media/embassy/config/preinit.sh").await; - - let res = match setup_or_init(cfg_path.clone()).await { - Err(e) => { - async move { - tracing::error!("{}", e.source); - tracing::debug!("{}", e.source); - crate::sound::BEETHOVEN.play().await?; - - let ctx = DiagnosticContext::init( - cfg_path, - if tokio::fs::metadata("/media/embassy/config/disk.guid") - .await - .is_ok() - { - Some(Arc::new( - tokio::fs::read_to_string("/media/embassy/config/disk.guid") // unique identifier for volume group - keeps track of the disk that goes with your embassy - .await? - .trim() - .to_owned(), - )) - } else { - None - }, - e, - ) - .await?; - - let server = WebServer::diagnostic( - SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), 80), - ctx.clone(), - ) - .await?; - - let shutdown = ctx.shutdown.subscribe().recv().await.unwrap(); - - server.shutdown().await; - - Ok(shutdown) - } - .await - } - Ok(s) => Ok(s), - }; - - run_script_if_exists("/media/embassy/config/postinit.sh").await; - - res -} - -pub fn main() { - let matches = clap::App::new("start-init") - .arg( - clap::Arg::with_name("config") - .short('c') - .long("config") - .takes_value(true), - ) - .get_matches(); - - let cfg_path = matches.value_of("config").map(|p| Path::new(p).to_owned()); - let res = { - let rt = tokio::runtime::Builder::new_multi_thread() - .enable_all() - .build() - .expect("failed to initialize runtime"); - rt.block_on(inner_main(cfg_path)) - }; - - match res { - Ok(Some(shutdown)) => shutdown.execute(), - Ok(None) => (), - Err(e) => { - eprintln!("{}", e.source); - tracing::debug!("{:?}", e.source); - drop(e.source); - std::process::exit(e.kind as i32) - } - } -} diff --git a/core/startos/src/bins/start_sdk.rs b/core/startos/src/bins/start_sdk.rs deleted file mode 100644 index 10219c48..00000000 --- a/core/startos/src/bins/start_sdk.rs +++ /dev/null @@ -1,61 +0,0 @@ -use rpc_toolkit::run_cli; -use rpc_toolkit::yajrc::RpcError; -use serde_json::Value; - -use crate::context::SdkContext; -use crate::util::logger::EmbassyLogger; -use crate::version::{Current, VersionT}; -use crate::Error; - -lazy_static::lazy_static! { - static ref VERSION_STRING: String = Current::new().semver().to_string(); -} - -fn inner_main() -> Result<(), Error> { - run_cli!({ - command: crate::portable_api, - app: app => app - .name("StartOS SDK") - .version(&**VERSION_STRING) - .arg( - clap::Arg::with_name("config") - .short('c') - .long("config") - .takes_value(true), - ), - context: matches => { - if let Err(_) = std::env::var("RUST_LOG") { - std::env::set_var("RUST_LOG", "embassy=warn,js_engine=warn"); - } - EmbassyLogger::init(); - SdkContext::init(matches)? - }, - exit: |e: RpcError| { - match e.data { - Some(Value::String(s)) => eprintln!("{}: {}", e.message, s), - Some(Value::Object(o)) => if let Some(Value::String(s)) = o.get("details") { - eprintln!("{}: {}", e.message, s); - if let Some(Value::String(s)) = o.get("debug") { - tracing::debug!("{}", s) - } - } - Some(a) => eprintln!("{}: {}", e.message, a), - None => eprintln!("{}", e.message), - } - std::process::exit(e.code); - } - }); - Ok(()) -} - -pub fn main() { - match inner_main() { - Ok(_) => (), - Err(e) => { - eprintln!("{}", e.source); - tracing::debug!("{:?}", e.source); - drop(e.source); - std::process::exit(e.kind as i32) - } - } -} diff --git a/core/startos/src/bins/startd.rs b/core/startos/src/bins/startd.rs deleted file mode 100644 index a773dd99..00000000 --- a/core/startos/src/bins/startd.rs +++ /dev/null @@ -1,187 +0,0 @@ -use std::net::{Ipv6Addr, SocketAddr}; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use color_eyre::eyre::eyre; -use futures::{FutureExt, TryFutureExt}; -use tokio::signal::unix::signal; -use tracing::instrument; - -use crate::context::{DiagnosticContext, RpcContext}; -use crate::net::web_server::WebServer; -use crate::shutdown::Shutdown; -use crate::system::launch_metrics_task; -use crate::util::logger::EmbassyLogger; -use crate::{Error, ErrorKind, ResultExt}; - -#[instrument(skip_all)] -async fn inner_main(cfg_path: Option) -> Result, Error> { - let (rpc_ctx, server, shutdown) = async { - let rpc_ctx = RpcContext::init( - cfg_path, - Arc::new( - tokio::fs::read_to_string("/media/embassy/config/disk.guid") // unique identifier for volume group - keeps track of the disk that goes with your embassy - .await? - .trim() - .to_owned(), - ), - ) - .await?; - crate::hostname::sync_hostname(&rpc_ctx.account.read().await.hostname).await?; - let server = WebServer::main( - SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), 80), - rpc_ctx.clone(), - ) - .await?; - - let mut shutdown_recv = rpc_ctx.shutdown.subscribe(); - - let sig_handler_ctx = rpc_ctx.clone(); - let sig_handler = tokio::spawn(async move { - use tokio::signal::unix::SignalKind; - futures::future::select_all( - [ - SignalKind::interrupt(), - SignalKind::quit(), - SignalKind::terminate(), - ] - .iter() - .map(|s| { - async move { - signal(*s) - .unwrap_or_else(|_| panic!("register {:?} handler", s)) - .recv() - .await - } - .boxed() - }), - ) - .await; - sig_handler_ctx - .shutdown - .send(None) - .map_err(|_| ()) - .expect("send shutdown signal"); - }); - - let metrics_ctx = rpc_ctx.clone(); - let metrics_task = tokio::spawn(async move { - launch_metrics_task(&metrics_ctx.metrics_cache, || { - metrics_ctx.shutdown.subscribe() - }) - .await - }); - - crate::sound::CHIME.play().await?; - - metrics_task - .map_err(|e| { - Error::new( - eyre!("{}", e).wrap_err("Metrics daemon panicked!"), - ErrorKind::Unknown, - ) - }) - .map_ok(|_| tracing::debug!("Metrics daemon Shutdown")) - .await?; - - let shutdown = shutdown_recv - .recv() - .await - .with_kind(crate::ErrorKind::Unknown)?; - - sig_handler.abort(); - - Ok::<_, Error>((rpc_ctx, server, shutdown)) - } - .await?; - server.shutdown().await; - rpc_ctx.shutdown().await?; - - tracing::info!("RPC Context is dropped"); - - Ok(shutdown) -} - -pub fn main() { - EmbassyLogger::init(); - - if !Path::new("/run/embassy/initialized").exists() { - super::start_init::main(); - std::fs::write("/run/embassy/initialized", "").unwrap(); - } - - let matches = clap::App::new("startd") - .arg( - clap::Arg::with_name("config") - .short('c') - .long("config") - .takes_value(true), - ) - .get_matches(); - - let cfg_path = matches.value_of("config").map(|p| Path::new(p).to_owned()); - - let res = { - let rt = tokio::runtime::Builder::new_multi_thread() - .enable_all() - .build() - .expect("failed to initialize runtime"); - rt.block_on(async { - match inner_main(cfg_path.clone()).await { - Ok(a) => Ok(a), - Err(e) => { - async { - tracing::error!("{}", e.source); - tracing::debug!("{:?}", e.source); - crate::sound::BEETHOVEN.play().await?; - let ctx = DiagnosticContext::init( - cfg_path, - if tokio::fs::metadata("/media/embassy/config/disk.guid") - .await - .is_ok() - { - Some(Arc::new( - tokio::fs::read_to_string("/media/embassy/config/disk.guid") // unique identifier for volume group - keeps track of the disk that goes with your embassy - .await? - .trim() - .to_owned(), - )) - } else { - None - }, - e, - ) - .await?; - - let server = WebServer::diagnostic( - SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), 80), - ctx.clone(), - ) - .await?; - - let mut shutdown = ctx.shutdown.subscribe(); - - let shutdown = - shutdown.recv().await.with_kind(crate::ErrorKind::Unknown)?; - - server.shutdown().await; - - Ok::<_, Error>(shutdown) - } - .await - } - } - }) - }; - - match res { - Ok(None) => (), - Ok(Some(s)) => s.execute(), - Err(e) => { - eprintln!("{}", e.source); - tracing::debug!("{:?}", e.source); - drop(e.source); - std::process::exit(e.kind as i32) - } - } -} diff --git a/core/startos/src/config/action.rs b/core/startos/src/config/action.rs deleted file mode 100644 index 27cd1683..00000000 --- a/core/startos/src/config/action.rs +++ /dev/null @@ -1,116 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; - -use color_eyre::eyre::eyre; -use models::ImageId; -use patch_db::HasModel; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use super::{Config, ConfigSpec}; -use crate::context::RpcContext; -use crate::dependencies::Dependencies; -use crate::prelude::*; -use crate::procedure::docker::DockerContainers; -use crate::procedure::{PackageProcedure, ProcedureName}; -use crate::s9pk::manifest::PackageId; -use crate::status::health_check::HealthCheckId; -use crate::util::Version; -use crate::volume::Volumes; -use crate::{Error, ResultExt}; - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct ConfigRes { - pub config: Option, - pub spec: ConfigSpec, -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[model = "Model"] -pub struct ConfigActions { - pub get: PackageProcedure, - pub set: PackageProcedure, -} -impl ConfigActions { - #[instrument(skip_all)] - pub fn validate( - &self, - _container: &Option, - eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - ) -> Result<(), Error> { - self.get - .validate(eos_version, volumes, image_ids, true) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Config Get"))?; - self.set - .validate(eos_version, volumes, image_ids, true) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Config Set"))?; - Ok(()) - } - #[instrument(skip_all)] - pub async fn get( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - ) -> Result { - self.get - .execute( - ctx, - pkg_id, - pkg_version, - ProcedureName::GetConfig, - volumes, - None::<()>, - None, - ) - .await - .and_then(|res| { - res.map_err(|e| Error::new(eyre!("{}", e.1), crate::ErrorKind::ConfigGen)) - }) - } - - #[instrument(skip_all)] - pub async fn set( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - dependencies: &Dependencies, - volumes: &Volumes, - input: &Config, - ) -> Result { - let res: SetResult = self - .set - .execute( - ctx, - pkg_id, - pkg_version, - ProcedureName::SetConfig, - volumes, - Some(input), - None, - ) - .await - .and_then(|res| { - res.map_err(|e| { - Error::new(eyre!("{}", e.1), crate::ErrorKind::ConfigRulesViolation) - }) - })?; - Ok(SetResult { - depends_on: res - .depends_on - .into_iter() - .filter(|(pkg, _)| dependencies.0.contains_key(pkg)) - .collect(), - }) - } -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct SetResult { - pub depends_on: BTreeMap>, -} diff --git a/core/startos/src/config/mod.rs b/core/startos/src/config/mod.rs deleted file mode 100644 index 06e7770b..00000000 --- a/core/startos/src/config/mod.rs +++ /dev/null @@ -1,287 +0,0 @@ -use std::collections::BTreeMap; -use std::path::PathBuf; -use std::sync::Arc; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use indexmap::IndexSet; -use itertools::Itertools; -use models::{ErrorKind, OptionExt}; -use patch_db::value::InternedString; -use patch_db::Value; -use regex::Regex; -use rpc_toolkit::command; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::display_none; -use crate::util::serde::{display_serializable, parse_stdin_deserializable, IoFormat}; -use crate::Error; - -pub mod action; -pub mod spec; -pub mod util; - -pub use spec::{ConfigSpec, Defaultable}; -use util::NumRange; - -use self::action::ConfigRes; -use self::spec::ValueSpecPointer; - -pub type Config = patch_db::value::InOMap; -pub trait TypeOf { - fn type_of(&self) -> &'static str; -} -impl TypeOf for Value { - fn type_of(&self) -> &'static str { - match self { - Value::Array(_) => "list", - Value::Bool(_) => "boolean", - Value::Null => "null", - Value::Number(_) => "number", - Value::Object(_) => "object", - Value::String(_) => "string", - } - } -} - -#[derive(Debug, thiserror::Error)] -pub enum ConfigurationError { - #[error("Timeout Error")] - TimeoutError(#[from] TimeoutError), - #[error("No Match: {0}")] - NoMatch(#[from] NoMatchWithPath), - #[error("System Error: {0}")] - SystemError(Error), - #[error("Permission Denied: {0}")] - PermissionDenied(ValueSpecPointer), -} -impl From for Error { - fn from(err: ConfigurationError) -> Self { - let kind = match &err { - ConfigurationError::SystemError(e) => e.kind, - _ => crate::ErrorKind::ConfigGen, - }; - crate::Error::new(err, kind) - } -} - -#[derive(Clone, Copy, Debug, thiserror::Error)] -#[error("Timeout Error")] -pub struct TimeoutError; - -#[derive(Clone, Debug, thiserror::Error)] -pub struct NoMatchWithPath { - pub path: Vec, - pub error: MatchError, -} -impl NoMatchWithPath { - pub fn new(error: MatchError) -> Self { - NoMatchWithPath { - path: Vec::new(), - error, - } - } - pub fn prepend(mut self, seg: InternedString) -> Self { - self.path.push(seg); - self - } -} -impl std::fmt::Display for NoMatchWithPath { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "{}: {}", self.path.iter().rev().join("."), self.error) - } -} -impl From for Error { - fn from(e: NoMatchWithPath) -> Self { - ConfigurationError::from(e).into() - } -} - -#[derive(Clone, Debug, thiserror::Error)] -pub enum MatchError { - #[error("String {0:?} Does Not Match Pattern {1}")] - Pattern(Arc, Regex), - #[error("String {0:?} Is Not In Enum {1:?}")] - Enum(Arc, IndexSet), - #[error("Field Is Not Nullable")] - NotNullable, - #[error("Length Mismatch: expected {0}, actual: {1}")] - LengthMismatch(NumRange, usize), - #[error("Invalid Type: expected {0}, actual: {1}")] - InvalidType(&'static str, &'static str), - #[error("Number Out Of Range: expected {0}, actual: {1}")] - OutOfRange(NumRange, f64), - #[error("Number Is Not Integral: {0}")] - NonIntegral(f64), - #[error("Variant {0:?} Is Not In Union {1:?}")] - Union(Arc, IndexSet), - #[error("Variant Is Missing Tag {0:?}")] - MissingTag(InternedString), - #[error("Property {0:?} Of Variant {1:?} Conflicts With Union Tag")] - PropertyMatchesUnionTag(InternedString, String), - #[error("Name of Property {0:?} Conflicts With Map Tag Name")] - PropertyNameMatchesMapTag(String), - #[error("Pointer Is Invalid: {0}")] - InvalidPointer(spec::ValueSpecPointer), - #[error("Object Key Is Invalid: {0}")] - InvalidKey(String), - #[error("Value In List Is Not Unique")] - ListUniquenessViolation, -} - -#[command(rename = "config-spec", cli_only, blocking, display(display_none))] -pub fn verify_spec(#[arg] path: PathBuf) -> Result<(), Error> { - let mut file = std::fs::File::open(&path)?; - let format = match path.extension().and_then(|s| s.to_str()) { - Some("yaml") | Some("yml") => IoFormat::Yaml, - Some("json") => IoFormat::Json, - Some("toml") => IoFormat::Toml, - Some("cbor") => IoFormat::Cbor, - _ => { - return Err(Error::new( - eyre!("Unknown file format. Expected one of yaml, json, toml, cbor."), - crate::ErrorKind::Deserialization, - )); - } - }; - let _: ConfigSpec = format.from_reader(&mut file)?; - - Ok(()) -} - -#[command(subcommands(get, set))] -pub fn config(#[arg] id: PackageId) -> Result { - Ok(id) -} - -#[command(display(display_serializable))] -#[instrument(skip_all)] -pub async fn get( - #[context] ctx: RpcContext, - #[parent_data] id: PackageId, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - let db = ctx.db.peek().await; - let manifest = db - .as_package_data() - .as_idx(&id) - .or_not_found(&id)? - .as_installed() - .or_not_found(&id)? - .as_manifest(); - let action = manifest - .as_config() - .de()? - .ok_or_else(|| Error::new(eyre!("{} has no config", id), crate::ErrorKind::NotFound))?; - - let volumes = manifest.as_volumes().de()?; - let version = manifest.as_version().de()?; - action.get(&ctx, &id, &version, &volumes).await -} - -#[command( - subcommands(self(set_impl(async, context(RpcContext))), set_dry), - display(display_none), - metadata(sync_db = true) -)] -#[instrument(skip_all)] -pub fn set( - #[parent_data] id: PackageId, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, - #[arg(long = "timeout")] timeout: Option, - #[arg(stdin, parse(parse_stdin_deserializable))] config: Option, -) -> Result<(PackageId, Option, Option), Error> { - Ok((id, config, timeout.map(|d| *d))) -} - -#[command(rename = "dry", display(display_serializable))] -#[instrument(skip_all)] -pub async fn set_dry( - #[context] ctx: RpcContext, - #[parent_data] (id, config, timeout): (PackageId, Option, Option), -) -> Result, Error> { - let breakages = BTreeMap::new(); - let overrides = Default::default(); - - let configure_context = ConfigureContext { - breakages, - timeout, - config, - dry_run: true, - overrides, - }; - let breakages = configure(&ctx, &id, configure_context).await?; - - Ok(breakages) -} - -pub struct ConfigureContext { - pub breakages: BTreeMap, - pub timeout: Option, - pub config: Option, - pub overrides: BTreeMap, - pub dry_run: bool, -} - -#[instrument(skip_all)] -pub async fn set_impl( - ctx: RpcContext, - (id, config, timeout): (PackageId, Option, Option), -) -> Result<(), Error> { - let breakages = BTreeMap::new(); - let overrides = Default::default(); - - let configure_context = ConfigureContext { - breakages, - timeout, - config, - dry_run: false, - overrides, - }; - configure(&ctx, &id, configure_context).await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn configure( - ctx: &RpcContext, - id: &PackageId, - configure_context: ConfigureContext, -) -> Result, Error> { - let db = ctx.db.peek().await; - let package = db - .as_package_data() - .as_idx(id) - .or_not_found(&id)? - .as_installed() - .or_not_found(&id)?; - let version = package.as_manifest().as_version().de()?; - ctx.managers - .get(&(id.clone(), version.clone())) - .await - .ok_or_else(|| { - Error::new( - eyre!("There is no manager running for {id:?} and {version:?}"), - ErrorKind::Unknown, - ) - })? - .configure(configure_context) - .await -} - -macro_rules! not_found { - ($x:expr) => { - crate::Error::new( - color_eyre::eyre::eyre!("Could not find {} at {}:{}", $x, module_path!(), line!()), - crate::ErrorKind::Incoherent, - ) - }; -} -pub(crate) use not_found; diff --git a/core/startos/src/config/spec.rs b/core/startos/src/config/spec.rs deleted file mode 100644 index 1e1877d6..00000000 --- a/core/startos/src/config/spec.rs +++ /dev/null @@ -1,2045 +0,0 @@ -use std::borrow::Cow; -use std::collections::{BTreeMap, BTreeSet}; -use std::fmt; -use std::fmt::Debug; -use std::hash::{Hash, Hasher}; -use std::iter::FromIterator; -use std::ops::RangeBounds; -use std::sync::Arc; -use std::time::Duration; - -use async_trait::async_trait; -use imbl::Vector; -use indexmap::{IndexMap, IndexSet}; -use itertools::Itertools; -use jsonpath_lib::Compiled as CompiledJsonPath; -use patch_db::value::{InternedString, Number, Value}; -use rand::{CryptoRng, Rng}; -use regex::Regex; -use serde::de::{MapAccess, Visitor}; -use serde::{Deserialize, Deserializer, Serialize, Serializer}; -use sqlx::PgPool; - -use super::util::{self, CharSet, NumRange, UniqueBy, STATIC_NULL}; -use super::{Config, MatchError, NoMatchWithPath, TimeoutError, TypeOf}; -use crate::config::ConfigurationError; -use crate::context::RpcContext; -use crate::net::interface::InterfaceId; -use crate::net::keys::Key; -use crate::prelude::*; -use crate::s9pk::manifest::{Manifest, PackageId}; - -// Config Value Specifications -#[async_trait] -pub trait ValueSpec { - // This function defines whether the value supplied in the argument is - // consistent with the spec in &self - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath>; - // This function checks whether the value spec is consistent with itself, - // since not all inVariant can be checked by the type - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath>; - // update is to fill in values for environment pointers recursively - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError>; - // returns all pointers that are live in the provided config - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath>; - // requires returns whether the app id is the target of a pointer within it - fn requires(&self, id: &PackageId, value: &Value) -> bool; - // defines if 2 values of this type are equal for the purpose of uniqueness - fn eq(&self, lhs: &Value, rhs: &Value) -> bool; -} - -// Config Value Default Generation -// -// This behavior is defined by two independent traits as well as a third that -// represents a conjunction of those two traits: -// -// DefaultableWith - defines an associated type describing the information it -// needs to be able to generate a default value, as well as a function for -// extracting relevant pieces of that information and using it to actually -// generate the default value -// -// HasDefaultSpec - only purpose is to summon the default spec for the type -// -// Defaultable - this is a redundant trait that may replace 'DefaultableWith' -// and 'HasDefaultSpec'. -pub trait DefaultableWith { - type DefaultSpec: Sync; - type Error: std::error::Error; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result; -} -pub trait HasDefaultSpec: DefaultableWith { - fn default_spec(&self) -> &Self::DefaultSpec; -} - -pub trait Defaultable { - type Error; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result; -} -impl Defaultable for T -where - T: HasDefaultSpec + DefaultableWith + Sync, - E: std::error::Error, -{ - type Error = E; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.gen_with(self.default_spec(), rng, timeout) - } -} - -// WithDefault - trivial wrapper that pairs a 'DefaultableWith' type with a -// default spec -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct WithDefault { - #[serde(flatten)] - pub inner: T, - pub default: T::DefaultSpec, -} -impl DefaultableWith for WithDefault -where - T: DefaultableWith + Sync + Send, - T::DefaultSpec: Send, -{ - type DefaultSpec = T::DefaultSpec; - type Error = T::Error; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.inner.gen_with(spec, rng, timeout) - } -} -impl HasDefaultSpec for WithDefault -where - T: DefaultableWith + Sync + Send, - T::DefaultSpec: Send, -{ - fn default_spec(&self) -> &Self::DefaultSpec { - &self.default - } -} -#[async_trait] -impl ValueSpec for WithDefault -where - T: ValueSpec + DefaultableWith + Send + Sync, - Self: Send + Sync, -{ - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - self.inner.matches(value) - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - self.inner.validate(manifest) - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - self.inner - .update(ctx, manifest, config_overrides, value) - .await - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - self.inner.pointers(value) - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - self.inner.requires(id, value) - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - self.inner.eq(lhs, rhs) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct WithNullable { - #[serde(flatten)] - pub inner: T, - pub nullable: bool, -} -#[async_trait] -impl ValueSpec for WithNullable -where - T: ValueSpec + Send + Sync, - Self: Send + Sync, -{ - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match (self.nullable, value) { - (true, &Value::Null) => Ok(()), - _ => self.inner.matches(value), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - self.inner.validate(manifest) - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - self.inner - .update(ctx, manifest, config_overrides, value) - .await - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - self.inner.pointers(value) - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - self.inner.requires(id, value) - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - self.inner.eq(lhs, rhs) - } -} - -impl DefaultableWith for WithNullable -where - T: DefaultableWith + Sync + Send, -{ - type DefaultSpec = T::DefaultSpec; - type Error = T::Error; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.inner.gen_with(spec, rng, timeout) - } -} - -impl Defaultable for WithNullable -where - T: Defaultable + Sync + Send, -{ - type Error = T::Error; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.inner.gen(rng, timeout) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct WithDescription { - #[serde(flatten)] - pub inner: T, - pub description: Option, - pub name: String, - #[serde(skip_serializing_if = "Option::is_none")] - pub warning: Option, -} -#[async_trait] -impl ValueSpec for WithDescription -where - T: ValueSpec + Sync + Send, - Self: Sync + Send, -{ - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - self.inner.matches(value) - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - self.inner.validate(manifest) - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - self.inner - .update(ctx, manifest, config_overrides, value) - .await - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - self.inner.pointers(value) - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - self.inner.requires(id, value) - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - self.inner.eq(lhs, rhs) - } -} - -impl DefaultableWith for WithDescription -where - T: DefaultableWith + Sync + Send, -{ - type DefaultSpec = T::DefaultSpec; - type Error = T::Error; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.inner.gen_with(spec, rng, timeout) - } -} - -impl Defaultable for WithDescription -where - T: Defaultable + Sync + Send, -{ - type Error = T::Error; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.inner.gen(rng, timeout) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -#[serde(tag = "type")] -pub enum ValueSpecAny { - Boolean(WithDescription>), - Enum(WithDescription>), - List(ValueSpecList), - Number(WithDescription>>), - Object(WithDescription), - String(WithDescription>>), - Union(WithDescription>), - Pointer(WithDescription), -} -impl ValueSpecAny { - pub fn name(&self) -> &'_ str { - match self { - ValueSpecAny::Boolean(b) => b.name.as_str(), - ValueSpecAny::Enum(e) => e.name.as_str(), - ValueSpecAny::List(l) => match l { - ValueSpecList::Enum(e) => e.name.as_str(), - ValueSpecList::Number(n) => n.name.as_str(), - ValueSpecList::Object(o) => o.name.as_str(), - ValueSpecList::String(s) => s.name.as_str(), - ValueSpecList::Union(u) => u.name.as_str(), - }, - ValueSpecAny::Number(n) => n.name.as_str(), - ValueSpecAny::Object(o) => o.name.as_str(), - ValueSpecAny::Pointer(p) => p.name.as_str(), - ValueSpecAny::String(s) => s.name.as_str(), - ValueSpecAny::Union(u) => u.name.as_str(), - } - } -} -#[async_trait] -impl ValueSpec for ValueSpecAny { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match self { - ValueSpecAny::Boolean(a) => a.matches(value), - ValueSpecAny::Enum(a) => a.matches(value), - ValueSpecAny::List(a) => a.matches(value), - ValueSpecAny::Number(a) => a.matches(value), - ValueSpecAny::Object(a) => a.matches(value), - ValueSpecAny::String(a) => a.matches(value), - ValueSpecAny::Union(a) => a.matches(value), - ValueSpecAny::Pointer(a) => a.matches(value), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - match self { - ValueSpecAny::Boolean(a) => a.validate(manifest), - ValueSpecAny::Enum(a) => a.validate(manifest), - ValueSpecAny::List(a) => a.validate(manifest), - ValueSpecAny::Number(a) => a.validate(manifest), - ValueSpecAny::Object(a) => a.validate(manifest), - ValueSpecAny::String(a) => a.validate(manifest), - ValueSpecAny::Union(a) => a.validate(manifest), - ValueSpecAny::Pointer(a) => a.validate(manifest), - } - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - match self { - ValueSpecAny::Boolean(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::Enum(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::List(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::Number(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::Object(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::String(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::Union(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecAny::Pointer(a) => a.update(ctx, manifest, config_overrides, value).await, - } - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - match self { - ValueSpecAny::Boolean(a) => a.pointers(value), - ValueSpecAny::Enum(a) => a.pointers(value), - ValueSpecAny::List(a) => a.pointers(value), - ValueSpecAny::Number(a) => a.pointers(value), - ValueSpecAny::Object(a) => a.pointers(value), - ValueSpecAny::String(a) => a.pointers(value), - ValueSpecAny::Union(a) => a.pointers(value), - ValueSpecAny::Pointer(a) => a.pointers(value), - } - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - match self { - ValueSpecAny::Boolean(a) => a.requires(id, value), - ValueSpecAny::Enum(a) => a.requires(id, value), - ValueSpecAny::List(a) => a.requires(id, value), - ValueSpecAny::Number(a) => a.requires(id, value), - ValueSpecAny::Object(a) => a.requires(id, value), - ValueSpecAny::String(a) => a.requires(id, value), - ValueSpecAny::Union(a) => a.requires(id, value), - ValueSpecAny::Pointer(a) => a.requires(id, value), - } - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match self { - ValueSpecAny::Boolean(a) => a.eq(lhs, rhs), - ValueSpecAny::Enum(a) => a.eq(lhs, rhs), - ValueSpecAny::List(a) => a.eq(lhs, rhs), - ValueSpecAny::Number(a) => a.eq(lhs, rhs), - ValueSpecAny::Object(a) => a.eq(lhs, rhs), - ValueSpecAny::String(a) => a.eq(lhs, rhs), - ValueSpecAny::Union(a) => a.eq(lhs, rhs), - ValueSpecAny::Pointer(a) => a.eq(lhs, rhs), - } - } -} -impl Defaultable for ValueSpecAny { - type Error = ConfigurationError; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - match self { - ValueSpecAny::Boolean(a) => a.gen(rng, timeout).map_err(crate::util::Never::absurd), - ValueSpecAny::Enum(a) => a.gen(rng, timeout).map_err(crate::util::Never::absurd), - ValueSpecAny::List(a) => a.gen(rng, timeout), - ValueSpecAny::Number(a) => a.gen(rng, timeout).map_err(crate::util::Never::absurd), - ValueSpecAny::Object(a) => a.gen(rng, timeout), - ValueSpecAny::String(a) => a.gen(rng, timeout).map_err(ConfigurationError::from), - ValueSpecAny::Union(a) => a.gen(rng, timeout), - ValueSpecAny::Pointer(a) => a.gen(rng, timeout), - } - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct ValueSpecBoolean {} -#[async_trait] -impl ValueSpec for ValueSpecBoolean { - fn matches(&self, val: &Value) -> Result<(), NoMatchWithPath> { - match val { - Value::Bool(_) => Ok(()), - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "boolean", - a.type_of(), - ))), - } - } - fn validate(&self, _manifest: &Manifest) -> Result<(), NoMatchWithPath> { - Ok(()) - } - async fn update( - &self, - _ctx: &RpcContext, - _manifest: &Manifest, - _config_overrides: &BTreeMap, - _value: &mut Value, - ) -> Result<(), ConfigurationError> { - Ok(()) - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - Ok(BTreeSet::new()) - } - fn requires(&self, _id: &PackageId, _value: &Value) -> bool { - false - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::Bool(lhs), Value::Bool(rhs)) => lhs == rhs, - _ => false, - } - } -} -impl DefaultableWith for ValueSpecBoolean { - type DefaultSpec = bool; - type Error = crate::util::Never; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(Value::Bool(*spec)) - } -} - -#[derive(Clone, Debug, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct ValueSpecEnum { - pub values: IndexSet, - pub value_names: BTreeMap, -} -impl<'de> serde::de::Deserialize<'de> for ValueSpecEnum { - fn deserialize>(deserializer: D) -> Result { - #[derive(Deserialize)] - #[serde(rename_all = "kebab-case")] - pub struct _ValueSpecEnum { - pub values: IndexSet, - #[serde(default)] - pub value_names: BTreeMap, - } - - let mut r#enum = _ValueSpecEnum::deserialize(deserializer)?; - for name in &r#enum.values { - if !r#enum.value_names.contains_key(name) { - r#enum.value_names.insert(name.clone(), name.clone()); - } - } - Ok(ValueSpecEnum { - values: r#enum.values, - value_names: r#enum.value_names, - }) - } -} -#[async_trait] -impl ValueSpec for ValueSpecEnum { - fn matches(&self, val: &Value) -> Result<(), NoMatchWithPath> { - match val { - Value::String(b) => { - if self.values.contains(&**b) { - Ok(()) - } else { - Err(NoMatchWithPath::new(MatchError::Enum( - b.clone(), - self.values.clone(), - ))) - } - } - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "string", - a.type_of(), - ))), - } - } - fn validate(&self, _manifest: &Manifest) -> Result<(), NoMatchWithPath> { - Ok(()) - } - async fn update( - &self, - _ctx: &RpcContext, - _manifest: &Manifest, - _config_overrides: &BTreeMap, - _value: &mut Value, - ) -> Result<(), ConfigurationError> { - Ok(()) - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - Ok(BTreeSet::new()) - } - fn requires(&self, _id: &PackageId, _value: &Value) -> bool { - false - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::String(lhs), Value::String(rhs)) => lhs == rhs, - _ => false, - } - } -} -impl DefaultableWith for ValueSpecEnum { - type DefaultSpec = Arc; - type Error = crate::util::Never; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(Value::String(spec.clone())) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct ListSpec { - pub spec: T, - pub range: NumRange, -} -#[async_trait] -impl ValueSpec for ListSpec -where - T: ValueSpec + Sync + Send, - Self: Sync + Send, -{ - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match value { - Value::Array(l) => { - if !self.range.contains(&l.len()) { - Err(NoMatchWithPath { - path: Vec::new(), - error: MatchError::LengthMismatch(self.range.clone(), l.len()), - }) - } else { - l.iter() - .enumerate() - .map(|(i, v)| { - self.spec - .matches(v) - .map_err(|e| e.prepend(InternedString::from_display(&i)))?; - if l.iter() - .enumerate() - .any(|(i2, v2)| i != i2 && self.spec.eq(v, v2)) - { - Err(NoMatchWithPath::new(MatchError::ListUniquenessViolation) - .prepend(InternedString::from_display(&i))) - } else { - Ok(()) - } - }) - .collect() - } - } - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "list", - a.type_of(), - ))), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - self.spec.validate(manifest) - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - if let Value::Array(ref mut ls) = value { - for (i, val) in ls.iter_mut().enumerate() { - match self.spec.update(ctx, manifest, config_overrides, val).await { - Err(ConfigurationError::NoMatch(e)) => Err(ConfigurationError::NoMatch( - e.prepend(InternedString::from_display(&i)), - )), - a => a, - }?; - } - Ok(()) - } else { - Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::InvalidType("list", value.type_of()), - ))) - } - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - Ok(BTreeSet::new()) - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - if let Value::Array(ref ls) = value { - ls.into_iter().any(|v| self.spec.requires(id, v)) - } else { - false - } - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::Array(lhs), Value::Array(rhs)) => { - lhs.iter().zip_longest(rhs.iter()).all(|zip| match zip { - itertools::EitherOrBoth::Both(lhs, rhs) => lhs == rhs, - _ => false, - }) - } - _ => false, - } - } -} - -impl DefaultableWith for ListSpec -where - T: DefaultableWith + Sync + Send, -{ - type DefaultSpec = Vec; - type Error = T::Error; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result { - let mut res = Vector::new(); - for spec_member in spec.iter() { - res.push_back(self.spec.gen_with(spec_member, rng, timeout)?); - } - Ok(Value::Array(res)) - } -} - -unsafe impl Sync for ValueSpecObject {} // TODO: remove -unsafe impl Send for ValueSpecObject {} // TODO: remove -unsafe impl Sync for ValueSpecUnion {} // TODO: remove -unsafe impl Send for ValueSpecUnion {} // TODO: remove - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -#[serde(tag = "subtype")] -pub enum ValueSpecList { - Enum(WithDescription>>), - Number(WithDescription>>), - Object(WithDescription>>), - String(WithDescription>>), - Union(WithDescription>>>), -} -#[async_trait] -impl ValueSpec for ValueSpecList { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match self { - ValueSpecList::Enum(a) => a.matches(value), - ValueSpecList::Number(a) => a.matches(value), - ValueSpecList::Object(a) => a.matches(value), - ValueSpecList::String(a) => a.matches(value), - ValueSpecList::Union(a) => a.matches(value), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - match self { - ValueSpecList::Enum(a) => a.validate(manifest), - ValueSpecList::Number(a) => a.validate(manifest), - ValueSpecList::Object(a) => a.validate(manifest), - ValueSpecList::String(a) => a.validate(manifest), - ValueSpecList::Union(a) => a.validate(manifest), - } - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - match self { - ValueSpecList::Enum(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecList::Number(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecList::Object(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecList::String(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecList::Union(a) => a.update(ctx, manifest, config_overrides, value).await, - } - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - match self { - ValueSpecList::Enum(a) => a.pointers(value), - ValueSpecList::Number(a) => a.pointers(value), - ValueSpecList::Object(a) => a.pointers(value), - ValueSpecList::String(a) => a.pointers(value), - ValueSpecList::Union(a) => a.pointers(value), - } - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - match self { - ValueSpecList::Enum(a) => a.requires(id, value), - ValueSpecList::Number(a) => a.requires(id, value), - ValueSpecList::Object(a) => a.requires(id, value), - ValueSpecList::String(a) => a.requires(id, value), - ValueSpecList::Union(a) => a.requires(id, value), - } - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match self { - ValueSpecList::Enum(a) => a.eq(lhs, rhs), - ValueSpecList::Number(a) => a.eq(lhs, rhs), - ValueSpecList::Object(a) => a.eq(lhs, rhs), - ValueSpecList::String(a) => a.eq(lhs, rhs), - ValueSpecList::Union(a) => a.eq(lhs, rhs), - } - } -} - -impl Defaultable for ValueSpecList { - type Error = ConfigurationError; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - match self { - ValueSpecList::Enum(a) => a.gen(rng, timeout).map_err(crate::util::Never::absurd), - ValueSpecList::Number(a) => a.gen(rng, timeout).map_err(crate::util::Never::absurd), - ValueSpecList::Object(a) => { - let mut ret = match a.gen(rng, timeout).unwrap() { - Value::Array(l) => l, - a => { - return Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::InvalidType("list", a.type_of()), - ))) - } - }; - while !( - a.inner.inner.range.start_bound(), - std::ops::Bound::Unbounded, - ) - .contains(&ret.len()) - { - ret.push_back( - a.inner - .inner - .spec - .gen(rng, timeout) - .map_err(ConfigurationError::from)?, - ); - } - Ok(Value::Array(ret)) - } - ValueSpecList::String(a) => a.gen(rng, timeout).map_err(ConfigurationError::from), - ValueSpecList::Union(a) => a.gen(rng, timeout).map_err(ConfigurationError::from), - } - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct ValueSpecNumber { - range: Option>, - #[serde(default)] - integral: bool, - #[serde(skip_serializing_if = "Option::is_none")] - units: Option, - #[serde(skip_serializing_if = "Option::is_none")] - #[serde(default)] - pub placeholder: Option, -} -#[async_trait] -impl ValueSpec for ValueSpecNumber { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match value { - Value::Number(n) => { - let n = n.as_f64().unwrap(); - if self.integral && n.floor() != n { - return Err(NoMatchWithPath::new(MatchError::NonIntegral(n))); - } - if let Some(range) = &self.range { - if !range.contains(&n) { - return Err(NoMatchWithPath::new(MatchError::OutOfRange( - range.clone(), - n, - ))); - } - } - Ok(()) - } - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "object", - a.type_of(), - ))), - } - } - fn validate(&self, _manifest: &Manifest) -> Result<(), NoMatchWithPath> { - Ok(()) - } - async fn update( - &self, - _ctx: &RpcContext, - _manifest: &Manifest, - _config_overrides: &BTreeMap, - _value: &mut Value, - ) -> Result<(), ConfigurationError> { - Ok(()) - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - Ok(BTreeSet::new()) - } - fn requires(&self, _id: &PackageId, _value: &Value) -> bool { - false - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::Number(lhs), Value::Number(rhs)) => lhs == rhs, - _ => false, - } - } -} -impl DefaultableWith for ValueSpecNumber { - type DefaultSpec = Option; - type Error = crate::util::Never; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(spec - .clone() - .map(|s| Value::Number(s)) - .unwrap_or(Value::Null)) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct ValueSpecObject { - pub spec: ConfigSpec, - pub display_as: Option, - #[serde(default)] - pub unique_by: UniqueBy, -} -#[async_trait] -impl ValueSpec for ValueSpecObject { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match value { - Value::Object(o) => self.spec.matches(o), - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "object", - a.type_of(), - ))), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - self.spec.validate(manifest) - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - if let Value::Object(o) = value { - self.spec.update(ctx, manifest, config_overrides, o).await - } else { - Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::InvalidType("object", value.type_of()), - ))) - } - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - if let Value::Object(o) = value { - self.spec.pointers(o) - } else { - Err(NoMatchWithPath::new(MatchError::InvalidType( - "object", - value.type_of(), - ))) - } - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - if let Value::Object(o) = value { - self.spec.requires(id, o) - } else { - false - } - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::Object(lhs), Value::Object(rhs)) => self.unique_by.eq(lhs, rhs), - _ => false, - } - } -} -impl DefaultableWith for ValueSpecObject { - type DefaultSpec = Config; - type Error = crate::util::Never; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(Value::Object(spec.clone())) - } -} -impl Defaultable for ValueSpecObject { - type Error = ConfigurationError; - - fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - self.spec.gen(rng, timeout).map(Value::Object) - } -} - -#[derive(Clone, Debug, Default, Serialize, Deserialize)] -pub struct ConfigSpec(pub IndexMap); -impl ConfigSpec { - pub fn matches(&self, value: &Config) -> Result<(), NoMatchWithPath> { - for (key, val) in self.0.iter() { - if let Some(v) = value.get(&**key) { - val.matches(v).map_err(|e| e.prepend(key.clone()))?; - } else { - val.matches(&Value::Null) - .map_err(|e| e.prepend(key.clone()))?; - } - } - Ok(()) - } - - pub fn gen( - &self, - rng: &mut R, - timeout: &Option, - ) -> Result { - let mut res = Config::new(); - for (key, val) in self.0.iter() { - res.insert(key.clone(), val.gen(rng, timeout)?); - } - Ok(res) - } - - pub fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - for (name, val) in &self.0 { - val.validate(manifest) - .map_err(|e| e.prepend(name.clone()))?; - } - Ok(()) - } - - pub async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - cfg: &mut Config, - ) -> Result<(), ConfigurationError> { - for (k, vs) in self.0.iter() { - match cfg.get_mut(k) { - None => { - let mut v = Value::Null; - vs.update(ctx, manifest, config_overrides, &mut v).await?; - cfg.insert(k.clone(), v); - } - Some(v) => match vs.update(ctx, manifest, config_overrides, v).await { - Err(ConfigurationError::NoMatch(e)) => { - Err(ConfigurationError::NoMatch(e.prepend(k.clone()))) - } - a => a, - }?, - }; - } - Ok(()) - } - - pub fn pointers(&self, cfg: &Config) -> Result, NoMatchWithPath> { - cfg.iter() - .filter_map(|(k, v)| self.0.get(k).map(|vs| (k, vs.pointers(v)))) - .fold(Ok(BTreeSet::::new()), |acc, v| { - match (acc, v) { - // propagate existing errors - (Err(e), _) => Err(e), - // create new error case - (Ok(_), (k, Err(e))) => Err(e.prepend(k.clone())), - // combine sets - (Ok(s0), (_, Ok(s1))) => Ok(BTreeSet::from_iter(s0.union(&s1).cloned())), - } - }) - } - - pub fn requires(&self, id: &PackageId, cfg: &Config) -> bool { - self.0 - .iter() - .any(|(k, v)| v.requires(id, cfg.get(k).unwrap_or(&STATIC_NULL))) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct Pattern { - #[serde(with = "util::serde_regex")] - pub pattern: Regex, - pub pattern_description: String, -} - -#[derive(Clone, Debug, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct ValueSpecString { - #[serde(flatten)] - pub pattern: Option, - pub textarea: bool, - pub copyable: bool, - pub masked: bool, - #[serde(skip_serializing_if = "Option::is_none")] - pub placeholder: Option, -} -impl<'de> Deserialize<'de> for ValueSpecString { - fn deserialize>(deserializer: D) -> Result { - struct ValueSpecStringVisitor; - impl<'de> Visitor<'de> for ValueSpecStringVisitor { - type Value = ValueSpecString; - fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result { - formatter.write_str("struct ValueSpecString") - } - fn visit_map>(self, mut map: V) -> Result { - let mut pattern = None; - let mut pattern_description = None; - let mut textarea = false; - let mut copyable = false; - let mut masked = false; - let mut placeholder = None; - while let Some::(key) = map.next_key()? { - if &key == "pattern" { - if pattern.is_some() { - return Err(serde::de::Error::duplicate_field("pattern")); - } else { - pattern = Some( - Regex::new(&map.next_value::()?) - .map_err(serde::de::Error::custom)?, - ); - } - } else if &key == "pattern-description" { - if pattern_description.is_some() { - return Err(serde::de::Error::duplicate_field("pattern-description")); - } else { - pattern_description = Some(map.next_value()?); - } - } else if &key == "textarea" { - textarea = map.next_value()?; - } else if &key == "copyable" { - copyable = map.next_value()?; - } else if &key == "masked" { - masked = map.next_value()?; - } else if &key == "placeholder" { - if placeholder.is_some() { - return Err(serde::de::Error::duplicate_field("placeholder")); - } else { - placeholder = Some(map.next_value()?); - } - } - } - let regex = match (pattern, pattern_description) { - (None, None) => None, - (Some(p), Some(d)) => Some(Pattern { - pattern: p, - pattern_description: d, - }), - (Some(_), None) => { - return Err(serde::de::Error::missing_field("pattern-description")); - } - (None, Some(_)) => { - return Err(serde::de::Error::missing_field("pattern")); - } - }; - Ok(ValueSpecString { - pattern: regex, - textarea, - copyable, - masked, - placeholder, - }) - } - } - const FIELDS: &[&str] = &[ - "pattern", - "pattern-description", - "textarea", - "copyable", - "masked", - "placeholder", - ]; - deserializer.deserialize_struct("ValueSpecString", FIELDS, ValueSpecStringVisitor) - } -} -#[async_trait] -impl ValueSpec for ValueSpecString { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match value { - Value::String(s) => { - if let Some(pattern) = &self.pattern { - if pattern.pattern.is_match(s) { - Ok(()) - } else { - Err(NoMatchWithPath::new(MatchError::Pattern( - s.clone(), - pattern.pattern.clone(), - ))) - } - } else { - Ok(()) - } - } - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "string", - a.type_of(), - ))), - } - } - fn validate(&self, _manifest: &Manifest) -> Result<(), NoMatchWithPath> { - Ok(()) - } - async fn update( - &self, - _ctx: &RpcContext, - _manifest: &Manifest, - _config_overrides: &BTreeMap, - _value: &mut Value, - ) -> Result<(), ConfigurationError> { - Ok(()) - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - Ok(BTreeSet::new()) - } - fn requires(&self, _id: &PackageId, _value: &Value) -> bool { - false - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::String(lhs), Value::String(rhs)) => lhs == rhs, - _ => false, - } - } -} -impl DefaultableWith for ValueSpecString { - type DefaultSpec = Option; - type Error = TimeoutError; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result { - if let Some(spec) = spec { - let now = timeout.as_ref().map(|_| std::time::Instant::now()); - loop { - let candidate = spec.gen(rng); - match (spec, &self.pattern) { - (DefaultString::Entropy(_), Some(pattern)) - if !pattern.pattern.is_match(&candidate) => {} - _ => { - return Ok(Value::String(candidate)); - } - } - if let (Some(now), Some(timeout)) = (now, timeout) { - if &now.elapsed() > timeout { - return Err(TimeoutError); - } - } else { - return Ok(Value::String(candidate)); - } - } - } else { - Ok(Value::Null) - } - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(untagged)] -pub enum DefaultString { - Literal(String), - Entropy(Entropy), -} -impl DefaultString { - pub fn gen(&self, rng: &mut R) -> Arc { - Arc::new(match self { - DefaultString::Literal(s) => s.clone(), - DefaultString::Entropy(e) => e.gen(rng), - }) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct Entropy { - pub charset: Option, - pub len: usize, -} -impl Entropy { - pub fn gen(&self, rng: &mut R) -> String { - let len = self.len; - let set = self - .charset - .as_ref() - .map(|cs| Cow::Borrowed(cs)) - .unwrap_or_else(|| Cow::Owned(Default::default())); - std::iter::repeat_with(|| set.gen(rng)).take(len).collect() - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct UnionTag { - pub id: InternedString, - pub name: String, - pub description: Option, - pub variant_names: BTreeMap, -} - -#[derive(Clone, Debug, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct ValueSpecUnion { - pub tag: UnionTag, - pub variants: BTreeMap, - pub display_as: Option, - pub unique_by: UniqueBy, -} - -impl<'de> serde::de::Deserialize<'de> for ValueSpecUnion { - fn deserialize>(deserializer: D) -> Result { - #[derive(Deserialize)] - #[serde(rename_all = "kebab-case")] - #[serde(untagged)] - pub enum _UnionTag { - Old(InternedString), - New(UnionTag), - } - #[derive(Deserialize)] - #[serde(rename_all = "kebab-case")] - pub struct _ValueSpecUnion { - pub variants: BTreeMap, - pub tag: _UnionTag, - pub display_as: Option, - #[serde(default)] - pub unique_by: UniqueBy, - } - - let u = _ValueSpecUnion::deserialize(deserializer)?; - Ok(ValueSpecUnion { - tag: match u.tag { - _UnionTag::Old(id) => UnionTag { - id: id.clone(), - name: id.to_string(), - description: None, - variant_names: u - .variants - .keys() - .map(|k| (k.to_owned(), k.to_owned())) - .collect(), - }, - _UnionTag::New(UnionTag { - id, - name, - description, - mut variant_names, - }) => UnionTag { - id, - name, - description, - variant_names: { - let mut iter = u.variants.keys(); - while variant_names.len() < u.variants.len() { - if let Some(variant) = iter.next() { - variant_names.insert(variant.to_owned(), variant.to_owned()); - } else { - break; - } - } - variant_names - }, - }, - }, - variants: u.variants, - display_as: u.display_as, - unique_by: u.unique_by, - }) - } -} - -#[async_trait] -impl ValueSpec for ValueSpecUnion { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match value { - Value::Object(o) => { - if let Some(Value::String(ref tag)) = o.get(&*self.tag.id) { - if let Some(obj_spec) = self.variants.get(&**tag) { - let mut without_tag = o.clone(); - without_tag.remove(&*self.tag.id); - obj_spec.matches(&without_tag) - } else { - Err(NoMatchWithPath::new(MatchError::Union( - tag.clone(), - self.variants.keys().cloned().collect(), - ))) - } - } else { - Err(NoMatchWithPath::new(MatchError::MissingTag( - self.tag.id.clone(), - ))) - } - } - Value::Null => Err(NoMatchWithPath::new(MatchError::NotNullable)), - a => Err(NoMatchWithPath::new(MatchError::InvalidType( - "object", - a.type_of(), - ))), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - for (name, variant) in &self.variants { - if variant.0.get(&*self.tag.id).is_some() { - return Err(NoMatchWithPath::new(MatchError::PropertyMatchesUnionTag( - self.tag.id.clone(), - name.clone(), - ))); - } - variant.validate(manifest)?; - } - Ok(()) - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - if let Value::Object(o) = value { - match o.get(&*self.tag.id) { - None => Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::MissingTag(self.tag.id.clone()), - ))), - Some(Value::String(tag)) => match self.variants.get(&**tag) { - None => Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::Union(tag.clone(), self.variants.keys().cloned().collect()), - ))), - Some(spec) => spec.update(ctx, manifest, config_overrides, o).await, - }, - Some(other) => Err(ConfigurationError::NoMatch( - NoMatchWithPath::new(MatchError::InvalidType("string", other.type_of())) - .prepend(self.tag.id.clone()), - )), - } - } else { - Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::InvalidType("object", value.type_of()), - ))) - } - } - fn pointers(&self, value: &Value) -> Result, NoMatchWithPath> { - if let Value::Object(o) = value { - match o.get(&*self.tag.id) { - None => Err(NoMatchWithPath::new(MatchError::MissingTag( - self.tag.id.clone(), - ))), - Some(Value::String(tag)) => match self.variants.get(&**tag) { - None => Err(NoMatchWithPath::new(MatchError::Union( - tag.clone(), - self.variants.keys().cloned().collect(), - ))), - Some(spec) => spec.pointers(o), - }, - Some(other) => Err(NoMatchWithPath::new(MatchError::InvalidType( - "string", - other.type_of(), - )) - .prepend(self.tag.id.clone())), - } - } else { - Err(NoMatchWithPath::new(MatchError::InvalidType( - "object", - value.type_of(), - ))) - } - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - if let Value::Object(o) = value { - match o.get(&*self.tag.id) { - Some(Value::String(tag)) => match self.variants.get(&**tag) { - None => false, - Some(spec) => spec.requires(id, o), - }, - _ => false, - } - } else { - false - } - } - fn eq(&self, lhs: &Value, rhs: &Value) -> bool { - match (lhs, rhs) { - (Value::Object(lhs), Value::Object(rhs)) => self.unique_by.eq(lhs, rhs), - _ => false, - } - } -} -impl DefaultableWith for ValueSpecUnion { - type DefaultSpec = Arc; - type Error = ConfigurationError; - - fn gen_with( - &self, - spec: &Self::DefaultSpec, - rng: &mut R, - timeout: &Option, - ) -> Result { - let variant = if let Some(v) = self.variants.get(&**spec) { - v - } else { - return Err(ConfigurationError::NoMatch(NoMatchWithPath::new( - MatchError::Union(spec.clone(), self.variants.keys().cloned().collect()), - ))); - }; - let cfg_res = variant.gen(rng, timeout)?; - - let mut tagged_cfg = Config::new(); - tagged_cfg.insert(self.tag.id.clone(), Value::String(spec.clone())); - tagged_cfg.extend(cfg_res.into_iter()); - - Ok(Value::Object(tagged_cfg)) - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(tag = "subtype")] -#[serde(rename_all = "kebab-case")] -pub enum ValueSpecPointer { - Package(PackagePointerSpec), - System(SystemPointerSpec), -} -impl fmt::Display for ValueSpecPointer { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - ValueSpecPointer::Package(p) => write!(f, "{}", p), - ValueSpecPointer::System(p) => write!(f, "{}", p), - } - } -} -impl Defaultable for ValueSpecPointer { - type Error = ConfigurationError; - fn gen( - &self, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(Value::Null) - } -} -#[async_trait] -impl ValueSpec for ValueSpecPointer { - fn matches(&self, value: &Value) -> Result<(), NoMatchWithPath> { - match self { - ValueSpecPointer::Package(a) => a.matches(value), - ValueSpecPointer::System(a) => a.matches(value), - } - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - match self { - ValueSpecPointer::Package(a) => a.validate(manifest), - ValueSpecPointer::System(a) => a.validate(manifest), - } - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - match self { - ValueSpecPointer::Package(a) => a.update(ctx, manifest, config_overrides, value).await, - ValueSpecPointer::System(a) => a.update(ctx, manifest, config_overrides, value).await, - } - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - let mut pointers = BTreeSet::new(); - pointers.insert(self.clone()); - Ok(pointers) - } - fn requires(&self, id: &PackageId, value: &Value) -> bool { - match self { - ValueSpecPointer::Package(a) => a.requires(id, value), - ValueSpecPointer::System(a) => a.requires(id, value), - } - } - fn eq(&self, _lhs: &Value, _rhs: &Value) -> bool { - false - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(tag = "target")] -#[serde(rename_all = "kebab-case")] -pub enum PackagePointerSpec { - TorKey(TorKeyPointer), - TorAddress(TorAddressPointer), - LanAddress(LanAddressPointer), - Config(ConfigPointer), -} -impl PackagePointerSpec { - pub fn package_id(&self) -> &PackageId { - match self { - PackagePointerSpec::TorKey(TorKeyPointer { package_id, .. }) => package_id, - PackagePointerSpec::TorAddress(TorAddressPointer { package_id, .. }) => package_id, - PackagePointerSpec::LanAddress(LanAddressPointer { package_id, .. }) => package_id, - PackagePointerSpec::Config(ConfigPointer { package_id, .. }) => package_id, - } - } - async fn deref( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - ) -> Result { - match &self { - PackagePointerSpec::TorKey(key) => key.deref(&manifest.id, &ctx.secret_store).await, - PackagePointerSpec::TorAddress(tor) => tor.deref(ctx).await, - PackagePointerSpec::LanAddress(lan) => lan.deref(ctx).await, - PackagePointerSpec::Config(cfg) => cfg.deref(ctx, config_overrides).await, - } - } -} -impl fmt::Display for PackagePointerSpec { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - PackagePointerSpec::TorKey(key) => write!(f, "{}", key), - PackagePointerSpec::TorAddress(tor) => write!(f, "{}", tor), - PackagePointerSpec::LanAddress(lan) => write!(f, "{}", lan), - PackagePointerSpec::Config(cfg) => write!(f, "{}", cfg), - } - } -} -impl Defaultable for PackagePointerSpec { - type Error = ConfigurationError; - fn gen( - &self, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(Value::Null) - } -} -#[async_trait] -impl ValueSpec for PackagePointerSpec { - fn matches(&self, _value: &Value) -> Result<(), NoMatchWithPath> { - Ok(()) - } - fn validate(&self, manifest: &Manifest) -> Result<(), NoMatchWithPath> { - if &manifest.id != self.package_id() - && !manifest.dependencies.0.contains_key(self.package_id()) - { - return Err(NoMatchWithPath::new(MatchError::InvalidPointer( - ValueSpecPointer::Package(self.clone()), - ))); - } - match self { - _ => Ok(()), - } - } - async fn update( - &self, - ctx: &RpcContext, - manifest: &Manifest, - config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - *value = self.deref(ctx, manifest, config_overrides).await?; - Ok(()) - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - let mut pointers = BTreeSet::new(); - pointers.insert(ValueSpecPointer::Package(self.clone())); - Ok(pointers) - } - fn requires(&self, id: &PackageId, _value: &Value) -> bool { - self.package_id() == id - } - fn eq(&self, _lhs: &Value, _rhs: &Value) -> bool { - false - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct TorAddressPointer { - pub package_id: PackageId, - interface: InterfaceId, -} -impl TorAddressPointer { - async fn deref(&self, ctx: &RpcContext) -> Result { - let addr = ctx - .db - .peek() - .await - .as_package_data() - .as_idx(&self.package_id) - .and_then(|pde| pde.as_installed()) - .and_then(|i| i.as_interface_addresses().as_idx(&self.interface)) - .and_then(|a| a.as_tor_address().de().transpose()) - .transpose() - .map_err(|e| ConfigurationError::SystemError(e))?; - Ok(addr.map(Arc::new).map(Value::String).unwrap_or(Value::Null)) - } -} -impl fmt::Display for TorAddressPointer { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - TorAddressPointer { - package_id, - interface, - } => write!(f, "{}: tor-address: {}", package_id, interface), - } - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct LanAddressPointer { - pub package_id: PackageId, - interface: InterfaceId, -} -impl fmt::Display for LanAddressPointer { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let LanAddressPointer { - package_id, - interface, - } = self; - write!(f, "{}: lan-address: {}", package_id, interface) - } -} -impl LanAddressPointer { - async fn deref(&self, ctx: &RpcContext) -> Result { - let addr = ctx - .db - .peek() - .await - .as_package_data() - .as_idx(&self.package_id) - .and_then(|pde| pde.as_installed()) - .and_then(|i| i.as_interface_addresses().as_idx(&self.interface)) - .and_then(|a| a.as_lan_address().de().transpose()) - .transpose() - .map_err(|e| ConfigurationError::SystemError(e))?; - Ok(addr - .to_owned() - .map(Arc::new) - .map(Value::String) - .unwrap_or(Value::Null)) - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct ConfigPointer { - package_id: PackageId, - selector: Arc, - multi: bool, -} -impl ConfigPointer { - pub fn select(&self, val: &Value) -> Value { - self.selector.select(self.multi, val) - } - async fn deref( - &self, - ctx: &RpcContext, - config_overrides: &BTreeMap, - ) -> Result { - if let Some(cfg) = config_overrides.get(&self.package_id) { - Ok(self.select(&Value::Object(cfg.clone()))) - } else { - let id = &self.package_id; - let db = ctx.db.peek().await; - let manifest = db.as_package_data().as_idx(id).map(|pde| pde.as_manifest()); - let cfg_actions = manifest.and_then(|m| m.as_config().transpose_ref()); - if let (Some(manifest), Some(cfg_actions)) = (manifest, cfg_actions) { - let cfg_res = cfg_actions - .de() - .map_err(|e| ConfigurationError::SystemError(e))? - .get( - ctx, - &self.package_id, - &manifest - .as_version() - .de() - .map_err(|e| ConfigurationError::SystemError(e))?, - &manifest - .as_volumes() - .de() - .map_err(|e| ConfigurationError::SystemError(e))?, - ) - .await - .map_err(|e| ConfigurationError::SystemError(e))?; - if let Some(cfg) = cfg_res.config { - Ok(self.select(&Value::Object(cfg))) - } else { - Ok(Value::Null) - } - } else { - Ok(Value::Null) - } - } - } -} -impl fmt::Display for ConfigPointer { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let ConfigPointer { - package_id, - selector, - .. - } = self; - write!(f, "{}: config: {}", package_id, selector) - } -} - -#[derive(Clone, Debug)] -pub struct ConfigSelector { - src: String, - compiled: CompiledJsonPath, -} -impl ConfigSelector { - fn select(&self, multi: bool, val: &Value) -> Value { - // WORKAROUND: jsonpath_lib uses a different imbl_value version than patch_db - // Convert through JSON to bridge the incompatible types - - // Step 1: Convert patch_db::Value to serde_json::Value - let json_val = match serde_json::to_value(val) { - Ok(v) => v, - Err(_) => return Value::Null, - }; - - // Step 2: Convert to the type jsonpath_lib expects (different imbl_value version) - // We can't name the type explicitly since we don't have a direct dependency on it - // So we use serde_json as the intermediary - let json_str = json_val.to_string(); - let jsonpath_val = match serde_json::from_str(&json_str) { - Ok(v) => v, - Err(_) => return Value::Null, - }; - - // Step 3: Run the jsonpath selection - let selected = match self.compiled.select(&jsonpath_val) { - Ok(vec) => vec, - Err(_) => return if multi { Value::Array(Vector::new()) } else { Value::Null }, - }; - - // Step 4: Convert results back through JSON to patch_db::Value - if multi { - let items: Vec = selected.into_iter() - .filter_map(|v| { - serde_json::to_string(v).ok() - .and_then(|s| serde_json::from_str(&s).ok()) - }) - .collect(); - Value::Array(items.into_iter().collect()) - } else { - selected.get(0) - .and_then(|v| serde_json::to_string(v).ok()) - .and_then(|s| serde_json::from_str(&s).ok()) - .unwrap_or(Value::Null) - } - } -} -impl fmt::Display for ConfigSelector { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{}", self.src) - } -} -impl Serialize for ConfigSelector { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serializer.serialize_str(&self.src) - } -} -impl<'de> Deserialize<'de> for ConfigSelector { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let src: String = Deserialize::deserialize(deserializer)?; - let compiled = CompiledJsonPath::compile(&src).map_err(serde::de::Error::custom)?; - Ok(Self { src, compiled }) - } -} -impl PartialEq for ConfigSelector { - fn eq(&self, other: &ConfigSelector) -> bool { - self.src == other.src - } -} -impl Eq for ConfigSelector {} -impl PartialOrd for ConfigSelector { - fn partial_cmp(&self, other: &Self) -> Option { - self.src.partial_cmp(&other.src) - } -} -impl Ord for ConfigSelector { - fn cmp(&self, other: &Self) -> std::cmp::Ordering { - self.src.cmp(&other.src) - } -} -impl Hash for ConfigSelector { - fn hash(&self, state: &mut H) { - self.src.hash(state) - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct TorKeyPointer { - package_id: PackageId, - interface: InterfaceId, -} -impl TorKeyPointer { - async fn deref( - &self, - source_package: &PackageId, - secrets: &PgPool, - ) -> Result { - if &self.package_id != source_package { - return Err(ConfigurationError::PermissionDenied( - ValueSpecPointer::Package(PackagePointerSpec::TorKey(self.clone())), - )); - } - let key = Key::for_interface( - secrets - .acquire() - .await - .map_err(|e| ConfigurationError::SystemError(e.into()))? - .as_mut(), - Some((self.package_id.clone(), self.interface.clone())), - ) - .await - .map_err(ConfigurationError::SystemError)?; - Ok(Value::String(Arc::new(base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &key.tor_key().as_bytes(), - )))) - } -} -impl fmt::Display for TorKeyPointer { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "{}: tor-key: {}", self.package_id, self.interface) - } -} - -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -#[serde(tag = "target")] -pub enum SystemPointerSpec {} -impl fmt::Display for SystemPointerSpec { - fn fmt(&self, _f: &mut fmt::Formatter) -> fmt::Result { - // write!(f, "SYSTEM: {}", match *self {}) - Ok(()) - } -} -impl SystemPointerSpec { - async fn deref(&self, _ctx: &RpcContext) -> Result { - #[allow(unreachable_code)] - Ok(match *self {}) - } -} -impl Defaultable for SystemPointerSpec { - type Error = ConfigurationError; - fn gen( - &self, - _rng: &mut R, - _timeout: &Option, - ) -> Result { - Ok(Value::Null) - } -} -#[async_trait] -impl ValueSpec for SystemPointerSpec { - fn matches(&self, _value: &Value) -> Result<(), NoMatchWithPath> { - Ok(()) - } - fn validate(&self, _manifest: &Manifest) -> Result<(), NoMatchWithPath> { - Ok(()) - } - async fn update( - &self, - ctx: &RpcContext, - _manifest: &Manifest, - _config_overrides: &BTreeMap, - value: &mut Value, - ) -> Result<(), ConfigurationError> { - *value = self.deref(ctx).await?; - Ok(()) - } - fn pointers(&self, _value: &Value) -> Result, NoMatchWithPath> { - let mut pointers = BTreeSet::new(); - pointers.insert(ValueSpecPointer::System(self.clone())); - #[allow(unreachable_code)] - Ok(pointers) - } - fn requires(&self, _id: &PackageId, _value: &Value) -> bool { - false - } - fn eq(&self, _lhs: &Value, _rhs: &Value) -> bool { - false - } -} - -#[test] -fn invalid_regex_produces_error() { - assert!( - serde_yaml::from_reader::<_, ConfigSpec>(std::io::Cursor::new(include_bytes!( - "../../test/config-spec/lnd-invalid-regex.yaml" - ))) - .is_err() - ) -} - -#[test] -fn missing_pattern_description_produces_error() { - assert!( - serde_yaml::from_reader::<_, ConfigSpec>(std::io::Cursor::new(include_bytes!( - "../../test/config-spec/lnd-missing-pattern-description.yaml" - ))) - .is_err() - ) -} - -#[test] -fn missing_pattern_produces_error() { - assert!( - serde_yaml::from_reader::<_, ConfigSpec>(std::io::Cursor::new(include_bytes!( - "../../test/config-spec/lnd-missing-pattern.yaml" - ))) - .is_err() - ) -} - -#[test] -fn regex_control() { - let spec = serde_yaml::from_reader::<_, ConfigSpec>(std::io::Cursor::new(include_bytes!( - "../../test/config-spec/lnd-correct.yaml" - ))) - .unwrap(); - println!("{}", serde_json::to_string_pretty(&spec).unwrap()); -} diff --git a/core/startos/src/config/util.rs b/core/startos/src/config/util.rs deleted file mode 100644 index 359c2447..00000000 --- a/core/startos/src/config/util.rs +++ /dev/null @@ -1,406 +0,0 @@ -use std::borrow::Cow; -use std::ops::{Bound, RangeBounds, RangeInclusive}; - -use patch_db::Value; -use rand::distributions::Distribution; -use rand::Rng; - -use super::Config; - -pub const STATIC_NULL: Value = Value::Null; - -#[derive(Clone, Debug)] -pub struct CharSet(pub Vec<(RangeInclusive, usize)>, usize); -impl CharSet { - pub fn contains(&self, c: &char) -> bool { - self.0.iter().any(|r| r.0.contains(c)) - } - pub fn gen(&self, rng: &mut R) -> char { - let mut idx = rng.gen_range(0..self.1); - for r in &self.0 { - if idx < r.1 { - return std::convert::TryFrom::try_from( - rand::distributions::Uniform::new_inclusive( - u32::from(*r.0.start()), - u32::from(*r.0.end()), - ) - .sample(rng), - ) - .unwrap(); - } else { - idx -= r.1; - } - } - unreachable!() - } -} -impl Default for CharSet { - fn default() -> Self { - CharSet(vec![('!'..='~', 94)], 94) - } -} -impl<'de> serde::de::Deserialize<'de> for CharSet { - fn deserialize(deserializer: D) -> Result - where - D: serde::de::Deserializer<'de>, - { - let s = String::deserialize(deserializer)?; - let mut res = Vec::new(); - let mut len = 0; - let mut a: Option = None; - let mut b: Option = None; - let mut in_range = false; - for c in s.chars() { - match c { - ',' => match (a, b, in_range) { - (Some(start), Some(end), _) => { - if !end.is_ascii() { - return Err(serde::de::Error::custom("Invalid Character")); - } - if start >= end { - return Err(serde::de::Error::custom("Invalid Bounds")); - } - let l = u32::from(end) - u32::from(start) + 1; - res.push((start..=end, l as usize)); - len += l as usize; - a = None; - b = None; - in_range = false; - } - (Some(start), None, false) => { - len += 1; - res.push((start..=start, 1)); - a = None; - } - (Some(_), None, true) => { - b = Some(','); - } - (None, None, false) => { - a = Some(','); - } - _ => { - return Err(serde::de::Error::custom("Syntax Error")); - } - }, - '-' => { - if a.is_none() { - a = Some('-'); - } else if !in_range { - in_range = true; - } else if b.is_none() { - b = Some('-') - } else { - return Err(serde::de::Error::custom("Syntax Error")); - } - } - _ => { - if a.is_none() { - a = Some(c); - } else if in_range && b.is_none() { - b = Some(c); - } else { - return Err(serde::de::Error::custom("Syntax Error")); - } - } - } - } - match (a, b) { - (Some(start), Some(end)) => { - if !end.is_ascii() { - return Err(serde::de::Error::custom("Invalid Character")); - } - if start >= end { - return Err(serde::de::Error::custom("Invalid Bounds")); - } - let l = u32::from(end) - u32::from(start) + 1; - res.push((start..=end, l as usize)); - len += l as usize; - } - (Some(c), None) => { - len += 1; - res.push((c..=c, 1)); - } - _ => (), - } - - Ok(CharSet(res, len)) - } -} -impl serde::ser::Serialize for CharSet { - fn serialize(&self, serializer: S) -> Result - where - S: serde::ser::Serializer, - { - <&str>::serialize( - &self - .0 - .iter() - .map(|r| match r.1 { - 1 => format!("{}", r.0.start()), - _ => format!("{}-{}", r.0.start(), r.0.end()), - }) - .collect::>() - .join(",") - .as_str(), - serializer, - ) - } -} - -pub trait MergeWith { - fn merge_with(&mut self, other: &serde_json::Value); -} - -impl MergeWith for serde_json::Value { - fn merge_with(&mut self, other: &serde_json::Value) { - use serde_json::Value::Object; - if let (Object(orig), Object(ref other)) = (self, other) { - for (key, val) in other.into_iter() { - match (orig.get_mut(key), val) { - (Some(new_orig @ Object(_)), other @ Object(_)) => { - new_orig.merge_with(other); - } - (None, _) => { - orig.insert(key.clone(), val.clone()); - } - _ => (), - } - } - } - } -} - -#[test] -fn merge_with_tests() { - use serde_json::json; - - let mut a = json!( - {"a": 1, "c": {"d": "123"}, "i": [1,2,3], "j": {}, "k":[1,2,3], "l": "test"} - ); - a.merge_with( - &json!({"a":"a", "b": "b", "c":{"d":"d", "e":"e"}, "f":{"g":"g"}, "h": [1,2,3], "i":"i", "j":[1,2,3], "k":{}}), - ); - assert_eq!( - a, - json!({"a": 1, "c": {"d": "123", "e":"e"}, "b":"b", "f": {"g":"g"}, "h":[1,2,3], "i":[1,2,3], "j": {}, "k":[1,2,3], "l": "test"}) - ) -} -pub mod serde_regex { - use regex::Regex; - use serde::*; - - pub fn serialize(regex: &Regex, serializer: S) -> Result - where - S: Serializer, - { - <&str>::serialize(®ex.as_str(), serializer) - } - - pub fn deserialize<'de, D>(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = String::deserialize(deserializer)?; - Regex::new(&s).map_err(|e| de::Error::custom(e)) - } -} - -#[derive(Clone, Debug)] -pub struct NumRange( - pub (Bound, Bound), -); -impl std::ops::Deref for NumRange -where - T: std::str::FromStr + std::fmt::Display + std::cmp::PartialOrd, -{ - type Target = (Bound, Bound); - - fn deref(&self) -> &Self::Target { - &self.0 - } -} -impl<'de, T> serde::de::Deserialize<'de> for NumRange -where - T: std::str::FromStr + std::fmt::Display + std::cmp::PartialOrd, - ::Err: std::fmt::Display, -{ - fn deserialize(deserializer: D) -> Result - where - D: serde::de::Deserializer<'de>, - { - let s = String::deserialize(deserializer)?; - let mut split = s.split(","); - let start = split - .next() - .map(|s| match s.get(..1) { - Some("(") => match s.get(1..2) { - Some("*") => Ok(Bound::Unbounded), - _ => s[1..] - .trim() - .parse() - .map(Bound::Excluded) - .map_err(|e| serde::de::Error::custom(e)), - }, - Some("[") => s[1..] - .trim() - .parse() - .map(Bound::Included) - .map_err(|e| serde::de::Error::custom(e)), - _ => Err(serde::de::Error::custom(format!( - "Could not parse left bound: {}", - s - ))), - }) - .transpose()? - .unwrap(); - let end = split - .next() - .map(|s| match s.get(s.len() - 1..) { - Some(")") => match s.get(s.len() - 2..s.len() - 1) { - Some("*") => Ok(Bound::Unbounded), - _ => s[..s.len() - 1] - .trim() - .parse() - .map(Bound::Excluded) - .map_err(|e| serde::de::Error::custom(e)), - }, - Some("]") => s[..s.len() - 1] - .trim() - .parse() - .map(Bound::Included) - .map_err(|e| serde::de::Error::custom(e)), - _ => Err(serde::de::Error::custom(format!( - "Could not parse right bound: {}", - s - ))), - }) - .transpose()? - .unwrap_or(Bound::Unbounded); - - Ok(NumRange((start, end))) - } -} -impl std::fmt::Display for NumRange -where - T: std::str::FromStr + std::fmt::Display + std::cmp::PartialOrd, -{ - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self.start_bound() { - Bound::Excluded(n) => write!(f, "({},", n)?, - Bound::Included(n) => write!(f, "[{},", n)?, - Bound::Unbounded => write!(f, "(*,")?, - }; - match self.end_bound() { - Bound::Excluded(n) => write!(f, "{})", n), - Bound::Included(n) => write!(f, "{}]", n), - Bound::Unbounded => write!(f, "*)"), - } - } -} -impl serde::ser::Serialize for NumRange -where - T: std::str::FromStr + std::fmt::Display + std::cmp::PartialOrd, -{ - fn serialize(&self, serializer: S) -> Result - where - S: serde::ser::Serializer, - { - <&str>::serialize(&format!("{}", self).as_str(), serializer) - } -} - -#[derive(Clone, Debug)] -pub enum UniqueBy { - Any(Vec), - All(Vec), - Exactly(String), - NotUnique, -} -impl UniqueBy { - pub fn eq(&self, lhs: &Config, rhs: &Config) -> bool { - match self { - UniqueBy::Any(any) => any.iter().any(|u| u.eq(lhs, rhs)), - UniqueBy::All(all) => all.iter().all(|u| u.eq(lhs, rhs)), - UniqueBy::Exactly(key) => lhs.get(&**key) == rhs.get(&**key), - UniqueBy::NotUnique => false, - } - } -} -impl Default for UniqueBy { - fn default() -> Self { - UniqueBy::NotUnique - } -} -impl<'de> serde::de::Deserialize<'de> for UniqueBy { - fn deserialize>(deserializer: D) -> Result { - struct Visitor; - impl<'de> serde::de::Visitor<'de> for Visitor { - type Value = UniqueBy; - fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { - write!(formatter, "a key, an \"any\" object, or an \"all\" object") - } - fn visit_str(self, v: &str) -> Result { - Ok(UniqueBy::Exactly(v.to_owned())) - } - fn visit_string(self, v: String) -> Result { - Ok(UniqueBy::Exactly(v)) - } - fn visit_map>( - self, - mut map: A, - ) -> Result { - let mut variant = None; - while let Some(key) = map.next_key::>()? { - match key.as_ref() { - "any" => { - return Ok(UniqueBy::Any(map.next_value()?)); - } - "all" => { - return Ok(UniqueBy::All(map.next_value()?)); - } - _ => { - variant = Some(key); - } - } - } - Err(serde::de::Error::unknown_variant( - variant.unwrap_or_default().as_ref(), - &["any", "all"], - )) - } - fn visit_unit(self) -> Result { - Ok(UniqueBy::NotUnique) - } - fn visit_none(self) -> Result { - Ok(UniqueBy::NotUnique) - } - } - deserializer.deserialize_any(Visitor) - } -} - -impl serde::ser::Serialize for UniqueBy { - fn serialize(&self, serializer: S) -> Result - where - S: serde::ser::Serializer, - { - use serde::ser::SerializeMap; - - match self { - UniqueBy::Any(any) => { - let mut map = serializer.serialize_map(Some(1))?; - map.serialize_key("any")?; - map.serialize_value(any)?; - map.end() - } - UniqueBy::All(all) => { - let mut map = serializer.serialize_map(Some(1))?; - map.serialize_key("all")?; - map.serialize_value(all)?; - map.end() - } - UniqueBy::Exactly(key) => serializer.serialize_str(key), - UniqueBy::NotUnique => serializer.serialize_unit(), - } - } -} diff --git a/core/startos/src/context/cli.rs b/core/startos/src/context/cli.rs deleted file mode 100644 index 020b7345..00000000 --- a/core/startos/src/context/cli.rs +++ /dev/null @@ -1,185 +0,0 @@ -use std::fs::File; -use std::io::BufReader; -use std::net::Ipv4Addr; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use cookie_store::{CookieStore, RawCookie}; -use josekit::jwk::Jwk; -use reqwest::Proxy; -use reqwest_cookie_store::CookieStoreMutex; -use rpc_toolkit::reqwest::{Client, Url}; -use rpc_toolkit::url::Host; -use rpc_toolkit::Context; -use serde::Deserialize; -use tracing::instrument; - -use super::setup::CURRENT_SECRET; -use crate::middleware::auth::LOCAL_AUTH_COOKIE_PATH; -use crate::util::config::{load_config_from_paths, local_config_path}; -use crate::ResultExt; - -#[derive(Debug, Default, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct CliContextConfig { - pub host: Option, - #[serde(deserialize_with = "crate::util::serde::deserialize_from_str_opt")] - #[serde(default)] - pub proxy: Option, - pub cookie_path: Option, -} - -#[derive(Debug)] -pub struct CliContextSeed { - pub base_url: Url, - pub rpc_url: Url, - pub client: Client, - pub cookie_store: Arc, - pub cookie_path: PathBuf, -} -impl Drop for CliContextSeed { - fn drop(&mut self) { - let tmp = format!("{}.tmp", self.cookie_path.display()); - let parent_dir = self.cookie_path.parent().unwrap_or(Path::new("/")); - if !parent_dir.exists() { - std::fs::create_dir_all(&parent_dir).unwrap(); - } - let mut writer = fd_lock_rs::FdLock::lock( - File::create(&tmp).unwrap(), - fd_lock_rs::LockType::Exclusive, - true, - ) - .unwrap(); - let mut store = self.cookie_store.lock().unwrap(); - store.remove("localhost", "", "local"); - store.save_json(&mut *writer).unwrap(); - writer.sync_all().unwrap(); - std::fs::rename(tmp, &self.cookie_path).unwrap(); - } -} - -const DEFAULT_HOST: Host<&'static str> = Host::Ipv4(Ipv4Addr::new(127, 0, 0, 1)); -const DEFAULT_PORT: u16 = 5959; - -#[derive(Debug, Clone)] -pub struct CliContext(Arc); -impl CliContext { - /// BLOCKING - #[instrument(skip_all)] - pub fn init(matches: &ArgMatches) -> Result { - let local_config_path = local_config_path(); - let base: CliContextConfig = load_config_from_paths( - matches - .values_of("config") - .into_iter() - .flatten() - .map(|p| Path::new(p)) - .chain(local_config_path.as_deref().into_iter()) - .chain(std::iter::once(Path::new(crate::util::config::CONFIG_PATH))), - )?; - let mut url = if let Some(host) = matches.value_of("host") { - host.parse()? - } else if let Some(host) = base.host { - host - } else { - "http://localhost".parse()? - }; - let proxy = if let Some(proxy) = matches.value_of("proxy") { - Some(proxy.parse()?) - } else { - base.proxy - }; - - let cookie_path = base.cookie_path.unwrap_or_else(|| { - local_config_path - .as_deref() - .unwrap_or_else(|| Path::new(crate::util::config::CONFIG_PATH)) - .parent() - .unwrap_or(Path::new("/")) - .join(".cookies.json") - }); - let cookie_store = Arc::new(CookieStoreMutex::new({ - let mut store = if cookie_path.exists() { - CookieStore::load_json(BufReader::new(File::open(&cookie_path)?)) - .map_err(|e| eyre!("{}", e)) - .with_kind(crate::ErrorKind::Deserialization)? - } else { - CookieStore::default() - }; - if let Ok(local) = std::fs::read_to_string(LOCAL_AUTH_COOKIE_PATH) { - store - .insert_raw( - &RawCookie::new("local", local), - &"http://localhost".parse()?, - ) - .with_kind(crate::ErrorKind::Network)?; - } - store - })); - - Ok(CliContext(Arc::new(CliContextSeed { - base_url: url.clone(), - rpc_url: { - url.path_segments_mut() - .map_err(|_| eyre!("Url cannot be base")) - .with_kind(crate::ErrorKind::ParseUrl)? - .push("rpc") - .push("v1"); - url - }, - client: { - let mut builder = Client::builder().cookie_provider(cookie_store.clone()); - if let Some(proxy) = proxy { - builder = - builder.proxy(Proxy::all(proxy).with_kind(crate::ErrorKind::ParseUrl)?) - } - builder.build().expect("cannot fail") - }, - cookie_store, - cookie_path, - }))) - } -} -impl AsRef for CliContext { - fn as_ref(&self) -> &Jwk { - &*CURRENT_SECRET - } -} -impl std::ops::Deref for CliContext { - type Target = CliContextSeed; - fn deref(&self) -> &Self::Target { - &*self.0 - } -} -impl Context for CliContext { - fn protocol(&self) -> &str { - self.0.base_url.scheme() - } - fn host(&self) -> Host<&str> { - self.0.base_url.host().unwrap_or(DEFAULT_HOST) - } - fn port(&self) -> u16 { - self.0.base_url.port().unwrap_or(DEFAULT_PORT) - } - fn path(&self) -> &str { - self.0.rpc_url.path() - } - fn url(&self) -> Url { - self.0.rpc_url.clone() - } - fn client(&self) -> &Client { - &self.0.client - } -} -/// When we had an empty proxy the system wasn't working like it used to, which allowed empty proxy -#[test] -fn test_cli_proxy_empty() { - serde_yaml::from_str::( - " - bind_rpc: - ", - ) - .unwrap(); -} diff --git a/core/startos/src/context/diagnostic.rs b/core/startos/src/context/diagnostic.rs deleted file mode 100644 index 151948d7..00000000 --- a/core/startos/src/context/diagnostic.rs +++ /dev/null @@ -1,83 +0,0 @@ -use std::ops::Deref; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use rpc_toolkit::yajrc::RpcError; -use rpc_toolkit::Context; -use serde::Deserialize; -use tokio::sync::broadcast::Sender; -use tracing::instrument; - -use crate::shutdown::Shutdown; -use crate::util::config::load_config_from_paths; -use crate::Error; - -#[derive(Debug, Default, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct DiagnosticContextConfig { - pub datadir: Option, -} -impl DiagnosticContextConfig { - #[instrument(skip_all)] - pub async fn load + Send + 'static>(path: Option

) -> Result { - tokio::task::spawn_blocking(move || { - load_config_from_paths( - path.as_ref() - .into_iter() - .map(|p| p.as_ref()) - .chain(std::iter::once(Path::new( - crate::util::config::DEVICE_CONFIG_PATH, - ))) - .chain(std::iter::once(Path::new(crate::util::config::CONFIG_PATH))), - ) - }) - .await - .unwrap() - } - - pub fn datadir(&self) -> &Path { - self.datadir - .as_deref() - .unwrap_or_else(|| Path::new("/embassy-data")) - } -} - -pub struct DiagnosticContextSeed { - pub datadir: PathBuf, - pub shutdown: Sender>, - pub error: Arc, - pub disk_guid: Option>, -} - -#[derive(Clone)] -pub struct DiagnosticContext(Arc); -impl DiagnosticContext { - #[instrument(skip_all)] - pub async fn init + Send + 'static>( - path: Option

, - disk_guid: Option>, - error: Error, - ) -> Result { - tracing::error!("Error: {}: Starting diagnostic UI", error); - tracing::debug!("{:?}", error); - - let cfg = DiagnosticContextConfig::load(path).await?; - - let (shutdown, _) = tokio::sync::broadcast::channel(1); - - Ok(Self(Arc::new(DiagnosticContextSeed { - datadir: cfg.datadir().to_owned(), - shutdown, - disk_guid, - error: Arc::new(error.into()), - }))) - } -} - -impl Context for DiagnosticContext {} -impl Deref for DiagnosticContext { - type Target = DiagnosticContextSeed; - fn deref(&self) -> &Self::Target { - &*self.0 - } -} diff --git a/core/startos/src/context/install.rs b/core/startos/src/context/install.rs deleted file mode 100644 index 87484b7e..00000000 --- a/core/startos/src/context/install.rs +++ /dev/null @@ -1,58 +0,0 @@ -use std::ops::Deref; -use std::path::Path; -use std::sync::Arc; - -use rpc_toolkit::Context; -use serde::Deserialize; -use tokio::sync::broadcast::Sender; -use tracing::instrument; - -use crate::net::utils::find_eth_iface; -use crate::util::config::load_config_from_paths; -use crate::Error; - -#[derive(Debug, Default, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct InstallContextConfig {} -impl InstallContextConfig { - #[instrument(skip_all)] - pub async fn load + Send + 'static>(path: Option

) -> Result { - tokio::task::spawn_blocking(move || { - load_config_from_paths( - path.as_ref() - .into_iter() - .map(|p| p.as_ref()) - .chain(std::iter::once(Path::new(crate::util::config::CONFIG_PATH))), - ) - }) - .await - .unwrap() - } -} - -pub struct InstallContextSeed { - pub ethernet_interface: String, - pub shutdown: Sender<()>, -} - -#[derive(Clone)] -pub struct InstallContext(Arc); -impl InstallContext { - #[instrument(skip_all)] - pub async fn init + Send + 'static>(path: Option

) -> Result { - let _cfg = InstallContextConfig::load(path.as_ref().map(|p| p.as_ref().to_owned())).await?; - let (shutdown, _) = tokio::sync::broadcast::channel(1); - Ok(Self(Arc::new(InstallContextSeed { - ethernet_interface: find_eth_iface().await?, - shutdown, - }))) - } -} - -impl Context for InstallContext {} -impl Deref for InstallContext { - type Target = InstallContextSeed; - fn deref(&self) -> &Self::Target { - &*self.0 - } -} diff --git a/core/startos/src/context/mod.rs b/core/startos/src/context/mod.rs deleted file mode 100644 index c4e8e775..00000000 --- a/core/startos/src/context/mod.rs +++ /dev/null @@ -1,44 +0,0 @@ -pub mod cli; -pub mod diagnostic; -pub mod install; -pub mod rpc; -pub mod sdk; -pub mod setup; - -pub use cli::CliContext; -pub use diagnostic::DiagnosticContext; -pub use install::InstallContext; -pub use rpc::RpcContext; -pub use sdk::SdkContext; -pub use setup::SetupContext; - -impl From for () { - fn from(_: CliContext) -> Self { - () - } -} -impl From for () { - fn from(_: DiagnosticContext) -> Self { - () - } -} -impl From for () { - fn from(_: RpcContext) -> Self { - () - } -} -impl From for () { - fn from(_: SdkContext) -> Self { - () - } -} -impl From for () { - fn from(_: SetupContext) -> Self { - () - } -} -impl From for () { - fn from(_: InstallContext) -> Self { - () - } -} diff --git a/core/startos/src/context/rpc.rs b/core/startos/src/context/rpc.rs deleted file mode 100644 index 5358a59b..00000000 --- a/core/startos/src/context/rpc.rs +++ /dev/null @@ -1,480 +0,0 @@ -use std::collections::BTreeMap; -use std::net::{Ipv4Addr, SocketAddr, SocketAddrV4}; -use std::ops::Deref; -use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::Arc; -use std::time::Duration; - -use helpers::to_tmp_path; -use josekit::jwk::Jwk; -use patch_db::json_ptr::JsonPointer; -use patch_db::PatchDb; -use reqwest::{Client, Proxy, Url}; -use rpc_toolkit::Context; -use serde::Deserialize; -use sqlx::postgres::PgConnectOptions; -use sqlx::PgPool; -use tokio::sync::{broadcast, oneshot, Mutex, RwLock}; -use tokio::time::Instant; -use tracing::instrument; - -use super::setup::CURRENT_SECRET; -use crate::account::AccountInfo; -use crate::core::rpc_continuations::{RequestGuid, RestHandler, RpcContinuation}; -use crate::db::model::{CurrentDependents, Database, PackageDataEntryMatchModelRef}; -use crate::db::prelude::PatchDbExt; -use crate::dependencies::compute_dependency_config_errs; -use crate::disk::OsPartitionInfo; -use crate::init::{check_time_is_synchronized, init_postgres}; -use crate::install::cleanup::{cleanup_failed, uninstall}; -use crate::manager::ManagerMap; -use crate::middleware::auth::HashSessionToken; -use crate::net::net_controller::NetController; -use crate::net::ssl::{root_ca_start_time, SslManager}; -use crate::net::wifi::WpaCli; -use crate::notifications::NotificationManager; -use crate::shutdown::Shutdown; -use crate::status::MainStatus; -use crate::system::get_mem_info; -use crate::util::config::load_config_from_paths; -use crate::util::lshw::{lshw, LshwDevice}; -use crate::{Error, ErrorKind, ResultExt}; - -#[derive(Debug, Default, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct RpcContextConfig { - pub wifi_interface: Option, - pub ethernet_interface: String, - pub os_partitions: OsPartitionInfo, - pub migration_batch_rows: Option, - pub migration_prefetch_rows: Option, - pub bind_rpc: Option, - pub tor_control: Option, - pub tor_socks: Option, - pub dns_bind: Option>, - pub revision_cache_size: Option, - pub datadir: Option, - pub log_server: Option, -} -impl RpcContextConfig { - pub async fn load + Send + 'static>(path: Option

) -> Result { - tokio::task::spawn_blocking(move || { - load_config_from_paths( - path.as_ref() - .into_iter() - .map(|p| p.as_ref()) - .chain(std::iter::once(Path::new( - crate::util::config::DEVICE_CONFIG_PATH, - ))) - .chain(std::iter::once(Path::new(crate::util::config::CONFIG_PATH))), - ) - }) - .await - .unwrap() - } - pub fn datadir(&self) -> &Path { - self.datadir - .as_deref() - .unwrap_or_else(|| Path::new("/embassy-data")) - } - pub async fn db(&self, account: &AccountInfo) -> Result { - let db_path = self.datadir().join("main").join("embassy.db"); - let db = PatchDb::open(&db_path) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, db_path.display().to_string()))?; - if !db.exists(&::default()).await { - db.put(&::default(), &Database::init(account)) - .await?; - } - Ok(db) - } - #[instrument(skip_all)] - pub async fn secret_store(&self) -> Result { - init_postgres(self.datadir()).await?; - let secret_store = - PgPool::connect_with(PgConnectOptions::new().database("secrets").username("root")) - .await?; - sqlx::migrate!() - .run(&secret_store) - .await - .with_kind(crate::ErrorKind::Database)?; - Ok(secret_store) - } -} - -pub struct RpcContextSeed { - is_closed: AtomicBool, - pub os_partitions: OsPartitionInfo, - pub wifi_interface: Option, - pub ethernet_interface: String, - pub datadir: PathBuf, - pub disk_guid: Arc, - pub db: PatchDb, - pub secret_store: PgPool, - pub account: RwLock, - pub net_controller: Arc, - pub managers: ManagerMap, - pub metrics_cache: RwLock>, - pub shutdown: broadcast::Sender>, - pub tor_socks: SocketAddr, - pub notification_manager: NotificationManager, - pub open_authed_websockets: Mutex>>>, - pub rpc_stream_continuations: Mutex>, - pub wifi_manager: Option>>, - pub current_secret: Arc, - pub client: Client, - pub hardware: Hardware, - pub start_time: Instant, -} - -pub struct Hardware { - pub devices: Vec, - pub ram: u64, -} - -#[derive(Clone)] -pub struct RpcContext(Arc); -impl RpcContext { - #[instrument(skip_all)] - pub async fn init + Send + Sync + 'static>( - cfg_path: Option

, - disk_guid: Arc, - ) -> Result { - let base = RpcContextConfig::load(cfg_path).await?; - tracing::info!("Loaded Config"); - let tor_proxy = base.tor_socks.unwrap_or(SocketAddr::V4(SocketAddrV4::new( - Ipv4Addr::new(127, 0, 0, 1), - 9050, - ))); - let (shutdown, _) = tokio::sync::broadcast::channel(1); - let secret_store = base.secret_store().await?; - tracing::info!("Opened Pg DB"); - let account = AccountInfo::load(&secret_store).await?; - let db = base.db(&account).await?; - tracing::info!("Opened PatchDB"); - let net_controller = Arc::new( - NetController::init( - base.tor_control - .unwrap_or(SocketAddr::from(([127, 0, 0, 1], 9051))), - tor_proxy, - base.dns_bind - .as_deref() - .unwrap_or(&[SocketAddr::from(([127, 0, 0, 1], 53))]), - SslManager::new(&account, root_ca_start_time().await?)?, - &account.hostname, - &account.key, - ) - .await?, - ); - tracing::info!("Initialized Net Controller"); - let managers = ManagerMap::default(); - let metrics_cache = RwLock::>::new(None); - let notification_manager = NotificationManager::new(secret_store.clone()); - tracing::info!("Initialized Notification Manager"); - let tor_proxy_url = format!("socks5h://{tor_proxy}"); - let devices = lshw().await?; - let ram = get_mem_info().await?.total.0 as u64 * 1024 * 1024; - - if !db.peek().await.as_server_info().as_ntp_synced().de()? { - let db = db.clone(); - tokio::spawn(async move { - while !check_time_is_synchronized().await.unwrap() { - tokio::time::sleep(Duration::from_secs(30)).await; - } - db.mutate(|v| v.as_server_info_mut().as_ntp_synced_mut().ser(&true)) - .await - .unwrap() - }); - } - - let seed = Arc::new(RpcContextSeed { - is_closed: AtomicBool::new(false), - datadir: base.datadir().to_path_buf(), - os_partitions: base.os_partitions, - wifi_interface: base.wifi_interface.clone(), - ethernet_interface: base.ethernet_interface, - disk_guid, - db, - secret_store, - account: RwLock::new(account), - net_controller, - managers, - metrics_cache, - shutdown, - tor_socks: tor_proxy, - notification_manager, - open_authed_websockets: Mutex::new(BTreeMap::new()), - rpc_stream_continuations: Mutex::new(BTreeMap::new()), - wifi_manager: base - .wifi_interface - .map(|i| Arc::new(RwLock::new(WpaCli::init(i)))), - current_secret: Arc::new( - Jwk::generate_ec_key(josekit::jwk::alg::ec::EcCurve::P256).map_err(|e| { - tracing::debug!("{:?}", e); - tracing::error!("Couldn't generate ec key"); - Error::new( - color_eyre::eyre::eyre!("Couldn't generate ec key"), - crate::ErrorKind::Unknown, - ) - })?, - ), - client: Client::builder() - .proxy(Proxy::custom(move |url| { - if url.host_str().map_or(false, |h| h.ends_with(".onion")) { - Some(tor_proxy_url.clone()) - } else { - None - } - })) - .build() - .with_kind(crate::ErrorKind::ParseUrl)?, - hardware: Hardware { devices, ram }, - start_time: Instant::now(), - }); - - let res = Self(seed.clone()); - res.cleanup_and_initialize().await?; - tracing::info!("Cleaned up transient states"); - Ok(res) - } - - #[instrument(skip_all)] - pub async fn shutdown(self) -> Result<(), Error> { - self.managers.empty().await?; - self.secret_store.close().await; - self.is_closed.store(true, Ordering::SeqCst); - tracing::info!("RPC Context is shutdown"); - // TODO: shutdown http servers - Ok(()) - } - - #[instrument(skip(self))] - pub async fn cleanup_and_initialize(&self) -> Result<(), Error> { - self.db - .mutate(|f| { - let mut current_dependents = f - .as_package_data() - .keys()? - .into_iter() - .map(|k| (k.clone(), BTreeMap::new())) - .collect::>(); - for (package_id, package) in f.as_package_data_mut().as_entries_mut()? { - for (k, v) in package - .as_installed_mut() - .into_iter() - .flat_map(|i| i.clone().into_current_dependencies().into_entries()) - .flatten() - { - let mut entry: BTreeMap<_, _> = - current_dependents.remove(&k).unwrap_or_default(); - entry.insert(package_id.clone(), v.de()?); - current_dependents.insert(k, entry); - } - } - for (package_id, current_dependents) in current_dependents { - if let Some(deps) = f - .as_package_data_mut() - .as_idx_mut(&package_id) - .and_then(|pde| pde.expect_as_installed_mut().ok()) - .map(|i| i.as_installed_mut().as_current_dependents_mut()) - { - deps.ser(&CurrentDependents(current_dependents))?; - } else if let Some(deps) = f - .as_package_data_mut() - .as_idx_mut(&package_id) - .and_then(|pde| pde.expect_as_removing_mut().ok()) - .map(|i| i.as_removing_mut().as_current_dependents_mut()) - { - deps.ser(&CurrentDependents(current_dependents))?; - } - } - Ok(()) - }) - .await?; - - let peek = self.db.peek().await; - - for (package_id, package) in peek.as_package_data().as_entries()?.into_iter() { - let action = match package.as_match() { - PackageDataEntryMatchModelRef::Installing(_) - | PackageDataEntryMatchModelRef::Restoring(_) - | PackageDataEntryMatchModelRef::Updating(_) => { - cleanup_failed(self, &package_id).await - } - PackageDataEntryMatchModelRef::Removing(_) => { - uninstall( - self, - self.secret_store.acquire().await?.as_mut(), - &package_id, - ) - .await - } - PackageDataEntryMatchModelRef::Installed(m) => { - let version = m.as_manifest().as_version().clone().de()?; - let volumes = m.as_manifest().as_volumes().de()?; - for (volume_id, volume_info) in &*volumes { - let tmp_path = to_tmp_path(volume_info.path_for( - &self.datadir, - &package_id, - &version, - volume_id, - )) - .with_kind(ErrorKind::Filesystem)?; - if tokio::fs::metadata(&tmp_path).await.is_ok() { - tokio::fs::remove_dir_all(&tmp_path).await?; - } - } - Ok(()) - } - _ => continue, - }; - if let Err(e) = action { - tracing::error!("Failed to clean up package {}: {}", package_id, e); - tracing::debug!("{:?}", e); - } - } - let peek = self - .db - .mutate(|v| { - for (_, pde) in v.as_package_data_mut().as_entries_mut()? { - let status = pde - .expect_as_installed_mut()? - .as_installed_mut() - .as_status_mut() - .as_main_mut(); - let running = status.clone().de()?.running(); - status.ser(&if running { - MainStatus::Starting - } else { - MainStatus::Stopped - })?; - } - Ok(v.clone()) - }) - .await?; - self.managers.init(self.clone(), peek.clone()).await?; - tracing::info!("Initialized Package Managers"); - - let mut all_dependency_config_errs = BTreeMap::new(); - for (package_id, package) in peek.as_package_data().as_entries()?.into_iter() { - let package = package.clone(); - if let Some(current_dependencies) = package - .as_installed() - .and_then(|x| x.as_current_dependencies().de().ok()) - { - let manifest = package.as_manifest().de()?; - all_dependency_config_errs.insert( - package_id.clone(), - compute_dependency_config_errs( - self, - &peek, - &manifest, - ¤t_dependencies, - &Default::default(), - ) - .await?, - ); - } - } - self.db - .mutate(|v| { - for (package_id, errs) in all_dependency_config_errs { - if let Some(config_errors) = v - .as_package_data_mut() - .as_idx_mut(&package_id) - .and_then(|pde| pde.as_installed_mut()) - .map(|i| i.as_status_mut().as_dependency_config_errors_mut()) - { - config_errors.ser(&errs)?; - } - } - Ok(()) - }) - .await?; - - Ok(()) - } - - #[instrument(skip_all)] - pub async fn clean_continuations(&self) { - let mut continuations = self.rpc_stream_continuations.lock().await; - let mut to_remove = Vec::new(); - for (guid, cont) in &*continuations { - if cont.is_timed_out() { - to_remove.push(guid.clone()); - } - } - for guid in to_remove { - continuations.remove(&guid); - } - } - - #[instrument(skip_all)] - pub async fn add_continuation(&self, guid: RequestGuid, handler: RpcContinuation) { - self.clean_continuations().await; - self.rpc_stream_continuations - .lock() - .await - .insert(guid, handler); - } - - pub async fn get_continuation_handler(&self, guid: &RequestGuid) -> Option { - let mut continuations = self.rpc_stream_continuations.lock().await; - if let Some(cont) = continuations.remove(guid) { - cont.into_handler().await - } else { - None - } - } - - pub async fn get_ws_continuation_handler(&self, guid: &RequestGuid) -> Option { - let continuations = self.rpc_stream_continuations.lock().await; - if matches!(continuations.get(guid), Some(RpcContinuation::WebSocket(_))) { - drop(continuations); - self.get_continuation_handler(guid).await - } else { - None - } - } - - pub async fn get_rest_continuation_handler(&self, guid: &RequestGuid) -> Option { - let continuations = self.rpc_stream_continuations.lock().await; - if matches!(continuations.get(guid), Some(RpcContinuation::Rest(_))) { - drop(continuations); - self.get_continuation_handler(guid).await - } else { - None - } - } -} -impl AsRef for RpcContext { - fn as_ref(&self) -> &Jwk { - &CURRENT_SECRET - } -} -impl Context for RpcContext {} -impl Deref for RpcContext { - type Target = RpcContextSeed; - fn deref(&self) -> &Self::Target { - #[cfg(feature = "unstable")] - if self.0.is_closed.load(Ordering::SeqCst) { - panic!( - "RpcContext used after shutdown! {}", - tracing_error::SpanTrace::capture() - ); - } - &self.0 - } -} -impl Drop for RpcContext { - fn drop(&mut self) { - #[cfg(feature = "unstable")] - if self.0.is_closed.load(Ordering::SeqCst) { - tracing::info!( - "RpcContext dropped. {} left.", - Arc::strong_count(&self.0) - 1 - ); - } - } -} diff --git a/core/startos/src/context/sdk.rs b/core/startos/src/context/sdk.rs deleted file mode 100644 index 7ba7a6bf..00000000 --- a/core/startos/src/context/sdk.rs +++ /dev/null @@ -1,76 +0,0 @@ -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use rpc_toolkit::Context; -use serde::Deserialize; -use tracing::instrument; - -use crate::prelude::*; -use crate::util::config::{load_config_from_paths, local_config_path}; - -#[derive(Debug, Default, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct SdkContextConfig { - pub developer_key_path: Option, -} - -#[derive(Debug)] -pub struct SdkContextSeed { - pub developer_key_path: PathBuf, -} - -#[derive(Debug, Clone)] -pub struct SdkContext(Arc); -impl SdkContext { - /// BLOCKING - #[instrument(skip_all)] - pub fn init(matches: &ArgMatches) -> Result { - let local_config_path = local_config_path(); - let base: SdkContextConfig = load_config_from_paths( - matches - .values_of("config") - .into_iter() - .flatten() - .map(|p| Path::new(p)) - .chain(local_config_path.as_deref().into_iter()) - .chain(std::iter::once(Path::new(crate::util::config::CONFIG_PATH))), - )?; - Ok(SdkContext(Arc::new(SdkContextSeed { - developer_key_path: base.developer_key_path.unwrap_or_else(|| { - local_config_path - .as_deref() - .unwrap_or_else(|| Path::new(crate::util::config::CONFIG_PATH)) - .parent() - .unwrap_or(Path::new("/")) - .join("developer.key.pem") - }), - }))) - } - /// BLOCKING - #[instrument(skip_all)] - pub fn developer_key(&self) -> Result { - if !self.developer_key_path.exists() { - return Err(Error::new(eyre!("Developer Key does not exist! Please run `start-sdk init` before running this command."), crate::ErrorKind::Uninitialized)); - } - let pair = ::from_pkcs8_pem( - &std::fs::read_to_string(&self.developer_key_path)?, - ) - .with_kind(crate::ErrorKind::Pem)?; - let secret = ed25519_dalek::SecretKey::try_from(&pair.secret_key[..]).map_err(|_| { - Error::new( - eyre!("pkcs8 key is of incorrect length"), - ErrorKind::OpenSsl, - ) - })?; - Ok(secret.into()) - } -} -impl std::ops::Deref for SdkContext { - type Target = SdkContextSeed; - fn deref(&self) -> &Self::Target { - &*self.0 - } -} -impl Context for SdkContext {} diff --git a/core/startos/src/context/setup.rs b/core/startos/src/context/setup.rs deleted file mode 100644 index 7ae161b0..00000000 --- a/core/startos/src/context/setup.rs +++ /dev/null @@ -1,149 +0,0 @@ -use std::ops::Deref; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use josekit::jwk::Jwk; -use patch_db::json_ptr::JsonPointer; -use patch_db::PatchDb; -use rpc_toolkit::yajrc::RpcError; -use rpc_toolkit::Context; -use serde::{Deserialize, Serialize}; -use sqlx::postgres::PgConnectOptions; -use sqlx::PgPool; -use tokio::sync::broadcast::Sender; -use tokio::sync::RwLock; -use tracing::instrument; - -use crate::account::AccountInfo; -use crate::db::model::Database; -use crate::disk::OsPartitionInfo; -use crate::init::init_postgres; -use crate::setup::SetupStatus; -use crate::util::config::load_config_from_paths; -use crate::{Error, ResultExt}; - -lazy_static::lazy_static! { - pub static ref CURRENT_SECRET: Jwk = Jwk::generate_ec_key(josekit::jwk::alg::ec::EcCurve::P256).unwrap_or_else(|e| { - tracing::debug!("{:?}", e); - tracing::error!("Couldn't generate ec key"); - panic!("Couldn't generate ec key") - }); -} - -#[derive(Clone, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct SetupResult { - pub tor_address: String, - pub lan_address: String, - pub root_ca: String, -} - -#[derive(Debug, Default, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct SetupContextConfig { - pub os_partitions: OsPartitionInfo, - pub migration_batch_rows: Option, - pub migration_prefetch_rows: Option, - pub datadir: Option, - #[serde(default)] - pub disable_encryption: bool, -} -impl SetupContextConfig { - #[instrument(skip_all)] - pub async fn load + Send + 'static>(path: Option

) -> Result { - tokio::task::spawn_blocking(move || { - load_config_from_paths( - path.as_ref() - .into_iter() - .map(|p| p.as_ref()) - .chain(std::iter::once(Path::new( - crate::util::config::DEVICE_CONFIG_PATH, - ))) - .chain(std::iter::once(Path::new(crate::util::config::CONFIG_PATH))), - ) - }) - .await - .unwrap() - } - pub fn datadir(&self) -> &Path { - self.datadir - .as_deref() - .unwrap_or_else(|| Path::new("/embassy-data")) - } -} - -pub struct SetupContextSeed { - pub os_partitions: OsPartitionInfo, - pub config_path: Option, - pub migration_batch_rows: usize, - pub migration_prefetch_rows: usize, - pub disable_encryption: bool, - pub shutdown: Sender<()>, - pub datadir: PathBuf, - pub selected_v2_drive: RwLock>, - pub cached_product_key: RwLock>>, - pub setup_status: RwLock>>, - pub setup_result: RwLock, SetupResult)>>, -} - -impl AsRef for SetupContextSeed { - fn as_ref(&self) -> &Jwk { - &*CURRENT_SECRET - } -} - -#[derive(Clone)] -pub struct SetupContext(Arc); -impl SetupContext { - #[instrument(skip_all)] - pub async fn init + Send + 'static>(path: Option

) -> Result { - let cfg = SetupContextConfig::load(path.as_ref().map(|p| p.as_ref().to_owned())).await?; - let (shutdown, _) = tokio::sync::broadcast::channel(1); - let datadir = cfg.datadir().to_owned(); - Ok(Self(Arc::new(SetupContextSeed { - os_partitions: cfg.os_partitions, - config_path: path.as_ref().map(|p| p.as_ref().to_owned()), - migration_batch_rows: cfg.migration_batch_rows.unwrap_or(25000), - migration_prefetch_rows: cfg.migration_prefetch_rows.unwrap_or(100_000), - disable_encryption: cfg.disable_encryption, - shutdown, - datadir, - selected_v2_drive: RwLock::new(None), - cached_product_key: RwLock::new(None), - setup_status: RwLock::new(None), - setup_result: RwLock::new(None), - }))) - } - #[instrument(skip_all)] - pub async fn db(&self, account: &AccountInfo) -> Result { - let db_path = self.datadir.join("main").join("embassy.db"); - let db = PatchDb::open(&db_path) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, db_path.display().to_string()))?; - if !db.exists(&::default()).await { - db.put(&::default(), &Database::init(account)) - .await?; - } - Ok(db) - } - #[instrument(skip_all)] - pub async fn secret_store(&self) -> Result { - init_postgres(&self.datadir).await?; - let secret_store = - PgPool::connect_with(PgConnectOptions::new().database("secrets").username("root")) - .await?; - sqlx::migrate!() - .run(&secret_store) - .await - .with_kind(crate::ErrorKind::Database)?; - Ok(secret_store) - } -} - -impl Context for SetupContext {} -impl Deref for SetupContext { - type Target = SetupContextSeed; - fn deref(&self) -> &Self::Target { - &*self.0 - } -} diff --git a/core/startos/src/control.rs b/core/startos/src/control.rs deleted file mode 100644 index 58e39ac1..00000000 --- a/core/startos/src/control.rs +++ /dev/null @@ -1,92 +0,0 @@ -use color_eyre::eyre::eyre; -use rpc_toolkit::command; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::status::MainStatus; -use crate::util::display_none; -use crate::Error; - -#[command(display(display_none), metadata(sync_db = true))] -#[instrument(skip_all)] -pub async fn start(#[context] ctx: RpcContext, #[arg] id: PackageId) -> Result<(), Error> { - let peek = ctx.db.peek().await; - let version = peek - .as_package_data() - .as_idx(&id) - .or_not_found(&id)? - .as_installed() - .or_not_found(&id)? - .as_manifest() - .as_version() - .de()?; - - ctx.managers - .get(&(id, version)) - .await - .ok_or_else(|| Error::new(eyre!("Manager not found"), crate::ErrorKind::InvalidRequest))? - .start() - .await; - - Ok(()) -} - -#[command(display(display_none), metadata(sync_db = true))] -pub async fn stop(#[context] ctx: RpcContext, #[arg] id: PackageId) -> Result { - let peek = ctx.db.peek().await; - let version = peek - .as_package_data() - .as_idx(&id) - .or_not_found(&id)? - .as_installed() - .or_not_found(&id)? - .as_manifest() - .as_version() - .de()?; - - let last_statuts = ctx - .db - .mutate(|v| { - v.as_package_data_mut() - .as_idx_mut(&id) - .and_then(|x| x.as_installed_mut()) - .ok_or_else(|| Error::new(eyre!("{} is not installed", id), ErrorKind::NotFound))? - .as_status_mut() - .as_main_mut() - .replace(&MainStatus::Stopping) - }) - .await?; - - ctx.managers - .get(&(id, version)) - .await - .ok_or_else(|| Error::new(eyre!("Manager not found"), crate::ErrorKind::InvalidRequest))? - .stop() - .await; - - Ok(last_statuts) -} - -#[command(display(display_none), metadata(sync_db = true))] -pub async fn restart(#[context] ctx: RpcContext, #[arg] id: PackageId) -> Result<(), Error> { - let peek = ctx.db.peek().await; - let version = peek - .as_package_data() - .as_idx(&id) - .or_not_found(&id)? - .expect_as_installed()? - .as_manifest() - .as_version() - .de()?; - - ctx.managers - .get(&(id, version)) - .await - .ok_or_else(|| Error::new(eyre!("Manager not found"), crate::ErrorKind::InvalidRequest))? - .restart() - .await; - - Ok(()) -} diff --git a/core/startos/src/core/mod.rs b/core/startos/src/core/mod.rs deleted file mode 100644 index 7c2dbbb0..00000000 --- a/core/startos/src/core/mod.rs +++ /dev/null @@ -1 +0,0 @@ -pub mod rpc_continuations; diff --git a/core/startos/src/core/rpc_continuations.rs b/core/startos/src/core/rpc_continuations.rs deleted file mode 100644 index 45a1c1b0..00000000 --- a/core/startos/src/core/rpc_continuations.rs +++ /dev/null @@ -1,116 +0,0 @@ -use std::sync::Arc; -use std::time::Duration; - -use futures::future::BoxFuture; -use futures::FutureExt; -use helpers::TimedResource; -use hyper::upgrade::Upgraded; -use hyper::{Body, Error as HyperError, Request, Response}; -use rand::RngCore; -use tokio::task::JoinError; -use tokio_tungstenite::WebSocketStream; - -use crate::{Error, ResultExt}; - -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize, serde::Deserialize)] -pub struct RequestGuid = String>(Arc); -impl RequestGuid { - pub fn new() -> Self { - let mut buf = [0; 40]; - rand::thread_rng().fill_bytes(&mut buf); - RequestGuid(Arc::new(base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &buf, - ))) - } - - pub fn from(r: &str) -> Option { - if r.len() != 64 { - return None; - } - for c in r.chars() { - if !(c >= 'A' && c <= 'Z' || c >= '2' && c <= '7') { - return None; - } - } - Some(RequestGuid(Arc::new(r.to_owned()))) - } -} -#[test] -fn parse_guid() { - println!( - "{:?}", - RequestGuid::from(&format!("{}", RequestGuid::new())) - ) -} - -impl> std::fmt::Display for RequestGuid { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - (&*self.0).as_ref().fmt(f) - } -} - -pub type RestHandler = Box< - dyn FnOnce(Request) -> BoxFuture<'static, Result, crate::Error>> + Send, ->; - -pub type WebSocketHandler = Box< - dyn FnOnce( - BoxFuture<'static, Result, HyperError>, JoinError>>, - ) -> BoxFuture<'static, Result<(), Error>> - + Send, ->; - -pub enum RpcContinuation { - Rest(TimedResource), - WebSocket(TimedResource), -} -impl RpcContinuation { - pub fn rest(handler: RestHandler, timeout: Duration) -> Self { - RpcContinuation::Rest(TimedResource::new(handler, timeout)) - } - pub fn ws(handler: WebSocketHandler, timeout: Duration) -> Self { - RpcContinuation::WebSocket(TimedResource::new(handler, timeout)) - } - pub fn is_timed_out(&self) -> bool { - match self { - RpcContinuation::Rest(a) => a.is_timed_out(), - RpcContinuation::WebSocket(a) => a.is_timed_out(), - } - } - pub async fn into_handler(self) -> Option { - match self { - RpcContinuation::Rest(handler) => handler.get().await, - RpcContinuation::WebSocket(handler) => { - if let Some(handler) = handler.get().await { - Some(Box::new( - |req: Request| -> BoxFuture<'static, Result, Error>> { - async move { - let (parts, body) = req.into_parts(); - let req = Request::from_parts(parts, body); - let (res, ws_fut) = hyper_ws_listener::create_ws(req) - .with_kind(crate::ErrorKind::Network)?; - if let Some(ws_fut) = ws_fut { - tokio::task::spawn(async move { - match handler(ws_fut.boxed()).await { - Ok(()) => (), - Err(e) => { - tracing::error!("WebSocket Closed: {}", e); - tracing::debug!("{:?}", e); - } - } - }); - } - - Ok(res) - } - .boxed() - }, - )) - } else { - None - } - } - } - } -} diff --git a/core/startos/src/db/mod.rs b/core/startos/src/db/mod.rs deleted file mode 100644 index 03ad9433..00000000 --- a/core/startos/src/db/mod.rs +++ /dev/null @@ -1,370 +0,0 @@ -pub mod model; -pub mod package; -pub mod prelude; - -use std::future::Future; -use std::path::PathBuf; -use std::sync::Arc; - -use futures::{FutureExt, SinkExt, StreamExt}; -use patch_db::json_ptr::JsonPointer; -use patch_db::{Dump, Revision}; -use rpc_toolkit::command; -use rpc_toolkit::hyper::upgrade::Upgraded; -use rpc_toolkit::hyper::{Body, Error as HyperError, Request, Response}; -use rpc_toolkit::yajrc::RpcError; -use serde::{Deserialize, Serialize}; -use serde_json::Value; -use tokio::sync::oneshot; -use tokio::task::JoinError; -use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode; -use tokio_tungstenite::tungstenite::protocol::CloseFrame; -use tokio_tungstenite::tungstenite::Message; -use tokio_tungstenite::WebSocketStream; -use tracing::instrument; - -use crate::context::{CliContext, RpcContext}; -use crate::middleware::auth::{HasValidSession, HashSessionToken}; -use crate::prelude::*; -use crate::util::display_none; -use crate::util::serde::{display_serializable, IoFormat}; - -#[instrument(skip_all)] -async fn ws_handler< - WSFut: Future, HyperError>, JoinError>>, ->( - ctx: RpcContext, - session: Option<(HasValidSession, HashSessionToken)>, - ws_fut: WSFut, -) -> Result<(), Error> { - let (dump, sub) = ctx.db.dump_and_sub().await; - let mut stream = ws_fut - .await - .with_kind(ErrorKind::Network)? - .with_kind(ErrorKind::Unknown)?; - - if let Some((session, token)) = session { - let kill = subscribe_to_session_kill(&ctx, token).await; - send_dump(session, &mut stream, dump).await?; - - deal_with_messages(session, kill, sub, stream).await?; - } else { - stream - .close(Some(CloseFrame { - code: CloseCode::Error, - reason: "UNAUTHORIZED".into(), - })) - .await - .with_kind(ErrorKind::Network)?; - } - - Ok(()) -} - -async fn subscribe_to_session_kill( - ctx: &RpcContext, - token: HashSessionToken, -) -> oneshot::Receiver<()> { - let (send, recv) = oneshot::channel(); - let mut guard = ctx.open_authed_websockets.lock().await; - if !guard.contains_key(&token) { - guard.insert(token, vec![send]); - } else { - guard.get_mut(&token).unwrap().push(send); - } - recv -} - -#[instrument(skip_all)] -async fn deal_with_messages( - _has_valid_authentication: HasValidSession, - mut kill: oneshot::Receiver<()>, - mut sub: patch_db::Subscriber, - mut stream: WebSocketStream, -) -> Result<(), Error> { - let mut timer = tokio::time::interval(tokio::time::Duration::from_secs(5)); - - loop { - futures::select! { - _ = (&mut kill).fuse() => { - tracing::info!("Closing WebSocket: Reason: Session Terminated"); - stream - .close(Some(CloseFrame { - code: CloseCode::Error, - reason: "UNAUTHORIZED".into(), - })) - .await - .with_kind(ErrorKind::Network)?; - return Ok(()) - } - new_rev = sub.recv().fuse() => { - let rev = new_rev.expect("UNREACHABLE: patch-db is dropped"); - stream - .send(Message::Text(serde_json::to_string(&rev).with_kind(ErrorKind::Serialization)?)) - .await - .with_kind(ErrorKind::Network)?; - } - message = stream.next().fuse() => { - let message = message.transpose().with_kind(ErrorKind::Network)?; - match message { - None => { - tracing::info!("Closing WebSocket: Stream Finished"); - return Ok(()) - } - _ => (), - } - } - // This is trying to give a health checks to the home to keep the ui alive. - _ = timer.tick().fuse() => { - stream - .send(Message::Ping(vec![])) - .await - .with_kind(crate::ErrorKind::Network)?; - } - } - } -} - -async fn send_dump( - _has_valid_authentication: HasValidSession, - stream: &mut WebSocketStream, - dump: Dump, -) -> Result<(), Error> { - stream - .send(Message::Text( - serde_json::to_string(&dump).with_kind(ErrorKind::Serialization)?, - )) - .await - .with_kind(ErrorKind::Network)?; - Ok(()) -} - -pub async fn subscribe(ctx: RpcContext, req: Request) -> Result, Error> { - let (parts, body) = req.into_parts(); - let session = match async { - let token = HashSessionToken::from_request_parts(&parts)?; - let session = HasValidSession::from_request_parts(&parts, &ctx).await?; - Ok::<_, Error>((session, token)) - } - .await - { - Ok(a) => Some(a), - Err(e) => { - if e.kind != ErrorKind::Authorization { - tracing::error!("Error Authenticating Websocket: {}", e); - tracing::debug!("{:?}", e); - } - None - } - }; - let req = Request::from_parts(parts, body); - let (res, ws_fut) = hyper_ws_listener::create_ws(req).with_kind(ErrorKind::Network)?; - if let Some(ws_fut) = ws_fut { - tokio::task::spawn(async move { - match ws_handler(ctx, session, ws_fut).await { - Ok(()) => (), - Err(e) => { - tracing::error!("WebSocket Closed: {}", e); - tracing::debug!("{:?}", e); - } - } - }); - } - - Ok(res) -} - -#[command(subcommands(dump, put, apply))] -pub fn db() -> Result<(), RpcError> { - Ok(()) -} - -#[derive(Deserialize, Serialize)] -#[serde(untagged)] -pub enum RevisionsRes { - Revisions(Vec>), - Dump(Dump), -} - -#[instrument(skip_all)] -async fn cli_dump( - ctx: CliContext, - _format: Option, - path: Option, -) -> Result { - let dump = if let Some(path) = path { - PatchDb::open(path).await?.dump().await - } else { - rpc_toolkit::command_helpers::call_remote( - ctx, - "db.dump", - serde_json::json!({}), - std::marker::PhantomData::, - ) - .await? - .result? - }; - - Ok(dump) -} - -#[command( - custom_cli(cli_dump(async, context(CliContext))), - display(display_serializable) -)] -pub async fn dump( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, - #[allow(unused_variables)] - #[arg] - path: Option, -) -> Result { - Ok(ctx.db.dump().await) -} - -fn apply_expr(input: jaq_core::Val, expr: &str) -> Result { - let (expr, errs) = jaq_core::parse::parse(expr, jaq_core::parse::main()); - - let Some(expr) = expr else { - return Err(Error::new( - eyre!("Failed to parse expression: {:?}", errs), - crate::ErrorKind::InvalidRequest, - )); - }; - - let mut errs = Vec::new(); - - let mut defs = jaq_core::Definitions::core(); - for def in jaq_std::std() { - defs.insert(def, &mut errs); - } - - let filter = defs.finish(expr, Vec::new(), &mut errs); - - if !errs.is_empty() { - return Err(Error::new( - eyre!("Failed to compile expression: {:?}", errs), - crate::ErrorKind::InvalidRequest, - )); - }; - - let inputs = jaq_core::RcIter::new(std::iter::empty()); - let mut res_iter = filter.run(jaq_core::Ctx::new([], &inputs), input); - - let Some(res) = res_iter - .next() - .transpose() - .map_err(|e| eyre!("{e}")) - .with_kind(crate::ErrorKind::Deserialization)? - else { - return Err(Error::new( - eyre!("expr returned no results"), - crate::ErrorKind::InvalidRequest, - )); - }; - - if res_iter.next().is_some() { - return Err(Error::new( - eyre!("expr returned too many results"), - crate::ErrorKind::InvalidRequest, - )); - } - - Ok(res) -} - -#[instrument(skip_all)] -async fn cli_apply(ctx: CliContext, expr: String, path: Option) -> Result<(), RpcError> { - if let Some(path) = path { - PatchDb::open(path) - .await? - .mutate(|db| { - let res = apply_expr( - serde_json::to_value(patch_db::Value::from(db.clone())) - .with_kind(ErrorKind::Deserialization)? - .into(), - &expr, - )?; - - db.ser( - &serde_json::from_value::(res.clone().into()).with_ctx( - |_| { - ( - crate::ErrorKind::Deserialization, - "result does not match database model", - ) - }, - )?, - ) - }) - .await?; - } else { - rpc_toolkit::command_helpers::call_remote( - ctx, - "db.apply", - serde_json::json!({ "expr": expr }), - std::marker::PhantomData::<()>, - ) - .await? - .result?; - } - - Ok(()) -} - -#[command( - custom_cli(cli_apply(async, context(CliContext))), - display(display_none) -)] -pub async fn apply( - #[context] ctx: RpcContext, - #[arg] expr: String, - #[allow(unused_variables)] - #[arg] - path: Option, -) -> Result<(), Error> { - ctx.db - .mutate(|db| { - let res = apply_expr( - serde_json::to_value(patch_db::Value::from(db.clone())) - .with_kind(ErrorKind::Deserialization)? - .into(), - &expr, - )?; - - db.ser( - &serde_json::from_value::(res.clone().into()).with_ctx(|_| { - ( - crate::ErrorKind::Deserialization, - "result does not match database model", - ) - })?, - ) - }) - .await -} - -#[command(subcommands(ui))] -pub fn put() -> Result<(), RpcError> { - Ok(()) -} - -#[command(display(display_serializable))] -#[instrument(skip_all)] -pub async fn ui( - #[context] ctx: RpcContext, - #[arg] pointer: JsonPointer, - #[arg] value: Value, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result<(), Error> { - let ptr = "/ui" - .parse::() - .with_kind(ErrorKind::Database)? - + &pointer; - ctx.db.put(&ptr, &value).await?; - Ok(()) -} diff --git a/core/startos/src/db/model.rs b/core/startos/src/db/model.rs deleted file mode 100644 index abd3a2db..00000000 --- a/core/startos/src/db/model.rs +++ /dev/null @@ -1,530 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; -use std::net::{Ipv4Addr, Ipv6Addr}; -use std::sync::Arc; - -use chrono::{DateTime, Utc}; -use emver::VersionRange; -use ipnet::{Ipv4Net, Ipv6Net}; -use isocountry::CountryCode; -use itertools::Itertools; -use models::{DataUrl, HealthCheckId, InterfaceId}; -use openssl::hash::MessageDigest; -use patch_db::value::InternedString; -use patch_db::{HasModel, Value}; -use reqwest::Url; -use serde::{Deserialize, Serialize}; -use ssh_key::public::Ed25519PublicKey; - -use crate::account::AccountInfo; -use crate::config::spec::PackagePointerSpec; -use crate::install::progress::InstallProgress; -use crate::net::utils::{get_iface_ipv4_addr, get_iface_ipv6_addr}; -use crate::prelude::*; -use crate::s9pk::manifest::{Manifest, PackageId}; -use crate::status::Status; -use crate::util::cpupower::{Governor}; -use crate::util::Version; -use crate::version::{Current, VersionT}; -use crate::{ARCH, PLATFORM}; - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -// #[macro_debug] -pub struct Database { - pub server_info: ServerInfo, - pub package_data: AllPackageData, - pub ui: Value, -} -impl Database { - pub fn init(account: &AccountInfo) -> Self { - let lan_address = account.hostname.lan_address().parse().unwrap(); - Database { - server_info: ServerInfo { - arch: get_arch(), - platform: get_platform(), - id: account.server_id.clone(), - version: Current::new().semver().into(), - hostname: account.hostname.no_dot_host_name(), - last_backup: None, - last_wifi_region: None, - eos_version_compat: Current::new().compat().clone(), - lan_address, - tor_address: format!("https://{}", account.key.tor_address()) - .parse() - .unwrap(), - ip_info: BTreeMap::new(), - status_info: ServerStatus { - backup_progress: None, - updated: false, - update_progress: None, - shutting_down: false, - restarting: false, - }, - wifi: WifiInfo { - ssids: Vec::new(), - connected: None, - selected: None, - }, - unread_notification_count: 0, - connection_addresses: ConnectionAddresses { - tor: Vec::new(), - clearnet: Vec::new(), - }, - password_hash: account.password.clone(), - pubkey: ssh_key::PublicKey::from(Ed25519PublicKey::from(&account.key.ssh_key())) - .to_openssh() - .unwrap(), - ca_fingerprint: account - .root_ca_cert - .digest(MessageDigest::sha256()) - .unwrap() - .iter() - .map(|x| format!("{x:X}")) - .join(":"), - ntp_synced: false, - zram: true, - governor: None, - }, - package_data: AllPackageData::default(), - ui: serde_json::from_str(include_str!(concat!( - env!("CARGO_MANIFEST_DIR"), - "/../../web/patchdb-ui-seed.json" - ))) - .unwrap(), - } - } -} - -pub type DatabaseModel = Model; - -fn get_arch() -> InternedString { - (*ARCH).into() -} - -fn get_platform() -> InternedString { - (&*PLATFORM).into() -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct ServerInfo { - #[serde(default = "get_arch")] - pub arch: InternedString, - #[serde(default = "get_platform")] - pub platform: InternedString, - pub id: String, - pub hostname: String, - pub version: Version, - pub last_backup: Option>, - /// Used in the wifi to determine the region to set the system to - pub last_wifi_region: Option, - pub eos_version_compat: VersionRange, - pub lan_address: Url, - pub tor_address: Url, - pub ip_info: BTreeMap, - #[serde(default)] - pub status_info: ServerStatus, - pub wifi: WifiInfo, - pub unread_notification_count: u64, - pub connection_addresses: ConnectionAddresses, - pub password_hash: String, - pub pubkey: String, - pub ca_fingerprint: String, - #[serde(default)] - pub ntp_synced: bool, - #[serde(default)] - pub zram: bool, - pub governor: Option, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct IpInfo { - pub ipv4_range: Option, - pub ipv4: Option, - pub ipv6_range: Option, - pub ipv6: Option, -} -impl IpInfo { - pub async fn for_interface(iface: &str) -> Result { - let (ipv4, ipv4_range) = get_iface_ipv4_addr(iface).await?.unzip(); - let (ipv6, ipv6_range) = get_iface_ipv6_addr(iface).await?.unzip(); - Ok(Self { - ipv4_range, - ipv4, - ipv6_range, - ipv6, - }) - } -} - -#[derive(Debug, Default, Deserialize, Serialize, HasModel)] -#[model = "Model"] -pub struct BackupProgress { - pub complete: bool, -} - -#[derive(Debug, Default, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct ServerStatus { - pub backup_progress: Option>, - pub updated: bool, - pub update_progress: Option, - #[serde(default)] - pub shutting_down: bool, - #[serde(default)] - pub restarting: bool, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct UpdateProgress { - pub size: Option, - pub downloaded: u64, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct WifiInfo { - pub ssids: Vec, - pub selected: Option, - pub connected: Option, -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct ServerSpecs { - pub cpu: String, - pub disk: String, - pub memory: String, -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct ConnectionAddresses { - pub tor: Vec, - pub clearnet: Vec, -} - -#[derive(Debug, Default, Deserialize, Serialize)] -pub struct AllPackageData(pub BTreeMap); -impl Map for AllPackageData { - type Key = PackageId; - type Value = PackageDataEntry; -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct StaticFiles { - license: String, - instructions: String, - icon: String, -} -impl StaticFiles { - pub fn local(id: &PackageId, version: &Version, icon_type: &str) -> Self { - StaticFiles { - license: format!("/public/package-data/{}/{}/LICENSE.md", id, version), - instructions: format!("/public/package-data/{}/{}/INSTRUCTIONS.md", id, version), - icon: format!("/public/package-data/{}/{}/icon.{}", id, version, icon_type), - } - } -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct PackageDataEntryInstalling { - pub static_files: StaticFiles, - pub manifest: Manifest, - pub install_progress: Arc, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct PackageDataEntryUpdating { - pub static_files: StaticFiles, - pub manifest: Manifest, - pub installed: InstalledPackageInfo, - pub install_progress: Arc, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct PackageDataEntryRestoring { - pub static_files: StaticFiles, - pub manifest: Manifest, - pub install_progress: Arc, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct PackageDataEntryRemoving { - pub static_files: StaticFiles, - pub manifest: Manifest, - pub removing: InstalledPackageInfo, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct PackageDataEntryInstalled { - pub static_files: StaticFiles, - pub manifest: Manifest, - pub installed: InstalledPackageInfo, -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(tag = "state")] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -// #[macro_debug] -pub enum PackageDataEntry { - Installing(PackageDataEntryInstalling), - Updating(PackageDataEntryUpdating), - Restoring(PackageDataEntryRestoring), - Removing(PackageDataEntryRemoving), - Installed(PackageDataEntryInstalled), -} -impl Model { - pub fn expect_into_installed(self) -> Result, Error> { - if let PackageDataEntryMatchModel::Installed(a) = self.into_match() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in installed state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn expect_as_installed(&self) -> Result<&Model, Error> { - if let PackageDataEntryMatchModelRef::Installed(a) = self.as_match() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in installed state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn expect_as_installed_mut( - &mut self, - ) -> Result<&mut Model, Error> { - if let PackageDataEntryMatchModelMut::Installed(a) = self.as_match_mut() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in installed state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn expect_into_removing(self) -> Result, Error> { - if let PackageDataEntryMatchModel::Removing(a) = self.into_match() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in removing state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn expect_as_removing(&self) -> Result<&Model, Error> { - if let PackageDataEntryMatchModelRef::Removing(a) = self.as_match() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in removing state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn expect_as_removing_mut( - &mut self, - ) -> Result<&mut Model, Error> { - if let PackageDataEntryMatchModelMut::Removing(a) = self.as_match_mut() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in removing state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn expect_as_installing_mut( - &mut self, - ) -> Result<&mut Model, Error> { - if let PackageDataEntryMatchModelMut::Installing(a) = self.as_match_mut() { - Ok(a) - } else { - Err(Error::new( - eyre!("package is not in installing state"), - ErrorKind::InvalidRequest, - )) - } - } - pub fn into_manifest(self) -> Model { - match self.into_match() { - PackageDataEntryMatchModel::Installing(a) => a.into_manifest(), - PackageDataEntryMatchModel::Updating(a) => a.into_installed().into_manifest(), - PackageDataEntryMatchModel::Restoring(a) => a.into_manifest(), - PackageDataEntryMatchModel::Removing(a) => a.into_manifest(), - PackageDataEntryMatchModel::Installed(a) => a.into_manifest(), - PackageDataEntryMatchModel::Error(_) => Model::from(Value::Null), - } - } - pub fn as_manifest(&self) -> &Model { - match self.as_match() { - PackageDataEntryMatchModelRef::Installing(a) => a.as_manifest(), - PackageDataEntryMatchModelRef::Updating(a) => a.as_installed().as_manifest(), - PackageDataEntryMatchModelRef::Restoring(a) => a.as_manifest(), - PackageDataEntryMatchModelRef::Removing(a) => a.as_manifest(), - PackageDataEntryMatchModelRef::Installed(a) => a.as_manifest(), - PackageDataEntryMatchModelRef::Error(_) => (&Value::Null).into(), - } - } - pub fn into_installed(self) -> Option> { - match self.into_match() { - PackageDataEntryMatchModel::Installing(_) => None, - PackageDataEntryMatchModel::Updating(a) => Some(a.into_installed()), - PackageDataEntryMatchModel::Restoring(_) => None, - PackageDataEntryMatchModel::Removing(_) => None, - PackageDataEntryMatchModel::Installed(a) => Some(a.into_installed()), - PackageDataEntryMatchModel::Error(_) => None, - } - } - pub fn as_installed(&self) -> Option<&Model> { - match self.as_match() { - PackageDataEntryMatchModelRef::Installing(_) => None, - PackageDataEntryMatchModelRef::Updating(a) => Some(a.as_installed()), - PackageDataEntryMatchModelRef::Restoring(_) => None, - PackageDataEntryMatchModelRef::Removing(_) => None, - PackageDataEntryMatchModelRef::Installed(a) => Some(a.as_installed()), - PackageDataEntryMatchModelRef::Error(_) => None, - } - } - pub fn as_installed_mut(&mut self) -> Option<&mut Model> { - match self.as_match_mut() { - PackageDataEntryMatchModelMut::Installing(_) => None, - PackageDataEntryMatchModelMut::Updating(a) => Some(a.as_installed_mut()), - PackageDataEntryMatchModelMut::Restoring(_) => None, - PackageDataEntryMatchModelMut::Removing(_) => None, - PackageDataEntryMatchModelMut::Installed(a) => Some(a.as_installed_mut()), - PackageDataEntryMatchModelMut::Error(_) => None, - } - } - pub fn as_install_progress(&self) -> Option<&Model>> { - match self.as_match() { - PackageDataEntryMatchModelRef::Installing(a) => Some(a.as_install_progress()), - PackageDataEntryMatchModelRef::Updating(a) => Some(a.as_install_progress()), - PackageDataEntryMatchModelRef::Restoring(a) => Some(a.as_install_progress()), - PackageDataEntryMatchModelRef::Removing(_) => None, - PackageDataEntryMatchModelRef::Installed(_) => None, - PackageDataEntryMatchModelRef::Error(_) => None, - } - } - pub fn as_install_progress_mut(&mut self) -> Option<&mut Model>> { - match self.as_match_mut() { - PackageDataEntryMatchModelMut::Installing(a) => Some(a.as_install_progress_mut()), - PackageDataEntryMatchModelMut::Updating(a) => Some(a.as_install_progress_mut()), - PackageDataEntryMatchModelMut::Restoring(a) => Some(a.as_install_progress_mut()), - PackageDataEntryMatchModelMut::Removing(_) => None, - PackageDataEntryMatchModelMut::Installed(_) => None, - PackageDataEntryMatchModelMut::Error(_) => None, - } - } -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct InstalledPackageInfo { - pub status: Status, - pub marketplace_url: Option, - #[serde(default)] - #[serde(with = "crate::util::serde::ed25519_pubkey")] - pub developer_key: ed25519_dalek::VerifyingKey, - pub manifest: Manifest, - pub last_backup: Option>, - pub dependency_info: BTreeMap, - pub current_dependents: CurrentDependents, - pub current_dependencies: CurrentDependencies, - pub interface_addresses: InterfaceAddressMap, -} - -#[derive(Debug, Clone, Default, Deserialize, Serialize)] -pub struct CurrentDependents(pub BTreeMap); -impl CurrentDependents { - pub fn map( - mut self, - transform: impl Fn( - BTreeMap, - ) -> BTreeMap, - ) -> Self { - self.0 = transform(self.0); - self - } -} -impl Map for CurrentDependents { - type Key = PackageId; - type Value = CurrentDependencyInfo; -} - -#[derive(Debug, Clone, Default, Deserialize, Serialize)] -pub struct CurrentDependencies(pub BTreeMap); -impl CurrentDependencies { - pub fn map( - mut self, - transform: impl Fn( - BTreeMap, - ) -> BTreeMap, - ) -> Self { - self.0 = transform(self.0); - self - } -} -impl Map for CurrentDependencies { - type Key = PackageId; - type Value = CurrentDependencyInfo; -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct StaticDependencyInfo { - pub title: String, - pub icon: DataUrl<'static>, -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct CurrentDependencyInfo { - #[serde(default)] - pub pointers: BTreeSet, - pub health_checks: BTreeSet, -} - -#[derive(Debug, Deserialize, Serialize)] -pub struct InterfaceAddressMap(pub BTreeMap); -impl Map for InterfaceAddressMap { - type Key = InterfaceId; - type Value = InterfaceAddresses; -} - -#[derive(Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct InterfaceAddresses { - pub tor_address: Option, - pub lan_address: Option, -} diff --git a/core/startos/src/db/package.rs b/core/startos/src/db/package.rs deleted file mode 100644 index fe6f9380..00000000 --- a/core/startos/src/db/package.rs +++ /dev/null @@ -1,22 +0,0 @@ -use models::Version; - -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; - -pub fn get_packages(db: Peeked) -> Result, Error> { - Ok(db - .as_package_data() - .keys()? - .into_iter() - .flat_map(|package_id| { - let version = db - .as_package_data() - .as_idx(&package_id)? - .as_manifest() - .as_version() - .de() - .ok()?; - Some((package_id, version)) - }) - .collect()) -} diff --git a/core/startos/src/db/prelude.rs b/core/startos/src/db/prelude.rs deleted file mode 100644 index 922a4750..00000000 --- a/core/startos/src/db/prelude.rs +++ /dev/null @@ -1,382 +0,0 @@ -use std::collections::BTreeMap; -use std::marker::PhantomData; -use std::panic::UnwindSafe; - -use patch_db::value::InternedString; -pub use patch_db::{HasModel, PatchDb, Value}; -use serde::de::DeserializeOwned; -use serde::Serialize; - -use crate::db::model::DatabaseModel; -use crate::prelude::*; - -pub type Peeked = Model; - -pub fn to_value(value: &T) -> Result -where - T: Serialize, -{ - patch_db::value::to_value(value).with_kind(ErrorKind::Serialization) -} - -pub fn from_value(value: Value) -> Result -where - T: DeserializeOwned, -{ - patch_db::value::from_value(value).with_kind(ErrorKind::Deserialization) -} - -#[async_trait::async_trait] -pub trait PatchDbExt { - async fn peek(&self) -> DatabaseModel; - async fn mutate( - &self, - f: impl FnOnce(&mut DatabaseModel) -> Result + UnwindSafe + Send, - ) -> Result; - async fn map_mutate( - &self, - f: impl FnOnce(DatabaseModel) -> Result + UnwindSafe + Send, - ) -> Result; -} -#[async_trait::async_trait] -impl PatchDbExt for PatchDb { - async fn peek(&self) -> DatabaseModel { - DatabaseModel::from(self.dump().await.value) - } - async fn mutate( - &self, - f: impl FnOnce(&mut DatabaseModel) -> Result + UnwindSafe + Send, - ) -> Result { - Ok(self - .apply_function(|mut v| { - let model = <&mut DatabaseModel>::from(&mut v); - let res = f(model)?; - Ok::<_, Error>((v, res)) - }) - .await? - .1) - } - async fn map_mutate( - &self, - f: impl FnOnce(DatabaseModel) -> Result + UnwindSafe + Send, - ) -> Result { - Ok(DatabaseModel::from( - self.apply_function(|v| f(DatabaseModel::from(v)).map(|a| (a.into(), ()))) - .await? - .0, - )) - } -} - -/// &mut Model <=> &mut Value -#[repr(transparent)] -#[derive(Debug)] -pub struct Model { - value: Value, - phantom: PhantomData, -} -impl Model { - pub fn de(&self) -> Result { - from_value(self.value.clone()) - } -} -impl Model { - pub fn new(value: &T) -> Result { - Ok(Self::from(to_value(value)?)) - } - pub fn ser(&mut self, value: &T) -> Result<(), Error> { - self.value = to_value(value)?; - Ok(()) - } -} - -impl Model { - pub fn replace(&mut self, value: &T) -> Result { - let orig = self.de()?; - self.ser(value)?; - Ok(orig) - } -} -impl Clone for Model { - fn clone(&self) -> Self { - Self { - value: self.value.clone(), - phantom: PhantomData, - } - } -} -impl From for Model { - fn from(value: Value) -> Self { - Self { - value, - phantom: PhantomData, - } - } -} -impl From> for Value { - fn from(value: Model) -> Self { - value.value - } -} -impl<'a, T> From<&'a Value> for &'a Model { - fn from(value: &'a Value) -> Self { - unsafe { std::mem::transmute(value) } - } -} -impl<'a, T> From<&'a Model> for &'a Value { - fn from(value: &'a Model) -> Self { - unsafe { std::mem::transmute(value) } - } -} -impl<'a, T> From<&'a mut Value> for &mut Model { - fn from(value: &'a mut Value) -> Self { - unsafe { std::mem::transmute(value) } - } -} -impl<'a, T> From<&'a mut Model> for &mut Value { - fn from(value: &'a mut Model) -> Self { - unsafe { std::mem::transmute(value) } - } -} -impl patch_db::Model for Model { - type Model = Model; -} - -impl Model> { - pub fn transpose(self) -> Option> { - use patch_db::ModelExt; - if self.value.is_null() { - None - } else { - Some(self.transmute(|a| a)) - } - } - pub fn transpose_ref(&self) -> Option<&Model> { - use patch_db::ModelExt; - if self.value.is_null() { - None - } else { - Some(self.transmute_ref(|a| a)) - } - } - pub fn transpose_mut(&mut self) -> Option<&mut Model> { - use patch_db::ModelExt; - if self.value.is_null() { - None - } else { - Some(self.transmute_mut(|a| a)) - } - } - pub fn from_option(opt: Option>) -> Self { - use patch_db::ModelExt; - match opt { - Some(a) => a.transmute(|a| a), - None => Self::from_value(Value::Null), - } - } -} - -pub trait Map: DeserializeOwned + Serialize { - type Key; - type Value; -} - -impl Map for BTreeMap -where - A: serde::Serialize + serde::de::DeserializeOwned + Ord, - B: serde::Serialize + serde::de::DeserializeOwned, -{ - type Key = A; - type Value = B; -} - -impl Model -where - T::Key: AsRef, - T::Value: Serialize, -{ - pub fn insert(&mut self, key: &T::Key, value: &T::Value) -> Result<(), Error> { - use serde::ser::Error; - let v = patch_db::value::to_value(value)?; - match &mut self.value { - Value::Object(o) => { - o.insert(InternedString::intern(key.as_ref()), v); - Ok(()) - } - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Serialization, - } - .into()), - } - } - pub fn insert_model(&mut self, key: &T::Key, value: Model) -> Result<(), Error> { - use patch_db::ModelExt; - use serde::ser::Error; - let v = value.into_value(); - match &mut self.value { - Value::Object(o) => { - o.insert(InternedString::intern(key.as_ref()), v); - Ok(()) - } - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Serialization, - } - .into()), - } - } -} - -impl Model -where - T::Key: DeserializeOwned + Ord + Clone, -{ - pub fn keys(&self) -> Result, Error> { - use serde::de::Error; - use serde::Deserialize; - match &self.value { - Value::Object(o) => o - .keys() - .cloned() - .map(|k| { - T::Key::deserialize(patch_db::value::de::InternedStringDeserializer::from(k)) - .map_err(|e| { - patch_db::value::Error { - kind: patch_db::value::ErrorKind::Deserialization, - source: e, - } - .into() - }) - }) - .collect(), - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Deserialization, - } - .into()), - } - } - - pub fn into_entries(self) -> Result)>, Error> { - use patch_db::ModelExt; - use serde::de::Error; - use serde::Deserialize; - match self.value { - Value::Object(o) => o - .into_iter() - .map(|(k, v)| { - Ok(( - T::Key::deserialize(patch_db::value::de::InternedStringDeserializer::from( - k, - )) - .with_kind(ErrorKind::Deserialization)?, - Model::from_value(v), - )) - }) - .collect(), - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Deserialization, - } - .into()), - } - } - pub fn as_entries(&self) -> Result)>, Error> { - use patch_db::ModelExt; - use serde::de::Error; - use serde::Deserialize; - match &self.value { - Value::Object(o) => o - .iter() - .map(|(k, v)| { - Ok(( - T::Key::deserialize(patch_db::value::de::InternedStringDeserializer::from( - k.clone(), - )) - .with_kind(ErrorKind::Deserialization)?, - Model::value_as(v), - )) - }) - .collect(), - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Deserialization, - } - .into()), - } - } - pub fn as_entries_mut(&mut self) -> Result)>, Error> { - use patch_db::ModelExt; - use serde::de::Error; - use serde::Deserialize; - match &mut self.value { - Value::Object(o) => o - .iter_mut() - .map(|(k, v)| { - Ok(( - T::Key::deserialize(patch_db::value::de::InternedStringDeserializer::from( - k.clone(), - )) - .with_kind(ErrorKind::Deserialization)?, - Model::value_as_mut(v), - )) - }) - .collect(), - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Deserialization, - } - .into()), - } - } -} -impl Model -where - T::Key: AsRef, -{ - pub fn into_idx(self, key: &T::Key) -> Option> { - use patch_db::ModelExt; - match &self.value { - Value::Object(o) if o.contains_key(key.as_ref()) => Some(self.transmute(|v| { - use patch_db::value::index::Index; - key.as_ref().index_into_owned(v).unwrap() - })), - _ => None, - } - } - pub fn as_idx<'a>(&'a self, key: &T::Key) -> Option<&'a Model> { - use patch_db::ModelExt; - match &self.value { - Value::Object(o) if o.contains_key(key.as_ref()) => Some(self.transmute_ref(|v| { - use patch_db::value::index::Index; - key.as_ref().index_into(v).unwrap() - })), - _ => None, - } - } - pub fn as_idx_mut<'a>(&'a mut self, key: &T::Key) -> Option<&'a mut Model> { - use patch_db::ModelExt; - match &mut self.value { - Value::Object(o) if o.contains_key(key.as_ref()) => Some(self.transmute_mut(|v| { - use patch_db::value::index::Index; - key.as_ref().index_or_insert(v) - })), - _ => None, - } - } - pub fn remove(&mut self, key: &T::Key) -> Result>, Error> { - use serde::ser::Error; - match &mut self.value { - Value::Object(o) => { - let v = o.remove(key.as_ref()); - Ok(v.map(patch_db::ModelExt::from_value)) - } - v => Err(patch_db::value::Error { - source: patch_db::value::ErrorSource::custom(format!("expected object found {v}")), - kind: patch_db::value::ErrorKind::Serialization, - } - .into()), - } - } -} diff --git a/core/startos/src/dependencies.rs b/core/startos/src/dependencies.rs deleted file mode 100644 index dfddecd9..00000000 --- a/core/startos/src/dependencies.rs +++ /dev/null @@ -1,363 +0,0 @@ -use std::collections::BTreeMap; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use emver::VersionRange; -use models::OptionExt; -use rand::SeedableRng; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use crate::config::action::ConfigRes; -use crate::config::spec::PackagePointerSpec; -use crate::config::{not_found, Config, ConfigSpec, ConfigureContext}; -use crate::context::RpcContext; -use crate::db::model::{CurrentDependencies, Database}; -use crate::prelude::*; -use crate::procedure::{NoOutput, PackageProcedure, ProcedureName}; -use crate::s9pk::manifest::{Manifest, PackageId}; -use crate::status::DependencyConfigErrors; -use crate::util::serde::display_serializable; -use crate::util::{display_none, Version}; -use crate::volume::Volumes; -use crate::Error; - -#[command(subcommands(configure))] -pub fn dependency() -> Result<(), Error> { - Ok(()) -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize, HasModel)] -#[model = "Model"] -pub struct Dependencies(pub BTreeMap); -impl Map for Dependencies { - type Key = PackageId; - type Value = DepInfo; -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -#[serde(tag = "type")] -pub enum DependencyRequirement { - OptIn { how: String }, - OptOut { how: String }, - Required, -} -impl DependencyRequirement { - pub fn required(&self) -> bool { - matches!(self, &DependencyRequirement::Required) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct DepInfo { - pub version: VersionRange, - pub requirement: DependencyRequirement, - pub description: Option, - #[serde(default)] - pub config: Option, -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct DependencyConfig { - check: PackageProcedure, - auto_configure: PackageProcedure, -} -impl DependencyConfig { - pub async fn check( - &self, - ctx: &RpcContext, - dependent_id: &PackageId, - dependent_version: &Version, - dependent_volumes: &Volumes, - dependency_id: &PackageId, - dependency_config: &Config, - ) -> Result, Error> { - Ok(self - .check - .sandboxed( - ctx, - dependent_id, - dependent_version, - dependent_volumes, - Some(dependency_config), - None, - ProcedureName::Check(dependency_id.clone()), - ) - .await? - .map_err(|(_, e)| e)) - } - pub async fn auto_configure( - &self, - ctx: &RpcContext, - dependent_id: &PackageId, - dependent_version: &Version, - dependent_volumes: &Volumes, - old: &Config, - ) -> Result { - self.auto_configure - .sandboxed( - ctx, - dependent_id, - dependent_version, - dependent_volumes, - Some(old), - None, - ProcedureName::AutoConfig(dependent_id.clone()), - ) - .await? - .map_err(|e| Error::new(eyre!("{}", e.1), crate::ErrorKind::AutoConfigure)) - } -} - -#[command( - subcommands(self(configure_impl(async)), configure_dry), - display(display_none) -)] -pub async fn configure( - #[arg(rename = "dependent-id")] dependent_id: PackageId, - #[arg(rename = "dependency-id")] dependency_id: PackageId, -) -> Result<(PackageId, PackageId), Error> { - Ok((dependent_id, dependency_id)) -} - -pub async fn configure_impl( - ctx: RpcContext, - (pkg_id, dep_id): (PackageId, PackageId), -) -> Result<(), Error> { - let breakages = BTreeMap::new(); - let overrides = Default::default(); - let ConfigDryRes { - old_config: _, - new_config, - spec: _, - } = configure_logic(ctx.clone(), (pkg_id, dep_id.clone())).await?; - - let configure_context = ConfigureContext { - breakages, - timeout: Some(Duration::from_secs(3).into()), - config: Some(new_config), - dry_run: false, - overrides, - }; - crate::config::configure(&ctx, &dep_id, configure_context).await?; - Ok(()) -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct ConfigDryRes { - pub old_config: Config, - pub new_config: Config, - pub spec: ConfigSpec, -} - -#[command(rename = "dry", display(display_serializable))] -#[instrument(skip_all)] -pub async fn configure_dry( - #[context] ctx: RpcContext, - #[parent_data] (pkg_id, dependency_id): (PackageId, PackageId), -) -> Result { - configure_logic(ctx, (pkg_id, dependency_id)).await -} - -pub async fn configure_logic( - ctx: RpcContext, - (pkg_id, dependency_id): (PackageId, PackageId), -) -> Result { - let db = ctx.db.peek().await; - let pkg = db - .as_package_data() - .as_idx(&pkg_id) - .or_not_found(&pkg_id)? - .as_installed() - .or_not_found(&pkg_id)?; - let pkg_version = pkg.as_manifest().as_version().de()?; - let pkg_volumes = pkg.as_manifest().as_volumes().de()?; - let dependency = db - .as_package_data() - .as_idx(&dependency_id) - .or_not_found(&dependency_id)? - .as_installed() - .or_not_found(&dependency_id)?; - let dependency_config_action = dependency - .as_manifest() - .as_config() - .de()? - .ok_or_else(|| not_found!("Manifest Config"))?; - let dependency_version = dependency.as_manifest().as_version().de()?; - let dependency_volumes = dependency.as_manifest().as_volumes().de()?; - let dependency = pkg - .as_manifest() - .as_dependencies() - .as_idx(&dependency_id) - .or_not_found(&dependency_id)?; - - let ConfigRes { - config: maybe_config, - spec, - } = dependency_config_action - .get( - &ctx, - &dependency_id, - &dependency_version, - &dependency_volumes, - ) - .await?; - - let old_config = if let Some(config) = maybe_config { - config - } else { - spec.gen( - &mut rand::rngs::StdRng::from_entropy(), - &Some(Duration::new(10, 0)), - )? - }; - - let new_config = dependency - .as_config() - .de()? - .ok_or_else(|| not_found!("Config"))? - .auto_configure - .sandboxed( - &ctx, - &pkg_id, - &pkg_version, - &pkg_volumes, - Some(&old_config), - None, - ProcedureName::AutoConfig(dependency_id.clone()), - ) - .await? - .map_err(|e| Error::new(eyre!("{}", e.1), crate::ErrorKind::AutoConfigure))?; - - Ok(ConfigDryRes { - old_config, - new_config, - spec, - }) -} - -#[instrument(skip_all)] -pub fn add_dependent_to_current_dependents_lists( - db: &mut Model, - dependent_id: &PackageId, - current_dependencies: &CurrentDependencies, -) -> Result<(), Error> { - for (dependency, dep_info) in ¤t_dependencies.0 { - if let Some(dependency_dependents) = db - .as_package_data_mut() - .as_idx_mut(dependency) - .and_then(|pde| pde.as_installed_mut()) - .map(|i| i.as_current_dependents_mut()) - { - dependency_dependents.insert(dependent_id, dep_info)?; - } - } - Ok(()) -} - -pub fn set_dependents_with_live_pointers_to_needs_config( - db: &mut Peeked, - id: &PackageId, -) -> Result, Error> { - let mut res = Vec::new(); - for (dep, info) in db - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .as_installed() - .or_not_found(id)? - .as_current_dependents() - .de()? - .0 - { - if info.pointers.iter().any(|ptr| match ptr { - // dependency id matches the package being uninstalled - PackagePointerSpec::TorAddress(ptr) => &ptr.package_id == id && &dep != id, - PackagePointerSpec::LanAddress(ptr) => &ptr.package_id == id && &dep != id, - // we never need to retarget these - PackagePointerSpec::TorKey(_) => false, - PackagePointerSpec::Config(_) => false, - }) { - let installed = db - .as_package_data_mut() - .as_idx_mut(&dep) - .or_not_found(&dep)? - .as_installed_mut() - .or_not_found(&dep)?; - let version = installed.as_manifest().as_version().de()?; - let configured = installed.as_status_mut().as_configured_mut(); - if configured.de()? { - configured.ser(&false)?; - res.push((dep, version)); - } - } - } - Ok(res) -} - -#[instrument(skip_all)] -pub async fn compute_dependency_config_errs( - ctx: &RpcContext, - db: &Peeked, - manifest: &Manifest, - current_dependencies: &CurrentDependencies, - dependency_config: &BTreeMap, -) -> Result { - let mut dependency_config_errs = BTreeMap::new(); - for (dependency, _dep_info) in current_dependencies - .0 - .iter() - .filter(|(dep_id, _)| dep_id != &&manifest.id) - { - // check if config passes dependency check - if let Some(cfg) = &manifest - .dependencies - .0 - .get(dependency) - .or_not_found(dependency)? - .config - { - if let Err(error) = cfg - .check( - ctx, - &manifest.id, - &manifest.version, - &manifest.volumes, - dependency, - &if let Some(config) = dependency_config.get(dependency) { - config.clone() - } else if let Some(manifest) = db - .as_package_data() - .as_idx(dependency) - .and_then(|pde| pde.as_installed()) - .map(|i| i.as_manifest().de()) - .transpose()? - { - if let Some(config) = &manifest.config { - config - .get(ctx, &manifest.id, &manifest.version, &manifest.volumes) - .await? - .config - .unwrap_or_default() - } else { - Config::default() - } - } else { - Config::default() - }, - ) - .await? - { - dependency_config_errs.insert(dependency.clone(), error); - } - } - } - Ok(DependencyConfigErrors(dependency_config_errs)) -} diff --git a/core/startos/src/developer/mod.rs b/core/startos/src/developer/mod.rs deleted file mode 100644 index 8722a4a1..00000000 --- a/core/startos/src/developer/mod.rs +++ /dev/null @@ -1,55 +0,0 @@ -use std::fs::File; -use std::io::Write; -use std::path::Path; - -use ed25519::pkcs8::EncodePrivateKey; -use ed25519::PublicKeyBytes; -use ed25519_dalek::{SigningKey, VerifyingKey}; -use rpc_toolkit::command; -use tracing::instrument; - -use crate::context::SdkContext; -use crate::util::display_none; -use crate::{Error, ResultExt}; - -#[command(cli_only, blocking, display(display_none))] -#[instrument(skip_all)] -pub fn init(#[context] ctx: SdkContext) -> Result<(), Error> { - if !ctx.developer_key_path.exists() { - let parent = ctx.developer_key_path.parent().unwrap_or(Path::new("/")); - if !parent.exists() { - std::fs::create_dir_all(parent) - .with_ctx(|_| (crate::ErrorKind::Filesystem, parent.display().to_string()))?; - } - tracing::info!("Generating new developer key..."); - let secret = SigningKey::generate(&mut rand::thread_rng()); - tracing::info!("Writing key to {}", ctx.developer_key_path.display()); - let keypair_bytes = ed25519::KeypairBytes { - secret_key: secret.to_bytes(), - public_key: Some(PublicKeyBytes(VerifyingKey::from(&secret).to_bytes())), - }; - let mut dev_key_file = File::create(&ctx.developer_key_path)?; - dev_key_file.write_all( - keypair_bytes - .to_pkcs8_pem(base64ct::LineEnding::default()) - .with_kind(crate::ErrorKind::Pem)? - .as_bytes(), - )?; - dev_key_file.sync_all()?; - println!( - "New developer key generated at {}", - ctx.developer_key_path.display() - ); - } else { - println!( - "Developer key already exists at {}", - ctx.developer_key_path.display() - ); - } - Ok(()) -} - -#[command(subcommands(crate::s9pk::verify, crate::config::verify_spec))] -pub fn verify() -> Result<(), Error> { - Ok(()) -} diff --git a/core/startos/src/diagnostic.rs b/core/startos/src/diagnostic.rs deleted file mode 100644 index aad95a5e..00000000 --- a/core/startos/src/diagnostic.rs +++ /dev/null @@ -1,72 +0,0 @@ -use std::path::Path; -use std::sync::Arc; - -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; - -use crate::context::DiagnosticContext; -use crate::disk::repair; -use crate::init::SYSTEM_REBUILD_PATH; -use crate::logs::{fetch_logs, LogResponse, LogSource}; -use crate::shutdown::Shutdown; -use crate::util::display_none; -use crate::Error; - -#[command(subcommands(error, logs, exit, restart, forget_disk, disk, rebuild))] -pub fn diagnostic() -> Result<(), Error> { - Ok(()) -} - -#[command] -pub fn error(#[context] ctx: DiagnosticContext) -> Result, Error> { - Ok(ctx.error.clone()) -} - -#[command(rpc_only)] -pub async fn logs( - #[arg] limit: Option, - #[arg] cursor: Option, - #[arg] before: bool, -) -> Result { - Ok(fetch_logs(LogSource::System, limit, cursor, before).await?) -} - -#[command(display(display_none))] -pub fn exit(#[context] ctx: DiagnosticContext) -> Result<(), Error> { - ctx.shutdown.send(None).expect("receiver dropped"); - Ok(()) -} - -#[command(display(display_none))] -pub fn restart(#[context] ctx: DiagnosticContext) -> Result<(), Error> { - ctx.shutdown - .send(Some(Shutdown { - export_args: ctx - .disk_guid - .clone() - .map(|guid| (guid, ctx.datadir.clone())), - restart: true, - })) - .expect("receiver dropped"); - Ok(()) -} - -#[command(display(display_none))] -pub async fn rebuild(#[context] ctx: DiagnosticContext) -> Result<(), Error> { - tokio::fs::write(SYSTEM_REBUILD_PATH, b"").await?; - restart(ctx) -} - -#[command(subcommands(forget_disk, repair))] -pub fn disk() -> Result<(), Error> { - Ok(()) -} - -#[command(rename = "forget", display(display_none))] -pub async fn forget_disk() -> Result<(), Error> { - let disk_guid = Path::new("/media/embassy/config/disk.guid"); - if tokio::fs::metadata(disk_guid).await.is_ok() { - tokio::fs::remove_file(disk_guid).await?; - } - Ok(()) -} diff --git a/core/startos/src/disk/fsck/btrfs.rs b/core/startos/src/disk/fsck/btrfs.rs deleted file mode 100644 index 2a198cd8..00000000 --- a/core/startos/src/disk/fsck/btrfs.rs +++ /dev/null @@ -1,33 +0,0 @@ -use std::io::Cursor; -use std::path::Path; - -use tokio::process::Command; -use tracing::instrument; - -use crate::disk::fsck::RequiresReboot; -use crate::util::Invoke; -use crate::Error; - -#[instrument(skip_all)] -pub async fn btrfs_check_readonly(logicalname: impl AsRef) -> Result { - Command::new("btrfs") - .arg("check") - .arg("--readonly") - .arg(logicalname.as_ref()) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - - Ok(RequiresReboot(false)) -} - -pub async fn btrfs_check_repair(logicalname: impl AsRef) -> Result { - Command::new("btrfs") - .arg("check") - .arg("--repair") - .arg(logicalname.as_ref()) - .input(Some(&mut Cursor::new(b"y\n"))) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - - Ok(RequiresReboot(false)) -} diff --git a/core/startos/src/disk/fsck/ext4.rs b/core/startos/src/disk/fsck/ext4.rs deleted file mode 100644 index 7bcbbc8b..00000000 --- a/core/startos/src/disk/fsck/ext4.rs +++ /dev/null @@ -1,95 +0,0 @@ -use std::ffi::OsStr; -use std::path::Path; - -use color_eyre::eyre::eyre; -use futures::future::BoxFuture; -use futures::FutureExt; -use tokio::process::Command; -use tracing::instrument; - -use crate::disk::fsck::RequiresReboot; -use crate::Error; - -#[instrument(skip_all)] -pub async fn e2fsck_preen( - logicalname: impl AsRef + std::fmt::Debug, -) -> Result { - e2fsck_runner(Command::new("e2fsck").arg("-p"), logicalname).await -} - -fn backup_existing_undo_file<'a>(path: &'a Path) -> BoxFuture<'a, Result<(), Error>> { - async move { - if tokio::fs::metadata(path).await.is_ok() { - let bak = path.with_extension(format!( - "{}.bak", - path.extension() - .and_then(|s| s.to_str()) - .unwrap_or_default() - )); - backup_existing_undo_file(&bak).await?; - tokio::fs::rename(path, &bak).await?; - } - Ok(()) - } - .boxed() -} - -#[instrument(skip_all)] -pub async fn e2fsck_aggressive( - logicalname: impl AsRef + std::fmt::Debug, -) -> Result { - let undo_path = Path::new("/media/embassy/config") - .join( - logicalname - .as_ref() - .file_name() - .unwrap_or(OsStr::new("unknown")), - ) - .with_extension("e2undo"); - backup_existing_undo_file(&undo_path).await?; - e2fsck_runner( - Command::new("e2fsck").arg("-y").arg("-z").arg(undo_path), - logicalname, - ) - .await -} - -async fn e2fsck_runner( - e2fsck_cmd: &mut Command, - logicalname: impl AsRef + std::fmt::Debug, -) -> Result { - let e2fsck_out = e2fsck_cmd.arg(logicalname.as_ref()).output().await?; - let e2fsck_stderr = String::from_utf8(e2fsck_out.stderr)?; - let code = e2fsck_out.status.code().ok_or_else(|| { - Error::new( - eyre!("e2fsck: process terminated by signal"), - crate::ErrorKind::DiskManagement, - ) - })?; - if code & 4 != 0 { - tracing::error!( - "some filesystem errors NOT corrected on {}:\n{}", - logicalname.as_ref().display(), - e2fsck_stderr, - ); - } else if code & 1 != 0 { - tracing::warn!( - "filesystem errors corrected on {}:\n{}", - logicalname.as_ref().display(), - e2fsck_stderr, - ); - } - if code < 8 { - if code & 2 != 0 { - tracing::warn!("reboot required"); - Ok(RequiresReboot(true)) - } else { - Ok(RequiresReboot(false)) - } - } else { - Err(Error::new( - eyre!("e2fsck: {}", e2fsck_stderr), - crate::ErrorKind::DiskManagement, - )) - } -} diff --git a/core/startos/src/disk/fsck/mod.rs b/core/startos/src/disk/fsck/mod.rs deleted file mode 100644 index 6758ddd5..00000000 --- a/core/startos/src/disk/fsck/mod.rs +++ /dev/null @@ -1,70 +0,0 @@ -use std::path::Path; - -use color_eyre::eyre::eyre; -use tokio::process::Command; - -use crate::disk::fsck::btrfs::{btrfs_check_readonly, btrfs_check_repair}; -use crate::disk::fsck::ext4::{e2fsck_aggressive, e2fsck_preen}; -use crate::util::Invoke; -use crate::Error; - -pub mod btrfs; -pub mod ext4; - -#[derive(Debug, Clone, Copy, serde::Serialize, serde::Deserialize)] -#[must_use] -pub struct RequiresReboot(pub bool); -impl std::ops::BitOrAssign for RequiresReboot { - fn bitor_assign(&mut self, rhs: Self) { - self.0 |= rhs.0 - } -} - -#[derive(Debug, Clone, Copy)] -pub enum RepairStrategy { - Preen, - Aggressive, -} -impl RepairStrategy { - pub async fn fsck( - &self, - logicalname: impl AsRef + std::fmt::Debug, - ) -> Result { - match &*String::from_utf8( - Command::new("grub-probe") - .arg("-d") - .arg(logicalname.as_ref()) - .invoke(crate::ErrorKind::DiskManagement) - .await?, - )? - .trim() - { - "ext2" => self.e2fsck(logicalname).await, - "btrfs" => self.btrfs_check(logicalname).await, - fs => { - return Err(Error::new( - eyre!("Unknown filesystem {fs}"), - crate::ErrorKind::DiskManagement, - )) - } - } - } - pub async fn e2fsck( - &self, - logicalname: impl AsRef + std::fmt::Debug, - ) -> Result { - match self { - RepairStrategy::Preen => e2fsck_preen(logicalname).await, - RepairStrategy::Aggressive => e2fsck_aggressive(logicalname).await, - } - } - pub async fn btrfs_check( - &self, - logicalname: impl AsRef + std::fmt::Debug, - ) -> Result { - match self { - RepairStrategy::Preen => btrfs_check_readonly(logicalname).await, - RepairStrategy::Aggressive => btrfs_check_repair(logicalname).await, - } - } -} diff --git a/core/startos/src/disk/main.rs b/core/startos/src/disk/main.rs deleted file mode 100644 index 74f6db73..00000000 --- a/core/startos/src/disk/main.rs +++ /dev/null @@ -1,337 +0,0 @@ -use std::collections::BTreeMap; -use std::path::{Path, PathBuf}; - -use color_eyre::eyre::eyre; -use tokio::process::Command; -use tracing::instrument; - -use super::fsck::{RepairStrategy, RequiresReboot}; -use super::util::pvscan; -use crate::disk::mount::filesystem::block_dev::mount; -use crate::disk::mount::filesystem::ReadWrite; -use crate::disk::mount::util::unmount; -use crate::util::Invoke; -use crate::{Error, ErrorKind, ResultExt}; - -pub const PASSWORD_PATH: &'static str = "/run/embassy/password"; -pub const DEFAULT_PASSWORD: &'static str = "password"; -pub const MAIN_FS_SIZE: FsSize = FsSize::Gigabytes(8); - -#[instrument(skip_all)] -pub async fn create( - disks: &I, - pvscan: &BTreeMap>, - datadir: impl AsRef, - password: Option<&str>, -) -> Result -where - for<'a> &'a I: IntoIterator, - P: AsRef, -{ - let guid = create_pool(disks, pvscan, password.is_some()).await?; - create_all_fs(&guid, &datadir, password).await?; - export(&guid, datadir).await?; - Ok(guid) -} - -#[instrument(skip_all)] -pub async fn create_pool( - disks: &I, - pvscan: &BTreeMap>, - encrypted: bool, -) -> Result -where - for<'a> &'a I: IntoIterator, - P: AsRef, -{ - Command::new("dmsetup") - .arg("remove_all") // TODO: find a higher finesse way to do this for portability reasons - .invoke(crate::ErrorKind::DiskManagement) - .await?; - for disk in disks { - if pvscan.contains_key(disk.as_ref()) { - Command::new("pvremove") - .arg("-yff") - .arg(disk.as_ref()) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - } - tokio::fs::write(disk.as_ref(), &[0; 2048]).await?; // wipe partition table - Command::new("pvcreate") - .arg("-yff") - .arg(disk.as_ref()) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - } - let mut guid = format!( - "EMBASSY_{}", - base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &rand::random::<[u8; 32]>(), - ) - ); - if !encrypted { - guid += "_UNENC"; - } - let mut cmd = Command::new("vgcreate"); - cmd.arg("-y").arg(&guid); - for disk in disks { - cmd.arg(disk.as_ref()); - } - cmd.invoke(crate::ErrorKind::DiskManagement).await?; - Ok(guid) -} - -#[derive(Debug, Clone, Copy)] -pub enum FsSize { - Gigabytes(usize), - FreePercentage(usize), -} - -#[instrument(skip_all)] -pub async fn create_fs>( - guid: &str, - datadir: P, - name: &str, - size: FsSize, - password: Option<&str>, -) -> Result<(), Error> { - let mut cmd = Command::new("lvcreate"); - match size { - FsSize::Gigabytes(a) => cmd.arg("-L").arg(format!("{}G", a)), - FsSize::FreePercentage(a) => cmd.arg("-l").arg(format!("{}%FREE", a)), - }; - cmd.arg("-y") - .arg("-n") - .arg(name) - .arg(guid) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - let mut blockdev_path = Path::new("/dev").join(guid).join(name); - if let Some(password) = password { - if let Some(parent) = Path::new(PASSWORD_PATH).parent() { - tokio::fs::create_dir_all(parent).await?; - } - tokio::fs::write(PASSWORD_PATH, password) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, PASSWORD_PATH))?; - Command::new("cryptsetup") - .arg("-q") - .arg("luksFormat") - .arg(format!("--key-file={}", PASSWORD_PATH)) - .arg(format!("--keyfile-size={}", password.len())) - .arg(&blockdev_path) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Command::new("cryptsetup") - .arg("-q") - .arg("luksOpen") - .arg("--allow-discards") - .arg(format!("--key-file={}", PASSWORD_PATH)) - .arg(format!("--keyfile-size={}", password.len())) - .arg(&blockdev_path) - .arg(format!("{}_{}", guid, name)) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - tokio::fs::remove_file(PASSWORD_PATH) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, PASSWORD_PATH))?; - blockdev_path = Path::new("/dev/mapper").join(format!("{}_{}", guid, name)); - } - Command::new("mkfs.btrfs") - .arg(&blockdev_path) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - mount(&blockdev_path, datadir.as_ref().join(name), ReadWrite).await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn create_all_fs>( - guid: &str, - datadir: P, - password: Option<&str>, -) -> Result<(), Error> { - create_fs(guid, &datadir, "main", MAIN_FS_SIZE, password).await?; - create_fs( - guid, - &datadir, - "package-data", - FsSize::FreePercentage(100), - password, - ) - .await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn unmount_fs>(guid: &str, datadir: P, name: &str) -> Result<(), Error> { - unmount(datadir.as_ref().join(name)).await?; - if !guid.ends_with("_UNENC") { - Command::new("cryptsetup") - .arg("-q") - .arg("luksClose") - .arg(format!("{}_{}", guid, name)) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - } - - Ok(()) -} - -#[instrument(skip_all)] -pub async fn unmount_all_fs>(guid: &str, datadir: P) -> Result<(), Error> { - unmount_fs(guid, &datadir, "main").await?; - unmount_fs(guid, &datadir, "package-data").await?; - Command::new("dmsetup") - .arg("remove_all") // TODO: find a higher finesse way to do this for portability reasons - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn export>(guid: &str, datadir: P) -> Result<(), Error> { - Command::new("sync").invoke(ErrorKind::Filesystem).await?; - unmount_all_fs(guid, datadir).await?; - Command::new("vgchange") - .arg("-an") - .arg(guid) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Command::new("vgexport") - .arg(guid) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn import>( - guid: &str, - datadir: P, - repair: RepairStrategy, - password: Option<&str>, -) -> Result { - let scan = pvscan().await?; - if scan - .values() - .filter_map(|a| a.as_ref()) - .filter(|a| a.starts_with("EMBASSY_")) - .next() - .is_none() - { - return Err(Error::new( - eyre!("StartOS disk not found."), - crate::ErrorKind::DiskNotAvailable, - )); - } - if !scan - .values() - .filter_map(|a| a.as_ref()) - .any(|id| id == guid) - { - return Err(Error::new( - eyre!("A StartOS disk was found, but it is not the correct disk for this device."), - crate::ErrorKind::IncorrectDisk, - )); - } - Command::new("dmsetup") - .arg("remove_all") // TODO: find a higher finesse way to do this for portability reasons - .invoke(crate::ErrorKind::DiskManagement) - .await?; - match Command::new("vgimport") - .arg(guid) - .invoke(crate::ErrorKind::DiskManagement) - .await - { - Ok(_) => Ok(()), - Err(e) - if format!("{}", e.source) - .lines() - .any(|l| l.trim() == format!("Volume group \"{}\" is not exported", guid)) => - { - Ok(()) - } - Err(e) => Err(e), - }?; - Command::new("vgchange") - .arg("-ay") - .arg(guid) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - mount_all_fs(guid, datadir, repair, password).await -} - -#[instrument(skip_all)] -pub async fn mount_fs>( - guid: &str, - datadir: P, - name: &str, - repair: RepairStrategy, - password: Option<&str>, -) -> Result { - let orig_path = Path::new("/dev").join(guid).join(name); - let mut blockdev_path = orig_path.clone(); - let full_name = format!("{}_{}", guid, name); - if !guid.ends_with("_UNENC") { - let password = password.unwrap_or(DEFAULT_PASSWORD); - if let Some(parent) = Path::new(PASSWORD_PATH).parent() { - tokio::fs::create_dir_all(parent).await?; - } - tokio::fs::write(PASSWORD_PATH, password) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, PASSWORD_PATH))?; - Command::new("cryptsetup") - .arg("-q") - .arg("luksOpen") - .arg(format!("--key-file={}", PASSWORD_PATH)) - .arg(format!("--keyfile-size={}", password.len())) - .arg(&blockdev_path) - .arg(&full_name) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - tokio::fs::remove_file(PASSWORD_PATH) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, PASSWORD_PATH))?; - blockdev_path = Path::new("/dev/mapper").join(&full_name); - } - let reboot = repair.fsck(&blockdev_path).await?; - - if !guid.ends_with("_UNENC") { - // Backup LUKS header if e2fsck succeeded - let luks_folder = Path::new("/media/embassy/config/luks"); - tokio::fs::create_dir_all(luks_folder).await?; - let tmp_luks_bak = luks_folder.join(format!(".{full_name}.luks.bak.tmp")); - if tokio::fs::metadata(&tmp_luks_bak).await.is_ok() { - tokio::fs::remove_file(&tmp_luks_bak).await?; - } - let luks_bak = luks_folder.join(format!("{full_name}.luks.bak")); - Command::new("cryptsetup") - .arg("-q") - .arg("luksHeaderBackup") - .arg("--header-backup-file") - .arg(&tmp_luks_bak) - .arg(&orig_path) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - tokio::fs::rename(&tmp_luks_bak, &luks_bak).await?; - } - - mount(&blockdev_path, datadir.as_ref().join(name), ReadWrite).await?; - - Ok(reboot) -} - -#[instrument(skip_all)] -pub async fn mount_all_fs>( - guid: &str, - datadir: P, - repair: RepairStrategy, - password: Option<&str>, -) -> Result { - let mut reboot = RequiresReboot(false); - reboot |= mount_fs(guid, &datadir, "main", repair, password).await?; - reboot |= mount_fs(guid, &datadir, "package-data", repair, password).await?; - Ok(reboot) -} diff --git a/core/startos/src/disk/mod.rs b/core/startos/src/disk/mod.rs deleted file mode 100644 index 485d2570..00000000 --- a/core/startos/src/disk/mod.rs +++ /dev/null @@ -1,118 +0,0 @@ -use std::path::{Path, PathBuf}; - -use clap::ArgMatches; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; - -use crate::context::RpcContext; -use crate::disk::util::DiskInfo; -use crate::util::display_none; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::Error; - -pub mod fsck; -pub mod main; -pub mod mount; -pub mod util; - -pub const BOOT_RW_PATH: &str = "/media/boot-rw"; -pub const REPAIR_DISK_PATH: &str = "/media/embassy/config/repair-disk"; - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct OsPartitionInfo { - pub efi: Option, - pub bios: Option, - pub boot: PathBuf, - pub root: PathBuf, -} -impl OsPartitionInfo { - pub fn contains(&self, logicalname: impl AsRef) -> bool { - self.efi - .as_ref() - .map(|p| p == logicalname.as_ref()) - .unwrap_or(false) - || self - .bios - .as_ref() - .map(|p| p == logicalname.as_ref()) - .unwrap_or(false) - || &*self.boot == logicalname.as_ref() - || &*self.root == logicalname.as_ref() - } -} - -#[command(subcommands(list, repair))] -pub fn disk() -> Result<(), Error> { - Ok(()) -} - -fn display_disk_info(info: Vec, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(info, matches); - } - - let mut table = Table::new(); - table.add_row(row![bc => - "LOGICALNAME", - "LABEL", - "CAPACITY", - "USED", - "EMBASSY OS VERSION" - ]); - for disk in info { - let row = row![ - disk.logicalname.display(), - "N/A", - &format!("{:.2} GiB", disk.capacity as f64 / 1024.0 / 1024.0 / 1024.0), - "N/A", - "N/A", - ]; - table.add_row(row); - for part in disk.partitions { - let row = row![ - part.logicalname.display(), - if let Some(label) = part.label.as_ref() { - label - } else { - "N/A" - }, - part.capacity, - if let Some(used) = part - .used - .map(|u| format!("{:.2} GiB", u as f64 / 1024.0 / 1024.0 / 1024.0)) - .as_ref() - { - used - } else { - "N/A" - }, - if let Some(eos) = part.embassy_os.as_ref() { - eos.version.as_str() - } else { - "N/A" - }, - ]; - table.add_row(row); - } - } - table.print_tty(false).unwrap(); -} - -#[command(display(display_disk_info))] -pub async fn list( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg] - format: Option, -) -> Result, Error> { - crate::disk::util::list(&ctx.os_partitions).await -} - -#[command(display(display_none))] -pub async fn repair() -> Result<(), Error> { - tokio::fs::write(REPAIR_DISK_PATH, b"").await?; - Ok(()) -} diff --git a/core/startos/src/disk/mount/backup.rs b/core/startos/src/disk/mount/backup.rs deleted file mode 100644 index a1905624..00000000 --- a/core/startos/src/disk/mount/backup.rs +++ /dev/null @@ -1,262 +0,0 @@ -use std::path::{Path, PathBuf}; - -use color_eyre::eyre::eyre; -use helpers::AtomicFile; -use tokio::io::AsyncWriteExt; -use tracing::instrument; - -use super::filesystem::ecryptfs::EcryptFS; -use super::guard::{GenericMountGuard, TmpMountGuard}; -use super::util::{bind, unmount}; -use crate::auth::check_password; -use crate::backup::target::BackupInfo; -use crate::disk::mount::filesystem::ReadWrite; -use crate::disk::util::EmbassyOsRecoveryInfo; -use crate::middleware::encrypt::{decrypt_slice, encrypt_slice}; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::IoFormat; -use crate::util::FileLock; -use crate::volume::BACKUP_DIR; -use crate::{Error, ErrorKind, ResultExt}; - -pub struct BackupMountGuard { - backup_disk_mount_guard: Option, - encrypted_guard: Option, - enc_key: String, - pub unencrypted_metadata: EmbassyOsRecoveryInfo, - pub metadata: BackupInfo, -} -impl BackupMountGuard { - fn backup_disk_path(&self) -> &Path { - if let Some(guard) = &self.backup_disk_mount_guard { - guard.as_ref() - } else { - unreachable!() - } - } - - #[instrument(skip_all)] - pub async fn mount(backup_disk_mount_guard: G, password: &str) -> Result { - let backup_disk_path = backup_disk_mount_guard.as_ref(); - let unencrypted_metadata_path = - backup_disk_path.join("EmbassyBackups/unencrypted-metadata.cbor"); - let mut unencrypted_metadata: EmbassyOsRecoveryInfo = - if tokio::fs::metadata(&unencrypted_metadata_path) - .await - .is_ok() - { - IoFormat::Cbor.from_slice( - &tokio::fs::read(&unencrypted_metadata_path) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - unencrypted_metadata_path.display().to_string(), - ) - })?, - )? - } else { - Default::default() - }; - let enc_key = if let (Some(hash), Some(wrapped_key)) = ( - unencrypted_metadata.password_hash.as_ref(), - unencrypted_metadata.wrapped_key.as_ref(), - ) { - let wrapped_key = - base32::decode(base32::Alphabet::RFC4648 { padding: true }, wrapped_key) - .ok_or_else(|| { - Error::new( - eyre!("failed to decode wrapped key"), - crate::ErrorKind::Backup, - ) - })?; - check_password(hash, password)?; - String::from_utf8(decrypt_slice(wrapped_key, password))? - } else { - base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &rand::random::<[u8; 32]>()[..], - ) - }; - - if unencrypted_metadata.password_hash.is_none() { - unencrypted_metadata.password_hash = Some( - argon2::hash_encoded( - password.as_bytes(), - &rand::random::<[u8; 16]>()[..], - &argon2::Config::rfc9106_low_mem(), - ) - .with_kind(crate::ErrorKind::PasswordHashGeneration)?, - ); - } - if unencrypted_metadata.wrapped_key.is_none() { - unencrypted_metadata.wrapped_key = Some(base32::encode( - base32::Alphabet::RFC4648 { padding: true }, - &encrypt_slice(&enc_key, password), - )); - } - - let crypt_path = backup_disk_path.join("EmbassyBackups/crypt"); - if tokio::fs::metadata(&crypt_path).await.is_err() { - tokio::fs::create_dir_all(&crypt_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - crypt_path.display().to_string(), - ) - })?; - } - let encrypted_guard = - TmpMountGuard::mount(&EcryptFS::new(&crypt_path, &enc_key), ReadWrite).await?; - - let metadata_path = encrypted_guard.as_ref().join("metadata.cbor"); - let metadata: BackupInfo = if tokio::fs::metadata(&metadata_path).await.is_ok() { - IoFormat::Cbor.from_slice(&tokio::fs::read(&metadata_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - metadata_path.display().to_string(), - ) - })?)? - } else { - Default::default() - }; - - Ok(Self { - backup_disk_mount_guard: Some(backup_disk_mount_guard), - encrypted_guard: Some(encrypted_guard), - enc_key, - unencrypted_metadata, - metadata, - }) - } - - pub fn change_password(&mut self, new_password: &str) -> Result<(), Error> { - self.unencrypted_metadata.password_hash = Some( - argon2::hash_encoded( - new_password.as_bytes(), - &rand::random::<[u8; 16]>()[..], - &argon2::Config::rfc9106_low_mem(), - ) - .with_kind(crate::ErrorKind::PasswordHashGeneration)?, - ); - self.unencrypted_metadata.wrapped_key = Some(base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &encrypt_slice(&self.enc_key, new_password), - )); - Ok(()) - } - - #[instrument(skip_all)] - pub async fn mount_package_backup( - &self, - id: &PackageId, - ) -> Result { - let lock = FileLock::new(Path::new(BACKUP_DIR).join(format!("{}.lock", id)), false).await?; - let mountpoint = Path::new(BACKUP_DIR).join(id); - bind(self.as_ref().join(id), &mountpoint, false).await?; - Ok(PackageBackupMountGuard { - mountpoint: Some(mountpoint), - lock: Some(lock), - }) - } - - #[instrument(skip_all)] - pub async fn save(&self) -> Result<(), Error> { - let metadata_path = self.as_ref().join("metadata.cbor"); - let backup_disk_path = self.backup_disk_path(); - let mut file = AtomicFile::new(&metadata_path, None::) - .await - .with_kind(ErrorKind::Filesystem)?; - file.write_all(&IoFormat::Cbor.to_vec(&self.metadata)?) - .await?; - file.save().await.with_kind(ErrorKind::Filesystem)?; - let unencrypted_metadata_path = - backup_disk_path.join("EmbassyBackups/unencrypted-metadata.cbor"); - let mut file = AtomicFile::new(&unencrypted_metadata_path, None::) - .await - .with_kind(ErrorKind::Filesystem)?; - file.write_all(&IoFormat::Cbor.to_vec(&self.unencrypted_metadata)?) - .await?; - file.save().await.with_kind(ErrorKind::Filesystem)?; - Ok(()) - } - - #[instrument(skip_all)] - pub async fn unmount(mut self) -> Result<(), Error> { - if let Some(guard) = self.encrypted_guard.take() { - guard.unmount().await?; - } - if let Some(guard) = self.backup_disk_mount_guard.take() { - guard.unmount().await?; - } - Ok(()) - } - - #[instrument(skip_all)] - pub async fn save_and_unmount(self) -> Result<(), Error> { - self.save().await?; - self.unmount().await?; - Ok(()) - } -} -impl AsRef for BackupMountGuard { - fn as_ref(&self) -> &Path { - if let Some(guard) = &self.encrypted_guard { - guard.as_ref() - } else { - unreachable!() - } - } -} -impl Drop for BackupMountGuard { - fn drop(&mut self) { - let first = self.encrypted_guard.take(); - let second = self.backup_disk_mount_guard.take(); - tokio::spawn(async move { - if let Some(guard) = first { - guard.unmount().await.unwrap(); - } - if let Some(guard) = second { - guard.unmount().await.unwrap(); - } - }); - } -} - -pub struct PackageBackupMountGuard { - mountpoint: Option, - lock: Option, -} -impl PackageBackupMountGuard { - pub async fn unmount(mut self) -> Result<(), Error> { - if let Some(mountpoint) = self.mountpoint.take() { - unmount(&mountpoint).await?; - } - if let Some(lock) = self.lock.take() { - lock.unlock().await?; - } - Ok(()) - } -} -impl AsRef for PackageBackupMountGuard { - fn as_ref(&self) -> &Path { - if let Some(mountpoint) = &self.mountpoint { - mountpoint - } else { - unreachable!() - } - } -} -impl Drop for PackageBackupMountGuard { - fn drop(&mut self) { - let mountpoint = self.mountpoint.take(); - let lock = self.lock.take(); - tokio::spawn(async move { - if let Some(mountpoint) = mountpoint { - unmount(&mountpoint).await.unwrap(); - } - if let Some(lock) = lock { - lock.unlock().await.unwrap(); - } - }); - } -} diff --git a/core/startos/src/disk/mount/filesystem/bind.rs b/core/startos/src/disk/mount/filesystem/bind.rs deleted file mode 100644 index 8799372e..00000000 --- a/core/startos/src/disk/mount/filesystem/bind.rs +++ /dev/null @@ -1,54 +0,0 @@ -use std::os::unix::ffi::OsStrExt; -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use sha2::Sha256; - -use super::{FileSystem, MountType, ReadOnly}; -use crate::disk::mount::util::bind; -use crate::{Error, ResultExt}; - -pub struct Bind> { - src_dir: SrcDir, -} -impl> Bind { - pub fn new(src_dir: SrcDir) -> Self { - Self { src_dir } - } -} -#[async_trait] -impl + Send + Sync> FileSystem for Bind { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error> { - bind( - self.src_dir.as_ref(), - mountpoint, - matches!(mount_type, ReadOnly), - ) - .await - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("Bind"); - sha.update( - tokio::fs::canonicalize(self.src_dir.as_ref()) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - self.src_dir.as_ref().display().to_string(), - ) - })? - .as_os_str() - .as_bytes(), - ); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/block_dev.rs b/core/startos/src/disk/mount/filesystem/block_dev.rs deleted file mode 100644 index e21f0c42..00000000 --- a/core/startos/src/disk/mount/filesystem/block_dev.rs +++ /dev/null @@ -1,67 +0,0 @@ -use std::os::unix::ffi::OsStrExt; -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use serde::{Deserialize, Serialize}; -use sha2::Sha256; - -use super::{FileSystem, MountType, ReadOnly}; -use crate::util::Invoke; -use crate::{Error, ResultExt}; - -pub async fn mount( - logicalname: impl AsRef, - mountpoint: impl AsRef, - mount_type: MountType, -) -> Result<(), Error> { - tokio::fs::create_dir_all(mountpoint.as_ref()).await?; - let mut cmd = tokio::process::Command::new("mount"); - cmd.arg(logicalname.as_ref()).arg(mountpoint.as_ref()); - if mount_type == ReadOnly { - cmd.arg("-o").arg("ro"); - } - cmd.invoke(crate::ErrorKind::Filesystem).await?; - Ok(()) -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct BlockDev> { - logicalname: LogicalName, -} -impl> BlockDev { - pub fn new(logicalname: LogicalName) -> Self { - BlockDev { logicalname } - } -} -#[async_trait] -impl + Send + Sync> FileSystem for BlockDev { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error> { - mount(self.logicalname.as_ref(), mountpoint, mount_type).await - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("BlockDev"); - sha.update( - tokio::fs::canonicalize(self.logicalname.as_ref()) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - self.logicalname.as_ref().display().to_string(), - ) - })? - .as_os_str() - .as_bytes(), - ); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/cifs.rs b/core/startos/src/disk/mount/filesystem/cifs.rs deleted file mode 100644 index 91b477fc..00000000 --- a/core/startos/src/disk/mount/filesystem/cifs.rs +++ /dev/null @@ -1,107 +0,0 @@ -use std::net::IpAddr; -use std::os::unix::ffi::OsStrExt; -use std::path::{Path, PathBuf}; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use serde::{Deserialize, Serialize}; -use sha2::Sha256; -use tokio::process::Command; -use tracing::instrument; - -use super::{FileSystem, MountType, ReadOnly}; -use crate::disk::mount::guard::TmpMountGuard; -use crate::util::Invoke; -use crate::Error; - -async fn resolve_hostname(hostname: &str) -> Result { - if let Ok(addr) = hostname.parse() { - return Ok(addr); - } - if hostname.ends_with(".local") { - return Ok(IpAddr::V4(crate::net::mdns::resolve_mdns(hostname).await?)); - } - Ok(String::from_utf8( - Command::new("nmblookup") - .arg(hostname) - .invoke(crate::ErrorKind::Network) - .await?, - )? - .split(" ") - .next() - .unwrap() - .trim() - .parse()?) -} - -#[instrument(skip_all)] -pub async fn mount_cifs( - hostname: &str, - path: impl AsRef, - username: &str, - password: Option<&str>, - mountpoint: impl AsRef, - mount_type: MountType, -) -> Result<(), Error> { - tokio::fs::create_dir_all(mountpoint.as_ref()).await?; - let ip: IpAddr = resolve_hostname(hostname).await?; - let absolute_path = Path::new("/").join(path.as_ref()); - let mut cmd = Command::new("mount"); - cmd.arg("-t") - .arg("cifs") - .env("USER", username) - .env("PASSWD", password.unwrap_or_default()) - .arg(format!("//{}{}", ip, absolute_path.display())) - .arg(mountpoint.as_ref()); - if mount_type == ReadOnly { - cmd.arg("-o").arg("ro,noserverino"); - } else { - cmd.arg("-o").arg("noserverino"); - } - cmd.invoke(crate::ErrorKind::Filesystem).await?; - Ok(()) -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Cifs { - pub hostname: String, - pub path: PathBuf, - pub username: String, - pub password: Option, -} -impl Cifs { - pub async fn mountable(&self) -> Result<(), Error> { - let guard = TmpMountGuard::mount(self, ReadOnly).await?; - guard.unmount().await?; - Ok(()) - } -} -#[async_trait] -impl FileSystem for Cifs { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error> { - mount_cifs( - &self.hostname, - &self.path, - &self.username, - self.password.as_ref().map(|p| p.as_str()), - mountpoint, - mount_type, - ) - .await - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("Cifs"); - sha.update(self.hostname.as_bytes()); - sha.update(self.path.as_os_str().as_bytes()); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/ecryptfs.rs b/core/startos/src/disk/mount/filesystem/ecryptfs.rs deleted file mode 100644 index 78570f49..00000000 --- a/core/startos/src/disk/mount/filesystem/ecryptfs.rs +++ /dev/null @@ -1,71 +0,0 @@ -use std::os::unix::ffi::OsStrExt; -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use sha2::Sha256; - -use super::{FileSystem, MountType}; -use crate::util::Invoke; -use crate::{Error, ResultExt}; - -pub async fn mount_ecryptfs, P1: AsRef>( - src: P0, - dst: P1, - key: &str, -) -> Result<(), Error> { - tokio::fs::create_dir_all(dst.as_ref()).await?; - tokio::process::Command::new("mount") - .arg("-t") - .arg("ecryptfs") - .arg(src.as_ref()) - .arg(dst.as_ref()) - .arg("-o") - // for more information `man ecryptfs` - .arg(format!("key=passphrase:passphrase_passwd={},ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=y,no_sig_cache", key)) - .input(Some(&mut std::io::Cursor::new(b"\n"))) - .invoke(crate::ErrorKind::Filesystem).await?; - Ok(()) -} - -pub struct EcryptFS, Key: AsRef> { - encrypted_dir: EncryptedDir, - key: Key, -} -impl, Key: AsRef> EcryptFS { - pub fn new(encrypted_dir: EncryptedDir, key: Key) -> Self { - EcryptFS { encrypted_dir, key } - } -} -#[async_trait] -impl + Send + Sync, Key: AsRef + Send + Sync> FileSystem - for EcryptFS -{ - async fn mount + Send + Sync>( - &self, - mountpoint: P, - _mount_type: MountType, // ignored - inherited from parent fs - ) -> Result<(), Error> { - mount_ecryptfs(self.encrypted_dir.as_ref(), mountpoint, self.key.as_ref()).await - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("EcryptFS"); - sha.update( - tokio::fs::canonicalize(self.encrypted_dir.as_ref()) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - self.encrypted_dir.as_ref().display().to_string(), - ) - })? - .as_os_str() - .as_bytes(), - ); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/efivarfs.rs b/core/startos/src/disk/mount/filesystem/efivarfs.rs deleted file mode 100644 index ad9d7994..00000000 --- a/core/startos/src/disk/mount/filesystem/efivarfs.rs +++ /dev/null @@ -1,39 +0,0 @@ -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use sha2::Sha256; - -use super::{FileSystem, MountType, ReadOnly}; -use crate::util::Invoke; -use crate::Error; - -pub struct EfiVarFs; -#[async_trait] -impl FileSystem for EfiVarFs { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error> { - tokio::fs::create_dir_all(mountpoint.as_ref()).await?; - let mut cmd = tokio::process::Command::new("mount"); - cmd.arg("-t") - .arg("efivarfs") - .arg("efivarfs") - .arg(mountpoint.as_ref()); - if mount_type == ReadOnly { - cmd.arg("-o").arg("ro"); - } - cmd.invoke(crate::ErrorKind::Filesystem).await?; - Ok(()) - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("EfiVarFs"); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/httpdirfs.rs b/core/startos/src/disk/mount/filesystem/httpdirfs.rs deleted file mode 100644 index fda437ec..00000000 --- a/core/startos/src/disk/mount/filesystem/httpdirfs.rs +++ /dev/null @@ -1,52 +0,0 @@ -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use reqwest::Url; -use serde::{Deserialize, Serialize}; -use sha2::Sha256; - -use super::{FileSystem, MountType}; -use crate::util::Invoke; -use crate::Error; - -pub async fn mount_httpdirfs(url: &Url, mountpoint: impl AsRef) -> Result<(), Error> { - tokio::fs::create_dir_all(mountpoint.as_ref()).await?; - let mut cmd = tokio::process::Command::new("httpdirfs"); - cmd.arg("--cache") - .arg("--single-file-mode") - .arg(url.as_str()) - .arg(mountpoint.as_ref()); - cmd.invoke(crate::ErrorKind::Filesystem).await?; - Ok(()) -} - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct HttpDirFS { - url: Url, -} -impl HttpDirFS { - pub fn new(url: Url) -> Self { - HttpDirFS { url } - } -} -#[async_trait] -impl FileSystem for HttpDirFS { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - _mount_type: MountType, - ) -> Result<(), Error> { - mount_httpdirfs(&self.url, mountpoint).await - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("HttpDirFS"); - sha.update(self.url.as_str()); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/label.rs b/core/startos/src/disk/mount/filesystem/label.rs deleted file mode 100644 index b1e4f721..00000000 --- a/core/startos/src/disk/mount/filesystem/label.rs +++ /dev/null @@ -1,52 +0,0 @@ -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::{Digest, OutputSizeUser}; -use sha2::Sha256; - -use super::{FileSystem, MountType, ReadOnly}; -use crate::util::Invoke; -use crate::Error; - -pub async fn mount_label( - label: &str, - mountpoint: impl AsRef, - mount_type: MountType, -) -> Result<(), Error> { - tokio::fs::create_dir_all(mountpoint.as_ref()).await?; - let mut cmd = tokio::process::Command::new("mount"); - cmd.arg("-L").arg(label).arg(mountpoint.as_ref()); - if mount_type == ReadOnly { - cmd.arg("-o").arg("ro"); - } - cmd.invoke(crate::ErrorKind::Filesystem).await?; - Ok(()) -} - -pub struct Label> { - label: S, -} -impl> Label { - pub fn new(label: S) -> Self { - Label { label } - } -} -#[async_trait] -impl + Send + Sync> FileSystem for Label { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error> { - mount_label(self.label.as_ref(), mountpoint, mount_type).await - } - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error> { - let mut sha = Sha256::new(); - sha.update("Label"); - sha.update(self.label.as_ref().as_bytes()); - Ok(sha.finalize()) - } -} diff --git a/core/startos/src/disk/mount/filesystem/mod.rs b/core/startos/src/disk/mount/filesystem/mod.rs deleted file mode 100644 index 00247e0d..00000000 --- a/core/startos/src/disk/mount/filesystem/mod.rs +++ /dev/null @@ -1,36 +0,0 @@ -use std::path::Path; - -use async_trait::async_trait; -use digest::generic_array::GenericArray; -use digest::OutputSizeUser; -use sha2::Sha256; - -use crate::Error; - -pub mod bind; -pub mod block_dev; -pub mod cifs; -pub mod ecryptfs; -pub mod efivarfs; -pub mod httpdirfs; -pub mod label; - -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] -pub enum MountType { - ReadOnly, - ReadWrite, -} - -pub use MountType::*; - -#[async_trait] -pub trait FileSystem { - async fn mount + Send + Sync>( - &self, - mountpoint: P, - mount_type: MountType, - ) -> Result<(), Error>; - async fn source_hash( - &self, - ) -> Result::OutputSize>, Error>; -} diff --git a/core/startos/src/disk/mount/guard.rs b/core/startos/src/disk/mount/guard.rs deleted file mode 100644 index 617afeb0..00000000 --- a/core/startos/src/disk/mount/guard.rs +++ /dev/null @@ -1,142 +0,0 @@ -use std::collections::BTreeMap; -use std::path::{Path, PathBuf}; -use std::sync::{Arc, Weak}; - -use lazy_static::lazy_static; -use models::ResultExt; -use tokio::sync::Mutex; -use tracing::instrument; - -use super::filesystem::{FileSystem, MountType, ReadOnly, ReadWrite}; -use super::util::unmount; -use crate::util::Invoke; -use crate::Error; - -pub const TMP_MOUNTPOINT: &'static str = "/media/embassy/tmp"; - -#[async_trait::async_trait] -pub trait GenericMountGuard: AsRef + std::fmt::Debug + Send + Sync + 'static { - async fn unmount(mut self) -> Result<(), Error>; -} - -#[derive(Debug)] -pub struct MountGuard { - mountpoint: PathBuf, - mounted: bool, -} -impl MountGuard { - pub async fn mount( - filesystem: &impl FileSystem, - mountpoint: impl AsRef, - mount_type: MountType, - ) -> Result { - let mountpoint = mountpoint.as_ref().to_owned(); - filesystem.mount(&mountpoint, mount_type).await?; - Ok(MountGuard { - mountpoint, - mounted: true, - }) - } - pub async fn unmount(mut self, delete_mountpoint: bool) -> Result<(), Error> { - if self.mounted { - unmount(&self.mountpoint).await?; - if delete_mountpoint { - match tokio::fs::remove_dir(&self.mountpoint).await { - Err(e) if e.raw_os_error() == Some(39) => Ok(()), // directory not empty - a => a, - } - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("rm {}", self.mountpoint.display()), - ) - })?; - } - self.mounted = false; - } - Ok(()) - } -} -impl AsRef for MountGuard { - fn as_ref(&self) -> &Path { - &self.mountpoint - } -} -impl Drop for MountGuard { - fn drop(&mut self) { - if self.mounted { - let mountpoint = std::mem::take(&mut self.mountpoint); - tokio::spawn(async move { unmount(mountpoint).await.unwrap() }); - } - } -} -#[async_trait::async_trait] -impl GenericMountGuard for MountGuard { - async fn unmount(mut self) -> Result<(), Error> { - MountGuard::unmount(self, false).await - } -} - -async fn tmp_mountpoint(source: &impl FileSystem) -> Result { - Ok(Path::new(TMP_MOUNTPOINT).join(base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &source.source_hash().await?, - ))) -} - -lazy_static! { - static ref TMP_MOUNTS: Mutex)>> = - Mutex::new(BTreeMap::new()); -} - -#[derive(Debug)] -pub struct TmpMountGuard { - guard: Arc, -} -impl TmpMountGuard { - /// DRAGONS: if you try to mount something as ro and rw at the same time, the ro mount will be upgraded to rw. - #[instrument(skip_all)] - pub async fn mount(filesystem: &impl FileSystem, mount_type: MountType) -> Result { - let mountpoint = tmp_mountpoint(filesystem).await?; - let mut tmp_mounts = TMP_MOUNTS.lock().await; - if !tmp_mounts.contains_key(&mountpoint) { - tmp_mounts.insert(mountpoint.clone(), (mount_type, Weak::new())); - } - let (prev_mt, weak_slot) = tmp_mounts.get_mut(&mountpoint).unwrap(); - if let Some(guard) = weak_slot.upgrade() { - // upgrade to rw - if *prev_mt == ReadOnly && mount_type == ReadWrite { - tokio::process::Command::new("mount") - .arg("-o") - .arg("remount,rw") - .arg(&mountpoint) - .invoke(crate::ErrorKind::Filesystem) - .await?; - *prev_mt = ReadWrite; - } - Ok(TmpMountGuard { guard }) - } else { - let guard = Arc::new(MountGuard::mount(filesystem, &mountpoint, mount_type).await?); - *weak_slot = Arc::downgrade(&guard); - *prev_mt = mount_type; - Ok(TmpMountGuard { guard }) - } - } - pub async fn unmount(self) -> Result<(), Error> { - if let Ok(guard) = Arc::try_unwrap(self.guard) { - guard.unmount(true).await?; - } - Ok(()) - } -} -impl AsRef for TmpMountGuard { - fn as_ref(&self) -> &Path { - (&*self.guard).as_ref() - } -} -#[async_trait::async_trait] -impl GenericMountGuard for TmpMountGuard { - async fn unmount(mut self) -> Result<(), Error> { - TmpMountGuard::unmount(self).await - } -} diff --git a/core/startos/src/disk/mount/mod.rs b/core/startos/src/disk/mount/mod.rs deleted file mode 100644 index 74e34e86..00000000 --- a/core/startos/src/disk/mount/mod.rs +++ /dev/null @@ -1,4 +0,0 @@ -pub mod backup; -pub mod filesystem; -pub mod guard; -pub mod util; diff --git a/core/startos/src/disk/mount/util.rs b/core/startos/src/disk/mount/util.rs deleted file mode 100644 index 392e5d67..00000000 --- a/core/startos/src/disk/mount/util.rs +++ /dev/null @@ -1,52 +0,0 @@ -use std::path::Path; - -use tracing::instrument; - -use crate::util::Invoke; -use crate::Error; - -#[instrument(skip_all)] -pub async fn bind, P1: AsRef>( - src: P0, - dst: P1, - read_only: bool, -) -> Result<(), Error> { - tracing::info!( - "Binding {} to {}", - src.as_ref().display(), - dst.as_ref().display() - ); - let is_mountpoint = tokio::process::Command::new("mountpoint") - .arg(dst.as_ref()) - .stdout(std::process::Stdio::null()) - .stderr(std::process::Stdio::null()) - .status() - .await?; - if is_mountpoint.success() { - unmount(dst.as_ref()).await?; - } - tokio::fs::create_dir_all(&src).await?; - tokio::fs::create_dir_all(&dst).await?; - let mut mount_cmd = tokio::process::Command::new("mount"); - mount_cmd.arg("--bind"); - if read_only { - mount_cmd.arg("-o").arg("ro"); - } - mount_cmd - .arg(src.as_ref()) - .arg(dst.as_ref()) - .invoke(crate::ErrorKind::Filesystem) - .await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn unmount>(mountpoint: P) -> Result<(), Error> { - tracing::debug!("Unmounting {}.", mountpoint.as_ref().display()); - tokio::process::Command::new("umount") - .arg("-l") - .arg(mountpoint.as_ref()) - .invoke(crate::ErrorKind::Filesystem) - .await?; - Ok(()) -} diff --git a/core/startos/src/disk/util.rs b/core/startos/src/disk/util.rs deleted file mode 100644 index 7051026c..00000000 --- a/core/startos/src/disk/util.rs +++ /dev/null @@ -1,489 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; -use std::path::{Path, PathBuf}; - -use color_eyre::eyre::{self, eyre}; -use futures::TryStreamExt; -use nom::bytes::complete::{tag, take_till1}; -use nom::character::complete::multispace1; -use nom::character::is_space; -use nom::combinator::{opt, rest}; -use nom::sequence::{pair, preceded, terminated}; -use nom::IResult; -use regex::Regex; -use serde::{Deserialize, Serialize}; -use tokio::process::Command; -use tracing::instrument; - -use super::mount::filesystem::block_dev::BlockDev; -use super::mount::filesystem::ReadOnly; -use super::mount::guard::TmpMountGuard; -use crate::disk::OsPartitionInfo; -use crate::util::serde::IoFormat; -use crate::util::{Invoke, Version}; -use crate::{Error, ResultExt as _}; - -#[derive(Clone, Copy, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub enum PartitionTable { - Mbr, - Gpt, -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct DiskInfo { - pub logicalname: PathBuf, - pub partition_table: Option, - pub vendor: Option, - pub model: Option, - pub partitions: Vec, - pub capacity: u64, - pub guid: Option, -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct PartitionInfo { - pub logicalname: PathBuf, - pub label: Option, - pub capacity: u64, - pub used: Option, - pub embassy_os: Option, - pub guid: Option, -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct EmbassyOsRecoveryInfo { - pub version: Version, - pub full: bool, - pub password_hash: Option, - pub wrapped_key: Option, -} - -const DISK_PATH: &str = "/dev/disk/by-path"; -const SYS_BLOCK_PATH: &str = "/sys/block"; - -lazy_static::lazy_static! { - static ref PARTITION_REGEX: Regex = Regex::new("-part[0-9]+$").unwrap(); -} - -#[instrument(skip_all)] -pub async fn get_partition_table>(path: P) -> Result, Error> { - Ok(String::from_utf8( - Command::new("fdisk") - .arg("-l") - .arg(path.as_ref()) - .invoke(crate::ErrorKind::BlockDevice) - .await?, - )? - .lines() - .find_map(|l| l.strip_prefix("Disklabel type:")) - .and_then(|t| match t.trim() { - "dos" => Some(PartitionTable::Mbr), - "gpt" => Some(PartitionTable::Gpt), - _ => None, - })) -} - -#[instrument(skip_all)] -pub async fn get_vendor>(path: P) -> Result, Error> { - let vendor = tokio::fs::read_to_string( - Path::new(SYS_BLOCK_PATH) - .join(path.as_ref().strip_prefix("/dev").map_err(|_| { - Error::new( - eyre!("not a canonical block device"), - crate::ErrorKind::BlockDevice, - ) - })?) - .join("device") - .join("vendor"), - ) - .await? - .trim() - .to_owned(); - Ok(if vendor.is_empty() { - None - } else { - Some(vendor) - }) -} - -#[instrument(skip_all)] -pub async fn get_model>(path: P) -> Result, Error> { - let model = tokio::fs::read_to_string( - Path::new(SYS_BLOCK_PATH) - .join(path.as_ref().strip_prefix("/dev").map_err(|_| { - Error::new( - eyre!("not a canonical block device"), - crate::ErrorKind::BlockDevice, - ) - })?) - .join("device") - .join("model"), - ) - .await? - .trim() - .to_owned(); - Ok(if model.is_empty() { None } else { Some(model) }) -} - -#[instrument(skip_all)] -pub async fn get_capacity>(path: P) -> Result { - Ok(String::from_utf8( - Command::new("blockdev") - .arg("--getsize64") - .arg(path.as_ref()) - .invoke(crate::ErrorKind::BlockDevice) - .await?, - )? - .trim() - .parse::()?) -} - -#[instrument(skip_all)] -pub async fn get_label>(path: P) -> Result, Error> { - let label = String::from_utf8( - Command::new("lsblk") - .arg("-no") - .arg("label") - .arg(path.as_ref()) - .invoke(crate::ErrorKind::BlockDevice) - .await?, - )? - .trim() - .to_owned(); - Ok(if label.is_empty() { None } else { Some(label) }) -} - -#[instrument(skip_all)] -pub async fn get_used>(path: P) -> Result { - Ok(String::from_utf8( - Command::new("df") - .arg("--output=used") - .arg("--block-size=1") - .arg(path.as_ref()) - .invoke(crate::ErrorKind::Filesystem) - .await?, - )? - .lines() - .skip(1) - .next() - .unwrap_or_default() - .trim() - .parse::()?) -} - -#[instrument(skip_all)] -pub async fn get_available>(path: P) -> Result { - Ok(String::from_utf8( - Command::new("df") - .arg("--output=avail") - .arg("--block-size=1") - .arg(path.as_ref()) - .invoke(crate::ErrorKind::Filesystem) - .await?, - )? - .lines() - .skip(1) - .next() - .unwrap_or_default() - .trim() - .parse::()?) -} - -#[instrument(skip_all)] -pub async fn get_percentage>(path: P) -> Result { - Ok(String::from_utf8( - Command::new("df") - .arg("--output=pcent") - .arg(path.as_ref()) - .invoke(crate::ErrorKind::Filesystem) - .await?, - )? - .lines() - .skip(1) - .next() - .unwrap_or_default() - .trim() - .strip_suffix("%") - .unwrap() - .parse::()?) -} - -#[instrument(skip_all)] -pub async fn pvscan() -> Result>, Error> { - let pvscan_out = Command::new("pvscan") - .invoke(crate::ErrorKind::DiskManagement) - .await?; - let pvscan_out_str = std::str::from_utf8(&pvscan_out)?; - Ok(parse_pvscan_output(pvscan_out_str)) -} - -pub async fn recovery_info( - mountpoint: impl AsRef, -) -> Result, Error> { - let backup_unencrypted_metadata_path = mountpoint - .as_ref() - .join("EmbassyBackups/unencrypted-metadata.cbor"); - if tokio::fs::metadata(&backup_unencrypted_metadata_path) - .await - .is_ok() - { - return Ok(Some( - IoFormat::Cbor.from_slice( - &tokio::fs::read(&backup_unencrypted_metadata_path) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - backup_unencrypted_metadata_path.display().to_string(), - ) - })?, - )?, - )); - } - - Ok(None) -} - -#[instrument(skip_all)] -pub async fn list(os: &OsPartitionInfo) -> Result, Error> { - struct DiskIndex { - parts: BTreeSet, - internal: bool, - } - let disk_guids = pvscan().await?; - let disks = tokio_stream::wrappers::ReadDirStream::new( - tokio::fs::read_dir(DISK_PATH) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, DISK_PATH))?, - ) - .map_err(|e| { - Error::new( - eyre::Error::from(e).wrap_err(DISK_PATH), - crate::ErrorKind::Filesystem, - ) - }) - .try_fold( - BTreeMap::::new(), - |mut disks, dir_entry| async move { - if let Some(disk_path) = dir_entry.path().file_name().and_then(|s| s.to_str()) { - let (disk_path, part_path) = if let Some(end) = PARTITION_REGEX.find(disk_path) { - ( - disk_path.strip_suffix(end.as_str()).unwrap_or_default(), - Some(disk_path), - ) - } else { - (disk_path, None) - }; - let disk_path = Path::new(DISK_PATH).join(disk_path); - let disk = tokio::fs::canonicalize(&disk_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - disk_path.display().to_string(), - ) - })?; - let part = if let Some(part_path) = part_path { - let part_path = Path::new(DISK_PATH).join(part_path); - let part = tokio::fs::canonicalize(&part_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - part_path.display().to_string(), - ) - })?; - Some(part) - } else { - None - }; - if !disks.contains_key(&disk) { - disks.insert( - disk.clone(), - DiskIndex { - parts: BTreeSet::new(), - internal: false, - }, - ); - } - if let Some(part) = part { - if os.contains(&part) { - disks.get_mut(&disk).unwrap().internal = true; - } else { - disks.get_mut(&disk).unwrap().parts.insert(part); - } - } - } - Ok(disks) - }, - ) - .await?; - - let mut res = Vec::with_capacity(disks.len()); - for (disk, index) in disks { - if index.internal { - for part in index.parts { - let mut disk_info = disk_info(disk.clone()).await; - let part_info = part_info(part).await; - disk_info.logicalname = part_info.logicalname.clone(); - disk_info.capacity = part_info.capacity; - if let Some(g) = disk_guids.get(&disk_info.logicalname) { - disk_info.guid = g.clone(); - } else { - disk_info.partitions = vec![part_info]; - } - res.push(disk_info); - } - } else { - let mut disk_info = disk_info(disk).await; - disk_info.partitions = Vec::with_capacity(index.parts.len()); - if let Some(g) = disk_guids.get(&disk_info.logicalname) { - disk_info.guid = g.clone(); - } else { - for part in index.parts { - let mut part_info = part_info(part).await; - if let Some(g) = disk_guids.get(&part_info.logicalname) { - part_info.guid = g.clone(); - } - disk_info.partitions.push(part_info); - } - } - res.push(disk_info); - } - } - - Ok(res) -} - -async fn disk_info(disk: PathBuf) -> DiskInfo { - let partition_table = get_partition_table(&disk) - .await - .map_err(|e| { - tracing::warn!( - "Could not get partition table of {}: {}", - disk.display(), - e.source - ) - }) - .unwrap_or_default(); - let vendor = get_vendor(&disk) - .await - .map_err(|e| tracing::warn!("Could not get vendor of {}: {}", disk.display(), e.source)) - .unwrap_or_default(); - let model = get_model(&disk) - .await - .map_err(|e| tracing::warn!("Could not get model of {}: {}", disk.display(), e.source)) - .unwrap_or_default(); - let capacity = get_capacity(&disk) - .await - .map_err(|e| tracing::warn!("Could not get capacity of {}: {}", disk.display(), e.source)) - .unwrap_or_default(); - DiskInfo { - logicalname: disk, - partition_table, - vendor, - model, - partitions: Vec::new(), - capacity, - guid: None, - } -} - -async fn part_info(part: PathBuf) -> PartitionInfo { - let mut embassy_os = None; - let label = get_label(&part) - .await - .map_err(|e| tracing::warn!("Could not get label of {}: {}", part.display(), e.source)) - .unwrap_or_default(); - let capacity = get_capacity(&part) - .await - .map_err(|e| tracing::warn!("Could not get capacity of {}: {}", part.display(), e.source)) - .unwrap_or_default(); - let mut used = None; - - match TmpMountGuard::mount(&BlockDev::new(&part), ReadOnly).await { - Err(e) => tracing::warn!("Could not collect usage information: {}", e.source), - Ok(mount_guard) => { - used = get_used(&mount_guard) - .await - .map_err(|e| { - tracing::warn!("Could not get usage of {}: {}", part.display(), e.source) - }) - .ok(); - if let Some(recovery_info) = match recovery_info(&mount_guard).await { - Ok(a) => a, - Err(e) => { - tracing::error!("Error fetching unencrypted backup metadata: {}", e); - None - } - } { - embassy_os = Some(recovery_info) - } - if let Err(e) = mount_guard.unmount().await { - tracing::error!("Error unmounting partition {}: {}", part.display(), e); - } - } - } - - PartitionInfo { - logicalname: part, - label, - capacity, - used, - embassy_os, - guid: None, - } -} - -fn parse_pvscan_output(pvscan_output: &str) -> BTreeMap> { - fn parse_line(line: &str) -> IResult<&str, (&str, Option<&str>)> { - let pv_parse = preceded( - tag(" PV "), - terminated(take_till1(|c| is_space(c as u8)), multispace1), - ); - let vg_parse = preceded( - opt(tag("is in exported ")), - preceded( - tag("VG "), - terminated(take_till1(|c| is_space(c as u8)), multispace1), - ), - ); - let mut parser = terminated(pair(pv_parse, opt(vg_parse)), rest); - parser(line) - } - let lines = pvscan_output.lines(); - let n = lines.clone().count(); - let entries = lines.take(n.saturating_sub(1)); - let mut ret = BTreeMap::new(); - for entry in entries { - match parse_line(entry) { - Ok((_, (pv, vg))) => { - ret.insert(PathBuf::from(pv), vg.map(|s| s.to_owned())); - } - Err(_) => { - tracing::warn!("Failed to parse pvscan output line: {}", entry); - } - } - } - ret -} - -#[test] -fn test_pvscan_parser() { - let s1 = r#" PV /dev/mapper/cryptdata VG data lvm2 [1.81 TiB / 0 free] - PV /dev/sdb lvm2 [931.51 GiB] - Total: 2 [2.72 TiB] / in use: 1 [1.81 TiB] / in no VG: 1 [931.51 GiB] -"#; - let s2 = r#" PV /dev/sdb VG EMBASSY_LZHJAENWGPCJJL6C6AXOD7OOOIJG7HFBV4GYRJH6HADXUCN4BRWQ lvm2 [931.51 GiB / 0 free] - Total: 1 [931.51 GiB] / in use: 1 [931.51 GiB] / in no VG: 0 [0 ] -"#; - let s3 = r#" PV /dev/mapper/cryptdata VG data lvm2 [1.81 TiB / 0 free] - Total: 1 [1.81 TiB] / in use: 1 [1.81 TiB] / in no VG: 0 [0 ] -"#; - let s4 = r#" PV /dev/sda is in exported VG EMBASSY_ZFHOCTYV3ZJMJW3OTFMG55LSQZLP667EDNZKDNUJKPJX5HE6S5HQ [931.51 GiB / 0 free] - Total: 1 [931.51 GiB] / in use: 1 [931.51 GiB] / in no VG: 0 [0 ] -"#; - println!("{:?}", parse_pvscan_output(s1)); - println!("{:?}", parse_pvscan_output(s2)); - println!("{:?}", parse_pvscan_output(s3)); - println!("{:?}", parse_pvscan_output(s4)); -} diff --git a/core/startos/src/error.rs b/core/startos/src/error.rs deleted file mode 100644 index 2b769b03..00000000 --- a/core/startos/src/error.rs +++ /dev/null @@ -1,60 +0,0 @@ -use color_eyre::eyre::eyre; -pub use models::{Error, ErrorKind, OptionExt, ResultExt}; - -#[derive(Debug, Default)] -pub struct ErrorCollection(Vec); -impl ErrorCollection { - pub fn new() -> Self { - Self::default() - } - - pub fn handle>(&mut self, result: Result) -> Option { - match result { - Ok(a) => Some(a), - Err(e) => { - self.0.push(e.into()); - None - } - } - } - - pub fn into_result(self) -> Result<(), Error> { - if self.0.is_empty() { - Ok(()) - } else { - Err(Error::new(eyre!("{}", self), ErrorKind::MultipleErrors)) - } - } -} -impl From for Result<(), Error> { - fn from(e: ErrorCollection) -> Self { - e.into_result() - } -} -impl> Extend> for ErrorCollection { - fn extend>>(&mut self, iter: I) { - for item in iter { - self.handle(item); - } - } -} -impl std::fmt::Display for ErrorCollection { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - for (idx, e) in self.0.iter().enumerate() { - if idx > 0 { - write!(f, "; ")?; - } - write!(f, "{}", e)?; - } - Ok(()) - } -} - -#[macro_export] -macro_rules! ensure_code { - ($x:expr, $c:expr, $fmt:expr $(, $arg:expr)*) => { - if !($x) { - return Err(crate::error::Error::new(color_eyre::eyre::eyre!($fmt, $($arg, )*), $c)); - } - }; -} diff --git a/core/startos/src/firmware.rs b/core/startos/src/firmware.rs deleted file mode 100644 index 7f9a4a27..00000000 --- a/core/startos/src/firmware.rs +++ /dev/null @@ -1,149 +0,0 @@ -use std::collections::BTreeSet; -use std::path::Path; - -use async_compression::tokio::bufread::GzipDecoder; -use clap::ArgMatches; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use tokio::fs::File; -use tokio::io::BufReader; -use tokio::process::Command; - -use crate::disk::fsck::RequiresReboot; -use crate::prelude::*; -use crate::util::Invoke; -use crate::PLATFORM; - -/// Part of the Firmware, look there for more about -#[derive(Clone, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct VersionMatcher { - /// Strip this prefix on the version matcher - semver_prefix: Option, - /// Match the semver to this range - semver_range: Option, - /// Strip this suffix on the version matcher - semver_suffix: Option, -} - -/// Inside a file that is firmware.json, we -/// wanted a structure that could help decide what to do -/// for each of the firmware versions -#[derive(Clone, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Firmware { - id: String, - /// This is the platform(s) the firmware was built for - platform: BTreeSet, - /// This usally comes from the dmidecode - system_product_name: Option, - /// The version comes from dmidecode, then we decide if it matches - bios_version: Option, - /// the hash of the firmware rom.gz - shasum: String, -} - -fn display_firmware_update_result(arg: RequiresReboot, _: &ArgMatches) { - if arg.0 { - println!("Firmware successfully updated! Reboot to apply changes."); - } else { - println!("No firmware update available."); - } -} - -/// We wanted to make sure during every init -/// that the firmware was the correct and updated for -/// systems like the Pure System that a new firmware -/// was released and the updates where pushed through the pure os. -#[command(rename = "update-firmware", display(display_firmware_update_result))] -pub async fn update_firmware() -> Result { - let system_product_name = String::from_utf8( - Command::new("dmidecode") - .arg("-s") - .arg("system-product-name") - .invoke(ErrorKind::Firmware) - .await?, - )? - .trim() - .to_owned(); - let bios_version = String::from_utf8( - Command::new("dmidecode") - .arg("-s") - .arg("bios-version") - .invoke(ErrorKind::Firmware) - .await?, - )? - .trim() - .to_owned(); - if system_product_name.is_empty() || bios_version.is_empty() { - return Ok(RequiresReboot(false)); - } - - let firmware_dir = Path::new("/usr/lib/startos/firmware"); - - for firmware in serde_json::from_str::>( - &tokio::fs::read_to_string("/usr/lib/startos/firmware.json").await?, - ) - .with_kind(ErrorKind::Deserialization)? - { - let id = firmware.id; - let matches_product_name = firmware - .system_product_name - .map_or(true, |spn| spn == system_product_name); - let matches_bios_version = firmware - .bios_version - .map_or(Some(true), |bv| { - let mut semver_str = bios_version.as_str(); - if let Some(prefix) = &bv.semver_prefix { - semver_str = semver_str.strip_prefix(prefix)?; - } - if let Some(suffix) = &bv.semver_suffix { - semver_str = semver_str.strip_suffix(suffix)?; - } - let semver = semver_str - .split(".") - .filter_map(|v| v.parse().ok()) - .chain(std::iter::repeat(0)) - .take(3) - .collect::>(); - let semver = semver::Version::new(semver[0], semver[1], semver[2]); - Some( - bv.semver_range - .as_ref() - .map_or(true, |r| r.matches(&semver)), - ) - }) - .unwrap_or(false); - if firmware.platform.contains(&*PLATFORM) && matches_product_name && matches_bios_version { - let filename = format!("{id}.rom.gz"); - let firmware_path = firmware_dir.join(&filename); - Command::new("sha256sum") - .arg("-c") - .input(Some(&mut std::io::Cursor::new(format!( - "{} {}", - firmware.shasum, - firmware_path.display() - )))) - .invoke(ErrorKind::Filesystem) - .await?; - let mut rdr = if tokio::fs::metadata(&firmware_path).await.is_ok() { - GzipDecoder::new(BufReader::new(File::open(&firmware_path).await?)) - } else { - return Err(Error::new( - eyre!("Firmware {id}.rom.gz not found in {firmware_dir:?}"), - ErrorKind::NotFound, - )); - }; - Command::new("flashrom") - .arg("-p") - .arg("internal") - .arg("-w-") - .input(Some(&mut rdr)) - .invoke(ErrorKind::Firmware) - .await?; - return Ok(RequiresReboot(true)); - } - } - - Ok(RequiresReboot(false)) -} diff --git a/core/startos/src/hostname.rs b/core/startos/src/hostname.rs deleted file mode 100644 index f68d5c9d..00000000 --- a/core/startos/src/hostname.rs +++ /dev/null @@ -1,83 +0,0 @@ -use rand::{thread_rng, Rng}; -use tokio::process::Command; -use tracing::instrument; - -use crate::util::Invoke; -use crate::{Error, ErrorKind}; -#[derive(Clone, serde::Deserialize, serde::Serialize, Debug)] -pub struct Hostname(pub String); - -lazy_static::lazy_static! { - static ref ADJECTIVES: Vec = include_str!("./assets/adjectives.txt").lines().map(|x| x.to_string()).collect(); - static ref NOUNS: Vec = include_str!("./assets/nouns.txt").lines().map(|x| x.to_string()).collect(); -} -impl AsRef for Hostname { - fn as_ref(&self) -> &str { - &self.0 - } -} - -impl Hostname { - pub fn lan_address(&self) -> String { - format!("https://{}.local", self.0) - } - - pub fn local_domain_name(&self) -> String { - format!("{}.local", self.0) - } - pub fn no_dot_host_name(&self) -> String { - self.0.to_owned() - } -} - -pub fn generate_hostname() -> Hostname { - let mut rng = thread_rng(); - let adjective = &ADJECTIVES[rng.gen_range(0..ADJECTIVES.len())]; - let noun = &NOUNS[rng.gen_range(0..NOUNS.len())]; - Hostname(format!("{adjective}-{noun}")) -} - -pub fn generate_id() -> String { - let id = uuid::Uuid::new_v4(); - id.to_string() -} - -#[instrument(skip_all)] -pub async fn get_current_hostname() -> Result { - let out = Command::new("hostname") - .invoke(ErrorKind::ParseSysInfo) - .await?; - let out_string = String::from_utf8(out)?; - Ok(Hostname(out_string.trim().to_owned())) -} - -#[instrument(skip_all)] -pub async fn set_hostname(hostname: &Hostname) -> Result<(), Error> { - let hostname: &String = &hostname.0; - Command::new("hostnamectl") - .arg("--static") - .arg("set-hostname") - .arg(hostname) - .invoke(ErrorKind::ParseSysInfo) - .await?; - Command::new("sed") - .arg("-i") - .arg(format!( - "s/\\(\\s\\)localhost\\( {hostname}\\)\\?/\\1localhost {hostname}/g" - )) - .arg("/etc/hosts") - .invoke(ErrorKind::ParseSysInfo) - .await?; - Ok(()) -} - -#[instrument(skip_all)] -pub async fn sync_hostname(hostname: &Hostname) -> Result<(), Error> { - set_hostname(hostname).await?; - Command::new("systemctl") - .arg("restart") - .arg("avahi-daemon") - .invoke(crate::ErrorKind::Network) - .await?; - Ok(()) -} diff --git a/core/startos/src/init.rs b/core/startos/src/init.rs deleted file mode 100644 index 74c3767e..00000000 --- a/core/startos/src/init.rs +++ /dev/null @@ -1,447 +0,0 @@ -use std::fs::Permissions; -use std::os::unix::fs::PermissionsExt; -use std::path::Path; -use std::time::{Duration, SystemTime}; - -use color_eyre::eyre::eyre; - -use models::ResultExt; -use rand::random; -use sqlx::{Pool, Postgres}; -use tokio::process::Command; -use tracing::instrument; - -use crate::account::AccountInfo; -use crate::context::rpc::RpcContextConfig; -use crate::db::model::ServerStatus; -use crate::disk::mount::util::unmount; -use crate::install::PKG_ARCHIVE_DIR; -use crate::middleware::auth::LOCAL_AUTH_COOKIE_PATH; -use crate::prelude::*; - -use crate::util::cpupower::{ - get_available_governors, get_preferred_governor, set_governor, -}; -use crate::util::docker::{create_bridge_network, CONTAINER_DATADIR, CONTAINER_TOOL}; -use crate::util::Invoke; -use crate::{Error, ARCH}; - -pub const SYSTEM_REBUILD_PATH: &str = "/media/embassy/config/system-rebuild"; -pub const STANDBY_MODE_PATH: &str = "/media/embassy/config/standby"; - -pub async fn check_time_is_synchronized() -> Result { - Ok(String::from_utf8( - Command::new("timedatectl") - .arg("show") - .arg("-p") - .arg("NTPSynchronized") - .invoke(crate::ErrorKind::Unknown) - .await?, - )? - .trim() - == "NTPSynchronized=yes") -} - -// must be idempotent -#[tracing::instrument(skip_all)] -pub async fn init_postgres(datadir: impl AsRef) -> Result<(), Error> { - let db_dir = datadir.as_ref().join("main/postgresql"); - if tokio::process::Command::new("mountpoint") - .arg("/var/lib/postgresql") - .stdout(std::process::Stdio::null()) - .stderr(std::process::Stdio::null()) - .status() - .await? - .success() - { - unmount("/var/lib/postgresql").await?; - } - let exists = tokio::fs::metadata(&db_dir).await.is_ok(); - if !exists { - Command::new("cp") - .arg("-ra") - .arg("/var/lib/postgresql") - .arg(&db_dir) - .invoke(crate::ErrorKind::Filesystem) - .await?; - } - Command::new("chown") - .arg("-R") - .arg("postgres:postgres") - .arg(&db_dir) - .invoke(crate::ErrorKind::Database) - .await?; - - let mut pg_paths = tokio::fs::read_dir("/usr/lib/postgresql").await?; - let mut pg_version = None; - while let Some(pg_path) = pg_paths.next_entry().await? { - let pg_path_version = pg_path - .file_name() - .to_str() - .map(|v| v.parse()) - .transpose()? - .unwrap_or(0); - if pg_path_version > pg_version.unwrap_or(0) { - pg_version = Some(pg_path_version) - } - } - let pg_version = pg_version.ok_or_else(|| { - Error::new( - eyre!("could not determine postgresql version"), - crate::ErrorKind::Database, - ) - })?; - - crate::disk::mount::util::bind(&db_dir, "/var/lib/postgresql", false).await?; - - let pg_version_string = pg_version.to_string(); - let pg_version_path = db_dir.join(&pg_version_string); - if exists - // maybe migrate - { - let incomplete_path = db_dir.join(format!("{pg_version}.migration.incomplete")); - if tokio::fs::metadata(&incomplete_path).await.is_ok() // previous migration was incomplete - && tokio::fs::metadata(&pg_version_path).await.is_ok() - { - tokio::fs::remove_dir_all(&pg_version_path).await?; - } - if tokio::fs::metadata(&pg_version_path).await.is_err() - // need to migrate - { - let conf_dir = Path::new("/etc/postgresql").join(pg_version.to_string()); - let conf_dir_tmp = { - let mut tmp = conf_dir.clone(); - tmp.set_extension("tmp"); - tmp - }; - if tokio::fs::metadata(&conf_dir).await.is_ok() { - Command::new("mv") - .arg(&conf_dir) - .arg(&conf_dir_tmp) - .invoke(ErrorKind::Filesystem) - .await?; - } - let mut old_version = pg_version; - while old_version > 13 - /* oldest pg version included in startos */ - { - old_version -= 1; - let old_datadir = db_dir.join(old_version.to_string()); - if tokio::fs::metadata(&old_datadir).await.is_ok() { - tokio::fs::File::create(&incomplete_path) - .await? - .sync_all() - .await?; - Command::new("pg_upgradecluster") - .arg(old_version.to_string()) - .arg("main") - .invoke(crate::ErrorKind::Database) - .await?; - break; - } - } - if tokio::fs::metadata(&conf_dir).await.is_ok() { - if tokio::fs::metadata(&conf_dir).await.is_ok() { - tokio::fs::remove_dir_all(&conf_dir).await?; - } - Command::new("mv") - .arg(&conf_dir_tmp) - .arg(&conf_dir) - .invoke(ErrorKind::Filesystem) - .await?; - } - tokio::fs::remove_file(&incomplete_path).await?; - } - if tokio::fs::metadata(&incomplete_path).await.is_ok() { - unreachable!() // paranoia - } - } - - Command::new("systemctl") - .arg("start") - .arg(format!("postgresql@{pg_version}-main.service")) - .invoke(crate::ErrorKind::Database) - .await?; - if !exists { - Command::new("sudo") - .arg("-u") - .arg("postgres") - .arg("createuser") - .arg("root") - .invoke(crate::ErrorKind::Database) - .await?; - Command::new("sudo") - .arg("-u") - .arg("postgres") - .arg("createdb") - .arg("secrets") - .arg("-O") - .arg("root") - .invoke(crate::ErrorKind::Database) - .await?; - } - - Ok(()) -} - -pub struct InitResult { - pub secret_store: Pool, - pub db: patch_db::PatchDb, -} - -#[instrument(skip_all)] -pub async fn init(cfg: &RpcContextConfig) -> Result { - tokio::fs::create_dir_all("/run/embassy") - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "mkdir -p /run/embassy"))?; - if tokio::fs::metadata(LOCAL_AUTH_COOKIE_PATH).await.is_err() { - tokio::fs::write( - LOCAL_AUTH_COOKIE_PATH, - base64::encode(random::<[u8; 32]>()).as_bytes(), - ) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("write {}", LOCAL_AUTH_COOKIE_PATH), - ) - })?; - tokio::fs::set_permissions(LOCAL_AUTH_COOKIE_PATH, Permissions::from_mode(0o046)).await?; - Command::new("chown") - .arg("root:embassy") - .arg(LOCAL_AUTH_COOKIE_PATH) - .invoke(crate::ErrorKind::Filesystem) - .await?; - } - - let secret_store = cfg.secret_store().await?; - tracing::info!("Opened Postgres"); - - crate::ssh::sync_keys_from_db(&secret_store, "/home/start9/.ssh/authorized_keys").await?; - tracing::info!("Synced SSH Keys"); - - let account = AccountInfo::load(&secret_store).await?; - let db = cfg.db(&account).await?; - tracing::info!("Opened PatchDB"); - let peek = db.peek().await; - let mut server_info = peek.as_server_info().de()?; - - // write to ca cert store - tokio::fs::write( - "/usr/local/share/ca-certificates/startos-root-ca.crt", - account.root_ca_cert.to_pem()?, - ) - .await?; - Command::new("update-ca-certificates") - .invoke(crate::ErrorKind::OpenSsl) - .await?; - - if let Some(wifi_interface) = &cfg.wifi_interface { - crate::net::wifi::synchronize_wpa_supplicant_conf( - &cfg.datadir().join("main"), - wifi_interface, - &server_info.last_wifi_region, - ) - .await?; - tracing::info!("Synchronized WiFi"); - } - - let should_rebuild = tokio::fs::metadata(SYSTEM_REBUILD_PATH).await.is_ok() - || &*server_info.version < &emver::Version::new(0, 3, 2, 0) - || (*ARCH == "x86_64" && &*server_info.version < &emver::Version::new(0, 3, 4, 0)); - - let log_dir = cfg.datadir().join("main/logs"); - if tokio::fs::metadata(&log_dir).await.is_err() { - tokio::fs::create_dir_all(&log_dir).await?; - } - let current_machine_id = tokio::fs::read_to_string("/etc/machine-id").await?; - let mut machine_ids = tokio::fs::read_dir(&log_dir).await?; - while let Some(machine_id) = machine_ids.next_entry().await? { - if machine_id.file_name().to_string_lossy().trim() != current_machine_id.trim() { - tokio::fs::remove_dir_all(machine_id.path()).await?; - } - } - crate::disk::mount::util::bind(&log_dir, "/var/log/journal", false).await?; - match Command::new("chattr") - .arg("-R") - .arg("+C") - .arg("/var/log/journal") - .invoke(ErrorKind::Filesystem) - .await - { - Ok(_) => Ok(()), - Err(e) if e.source.to_string().contains("Operation not supported") => Ok(()), - Err(e) => Err(e), - }?; - Command::new("systemctl") - .arg("restart") - .arg("systemd-journald") - .invoke(crate::ErrorKind::Journald) - .await?; - tracing::info!("Mounted Logs"); - - let tmp_dir = cfg.datadir().join("package-data/tmp"); - if should_rebuild && tokio::fs::metadata(&tmp_dir).await.is_ok() { - tokio::fs::remove_dir_all(&tmp_dir).await?; - } - if tokio::fs::metadata(&tmp_dir).await.is_err() { - tokio::fs::create_dir_all(&tmp_dir).await?; - } - let tmp_var = cfg.datadir().join(format!("package-data/tmp/var")); - if tokio::fs::metadata(&tmp_var).await.is_ok() { - tokio::fs::remove_dir_all(&tmp_var).await?; - } - crate::disk::mount::util::bind(&tmp_var, "/var/tmp", false).await?; - let tmp_docker = cfg - .datadir() - .join(format!("package-data/tmp/{CONTAINER_TOOL}")); - let tmp_docker_exists = tokio::fs::metadata(&tmp_docker).await.is_ok(); - if CONTAINER_TOOL == "docker" { - Command::new("systemctl") - .arg("stop") - .arg("docker") - .invoke(crate::ErrorKind::Docker) - .await?; - } - crate::disk::mount::util::bind(&tmp_docker, CONTAINER_DATADIR, false).await?; - - if CONTAINER_TOOL == "docker" { - Command::new("systemctl") - .arg("reset-failed") - .arg("docker") - .invoke(crate::ErrorKind::Docker) - .await?; - Command::new("systemctl") - .arg("start") - .arg("docker") - .invoke(crate::ErrorKind::Docker) - .await?; - } - tracing::info!("Mounted Docker Data"); - - if should_rebuild || !tmp_docker_exists { - if CONTAINER_TOOL == "docker" { - tracing::info!("Creating Docker Network"); - create_bridge_network("start9", "172.18.0.1/24", "br-start9").await?; - tracing::info!("Created Docker Network"); - } - - let datadir = cfg.datadir(); - tracing::info!("Loading System Docker Images"); - crate::install::rebuild_from("/usr/lib/startos/system-images", &datadir).await?; - tracing::info!("Loaded System Docker Images"); - - tracing::info!("Loading Package Docker Images"); - crate::install::rebuild_from(datadir.join(PKG_ARCHIVE_DIR), &datadir).await?; - tracing::info!("Loaded Package Docker Images"); - } - - if CONTAINER_TOOL == "podman" { - crate::util::docker::remove_container("netdummy", true).await?; - Command::new("podman") - .arg("run") - .arg("-d") - .arg("--rm") - .arg("--init") - .arg("--network=start9") - .arg("--name=netdummy") - .arg("start9/x_system/utils:latest") - .arg("sleep") - .arg("infinity") - .invoke(crate::ErrorKind::Docker) - .await?; - } - - tracing::info!("Enabling Docker QEMU Emulation"); - Command::new(CONTAINER_TOOL) - .arg("run") - .arg("--privileged") - .arg("--rm") - .arg("start9/x_system/binfmt") - .arg("--install") - .arg("all") - .invoke(crate::ErrorKind::Docker) - .await?; - tracing::info!("Enabled Docker QEMU Emulation"); - - let governor = if let Some(governor) = &server_info.governor { - if get_available_governors().await?.contains(governor) { - Some(governor) - } else { - tracing::warn!("CPU Governor \"{governor}\" Not Available"); - None - } - } else { - get_preferred_governor().await? - }; - if let Some(governor) = governor { - tracing::info!("Setting CPU Governor to \"{governor}\""); - set_governor(governor).await?; - tracing::info!("Set CPU Governor"); - } - - server_info.ntp_synced = false; - let mut not_made_progress = 0u32; - for _ in 0..1800 { - if check_time_is_synchronized().await? { - server_info.ntp_synced = true; - break; - } - let t = SystemTime::now(); - tokio::time::sleep(Duration::from_secs(1)).await; - if t.elapsed() - .map(|t| t > Duration::from_secs_f64(1.1)) - .unwrap_or(true) - { - not_made_progress = 0; - } else { - not_made_progress += 1; - } - if not_made_progress > 30 { - break; - } - } - if !server_info.ntp_synced { - tracing::warn!("Timed out waiting for system time to synchronize"); - } else { - tracing::info!("Syncronized system clock"); - } - - if server_info.zram { - crate::system::enable_zram().await? - } - server_info.ip_info = crate::net::dhcp::init_ips().await?; - server_info.status_info = ServerStatus { - updated: false, - update_progress: None, - backup_progress: None, - shutting_down: false, - restarting: false, - }; - - db.mutate(|v| { - v.as_server_info_mut().ser(&server_info)?; - Ok(()) - }) - .await?; - - crate::version::init(&db, &secret_store).await?; - - db.mutate(|d| { - let model = d.de()?; - d.ser(&model) - }) - .await?; - - if should_rebuild { - match tokio::fs::remove_file(SYSTEM_REBUILD_PATH).await { - Ok(()) => Ok(()), - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), - Err(e) => Err(e), - }?; - } - - tracing::info!("System initialized."); - - Ok(InitResult { secret_store, db }) -} diff --git a/core/startos/src/inspect.rs b/core/startos/src/inspect.rs deleted file mode 100644 index cd27bbb2..00000000 --- a/core/startos/src/inspect.rs +++ /dev/null @@ -1,92 +0,0 @@ -use std::path::PathBuf; - -use rpc_toolkit::command; - -use crate::s9pk::manifest::Manifest; -use crate::s9pk::reader::S9pkReader; -use crate::util::display_none; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::Error; - -#[command(subcommands(hash, manifest, license, icon, instructions, docker_images))] -pub fn inspect() -> Result<(), Error> { - Ok(()) -} - -#[command(cli_only)] -pub async fn hash(#[arg] path: PathBuf) -> Result { - Ok(S9pkReader::open(path, true) - .await? - .hash_str() - .unwrap() - .to_owned()) -} - -#[command(cli_only, display(display_serializable))] -pub async fn manifest( - #[arg] path: PathBuf, - #[arg(rename = "no-verify", long = "no-verify")] no_verify: bool, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - S9pkReader::open(path, !no_verify).await?.manifest().await -} - -#[command(cli_only, display(display_none))] -pub async fn license( - #[arg] path: PathBuf, - #[arg(rename = "no-verify", long = "no-verify")] no_verify: bool, -) -> Result<(), Error> { - tokio::io::copy( - &mut S9pkReader::open(path, !no_verify).await?.license().await?, - &mut tokio::io::stdout(), - ) - .await?; - Ok(()) -} - -#[command(cli_only, display(display_none))] -pub async fn icon( - #[arg] path: PathBuf, - #[arg(rename = "no-verify", long = "no-verify")] no_verify: bool, -) -> Result<(), Error> { - tokio::io::copy( - &mut S9pkReader::open(path, !no_verify).await?.icon().await?, - &mut tokio::io::stdout(), - ) - .await?; - Ok(()) -} - -#[command(cli_only, display(display_none))] -pub async fn instructions( - #[arg] path: PathBuf, - #[arg(rename = "no-verify", long = "no-verify")] no_verify: bool, -) -> Result<(), Error> { - tokio::io::copy( - &mut S9pkReader::open(path, !no_verify) - .await? - .instructions() - .await?, - &mut tokio::io::stdout(), - ) - .await?; - Ok(()) -} - -#[command(cli_only, display(display_none), rename = "docker-images")] -pub async fn docker_images( - #[arg] path: PathBuf, - #[arg(rename = "no-verify", long = "no-verify")] no_verify: bool, -) -> Result<(), Error> { - tokio::io::copy( - &mut S9pkReader::open(path, !no_verify) - .await? - .docker_images() - .await?, - &mut tokio::io::stdout(), - ) - .await?; - Ok(()) -} diff --git a/core/startos/src/install/cleanup.rs b/core/startos/src/install/cleanup.rs deleted file mode 100644 index d90ec502..00000000 --- a/core/startos/src/install/cleanup.rs +++ /dev/null @@ -1,241 +0,0 @@ -use std::path::PathBuf; -use std::sync::Arc; - -use models::OptionExt; -use sqlx::{Executor, Postgres}; -use tracing::instrument; - -use super::PKG_ARCHIVE_DIR; -use crate::context::RpcContext; -use crate::db::model::{ - CurrentDependencies, Database, PackageDataEntry, PackageDataEntryInstalled, - PackageDataEntryMatchModelRef, -}; -use crate::error::ErrorCollection; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::{Apply, Version}; -use crate::volume::{asset_dir, script_dir}; -use crate::Error; - -#[instrument(skip_all)] -pub async fn cleanup(ctx: &RpcContext, id: &PackageId, version: &Version) -> Result<(), Error> { - let mut errors = ErrorCollection::new(); - ctx.managers.remove(&(id.clone(), version.clone())).await; - // docker images start9/$APP_ID/*:$VERSION -q | xargs docker rmi - let images = crate::util::docker::images_for(id, version).await?; - errors.extend( - futures::future::join_all(images.into_iter().map(|sha| async { - let sha = sha; // move into future - crate::util::docker::remove_image(&sha).await - })) - .await, - ); - let pkg_archive_dir = ctx - .datadir - .join(PKG_ARCHIVE_DIR) - .join(id) - .join(version.as_str()); - if tokio::fs::metadata(&pkg_archive_dir).await.is_ok() { - tokio::fs::remove_dir_all(&pkg_archive_dir) - .await - .apply(|res| errors.handle(res)); - } - let assets_path = asset_dir(&ctx.datadir, id, version); - if tokio::fs::metadata(&assets_path).await.is_ok() { - tokio::fs::remove_dir_all(&assets_path) - .await - .apply(|res| errors.handle(res)); - } - let scripts_path = script_dir(&ctx.datadir, id, version); - if tokio::fs::metadata(&scripts_path).await.is_ok() { - tokio::fs::remove_dir_all(&scripts_path) - .await - .apply(|res| errors.handle(res)); - } - - errors.into_result() -} - -#[instrument(skip_all)] -pub async fn cleanup_failed(ctx: &RpcContext, id: &PackageId) -> Result<(), Error> { - if let Some(version) = match ctx - .db - .peek() - .await - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .as_match() - { - PackageDataEntryMatchModelRef::Installing(m) => Some(m.as_manifest().as_version().de()?), - PackageDataEntryMatchModelRef::Restoring(m) => Some(m.as_manifest().as_version().de()?), - PackageDataEntryMatchModelRef::Updating(m) => { - let manifest_version = m.as_manifest().as_version().de()?; - let installed = m.as_installed().as_manifest().as_version().de()?; - if manifest_version != installed { - Some(manifest_version) - } else { - None // do not remove existing data - } - } - _ => { - tracing::warn!("{}: Nothing to clean up!", id); - None - } - } { - cleanup(ctx, id, &version).await?; - } - - ctx.db - .mutate(|v| { - match v - .clone() - .into_package_data() - .into_idx(id) - .or_not_found(id)? - .as_match() - { - PackageDataEntryMatchModelRef::Installing(_) - | PackageDataEntryMatchModelRef::Restoring(_) => { - v.as_package_data_mut().remove(id)?; - } - PackageDataEntryMatchModelRef::Updating(pde) => { - v.as_package_data_mut() - .as_idx_mut(id) - .or_not_found(id)? - .ser(&PackageDataEntry::Installed(PackageDataEntryInstalled { - manifest: pde.as_installed().as_manifest().de()?, - static_files: pde.as_static_files().de()?, - installed: pde.as_installed().de()?, - }))?; - } - _ => (), - } - Ok(()) - }) - .await -} - -#[instrument(skip_all)] -pub fn remove_from_current_dependents_lists( - db: &mut Model, - id: &PackageId, - current_dependencies: &CurrentDependencies, -) -> Result<(), Error> { - for dep in current_dependencies.0.keys().chain(std::iter::once(id)) { - if let Some(current_dependents) = db - .as_package_data_mut() - .as_idx_mut(dep) - .and_then(|d| d.as_installed_mut()) - .map(|i| i.as_current_dependents_mut()) - { - current_dependents.remove(id)?; - } - } - Ok(()) -} - -#[instrument(skip_all)] -pub async fn uninstall(ctx: &RpcContext, secrets: &mut Ex, id: &PackageId) -> Result<(), Error> -where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, -{ - let db = ctx.db.peek().await; - let entry = db - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .expect_as_removing()?; - - let dependents_paths: Vec = entry - .as_removing() - .as_current_dependents() - .keys()? - .into_iter() - .filter(|x| x != id) - .flat_map(|x| db.as_package_data().as_idx(&x)) - .flat_map(|x| x.as_installed()) - .flat_map(|x| x.as_manifest().as_volumes().de()) - .flat_map(|x| x.values().cloned().collect::>()) - .flat_map(|x| x.pointer_path(&ctx.datadir)) - .collect(); - - let volume_dir = ctx - .datadir - .join(crate::volume::PKG_VOLUME_DIR) - .join(&*entry.as_manifest().as_id().de()?); - let version = entry.as_removing().as_manifest().as_version().de()?; - tracing::debug!( - "Cleaning up {:?} except for {:?}", - volume_dir, - dependents_paths - ); - cleanup(ctx, id, &version).await?; - cleanup_folder(volume_dir, Arc::new(dependents_paths)).await; - remove_network_keys(secrets, id).await?; - - ctx.db - .mutate(|d| { - d.as_package_data_mut().remove(id)?; - remove_from_current_dependents_lists( - d, - id, - &entry.as_removing().as_current_dependencies().de()?, - ) - }) - .await -} - -#[instrument(skip_all)] -pub async fn remove_network_keys(secrets: &mut Ex, id: &PackageId) -> Result<(), Error> -where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, -{ - sqlx::query!("DELETE FROM network_keys WHERE package = $1", &*id) - .execute(&mut *secrets) - .await?; - sqlx::query!("DELETE FROM tor WHERE package = $1", &*id) - .execute(&mut *secrets) - .await?; - Ok(()) -} - -/// Needed to remove, without removing the folders that are mounted in the other docker containers -pub fn cleanup_folder( - path: PathBuf, - dependents_volumes: Arc>, -) -> futures::future::BoxFuture<'static, ()> { - Box::pin(async move { - let meta_data = match tokio::fs::metadata(&path).await { - Ok(a) => a, - Err(_e) => { - return; - } - }; - if !meta_data.is_dir() { - tracing::error!("is_not dir, remove {:?}", path); - let _ = tokio::fs::remove_file(&path).await; - return; - } - if !dependents_volumes - .iter() - .any(|v| v.starts_with(&path) || v == &path) - { - tracing::error!("No parents, remove {:?}", path); - let _ = tokio::fs::remove_dir_all(&path).await; - return; - } - let mut read_dir = match tokio::fs::read_dir(&path).await { - Ok(a) => a, - Err(_e) => { - return; - } - }; - tracing::error!("Parents, recurse {:?}", path); - while let Some(entry) = read_dir.next_entry().await.ok().flatten() { - let entry_path = entry.path(); - cleanup_folder(entry_path, dependents_volumes.clone()).await; - } - }) -} diff --git a/core/startos/src/install/mod.rs b/core/startos/src/install/mod.rs deleted file mode 100644 index 01f405e7..00000000 --- a/core/startos/src/install/mod.rs +++ /dev/null @@ -1,1317 +0,0 @@ -use std::collections::BTreeMap; -use std::io::SeekFrom; -use std::marker::PhantomData; -use std::path::{Path, PathBuf}; -use std::sync::atomic::Ordering; -use std::sync::Arc; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use emver::VersionRange; -use futures::future::BoxFuture; -use futures::{FutureExt, StreamExt, TryStreamExt}; -use http::header::CONTENT_LENGTH; -use http::{Request, Response, StatusCode}; -use hyper::Body; -use models::{mime, DataUrl}; -use reqwest::Url; -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; -use serde_json::{json, Value}; -use tokio::fs::{File, OpenOptions}; -use tokio::io::{AsyncRead, AsyncSeek, AsyncSeekExt, AsyncWriteExt}; -use tokio::process::Command; -use tokio::sync::oneshot; -use tokio_stream::wrappers::ReadDirStream; -use tracing::instrument; - -use self::cleanup::{cleanup_failed, remove_from_current_dependents_lists}; -use crate::config::ConfigureContext; -use crate::context::{CliContext, RpcContext}; -use crate::core::rpc_continuations::{RequestGuid, RpcContinuation}; -use crate::db::model::{ - CurrentDependencies, CurrentDependencyInfo, CurrentDependents, InstalledPackageInfo, - PackageDataEntry, PackageDataEntryInstalled, PackageDataEntryInstalling, - PackageDataEntryMatchModelRef, PackageDataEntryRemoving, PackageDataEntryRestoring, - PackageDataEntryUpdating, StaticDependencyInfo, StaticFiles, -}; -use crate::dependencies::{ - add_dependent_to_current_dependents_lists, compute_dependency_config_errs, - set_dependents_with_live_pointers_to_needs_config, -}; -use crate::install::cleanup::cleanup; -use crate::install::progress::{InstallProgress, InstallProgressTracker}; -use crate::notifications::NotificationLevel; -use crate::prelude::*; -use crate::registry::marketplace::with_query_params; -use crate::s9pk::manifest::{Manifest, PackageId}; -use crate::s9pk::reader::S9pkReader; -use crate::status::{MainStatus, Status}; -use crate::util::docker::CONTAINER_TOOL; -use crate::util::io::response_to_reader; -use crate::util::serde::{display_serializable, Port}; -use crate::util::{display_none, AsyncFileExt, Invoke, Version}; -use crate::volume::{asset_dir, script_dir}; -use crate::{Error, ErrorKind, ResultExt}; - -pub mod cleanup; -pub mod progress; - -pub const PKG_ARCHIVE_DIR: &str = "package-data/archive"; -pub const PKG_PUBLIC_DIR: &str = "package-data/public"; -pub const PKG_WASM_DIR: &str = "package-data/wasm"; - -#[command(display(display_serializable))] -pub async fn list(#[context] ctx: RpcContext) -> Result { - Ok(ctx.db.peek().await.as_package_data().as_entries()? - .iter() - .filter_map(|(id, pde)| { - let status = match pde.as_match() { - PackageDataEntryMatchModelRef::Installed(_) => { - "installed" - } - PackageDataEntryMatchModelRef::Installing(_) => { - "installing" - } - PackageDataEntryMatchModelRef::Updating(_) => { - "updating" - } - PackageDataEntryMatchModelRef::Restoring(_) => { - "restoring" - } - PackageDataEntryMatchModelRef::Removing(_) => { - "removing" - } - PackageDataEntryMatchModelRef::Error(_) => { - "error" - } - }; - serde_json::to_value(json!({ "status":status, "id": id.clone(), "version": pde.as_manifest().as_version().de().ok()?})) - .ok() - }) - .collect()) -} - -#[derive(Debug, Clone, Copy, serde::Deserialize, serde::Serialize)] -#[serde(rename_all = "kebab-case")] -pub enum MinMax { - Min, - Max, -} -impl Default for MinMax { - fn default() -> Self { - MinMax::Max - } -} -impl std::str::FromStr for MinMax { - type Err = Error; - fn from_str(s: &str) -> Result { - match s { - "min" => Ok(MinMax::Min), - "max" => Ok(MinMax::Max), - _ => Err(Error::new( - eyre!("Must be one of \"min\", \"max\"."), - crate::ErrorKind::ParseVersion, - )), - } - } -} -impl std::fmt::Display for MinMax { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - MinMax::Min => write!(f, "min"), - MinMax::Max => write!(f, "max"), - } - } -} - -#[command( - custom_cli(cli_install(async, context(CliContext))), - display(display_none), - metadata(sync_db = true) -)] -#[instrument(skip_all)] -pub async fn install( - #[context] ctx: RpcContext, - #[arg] id: String, - #[arg(short = 'm', long = "marketplace-url", rename = "marketplace-url")] - marketplace_url: Option, - #[arg(short = 'v', long = "version-spec", rename = "version-spec")] version_spec: Option< - String, - >, - #[arg(long = "version-priority", rename = "version-priority")] version_priority: Option, -) -> Result<(), Error> { - let version_str = match &version_spec { - None => "*", - Some(v) => &*v, - }; - let version: VersionRange = version_str.parse()?; - let marketplace_url = - marketplace_url.unwrap_or_else(|| crate::DEFAULT_MARKETPLACE.parse().unwrap()); - let version_priority = version_priority.unwrap_or_default(); - let man: Manifest = ctx - .client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/manifest/{}?spec={}&version-priority={}", - marketplace_url, id, version, version_priority, - ) - .parse()?, - )) - .send() - .await - .with_kind(crate::ErrorKind::Registry)? - .error_for_status() - .with_kind(crate::ErrorKind::Registry)? - .json() - .await - .with_kind(crate::ErrorKind::Registry)?; - let s9pk = ctx - .client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/{}.s9pk?spec=={}&version-priority={}", - marketplace_url, id, man.version, version_priority, - ) - .parse()?, - )) - .send() - .await - .with_kind(crate::ErrorKind::Registry)? - .error_for_status()?; - - if *man.id != *id || !man.version.satisfies(&version) { - return Err(Error::new( - eyre!("Fetched package does not match requested id and version"), - ErrorKind::Registry, - )); - } - - let public_dir_path = ctx - .datadir - .join(PKG_PUBLIC_DIR) - .join(&man.id) - .join(man.version.as_str()); - tokio::fs::create_dir_all(&public_dir_path).await?; - - let icon_type = man.assets.icon_type(); - let (license_res, instructions_res, icon_res) = tokio::join!( - async { - tokio::io::copy( - &mut response_to_reader( - ctx.client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/license/{}?spec=={}", - marketplace_url, id, man.version, - ) - .parse()?, - )) - .send() - .await? - .error_for_status()?, - ), - &mut File::create(public_dir_path.join("LICENSE.md")).await?, - ) - .await?; - Ok::<_, color_eyre::eyre::Report>(()) - }, - async { - tokio::io::copy( - &mut response_to_reader( - ctx.client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/instructions/{}?spec=={}", - marketplace_url, id, man.version, - ) - .parse()?, - )) - .send() - .await? - .error_for_status()?, - ), - &mut File::create(public_dir_path.join("INSTRUCTIONS.md")).await?, - ) - .await?; - Ok::<_, color_eyre::eyre::Report>(()) - }, - async { - tokio::io::copy( - &mut response_to_reader( - ctx.client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/icon/{}?spec=={}", - marketplace_url, id, man.version, - ) - .parse()?, - )) - .send() - .await? - .error_for_status()?, - ), - &mut File::create(public_dir_path.join(format!("icon.{}", icon_type))).await?, - ) - .await?; - Ok::<_, color_eyre::eyre::Report>(()) - }, - ); - if let Err(e) = license_res { - tracing::warn!("Failed to pre-download license: {}", e); - } - if let Err(e) = instructions_res { - tracing::warn!("Failed to pre-download instructions: {}", e); - } - if let Err(e) = icon_res { - tracing::warn!("Failed to pre-download icon: {}", e); - } - - let progress = Arc::new(InstallProgress::new(s9pk.content_length())); - let static_files = StaticFiles::local(&man.id, &man.version, icon_type); - ctx.db - .mutate(|db| { - let pde = match db - .as_package_data() - .as_idx(&man.id) - .map(|x| x.de()) - .transpose()? - { - Some(PackageDataEntry::Installed(PackageDataEntryInstalled { - installed, - static_files, - .. - })) => PackageDataEntry::Updating(PackageDataEntryUpdating { - install_progress: progress.clone(), - static_files, - installed, - manifest: man.clone(), - }), - None => PackageDataEntry::Installing(PackageDataEntryInstalling { - install_progress: progress.clone(), - static_files, - manifest: man.clone(), - }), - _ => { - return Err(Error::new( - eyre!("Cannot install over a package in a transient state"), - crate::ErrorKind::InvalidRequest, - )) - } - }; - db.as_package_data_mut().insert(&man.id, &pde) - }) - .await?; - - let downloading = download_install_s9pk( - ctx.clone(), - man.clone(), - Some(marketplace_url), - Arc::new(InstallProgress::new(s9pk.content_length())), - response_to_reader(s9pk), - None, - ); - tokio::spawn(async move { - if let Err(e) = downloading.await { - let err_str = format!("Install of {}@{} Failed: {}", man.id, man.version, e); - tracing::error!("{}", err_str); - tracing::debug!("{:?}", e); - if let Err(e) = ctx - .notification_manager - .notify( - ctx.db.clone(), - Some(man.id), - NotificationLevel::Error, - String::from("Install Failed"), - err_str, - (), - None, - ) - .await - { - tracing::error!("Failed to issue Notification: {}", e); - tracing::debug!("{:?}", e); - } - } - Ok::<_, String>(()) - }); - - Ok(()) -} -#[command(rpc_only, display(display_none))] -#[instrument(skip_all)] -pub async fn sideload( - #[context] ctx: RpcContext, - #[arg] manifest: Manifest, - #[arg] icon: Option, -) -> Result { - let new_ctx = ctx.clone(); - let guid = RequestGuid::new(); - if let Some(icon) = icon { - use tokio::io::AsyncWriteExt; - - let public_dir_path = ctx - .datadir - .join(PKG_PUBLIC_DIR) - .join(&manifest.id) - .join(manifest.version.as_str()); - tokio::fs::create_dir_all(&public_dir_path).await?; - - let invalid_data_url = - || Error::new(eyre!("Invalid Icon Data URL"), ErrorKind::InvalidRequest); - let data = icon - .strip_prefix(&format!( - "data:image/{};base64,", - manifest.assets.icon_type() - )) - .ok_or_else(&invalid_data_url)?; - let mut icon_file = - File::create(public_dir_path.join(format!("icon.{}", manifest.assets.icon_type()))) - .await?; - icon_file - .write_all(&base64::decode(data).with_kind(ErrorKind::InvalidRequest)?) - .await?; - icon_file.sync_all().await?; - } - - let handler = Box::new(|req: Request| { - async move { - let content_length = match req.headers().get(CONTENT_LENGTH).map(|a| a.to_str()) { - None => None, - Some(Err(_)) => { - return Response::builder() - .status(StatusCode::BAD_REQUEST) - .body(Body::from("Invalid Content Length")) - .with_kind(ErrorKind::Network) - } - Some(Ok(a)) => match a.parse::() { - Err(_) => { - return Response::builder() - .status(StatusCode::BAD_REQUEST) - .body(Body::from("Invalid Content Length")) - .with_kind(ErrorKind::Network) - } - Ok(a) => Some(a), - }, - }; - let progress = Arc::new(InstallProgress::new(content_length)); - let install_progress = progress.clone(); - - new_ctx - .db - .mutate(|db| { - let pde = match db - .as_package_data() - .as_idx(&manifest.id) - .map(|x| x.de()) - .transpose()? - { - Some(PackageDataEntry::Installed(PackageDataEntryInstalled { - installed, - static_files, - .. - })) => PackageDataEntry::Updating(PackageDataEntryUpdating { - install_progress, - installed, - manifest: manifest.clone(), - static_files, - }), - None => PackageDataEntry::Installing(PackageDataEntryInstalling { - install_progress, - static_files: StaticFiles::local( - &manifest.id, - &manifest.version, - &manifest.assets.icon_type(), - ), - manifest: manifest.clone(), - }), - _ => { - return Err(Error::new( - eyre!("Cannot install over a package in a transient state"), - crate::ErrorKind::InvalidRequest, - )) - } - }; - db.as_package_data_mut().insert(&manifest.id, &pde) - }) - .await?; - - let (send, recv) = oneshot::channel(); - - tokio::spawn(async move { - if let Err(e) = download_install_s9pk( - new_ctx.clone(), - manifest.clone(), - None, - progress, - tokio_util::io::StreamReader::new(req.into_body().map_err(|e| { - std::io::Error::new( - match &e { - e if e.is_connect() => std::io::ErrorKind::ConnectionRefused, - e if e.is_timeout() => std::io::ErrorKind::TimedOut, - _ => std::io::ErrorKind::Other, - }, - e, - ) - })), - Some(send), - ) - .await - { - let err_str = format!( - "Install of {}@{} Failed: {}", - manifest.id, manifest.version, e - ); - tracing::error!("{}", err_str); - tracing::debug!("{:?}", e); - if let Err(e) = new_ctx - .notification_manager - .notify( - new_ctx.db.clone(), - Some(manifest.id.clone()), - NotificationLevel::Error, - String::from("Install Failed"), - err_str, - (), - None, - ) - .await - { - tracing::error!("Failed to issue Notification: {}", e); - tracing::debug!("{:?}", e); - } - } - }); - - if let Ok(_) = recv.await { - Response::builder() - .status(StatusCode::OK) - .body(Body::empty()) - .with_kind(ErrorKind::Network) - } else { - Response::builder() - .status(StatusCode::INTERNAL_SERVER_ERROR) - .body(Body::from("installation aborted before upload completed")) - .with_kind(ErrorKind::Network) - } - } - .boxed() - }); - ctx.add_continuation( - guid.clone(), - RpcContinuation::rest(handler, Duration::from_secs(30)), - ) - .await; - Ok(guid) -} - -#[instrument(skip_all)] -async fn cli_install( - ctx: CliContext, - target: String, - marketplace_url: Option, - version_spec: Option, - version_priority: Option, -) -> Result<(), RpcError> { - if target.ends_with(".s9pk") { - let path = PathBuf::from(target); - - // inspect manifest no verify - let mut reader = S9pkReader::open(&path, false).await?; - let manifest = reader.manifest().await?; - let icon = reader.icon().await?.to_vec().await?; - let icon_str = format!( - "data:image/{};base64,{}", - manifest.assets.icon_type(), - base64::encode(&icon) - ); - - // rpc call remote sideload - tracing::debug!("calling package.sideload"); - let guid = rpc_toolkit::command_helpers::call_remote( - ctx.clone(), - "package.sideload", - serde_json::json!({ "manifest": manifest, "icon": icon_str }), - PhantomData::, - ) - .await? - .result?; - tracing::debug!("package.sideload succeeded {:?}", guid); - - // hit continuation api with guid that comes back - let file = tokio::fs::File::open(path).await?; - let content_length = file.metadata().await?.len(); - let body = Body::wrap_stream(tokio_util::io::ReaderStream::new(file)); - let res = ctx - .client - .post(format!("{}rest/rpc/{}", ctx.base_url, guid,)) - .header(CONTENT_LENGTH, content_length) - .body(body) - .send() - .await?; - if res.status().as_u16() == 200 { - tracing::info!("Package Uploaded") - } else { - tracing::info!("Package Upload failed: {}", res.text().await?) - } - } else { - let params = match (target.split_once("@"), version_spec) { - (Some((pkg, v)), None) => { - serde_json::json!({ "id": pkg, "marketplace-url": marketplace_url, "version-spec": v, "version-priority": version_priority }) - } - (Some(_), Some(_)) => { - return Err(crate::Error::new( - eyre!("Invalid package id {}", target), - ErrorKind::InvalidRequest, - ) - .into()) - } - (None, Some(v)) => { - serde_json::json!({ "id": target, "marketplace-url": marketplace_url, "version-spec": v, "version-priority": version_priority }) - } - (None, None) => { - serde_json::json!({ "id": target, "marketplace-url": marketplace_url, "version-priority": version_priority }) - } - }; - tracing::debug!("calling package.install"); - rpc_toolkit::command_helpers::call_remote( - ctx, - "package.install", - params, - PhantomData::<()>, - ) - .await? - .result?; - tracing::debug!("package.install succeeded"); - } - Ok(()) -} - -#[command(display(display_none), metadata(sync_db = true))] -pub async fn uninstall( - #[context] ctx: RpcContext, - #[arg] id: PackageId, -) -> Result { - ctx.db - .mutate(|db| { - let (manifest, static_files, installed) = - match db.as_package_data().as_idx(&id).or_not_found(&id)?.de()? { - PackageDataEntry::Installed(PackageDataEntryInstalled { - manifest, - static_files, - installed, - }) => (manifest, static_files, installed), - _ => { - return Err(Error::new( - eyre!("Package is not installed."), - crate::ErrorKind::NotFound, - )); - } - }; - let pde = PackageDataEntry::Removing(PackageDataEntryRemoving { - manifest, - static_files, - removing: installed, - }); - db.as_package_data_mut().insert(&id, &pde) - }) - .await?; - - let return_id = id.clone(); - - tokio::spawn(async move { - if let Err(e) = async { - cleanup::uninstall(&ctx, ctx.secret_store.acquire().await?.as_mut(), &id).await - } - .await - { - let err_str = format!("Uninstall of {} Failed: {}", id, e); - tracing::error!("{}", err_str); - tracing::debug!("{:?}", e); - if let Err(e) = ctx - .notification_manager - .notify( - ctx.db.clone(), // allocating separate handle here because the lifetime of the previous one is the expression - Some(id), - NotificationLevel::Error, - String::from("Uninstall Failed"), - err_str, - (), - None, - ) - .await - { - tracing::error!("Failed to issue Notification: {}", e); - tracing::debug!("{:?}", e); - } - } - }); - - Ok(return_id) -} - -#[instrument(skip_all)] -pub async fn download_install_s9pk( - ctx: RpcContext, - temp_manifest: Manifest, - marketplace_url: Option, - progress: Arc, - mut s9pk: impl AsyncRead + Unpin, - download_complete: Option>, -) -> Result<(), Error> { - let pkg_id = &temp_manifest.id; - let version = &temp_manifest.version; - let db = ctx.db.peek().await; - - if let Result::<(), Error>::Err(e) = { - let ctx = ctx.clone(); - async move { - // // Build set of existing manifests - let mut manifests = Vec::new(); - for (_id, pkg) in db.as_package_data().as_entries()? { - let m = pkg.as_manifest().de()?; - manifests.push(m); - } - // Build map of current port -> ssl mappings - let port_map = ssl_port_status(&manifests); - tracing::info!("SSL Port Map: {:?}", &port_map); - - // if any of the requested interface lan configs conflict with current state, fail the install - for (_id, iface) in &temp_manifest.interfaces.0 { - if let Some(cfg) = &iface.lan_config { - for (p, lan) in cfg { - if p.0 == 80 && lan.ssl || p.0 == 443 && !lan.ssl { - return Err(Error::new( - eyre!("SSL Conflict with StartOS"), - ErrorKind::LanPortConflict, - )); - } - match port_map.get(&p) { - Some((ssl, pkg)) => { - if *ssl != lan.ssl { - return Err(Error::new( - eyre!("SSL Conflict with package: {}", pkg), - ErrorKind::LanPortConflict, - )); - } - } - None => { - continue; - } - } - } - } - } - - let pkg_archive_dir = ctx - .datadir - .join(PKG_ARCHIVE_DIR) - .join(pkg_id) - .join(version.as_str()); - tokio::fs::create_dir_all(&pkg_archive_dir).await?; - let pkg_archive = - pkg_archive_dir.join(AsRef::::as_ref(pkg_id).with_extension("s9pk")); - - File::delete(&pkg_archive).await?; - let mut dst = OpenOptions::new() - .create(true) - .write(true) - .read(true) - .open(&pkg_archive) - .await?; - - progress - .track_download_during(ctx.db.clone(), pkg_id, || async { - let mut progress_writer = - InstallProgressTracker::new(&mut dst, progress.clone()); - tokio::io::copy(&mut s9pk, &mut progress_writer).await?; - progress.download_complete(); - if let Some(complete) = download_complete { - complete.send(()).unwrap_or_default(); - } - Ok(()) - }) - .await?; - - dst.seek(SeekFrom::Start(0)).await?; - - let progress_reader = InstallProgressTracker::new(dst, progress.clone()); - let mut s9pk_reader = progress - .track_read_during(ctx.db.clone(), pkg_id, || { - S9pkReader::from_reader(progress_reader, true) - }) - .await?; - - install_s9pk( - ctx.clone(), - pkg_id, - version, - marketplace_url, - &mut s9pk_reader, - progress, - ) - .await?; - - Ok(()) - } - } - .await - { - if let Err(e) = cleanup_failed(&ctx, pkg_id).await { - tracing::error!("Failed to clean up {}@{}: {}", pkg_id, version, e); - tracing::debug!("{:?}", e); - } - - Err(e) - } else { - Ok::<_, Error>(()) - } -} - -#[instrument(skip_all)] -pub async fn install_s9pk( - ctx: RpcContext, - pkg_id: &PackageId, - version: &Version, - marketplace_url: Option, - rdr: &mut S9pkReader>, - progress: Arc, -) -> Result<(), Error> { - rdr.validate().await?; - rdr.validated(); - let developer_key = rdr.developer_key().clone(); - rdr.reset().await?; - let db = ctx.db.peek().await; - - tracing::info!("Install {}@{}: Unpacking Manifest", pkg_id, version); - let manifest = progress - .track_read_during(ctx.db.clone(), pkg_id, || rdr.manifest()) - .await?; - tracing::info!("Install {}@{}: Unpacked Manifest", pkg_id, version); - - tracing::info!("Install {}@{}: Fetching Dependency Info", pkg_id, version); - let mut dependency_info = BTreeMap::new(); - for (dep, info) in &manifest.dependencies.0 { - let manifest: Option = if let Some(local_man) = db - .as_package_data() - .as_idx(dep) - .map(|pde| pde.as_manifest().de()) - { - Some(local_man?) - } else if let Some(marketplace_url) = &marketplace_url { - match ctx - .client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/manifest/{}?spec={}", - marketplace_url, dep, info.version, - ) - .parse()?, - )) - .send() - .await - .with_kind(crate::ErrorKind::Registry)? - .error_for_status() - { - Ok(a) => Ok(Some( - a.json() - .await - .with_kind(crate::ErrorKind::Deserialization)?, - )), - Err(e) - if e.status() == Some(StatusCode::BAD_REQUEST) - || e.status() == Some(StatusCode::NOT_FOUND) => - { - Ok(None) - } - Err(e) => Err(e), - } - .with_kind(crate::ErrorKind::Registry)? - } else { - None - }; - - let icon_path = if let Some(manifest) = &manifest { - let dir = ctx - .datadir - .join(PKG_PUBLIC_DIR) - .join(&manifest.id) - .join(manifest.version.as_str()); - let icon_path = dir.join(format!("icon.{}", manifest.assets.icon_type())); - if tokio::fs::metadata(&icon_path).await.is_err() { - if let Some(marketplace_url) = &marketplace_url { - tokio::fs::create_dir_all(&dir).await?; - let icon = ctx - .client - .get(with_query_params( - ctx.clone(), - format!( - "{}/package/v0/icon/{}?spec={}", - marketplace_url, dep, info.version, - ) - .parse()?, - )) - .send() - .await - .with_kind(crate::ErrorKind::Registry)?; - let mut dst = File::create(&icon_path).await?; - tokio::io::copy(&mut response_to_reader(icon), &mut dst).await?; - dst.sync_all().await?; - Some(icon_path) - } else { - None - } - } else { - Some(icon_path) - } - } else { - None - }; - - dependency_info.insert( - dep.clone(), - StaticDependencyInfo { - title: manifest - .as_ref() - .map(|x| x.title.clone()) - .unwrap_or_else(|| dep.to_string()), - icon: if let Some(icon_path) = &icon_path { - DataUrl::from_path(icon_path).await? - } else { - DataUrl::from_slice("image/png", include_bytes!("./package-icon.png")) - }, - }, - ); - } - tracing::info!("Install {}@{}: Fetched Dependency Info", pkg_id, version); - - let icon = progress - .track_read_during(ctx.db.clone(), pkg_id, || { - unpack_s9pk(&ctx.datadir, &manifest, rdr) - }) - .await?; - - progress.unpack_complete.store(true, Ordering::SeqCst); - - progress - .track_read( - ctx.db.clone(), - pkg_id.clone(), - Arc::new(::std::sync::atomic::AtomicBool::new(true)), - ) - .await?; - - let peek = ctx.db.peek().await; - let prev = peek - .as_package_data() - .as_idx(pkg_id) - .or_not_found(pkg_id)? - .de()?; - let mut sql_tx = ctx.secret_store.begin().await?; - - tracing::info!("Install {}@{}: Creating volumes", pkg_id, version); - manifest.volumes.install(&ctx, pkg_id, version).await?; - tracing::info!("Install {}@{}: Created volumes", pkg_id, version); - - tracing::info!("Install {}@{}: Installing interfaces", pkg_id, version); - let interface_addresses = manifest.interfaces.install(sql_tx.as_mut(), pkg_id).await?; - tracing::info!( - "Install {}@{}: Installed interfaces {:?}", - pkg_id, - version, - interface_addresses - ); - - tracing::info!("Install {}@{}: Creating manager", pkg_id, version); - let manager = ctx.managers.add(ctx.clone(), manifest.clone()).await?; - tracing::info!("Install {}@{}: Created manager", pkg_id, version); - - let static_files = StaticFiles::local(pkg_id, version, manifest.assets.icon_type()); - let current_dependencies: CurrentDependencies = CurrentDependencies( - manifest - .dependencies - .0 - .iter() - .filter_map(|(id, info)| { - if info.requirement.required() { - Some((id.clone(), CurrentDependencyInfo::default())) - } else { - None - } - }) - .collect(), - ); - let mut dependents_static_dependency_info = BTreeMap::new(); - let current_dependents = { - let mut deps = BTreeMap::new(); - for package in db.as_package_data().keys()? { - if db - .as_package_data() - .as_idx(&package) - .or_not_found(&package)? - .as_installed() - .and_then(|i| i.as_dependency_info().as_idx(&pkg_id)) - .is_some() - { - dependents_static_dependency_info.insert(package.clone(), icon.clone()); - } - if let Some(dep) = db - .as_package_data() - .as_idx(&package) - .or_not_found(&package)? - .as_installed() - .and_then(|i| i.as_current_dependencies().as_idx(pkg_id)) - { - deps.insert(package, dep.de()?); - } - } - - CurrentDependents(deps) - }; - - let installed = InstalledPackageInfo { - status: Status { - configured: manifest.config.is_none(), - main: MainStatus::Stopped, - dependency_config_errors: compute_dependency_config_errs( - &ctx, - &peek, - &manifest, - ¤t_dependencies, - &Default::default(), - ) - .await?, - }, - marketplace_url, - developer_key, - manifest: manifest.clone(), - last_backup: match prev { - PackageDataEntry::Updating(PackageDataEntryUpdating { - installed: - InstalledPackageInfo { - last_backup: Some(time), - .. - }, - .. - }) => Some(time), - _ => None, - }, - dependency_info, - current_dependents: current_dependents.clone(), - current_dependencies: current_dependencies.clone(), - interface_addresses, - }; - let mut next = PackageDataEntryInstalled { - installed, - manifest: manifest.clone(), - static_files, - }; - - let mut auto_start = false; - let mut configured = false; - - let mut to_cleanup = None; - - if let PackageDataEntry::Updating(PackageDataEntryUpdating { - installed: prev, .. - }) = &prev - { - let prev_is_configured = prev.status.configured; - let prev_migration = prev - .manifest - .migrations - .to( - &ctx, - version, - pkg_id, - &prev.manifest.version, - &prev.manifest.volumes, - ) - .map(futures::future::Either::Left); - let migration = manifest - .migrations - .from( - &manifest.containers, - &ctx, - &prev.manifest.version, - pkg_id, - version, - &manifest.volumes, - ) - .map(futures::future::Either::Right); - - let viable_migration = if prev.manifest.version > manifest.version { - prev_migration.or(migration) - } else { - migration.or(prev_migration) - }; - - if let Some(f) = viable_migration { - configured = f.await?.configured && prev_is_configured; - } - if configured || manifest.config.is_none() { - auto_start = prev.status.main.running(); - } - if &prev.manifest.version != version { - to_cleanup = Some((prev.manifest.id.clone(), prev.manifest.version.clone())); - } - } else if let PackageDataEntry::Restoring(PackageDataEntryRestoring { .. }) = prev { - next.installed.marketplace_url = manifest - .backup - .restore(&ctx, pkg_id, version, &manifest.volumes) - .await?; - } - - sql_tx.commit().await?; - - let to_configure = ctx - .db - .mutate(|db| { - for (package, icon) in dependents_static_dependency_info { - db.as_package_data_mut() - .as_idx_mut(&package) - .or_not_found(&package)? - .as_installed_mut() - .or_not_found(&package)? - .as_dependency_info_mut() - .insert( - &pkg_id, - &StaticDependencyInfo { - icon, - title: manifest.title.clone(), - }, - )?; - } - db.as_package_data_mut() - .insert(&pkg_id, &PackageDataEntry::Installed(next))?; - if let PackageDataEntry::Updating(PackageDataEntryUpdating { - installed: prev, .. - }) = &prev - { - remove_from_current_dependents_lists(db, pkg_id, &prev.current_dependencies)?; - } - add_dependent_to_current_dependents_lists(db, pkg_id, ¤t_dependencies)?; - - set_dependents_with_live_pointers_to_needs_config(db, pkg_id) - }) - .await?; - - if let Some((id, version)) = to_cleanup { - cleanup(&ctx, &id, &version).await?; - } - - if configured && manifest.config.is_some() { - let breakages = BTreeMap::new(); - let overrides = Default::default(); - - let configure_context = ConfigureContext { - breakages, - timeout: None, - config: None, - dry_run: false, - overrides, - }; - manager.configure(configure_context).await?; - } - - for to_configure in to_configure.into_iter().filter(|(dep, _)| dep != pkg_id) { - if let Err(e) = async { - ctx.managers - .get(&to_configure) - .await - .or_not_found(format!("manager for {}", to_configure.0))? - .configure(ConfigureContext { - breakages: BTreeMap::new(), - timeout: None, - config: None, - overrides: BTreeMap::new(), - dry_run: false, - }) - .await - } - .await - { - tracing::error!("error configuring dependent: {e}"); - tracing::debug!("{e:?}") - } - } - - if auto_start { - manager.start().await; - } - - tracing::info!("Install {}@{}: Complete", pkg_id, version); - - Ok(()) -} - -#[instrument(skip_all)] -pub async fn unpack_s9pk( - datadir: impl AsRef, - manifest: &Manifest, - rdr: &mut S9pkReader, -) -> Result, Error> { - let datadir = datadir.as_ref(); - let pkg_id = &manifest.id; - let version = &manifest.version; - - let public_dir_path = datadir - .join(PKG_PUBLIC_DIR) - .join(pkg_id) - .join(version.as_str()); - tokio::fs::create_dir_all(&public_dir_path).await?; - - tracing::info!("Install {}@{}: Unpacking LICENSE.md", pkg_id, version); - let license_path = public_dir_path.join("LICENSE.md"); - let mut dst = File::create(&license_path).await?; - tokio::io::copy(&mut rdr.license().await?, &mut dst).await?; - dst.sync_all().await?; - tracing::info!("Install {}@{}: Unpacked LICENSE.md", pkg_id, version); - - tracing::info!("Install {}@{}: Unpacking INSTRUCTIONS.md", pkg_id, version); - let instructions_path = public_dir_path.join("INSTRUCTIONS.md"); - let mut dst = File::create(&instructions_path).await?; - tokio::io::copy(&mut rdr.instructions().await?, &mut dst).await?; - dst.sync_all().await?; - tracing::info!("Install {}@{}: Unpacked INSTRUCTIONS.md", pkg_id, version); - - let icon_filename = Path::new("icon").with_extension(manifest.assets.icon_type()); - let icon_path = public_dir_path.join(&icon_filename); - tracing::info!( - "Install {}@{}: Unpacking {}", - pkg_id, - version, - icon_path.display() - ); - let icon_buf = rdr.icon().await?.to_vec().await?; - let mut dst = File::create(&icon_path).await?; - dst.write_all(&icon_buf).await?; - dst.sync_all().await?; - let icon = DataUrl::from_vec( - mime(manifest.assets.icon_type()).unwrap_or("image/png"), - icon_buf, - ); - tracing::info!( - "Install {}@{}: Unpacked {}", - pkg_id, - version, - icon_filename.display() - ); - - tracing::info!("Install {}@{}: Unpacking Docker Images", pkg_id, version); - Command::new(CONTAINER_TOOL) - .arg("load") - .input(Some(&mut rdr.docker_images().await?)) - .invoke(ErrorKind::Docker) - .await?; - tracing::info!("Install {}@{}: Unpacked Docker Images", pkg_id, version,); - - tracing::info!("Install {}@{}: Unpacking Assets", pkg_id, version); - let asset_dir = asset_dir(datadir, pkg_id, version); - if tokio::fs::metadata(&asset_dir).await.is_ok() { - tokio::fs::remove_dir_all(&asset_dir).await?; - } - tokio::fs::create_dir_all(&asset_dir).await?; - let mut tar = tokio_tar::Archive::new(rdr.assets().await?); - tar.unpack(asset_dir).await?; - - let script_dir = script_dir(datadir, pkg_id, version); - if tokio::fs::metadata(&script_dir).await.is_err() { - tokio::fs::create_dir_all(&script_dir).await?; - } - if let Some(mut hdl) = rdr.scripts().await? { - tokio::io::copy( - &mut hdl, - &mut File::create(script_dir.join("embassy.js")).await?, - ) - .await?; - } - tracing::info!("Install {}@{}: Unpacked Assets", pkg_id, version); - - Ok(icon) -} - -#[instrument(skip_all)] -pub fn rebuild_from<'a>( - source: impl AsRef + 'a + Send + Sync, - datadir: impl AsRef + 'a + Send + Sync, -) -> BoxFuture<'a, Result<(), Error>> { - async move { - let source_dir = source.as_ref(); - let datadir = datadir.as_ref(); - if tokio::fs::metadata(&source_dir).await.is_ok() { - ReadDirStream::new(tokio::fs::read_dir(&source_dir).await?) - .map(|r| { - r.with_ctx(|_| (crate::ErrorKind::Filesystem, format!("{:?}", &source_dir))) - }) - .try_for_each(|entry| async move { - let m = entry.metadata().await?; - if m.is_file() { - let path = entry.path(); - let ext = path.extension().and_then(|ext| ext.to_str()); - if ext == Some("tar") || ext == Some("s9pk") { - if let Err(e) = async { - match ext { - Some("tar") => { - Command::new(CONTAINER_TOOL) - .arg("load") - .input(Some(&mut File::open(&path).await?)) - .invoke(ErrorKind::Docker) - .await?; - Ok::<_, Error>(()) - } - Some("s9pk") => { - let mut s9pk = S9pkReader::open(&path, true).await?; - unpack_s9pk(datadir, &s9pk.manifest().await?, &mut s9pk) - .await?; - Ok(()) - } - _ => unreachable!(), - } - } - .await - { - tracing::error!("Error unpacking {path:?}: {e}"); - tracing::debug!("{e:?}"); - } - Ok(()) - } else { - Ok(()) - } - } else if m.is_dir() { - rebuild_from(entry.path(), datadir).await?; - Ok(()) - } else { - Ok(()) - } - }) - .await - } else { - Ok(()) - } - } - .boxed() -} - -fn ssl_port_status(manifests: &Vec) -> BTreeMap { - let mut ret = BTreeMap::new(); - for m in manifests { - for (_id, iface) in &m.interfaces.0 { - match &iface.lan_config { - None => {} - Some(cfg) => { - for (p, lan) in cfg { - ret.insert(p.clone(), (lan.ssl, m.id.clone())); - } - } - } - } - } - ret -} diff --git a/core/startos/src/install/package-icon.png b/core/startos/src/install/package-icon.png deleted file mode 100644 index 265558cf..00000000 Binary files a/core/startos/src/install/package-icon.png and /dev/null differ diff --git a/core/startos/src/install/progress.rs b/core/startos/src/install/progress.rs deleted file mode 100644 index 61e58e0e..00000000 --- a/core/startos/src/install/progress.rs +++ /dev/null @@ -1,228 +0,0 @@ -use std::future::Future; -use std::io::SeekFrom; -use std::pin::Pin; -use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; -use std::sync::Arc; -use std::task::{Context, Poll}; -use std::time::Duration; - -use models::{OptionExt, PackageId}; -use serde::{Deserialize, Serialize}; -use tokio::io::{AsyncRead, AsyncSeek, AsyncWrite}; - -use crate::db::model::Database; -use crate::prelude::*; - -#[derive(Debug, Deserialize, Serialize, HasModel, Default)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct InstallProgress { - pub size: Option, - pub downloaded: AtomicU64, - pub download_complete: AtomicBool, - pub validated: AtomicU64, - pub validation_complete: AtomicBool, - pub unpacked: AtomicU64, - pub unpack_complete: AtomicBool, -} -impl InstallProgress { - pub fn new(size: Option) -> Self { - InstallProgress { - size, - downloaded: AtomicU64::new(0), - download_complete: AtomicBool::new(false), - validated: AtomicU64::new(0), - validation_complete: AtomicBool::new(false), - unpacked: AtomicU64::new(0), - unpack_complete: AtomicBool::new(false), - } - } - pub fn download_complete(&self) { - self.download_complete.store(true, Ordering::SeqCst) - } - pub async fn track_download(self: Arc, db: PatchDb, id: PackageId) -> Result<(), Error> { - let update = |d: &mut Model| { - d.as_package_data_mut() - .as_idx_mut(&id) - .or_not_found(&id)? - .as_install_progress_mut() - .or_not_found("install-progress")? - .ser(&self) - }; - while !self.download_complete.load(Ordering::SeqCst) { - db.mutate(&update).await?; - tokio::time::sleep(Duration::from_millis(300)).await; - } - db.mutate(&update).await - } - pub async fn track_download_during< - F: FnOnce() -> Fut, - Fut: Future>, - T, - >( - self: &Arc, - db: PatchDb, - id: &PackageId, - f: F, - ) -> Result { - let tracker = tokio::spawn(self.clone().track_download(db.clone(), id.clone())); - let res = f().await; - self.download_complete.store(true, Ordering::SeqCst); - tracker.await.unwrap()?; - res - } - pub async fn track_read( - self: Arc, - db: PatchDb, - id: PackageId, - complete: Arc, - ) -> Result<(), Error> { - let update = |d: &mut Model| { - d.as_package_data_mut() - .as_idx_mut(&id) - .or_not_found(&id)? - .as_install_progress_mut() - .or_not_found("install-progress")? - .ser(&self) - }; - while !complete.load(Ordering::SeqCst) { - db.mutate(&update).await?; - tokio::time::sleep(Duration::from_millis(300)).await; - } - db.mutate(&update).await - } - pub async fn track_read_during< - F: FnOnce() -> Fut, - Fut: Future>, - T, - >( - self: &Arc, - db: PatchDb, - id: &PackageId, - f: F, - ) -> Result { - let complete = Arc::new(AtomicBool::new(false)); - let tracker = tokio::spawn(self.clone().track_read( - db.clone(), - id.clone(), - complete.clone(), - )); - let res = f().await; - complete.store(true, Ordering::SeqCst); - tracker.await.unwrap()?; - res - } -} - -#[pin_project::pin_project] -#[derive(Debug)] -pub struct InstallProgressTracker { - #[pin] - inner: RW, - validating: bool, - progress: Arc, -} -impl InstallProgressTracker { - pub fn new(inner: RW, progress: Arc) -> Self { - InstallProgressTracker { - inner, - validating: true, - progress, - } - } - pub fn validated(&mut self) { - self.progress - .validation_complete - .store(true, Ordering::SeqCst); - self.validating = false; - } -} -impl AsyncWrite for InstallProgressTracker { - fn poll_write( - self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &[u8], - ) -> Poll> { - let this = self.project(); - match this.inner.poll_write(cx, buf) { - Poll::Ready(Ok(n)) => { - this.progress - .downloaded - .fetch_add(n as u64, Ordering::SeqCst); - Poll::Ready(Ok(n)) - } - a => a, - } - } - fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - let this = self.project(); - this.inner.poll_flush(cx) - } - fn poll_shutdown( - self: Pin<&mut Self>, - cx: &mut Context<'_>, - ) -> Poll> { - let this = self.project(); - this.inner.poll_shutdown(cx) - } - fn poll_write_vectored( - self: Pin<&mut Self>, - cx: &mut Context<'_>, - bufs: &[std::io::IoSlice<'_>], - ) -> Poll> { - let this = self.project(); - match this.inner.poll_write_vectored(cx, bufs) { - Poll::Ready(Ok(n)) => { - this.progress - .downloaded - .fetch_add(n as u64, Ordering::SeqCst); - Poll::Ready(Ok(n)) - } - a => a, - } - } -} -impl AsyncRead for InstallProgressTracker { - fn poll_read( - self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &mut tokio::io::ReadBuf<'_>, - ) -> Poll> { - let this = self.project(); - let prev = buf.filled().len() as u64; - match this.inner.poll_read(cx, buf) { - Poll::Ready(Ok(())) => { - if *this.validating { - &this.progress.validated - } else { - &this.progress.unpacked - } - .fetch_add(buf.filled().len() as u64 - prev, Ordering::SeqCst); - - Poll::Ready(Ok(())) - } - a => a, - } - } -} -impl AsyncSeek for InstallProgressTracker { - fn start_seek(self: Pin<&mut Self>, position: SeekFrom) -> std::io::Result<()> { - let this = self.project(); - this.inner.start_seek(position) - } - fn poll_complete(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - let this = self.project(); - match this.inner.poll_complete(cx) { - Poll::Ready(Ok(n)) => { - if *this.validating { - &this.progress.validated - } else { - &this.progress.unpacked - } - .store(n, Ordering::SeqCst); - Poll::Ready(Ok(n)) - } - a => a, - } - } -} diff --git a/core/startos/src/install/update.rs b/core/startos/src/install/update.rs deleted file mode 100644 index 69405121..00000000 --- a/core/startos/src/install/update.rs +++ /dev/null @@ -1,18 +0,0 @@ -use std::collections::BTreeMap; - -use rpc_toolkit::command; -use tracing::instrument; - -use crate::config::not_found; -use crate::context::RpcContext; -use crate::db::model::CurrentDependents; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::display_serializable; -use crate::util::Version; -use crate::Error; - -#[command(subcommands(dry))] -pub async fn update() -> Result<(), Error> { - Ok(()) -} diff --git a/core/startos/src/lib.rs b/core/startos/src/lib.rs deleted file mode 100644 index 5fde6513..00000000 --- a/core/startos/src/lib.rs +++ /dev/null @@ -1,158 +0,0 @@ -pub const DEFAULT_MARKETPLACE: &str = "https://registry.start9.com"; -// pub const COMMUNITY_MARKETPLACE: &str = "https://community-registry.start9.com"; -pub const BUFFER_SIZE: usize = 1024; -pub const HOST_IP: [u8; 4] = [172, 18, 0, 1]; -pub const TARGET: &str = current_platform::CURRENT_PLATFORM; -lazy_static::lazy_static! { - pub static ref ARCH: &'static str = { - let (arch, _) = TARGET.split_once("-").unwrap(); - arch - }; - pub static ref PLATFORM: String = { - if let Ok(platform) = std::fs::read_to_string("/usr/lib/startos/PLATFORM.txt") { - platform - } else { - ARCH.to_string() - } - }; - pub static ref SOURCE_DATE: SystemTime = { - std::fs::metadata(std::env::current_exe().unwrap()).unwrap().modified().unwrap() - }; -} - -pub mod account; -pub mod action; -pub mod auth; -pub mod backup; -pub mod bins; -pub mod config; -pub mod context; -pub mod control; -pub mod core; -pub mod db; -pub mod dependencies; -pub mod developer; -pub mod diagnostic; -pub mod disk; -pub mod error; -pub mod firmware; -pub mod hostname; -pub mod init; -pub mod inspect; -pub mod install; -pub mod logs; -pub mod manager; -pub mod middleware; -pub mod migration; -pub mod net; -pub mod notifications; -pub mod os_install; -pub mod prelude; -pub mod procedure; -pub mod properties; -pub mod registry; -pub mod s9pk; -pub mod setup; -pub mod shutdown; -pub mod sound; -pub mod ssh; -pub mod status; -pub mod system; -pub mod update; -pub mod util; -pub mod version; -pub mod volume; - -use std::time::SystemTime; - -pub use config::Config; -pub use error::{Error, ErrorKind, ResultExt}; -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; - -#[command(metadata(authenticated = false))] -pub fn echo(#[arg] message: String) -> Result { - Ok(message) -} - -#[command(subcommands( - version::git_info, - echo, - inspect::inspect, - server, - package, - net::net, - auth::auth, - db::db, - ssh::ssh, - net::wifi::wifi, - disk::disk, - notifications::notification, - backup::backup, - registry::marketplace::marketplace, -))] -pub fn main_api() -> Result<(), RpcError> { - Ok(()) -} - -#[command(subcommands( - system::time, - system::experimental, - system::logs, - system::kernel_logs, - system::metrics, - shutdown::shutdown, - shutdown::restart, - shutdown::rebuild, - update::update_system, - firmware::update_firmware, -))] -pub fn server() -> Result<(), RpcError> { - Ok(()) -} - -#[command(subcommands( - action::action, - install::install, - install::sideload, - install::uninstall, - install::list, - config::config, - control::start, - control::stop, - control::restart, - logs::logs, - properties::properties, - dependencies::dependency, - backup::package_backup, -))] -pub fn package() -> Result<(), RpcError> { - Ok(()) -} - -#[command(subcommands( - version::git_info, - s9pk::pack, - developer::verify, - developer::init, - inspect::inspect, - registry::admin::publish, -))] -pub fn portable_api() -> Result<(), RpcError> { - Ok(()) -} - -#[command(subcommands(version::git_info, echo, diagnostic::diagnostic))] -pub fn diagnostic_api() -> Result<(), RpcError> { - Ok(()) -} - -#[command(subcommands(version::git_info, echo, setup::setup))] -pub fn setup_api() -> Result<(), RpcError> { - Ok(()) -} - -#[command(subcommands(version::git_info, echo, os_install::install))] -pub fn install_api() -> Result<(), RpcError> { - Ok(()) -} diff --git a/core/startos/src/logs.rs b/core/startos/src/logs.rs deleted file mode 100644 index 691ae09b..00000000 --- a/core/startos/src/logs.rs +++ /dev/null @@ -1,551 +0,0 @@ -use std::future::Future; -use std::marker::PhantomData; -use std::ops::{Deref, DerefMut}; -use std::process::Stdio; -use std::time::{Duration, UNIX_EPOCH}; - -use chrono::{DateTime, Utc}; -use color_eyre::eyre::eyre; -use futures::stream::BoxStream; -use futures::{FutureExt, SinkExt, Stream, StreamExt, TryStreamExt}; -use hyper::upgrade::Upgraded; -use hyper::Error as HyperError; -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; -use serde::{Deserialize, Serialize}; -use tokio::io::{AsyncBufReadExt, BufReader}; -use tokio::process::{Child, Command}; -use tokio::task::JoinError; -use tokio_stream::wrappers::LinesStream; -use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode; -use tokio_tungstenite::tungstenite::protocol::CloseFrame; -use tokio_tungstenite::tungstenite::Message; -use tokio_tungstenite::WebSocketStream; -use tracing::instrument; - -use crate::context::{CliContext, RpcContext}; -use crate::core::rpc_continuations::{RequestGuid, RpcContinuation}; -use crate::error::ResultExt; -use crate::procedure::docker::DockerProcedure; -use crate::s9pk::manifest::PackageId; -use crate::util::display_none; -use crate::util::serde::Reversible; -use crate::{Error, ErrorKind}; - -#[pin_project::pin_project] -pub struct LogStream { - _child: Child, - #[pin] - entries: BoxStream<'static, Result>, -} -impl Deref for LogStream { - type Target = BoxStream<'static, Result>; - fn deref(&self) -> &Self::Target { - &self.entries - } -} -impl DerefMut for LogStream { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.entries - } -} -impl Stream for LogStream { - type Item = Result; - fn poll_next( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - let this = self.project(); - Stream::poll_next(this.entries, cx) - } - - fn size_hint(&self) -> (usize, Option) { - self.entries.size_hint() - } -} - -#[instrument(skip_all)] -async fn ws_handler< - WSFut: Future, HyperError>, JoinError>>, ->( - first_entry: Option, - mut logs: LogStream, - ws_fut: WSFut, -) -> Result<(), Error> { - let mut stream = ws_fut - .await - .with_kind(crate::ErrorKind::Network)? - .with_kind(crate::ErrorKind::Unknown)?; - - if let Some(first_entry) = first_entry { - stream - .send(Message::Text( - serde_json::to_string(&first_entry).with_kind(ErrorKind::Serialization)?, - )) - .await - .with_kind(ErrorKind::Network)?; - } - - let mut ws_closed = false; - while let Some(entry) = tokio::select! { - a = logs.try_next() => Some(a?), - a = stream.try_next() => { a.with_kind(crate::ErrorKind::Network)?; ws_closed = true; None } - } { - if let Some(entry) = entry { - let (_, log_entry) = entry.log_entry()?; - stream - .send(Message::Text( - serde_json::to_string(&log_entry).with_kind(ErrorKind::Serialization)?, - )) - .await - .with_kind(ErrorKind::Network)?; - } - } - - if !ws_closed { - stream - .close(Some(CloseFrame { - code: CloseCode::Normal, - reason: "Log Stream Finished".into(), - })) - .await - .with_kind(ErrorKind::Network)?; - } - - Ok(()) -} - -#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)] -#[serde(rename_all = "kebab-case")] -pub struct LogResponse { - entries: Reversible, - start_cursor: Option, - end_cursor: Option, -} -#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)] -#[serde(rename_all = "kebab-case")] -pub struct LogFollowResponse { - start_cursor: Option, - guid: RequestGuid, -} - -#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)] -pub struct LogEntry { - timestamp: DateTime, - message: String, -} -impl std::fmt::Display for LogEntry { - fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { - write!( - f, - "{} {}", - self.timestamp - .to_rfc3339_opts(chrono::SecondsFormat::Millis, true), - self.message - ) - } -} - -#[derive(Serialize, Deserialize, Debug)] -pub struct JournalctlEntry { - #[serde(rename = "__REALTIME_TIMESTAMP")] - pub timestamp: String, - #[serde(rename = "MESSAGE")] - #[serde(deserialize_with = "deserialize_log_message")] - pub message: String, - #[serde(rename = "__CURSOR")] - pub cursor: String, -} -impl JournalctlEntry { - fn log_entry(self) -> Result<(String, LogEntry), Error> { - Ok(( - self.cursor, - LogEntry { - timestamp: DateTime::::from( - UNIX_EPOCH + Duration::from_micros(self.timestamp.parse::()?), - ), - message: self.message, - }, - )) - } -} - -fn deserialize_log_message<'de, D: serde::de::Deserializer<'de>>( - deserializer: D, -) -> std::result::Result { - struct Visitor; - impl<'de> serde::de::Visitor<'de> for Visitor { - type Value = String; - fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(formatter, "a parsable string") - } - fn visit_str(self, v: &str) -> Result - where - E: serde::de::Error, - { - Ok(v.trim().to_owned()) - } - fn visit_unit(self) -> Result - where - E: serde::de::Error, - { - Ok(String::new()) - } - fn visit_seq(self, mut seq: A) -> Result - where - A: serde::de::SeqAccess<'de>, - { - String::from_utf8( - std::iter::repeat_with(|| seq.next_element::().transpose()) - .take_while(|a| a.is_some()) - .flatten() - .collect::, _>>()?, - ) - .map(|s| s.trim().to_owned()) - .map_err(serde::de::Error::custom) - } - } - deserializer.deserialize_any(Visitor) -} - -/// Defining how we are going to filter on a journalctl cli log. -/// Kernal: (-k --dmesg Show kernel message log from the current boot) -/// Unit: ( -u --unit=UNIT Show logs from the specified unit -/// --user-unit=UNIT Show logs from the specified user unit)) -/// System: Unit is startd, but we also filter on the comm -/// Container: Filtering containers, like podman/docker is done by filtering on the CONTAINER_NAME -#[derive(Debug)] -pub enum LogSource { - Kernel, - Unit(&'static str), - System, - Container(PackageId), -} - -pub const SYSTEM_UNIT: &str = "startd"; - -#[command( - custom_cli(cli_logs(async, context(CliContext))), - subcommands(self(logs_nofollow(async)), logs_follow), - display(display_none) -)] -pub async fn logs( - #[arg] id: PackageId, - #[arg(short = 'l', long = "limit")] limit: Option, - #[arg(short = 'c', long = "cursor")] cursor: Option, - #[arg(short = 'B', long = "before", default)] before: bool, - #[arg(short = 'f', long = "follow", default)] follow: bool, -) -> Result<(PackageId, Option, Option, bool, bool), Error> { - Ok((id, limit, cursor, before, follow)) -} -pub async fn cli_logs( - ctx: CliContext, - (id, limit, cursor, before, follow): (PackageId, Option, Option, bool, bool), -) -> Result<(), RpcError> { - if follow { - if cursor.is_some() { - return Err(RpcError::from(Error::new( - eyre!("The argument '--cursor ' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - if before { - return Err(RpcError::from(Error::new( - eyre!("The argument '--before' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - cli_logs_generic_follow(ctx, "package.logs.follow", Some(id), limit).await - } else { - cli_logs_generic_nofollow(ctx, "package.logs", Some(id), limit, cursor, before).await - } -} -pub async fn logs_nofollow( - _ctx: (), - (id, limit, cursor, before, _): (PackageId, Option, Option, bool, bool), -) -> Result { - fetch_logs(LogSource::Container(id), limit, cursor, before).await -} -#[command(rpc_only, rename = "follow", display(display_none))] -pub async fn logs_follow( - #[context] ctx: RpcContext, - #[parent_data] (id, limit, _, _, _): (PackageId, Option, Option, bool, bool), -) -> Result { - follow_logs(ctx, LogSource::Container(id), limit).await -} - -pub async fn cli_logs_generic_nofollow( - ctx: CliContext, - method: &str, - id: Option, - limit: Option, - cursor: Option, - before: bool, -) -> Result<(), RpcError> { - let res = rpc_toolkit::command_helpers::call_remote( - ctx.clone(), - method, - serde_json::json!({ - "id": id, - "limit": limit, - "cursor": cursor, - "before": before, - }), - PhantomData::, - ) - .await? - .result?; - - for entry in res.entries.iter() { - println!("{}", entry); - } - - Ok(()) -} - -pub async fn cli_logs_generic_follow( - ctx: CliContext, - method: &str, - id: Option, - limit: Option, -) -> Result<(), RpcError> { - let res = rpc_toolkit::command_helpers::call_remote( - ctx.clone(), - method, - serde_json::json!({ - "id": id, - "limit": limit, - }), - PhantomData::, - ) - .await? - .result?; - - let mut base_url = ctx.base_url.clone(); - let ws_scheme = match base_url.scheme() { - "https" => "wss", - "http" => "ws", - _ => { - return Err(Error::new( - eyre!("Cannot parse scheme from base URL"), - crate::ErrorKind::ParseUrl, - ) - .into()) - } - }; - base_url - .set_scheme(ws_scheme) - .map_err(|_| Error::new(eyre!("Cannot set URL scheme"), crate::ErrorKind::ParseUrl))?; - let (mut stream, _) = - // base_url is "http://127.0.0.1/", with a trailing slash, so we don't put a leading slash in this path: - tokio_tungstenite::connect_async(format!("{}ws/rpc/{}", base_url, res.guid)).await?; - while let Some(log) = stream.try_next().await? { - if let Message::Text(log) = log { - println!("{}", serde_json::from_str::(&log)?); - } - } - - Ok(()) -} - -pub async fn journalctl( - id: LogSource, - limit: usize, - cursor: Option<&str>, - before: bool, - follow: bool, -) -> Result { - let mut cmd = Command::new("journalctl"); - cmd.kill_on_drop(true); - - cmd.arg("--output=json"); - cmd.arg("--output-fields=MESSAGE"); - cmd.arg(format!("-n{}", limit)); - match id { - LogSource::Kernel => { - cmd.arg("-k"); - } - LogSource::Unit(id) => { - cmd.arg("-u"); - cmd.arg(id); - } - LogSource::System => { - cmd.arg("-u"); - cmd.arg(SYSTEM_UNIT); - cmd.arg(format!("_COMM={}", SYSTEM_UNIT)); - } - LogSource::Container(id) => { - #[cfg(not(feature = "docker"))] - cmd.arg(format!( - "SYSLOG_IDENTIFIER={}", - DockerProcedure::container_name(&id, None) - )); - #[cfg(feature = "docker")] - cmd.arg(format!( - "CONTAINER_NAME={}", - DockerProcedure::container_name(&id, None) - )); - } - }; - - let cursor_formatted = format!("--after-cursor={}", cursor.unwrap_or("")); - if cursor.is_some() { - cmd.arg(&cursor_formatted); - if before { - cmd.arg("--reverse"); - } - } - if follow { - cmd.arg("--follow"); - } - - let mut child = cmd.stdout(Stdio::piped()).spawn()?; - let out = BufReader::new( - child - .stdout - .take() - .ok_or_else(|| Error::new(eyre!("No stdout available"), crate::ErrorKind::Journald))?, - ); - - let journalctl_entries = LinesStream::new(out.lines()); - - let deserialized_entries = journalctl_entries - .map_err(|e| Error::new(e, crate::ErrorKind::Journald)) - .and_then(|s| { - futures::future::ready( - serde_json::from_str::(&s) - .with_kind(crate::ErrorKind::Deserialization), - ) - }); - - Ok(LogStream { - _child: child, - entries: deserialized_entries.boxed(), - }) -} - -#[instrument(skip_all)] -pub async fn fetch_logs( - id: LogSource, - limit: Option, - cursor: Option, - before: bool, -) -> Result { - let limit = limit.unwrap_or(50); - let mut stream = journalctl(id, limit, cursor.as_deref(), before, false).await?; - - let mut entries = Vec::with_capacity(limit); - let mut start_cursor = None; - - if let Some(first) = tokio::time::timeout(Duration::from_secs(1), stream.try_next()) - .await - .ok() - .transpose()? - .flatten() - { - let (cursor, entry) = first.log_entry()?; - start_cursor = Some(cursor); - entries.push(entry); - } - - let (mut end_cursor, entries) = stream - .try_fold( - (start_cursor.clone(), entries), - |(_, mut acc), entry| async move { - let (cursor, entry) = entry.log_entry()?; - acc.push(entry); - Ok((Some(cursor), acc)) - }, - ) - .await?; - let mut entries = Reversible::new(entries); - // reverse again so output is always in increasing chronological order - if cursor.is_some() && before { - entries.reverse(); - std::mem::swap(&mut start_cursor, &mut end_cursor); - } - Ok(LogResponse { - entries, - start_cursor, - end_cursor, - }) -} - -#[instrument(skip_all)] -pub async fn follow_logs( - ctx: RpcContext, - id: LogSource, - limit: Option, -) -> Result { - let limit = limit.unwrap_or(50); - let mut stream = journalctl(id, limit, None, false, true).await?; - - let mut start_cursor = None; - let mut first_entry = None; - - if let Some(first) = tokio::time::timeout(Duration::from_secs(1), stream.try_next()) - .await - .ok() - .transpose()? - .flatten() - { - let (cursor, entry) = first.log_entry()?; - start_cursor = Some(cursor); - first_entry = Some(entry); - } - - let guid = RequestGuid::new(); - ctx.add_continuation( - guid.clone(), - RpcContinuation::ws( - Box::new(move |ws_fut| ws_handler(first_entry, stream, ws_fut).boxed()), - Duration::from_secs(30), - ), - ) - .await; - Ok(LogFollowResponse { start_cursor, guid }) -} - -// #[tokio::test] -// pub async fn test_logs() { -// let response = fetch_logs( -// // change `tor.service` to an actual journald unit on your machine -// // LogSource::Service("tor.service"), -// // first run `docker run --name=hello-world.embassy --log-driver=journald hello-world` -// LogSource::Container("hello-world".parse().unwrap()), -// // Some(5), -// None, -// None, -// // Some("s=1b8c418e28534400856c27b211dd94fd;i=5a7;b=97571c13a1284f87bc0639b5cff5acbe;m=740e916;t=5ca073eea3445;x=f45bc233ca328348".to_owned()), -// false, -// true, -// ) -// .await -// .unwrap(); -// let serialized = serde_json::to_string_pretty(&response).unwrap(); -// println!("{}", serialized); -// } - -// #[tokio::test] -// pub async fn test_logs() { -// let mut cmd = Command::new("journalctl"); -// cmd.kill_on_drop(true); - -// cmd.arg("-f"); -// cmd.arg("CONTAINER_NAME=hello-world.embassy"); - -// let mut child = cmd.stdout(Stdio::piped()).spawn().unwrap(); -// let out = BufReader::new( -// child -// .stdout -// .take() -// .ok_or_else(|| Error::new(eyre!("No stdout available"), crate::ErrorKind::Journald)) -// .unwrap(), -// ); - -// let mut journalctl_entries = LinesStream::new(out.lines()); - -// while let Some(line) = journalctl_entries.try_next().await.unwrap() { -// dbg!(line); -// } -// } diff --git a/core/startos/src/main.rs b/core/startos/src/main.rs deleted file mode 100644 index 371cd5e7..00000000 --- a/core/startos/src/main.rs +++ /dev/null @@ -1,3 +0,0 @@ -fn main() { - startos::bins::startbox() -} diff --git a/core/startos/src/manager/health.rs b/core/startos/src/manager/health.rs deleted file mode 100644 index 30f18051..00000000 --- a/core/startos/src/manager/health.rs +++ /dev/null @@ -1,56 +0,0 @@ -use models::OptionExt; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::status::MainStatus; -use crate::Error; - -/// So, this is used for a service to run a health check cycle, go out and run the health checks, and store those in the db -#[instrument(skip_all)] -pub async fn check(ctx: &RpcContext, id: &PackageId) -> Result<(), Error> { - let (manifest, started) = { - let peeked = ctx.db.peek().await; - let pde = peeked - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .expect_as_installed()?; - - let manifest = pde.as_installed().as_manifest().de()?; - - let started = pde.as_installed().as_status().as_main().de()?.started(); - - (manifest, started) - }; - - let health_results = if let Some(started) = started { - tracing::debug!("Checking health of {}", id); - manifest - .health_checks - .check_all(ctx, started, id, &manifest.version, &manifest.volumes) - .await? - } else { - return Ok(()); - }; - - ctx.db - .mutate(|v| { - let pde = v - .as_package_data_mut() - .as_idx_mut(id) - .or_not_found(id)? - .expect_as_installed_mut()?; - let status = pde.as_installed_mut().as_status_mut().as_main_mut(); - - if let MainStatus::Running { health: _, started } = status.de()? { - status.ser(&MainStatus::Running { - health: health_results.clone(), - started, - })?; - } - Ok(()) - }) - .await -} diff --git a/core/startos/src/manager/manager_container.rs b/core/startos/src/manager/manager_container.rs deleted file mode 100644 index 32e11c2e..00000000 --- a/core/startos/src/manager/manager_container.rs +++ /dev/null @@ -1,300 +0,0 @@ -use std::sync::Arc; -use std::time::Duration; - -use models::OptionExt; -use tokio::sync::watch; -use tokio::sync::watch::Sender; -use tracing::instrument; - -use super::start_stop::StartStop; -use super::{manager_seed, run_main, ManagerPersistentContainer, RunMainResult}; -use crate::prelude::*; -use crate::procedure::NoOutput; -use crate::s9pk::manifest::Manifest; -use crate::status::MainStatus; -use crate::util::NonDetachingJoinHandle; -use crate::Error; - -pub type ManageContainerOverride = Arc>>; - -pub type Override = MainStatus; - -pub struct OverrideGuard { - override_main_status: Option, -} -impl OverrideGuard { - pub fn drop(self) {} -} -impl Drop for OverrideGuard { - fn drop(&mut self) { - if let Some(override_main_status) = self.override_main_status.take() { - override_main_status.send_modify(|x| { - *x = None; - }); - } - } -} - -/// This is the thing describing the state machine actor for a service -/// state and current running/ desired states. -pub struct ManageContainer { - pub(super) current_state: Arc>, - pub(super) desired_state: Arc>, - _service: NonDetachingJoinHandle<()>, - _save_state: NonDetachingJoinHandle<()>, - override_main_status: ManageContainerOverride, -} - -impl ManageContainer { - pub async fn new( - seed: Arc, - persistent_container: ManagerPersistentContainer, - ) -> Result { - let current_state = Arc::new(watch::channel(StartStop::Stop).0); - let desired_state = Arc::new( - watch::channel::( - get_status(seed.ctx.db.peek().await, &seed.manifest).into(), - ) - .0, - ); - let override_main_status: ManageContainerOverride = Arc::new(watch::channel(None).0); - let service = tokio::spawn(create_service_manager( - desired_state.clone(), - seed.clone(), - current_state.clone(), - persistent_container, - )) - .into(); - let save_state = tokio::spawn(save_state( - desired_state.clone(), - current_state.clone(), - override_main_status.clone(), - seed.clone(), - )) - .into(); - Ok(ManageContainer { - current_state, - desired_state, - _service: service, - override_main_status, - _save_state: save_state, - }) - } - - /// Set override is used during something like a restart of a service. We want to show certain statuses be different - /// from the actual status of the service. - pub fn set_override(&self, override_status: Override) -> Result { - let status = Some(override_status); - if self.override_main_status.borrow().is_some() { - return Err(Error::new( - eyre!("Already have an override"), - ErrorKind::InvalidRequest, - )); - } - self.override_main_status - .send_modify(|x| *x = status.clone()); - Ok(OverrideGuard { - override_main_status: Some(self.override_main_status.clone()), - }) - } - - /// Set the override, but don't have a guard to revert it. Used only on the mananger to do a shutdown. - pub(super) async fn lock_state_forever( - &self, - seed: &manager_seed::ManagerSeed, - ) -> Result<(), Error> { - let current_state = get_status(seed.ctx.db.peek().await, &seed.manifest); - self.override_main_status - .send_modify(|x| *x = Some(current_state)); - Ok(()) - } - - /// We want to set the state of the service, like to start or stop - pub fn to_desired(&self, new_state: StartStop) { - self.desired_state.send_modify(|x| *x = new_state); - } - - /// This is a tool to say wait for the service to be in a certain state. - pub async fn wait_for_desired(&self, new_state: StartStop) { - let mut current_state = self.current_state(); - self.to_desired(new_state); - while *current_state.borrow() != new_state { - current_state.changed().await.unwrap_or_default(); - } - } - - /// Getter - pub fn current_state(&self) -> watch::Receiver { - self.current_state.subscribe() - } - - /// Getter - pub fn desired_state(&self) -> watch::Receiver { - self.desired_state.subscribe() - } -} - -async fn create_service_manager( - desired_state: Arc>, - seed: Arc, - current_state: Arc>, - persistent_container: Arc>, -) { - let mut desired_state_receiver = desired_state.subscribe(); - let mut running_service: Option> = None; - let seed = seed.clone(); - loop { - let current: StartStop = *current_state.borrow(); - let desired: StartStop = *desired_state_receiver.borrow(); - match (current, desired) { - (StartStop::Start, StartStop::Start) => (), - (StartStop::Start, StartStop::Stop) => { - if persistent_container.is_none() { - if let Err(err) = seed.stop_container().await { - tracing::error!("Could not stop container"); - tracing::debug!("{:?}", err) - } - running_service = None; - } else if let Some(current_service) = running_service.take() { - tokio::select! { - _ = current_service => (), - _ = tokio::time::sleep(Duration::from_secs_f64(seed.manifest - .containers - .as_ref() - .and_then(|c| c.main.sigterm_timeout).map(|x| x.as_secs_f64()).unwrap_or_default())) => { - tracing::error!("Could not stop service"); - } - } - } - current_state.send_modify(|x| *x = StartStop::Stop); - } - (StartStop::Stop, StartStop::Start) => starting_service( - current_state.clone(), - desired_state.clone(), - seed.clone(), - persistent_container.clone(), - &mut running_service, - ), - (StartStop::Stop, StartStop::Stop) => (), - } - - if desired_state_receiver.changed().await.is_err() { - tracing::error!("Desired state error"); - break; - } - } -} - -async fn save_state( - desired_state: Arc>, - current_state: Arc>, - override_main_status: ManageContainerOverride, - seed: Arc, -) { - let mut desired_state_receiver = desired_state.subscribe(); - let mut current_state_receiver = current_state.subscribe(); - let mut override_main_status_receiver = override_main_status.subscribe(); - loop { - let current: StartStop = *current_state_receiver.borrow(); - let desired: StartStop = *desired_state_receiver.borrow(); - let override_status = override_main_status_receiver.borrow().clone(); - let status = match (override_status.clone(), current, desired) { - (Some(status), _, _) => status, - (_, StartStop::Start, StartStop::Start) => MainStatus::Running { - started: chrono::Utc::now(), - health: Default::default(), - }, - (_, StartStop::Start, StartStop::Stop) => MainStatus::Stopping, - (_, StartStop::Stop, StartStop::Start) => MainStatus::Starting, - (_, StartStop::Stop, StartStop::Stop) => MainStatus::Stopped, - }; - - let manifest = &seed.manifest; - if let Err(err) = seed - .ctx - .db - .mutate(|db| set_status(db, manifest, &status)) - .await - { - tracing::error!("Did not set status for {}", seed.container_name); - tracing::debug!("{:?}", err); - } - tokio::select! { - _ = desired_state_receiver.changed() =>{}, - _ = current_state_receiver.changed() => {}, - _ = override_main_status_receiver.changed() => {} - } - } -} - -fn starting_service( - current_state: Arc>, - desired_state: Arc>, - seed: Arc, - persistent_container: ManagerPersistentContainer, - running_service: &mut Option>, -) { - let set_running = { - let current_state = current_state.clone(); - Arc::new(move || { - current_state.send_modify(|x| *x = StartStop::Start); - }) - }; - let set_stopped = { move || current_state.send_modify(|x| *x = StartStop::Stop) }; - let running_main_loop = async move { - while desired_state.borrow().is_start() { - let result = run_main( - seed.clone(), - persistent_container.clone(), - set_running.clone(), - ) - .await; - set_stopped(); - run_main_log_result(result, seed.clone()).await; - } - }; - *running_service = Some(tokio::spawn(running_main_loop).into()); -} - -async fn run_main_log_result(result: RunMainResult, seed: Arc) { - match result { - Ok(Ok(NoOutput)) => (), // restart - Ok(Err(e)) => { - tracing::error!( - "The service {} has crashed with the following exit code: {}", - seed.manifest.id.clone(), - e.0 - ); - - tokio::time::sleep(Duration::from_secs(15)).await; - } - Err(e) => { - tracing::error!("failed to start service: {}", e); - tracing::debug!("{:?}", e); - } - } -} - -/// Used only in the mod where we are doing a backup -#[instrument(skip(db, manifest))] -pub(super) fn get_status(db: Peeked, manifest: &Manifest) -> MainStatus { - db.as_package_data() - .as_idx(&manifest.id) - .and_then(|x| x.as_installed()) - .filter(|x| x.as_manifest().as_version().de().ok() == Some(manifest.version.clone())) - .and_then(|x| x.as_status().as_main().de().ok()) - .unwrap_or(MainStatus::Stopped) -} - -#[instrument(skip(db, manifest))] -fn set_status(db: &mut Peeked, manifest: &Manifest, main_status: &MainStatus) -> Result<(), Error> { - let Some(installed) = db - .as_package_data_mut() - .as_idx_mut(&manifest.id) - .or_not_found(&manifest.id)? - .as_installed_mut() - else { - return Ok(()); - }; - installed.as_status_mut().as_main_mut().ser(main_status) -} diff --git a/core/startos/src/manager/manager_map.rs b/core/startos/src/manager/manager_map.rs deleted file mode 100644 index 07f128cc..00000000 --- a/core/startos/src/manager/manager_map.rs +++ /dev/null @@ -1,96 +0,0 @@ -use std::collections::BTreeMap; -use std::sync::Arc; - -use color_eyre::eyre::eyre; -use tokio::sync::RwLock; -use tracing::instrument; - -use super::Manager; -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::{Manifest, PackageId}; -use crate::util::Version; -use crate::Error; - -/// This is the structure to contain all the service managers -#[derive(Default)] -pub struct ManagerMap(RwLock>>); -impl ManagerMap { - #[instrument(skip_all)] - pub async fn init(&self, ctx: RpcContext, peeked: Peeked) -> Result<(), Error> { - let mut res = BTreeMap::new(); - for package in peeked.as_package_data().keys()? { - let man: Manifest = if let Some(manifest) = peeked - .as_package_data() - .as_idx(&package) - .and_then(|x| x.as_installed()) - .map(|x| x.as_manifest().de()) - { - manifest? - } else { - continue; - }; - - res.insert( - (package, man.version.clone()), - Arc::new(Manager::new(ctx.clone(), man).await?), - ); - } - *self.0.write().await = res; - Ok(()) - } - - /// Used during the install process - #[instrument(skip_all)] - pub async fn add(&self, ctx: RpcContext, manifest: Manifest) -> Result, Error> { - let mut lock = self.0.write().await; - let id = (manifest.id.clone(), manifest.version.clone()); - if let Some(man) = lock.remove(&id) { - man.exit().await; - } - let manager = Arc::new(Manager::new(ctx.clone(), manifest).await?); - lock.insert(id, manager.clone()); - Ok(manager) - } - - /// This is ran during the cleanup, so when we are uninstalling the service - #[instrument(skip_all)] - pub async fn remove(&self, id: &(PackageId, Version)) { - if let Some(man) = self.0.write().await.remove(id) { - man.exit().await; - } - } - - /// Used during a shutdown - #[instrument(skip_all)] - pub async fn empty(&self) -> Result<(), Error> { - let res = - futures::future::join_all(std::mem::take(&mut *self.0.write().await).into_iter().map( - |((id, version), man)| async move { - tracing::debug!("Manager for {}@{} shutting down", id, version); - man.shutdown().await?; - tracing::debug!("Manager for {}@{} is shutdown", id, version); - if let Err(e) = Arc::try_unwrap(man) { - tracing::trace!( - "Manager for {}@{} still has {} other open references", - id, - version, - Arc::strong_count(&e) - 1 - ); - } - Ok::<_, Error>(()) - }, - )) - .await; - res.into_iter().fold(Ok(()), |res, x| match (res, x) { - (Ok(()), x) => x, - (Err(e), Ok(())) => Err(e), - (Err(e1), Err(e2)) => Err(Error::new(eyre!("{}, {}", e1.source, e2.source), e1.kind)), - }) - } - - #[instrument(skip_all)] - pub async fn get(&self, id: &(PackageId, Version)) -> Option> { - self.0.read().await.get(id).cloned() - } -} diff --git a/core/startos/src/manager/manager_seed.rs b/core/startos/src/manager/manager_seed.rs deleted file mode 100644 index f90e7739..00000000 --- a/core/startos/src/manager/manager_seed.rs +++ /dev/null @@ -1,37 +0,0 @@ -use models::ErrorKind; - -use crate::context::RpcContext; -use crate::procedure::docker::DockerProcedure; -use crate::procedure::PackageProcedure; -use crate::s9pk::manifest::Manifest; -use crate::util::docker::stop_container; -use crate::Error; - -/// This is helper structure for a service, the seed of the data that is needed for the manager_container -pub struct ManagerSeed { - pub ctx: RpcContext, - pub manifest: Manifest, - pub container_name: String, -} - -impl ManagerSeed { - pub async fn stop_container(&self) -> Result<(), Error> { - match stop_container( - &self.container_name, - match &self.manifest.main { - PackageProcedure::Docker(DockerProcedure { - sigterm_timeout: Some(sigterm_timeout), - .. - }) => Some(**sigterm_timeout), - _ => None, - }, - None, - ) - .await - { - Err(e) if e.kind == ErrorKind::NotFound => (), // Already stopped - a => a?, - } - Ok(()) - } -} diff --git a/core/startos/src/manager/mod.rs b/core/startos/src/manager/mod.rs deleted file mode 100644 index 1151da2f..00000000 --- a/core/startos/src/manager/mod.rs +++ /dev/null @@ -1,888 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; -use std::net::Ipv4Addr; -use std::sync::Arc; -use std::task::Poll; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use container_init::ProcessGroupId; -use futures::future::BoxFuture; -use futures::{Future, FutureExt, TryFutureExt}; -use helpers::UnixRpcClient; -use models::{ErrorKind, OptionExt, PackageId}; -use nix::sys::signal::Signal; -use persistent_container::PersistentContainer; -use rand::SeedableRng; -use sqlx::Connection; -use start_stop::StartStop; -use tokio::sync::watch::{self, Sender}; -use tokio::sync::{oneshot, Mutex}; -use tracing::instrument; -use transition_state::TransitionState; - -use crate::backup::target::PackageBackupInfo; -use crate::backup::PackageBackupReport; -use crate::config::action::ConfigRes; -use crate::config::spec::ValueSpecPointer; -use crate::config::ConfigureContext; -use crate::context::RpcContext; -use crate::db::model::{CurrentDependencies, CurrentDependencyInfo}; -use crate::dependencies::{ - add_dependent_to_current_dependents_lists, compute_dependency_config_errs, -}; -use crate::disk::mount::backup::BackupMountGuard; -use crate::disk::mount::guard::TmpMountGuard; -use crate::install::cleanup::remove_from_current_dependents_lists; -use crate::net::net_controller::NetService; -use crate::net::vhost::AlpnInfo; -use crate::prelude::*; -use crate::procedure::docker::{DockerContainer, DockerProcedure, LongRunning}; -use crate::procedure::{NoOutput, ProcedureName}; -use crate::s9pk::manifest::Manifest; -use crate::status::MainStatus; -use crate::util::docker::{get_container_ip, kill_container}; -use crate::util::NonDetachingJoinHandle; -use crate::volume::Volume; -use crate::Error; - -pub mod health; -mod manager_container; -mod manager_map; -pub mod manager_seed; -mod persistent_container; -mod start_stop; -mod transition_state; - -pub use manager_map::ManagerMap; - -use self::manager_container::{get_status, ManageContainer}; -use self::manager_seed::ManagerSeed; - -pub const HEALTH_CHECK_COOLDOWN_SECONDS: u64 = 15; -pub const HEALTH_CHECK_GRACE_PERIOD_SECONDS: u64 = 5; - -type ManagerPersistentContainer = Arc>; -type BackupGuard = Arc>>; -pub enum BackupReturn { - Error(Error), - AlreadyRunning(PackageBackupReport), - Ran { - report: PackageBackupReport, - res: Result, - }, -} - -pub struct Gid { - next_gid: (watch::Sender, watch::Receiver), - main_gid: ( - watch::Sender, - watch::Receiver, - ), -} - -impl Default for Gid { - fn default() -> Self { - Self { - next_gid: watch::channel(1), - main_gid: watch::channel(ProcessGroupId(1)), - } - } -} -impl Gid { - pub fn new_gid(&self) -> ProcessGroupId { - let mut previous = 0; - self.next_gid.0.send_modify(|x| { - previous = *x; - *x = previous + 1; - }); - ProcessGroupId(previous) - } - - pub fn new_main_gid(&self) -> ProcessGroupId { - let gid = self.new_gid(); - self.main_gid.0.send(gid).unwrap_or_default(); - gid - } -} - -/// This is the controller of the services. Here is where we can control a service with a start, stop, restart, etc. -#[derive(Clone)] -pub struct Manager { - seed: Arc, - - manage_container: Arc, - transition: Arc>, - persistent_container: ManagerPersistentContainer, - - pub gid: Arc, -} -impl Manager { - pub async fn new(ctx: RpcContext, manifest: Manifest) -> Result { - let seed = Arc::new(ManagerSeed { - ctx, - container_name: DockerProcedure::container_name(&manifest.id, None), - manifest, - }); - - let persistent_container = Arc::new(PersistentContainer::init(&seed).await?); - let manage_container = Arc::new( - manager_container::ManageContainer::new(seed.clone(), persistent_container.clone()) - .await?, - ); - let (transition, _) = watch::channel(Default::default()); - let transition = Arc::new(transition); - Ok(Self { - seed, - manage_container, - transition, - persistent_container, - gid: Default::default(), - }) - } - - /// awaiting this does not wait for the start to complete - pub async fn start(&self) { - if self._is_transition_restart() { - return; - } - self._transition_abort().await; - self.manage_container.to_desired(StartStop::Start); - } - - /// awaiting this does not wait for the stop to complete - pub async fn stop(&self) { - self._transition_abort().await; - self.manage_container.to_desired(StartStop::Stop); - } - /// awaiting this does not wait for the restart to complete - pub async fn restart(&self) { - if self._is_transition_restart() - && *self.manage_container.desired_state().borrow() == StartStop::Stop - { - return; - } - if self.manage_container.desired_state().borrow().is_start() { - self._transition_replace(self._transition_restart()).await; - } - } - /// awaiting this does not wait for the restart to complete - pub async fn configure( - &self, - configure_context: ConfigureContext, - ) -> Result, Error> { - if self._is_transition_restart() { - self._transition_abort().await; - } else if self._is_transition_backup() { - return Err(Error::new( - eyre!("Can't configure because service is backing up"), - ErrorKind::InvalidRequest, - )); - } - let context = self.seed.ctx.clone(); - let id = self.seed.manifest.id.clone(); - - let breakages = configure(context, id, configure_context).await?; - - self.restart().await; - - Ok(breakages) - } - - /// awaiting this does not wait for the backup to complete - pub async fn backup(&self, backup_guard: BackupGuard) -> BackupReturn { - if self._is_transition_backup() { - return BackupReturn::AlreadyRunning(PackageBackupReport { - error: Some("Can't do backup because service is already backing up".to_owned()), - }); - } - let (transition_state, done) = self._transition_backup(backup_guard); - self._transition_replace(transition_state).await; - done.await - } - pub async fn exit(&self) { - self._transition_abort().await; - self.manage_container - .wait_for_desired(StartStop::Stop) - .await; - } - - /// A special exit that is overridden the start state, should only be called in the shutdown, where we remove other containers - async fn shutdown(&self) -> Result<(), Error> { - self.manage_container.lock_state_forever(&self.seed).await?; - - self.exit().await; - Ok(()) - } - - /// Used when we want to shutdown the service - pub async fn signal(&self, signal: Signal) -> Result<(), Error> { - let gid = self.gid.clone(); - send_signal(self, gid, signal).await - } - - /// Used as a getter, but also used in procedure - pub fn rpc_client(&self) -> Option> { - (*self.persistent_container) - .as_ref() - .map(|x| x.rpc_client()) - } - - async fn _transition_abort(&self) { - self.transition - .send_replace(Default::default()) - .abort() - .await; - } - async fn _transition_replace(&self, transition_state: TransitionState) { - self.transition.send_replace(transition_state).abort().await; - } - - pub(super) fn perform_restart(&self) -> impl Future> + 'static { - let manage_container = self.manage_container.clone(); - async move { - let restart_override = manage_container.set_override(MainStatus::Restarting)?; - manage_container.wait_for_desired(StartStop::Stop).await; - manage_container.wait_for_desired(StartStop::Start).await; - restart_override.drop(); - Ok(()) - } - } - fn _transition_restart(&self) -> TransitionState { - let transition = self.transition.clone(); - let restart = self.perform_restart(); - TransitionState::Restarting( - tokio::spawn(async move { - if let Err(err) = restart.await { - tracing::error!("Error restarting service: {}", err); - } - transition.send_replace(Default::default()); - }) - .into(), - ) - } - fn perform_backup( - &self, - backup_guard: BackupGuard, - ) -> impl Future, Error>> { - let manage_container = self.manage_container.clone(); - let seed = self.seed.clone(); - async move { - let peek = seed.ctx.db.peek().await; - let state_reverter = DesiredStateReverter::new(manage_container.clone()); - let override_guard = - manage_container.set_override(get_status(peek, &seed.manifest).backing_up())?; - manage_container.wait_for_desired(StartStop::Stop).await; - let backup_guard = backup_guard.lock().await; - let guard = backup_guard.mount_package_backup(&seed.manifest.id).await?; - - let return_value = seed.manifest.backup.create(seed.clone()).await; - guard.unmount().await?; - drop(backup_guard); - - let manifest_id = seed.manifest.id.clone(); - seed.ctx - .db - .mutate(|db| { - if let Some(progress) = db - .as_server_info_mut() - .as_status_info_mut() - .as_backup_progress_mut() - .transpose_mut() - .and_then(|p| p.as_idx_mut(&manifest_id)) - { - progress.as_complete_mut().ser(&true)?; - } - Ok(()) - }) - .await?; - - state_reverter.revert().await; - - override_guard.drop(); - Ok::<_, Error>(return_value) - } - } - fn _transition_backup( - &self, - backup_guard: BackupGuard, - ) -> (TransitionState, BoxFuture) { - let (send, done) = oneshot::channel(); - - let transition_state = self.transition.clone(); - ( - TransitionState::BackingUp( - tokio::spawn( - self.perform_backup(backup_guard) - .then(finish_up_backup_task(transition_state, send)), - ) - .into(), - ), - done.map_err(|err| Error::new(eyre!("Oneshot error: {err:?}"), ErrorKind::Unknown)) - .map(flatten_backup_error) - .boxed(), - ) - } - fn _is_transition_restart(&self) -> bool { - let transition = self.transition.borrow(); - matches!(*transition, TransitionState::Restarting(_)) - } - fn _is_transition_backup(&self) -> bool { - let transition = self.transition.borrow(); - matches!(*transition, TransitionState::BackingUp(_)) - } -} - -#[instrument(skip_all)] -async fn configure( - ctx: RpcContext, - id: PackageId, - mut configure_context: ConfigureContext, -) -> Result, Error> { - let db = ctx.db.peek().await; - let id = &id; - let ctx = &ctx; - let overrides = &mut configure_context.overrides; - // fetch data from db - let manifest = db - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .as_manifest() - .de()?; - - // get current config and current spec - let ConfigRes { - config: old_config, - spec, - } = manifest - .config - .as_ref() - .or_not_found("Manifest config")? - .get(ctx, id, &manifest.version, &manifest.volumes) - .await?; - - // determine new config to use - let mut config = if let Some(config) = configure_context.config.or_else(|| old_config.clone()) { - config - } else { - spec.gen( - &mut rand::rngs::StdRng::from_entropy(), - &configure_context.timeout, - )? - }; - - spec.validate(&manifest)?; - spec.matches(&config)?; // check that new config matches spec - - // TODO Commit or not? - spec.update(ctx, &manifest, overrides, &mut config).await?; // dereference pointers in the new config - - let manifest = db - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .as_installed() - .or_not_found(id)? - .as_manifest() - .de()?; - - let dependencies = &manifest.dependencies; - let mut current_dependencies: CurrentDependencies = CurrentDependencies( - dependencies - .0 - .iter() - .filter_map(|(id, info)| { - if info.requirement.required() { - Some((id.clone(), CurrentDependencyInfo::default())) - } else { - None - } - }) - .collect(), - ); - for ptr in spec.pointers(&config)? { - match ptr { - ValueSpecPointer::Package(pkg_ptr) => { - if let Some(info) = current_dependencies.0.get_mut(pkg_ptr.package_id()) { - info.pointers.insert(pkg_ptr); - } else { - let id = pkg_ptr.package_id().to_owned(); - let mut pointers = BTreeSet::new(); - pointers.insert(pkg_ptr); - current_dependencies.0.insert( - id, - CurrentDependencyInfo { - pointers, - health_checks: BTreeSet::new(), - }, - ); - } - } - ValueSpecPointer::System(_) => (), - } - } - - let action = manifest.config.as_ref().or_not_found(id)?; - let version = &manifest.version; - let volumes = &manifest.volumes; - if !configure_context.dry_run { - // run config action - let res = action - .set(ctx, id, version, &dependencies, volumes, &config) - .await?; - - // track dependencies with no pointers - for (package_id, health_checks) in res.depends_on.into_iter() { - if let Some(current_dependency) = current_dependencies.0.get_mut(&package_id) { - current_dependency.health_checks.extend(health_checks); - } else { - current_dependencies.0.insert( - package_id, - CurrentDependencyInfo { - pointers: BTreeSet::new(), - health_checks, - }, - ); - } - } - - // track dependency health checks - current_dependencies = current_dependencies.map(|x| { - x.into_iter() - .filter(|(dep_id, _)| { - if dep_id != id && !manifest.dependencies.0.contains_key(dep_id) { - tracing::warn!("Illegal dependency specified: {}", dep_id); - false - } else { - true - } - }) - .collect() - }); - } - - let dependency_config_errs = - compute_dependency_config_errs(&ctx, &db, &manifest, ¤t_dependencies, overrides) - .await?; - - // cache current config for dependents - configure_context - .overrides - .insert(id.clone(), config.clone()); - - // handle dependents - - let dependents = db - .as_package_data() - .as_idx(id) - .or_not_found(id)? - .as_installed() - .or_not_found(id)? - .as_current_dependents() - .de()?; - for (dependent, _dep_info) in dependents.0.iter().filter(|(dep_id, _)| dep_id != &id) { - // check if config passes dependent check - if let Some(cfg) = db - .as_package_data() - .as_idx(dependent) - .or_not_found(dependent)? - .as_installed() - .or_not_found(dependent)? - .as_manifest() - .as_dependencies() - .as_idx(id) - .or_not_found(id)? - .as_config() - .de()? - { - let manifest = db - .as_package_data() - .as_idx(dependent) - .or_not_found(dependent)? - .as_installed() - .or_not_found(dependent)? - .as_manifest() - .de()?; - if let Err(error) = cfg - .check( - ctx, - dependent, - &manifest.version, - &manifest.volumes, - id, - &config, - ) - .await? - { - configure_context.breakages.insert(dependent.clone(), error); - } - } - } - - if !configure_context.dry_run { - return ctx - .db - .mutate(move |db| { - remove_from_current_dependents_lists(db, id, ¤t_dependencies)?; - add_dependent_to_current_dependents_lists(db, id, ¤t_dependencies)?; - current_dependencies.0.remove(id); - for (dep, errs) in db - .as_package_data_mut() - .as_entries_mut()? - .into_iter() - .filter_map(|(id, pde)| { - pde.as_installed_mut() - .map(|i| (id, i.as_status_mut().as_dependency_config_errors_mut())) - }) - { - errs.remove(id)?; - if let Some(err) = configure_context.breakages.get(&dep) { - errs.insert(id, err)?; - } - } - let installed = db - .as_package_data_mut() - .as_idx_mut(id) - .or_not_found(id)? - .as_installed_mut() - .or_not_found(id)?; - installed - .as_current_dependencies_mut() - .ser(¤t_dependencies)?; - let status = installed.as_status_mut(); - status.as_configured_mut().ser(&true)?; - status - .as_dependency_config_errors_mut() - .ser(&dependency_config_errs)?; - Ok(configure_context.breakages) - }) - .await; // add new - } - - Ok(configure_context.breakages) -} - -struct DesiredStateReverter { - manage_container: Option>, - starting_state: StartStop, -} -impl DesiredStateReverter { - fn new(manage_container: Arc) -> Self { - let starting_state = *manage_container.desired_state().borrow(); - let manage_container = Some(manage_container); - Self { - starting_state, - manage_container, - } - } - async fn revert(mut self) { - if let Some(mut current_state) = self._revert() { - while *current_state.borrow() != self.starting_state { - current_state.changed().await.unwrap(); - } - } - } - fn _revert(&mut self) -> Option> { - if let Some(manage_container) = self.manage_container.take() { - manage_container.to_desired(self.starting_state); - - return Some(manage_container.desired_state()); - } - None - } -} -impl Drop for DesiredStateReverter { - fn drop(&mut self) { - self._revert(); - } -} - -type BackupDoneSender = oneshot::Sender>; - -fn finish_up_backup_task( - transition: Arc>, - send: BackupDoneSender, -) -> impl FnOnce(Result, Error>) -> BoxFuture<'static, ()> { - move |result| { - async move { - transition.send_replace(Default::default()); - send.send(match result { - Ok(a) => a, - Err(e) => Err(e), - }) - .unwrap_or_default(); - } - .boxed() - } -} - -fn response_to_report(response: &Result) -> PackageBackupReport { - PackageBackupReport { - error: response.as_ref().err().map(|e| e.to_string()), - } -} -fn flatten_backup_error(input: Result, Error>) -> BackupReturn { - match input { - Ok(a) => BackupReturn::Ran { - report: response_to_report(&a), - res: a, - }, - Err(err) => BackupReturn::Error(err), - } -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub enum Status { - Starting, - Running, - Stopped, - Paused, - Shutdown, -} - -#[derive(Debug, Clone, Copy)] -pub enum OnStop { - Restart, - Sleep, - Exit, -} - -type RunMainResult = Result, Error>; - -#[instrument(skip_all)] -async fn run_main( - seed: Arc, - persistent_container: ManagerPersistentContainer, - started: Arc, -) -> RunMainResult { - let mut runtime = NonDetachingJoinHandle::from(tokio::spawn(start_up_image(seed.clone()))); - let ip = match persistent_container.is_some() { - false => Some(match get_running_ip(&seed, &mut runtime).await { - GetRunningIp::Ip(x) => x, - GetRunningIp::Error(e) => return Err(e), - GetRunningIp::EarlyExit(x) => return Ok(x), - }), - true => None, - }; - - let svc = if let Some(ip) = ip { - let net = add_network_for_main(&seed, ip).await?; - started(); - Some(net) - } else { - None - }; - - let health = main_health_check_daemon(seed.clone()); - let res = tokio::select! { - a = runtime => a.map_err(|_| Error::new(eyre!("Manager runtime panicked!"), crate::ErrorKind::Docker)).and_then(|a| a), - _ = health => Err(Error::new(eyre!("Health check daemon exited!"), crate::ErrorKind::Unknown)) - }; - if let Some(svc) = svc { - remove_network_for_main(svc).await?; - } - res -} - -/// We want to start up the manifest, but in this case we want to know that we have generated the certificates. -/// Note for _generated_certificate: Needed to know that before we start the state we have generated the certificate -async fn start_up_image(seed: Arc) -> Result, Error> { - seed.manifest - .main - .execute::<(), NoOutput>( - &seed.ctx, - &seed.manifest.id, - &seed.manifest.version, - ProcedureName::Main, - &seed.manifest.volumes, - None, - None, - ) - .await -} - -async fn long_running_docker( - seed: &ManagerSeed, - container: &DockerContainer, -) -> Result<(LongRunning, UnixRpcClient), Error> { - container - .long_running_execute( - &seed.ctx, - &seed.manifest.id, - &seed.manifest.version, - &seed.manifest.volumes, - ) - .await -} - -enum GetRunningIp { - Ip(Ipv4Addr), - Error(Error), - EarlyExit(Result), -} - -async fn get_long_running_ip(seed: &ManagerSeed, runtime: &mut LongRunning) -> GetRunningIp { - loop { - match get_container_ip(&seed.container_name).await { - Ok(Some(ip_addr)) => return GetRunningIp::Ip(ip_addr), - Ok(None) => (), - Err(e) if e.kind == ErrorKind::NotFound => (), - Err(e) => return GetRunningIp::Error(e), - } - if let Poll::Ready(res) = futures::poll!(&mut runtime.running_output) { - match res { - Ok(_) => return GetRunningIp::EarlyExit(Ok(NoOutput)), - Err(_e) => { - return GetRunningIp::Error(Error::new( - eyre!("Manager runtime panicked!"), - crate::ErrorKind::Docker, - )) - } - } - } - } -} - -#[instrument(skip(seed))] -async fn add_network_for_main( - seed: &ManagerSeed, - ip: std::net::Ipv4Addr, -) -> Result { - let mut svc = seed - .ctx - .net_controller - .create_service(seed.manifest.id.clone(), ip) - .await?; - // DEPRECATED - let mut secrets = seed.ctx.secret_store.acquire().await?; - let mut tx = secrets.begin().await?; - for (id, interface) in &seed.manifest.interfaces.0 { - for (external, internal) in interface.lan_config.iter().flatten() { - svc.add_lan( - tx.as_mut(), - id.clone(), - external.0, - internal.internal, - Err(AlpnInfo::Specified(vec![])), - ) - .await?; - } - for (external, internal) in interface.tor_config.iter().flat_map(|t| &t.port_mapping) { - svc.add_tor(tx.as_mut(), id.clone(), external.0, internal.0) - .await?; - } - } - for volume in seed.manifest.volumes.values() { - if let Volume::Certificate { interface_id } = volume { - svc.export_cert(tx.as_mut(), interface_id, ip.into()) - .await?; - } - } - tx.commit().await?; - Ok(svc) -} - -#[instrument(skip(svc))] -async fn remove_network_for_main(svc: NetService) -> Result<(), Error> { - svc.remove_all().await -} - -async fn main_health_check_daemon(seed: Arc) { - tokio::time::sleep(Duration::from_secs(HEALTH_CHECK_GRACE_PERIOD_SECONDS)).await; - loop { - if let Err(e) = health::check(&seed.ctx, &seed.manifest.id).await { - tracing::error!( - "Failed to run health check for {}: {}", - &seed.manifest.id, - e - ); - tracing::debug!("{:?}", e); - } - tokio::time::sleep(Duration::from_secs(HEALTH_CHECK_COOLDOWN_SECONDS)).await; - } -} - -type RuntimeOfCommand = NonDetachingJoinHandle, Error>>; - -#[instrument(skip(seed, runtime))] -async fn get_running_ip(seed: &ManagerSeed, mut runtime: &mut RuntimeOfCommand) -> GetRunningIp { - loop { - match get_container_ip(&seed.container_name).await { - Ok(Some(ip_addr)) => return GetRunningIp::Ip(ip_addr), - Ok(None) => (), - Err(e) if e.kind == ErrorKind::NotFound => (), - Err(e) => return GetRunningIp::Error(e), - } - if let Poll::Ready(res) = futures::poll!(&mut runtime) { - match res { - Ok(Ok(response)) => return GetRunningIp::EarlyExit(response), - Err(e) => { - return GetRunningIp::Error(Error::new( - match e.try_into_panic() { - Ok(e) => { - eyre!( - "Manager runtime panicked: {}", - e.downcast_ref::<&'static str>().unwrap_or(&"UNKNOWN") - ) - } - _ => eyre!("Manager runtime cancelled!"), - }, - crate::ErrorKind::Docker, - )) - } - Ok(Err(e)) => { - return GetRunningIp::Error(Error::new( - eyre!("Manager runtime returned error: {}", e), - crate::ErrorKind::Docker, - )) - } - } - } - } -} - -async fn send_signal(manager: &Manager, gid: Arc, signal: Signal) -> Result<(), Error> { - // stop health checks from committing their results - // shared - // .commit_health_check_results - // .store(false, Ordering::SeqCst); - - if let Some(rpc_client) = manager.rpc_client() { - let main_gid = *gid.main_gid.0.borrow(); - let next_gid = gid.new_gid(); - #[cfg(feature = "js-engine")] - if let Err(e) = crate::procedure::js_scripts::JsProcedure::default() - .execute::<_, NoOutput>( - &manager.seed.ctx.datadir, - &manager.seed.manifest.id, - &manager.seed.manifest.version, - ProcedureName::Signal, - &manager.seed.manifest.volumes, - Some(container_init::SignalGroupParams { - gid: main_gid, - signal: signal as u32, - }), - None, // TODO - next_gid, - Some(rpc_client), - ) - .await? - { - tracing::error!("Failed to send js signal: {}", e.1); - tracing::debug!("{:?}", e); - } - } else { - // send signal to container - kill_container(&manager.seed.container_name, Some(signal)) - .await - .or_else(|e| { - if e.kind == ErrorKind::NotFound { - Ok(()) - } else { - Err(e) - } - })?; - } - - Ok(()) -} diff --git a/core/startos/src/manager/persistent_container.rs b/core/startos/src/manager/persistent_container.rs deleted file mode 100644 index d9868a62..00000000 --- a/core/startos/src/manager/persistent_container.rs +++ /dev/null @@ -1,101 +0,0 @@ -use std::sync::Arc; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use helpers::UnixRpcClient; -use tokio::sync::oneshot; -use tokio::sync::watch::{self, Receiver}; -use tracing::instrument; - -use super::manager_seed::ManagerSeed; -use super::{ - add_network_for_main, get_long_running_ip, long_running_docker, remove_network_for_main, - GetRunningIp, -}; -use crate::procedure::docker::DockerContainer; -use crate::util::NonDetachingJoinHandle; -use crate::Error; - -/// Persistant container are the old containers that need to run all the time -/// The goal is that all services will be persistent containers, waiting to run the main system. -pub struct PersistentContainer { - _running_docker: NonDetachingJoinHandle<()>, - pub rpc_client: Receiver>, -} - -impl PersistentContainer { - #[instrument(skip_all)] - pub async fn init(seed: &Arc) -> Result, Error> { - Ok(if let Some(containers) = &seed.manifest.containers { - let (running_docker, rpc_client) = - spawn_persistent_container(seed.clone(), containers.main.clone()).await?; - Some(Self { - _running_docker: running_docker, - rpc_client, - }) - } else { - None - }) - } - - pub fn rpc_client(&self) -> Arc { - self.rpc_client.borrow().clone() - } -} - -pub async fn spawn_persistent_container( - seed: Arc, - container: DockerContainer, -) -> Result<(NonDetachingJoinHandle<()>, Receiver>), Error> { - let (send_inserter, inserter) = oneshot::channel(); - Ok(( - tokio::task::spawn(async move { - let mut inserter_send: Option>> = None; - let mut send_inserter: Option>>> = Some(send_inserter); - loop { - if let Err(e) = async { - let (mut runtime, inserter) = - long_running_docker(&seed, &container).await?; - - - let ip = match get_long_running_ip(&seed, &mut runtime).await { - GetRunningIp::Ip(x) => x, - GetRunningIp::Error(e) => return Err(e), - GetRunningIp::EarlyExit(e) => { - tracing::error!("Early Exit"); - tracing::debug!("{:?}", e); - return Ok(()); - } - }; - let svc = add_network_for_main(&seed, ip).await?; - - if let Some(inserter_send) = inserter_send.as_mut() { - let _ = inserter_send.send(Arc::new(inserter)); - } else { - let (s, r) = watch::channel(Arc::new(inserter)); - inserter_send = Some(s); - if let Some(send_inserter) = send_inserter.take() { - let _ = send_inserter.send(r); - } - } - - let res = tokio::select! { - a = runtime.running_output => a.map_err(|_| Error::new(eyre!("Manager runtime panicked!"), crate::ErrorKind::Docker)).map(|_| ()), - }; - - remove_network_for_main(svc).await?; - - res - }.await { - tracing::error!("Error in persistent container: {}", e); - tracing::debug!("{:?}", e); - } else { - break; - } - tokio::time::sleep(Duration::from_millis(200)).await; - } - }) - .into(), - inserter.await.map_err(|_| Error::new(eyre!("Container handle dropped before inserter sent"), crate::ErrorKind::Unknown))?, - )) -} diff --git a/core/startos/src/manager/start_stop.rs b/core/startos/src/manager/start_stop.rs deleted file mode 100644 index 3842abe5..00000000 --- a/core/startos/src/manager/start_stop.rs +++ /dev/null @@ -1,32 +0,0 @@ -use crate::status::MainStatus; - -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum StartStop { - Start, - Stop, -} - -impl StartStop { - pub(crate) fn is_start(&self) -> bool { - matches!(self, StartStop::Start) - } -} -impl From for StartStop { - fn from(value: MainStatus) -> Self { - match value { - MainStatus::Stopped => StartStop::Stop, - MainStatus::Restarting => StartStop::Start, - MainStatus::Stopping => StartStop::Stop, - MainStatus::Starting => StartStop::Start, - MainStatus::Running { - started: _, - health: _, - } => StartStop::Start, - MainStatus::BackingUp { started, health: _ } if started.is_some() => StartStop::Start, - MainStatus::BackingUp { - started: _, - health: _, - } => StartStop::Stop, - } - } -} diff --git a/core/startos/src/manager/transition_state.rs b/core/startos/src/manager/transition_state.rs deleted file mode 100644 index 122c0f70..00000000 --- a/core/startos/src/manager/transition_state.rs +++ /dev/null @@ -1,35 +0,0 @@ -use helpers::NonDetachingJoinHandle; - -/// Used only in the manager/mod and is used to keep track of the state of the manager during the -/// transitional states -pub(super) enum TransitionState { - BackingUp(NonDetachingJoinHandle<()>), - Restarting(NonDetachingJoinHandle<()>), - None, -} - -impl TransitionState { - pub(super) fn take(&mut self) -> Self { - std::mem::take(self) - } - pub(super) fn into_join_handle(self) -> Option> { - Some(match self { - TransitionState::BackingUp(a) => a, - TransitionState::Restarting(a) => a, - TransitionState::None => return None, - }) - } - pub(super) async fn abort(&mut self) { - if let Some(s) = self.take().into_join_handle() { - if s.wait_for_abort().await.is_ok() { - tracing::trace!("transition completed before abort"); - } - } - } -} - -impl Default for TransitionState { - fn default() -> Self { - TransitionState::None - } -} diff --git a/core/startos/src/middleware/auth.rs b/core/startos/src/middleware/auth.rs deleted file mode 100644 index 611923ad..00000000 --- a/core/startos/src/middleware/auth.rs +++ /dev/null @@ -1,284 +0,0 @@ -use std::borrow::Borrow; -use std::sync::Arc; -use std::time::{Duration, Instant}; - -use basic_cookies::Cookie; -use color_eyre::eyre::eyre; -use digest::Digest; -use futures::future::BoxFuture; -use futures::FutureExt; -use http::StatusCode; -use rpc_toolkit::command_helpers::prelude::RequestParts; -use rpc_toolkit::hyper::header::COOKIE; -use rpc_toolkit::hyper::http::Error as HttpError; -use rpc_toolkit::hyper::{Body, Request, Response}; -use rpc_toolkit::rpc_server_helpers::{ - noop4, to_response, DynMiddleware, DynMiddlewareStage2, DynMiddlewareStage3, -}; -use rpc_toolkit::yajrc::RpcMethod; -use rpc_toolkit::Metadata; -use serde::{Deserialize, Serialize}; -use sha2::Sha256; -use tokio::sync::Mutex; - -use crate::context::RpcContext; -use crate::{Error, ResultExt}; - -pub const LOCAL_AUTH_COOKIE_PATH: &str = "/run/embassy/rpc.authcookie"; - -pub trait AsLogoutSessionId { - fn as_logout_session_id(self) -> String; -} - -/// Will need to know when we have logged out from a route -#[derive(Serialize, Deserialize)] -pub struct HasLoggedOutSessions(()); - -impl HasLoggedOutSessions { - pub async fn new( - logged_out_sessions: impl IntoIterator, - ctx: &RpcContext, - ) -> Result { - let mut open_authed_websockets = ctx.open_authed_websockets.lock().await; - let mut sqlx_conn = ctx.secret_store.acquire().await?; - for session in logged_out_sessions { - let session = session.as_logout_session_id(); - sqlx::query!( - "UPDATE session SET logged_out = CURRENT_TIMESTAMP WHERE id = $1", - session - ) - .execute(sqlx_conn.as_mut()) - .await?; - for socket in open_authed_websockets.remove(&session).unwrap_or_default() { - let _ = socket.send(()); - } - } - Ok(HasLoggedOutSessions(())) - } -} - -/// Used when we need to know that we have logged in with a valid user -#[derive(Clone, Copy)] -pub struct HasValidSession(()); - -impl HasValidSession { - pub async fn from_request_parts( - request_parts: &RequestParts, - ctx: &RpcContext, - ) -> Result { - if let Some(cookie_header) = request_parts.headers.get(COOKIE) { - let cookies = Cookie::parse( - cookie_header - .to_str() - .with_kind(crate::ErrorKind::Authorization)?, - ) - .with_kind(crate::ErrorKind::Authorization)?; - if let Some(cookie) = cookies.iter().find(|c| c.get_name() == "local") { - if let Ok(s) = Self::from_local(cookie).await { - return Ok(s); - } - } - if let Some(cookie) = cookies.iter().find(|c| c.get_name() == "session") { - if let Ok(s) = Self::from_session(&HashSessionToken::from_cookie(cookie), ctx).await - { - return Ok(s); - } - } - } - Err(Error::new( - eyre!("UNAUTHORIZED"), - crate::ErrorKind::Authorization, - )) - } - - pub async fn from_session(session: &HashSessionToken, ctx: &RpcContext) -> Result { - let session_hash = session.hashed(); - let session = sqlx::query!("UPDATE session SET last_active = CURRENT_TIMESTAMP WHERE id = $1 AND logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP", session_hash) - .execute(ctx.secret_store.acquire().await?.as_mut()) - .await?; - if session.rows_affected() == 0 { - return Err(Error::new( - eyre!("UNAUTHORIZED"), - crate::ErrorKind::Authorization, - )); - } - Ok(Self(())) - } - - pub async fn from_local(local: &Cookie<'_>) -> Result { - let token = tokio::fs::read_to_string(LOCAL_AUTH_COOKIE_PATH).await?; - if local.get_value() == &*token { - Ok(Self(())) - } else { - Err(Error::new( - eyre!("UNAUTHORIZED"), - crate::ErrorKind::Authorization, - )) - } - } -} - -/// When we have a need to create a new session, -/// Or when we are using internal valid authenticated service. -#[derive(Debug, Clone)] -pub struct HashSessionToken { - hashed: String, - token: String, -} -impl HashSessionToken { - pub fn new() -> Self { - let token = base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &rand::random::<[u8; 16]>(), - ) - .to_lowercase(); - let hashed = Self::hash(&token); - Self { hashed, token } - } - pub fn from_cookie(cookie: &Cookie) -> Self { - let token = cookie.get_value().to_owned(); - let hashed = Self::hash(&token); - Self { hashed, token } - } - - pub fn from_request_parts(request_parts: &RequestParts) -> Result { - if let Some(cookie_header) = request_parts.headers.get(COOKIE) { - let cookies = Cookie::parse( - cookie_header - .to_str() - .with_kind(crate::ErrorKind::Authorization)?, - ) - .with_kind(crate::ErrorKind::Authorization)?; - if let Some(session) = cookies.iter().find(|c| c.get_name() == "session") { - return Ok(Self::from_cookie(session)); - } - } - Err(Error::new( - eyre!("UNAUTHORIZED"), - crate::ErrorKind::Authorization, - )) - } - - pub fn header_value(&self) -> Result { - http::HeaderValue::from_str(&format!( - "session={}; Path=/; SameSite=Lax; Expires=Fri, 31 Dec 9999 23:59:59 GMT;", - self.token - )) - .with_kind(crate::ErrorKind::Unknown) - } - - pub fn hashed(&self) -> &str { - self.hashed.as_str() - } - - pub fn as_hash(self) -> String { - self.hashed - } - fn hash(token: &str) -> String { - let mut hasher = Sha256::new(); - hasher.update(token.as_bytes()); - base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - hasher.finalize().as_slice(), - ) - .to_lowercase() - } -} -impl AsLogoutSessionId for HashSessionToken { - fn as_logout_session_id(self) -> String { - self.hashed - } -} -impl PartialEq for HashSessionToken { - fn eq(&self, other: &Self) -> bool { - self.hashed == other.hashed - } -} -impl Eq for HashSessionToken {} -impl PartialOrd for HashSessionToken { - fn partial_cmp(&self, other: &Self) -> Option { - self.hashed.partial_cmp(&other.hashed) - } -} -impl Ord for HashSessionToken { - fn cmp(&self, other: &Self) -> std::cmp::Ordering { - self.hashed.cmp(&other.hashed) - } -} -impl Borrow for HashSessionToken { - fn borrow(&self) -> &String { - &self.hashed - } -} - -pub fn auth(ctx: RpcContext) -> DynMiddleware { - let rate_limiter = Arc::new(Mutex::new((0_usize, Instant::now()))); - Box::new( - move |req: &mut Request, - metadata: M| - -> BoxFuture>, HttpError>> { - let ctx = ctx.clone(); - let rate_limiter = rate_limiter.clone(); - async move { - let mut header_stub = Request::new(Body::empty()); - *header_stub.headers_mut() = req.headers().clone(); - let m2: DynMiddlewareStage2 = Box::new(move |req, rpc_req| { - async move { - if let Err(e) = HasValidSession::from_request_parts(req, &ctx).await { - if metadata - .get(rpc_req.method.as_str(), "authenticated") - .unwrap_or(true) - { - let (res_parts, _) = Response::new(()).into_parts(); - return Ok(Err(to_response( - &req.headers, - res_parts, - Err(e.into()), - |_| StatusCode::OK, - )?)); - } else if rpc_req.method.as_str() == "auth.login" { - let guard = rate_limiter.lock().await; - if guard.1.elapsed() < Duration::from_secs(20) { - if guard.0 >= 3 { - let (res_parts, _) = Response::new(()).into_parts(); - return Ok(Err(to_response( - &req.headers, - res_parts, - Err(Error::new( - eyre!( - "Please limit login attempts to 3 per 20 seconds." - ), - crate::ErrorKind::RateLimited, - ) - .into()), - |_| StatusCode::OK, - )?)); - } - } - } - } - let m3: DynMiddlewareStage3 = Box::new(move |_, res| { - async move { - let mut guard = rate_limiter.lock().await; - if guard.1.elapsed() < Duration::from_secs(20) { - if res.is_err() { - guard.0 += 1; - } - } else { - guard.0 = 0; - } - guard.1 = Instant::now(); - Ok(Ok(noop4())) - } - .boxed() - }); - Ok(Ok(m3)) - } - .boxed() - }); - Ok(Ok(m2)) - } - .boxed() - }, - ) -} diff --git a/core/startos/src/middleware/cors.rs b/core/startos/src/middleware/cors.rs deleted file mode 100644 index 5f33bc08..00000000 --- a/core/startos/src/middleware/cors.rs +++ /dev/null @@ -1,61 +0,0 @@ -use futures::FutureExt; -use http::HeaderValue; -use hyper::header::HeaderMap; -use rpc_toolkit::hyper::http::Error as HttpError; -use rpc_toolkit::hyper::{Body, Method, Request, Response}; -use rpc_toolkit::rpc_server_helpers::{ - DynMiddlewareStage2, DynMiddlewareStage3, DynMiddlewareStage4, -}; -use rpc_toolkit::Metadata; - -fn get_cors_headers(req: &Request) -> HeaderMap { - let mut res = HeaderMap::new(); - if let Some(origin) = req.headers().get("Origin") { - res.insert("Access-Control-Allow-Origin", origin.clone()); - } - if let Some(method) = req.headers().get("Access-Control-Request-Method") { - res.insert("Access-Control-Allow-Methods", method.clone()); - } - if let Some(headers) = req.headers().get("Access-Control-Request-Headers") { - res.insert("Access-Control-Allow-Headers", headers.clone()); - } - res.insert( - "Access-Control-Allow-Credentials", - HeaderValue::from_static("true"), - ); - res -} - -pub async fn cors( - req: &mut Request, - _metadata: M, -) -> Result>, HttpError> { - let headers = get_cors_headers(req); - if req.method() == Method::OPTIONS { - Ok(Err({ - let mut res = Response::new(Body::empty()); - res.headers_mut().extend(headers.into_iter()); - res - })) - } else { - Ok(Ok(Box::new(|_, _| { - async move { - let res: DynMiddlewareStage3 = Box::new(|_, _| { - async move { - let res: DynMiddlewareStage4 = Box::new(|res| { - async move { - res.headers_mut().extend(headers.into_iter()); - Ok::<_, HttpError>(()) - } - .boxed() - }); - Ok::<_, HttpError>(Ok(res)) - } - .boxed() - }); - Ok::<_, HttpError>(Ok(res)) - } - .boxed() - }))) - } -} diff --git a/core/startos/src/middleware/db.rs b/core/startos/src/middleware/db.rs deleted file mode 100644 index c3ceadda..00000000 --- a/core/startos/src/middleware/db.rs +++ /dev/null @@ -1,50 +0,0 @@ -use futures::future::BoxFuture; -use futures::FutureExt; -use http::HeaderValue; -use rpc_toolkit::hyper::http::Error as HttpError; -use rpc_toolkit::hyper::{Body, Request, Response}; -use rpc_toolkit::rpc_server_helpers::{ - noop4, DynMiddleware, DynMiddlewareStage2, DynMiddlewareStage3, -}; -use rpc_toolkit::yajrc::RpcMethod; -use rpc_toolkit::Metadata; - -use crate::context::RpcContext; - -pub fn db(ctx: RpcContext) -> DynMiddleware { - Box::new( - move |_: &mut Request, - metadata: M| - -> BoxFuture>, HttpError>> { - let ctx = ctx.clone(); - async move { - let m2: DynMiddlewareStage2 = Box::new(move |_req, rpc_req| { - async move { - let sync_db = metadata - .get(rpc_req.method.as_str(), "sync_db") - .unwrap_or(false); - - let m3: DynMiddlewareStage3 = Box::new(move |res, _| { - async move { - if sync_db { - res.headers.append( - "X-Patch-Sequence", - HeaderValue::from_str( - &ctx.db.sequence().await.to_string(), - )?, - ); - } - Ok(Ok(noop4())) - } - .boxed() - }); - Ok(Ok(m3)) - } - .boxed() - }); - Ok(Ok(m2)) - } - .boxed() - }, - ) -} diff --git a/core/startos/src/middleware/diagnostic.rs b/core/startos/src/middleware/diagnostic.rs deleted file mode 100644 index 959b8ea2..00000000 --- a/core/startos/src/middleware/diagnostic.rs +++ /dev/null @@ -1,39 +0,0 @@ -use futures::FutureExt; -use rpc_toolkit::hyper::http::Error as HttpError; -use rpc_toolkit::hyper::{Body, Request, Response}; -use rpc_toolkit::rpc_server_helpers::{noop4, DynMiddlewareStage2, DynMiddlewareStage3}; -use rpc_toolkit::yajrc::RpcMethod; -use rpc_toolkit::Metadata; - -use crate::Error; - -pub async fn diagnostic( - _req: &mut Request, - _metadata: M, -) -> Result>, HttpError> { - Ok(Ok(Box::new(|_, rpc_req| { - let method = rpc_req.method.as_str().to_owned(); - async move { - let res: DynMiddlewareStage3 = Box::new(|_, rpc_res| { - async move { - if let Err(e) = rpc_res { - if e.code == -32601 { - *e = Error::new( - color_eyre::eyre::eyre!( - "{} is not available on the Diagnostic API", - method - ), - crate::ErrorKind::DiagnosticMode, - ) - .into(); - } - } - Ok(Ok(noop4())) - } - .boxed() - }); - Ok::<_, HttpError>(Ok(res)) - } - .boxed() - }))) -} diff --git a/core/startos/src/middleware/encrypt.rs b/core/startos/src/middleware/encrypt.rs deleted file mode 100644 index 94167b7e..00000000 --- a/core/startos/src/middleware/encrypt.rs +++ /dev/null @@ -1,115 +0,0 @@ -use aes::cipher::{CipherKey, NewCipher, Nonce, StreamCipher}; -use aes::Aes256Ctr; -use hmac::Hmac; -use josekit::jwk::Jwk; -use serde::{Deserialize, Serialize}; -use sha2::Sha256; -use tracing::instrument; - -pub fn pbkdf2(password: impl AsRef<[u8]>, salt: impl AsRef<[u8]>) -> CipherKey { - let mut aeskey = CipherKey::::default(); - pbkdf2::pbkdf2::>( - password.as_ref(), - salt.as_ref(), - 1000, - aeskey.as_mut_slice(), - ) - .unwrap(); - aeskey -} - -pub fn encrypt_slice(input: impl AsRef<[u8]>, password: impl AsRef<[u8]>) -> Vec { - let prefix: [u8; 32] = rand::random(); - let aeskey = pbkdf2(password.as_ref(), &prefix[16..]); - let ctr = Nonce::::from_slice(&prefix[..16]); - let mut aes = Aes256Ctr::new(&aeskey, ctr); - let mut res = Vec::with_capacity(32 + input.as_ref().len()); - res.extend_from_slice(&prefix[..]); - res.extend_from_slice(input.as_ref()); - aes.apply_keystream(&mut res[32..]); - res -} - -pub fn decrypt_slice(input: impl AsRef<[u8]>, password: impl AsRef<[u8]>) -> Vec { - if input.as_ref().len() < 32 { - return Vec::new(); - } - let (prefix, rest) = input.as_ref().split_at(32); - let aeskey = pbkdf2(password.as_ref(), &prefix[16..]); - let ctr = Nonce::::from_slice(&prefix[..16]); - let mut aes = Aes256Ctr::new(&aeskey, ctr); - let mut res = rest.to_vec(); - aes.apply_keystream(&mut res); - res -} - -#[derive(Debug, Clone, Deserialize, Serialize)] -pub struct EncryptedWire { - encrypted: serde_json::Value, -} -impl EncryptedWire { - #[instrument(skip_all)] - pub fn decrypt(self, current_secret: impl AsRef) -> Option { - let current_secret = current_secret.as_ref(); - - let decrypter = match josekit::jwe::alg::ecdh_es::EcdhEsJweAlgorithm::EcdhEs - .decrypter_from_jwk(current_secret) - { - Ok(a) => a, - Err(e) => { - tracing::warn!("Could not setup awk"); - tracing::debug!("{:?}", e); - return None; - } - }; - let encrypted = match serde_json::to_string(&self.encrypted) { - Ok(a) => a, - Err(e) => { - tracing::warn!("Could not deserialize"); - tracing::debug!("{:?}", e); - - return None; - } - }; - let (decoded, _) = match josekit::jwe::deserialize_json(&encrypted, &decrypter) { - Ok(a) => a, - Err(e) => { - tracing::warn!("Could not decrypt"); - tracing::debug!("{:?}", e); - return None; - } - }; - match String::from_utf8(decoded) { - Ok(a) => Some(a), - Err(e) => { - tracing::warn!("Could not decrypt into utf8"); - tracing::debug!("{:?}", e); - return None; - } - } - } -} - -/// We created this test by first making the private key, then restoring from this private key for recreatability. -/// After this the frontend then encoded an password, then we are testing that the output that we got (hand coded) -/// will be the shape we want. -#[test] -fn test_gen_awk() { - let private_key: Jwk = serde_json::from_str( - r#"{ - "kty": "EC", - "crv": "P-256", - "d": "3P-MxbUJtEhdGGpBCRFXkUneGgdyz_DGZWfIAGSCHOU", - "x": "yHTDYSfjU809fkSv9MmN4wuojf5c3cnD7ZDN13n-jz4", - "y": "8Mpkn744A5KDag0DmX2YivB63srjbugYZzWc3JOpQXI" - }"#, - ) - .unwrap(); - let encrypted: EncryptedWire = serde_json::from_str(r#"{ - "encrypted": { "protected": "eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiRUNESC1FUyIsImtpZCI6ImgtZnNXUVh2Tm95dmJEazM5dUNsQ0NUdWc5N3MyZnJockJnWUVBUWVtclUiLCJlcGsiOnsia3R5IjoiRUMiLCJjcnYiOiJQLTI1NiIsIngiOiJmRkF0LXNWYWU2aGNkdWZJeUlmVVdUd3ZvWExaTkdKRHZIWVhIckxwOXNNIiwieSI6IjFvVFN6b00teHlFZC1SLUlBaUFHdXgzS1dJZmNYZHRMQ0JHLUh6MVkzY2sifX0", "iv": "NbwvfvWOdLpZfYRIZUrkcw", "ciphertext": "Zc5Br5kYOlhPkIjQKOLMJw", "tag": "EPoch52lDuCsbUUulzZGfg" } - }"#).unwrap(); - assert_eq!( - "testing12345", - &encrypted.decrypt(std::sync::Arc::new(private_key)).unwrap() - ); -} diff --git a/core/startos/src/middleware/mod.rs b/core/startos/src/middleware/mod.rs deleted file mode 100644 index 5af2b812..00000000 --- a/core/startos/src/middleware/mod.rs +++ /dev/null @@ -1,5 +0,0 @@ -pub mod auth; -pub mod cors; -pub mod db; -pub mod diagnostic; -pub mod encrypt; diff --git a/core/startos/src/migrate.load b/core/startos/src/migrate.load deleted file mode 100644 index 16e7e7c0..00000000 --- a/core/startos/src/migrate.load +++ /dev/null @@ -1,7 +0,0 @@ -load database - from sqlite://{sqlite_path} - into postgresql://root@unix:/var/run/postgresql:5432/secrets - - with include no drop, truncate, reset sequences, data only, workers = 1, concurrency = 1, max parallel create index = 1, batch rows = {batch_rows}, prefetch rows = {prefetch_rows} - - excluding table names like '_sqlx_migrations', 'notifications'; diff --git a/core/startos/src/migration.rs b/core/startos/src/migration.rs deleted file mode 100644 index 13f14c7c..00000000 --- a/core/startos/src/migration.rs +++ /dev/null @@ -1,141 +0,0 @@ -use std::collections::BTreeSet; - -use color_eyre::eyre::eyre; -use emver::VersionRange; -use futures::{Future, FutureExt}; -use indexmap::IndexMap; -use models::ImageId; -use patch_db::HasModel; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::prelude::*; -use crate::procedure::docker::DockerContainers; -use crate::procedure::{PackageProcedure, ProcedureName}; -use crate::s9pk::manifest::PackageId; -use crate::util::Version; -use crate::volume::Volumes; -use crate::{Error, ResultExt}; - -#[derive(Clone, Debug, Default, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct Migrations { - pub from: IndexMap, - pub to: IndexMap, -} -impl Migrations { - #[instrument(skip_all)] - pub fn validate( - &self, - _container: &Option, - eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - ) -> Result<(), Error> { - for (version, migration) in &self.from { - migration - .validate(eos_version, volumes, image_ids, true) - .with_ctx(|_| { - ( - crate::ErrorKind::ValidateS9pk, - format!("Migration from {}", version), - ) - })?; - } - for (version, migration) in &self.to { - migration - .validate(eos_version, volumes, image_ids, true) - .with_ctx(|_| { - ( - crate::ErrorKind::ValidateS9pk, - format!("Migration to {}", version), - ) - })?; - } - Ok(()) - } - - #[instrument(skip_all)] - pub fn from<'a>( - &'a self, - _container: &'a Option, - ctx: &'a RpcContext, - version: &'a Version, - pkg_id: &'a PackageId, - pkg_version: &'a Version, - volumes: &'a Volumes, - ) -> Option> + 'a> { - if let Some((_, migration)) = self - .from - .iter() - .find(|(range, _)| version.satisfies(*range)) - { - Some(async move { - migration - .execute( - ctx, - pkg_id, - pkg_version, - ProcedureName::Migration, // Migrations cannot be executed concurrently - volumes, - Some(version), - None, - ) - .map(|r| { - r.and_then(|r| { - r.map_err(|e| { - Error::new(eyre!("{}", e.1), crate::ErrorKind::MigrationFailed) - }) - }) - }) - .await - }) - } else { - None - } - } - - #[instrument(skip_all)] - pub fn to<'a>( - &'a self, - ctx: &'a RpcContext, - version: &'a Version, - pkg_id: &'a PackageId, - pkg_version: &'a Version, - volumes: &'a Volumes, - ) -> Option> + 'a> { - if let Some((_, migration)) = self.to.iter().find(|(range, _)| version.satisfies(*range)) { - Some(async move { - migration - .execute( - ctx, - pkg_id, - pkg_version, - ProcedureName::Migration, - volumes, - Some(version), - None, - ) - .map(|r| { - r.and_then(|r| { - r.map_err(|e| { - Error::new(eyre!("{}", e.1), crate::ErrorKind::MigrationFailed) - }) - }) - }) - .await - }) - } else { - None - } - } -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct MigrationRes { - pub configured: bool, -} diff --git a/core/startos/src/net/cert-local.csr.conf.template b/core/startos/src/net/cert-local.csr.conf.template deleted file mode 100644 index 223dd61b..00000000 --- a/core/startos/src/net/cert-local.csr.conf.template +++ /dev/null @@ -1,10 +0,0 @@ -[req] -default_bits = 4096 -default_md = sha256 -distinguished_name = req_distinguished_name -prompt = no - -[req_distinguished_name] -CN = {hostname}.local -O = Start9 Labs -OU = StartOS \ No newline at end of file diff --git a/core/startos/src/net/dhcp.rs b/core/startos/src/net/dhcp.rs deleted file mode 100644 index cbe7ff19..00000000 --- a/core/startos/src/net/dhcp.rs +++ /dev/null @@ -1,82 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; -use std::net::IpAddr; - -use futures::TryStreamExt; -use rpc_toolkit::command; -use tokio::sync::RwLock; - -use crate::context::RpcContext; -use crate::db::model::IpInfo; -use crate::net::utils::{iface_is_physical, list_interfaces}; -use crate::prelude::*; -use crate::util::display_none; -use crate::Error; - -lazy_static::lazy_static! { - static ref CACHED_IPS: RwLock> = RwLock::new(BTreeSet::new()); -} - -async fn _ips() -> Result, Error> { - Ok(init_ips() - .await? - .values() - .flat_map(|i| { - std::iter::empty() - .chain(i.ipv4.map(IpAddr::from)) - .chain(i.ipv6.map(IpAddr::from)) - }) - .collect()) -} - -pub async fn ips() -> Result, Error> { - let ips = CACHED_IPS.read().await.clone(); - if !ips.is_empty() { - return Ok(ips); - } - let ips = _ips().await?; - *CACHED_IPS.write().await = ips.clone(); - Ok(ips) -} - -pub async fn init_ips() -> Result, Error> { - let mut res = BTreeMap::new(); - let mut ifaces = list_interfaces(); - while let Some(iface) = ifaces.try_next().await? { - if iface_is_physical(&iface).await { - let ip_info = IpInfo::for_interface(&iface).await?; - res.insert(iface, ip_info); - } - } - Ok(res) -} - -#[command(subcommands(update))] -pub async fn dhcp() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_none))] -pub async fn update(#[context] ctx: RpcContext, #[arg] interface: String) -> Result<(), Error> { - if iface_is_physical(&interface).await { - let ip_info = IpInfo::for_interface(&interface).await?; - ctx.db - .mutate(|db| { - db.as_server_info_mut() - .as_ip_info_mut() - .insert(&interface, &ip_info) - }) - .await?; - - let mut cached = CACHED_IPS.write().await; - if cached.is_empty() { - *cached = _ips().await?; - } else { - cached.extend( - std::iter::empty() - .chain(ip_info.ipv4.map(IpAddr::from)) - .chain(ip_info.ipv6.map(IpAddr::from)), - ); - } - } - Ok(()) -} diff --git a/core/startos/src/net/dns.rs b/core/startos/src/net/dns.rs deleted file mode 100644 index 7b2784a5..00000000 --- a/core/startos/src/net/dns.rs +++ /dev/null @@ -1,228 +0,0 @@ -use std::borrow::Borrow; -use std::collections::BTreeMap; -use std::net::{Ipv4Addr, SocketAddr}; -use std::sync::{Arc, Weak}; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use futures::TryFutureExt; -use helpers::NonDetachingJoinHandle; -use models::PackageId; -use tokio::net::{TcpListener, UdpSocket}; -use tokio::process::Command; -use tokio::sync::RwLock; -use tracing::instrument; -use trust_dns_server::authority::MessageResponseBuilder; -use trust_dns_server::proto::op::{Header, ResponseCode}; -use trust_dns_server::proto::rr::{Name, Record, RecordType}; -use trust_dns_server::server::{Request, RequestHandler, ResponseHandler, ResponseInfo}; -use trust_dns_server::ServerFuture; - -use crate::util::Invoke; -use crate::{Error, ErrorKind, ResultExt}; - -pub struct DnsController { - services: Weak, BTreeMap>>>>, - #[allow(dead_code)] - dns_server: NonDetachingJoinHandle>, -} - -struct Resolver { - services: Arc, BTreeMap>>>>, -} -impl Resolver { - async fn resolve(&self, name: &Name) -> Option> { - match name.iter().next_back() { - Some(b"embassy") => { - if let Some(pkg) = name.iter().rev().skip(1).next() { - if let Some(ip) = self.services.read().await.get(&Some( - std::str::from_utf8(pkg) - .unwrap_or_default() - .parse() - .unwrap_or_default(), - )) { - Some( - ip.iter() - .filter(|(_, rc)| rc.strong_count() > 0) - .map(|(ip, _)| *ip) - .collect(), - ) - } else { - None - } - } else if let Some(ip) = self.services.read().await.get(&None) { - Some( - ip.iter() - .filter(|(_, rc)| rc.strong_count() > 0) - .map(|(ip, _)| *ip) - .collect(), - ) - } else { - None - } - } - _ => None, - } - } -} - -#[async_trait::async_trait] -impl RequestHandler for Resolver { - async fn handle_request( - &self, - request: &Request, - mut response_handle: R, - ) -> ResponseInfo { - let query = request.request_info().query; - if let Some(ip) = self.resolve(query.name().borrow()).await { - match query.query_type() { - RecordType::A => { - response_handle - .send_response( - MessageResponseBuilder::from_message_request(&*request).build( - Header::response_from_request(request.header()), - &ip.into_iter() - .map(|ip| { - Record::from_rdata( - request.request_info().query.name().to_owned().into(), - 0, - trust_dns_server::proto::rr::RData::A(ip.into()), - ) - }) - .collect::>(), - [], - [], - [], - ), - ) - .await - } - a => { - if a != RecordType::AAAA { - tracing::warn!( - "Non A-Record requested for {}: {:?}", - query.name(), - query.query_type() - ); - } - let mut res = Header::response_from_request(request.header()); - res.set_response_code(ResponseCode::NXDomain); - response_handle - .send_response( - MessageResponseBuilder::from_message_request(&*request).build( - res.into(), - [], - [], - [], - [], - ), - ) - .await - } - } - } else { - let mut res = Header::response_from_request(request.header()); - res.set_response_code(ResponseCode::NXDomain); - response_handle - .send_response( - MessageResponseBuilder::from_message_request(&*request).build( - res.into(), - [], - [], - [], - [], - ), - ) - .await - } - .unwrap_or_else(|e| { - tracing::error!("{}", e); - tracing::debug!("{:?}", e); - let mut res = Header::response_from_request(request.header()); - res.set_response_code(ResponseCode::ServFail); - res.into() - }) - } -} - -impl DnsController { - #[instrument(skip_all)] - pub async fn init(bind: &[SocketAddr]) -> Result { - let services = Arc::new(RwLock::new(BTreeMap::new())); - - let mut server = ServerFuture::new(Resolver { - services: services.clone(), - }); - server.register_listener( - TcpListener::bind(bind) - .await - .with_kind(ErrorKind::Network)?, - Duration::from_secs(30), - ); - server.register_socket(UdpSocket::bind(bind).await.with_kind(ErrorKind::Network)?); - - Command::new("resolvectl") - .arg("dns") - .arg("br-start9") - .arg("127.0.0.1") - .invoke(ErrorKind::Network) - .await?; - Command::new("resolvectl") - .arg("domain") - .arg("br-start9") - .arg("embassy") - .invoke(ErrorKind::Network) - .await?; - - let dns_server = tokio::spawn( - server - .block_until_done() - .map_err(|e| Error::new(e, ErrorKind::Network)), - ) - .into(); - - Ok(Self { - services: Arc::downgrade(&services), - dns_server, - }) - } - - pub async fn add(&self, pkg_id: Option, ip: Ipv4Addr) -> Result, Error> { - if let Some(services) = Weak::upgrade(&self.services) { - let mut writable = services.write().await; - let mut ips = writable.remove(&pkg_id).unwrap_or_default(); - let rc = if let Some(rc) = Weak::upgrade(&ips.remove(&ip).unwrap_or_default()) { - rc - } else { - Arc::new(()) - }; - ips.insert(ip, Arc::downgrade(&rc)); - writable.insert(pkg_id, ips); - Ok(rc) - } else { - Err(Error::new( - eyre!("DNS Server Thread has exited"), - crate::ErrorKind::Network, - )) - } - } - - pub async fn gc(&self, pkg_id: Option, ip: Ipv4Addr) -> Result<(), Error> { - if let Some(services) = Weak::upgrade(&self.services) { - let mut writable = services.write().await; - let mut ips = writable.remove(&pkg_id).unwrap_or_default(); - if let Some(rc) = Weak::upgrade(&ips.remove(&ip).unwrap_or_default()) { - ips.insert(ip, Arc::downgrade(&rc)); - } - if !ips.is_empty() { - writable.insert(pkg_id, ips); - } - Ok(()) - } else { - Err(Error::new( - eyre!("DNS Server Thread has exited"), - crate::ErrorKind::Network, - )) - } - } -} diff --git a/core/startos/src/net/interface.rs b/core/startos/src/net/interface.rs deleted file mode 100644 index a055bb27..00000000 --- a/core/startos/src/net/interface.rs +++ /dev/null @@ -1,122 +0,0 @@ -use std::collections::BTreeMap; - -use indexmap::IndexSet; -pub use models::InterfaceId; -use serde::{Deserialize, Deserializer, Serialize}; -use sqlx::{Executor, Postgres}; -use tracing::instrument; - -use crate::db::model::{InterfaceAddressMap, InterfaceAddresses}; -use crate::net::keys::Key; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::Port; -use crate::{Error, ResultExt}; - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Interfaces(pub BTreeMap); // TODO -impl Interfaces { - #[instrument(skip_all)] - pub fn validate(&self) -> Result<(), Error> { - for (_, interface) in &self.0 { - interface.validate().with_ctx(|_| { - ( - crate::ErrorKind::ValidateS9pk, - format!("Interface {}", interface.name), - ) - })?; - } - Ok(()) - } - #[instrument(skip_all)] - pub async fn install( - &self, - secrets: &mut Ex, - package_id: &PackageId, - ) -> Result - where - for<'a> &'a mut Ex: Executor<'a, Database = Postgres>, - { - let mut interface_addresses = InterfaceAddressMap(BTreeMap::new()); - for (id, iface) in &self.0 { - let mut addrs = InterfaceAddresses { - tor_address: None, - lan_address: None, - }; - if iface.tor_config.is_some() || iface.lan_config.is_some() { - let key = - Key::for_interface(secrets, Some((package_id.clone(), id.clone()))).await?; - if iface.tor_config.is_some() { - addrs.tor_address = Some(key.tor_address().to_string()); - } - if iface.lan_config.is_some() { - addrs.lan_address = Some(key.local_address()); - } - } - interface_addresses.0.insert(id.clone(), addrs); - } - Ok(interface_addresses) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Interface { - pub name: String, - pub description: String, - pub tor_config: Option, - pub lan_config: Option>, - pub ui: bool, - pub protocols: IndexSet, -} -impl Interface { - #[instrument(skip_all)] - pub fn validate(&self) -> Result<(), color_eyre::eyre::Report> { - if self.tor_config.is_some() && !self.protocols.contains("tcp") { - color_eyre::eyre::bail!("must support tcp to set up a tor hidden service"); - } - if self.lan_config.is_some() && !self.protocols.contains("http") { - color_eyre::eyre::bail!("must support http to set up a lan service"); - } - if self.ui && !(self.protocols.contains("http") || self.protocols.contains("https")) { - color_eyre::eyre::bail!("must support http or https to serve a ui"); - } - Ok(()) - } -} - -#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct TorConfig { - pub port_mapping: BTreeMap, -} - -#[derive(Clone, Debug, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct LanPortConfig { - pub ssl: bool, - pub internal: u16, -} -impl<'de> Deserialize<'de> for LanPortConfig { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - #[derive(Deserialize)] - #[serde(rename_all = "kebab-case")] - struct PermissiveLanPortConfig { - ssl: bool, - internal: Option, - mapping: Option, - } - - let config = PermissiveLanPortConfig::deserialize(deserializer)?; - Ok(LanPortConfig { - ssl: config.ssl, - internal: config - .internal - .or(config.mapping) - .ok_or_else(|| serde::de::Error::missing_field("internal"))?, - }) - } -} diff --git a/core/startos/src/net/keys.rs b/core/startos/src/net/keys.rs deleted file mode 100644 index 504bd276..00000000 --- a/core/startos/src/net/keys.rs +++ /dev/null @@ -1,385 +0,0 @@ -use std::collections::BTreeMap; - -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use models::{Id, InterfaceId, PackageId}; -use openssl::pkey::{PKey, Private}; -use openssl::sha::Sha256; -use openssl::x509::X509; -use p256::elliptic_curve::pkcs8::EncodePrivateKey; -use rpc_toolkit::command; -use sqlx::{Acquire, PgExecutor}; -use ssh_key::private::Ed25519PrivateKey; -use torut::onion::{OnionAddressV3, TorSecretKeyV3}; -use zeroize::Zeroize; - -use crate::config::{configure, ConfigureContext}; -use crate::context::RpcContext; -use crate::control::restart; -use crate::disk::fsck::RequiresReboot; -use crate::net::ssl::CertPair; -use crate::prelude::*; -use crate::util::crypto::ed25519_expand_key; - -// TODO: delete once we may change tor addresses -async fn compat( - secrets: impl PgExecutor<'_>, - interface: &Option<(PackageId, InterfaceId)>, -) -> Result, Error> { - if let Some((package, interface)) = interface { - if let Some(r) = sqlx::query!( - "SELECT key FROM tor WHERE package = $1 AND interface = $2", - package, - interface - ) - .fetch_optional(secrets) - .await? - { - Ok(Some(<[u8; 64]>::try_from(r.key).map_err(|e| { - Error::new( - eyre!("expected vec of len 64, got len {}", e.len()), - ErrorKind::ParseDbField, - ) - })?)) - } else { - Ok(None) - } - } else if let Some(key) = sqlx::query!("SELECT tor_key FROM account WHERE id = 0") - .fetch_one(secrets) - .await? - .tor_key - { - Ok(Some(<[u8; 64]>::try_from(key).map_err(|e| { - Error::new( - eyre!("expected vec of len 64, got len {}", e.len()), - ErrorKind::ParseDbField, - ) - })?)) - } else { - Ok(None) - } -} - -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct Key { - interface: Option<(PackageId, InterfaceId)>, - base: [u8; 32], - tor_key: [u8; 64], // Does NOT necessarily match base -} -impl Key { - pub fn interface(&self) -> Option<(PackageId, InterfaceId)> { - self.interface.clone() - } - pub fn as_bytes(&self) -> [u8; 32] { - self.base - } - pub fn internal_address(&self) -> String { - self.interface - .as_ref() - .map(|(pkg_id, _)| format!("{}.embassy", pkg_id)) - .unwrap_or_else(|| "embassy".to_owned()) - } - pub fn tor_key(&self) -> TorSecretKeyV3 { - self.tor_key.into() - } - pub fn tor_address(&self) -> OnionAddressV3 { - self.tor_key().public().get_onion_address() - } - pub fn base_address(&self) -> String { - self.tor_key() - .public() - .get_onion_address() - .get_address_without_dot_onion() - } - pub fn local_address(&self) -> String { - self.base_address() + ".local" - } - pub fn openssl_key_ed25519(&self) -> PKey { - PKey::private_key_from_raw_bytes(&self.base, openssl::pkey::Id::ED25519).unwrap() - } - pub fn openssl_key_nistp256(&self) -> PKey { - let mut buf = self.base; - loop { - if let Ok(k) = p256::SecretKey::from_slice(&buf) { - return PKey::private_key_from_pkcs8(&*k.to_pkcs8_der().unwrap().as_bytes()) - .unwrap(); - } - let mut sha = Sha256::new(); - sha.update(&buf); - buf = sha.finish(); - } - } - pub fn ssh_key(&self) -> Ed25519PrivateKey { - Ed25519PrivateKey::from_bytes(&self.base) - } - pub(crate) fn from_pair( - interface: Option<(PackageId, InterfaceId)>, - bytes: [u8; 32], - tor_key: [u8; 64], - ) -> Self { - Self { - interface, - tor_key, - base: bytes, - } - } - pub fn from_bytes(interface: Option<(PackageId, InterfaceId)>, bytes: [u8; 32]) -> Self { - Self::from_pair(interface, bytes, ed25519_expand_key(&bytes)) - } - pub fn new(interface: Option<(PackageId, InterfaceId)>) -> Self { - Self::from_bytes(interface, rand::random()) - } - pub(super) fn with_certs(self, certs: CertPair, int: X509, root: X509) -> KeyInfo { - KeyInfo { - key: self, - certs, - int, - root, - } - } - pub async fn for_package( - secrets: impl PgExecutor<'_>, - package: &PackageId, - ) -> Result, Error> { - sqlx::query!( - r#" - SELECT - network_keys.package, - network_keys.interface, - network_keys.key, - tor.key AS "tor_key?" - FROM - network_keys - LEFT JOIN - tor - ON - network_keys.package = tor.package - AND - network_keys.interface = tor.interface - WHERE - network_keys.package = $1 - "#, - package - ) - .fetch_all(secrets) - .await? - .into_iter() - .map(|row| { - let interface = Some(( - package.clone(), - InterfaceId::from(Id::try_from(row.interface)?), - )); - let bytes = row.key.try_into().map_err(|e: Vec| { - Error::new( - eyre!("Invalid length for network key {} expected 32", e.len()), - crate::ErrorKind::Database, - ) - })?; - Ok(match row.tor_key { - Some(tor_key) => Key::from_pair( - interface, - bytes, - tor_key.try_into().map_err(|e: Vec| { - Error::new( - eyre!("Invalid length for tor key {} expected 64", e.len()), - crate::ErrorKind::Database, - ) - })?, - ), - None => Key::from_bytes(interface, bytes), - }) - }) - .collect() - } - pub async fn for_interface( - secrets: &mut Ex, - interface: Option<(PackageId, InterfaceId)>, - ) -> Result - where - for<'a> &'a mut Ex: PgExecutor<'a>, - { - let tentative = rand::random::<[u8; 32]>(); - let actual = if let Some((pkg, iface)) = &interface { - let k = tentative.as_slice(); - let actual = sqlx::query!( - "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO UPDATE SET package = EXCLUDED.package RETURNING key", - pkg, - iface, - k, - ) - .fetch_one(&mut *secrets) - .await?.key; - let mut bytes = tentative; - bytes.clone_from_slice(actual.get(0..32).ok_or_else(|| { - Error::new( - eyre!("Invalid key size returned from DB"), - crate::ErrorKind::Database, - ) - })?); - bytes - } else { - let actual = sqlx::query!("SELECT network_key FROM account WHERE id = 0") - .fetch_one(&mut *secrets) - .await? - .network_key; - let mut bytes = tentative; - bytes.clone_from_slice(actual.get(0..32).ok_or_else(|| { - Error::new( - eyre!("Invalid key size returned from DB"), - crate::ErrorKind::Database, - ) - })?); - bytes - }; - let mut res = Self::from_bytes(interface, actual); - if let Some(tor_key) = compat(secrets, &res.interface).await? { - res.tor_key = tor_key; - } - Ok(res) - } -} -impl Drop for Key { - fn drop(&mut self) { - self.base.zeroize(); - self.tor_key.zeroize(); - } -} - -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] -pub struct KeyInfo { - key: Key, - certs: CertPair, - int: X509, - root: X509, -} -impl KeyInfo { - pub fn key(&self) -> &Key { - &self.key - } - pub fn certs(&self) -> &CertPair { - &self.certs - } - pub fn int_ca(&self) -> &X509 { - &self.int - } - pub fn root_ca(&self) -> &X509 { - &self.root - } - pub fn fullchain_ed25519(&self) -> Vec<&X509> { - vec![&self.certs.ed25519, &self.int, &self.root] - } - pub fn fullchain_nistp256(&self) -> Vec<&X509> { - vec![&self.certs.nistp256, &self.int, &self.root] - } -} - -#[test] -pub fn test_keygen() { - let key = Key::new(None); - key.tor_key(); - key.openssl_key_nistp256(); -} - -fn display_requires_reboot(arg: RequiresReboot, _matches: &ArgMatches) { - if arg.0 { - println!("Server must be restarted for changes to take effect"); - } -} - -#[command(rename = "rotate-key", display(display_requires_reboot))] -pub async fn rotate_key( - #[context] ctx: RpcContext, - #[arg] package: Option, - #[arg] interface: Option, -) -> Result { - let mut pgcon = ctx.secret_store.acquire().await?; - let mut tx = pgcon.begin().await?; - if let Some(package) = package { - let Some(interface) = interface else { - return Err(Error::new( - eyre!("Must specify interface"), - ErrorKind::InvalidRequest, - )); - }; - sqlx::query!( - "DELETE FROM tor WHERE package = $1 AND interface = $2", - &package, - &interface, - ) - .execute(&mut *tx) - .await?; - sqlx::query!( - "DELETE FROM network_keys WHERE package = $1 AND interface = $2", - &package, - &interface, - ) - .execute(&mut *tx) - .await?; - let new_key = - Key::for_interface(&mut *tx, Some((package.clone(), interface.clone()))).await?; - let needs_config = ctx - .db - .mutate(|v| { - let installed = v - .as_package_data_mut() - .as_idx_mut(&package) - .or_not_found(&package)? - .as_installed_mut() - .or_not_found("installed")?; - let addrs = installed - .as_interface_addresses_mut() - .as_idx_mut(&interface) - .or_not_found(&interface)?; - if let Some(lan) = addrs.as_lan_address_mut().transpose_mut() { - lan.ser(&new_key.local_address())?; - } - if let Some(lan) = addrs.as_tor_address_mut().transpose_mut() { - lan.ser(&new_key.tor_address().to_string())?; - } - - if installed - .as_manifest() - .as_config() - .transpose_ref() - .is_some() - { - installed - .as_status_mut() - .as_configured_mut() - .replace(&false) - } else { - Ok(false) - } - }) - .await?; - tx.commit().await?; - if needs_config { - configure( - &ctx, - &package, - ConfigureContext { - breakages: BTreeMap::new(), - timeout: None, - config: None, - overrides: BTreeMap::new(), - dry_run: false, - }, - ) - .await?; - } else { - restart(ctx, package).await?; - } - Ok(RequiresReboot(false)) - } else { - sqlx::query!("UPDATE account SET tor_key = NULL, network_key = gen_random_bytes(32)") - .execute(&mut *tx) - .await?; - let new_key = Key::for_interface(&mut *tx, None).await?; - let url = format!("https://{}", new_key.tor_address()).parse()?; - ctx.db - .mutate(|v| v.as_server_info_mut().as_tor_address_mut().ser(&url)) - .await?; - tx.commit().await?; - Ok(RequiresReboot(true)) - } -} diff --git a/core/startos/src/net/mdns.rs b/core/startos/src/net/mdns.rs deleted file mode 100644 index 21054241..00000000 --- a/core/startos/src/net/mdns.rs +++ /dev/null @@ -1,100 +0,0 @@ -use std::collections::BTreeMap; -use std::net::Ipv4Addr; -use std::sync::{Arc, Weak}; - -use color_eyre::eyre::eyre; -use tokio::process::{Child, Command}; -use tokio::sync::Mutex; -use tracing::instrument; - -use crate::util::Invoke; -use crate::{Error, ResultExt}; - -pub async fn resolve_mdns(hostname: &str) -> Result { - Ok(String::from_utf8( - Command::new("avahi-resolve-host-name") - .kill_on_drop(true) - .arg("-4") - .arg(hostname) - .invoke(crate::ErrorKind::Network) - .await?, - )? - .split_once("\t") - .ok_or_else(|| { - Error::new( - eyre!("Failed to resolve hostname: {}", hostname), - crate::ErrorKind::Network, - ) - })? - .1 - .trim() - .parse()?) -} - -pub struct MdnsController(Mutex); -impl MdnsController { - pub async fn init() -> Result { - Ok(MdnsController(Mutex::new( - MdnsControllerInner::init().await?, - ))) - } - pub async fn add(&self, alias: String) -> Result, Error> { - self.0.lock().await.add(alias).await - } - pub async fn gc(&self, alias: String) -> Result<(), Error> { - self.0.lock().await.gc(alias).await - } -} - -pub struct MdnsControllerInner { - alias_cmd: Option, - services: BTreeMap>, -} - -impl MdnsControllerInner { - #[instrument(skip_all)] - async fn init() -> Result { - let mut res = MdnsControllerInner { - alias_cmd: None, - services: BTreeMap::new(), - }; - res.sync().await?; - Ok(res) - } - #[instrument(skip_all)] - async fn sync(&mut self) -> Result<(), Error> { - if let Some(mut cmd) = self.alias_cmd.take() { - cmd.kill().await.with_kind(crate::ErrorKind::Network)?; - } - self.alias_cmd = Some( - Command::new("avahi-alias") - .kill_on_drop(true) - .args( - self.services - .iter() - .filter(|(_, rc)| rc.strong_count() > 0) - .map(|(s, _)| s), - ) - .spawn()?, - ); - Ok(()) - } - async fn add(&mut self, alias: String) -> Result, Error> { - let rc = if let Some(rc) = Weak::upgrade(&self.services.remove(&alias).unwrap_or_default()) - { - rc - } else { - Arc::new(()) - }; - self.services.insert(alias, Arc::downgrade(&rc)); - self.sync().await?; - Ok(rc) - } - async fn gc(&mut self, alias: String) -> Result<(), Error> { - if let Some(rc) = Weak::upgrade(&self.services.remove(&alias).unwrap_or_default()) { - self.services.insert(alias, Arc::downgrade(&rc)); - } - self.sync().await?; - Ok(()) - } -} diff --git a/core/startos/src/net/mod.rs b/core/startos/src/net/mod.rs deleted file mode 100644 index 50935fb1..00000000 --- a/core/startos/src/net/mod.rs +++ /dev/null @@ -1,32 +0,0 @@ -use std::sync::Arc; - -use futures::future::BoxFuture; -use hyper::{Body, Error as HyperError, Request, Response}; -use rpc_toolkit::command; - -use crate::Error; - -pub mod dhcp; -pub mod dns; -pub mod interface; -pub mod keys; -pub mod mdns; -pub mod net_controller; -pub mod ssl; -pub mod static_server; -pub mod tor; -pub mod utils; -pub mod vhost; -pub mod web_server; -pub mod wifi; - -pub const PACKAGE_CERT_PATH: &str = "/var/lib/embassy/ssl"; - -#[command(subcommands(tor::tor, dhcp::dhcp, ssl::ssl, keys::rotate_key))] -pub fn net() -> Result<(), Error> { - Ok(()) -} - -pub type HttpHandler = Arc< - dyn Fn(Request) -> BoxFuture<'static, Result, HyperError>> + Send + Sync, ->; diff --git a/core/startos/src/net/net_controller.rs b/core/startos/src/net/net_controller.rs deleted file mode 100644 index e2e77ed6..00000000 --- a/core/startos/src/net/net_controller.rs +++ /dev/null @@ -1,369 +0,0 @@ -use std::collections::BTreeMap; -use std::net::{IpAddr, Ipv4Addr, SocketAddr}; -use std::sync::{Arc, Weak}; - -use color_eyre::eyre::eyre; -use models::InterfaceId; -use sqlx::PgExecutor; -use tracing::instrument; - -use crate::error::ErrorCollection; -use crate::hostname::Hostname; -use crate::net::dns::DnsController; -use crate::net::keys::Key; -use crate::net::mdns::MdnsController; -use crate::net::ssl::{export_cert, export_key, SslManager}; -use crate::net::tor::TorController; -use crate::net::vhost::{AlpnInfo, VHostController}; -use crate::s9pk::manifest::PackageId; -use crate::volume::cert_dir; -use crate::{Error, HOST_IP}; - -pub struct NetController { - pub(super) tor: TorController, - pub(super) mdns: MdnsController, - pub(super) vhost: VHostController, - pub(super) dns: DnsController, - pub(super) ssl: Arc, - pub(super) os_bindings: Vec>, -} - -impl NetController { - #[instrument(skip_all)] - pub async fn init( - tor_control: SocketAddr, - tor_socks: SocketAddr, - dns_bind: &[SocketAddr], - ssl: SslManager, - hostname: &Hostname, - os_key: &Key, - ) -> Result { - let ssl = Arc::new(ssl); - let mut res = Self { - tor: TorController::new(tor_control, tor_socks), - mdns: MdnsController::init().await?, - vhost: VHostController::new(ssl.clone()), - dns: DnsController::init(dns_bind).await?, - ssl, - os_bindings: Vec::new(), - }; - res.add_os_bindings(hostname, os_key).await?; - Ok(res) - } - - async fn add_os_bindings(&mut self, hostname: &Hostname, key: &Key) -> Result<(), Error> { - let alpn = Err(AlpnInfo::Specified(vec!["http/1.1".into(), "h2".into()])); - - // Internal DNS - self.vhost - .add( - key.clone(), - Some("embassy".into()), - 443, - ([127, 0, 0, 1], 80).into(), - alpn.clone(), - ) - .await?; - self.os_bindings - .push(self.dns.add(None, HOST_IP.into()).await?); - - // LAN IP - self.os_bindings.push( - self.vhost - .add( - key.clone(), - None, - 443, - ([127, 0, 0, 1], 80).into(), - alpn.clone(), - ) - .await?, - ); - - // localhost - self.os_bindings.push( - self.vhost - .add( - key.clone(), - Some("localhost".into()), - 443, - ([127, 0, 0, 1], 80).into(), - alpn.clone(), - ) - .await?, - ); - self.os_bindings.push( - self.vhost - .add( - key.clone(), - Some(hostname.no_dot_host_name()), - 443, - ([127, 0, 0, 1], 80).into(), - alpn.clone(), - ) - .await?, - ); - - // LAN mDNS - self.os_bindings.push( - self.vhost - .add( - key.clone(), - Some(hostname.local_domain_name()), - 443, - ([127, 0, 0, 1], 80).into(), - alpn.clone(), - ) - .await?, - ); - - // Tor (http) - self.os_bindings.push( - self.tor - .add(key.tor_key(), 80, ([127, 0, 0, 1], 80).into()) - .await?, - ); - - // Tor (https) - self.os_bindings.push( - self.vhost - .add( - key.clone(), - Some(key.tor_address().to_string()), - 443, - ([127, 0, 0, 1], 80).into(), - alpn.clone(), - ) - .await?, - ); - self.os_bindings.push( - self.tor - .add(key.tor_key(), 443, ([127, 0, 0, 1], 443).into()) - .await?, - ); - - Ok(()) - } - - #[instrument(skip_all)] - pub async fn create_service( - self: &Arc, - package: PackageId, - ip: Ipv4Addr, - ) -> Result { - let dns = self.dns.add(Some(package.clone()), ip).await?; - - Ok(NetService { - shutdown: false, - id: package, - ip, - dns, - controller: Arc::downgrade(self), - tor: BTreeMap::new(), - lan: BTreeMap::new(), - }) - } - - async fn add_tor( - &self, - key: &Key, - external: u16, - target: SocketAddr, - ) -> Result>, Error> { - let mut rcs = Vec::with_capacity(1); - rcs.push(self.tor.add(key.tor_key(), external, target).await?); - Ok(rcs) - } - - async fn remove_tor(&self, key: &Key, external: u16, rcs: Vec>) -> Result<(), Error> { - drop(rcs); - self.tor.gc(Some(key.tor_key()), Some(external)).await - } - - async fn add_lan( - &self, - key: Key, - external: u16, - target: SocketAddr, - connect_ssl: Result<(), AlpnInfo>, - ) -> Result>, Error> { - let mut rcs = Vec::with_capacity(2); - rcs.push( - self.vhost - .add( - key.clone(), - Some(key.local_address()), - external, - target.into(), - connect_ssl, - ) - .await?, - ); - rcs.push(self.mdns.add(key.base_address()).await?); - Ok(rcs) - } - - async fn remove_lan(&self, key: &Key, external: u16, rcs: Vec>) -> Result<(), Error> { - drop(rcs); - self.mdns.gc(key.base_address()).await?; - self.vhost.gc(Some(key.local_address()), external).await - } -} - -pub struct NetService { - shutdown: bool, - id: PackageId, - ip: Ipv4Addr, - dns: Arc<()>, - controller: Weak, - tor: BTreeMap<(InterfaceId, u16), (Key, Vec>)>, - lan: BTreeMap<(InterfaceId, u16), (Key, Vec>)>, -} -impl NetService { - fn net_controller(&self) -> Result, Error> { - Weak::upgrade(&self.controller).ok_or_else(|| { - Error::new( - eyre!("NetController is shutdown"), - crate::ErrorKind::Network, - ) - }) - } - pub async fn add_tor( - &mut self, - secrets: &mut Ex, - id: InterfaceId, - external: u16, - internal: u16, - ) -> Result<(), Error> - where - for<'a> &'a mut Ex: PgExecutor<'a>, - { - let key = Key::for_interface(secrets, Some((self.id.clone(), id.clone()))).await?; - let ctrl = self.net_controller()?; - let tor_idx = (id, external); - let mut tor = self - .tor - .remove(&tor_idx) - .unwrap_or_else(|| (key.clone(), Vec::new())); - tor.1.append( - &mut ctrl - .add_tor(&key, external, SocketAddr::new(self.ip.into(), internal)) - .await?, - ); - self.tor.insert(tor_idx, tor); - Ok(()) - } - pub async fn remove_tor(&mut self, id: InterfaceId, external: u16) -> Result<(), Error> { - let ctrl = self.net_controller()?; - if let Some((key, rcs)) = self.tor.remove(&(id, external)) { - ctrl.remove_tor(&key, external, rcs).await?; - } - Ok(()) - } - pub async fn add_lan( - &mut self, - secrets: &mut Ex, - id: InterfaceId, - external: u16, - internal: u16, - connect_ssl: Result<(), AlpnInfo>, - ) -> Result<(), Error> - where - for<'a> &'a mut Ex: PgExecutor<'a>, - { - let key = Key::for_interface(secrets, Some((self.id.clone(), id.clone()))).await?; - let ctrl = self.net_controller()?; - let lan_idx = (id, external); - let mut lan = self - .lan - .remove(&lan_idx) - .unwrap_or_else(|| (key.clone(), Vec::new())); - lan.1.append( - &mut ctrl - .add_lan( - key, - external, - SocketAddr::new(self.ip.into(), internal), - connect_ssl, - ) - .await?, - ); - self.lan.insert(lan_idx, lan); - Ok(()) - } - pub async fn remove_lan(&mut self, id: InterfaceId, external: u16) -> Result<(), Error> { - let ctrl = self.net_controller()?; - if let Some((key, rcs)) = self.lan.remove(&(id, external)) { - ctrl.remove_lan(&key, external, rcs).await?; - } - Ok(()) - } - pub async fn export_cert( - &self, - secrets: &mut Ex, - id: &InterfaceId, - ip: IpAddr, - ) -> Result<(), Error> - where - for<'a> &'a mut Ex: PgExecutor<'a>, - { - let key = Key::for_interface(secrets, Some((self.id.clone(), id.clone()))).await?; - let ctrl = self.net_controller()?; - let cert = ctrl.ssl.with_certs(key, ip).await?; - let cert_dir = cert_dir(&self.id, id); - tokio::fs::create_dir_all(&cert_dir).await?; - export_key( - &cert.key().openssl_key_nistp256(), - &cert_dir.join(format!("{id}.key.pem")), - ) - .await?; - export_cert( - &cert.fullchain_nistp256(), - &cert_dir.join(format!("{id}.cert.pem")), - ) - .await?; // TODO: can upgrade to ed25519? - Ok(()) - } - pub async fn remove_all(mut self) -> Result<(), Error> { - self.shutdown = true; - let mut errors = ErrorCollection::new(); - if let Some(ctrl) = Weak::upgrade(&self.controller) { - for ((_, external), (key, rcs)) in std::mem::take(&mut self.lan) { - errors.handle(ctrl.remove_lan(&key, external, rcs).await); - } - for ((_, external), (key, rcs)) in std::mem::take(&mut self.tor) { - errors.handle(ctrl.remove_tor(&key, external, rcs).await); - } - std::mem::take(&mut self.dns); - errors.handle(ctrl.dns.gc(Some(self.id.clone()), self.ip).await); - errors.into_result() - } else { - tracing::warn!("NetService dropped after NetController is shutdown"); - Err(Error::new( - eyre!("NetController is shutdown"), - crate::ErrorKind::Network, - )) - } - } -} - -impl Drop for NetService { - fn drop(&mut self) { - if !self.shutdown { - tracing::debug!("Dropping NetService for {}", self.id); - let svc = std::mem::replace( - self, - NetService { - shutdown: true, - id: Default::default(), - ip: Ipv4Addr::new(0, 0, 0, 0), - dns: Default::default(), - controller: Default::default(), - tor: Default::default(), - lan: Default::default(), - }, - ); - tokio::spawn(async move { svc.remove_all().await.unwrap() }); - } - } -} diff --git a/core/startos/src/net/ssl.rs b/core/startos/src/net/ssl.rs deleted file mode 100644 index 1f9397ad..00000000 --- a/core/startos/src/net/ssl.rs +++ /dev/null @@ -1,458 +0,0 @@ -use std::cmp::Ordering; -use std::collections::{BTreeMap, BTreeSet}; -use std::net::IpAddr; -use std::path::Path; -use std::time::{SystemTime, UNIX_EPOCH}; - -use futures::FutureExt; -use libc::time_t; -use openssl::asn1::{Asn1Integer, Asn1Time}; -use openssl::bn::{BigNum, MsbOption}; -use openssl::ec::{EcGroup, EcKey}; -use openssl::hash::MessageDigest; -use openssl::nid::Nid; -use openssl::pkey::{PKey, Private}; -use openssl::x509::{X509Builder, X509Extension, X509NameBuilder, X509}; -use openssl::*; -use rpc_toolkit::command; -use tokio::sync::{Mutex, RwLock}; -use tracing::instrument; - -use crate::account::AccountInfo; -use crate::context::RpcContext; -use crate::hostname::Hostname; -use crate::init::check_time_is_synchronized; -use crate::net::dhcp::ips; -use crate::net::keys::{Key, KeyInfo}; -use crate::{Error, ErrorKind, ResultExt, SOURCE_DATE}; - -static CERTIFICATE_VERSION: i32 = 2; // X509 version 3 is actually encoded as '2' in the cert because fuck you. - -fn unix_time(time: SystemTime) -> time_t { - time.duration_since(UNIX_EPOCH) - .map(|d| d.as_secs() as time_t) - .or_else(|_| UNIX_EPOCH.elapsed().map(|d| -(d.as_secs() as time_t))) - .unwrap_or_default() -} - -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] -pub struct CertPair { - pub ed25519: X509, - pub nistp256: X509, -} -impl CertPair { - fn updated( - pair: Option<&Self>, - hostname: &Hostname, - signer: (&PKey, &X509), - applicant: &Key, - ip: BTreeSet, - ) -> Result<(Self, bool), Error> { - let mut updated = false; - let mut updated_cert = |cert: Option<&X509>, osk: PKey| -> Result { - let mut ips = BTreeSet::new(); - if let Some(cert) = cert { - ips.extend( - cert.subject_alt_names() - .iter() - .flatten() - .filter_map(|a| a.ipaddress()) - .filter_map(|a| match a.len() { - 4 => Some::(<[u8; 4]>::try_from(a).unwrap().into()), - 16 => Some::(<[u8; 16]>::try_from(a).unwrap().into()), - _ => None, - }), - ); - if cert - .not_before() - .compare(Asn1Time::days_from_now(0)?.as_ref())? - == Ordering::Less - && cert - .not_after() - .compare(Asn1Time::days_from_now(30)?.as_ref())? - == Ordering::Greater - && ips.is_superset(&ip) - { - return Ok(cert.clone()); - } - } - ips.extend(ip.iter().copied()); - updated = true; - make_leaf_cert(signer, (&osk, &SANInfo::new(&applicant, hostname, ips))) - }; - Ok(( - Self { - ed25519: updated_cert(pair.map(|c| &c.ed25519), applicant.openssl_key_ed25519())?, - nistp256: updated_cert( - pair.map(|c| &c.nistp256), - applicant.openssl_key_nistp256(), - )?, - }, - updated, - )) - } -} - -pub async fn root_ca_start_time() -> Result { - Ok(if check_time_is_synchronized().await? { - SystemTime::now() - } else { - *SOURCE_DATE - }) -} - -#[derive(Debug)] -pub struct SslManager { - hostname: Hostname, - root_cert: X509, - int_key: PKey, - int_cert: X509, - cert_cache: RwLock>, -} -impl SslManager { - pub fn new(account: &AccountInfo, start_time: SystemTime) -> Result { - let int_key = generate_key()?; - let int_cert = make_int_cert( - (&account.root_ca_key, &account.root_ca_cert), - &int_key, - start_time, - )?; - Ok(Self { - hostname: account.hostname.clone(), - root_cert: account.root_ca_cert.clone(), - int_key, - int_cert, - cert_cache: RwLock::new(BTreeMap::new()), - }) - } - pub async fn with_certs(&self, key: Key, ip: IpAddr) -> Result { - let mut ips = ips().await?; - ips.insert(ip); - let (pair, updated) = CertPair::updated( - self.cert_cache.read().await.get(&key), - &self.hostname, - (&self.int_key, &self.int_cert), - &key, - ips, - )?; - if updated { - self.cert_cache - .write() - .await - .insert(key.clone(), pair.clone()); - } - - Ok(key.with_certs(pair, self.int_cert.clone(), self.root_cert.clone())) - } -} - -const EC_CURVE_NAME: nid::Nid = nid::Nid::X9_62_PRIME256V1; -lazy_static::lazy_static! { - static ref EC_GROUP: EcGroup = EcGroup::from_curve_name(EC_CURVE_NAME).unwrap(); - static ref SSL_MUTEX: Mutex<()> = Mutex::new(()); // TODO: make thread safe -} - -pub async fn export_key(key: &PKey, target: &Path) -> Result<(), Error> { - tokio::fs::write(target, key.private_key_to_pem_pkcs8()?) - .map(|res| res.with_ctx(|_| (ErrorKind::Filesystem, target.display().to_string()))) - .await?; - Ok(()) -} -pub async fn export_cert(chain: &[&X509], target: &Path) -> Result<(), Error> { - tokio::fs::write( - target, - chain - .into_iter() - .flat_map(|c| c.to_pem().unwrap()) - .collect::>(), - ) - .await?; - Ok(()) -} - -#[instrument(skip_all)] -fn rand_serial() -> Result { - let mut bn = BigNum::new()?; - bn.rand(64, MsbOption::MAYBE_ZERO, false)?; - let asn1 = Asn1Integer::from_bn(&bn)?; - Ok(asn1) -} -#[instrument(skip_all)] -pub fn generate_key() -> Result, Error> { - let new_key = EcKey::generate(EC_GROUP.as_ref())?; - let key = PKey::from_ec_key(new_key)?; - Ok(key) -} - -#[instrument(skip_all)] -pub fn make_root_cert( - root_key: &PKey, - hostname: &Hostname, - start_time: SystemTime, -) -> Result { - let mut builder = X509Builder::new()?; - builder.set_version(CERTIFICATE_VERSION)?; - - let unix_start_time = unix_time(start_time); - - let embargo = Asn1Time::from_unix(unix_start_time - 86400)?; - builder.set_not_before(&embargo)?; - - let expiration = Asn1Time::from_unix(unix_start_time + (10 * 364 * 86400))?; - builder.set_not_after(&expiration)?; - - builder.set_serial_number(&*rand_serial()?)?; - - let mut subject_name_builder = X509NameBuilder::new()?; - subject_name_builder.append_entry_by_text("CN", &format!("{} Local Root CA", &*hostname.0))?; - subject_name_builder.append_entry_by_text("O", "Start9")?; - subject_name_builder.append_entry_by_text("OU", "StartOS")?; - let subject_name = subject_name_builder.build(); - builder.set_subject_name(&subject_name)?; - - builder.set_issuer_name(&subject_name)?; - - builder.set_pubkey(&root_key)?; - - // Extensions - let cfg = conf::Conf::new(conf::ConfMethod::default())?; - let ctx = builder.x509v3_context(None, Some(&cfg)); - // subjectKeyIdentifier = hash - let subject_key_identifier = - X509Extension::new_nid(Some(&cfg), Some(&ctx), Nid::SUBJECT_KEY_IDENTIFIER, "hash")?; - // basicConstraints = critical, CA:true, pathlen:0 - let basic_constraints = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::BASIC_CONSTRAINTS, - "critical,CA:true", - )?; - // keyUsage = critical, digitalSignature, cRLSign, keyCertSign - let key_usage = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::KEY_USAGE, - "critical,digitalSignature,cRLSign,keyCertSign", - )?; - builder.append_extension(subject_key_identifier)?; - builder.append_extension(basic_constraints)?; - builder.append_extension(key_usage)?; - builder.sign(&root_key, MessageDigest::sha256())?; - let cert = builder.build(); - Ok(cert) -} -#[instrument(skip_all)] -pub fn make_int_cert( - signer: (&PKey, &X509), - applicant: &PKey, - start_time: SystemTime, -) -> Result { - let mut builder = X509Builder::new()?; - builder.set_version(CERTIFICATE_VERSION)?; - - let unix_start_time = unix_time(start_time); - - let embargo = Asn1Time::from_unix(unix_start_time - 86400)?; - builder.set_not_before(&embargo)?; - - let expiration = Asn1Time::from_unix(unix_start_time + (10 * 364 * 86400))?; - builder.set_not_after(&expiration)?; - - builder.set_serial_number(&*rand_serial()?)?; - - let mut subject_name_builder = X509NameBuilder::new()?; - subject_name_builder.append_entry_by_text("CN", "StartOS Local Intermediate CA")?; - subject_name_builder.append_entry_by_text("O", "Start9")?; - subject_name_builder.append_entry_by_text("OU", "StartOS")?; - let subject_name = subject_name_builder.build(); - builder.set_subject_name(&subject_name)?; - - builder.set_issuer_name(signer.1.subject_name())?; - - builder.set_pubkey(&applicant)?; - - let cfg = conf::Conf::new(conf::ConfMethod::default())?; - let ctx = builder.x509v3_context(Some(&signer.1), Some(&cfg)); - // subjectKeyIdentifier = hash - let subject_key_identifier = - X509Extension::new_nid(Some(&cfg), Some(&ctx), Nid::SUBJECT_KEY_IDENTIFIER, "hash")?; - // authorityKeyIdentifier = keyid:always,issuer - let authority_key_identifier = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::AUTHORITY_KEY_IDENTIFIER, - "keyid:always,issuer", - )?; - // basicConstraints = critical, CA:true, pathlen:0 - let basic_constraints = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::BASIC_CONSTRAINTS, - "critical,CA:true,pathlen:0", - )?; - // keyUsage = critical, digitalSignature, cRLSign, keyCertSign - let key_usage = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::KEY_USAGE, - "critical,digitalSignature,cRLSign,keyCertSign", - )?; - builder.append_extension(subject_key_identifier)?; - builder.append_extension(authority_key_identifier)?; - builder.append_extension(basic_constraints)?; - builder.append_extension(key_usage)?; - builder.sign(&signer.0, MessageDigest::sha256())?; - let cert = builder.build(); - Ok(cert) -} - -#[derive(Debug, PartialEq, Eq, PartialOrd, Ord)] -pub enum MaybeWildcard { - WithWildcard(String), - WithoutWildcard(String), -} -impl MaybeWildcard { - pub fn as_str(&self) -> &str { - match self { - MaybeWildcard::WithWildcard(s) => s.as_str(), - MaybeWildcard::WithoutWildcard(s) => s.as_str(), - } - } -} -impl std::fmt::Display for MaybeWildcard { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - MaybeWildcard::WithWildcard(dns) => write!(f, "DNS:{dns},DNS:*.{dns}"), - MaybeWildcard::WithoutWildcard(dns) => write!(f, "DNS:{dns}"), - } - } -} - -#[derive(Debug)] -pub struct SANInfo { - pub dns: BTreeSet, - pub ips: BTreeSet, -} -impl SANInfo { - pub fn new(key: &Key, hostname: &Hostname, ips: BTreeSet) -> Self { - let mut dns = BTreeSet::new(); - if let Some((id, _)) = key.interface() { - dns.insert(MaybeWildcard::WithWildcard(format!("{id}.embassy"))); - dns.insert(MaybeWildcard::WithWildcard(key.local_address().to_string())); - } else { - dns.insert(MaybeWildcard::WithoutWildcard("embassy".to_owned())); - dns.insert(MaybeWildcard::WithWildcard(hostname.local_domain_name())); - dns.insert(MaybeWildcard::WithoutWildcard(hostname.no_dot_host_name())); - dns.insert(MaybeWildcard::WithoutWildcard("localhost".to_owned())); - } - dns.insert(MaybeWildcard::WithWildcard(key.tor_address().to_string())); - Self { dns, ips } - } -} -impl std::fmt::Display for SANInfo { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let mut written = false; - for dns in &self.dns { - if written { - write!(f, ",")?; - } - written = true; - write!(f, "{dns}")?; - } - for ip in &self.ips { - if written { - write!(f, ",")?; - } - written = true; - write!(f, "IP:{ip}")?; - } - Ok(()) - } -} - -#[instrument(skip_all)] -pub fn make_leaf_cert( - signer: (&PKey, &X509), - applicant: (&PKey, &SANInfo), -) -> Result { - let mut builder = X509Builder::new()?; - builder.set_version(CERTIFICATE_VERSION)?; - - let embargo = Asn1Time::from_unix(unix_time(SystemTime::now()) - 86400)?; - builder.set_not_before(&embargo)?; - - // Google Apple and Mozilla reject certificate horizons longer than 398 days - // https://techbeacon.com/security/google-apple-mozilla-enforce-1-year-max-security-certifications - let expiration = Asn1Time::days_from_now(397)?; - builder.set_not_after(&expiration)?; - - builder.set_serial_number(&*rand_serial()?)?; - - let mut subject_name_builder = X509NameBuilder::new()?; - subject_name_builder.append_entry_by_text( - "CN", - applicant - .1 - .dns - .first() - .map(MaybeWildcard::as_str) - .unwrap_or("localhost"), - )?; - subject_name_builder.append_entry_by_text("O", "Start9")?; - subject_name_builder.append_entry_by_text("OU", "StartOS")?; - let subject_name = subject_name_builder.build(); - builder.set_subject_name(&subject_name)?; - - builder.set_issuer_name(signer.1.subject_name())?; - - builder.set_pubkey(&applicant.0)?; - - // Extensions - let cfg = conf::Conf::new(conf::ConfMethod::default())?; - let ctx = builder.x509v3_context(Some(&signer.1), Some(&cfg)); - // subjectKeyIdentifier = hash - let subject_key_identifier = - X509Extension::new_nid(Some(&cfg), Some(&ctx), Nid::SUBJECT_KEY_IDENTIFIER, "hash")?; - // authorityKeyIdentifier = keyid:always,issuer - let authority_key_identifier = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::AUTHORITY_KEY_IDENTIFIER, - "keyid,issuer:always", - )?; - let basic_constraints = - X509Extension::new_nid(Some(&cfg), Some(&ctx), Nid::BASIC_CONSTRAINTS, "CA:FALSE")?; - let key_usage = X509Extension::new_nid( - Some(&cfg), - Some(&ctx), - Nid::KEY_USAGE, - "critical,digitalSignature,keyEncipherment", - )?; - - let san_string = applicant.1.to_string(); - let subject_alt_name = - X509Extension::new_nid(Some(&cfg), Some(&ctx), Nid::SUBJECT_ALT_NAME, &san_string)?; - builder.append_extension(subject_key_identifier)?; - builder.append_extension(authority_key_identifier)?; - builder.append_extension(subject_alt_name)?; - builder.append_extension(basic_constraints)?; - builder.append_extension(key_usage)?; - - builder.sign(&signer.0, MessageDigest::sha256())?; - - let cert = builder.build(); - Ok(cert) -} - -#[command(subcommands(size))] -pub async fn ssl() -> Result<(), Error> { - Ok(()) -} - -#[command] -pub async fn size(#[context] ctx: RpcContext) -> Result { - Ok(format!( - "Cert Catch size: {}", - ctx.net_controller.ssl.cert_cache.read().await.len() - )) -} diff --git a/core/startos/src/net/static_server.rs b/core/startos/src/net/static_server.rs deleted file mode 100644 index 761566a2..00000000 --- a/core/startos/src/net/static_server.rs +++ /dev/null @@ -1,586 +0,0 @@ -use std::fs::Metadata; -use std::future::Future; -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use std::time::UNIX_EPOCH; - -use async_compression::tokio::bufread::GzipEncoder; -use color_eyre::eyre::eyre; -use digest::Digest; -use futures::FutureExt; -use http::header::ACCEPT_ENCODING; -use http::request::Parts as RequestParts; -use hyper::{Body, Method, Request, Response, StatusCode}; -use include_dir::{include_dir, Dir}; -use new_mime_guess::MimeGuess; -use openssl::hash::MessageDigest; -use openssl::x509::X509; -use rpc_toolkit::rpc_handler; -use tokio::fs::File; -use tokio::io::BufReader; -use tokio_util::io::ReaderStream; - -use crate::context::{DiagnosticContext, InstallContext, RpcContext, SetupContext}; -use crate::core::rpc_continuations::RequestGuid; -use crate::db::subscribe; -use crate::hostname::Hostname; -use crate::install::PKG_PUBLIC_DIR; -use crate::middleware::auth::{auth as auth_middleware, HasValidSession}; -use crate::middleware::cors::cors; -use crate::middleware::db::db as db_middleware; -use crate::middleware::diagnostic::diagnostic as diagnostic_middleware; -use crate::net::HttpHandler; -use crate::{diagnostic_api, install_api, main_api, setup_api, Error, ErrorKind, ResultExt}; - -static NOT_FOUND: &[u8] = b"Not Found"; -static METHOD_NOT_ALLOWED: &[u8] = b"Method Not Allowed"; -static NOT_AUTHORIZED: &[u8] = b"Not Authorized"; - -static EMBEDDED_UIS: Dir<'_> = include_dir!("$CARGO_MANIFEST_DIR/../../web/dist/static"); - -const PROXY_STRIP_HEADERS: &[&str] = &["cookie", "host", "origin", "referer", "user-agent"]; - -fn status_fn(_: i32) -> StatusCode { - StatusCode::OK -} - -#[derive(Clone)] -pub enum UiMode { - Setup, - Diag, - Install, - Main, -} - -impl UiMode { - fn path(&self, path: &str) -> PathBuf { - match self { - Self::Setup => Path::new("setup-wizard").join(path), - Self::Diag => Path::new("diagnostic-ui").join(path), - Self::Install => Path::new("install-wizard").join(path), - Self::Main => Path::new("ui").join(path), - } - } -} - -pub async fn setup_ui_file_router(ctx: SetupContext) -> Result { - let handler: HttpHandler = Arc::new(move |req| { - let ctx = ctx.clone(); - - let ui_mode = UiMode::Setup; - async move { - let res = match req.uri().path() { - path if path.starts_with("/rpc/") => { - let rpc_handler = rpc_handler!({ - command: setup_api, - context: ctx, - status: status_fn, - middleware: [ - cors, - ] - }); - - rpc_handler(req) - .await - .map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network)) - } - _ => alt_ui(req, ui_mode).await, - }; - - match res { - Ok(data) => Ok(data), - Err(err) => Ok(server_error(err)), - } - } - .boxed() - }); - - Ok(handler) -} - -pub async fn diag_ui_file_router(ctx: DiagnosticContext) -> Result { - let handler: HttpHandler = Arc::new(move |req| { - let ctx = ctx.clone(); - let ui_mode = UiMode::Diag; - async move { - let res = match req.uri().path() { - path if path.starts_with("/rpc/") => { - let rpc_handler = rpc_handler!({ - command: diagnostic_api, - context: ctx, - status: status_fn, - middleware: [ - cors, - diagnostic_middleware, - ] - }); - - rpc_handler(req) - .await - .map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network)) - } - _ => alt_ui(req, ui_mode).await, - }; - - match res { - Ok(data) => Ok(data), - Err(err) => Ok(server_error(err)), - } - } - .boxed() - }); - - Ok(handler) -} - -pub async fn install_ui_file_router(ctx: InstallContext) -> Result { - let handler: HttpHandler = Arc::new(move |req| { - let ctx = ctx.clone(); - let ui_mode = UiMode::Install; - async move { - let res = match req.uri().path() { - path if path.starts_with("/rpc/") => { - let rpc_handler = rpc_handler!({ - command: install_api, - context: ctx, - status: status_fn, - middleware: [ - cors, - ] - }); - - rpc_handler(req) - .await - .map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network)) - } - _ => alt_ui(req, ui_mode).await, - }; - - match res { - Ok(data) => Ok(data), - Err(err) => Ok(server_error(err)), - } - } - .boxed() - }); - - Ok(handler) -} - -pub async fn main_ui_server_router(ctx: RpcContext) -> Result { - let handler: HttpHandler = Arc::new(move |req| { - let ctx = ctx.clone(); - - async move { - let res = match req.uri().path() { - path if path.starts_with("/rpc/") => { - let auth_middleware = auth_middleware(ctx.clone()); - let db_middleware = db_middleware(ctx.clone()); - let rpc_handler = rpc_handler!({ - command: main_api, - context: ctx, - status: status_fn, - middleware: [ - cors, - auth_middleware, - db_middleware, - ] - }); - - rpc_handler(req) - .await - .map_err(|err| Error::new(eyre!("{}", err), crate::ErrorKind::Network)) - } - "/ws/db" => subscribe(ctx, req).await, - path if path.starts_with("/ws/rpc/") => { - match RequestGuid::from(path.strip_prefix("/ws/rpc/").unwrap()) { - None => { - tracing::debug!("No Guid Path"); - Ok::<_, Error>(bad_request()) - } - Some(guid) => match ctx.get_ws_continuation_handler(&guid).await { - Some(cont) => match cont(req).await { - Ok::<_, Error>(r) => Ok::<_, Error>(r), - Err(err) => Ok::<_, Error>(server_error(err)), - }, - _ => Ok::<_, Error>(not_found()), - }, - } - } - path if path.starts_with("/rest/rpc/") => { - match RequestGuid::from(path.strip_prefix("/rest/rpc/").unwrap()) { - None => { - tracing::debug!("No Guid Path"); - Ok::<_, Error>(bad_request()) - } - Some(guid) => match ctx.get_rest_continuation_handler(&guid).await { - None => Ok::<_, Error>(not_found()), - Some(cont) => match cont(req).await { - Ok::<_, Error>(r) => Ok::<_, Error>(r), - Err(e) => Ok::<_, Error>(server_error(e)), - }, - }, - } - } - _ => main_embassy_ui(req, ctx).await, - }; - - match res { - Ok(data) => Ok(data), - Err(err) => Ok(server_error(err)), - } - } - .boxed() - }); - - Ok(handler) -} - -async fn alt_ui(req: Request, ui_mode: UiMode) -> Result, Error> { - let (request_parts, _body) = req.into_parts(); - match &request_parts.method { - &Method::GET => { - let uri_path = ui_mode.path( - request_parts - .uri - .path() - .strip_prefix('/') - .unwrap_or(request_parts.uri.path()), - ); - - let file = EMBEDDED_UIS - .get_file(&*uri_path) - .or_else(|| EMBEDDED_UIS.get_file(&*ui_mode.path("index.html"))); - - if let Some(file) = file { - FileData::from_embedded(&request_parts, file) - .into_response(&request_parts) - .await - } else { - Ok(not_found()) - } - } - _ => Ok(method_not_allowed()), - } -} - -async fn if_authorized< - F: FnOnce() -> Fut, - Fut: Future, Error>> + Send + Sync, ->( - ctx: &RpcContext, - parts: &RequestParts, - f: F, -) -> Result, Error> { - if let Err(e) = HasValidSession::from_request_parts(parts, ctx).await { - un_authorized(e, parts.uri.path()) - } else { - f().await - } -} - -async fn main_embassy_ui(req: Request, ctx: RpcContext) -> Result, Error> { - let (request_parts, _body) = req.into_parts(); - match ( - &request_parts.method, - request_parts - .uri - .path() - .strip_prefix('/') - .unwrap_or(request_parts.uri.path()) - .split_once('/'), - ) { - (&Method::GET, Some(("public", path))) => { - if_authorized(&ctx, &request_parts, || async { - let sub_path = Path::new(path); - if let Ok(rest) = sub_path.strip_prefix("package-data") { - FileData::from_path( - &request_parts, - &ctx.datadir.join(PKG_PUBLIC_DIR).join(rest), - ) - .await? - .into_response(&request_parts) - .await - } else { - Ok(not_found()) - } - }) - .await - } - (&Method::GET, Some(("proxy", target))) => { - if_authorized(&ctx, &request_parts, || async { - let target = urlencoding::decode(target)?; - let res = ctx - .client - .get(target.as_ref()) - .headers( - request_parts - .headers - .iter() - .filter(|(h, _)| { - !PROXY_STRIP_HEADERS - .iter() - .any(|bad| h.as_str().eq_ignore_ascii_case(bad)) - }) - .map(|(h, v)| (h.clone(), v.clone())) - .collect(), - ) - .send() - .await - .with_kind(crate::ErrorKind::Network)?; - let mut hres = Response::builder().status(res.status()); - for (h, v) in res.headers().clone() { - if let Some(h) = h { - hres = hres.header(h, v); - } - } - hres.body(Body::wrap_stream(res.bytes_stream())) - .with_kind(crate::ErrorKind::Network) - }) - .await - } - (&Method::GET, Some(("eos", "local.crt"))) => { - let account = ctx.account.read().await; - cert_send(&account.root_ca_cert, &account.hostname) - } - (&Method::GET, _) => { - let uri_path = UiMode::Main.path( - request_parts - .uri - .path() - .strip_prefix('/') - .unwrap_or(request_parts.uri.path()), - ); - - let file = EMBEDDED_UIS - .get_file(&*uri_path) - .or_else(|| EMBEDDED_UIS.get_file(&*UiMode::Main.path("index.html"))); - - if let Some(file) = file { - FileData::from_embedded(&request_parts, file) - .into_response(&request_parts) - .await - } else { - Ok(not_found()) - } - } - _ => Ok(method_not_allowed()), - } -} - -fn un_authorized(err: Error, path: &str) -> Result, Error> { - tracing::warn!("unauthorized for {} @{:?}", err, path); - tracing::debug!("{:?}", err); - Ok(Response::builder() - .status(StatusCode::UNAUTHORIZED) - .body(NOT_AUTHORIZED.into()) - .unwrap()) -} - -/// HTTP status code 404 -fn not_found() -> Response { - Response::builder() - .status(StatusCode::NOT_FOUND) - .body(NOT_FOUND.into()) - .unwrap() -} - -/// HTTP status code 405 -fn method_not_allowed() -> Response { - Response::builder() - .status(StatusCode::METHOD_NOT_ALLOWED) - .body(METHOD_NOT_ALLOWED.into()) - .unwrap() -} - -fn server_error(err: Error) -> Response { - Response::builder() - .status(StatusCode::INTERNAL_SERVER_ERROR) - .body(err.to_string().into()) - .unwrap() -} - -fn bad_request() -> Response { - Response::builder() - .status(StatusCode::BAD_REQUEST) - .body(Body::empty()) - .unwrap() -} - -fn cert_send(cert: &X509, hostname: &Hostname) -> Result, Error> { - let pem = cert.to_pem()?; - Response::builder() - .status(StatusCode::OK) - .header( - http::header::ETAG, - base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - &*cert.digest(MessageDigest::sha256())?, - ) - .to_lowercase(), - ) - .header(http::header::CONTENT_TYPE, "application/x-x509-ca-cert") - .header(http::header::CONTENT_LENGTH, pem.len()) - .header( - http::header::CONTENT_DISPOSITION, - format!("attachment; filename={}.crt", &hostname.0), - ) - .body(Body::from(pem)) - .with_kind(ErrorKind::Network) -} - -struct FileData { - data: Body, - len: Option, - encoding: Option<&'static str>, - e_tag: String, - mime: Option, -} -impl FileData { - fn from_embedded(req: &RequestParts, file: &'static include_dir::File<'static>) -> Self { - let path = file.path(); - let (encoding, data) = req - .headers - .get_all(ACCEPT_ENCODING) - .into_iter() - .filter_map(|h| h.to_str().ok()) - .flat_map(|s| s.split(",")) - .filter_map(|s| s.split(";").next()) - .map(|s| s.trim()) - .fold((None, file.contents()), |acc, e| { - if let Some(file) = (e == "br") - .then_some(()) - .and_then(|_| EMBEDDED_UIS.get_file(format!("{}.br", path.display()))) - { - (Some("br"), file.contents()) - } else if let Some(file) = (e == "gzip" && acc.0 != Some("br")) - .then_some(()) - .and_then(|_| EMBEDDED_UIS.get_file(format!("{}.gz", path.display()))) - { - (Some("gzip"), file.contents()) - } else { - acc - } - }); - - Self { - len: Some(data.len() as u64), - encoding, - data: data.into(), - e_tag: e_tag(path, None), - mime: MimeGuess::from_path(path) - .first() - .map(|m| m.essence_str().to_owned()), - } - } - - async fn from_path(req: &RequestParts, path: &Path) -> Result { - let encoding = req - .headers - .get_all(ACCEPT_ENCODING) - .into_iter() - .filter_map(|h| h.to_str().ok()) - .flat_map(|s| s.split(",")) - .filter_map(|s| s.split(";").next()) - .map(|s| s.trim()) - .any(|e| e == "gzip") - .then_some("gzip"); - - let file = File::open(path) - .await - .with_ctx(|_| (ErrorKind::Filesystem, path.display().to_string()))?; - let metadata = file - .metadata() - .await - .with_ctx(|_| (ErrorKind::Filesystem, path.display().to_string()))?; - - let e_tag = e_tag(path, Some(&metadata)); - - let (len, data) = if encoding == Some("gzip") { - ( - None, - Body::wrap_stream(ReaderStream::new(GzipEncoder::new(BufReader::new(file)))), - ) - } else { - ( - Some(metadata.len()), - Body::wrap_stream(ReaderStream::new(file)), - ) - }; - - Ok(Self { - data, - len, - encoding, - e_tag, - mime: MimeGuess::from_path(path) - .first() - .map(|m| m.essence_str().to_owned()), - }) - } - - async fn into_response(self, req: &RequestParts) -> Result, Error> { - let mut builder = Response::builder(); - if let Some(mime) = self.mime { - builder = builder.header(http::header::CONTENT_TYPE, &*mime); - } - builder = builder.header(http::header::ETAG, &*self.e_tag); - builder = builder.header( - http::header::CACHE_CONTROL, - "public, max-age=21000000, immutable", - ); - - if req - .headers - .get_all(http::header::CONNECTION) - .iter() - .flat_map(|s| s.to_str().ok()) - .flat_map(|s| s.split(",")) - .any(|s| s.trim() == "keep-alive") - { - builder = builder.header(http::header::CONNECTION, "keep-alive"); - } - - if req - .headers - .get("if-none-match") - .and_then(|h| h.to_str().ok()) - == Some(self.e_tag.as_ref()) - { - builder = builder.status(StatusCode::NOT_MODIFIED); - builder.body(Body::empty()) - } else { - if let Some(len) = self.len { - builder = builder.header(http::header::CONTENT_LENGTH, len); - } - if let Some(encoding) = self.encoding { - builder = builder.header(http::header::CONTENT_ENCODING, encoding); - } - - builder.body(self.data) - } - .with_kind(ErrorKind::Network) - } -} - -fn e_tag(path: &Path, metadata: Option<&Metadata>) -> String { - let mut hasher = sha2::Sha256::new(); - hasher.update(format!("{:?}", path).as_bytes()); - if let Some(modified) = metadata.and_then(|m| m.modified().ok()) { - hasher.update( - format!( - "{}", - modified - .duration_since(UNIX_EPOCH) - .unwrap_or_default() - .as_secs() - ) - .as_bytes(), - ); - } - let res = hasher.finalize(); - format!( - "\"{}\"", - base32::encode(base32::Alphabet::RFC4648 { padding: false }, res.as_slice()).to_lowercase() - ) -} diff --git a/core/startos/src/net/tor.rs b/core/startos/src/net/tor.rs deleted file mode 100644 index 1bf4c5f4..00000000 --- a/core/startos/src/net/tor.rs +++ /dev/null @@ -1,736 +0,0 @@ -use std::collections::BTreeMap; -use std::net::SocketAddr; -use std::sync::atomic::AtomicBool; -use std::sync::{Arc, Weak}; -use std::time::Duration; - -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use futures::future::BoxFuture; -use futures::{FutureExt, TryStreamExt}; -use helpers::NonDetachingJoinHandle; -use itertools::Itertools; -use lazy_static::lazy_static; -use regex::Regex; -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; -use tokio::net::TcpStream; -use tokio::process::Command; -use tokio::sync::{mpsc, oneshot}; -use tokio::time::Instant; -use torut::control::{AsyncEvent, AuthenticatedConn, ConnError}; -use torut::onion::{OnionAddressV3, TorSecretKeyV3}; -use tracing::instrument; - -use crate::context::{CliContext, RpcContext}; -use crate::logs::{ - cli_logs_generic_follow, cli_logs_generic_nofollow, fetch_logs, follow_logs, journalctl, - LogFollowResponse, LogResponse, LogSource, -}; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::util::{display_none, Invoke}; -use crate::{Error, ErrorKind, ResultExt as _}; - -pub const SYSTEMD_UNIT: &str = "tor@default"; -const STARTING_HEALTH_TIMEOUT: u64 = 120; // 2min - -enum ErrorLogSeverity { - Fatal { wipe_state: bool }, - Unknown { wipe_state: bool }, -} - -lazy_static! { - static ref LOG_REGEXES: Vec<(Regex, ErrorLogSeverity)> = vec![( - Regex::new("This could indicate a route manipulation attack, network overload, bad local network connectivity, or a bug\\.").unwrap(), - ErrorLogSeverity::Unknown { wipe_state: true } - ),( - Regex::new("died due to an invalid selected path").unwrap(), - ErrorLogSeverity::Fatal { wipe_state: false } - ),( - Regex::new("Tor has not observed any network activity for the past").unwrap(), - ErrorLogSeverity::Unknown { wipe_state: false } - )]; - static ref PROGRESS_REGEX: Regex = Regex::new("PROGRESS=([0-9]+)").unwrap(); -} - -#[command(subcommands(list_services, logs, reset))] -pub fn tor() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_none))] -pub async fn reset( - #[context] ctx: RpcContext, - #[arg(rename = "wipe-state", short = 'w', long = "wipe-state")] wipe_state: bool, - #[arg] reason: String, -) -> Result<(), Error> { - ctx.net_controller - .tor - .reset(wipe_state, Error::new(eyre!("{reason}"), ErrorKind::Tor)) - .await -} - -fn display_services(services: Vec, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(services, matches); - } - - let mut table = Table::new(); - for service in services { - let row = row![&service.to_string()]; - table.add_row(row); - } - table.print_tty(false).unwrap(); -} - -#[command(rename = "list-services", display(display_services))] -pub async fn list_services( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result, Error> { - ctx.net_controller.tor.list_services().await -} - -#[command( - custom_cli(cli_logs(async, context(CliContext))), - subcommands(self(logs_nofollow(async)), logs_follow), - display(display_none) -)] -pub async fn logs( - #[arg(short = 'l', long = "limit")] limit: Option, - #[arg(short = 'c', long = "cursor")] cursor: Option, - #[arg(short = 'B', long = "before", default)] before: bool, - #[arg(short = 'f', long = "follow", default)] follow: bool, -) -> Result<(Option, Option, bool, bool), Error> { - Ok((limit, cursor, before, follow)) -} -pub async fn cli_logs( - ctx: CliContext, - (limit, cursor, before, follow): (Option, Option, bool, bool), -) -> Result<(), RpcError> { - if follow { - if cursor.is_some() { - return Err(RpcError::from(Error::new( - eyre!("The argument '--cursor ' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - if before { - return Err(RpcError::from(Error::new( - eyre!("The argument '--before' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - cli_logs_generic_follow(ctx, "net.tor.logs.follow", None, limit).await - } else { - cli_logs_generic_nofollow(ctx, "net.tor.logs", None, limit, cursor, before).await - } -} -pub async fn logs_nofollow( - _ctx: (), - (limit, cursor, before, _): (Option, Option, bool, bool), -) -> Result { - fetch_logs(LogSource::Unit(SYSTEMD_UNIT), limit, cursor, before).await -} - -#[command(rpc_only, rename = "follow", display(display_none))] -pub async fn logs_follow( - #[context] ctx: RpcContext, - #[parent_data] (limit, _, _, _): (Option, Option, bool, bool), -) -> Result { - follow_logs(ctx, LogSource::Unit(SYSTEMD_UNIT), limit).await -} - -fn event_handler(_event: AsyncEvent<'static>) -> BoxFuture<'static, Result<(), ConnError>> { - async move { Ok(()) }.boxed() -} - -pub struct TorController(TorControl); -impl TorController { - pub fn new(tor_control: SocketAddr, tor_socks: SocketAddr) -> Self { - TorController(TorControl::new(tor_control, tor_socks)) - } - - pub async fn add( - &self, - key: TorSecretKeyV3, - external: u16, - target: SocketAddr, - ) -> Result, Error> { - let (reply, res) = oneshot::channel(); - self.0 - .send - .send(TorCommand::AddOnion { - key, - external, - target, - reply, - }) - .ok() - .ok_or_else(|| Error::new(eyre!("TorControl died"), ErrorKind::Tor))?; - res.await - .ok() - .ok_or_else(|| Error::new(eyre!("TorControl died"), ErrorKind::Tor)) - } - - pub async fn gc( - &self, - key: Option, - external: Option, - ) -> Result<(), Error> { - self.0 - .send - .send(TorCommand::GC { key, external }) - .ok() - .ok_or_else(|| Error::new(eyre!("TorControl died"), ErrorKind::Tor)) - } - - pub async fn reset(&self, wipe_state: bool, context: Error) -> Result<(), Error> { - self.0 - .send - .send(TorCommand::Reset { - wipe_state, - context, - }) - .ok() - .ok_or_else(|| Error::new(eyre!("TorControl died"), ErrorKind::Tor)) - } - - pub async fn list_services(&self) -> Result, Error> { - let (reply, res) = oneshot::channel(); - self.0 - .send - .send(TorCommand::GetInfo { - query: "onions/current".into(), - reply, - }) - .ok() - .ok_or_else(|| Error::new(eyre!("TorControl died"), ErrorKind::Tor))?; - res.await - .ok() - .ok_or_else(|| Error::new(eyre!("TorControl died"), ErrorKind::Tor))?? - .lines() - .map(|l| l.trim()) - .filter(|l| !l.is_empty()) - .map(|l| l.parse().with_kind(ErrorKind::Tor)) - .collect() - } -} - -type AuthenticatedConnection = AuthenticatedConn< - TcpStream, - Box) -> BoxFuture<'static, Result<(), ConnError>> + Send + Sync>, ->; - -enum TorCommand { - AddOnion { - key: TorSecretKeyV3, - external: u16, - target: SocketAddr, - reply: oneshot::Sender>, - }, - GC { - key: Option, - external: Option, - }, - GetInfo { - query: String, - reply: oneshot::Sender>, - }, - Reset { - wipe_state: bool, - context: Error, - }, -} - -#[instrument(skip_all)] -async fn torctl( - tor_control: SocketAddr, - tor_socks: SocketAddr, - recv: &mut mpsc::UnboundedReceiver, - services: &mut BTreeMap<[u8; 64], BTreeMap>>>, - wipe_state: &AtomicBool, - health_timeout: &mut Duration, -) -> Result<(), Error> { - let bootstrap = async { - if Command::new("systemctl") - .arg("is-active") - .arg("--quiet") - .arg("tor") - .invoke(ErrorKind::Tor) - .await - .is_ok() - { - Command::new("systemctl") - .arg("stop") - .arg("tor") - .invoke(ErrorKind::Tor) - .await?; - for _ in 0..30 { - if TcpStream::connect(tor_control).await.is_err() { - break; - } - tokio::time::sleep(Duration::from_secs(1)).await; - } - if TcpStream::connect(tor_control).await.is_ok() { - return Err(Error::new( - eyre!("Tor is failing to shut down"), - ErrorKind::Tor, - )); - } - } - if wipe_state.load(std::sync::atomic::Ordering::SeqCst) { - tokio::fs::remove_dir_all("/var/lib/tor").await?; - wipe_state.store(false, std::sync::atomic::Ordering::SeqCst); - } - tokio::fs::create_dir_all("/var/lib/tor").await?; - Command::new("chown") - .arg("-R") - .arg("debian-tor") - .arg("/var/lib/tor") - .invoke(ErrorKind::Filesystem) - .await?; - Command::new("systemctl") - .arg("start") - .arg("tor") - .invoke(ErrorKind::Tor) - .await?; - - let logs = journalctl(LogSource::Unit(SYSTEMD_UNIT), 0, None, false, true).await?; - - let mut tcp_stream = None; - for _ in 0..60 { - if let Ok(conn) = TcpStream::connect(tor_control).await { - tcp_stream = Some(conn); - break; - } - tokio::time::sleep(Duration::from_secs(1)).await; - } - let tcp_stream = tcp_stream.ok_or_else(|| { - Error::new(eyre!("Timed out waiting for tor to start"), ErrorKind::Tor) - })?; - tracing::info!("Tor is started"); - - let mut conn = torut::control::UnauthenticatedConn::new(tcp_stream); - let auth = conn - .load_protocol_info() - .await? - .make_auth_data()? - .ok_or_else(|| eyre!("Cookie Auth Not Available")) - .with_kind(crate::ErrorKind::Tor)?; - conn.authenticate(&auth).await?; - let mut connection: AuthenticatedConnection = conn.into_authenticated().await; - connection.set_async_event_handler(Some(Box::new(|event| event_handler(event)))); - - let mut bootstrapped = false; - let mut last_increment = (String::new(), Instant::now()); - for _ in 0..300 { - match connection.get_info("status/bootstrap-phase").await { - Ok(a) => { - if a.contains("TAG=done") { - bootstrapped = true; - break; - } - if let Some(p) = PROGRESS_REGEX.captures(&a) { - if let Some(p) = p.get(1) { - if p.as_str() != &*last_increment.0 { - last_increment = (p.as_str().into(), Instant::now()); - } - } - } - } - Err(e) => { - let e = Error::from(e); - tracing::error!("{}", e); - tracing::debug!("{:?}", e); - } - } - if last_increment.1.elapsed() > Duration::from_secs(30) { - return Err(Error::new( - eyre!("Tor stuck bootstrapping at {}%", last_increment.0), - ErrorKind::Tor, - )); - } - tokio::time::sleep(Duration::from_secs(1)).await; - } - if !bootstrapped { - return Err(Error::new( - eyre!("Timed out waiting for tor to bootstrap"), - ErrorKind::Tor, - )); - } - Ok((connection, logs)) - }; - let pre_handler = async { - while let Some(command) = recv.recv().await { - match command { - TorCommand::AddOnion { - key, - external, - target, - reply, - } => { - let mut service = if let Some(service) = services.remove(&key.as_bytes()) { - service - } else { - BTreeMap::new() - }; - let mut binding = service.remove(&external).unwrap_or_default(); - let rc = if let Some(rc) = - Weak::upgrade(&binding.remove(&target).unwrap_or_default()) - { - rc - } else { - Arc::new(()) - }; - binding.insert(target, Arc::downgrade(&rc)); - service.insert(external, binding); - services.insert(key.as_bytes(), service); - reply.send(rc).unwrap_or_default(); - } - TorCommand::GetInfo { reply, .. } => { - reply - .send(Err(Error::new( - eyre!("Tor has not finished bootstrapping..."), - ErrorKind::Tor, - ))) - .unwrap_or_default(); - } - TorCommand::GC { .. } => (), - TorCommand::Reset { - wipe_state: new_wipe_state, - context, - } => { - wipe_state.fetch_or(new_wipe_state, std::sync::atomic::Ordering::SeqCst); - return Err(context); - } - } - } - Ok(()) - }; - - let (mut connection, mut logs) = tokio::select! { - res = bootstrap => res?, - res = pre_handler => return res, - }; - - let hck_key = TorSecretKeyV3::generate(); - connection - .add_onion_v3( - &hck_key, - false, - false, - false, - None, - &mut [(80, SocketAddr::from(([127, 0, 0, 1], 80)))].iter(), - ) - .await?; - - for (key, service) in std::mem::take(services) { - let key = TorSecretKeyV3::from(key); - let bindings = service - .iter() - .flat_map(|(ext, int)| { - int.iter() - .find(|(_, rc)| rc.strong_count() > 0) - .map(|(addr, _)| (*ext, SocketAddr::from(*addr))) - }) - .collect::>(); - if !bindings.is_empty() { - services.insert(key.as_bytes(), service); - connection - .add_onion_v3(&key, false, false, false, None, &mut bindings.iter()) - .await?; - } - } - - let handler = async { - while let Some(command) = recv.recv().await { - match command { - TorCommand::AddOnion { - key, - external, - target, - reply, - } => { - let mut rm_res = Ok(()); - let onion_base = key - .public() - .get_onion_address() - .get_address_without_dot_onion(); - let mut service = if let Some(service) = services.remove(&key.as_bytes()) { - rm_res = connection.del_onion(&onion_base).await; - service - } else { - BTreeMap::new() - }; - let mut binding = service.remove(&external).unwrap_or_default(); - let rc = if let Some(rc) = - Weak::upgrade(&binding.remove(&target).unwrap_or_default()) - { - rc - } else { - Arc::new(()) - }; - binding.insert(target, Arc::downgrade(&rc)); - service.insert(external, binding); - let bindings = service - .iter() - .flat_map(|(ext, int)| { - int.iter() - .find(|(_, rc)| rc.strong_count() > 0) - .map(|(addr, _)| (*ext, SocketAddr::from(*addr))) - }) - .collect::>(); - services.insert(key.as_bytes(), service); - reply.send(rc).unwrap_or_default(); - rm_res?; - connection - .add_onion_v3(&key, false, false, false, None, &mut bindings.iter()) - .await?; - } - TorCommand::GC { key, external } => { - for key in if key.is_some() { - itertools::Either::Left(key.into_iter().map(|k| k.as_bytes())) - } else { - itertools::Either::Right(services.keys().cloned().collect_vec().into_iter()) - } { - let key = TorSecretKeyV3::from(key); - let onion_base = key - .public() - .get_onion_address() - .get_address_without_dot_onion(); - if let Some(mut service) = services.remove(&key.as_bytes()) { - for external in if external.is_some() { - itertools::Either::Left(external.into_iter()) - } else { - itertools::Either::Right( - service.keys().copied().collect_vec().into_iter(), - ) - } { - if let Some(mut binding) = service.remove(&external) { - binding = binding - .into_iter() - .filter(|(_, rc)| rc.strong_count() > 0) - .collect(); - if !binding.is_empty() { - service.insert(external, binding); - } - } - } - let rm_res = connection.del_onion(&onion_base).await; - if !service.is_empty() { - let bindings = service - .iter() - .flat_map(|(ext, int)| { - int.iter() - .find(|(_, rc)| rc.strong_count() > 0) - .map(|(addr, _)| (*ext, SocketAddr::from(*addr))) - }) - .collect::>(); - if !bindings.is_empty() { - services.insert(key.as_bytes(), service); - } - rm_res?; - if !bindings.is_empty() { - connection - .add_onion_v3( - &key, - false, - false, - false, - None, - &mut bindings.iter(), - ) - .await?; - } - } else { - rm_res?; - } - } - } - } - TorCommand::GetInfo { query, reply } => { - reply - .send(connection.get_info(&query).await.with_kind(ErrorKind::Tor)) - .unwrap_or_default(); - } - TorCommand::Reset { - wipe_state: new_wipe_state, - context, - } => { - wipe_state.fetch_or(new_wipe_state, std::sync::atomic::Ordering::SeqCst); - return Err(context); - } - } - } - Ok(()) - }; - let log_parser = async { - while let Some(log) = logs.try_next().await? { - for (regex, severity) in &*LOG_REGEXES { - if regex.is_match(&log.message) { - let (check, wipe_state) = match severity { - ErrorLogSeverity::Fatal { wipe_state } => (false, *wipe_state), - ErrorLogSeverity::Unknown { wipe_state } => (true, *wipe_state), - }; - if !check - || tokio::time::timeout( - Duration::from_secs(30), - tokio_socks::tcp::Socks5Stream::connect( - tor_socks, - (hck_key.public().get_onion_address().to_string(), 80), - ), - ) - .await - .map_err(|e| tracing::warn!("Tor is confirmed to be down: {e}")) - .and_then(|a| { - a.map_err(|e| tracing::warn!("Tor is confirmed to be down: {e}")) - }) - .is_err() - { - if wipe_state { - Command::new("systemctl") - .arg("stop") - .arg("tor") - .invoke(ErrorKind::Tor) - .await?; - tokio::fs::remove_dir_all("/var/lib/tor").await?; - } - return Err(Error::new(eyre!("{}", log.message), ErrorKind::Tor)); - } - } - } - } - Err(Error::new(eyre!("Log stream terminated"), ErrorKind::Tor)) - }; - let health_checker = async { - let mut last_success = Instant::now(); - loop { - tokio::time::sleep(Duration::from_secs(30)).await; - if tokio::time::timeout( - Duration::from_secs(30), - tokio_socks::tcp::Socks5Stream::connect( - tor_socks, - (hck_key.public().get_onion_address().to_string(), 80), - ), - ) - .await - .map_err(|e| e.to_string()) - .and_then(|e| e.map_err(|e| e.to_string())) - .is_err() - { - if last_success.elapsed() > *health_timeout { - let err = Error::new(eyre!("Tor health check failed for longer than current timeout ({health_timeout:?})"), crate::ErrorKind::Tor); - *health_timeout *= 2; - wipe_state.store(true, std::sync::atomic::Ordering::SeqCst); - return Err(err); - } - } else { - last_success = Instant::now(); - } - } - }; - - tokio::select! { - res = handler => res?, - res = log_parser => res?, - res = health_checker => res?, - } - - Ok(()) -} - -struct TorControl { - _thread: NonDetachingJoinHandle<()>, - send: mpsc::UnboundedSender, -} -impl TorControl { - pub fn new(tor_control: SocketAddr, tor_socks: SocketAddr) -> Self { - let (send, mut recv) = mpsc::unbounded_channel(); - Self { - _thread: tokio::spawn(async move { - let mut services = BTreeMap::new(); - let wipe_state = AtomicBool::new(false); - let mut health_timeout = Duration::from_secs(STARTING_HEALTH_TIMEOUT); - while let Err(e) = torctl( - tor_control, - tor_socks, - &mut recv, - &mut services, - &wipe_state, - &mut health_timeout, - ) - .await - { - tracing::error!("{e}: Restarting tor"); - tracing::debug!("{e:?}"); - } - tracing::info!("TorControl is shut down.") - }) - .into(), - send, - } - } -} - -#[tokio::test] -#[ignore] -async fn test() { - let mut conn = torut::control::UnauthenticatedConn::new( - TcpStream::connect(SocketAddr::from(([127, 0, 0, 1], 9051))) - .await - .unwrap(), // TODO - ); - let auth = conn - .load_protocol_info() - .await - .unwrap() - .make_auth_data() - .unwrap() - .ok_or_else(|| eyre!("Cookie Auth Not Available")) - .with_kind(crate::ErrorKind::Tor) - .unwrap(); - conn.authenticate(&auth).await.unwrap(); - let mut connection: AuthenticatedConn< - TcpStream, - fn(AsyncEvent<'static>) -> BoxFuture<'static, Result<(), ConnError>>, - > = conn.into_authenticated().await; - let tor_key = torut::onion::TorSecretKeyV3::generate(); - connection.get_conf("SocksPort").await.unwrap(); - connection - .add_onion_v3( - &tor_key, - false, - false, - false, - None, - &mut [(443_u16, SocketAddr::from(([127, 0, 0, 1], 8443)))].iter(), - ) - .await - .unwrap(); - connection - .del_onion( - &tor_key - .public() - .get_onion_address() - .get_address_without_dot_onion(), - ) - .await - .unwrap(); - connection - .add_onion_v3( - &tor_key, - false, - false, - false, - None, - &mut [(8443_u16, SocketAddr::from(([127, 0, 0, 1], 8443)))].iter(), - ) - .await - .unwrap(); -} diff --git a/core/startos/src/net/utils.rs b/core/startos/src/net/utils.rs deleted file mode 100644 index e496bd1f..00000000 --- a/core/startos/src/net/utils.rs +++ /dev/null @@ -1,166 +0,0 @@ -use std::convert::Infallible; -use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr}; -use std::path::Path; - -use async_stream::try_stream; -use color_eyre::eyre::eyre; -use futures::stream::BoxStream; -use futures::{StreamExt, TryStreamExt}; -use ipnet::{Ipv4Net, Ipv6Net}; -use tokio::net::{TcpListener, TcpStream}; -use tokio::process::Command; - -use crate::util::Invoke; -use crate::Error; - -fn parse_iface_ip(output: &str) -> Result, Error> { - let output = output.trim(); - if output.is_empty() { - return Ok(Vec::new()); - } - let mut res = Vec::new(); - for line in output.lines() { - if let Some(ip) = line.split_ascii_whitespace().nth(3) { - res.push(ip) - } else { - return Err(Error::new( - eyre!("malformed output from `ip`"), - crate::ErrorKind::Network, - )); - } - } - Ok(res) -} - -pub async fn get_iface_ipv4_addr(iface: &str) -> Result, Error> { - Ok(parse_iface_ip(&String::from_utf8( - Command::new("ip") - .arg("-4") - .arg("-o") - .arg("addr") - .arg("show") - .arg(iface) - .invoke(crate::ErrorKind::Network) - .await?, - )?)? - .into_iter() - .map(|s| Ok::<_, Error>((s.split("/").next().unwrap().parse()?, s.parse()?))) - .next() - .transpose()?) -} - -pub async fn get_iface_ipv6_addr(iface: &str) -> Result, Error> { - Ok(parse_iface_ip(&String::from_utf8( - Command::new("ip") - .arg("-6") - .arg("-o") - .arg("addr") - .arg("show") - .arg(iface) - .invoke(crate::ErrorKind::Network) - .await?, - )?)? - .into_iter() - .find(|ip| !ip.starts_with("fe80::")) - .map(|s| Ok::<_, Error>((s.split("/").next().unwrap().parse()?, s.parse()?))) - .transpose()?) -} - -pub async fn iface_is_physical(iface: &str) -> bool { - tokio::fs::metadata(Path::new("/sys/class/net").join(iface).join("device")) - .await - .is_ok() -} - -pub async fn iface_is_wireless(iface: &str) -> bool { - tokio::fs::metadata(Path::new("/sys/class/net").join(iface).join("wireless")) - .await - .is_ok() -} - -pub fn list_interfaces() -> BoxStream<'static, Result> { - try_stream! { - let mut ifaces = tokio::fs::read_dir("/sys/class/net").await?; - while let Some(iface) = ifaces.next_entry().await? { - if let Some(iface) = iface.file_name().into_string().ok() { - yield iface; - } - } - } - .boxed() -} - -pub async fn find_wifi_iface() -> Result, Error> { - let mut ifaces = list_interfaces(); - while let Some(iface) = ifaces.try_next().await? { - if iface_is_wireless(&iface).await { - return Ok(Some(iface)); - } - } - Ok(None) -} - -pub async fn find_eth_iface() -> Result { - let mut ifaces = list_interfaces(); - while let Some(iface) = ifaces.try_next().await? { - if iface_is_physical(&iface).await && !iface_is_wireless(&iface).await { - return Ok(iface); - } - } - Err(Error::new( - eyre!("Could not detect ethernet interface"), - crate::ErrorKind::Network, - )) -} - -#[pin_project::pin_project] -pub struct SingleAccept(Option); -impl SingleAccept { - pub fn new(conn: T) -> Self { - Self(Some(conn)) - } -} -impl hyper::server::accept::Accept for SingleAccept { - type Conn = T; - type Error = Infallible; - fn poll_accept( - self: std::pin::Pin<&mut Self>, - _cx: &mut std::task::Context<'_>, - ) -> std::task::Poll>> { - std::task::Poll::Ready(self.project().0.take().map(Ok)) - } -} - -pub struct TcpListeners { - listeners: Vec, -} -impl TcpListeners { - pub fn new(listeners: impl IntoIterator) -> Self { - Self { - listeners: listeners.into_iter().collect(), - } - } - - pub async fn accept(&self) -> std::io::Result<(TcpStream, SocketAddr)> { - futures::future::select_all(self.listeners.iter().map(|l| Box::pin(l.accept()))) - .await - .0 - } -} -impl hyper::server::accept::Accept for TcpListeners { - type Conn = TcpStream; - type Error = std::io::Error; - - fn poll_accept( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll>> { - for listener in self.listeners.iter() { - let poll = listener.poll_accept(cx); - if poll.is_ready() { - return poll.map(|a| a.map(|a| a.0)).map(Some); - } - } - std::task::Poll::Pending - } -} diff --git a/core/startos/src/net/vhost.rs b/core/startos/src/net/vhost.rs deleted file mode 100644 index bfbba057..00000000 --- a/core/startos/src/net/vhost.rs +++ /dev/null @@ -1,412 +0,0 @@ -use std::collections::BTreeMap; -use std::convert::Infallible; -use std::net::{IpAddr, Ipv6Addr, SocketAddr}; -use std::str::FromStr; -use std::sync::{Arc, Weak}; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use helpers::NonDetachingJoinHandle; -use http::{Response, Uri}; -use hyper::service::{make_service_fn, service_fn}; -use hyper::Body; -use models::ResultExt; -use tokio::net::{TcpListener, TcpStream}; -use tokio::sync::{Mutex, RwLock}; -use tokio_rustls::rustls::server::Acceptor; -use tokio_rustls::rustls::{RootCertStore, ServerConfig}; -use tokio_rustls::{LazyConfigAcceptor, TlsConnector}; -use tracing::instrument; - -use crate::net::keys::Key; -use crate::net::ssl::SslManager; -use crate::net::utils::SingleAccept; -use crate::prelude::*; -use crate::util::io::{BackTrackingReader, TimeoutStream}; - -// not allowed: <=1024, >=32768, 5355, 5432, 9050, 6010, 9051, 5353 - -pub struct VHostController { - ssl: Arc, - servers: Mutex>, -} -impl VHostController { - pub fn new(ssl: Arc) -> Self { - Self { - ssl, - servers: Mutex::new(BTreeMap::new()), - } - } - #[instrument(skip_all)] - pub async fn add( - &self, - key: Key, - hostname: Option, - external: u16, - target: SocketAddr, - connect_ssl: Result<(), AlpnInfo>, - ) -> Result, Error> { - let mut writable = self.servers.lock().await; - let server = if let Some(server) = writable.remove(&external) { - server - } else { - VHostServer::new(external, self.ssl.clone()).await? - }; - let rc = server - .add( - hostname, - TargetInfo { - addr: target, - connect_ssl, - key, - }, - ) - .await; - writable.insert(external, server); - Ok(rc?) - } - #[instrument(skip_all)] - pub async fn gc(&self, hostname: Option, external: u16) -> Result<(), Error> { - let mut writable = self.servers.lock().await; - if let Some(server) = writable.remove(&external) { - server.gc(hostname).await?; - if !server.is_empty().await? { - writable.insert(external, server); - } - } - Ok(()) - } -} - -#[derive(Clone, PartialEq, Eq, PartialOrd, Ord)] -struct TargetInfo { - addr: SocketAddr, - connect_ssl: Result<(), AlpnInfo>, - key: Key, -} - -#[derive(Clone, PartialEq, Eq, PartialOrd, Ord)] -pub enum AlpnInfo { - Reflect, - Specified(Vec>), -} - -struct VHostServer { - mapping: Weak, BTreeMap>>>>, - _thread: NonDetachingJoinHandle<()>, -} -impl VHostServer { - #[instrument(skip_all)] - async fn new(port: u16, ssl: Arc) -> Result { - // check if port allowed - let listener = TcpListener::bind(SocketAddr::new(Ipv6Addr::UNSPECIFIED.into(), port)) - .await - .with_kind(crate::ErrorKind::Network)?; - let mapping = Arc::new(RwLock::new(BTreeMap::new())); - Ok(Self { - mapping: Arc::downgrade(&mapping), - _thread: tokio::spawn(async move { - loop { - match listener.accept().await { - Ok((stream, _)) => { - let stream = - Box::pin(TimeoutStream::new(stream, Duration::from_secs(300))); - let mut stream = BackTrackingReader::new(stream); - stream.start_buffering(); - let mapping = mapping.clone(); - let ssl = ssl.clone(); - tokio::spawn(async move { - if let Err(e) = async { - let mid = match LazyConfigAcceptor::new( - Acceptor::default(), - &mut stream, - ) - .await - { - Ok(a) => a, - Err(_) => { - stream.rewind(); - return hyper::server::Server::builder( - SingleAccept::new(stream), - ) - .serve(make_service_fn(|_| async { - Ok::<_, Infallible>(service_fn(|req| async move { - let host = req - .headers() - .get(http::header::HOST) - .and_then(|host| host.to_str().ok()); - let uri = Uri::from_parts({ - let mut parts = - req.uri().to_owned().into_parts(); - parts.authority = host - .map(FromStr::from_str) - .transpose()?; - parts - })?; - Response::builder() - .status( - http::StatusCode::TEMPORARY_REDIRECT, - ) - .header( - http::header::LOCATION, - uri.to_string(), - ) - .body(Body::default()) - })) - })) - .await - .with_kind(crate::ErrorKind::Network); - } - }; - let target_name = - mid.client_hello().server_name().map(|s| s.to_owned()); - let target = { - let mapping = mapping.read().await; - mapping - .get(&target_name) - .into_iter() - .flatten() - .find(|(_, rc)| rc.strong_count() > 0) - .or_else(|| { - if target_name - .map(|s| s.parse::().is_ok()) - .unwrap_or(true) - { - mapping - .get(&None) - .into_iter() - .flatten() - .find(|(_, rc)| rc.strong_count() > 0) - } else { - None - } - }) - .map(|(target, _)| target.clone()) - }; - if let Some(target) = target { - let mut tcp_stream = - TcpStream::connect(target.addr).await?; - let key = - ssl.with_certs(target.key, target.addr.ip()).await?; - let cfg = ServerConfig::builder() - .with_safe_defaults() - .with_no_client_auth(); - let mut cfg = - if mid.client_hello().signature_schemes().contains( - &tokio_rustls::rustls::SignatureScheme::ED25519, - ) { - cfg.with_single_cert( - key.fullchain_ed25519() - .into_iter() - .map(|c| { - Ok(tokio_rustls::rustls::Certificate( - c.to_der()?, - )) - }) - .collect::>()?, - tokio_rustls::rustls::PrivateKey( - key.key() - .openssl_key_ed25519() - .private_key_to_der()?, - ), - ) - } else { - cfg.with_single_cert( - key.fullchain_nistp256() - .into_iter() - .map(|c| { - Ok(tokio_rustls::rustls::Certificate( - c.to_der()?, - )) - }) - .collect::>()?, - tokio_rustls::rustls::PrivateKey( - key.key() - .openssl_key_nistp256() - .private_key_to_der()?, - ), - ) - } - .with_kind(crate::ErrorKind::OpenSsl)?; - match target.connect_ssl { - Ok(()) => { - let mut client_cfg = - tokio_rustls::rustls::ClientConfig::builder() - .with_safe_defaults() - .with_root_certificates({ - let mut store = RootCertStore::empty(); - store.add( - &tokio_rustls::rustls::Certificate( - key.root_ca().to_der()?, - ), - ).with_kind(crate::ErrorKind::OpenSsl)?; - store - }) - .with_no_client_auth(); - client_cfg.alpn_protocols = mid - .client_hello() - .alpn() - .into_iter() - .flatten() - .map(|x| x.to_vec()) - .collect(); - let mut target_stream = - TlsConnector::from(Arc::new(client_cfg)) - .connect_with( - key.key() - .internal_address() - .as_str() - .try_into() - .with_kind( - crate::ErrorKind::OpenSsl, - )?, - tcp_stream, - |conn| { - cfg.alpn_protocols.extend( - conn.alpn_protocol() - .into_iter() - .map(|p| p.to_vec()), - ) - }, - ) - .await - .with_kind(crate::ErrorKind::OpenSsl)?; - let mut tls_stream = - match mid.into_stream(Arc::new(cfg)).await { - Ok(a) => a, - Err(e) => { - tracing::trace!( "VHostController: failed to accept TLS connection on port {port}: {e}"); - tracing::trace!("{e:?}"); - return Ok(()) - } - }; - tls_stream.get_mut().0.stop_buffering(); - tokio::io::copy_bidirectional( - &mut tls_stream, - &mut target_stream, - ) - .await - } - Err(AlpnInfo::Reflect) => { - for proto in - mid.client_hello().alpn().into_iter().flatten() - { - cfg.alpn_protocols.push(proto.into()); - } - let mut tls_stream = - match mid.into_stream(Arc::new(cfg)).await { - Ok(a) => a, - Err(e) => { - tracing::trace!( "VHostController: failed to accept TLS connection on port {port}: {e}"); - tracing::trace!("{e:?}"); - return Ok(()) - } - }; - tls_stream.get_mut().0.stop_buffering(); - tokio::io::copy_bidirectional( - &mut tls_stream, - &mut tcp_stream, - ) - .await - } - Err(AlpnInfo::Specified(alpn)) => { - cfg.alpn_protocols = alpn; - let mut tls_stream = - match mid.into_stream(Arc::new(cfg)).await { - Ok(a) => a, - Err(e) => { - tracing::trace!( "VHostController: failed to accept TLS connection on port {port}: {e}"); - tracing::trace!("{e:?}"); - return Ok(()) - } - }; - tls_stream.get_mut().0.stop_buffering(); - tokio::io::copy_bidirectional( - &mut tls_stream, - &mut tcp_stream, - ) - .await - } - } - .map_or_else( - |e| { - use std::io::ErrorKind as E; - match e.kind() { - E::UnexpectedEof | E::BrokenPipe | E::ConnectionAborted | E::ConnectionReset | E::ConnectionRefused | E::TimedOut | E::Interrupted | E::NotConnected => Ok(()), - _ => Err(e), - }}, - |_| Ok(()), - )?; - } else { - // 503 - } - Ok::<_, Error>(()) - } - .await - { - tracing::error!("Error in VHostController on port {port}: {e}"); - tracing::debug!("{e:?}") - } - }); - } - Err(e) => { - tracing::trace!( - "VHostController: failed to accept connection on port {port}: {e}" - ); - tracing::trace!("{e:?}"); - } - } - } - }) - .into(), - }) - } - async fn add(&self, hostname: Option, target: TargetInfo) -> Result, Error> { - if let Some(mapping) = Weak::upgrade(&self.mapping) { - let mut writable = mapping.write().await; - let mut targets = writable.remove(&hostname).unwrap_or_default(); - let rc = if let Some(rc) = Weak::upgrade(&targets.remove(&target).unwrap_or_default()) { - rc - } else { - Arc::new(()) - }; - targets.insert(target, Arc::downgrade(&rc)); - writable.insert(hostname, targets); - Ok(rc) - } else { - Err(Error::new( - eyre!("VHost Service Thread has exited"), - crate::ErrorKind::Network, - )) - } - } - async fn gc(&self, hostname: Option) -> Result<(), Error> { - if let Some(mapping) = Weak::upgrade(&self.mapping) { - let mut writable = mapping.write().await; - let mut targets = writable.remove(&hostname).unwrap_or_default(); - targets = targets - .into_iter() - .filter(|(_, rc)| rc.strong_count() > 0) - .collect(); - if !targets.is_empty() { - writable.insert(hostname, targets); - } - Ok(()) - } else { - Err(Error::new( - eyre!("VHost Service Thread has exited"), - crate::ErrorKind::Network, - )) - } - } - async fn is_empty(&self) -> Result { - if let Some(mapping) = Weak::upgrade(&self.mapping) { - Ok(mapping.read().await.is_empty()) - } else { - Err(Error::new( - eyre!("VHost Service Thread has exited"), - crate::ErrorKind::Network, - )) - } - } -} diff --git a/core/startos/src/net/web_server.rs b/core/startos/src/net/web_server.rs deleted file mode 100644 index c2e25a41..00000000 --- a/core/startos/src/net/web_server.rs +++ /dev/null @@ -1,61 +0,0 @@ -use std::convert::Infallible; -use std::net::SocketAddr; - -use futures::future::ready; -use futures::FutureExt; -use helpers::NonDetachingJoinHandle; -use hyper::service::{make_service_fn, service_fn}; -use hyper::Server; -use tokio::sync::oneshot; - -use crate::context::{DiagnosticContext, InstallContext, RpcContext, SetupContext}; -use crate::net::static_server::{ - diag_ui_file_router, install_ui_file_router, main_ui_server_router, setup_ui_file_router, -}; -use crate::net::HttpHandler; -use crate::Error; - -pub struct WebServer { - shutdown: oneshot::Sender<()>, - thread: NonDetachingJoinHandle<()>, -} -impl WebServer { - pub fn new(bind: SocketAddr, router: HttpHandler) -> Self { - let (shutdown, shutdown_recv) = oneshot::channel(); - let thread = NonDetachingJoinHandle::from(tokio::spawn(async move { - let server = Server::bind(&bind) - .http1_preserve_header_case(true) - .http1_title_case_headers(true) - .serve(make_service_fn(move |_| { - let router = router.clone(); - ready(Ok::<_, Infallible>(service_fn(move |req| router(req)))) - })) - .with_graceful_shutdown(shutdown_recv.map(|_| ())); - if let Err(e) = server.await { - tracing::error!("Spawning hyper server error: {}", e); - } - })); - Self { shutdown, thread } - } - - pub async fn shutdown(self) { - self.shutdown.send(()).unwrap_or_default(); - self.thread.await.unwrap() - } - - pub async fn main(bind: SocketAddr, ctx: RpcContext) -> Result { - Ok(Self::new(bind, main_ui_server_router(ctx).await?)) - } - - pub async fn setup(bind: SocketAddr, ctx: SetupContext) -> Result { - Ok(Self::new(bind, setup_ui_file_router(ctx).await?)) - } - - pub async fn diagnostic(bind: SocketAddr, ctx: DiagnosticContext) -> Result { - Ok(Self::new(bind, diag_ui_file_router(ctx).await?)) - } - - pub async fn install(bind: SocketAddr, ctx: InstallContext) -> Result { - Ok(Self::new(bind, install_ui_file_router(ctx).await?)) - } -} diff --git a/core/startos/src/net/wifi.rs b/core/startos/src/net/wifi.rs deleted file mode 100644 index 8429f920..00000000 --- a/core/startos/src/net/wifi.rs +++ /dev/null @@ -1,828 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet, HashMap}; -use std::path::Path; -use std::sync::Arc; -use std::time::Duration; - -use clap::ArgMatches; -use isocountry::CountryCode; -use lazy_static::lazy_static; -use regex::Regex; -use rpc_toolkit::command; -use tokio::process::Command; -use tokio::sync::RwLock; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::prelude::*; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::util::{display_none, Invoke}; -use crate::{Error, ErrorKind}; - -type WifiManager = Arc>; - -pub fn wifi_manager(ctx: &RpcContext) -> Result<&WifiManager, Error> { - if let Some(wifi_manager) = ctx.wifi_manager.as_ref() { - Ok(wifi_manager) - } else { - Err(Error::new( - color_eyre::eyre::eyre!("No WiFi interface available"), - ErrorKind::Wifi, - )) - } -} - -#[command(subcommands(add, connect, delete, get, country, available))] -pub async fn wifi() -> Result<(), Error> { - Ok(()) -} - -#[command(subcommands(get_available))] -pub async fn available() -> Result<(), Error> { - Ok(()) -} - -#[command(subcommands(set_country))] -pub async fn country() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn add( - #[context] ctx: RpcContext, - #[arg] ssid: String, - #[arg] password: String, -) -> Result<(), Error> { - let wifi_manager = wifi_manager(&ctx)?; - if !ssid.is_ascii() { - return Err(Error::new( - color_eyre::eyre::eyre!("SSID may not have special characters"), - ErrorKind::Wifi, - )); - } - if !password.is_ascii() { - return Err(Error::new( - color_eyre::eyre::eyre!("WiFi Password may not have special characters"), - ErrorKind::Wifi, - )); - } - async fn add_procedure( - db: PatchDb, - wifi_manager: WifiManager, - ssid: &Ssid, - password: &Psk, - ) -> Result<(), Error> { - tracing::info!("Adding new WiFi network: '{}'", ssid.0); - let mut wpa_supplicant = wifi_manager.write().await; - wpa_supplicant.add_network(db, ssid, password).await?; - drop(wpa_supplicant); - Ok(()) - } - if let Err(err) = add_procedure( - ctx.db.clone(), - wifi_manager.clone(), - &Ssid(ssid.clone()), - &Psk(password.clone()), - ) - .await - { - tracing::error!("Failed to add new WiFi network '{}': {}", ssid, err); - tracing::debug!("{:?}", err); - return Err(Error::new( - color_eyre::eyre::eyre!("Failed adding {}", ssid), - ErrorKind::Wifi, - )); - } - Ok(()) -} - -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn connect(#[context] ctx: RpcContext, #[arg] ssid: String) -> Result<(), Error> { - let wifi_manager = wifi_manager(&ctx)?; - if !ssid.is_ascii() { - return Err(Error::new( - color_eyre::eyre::eyre!("SSID may not have special characters"), - ErrorKind::Wifi, - )); - } - async fn connect_procedure( - db: PatchDb, - wifi_manager: WifiManager, - ssid: &Ssid, - ) -> Result<(), Error> { - let wpa_supplicant = wifi_manager.read().await; - let current = wpa_supplicant.get_current_network().await?; - drop(wpa_supplicant); - let mut wpa_supplicant = wifi_manager.write().await; - let connected = wpa_supplicant.select_network(db.clone(), ssid).await?; - if connected { - tracing::info!("Successfully connected to WiFi: '{}'", ssid.0); - } else { - tracing::info!("Failed to connect to WiFi: '{}'", ssid.0); - match current { - None => { - tracing::info!("No WiFi to revert to!"); - } - Some(current) => { - wpa_supplicant.select_network(db, ¤t).await?; - } - } - } - Ok(()) - } - - if let Err(err) = - connect_procedure(ctx.db.clone(), wifi_manager.clone(), &Ssid(ssid.clone())).await - { - tracing::error!("Failed to connect to WiFi network '{}': {}", &ssid, err); - return Err(Error::new( - color_eyre::eyre::eyre!("Can't connect to {}", ssid), - ErrorKind::Wifi, - )); - } - Ok(()) -} - -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn delete(#[context] ctx: RpcContext, #[arg] ssid: String) -> Result<(), Error> { - let wifi_manager = wifi_manager(&ctx)?; - if !ssid.is_ascii() { - return Err(Error::new( - color_eyre::eyre::eyre!("SSID may not have special characters"), - ErrorKind::Wifi, - )); - } - let wpa_supplicant = wifi_manager.read().await; - let current = wpa_supplicant.get_current_network().await?; - drop(wpa_supplicant); - let mut wpa_supplicant = wifi_manager.write().await; - let ssid = Ssid(ssid); - let is_current_being_removed = matches!(current, Some(current) if current == ssid); - let is_current_removed_and_no_hardwire = - is_current_being_removed && !interface_connected(&ctx.ethernet_interface).await?; - if is_current_removed_and_no_hardwire { - return Err(Error::new(color_eyre::eyre::eyre!("Forbidden: Deleting this network would make your server unreachable. Either connect to ethernet or connect to a different WiFi network to remedy this."), ErrorKind::Wifi)); - } - - wpa_supplicant.remove_network(ctx.db.clone(), &ssid).await?; - Ok(()) -} -#[derive(serde::Serialize, serde::Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct WiFiInfo { - ssids: HashMap, - connected: Option, - country: Option, - ethernet: bool, - available_wifi: Vec, -} -#[derive(serde::Serialize, serde::Deserialize, Clone)] -#[serde(rename_all = "kebab-case")] -pub struct WifiListInfo { - strength: SignalStrength, - security: Vec, -} -#[derive(serde::Serialize, serde::Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct WifiListOut { - ssid: Ssid, - strength: SignalStrength, - security: Vec, -} -pub type WifiList = HashMap; -fn display_wifi_info(info: WiFiInfo, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(info, matches); - } - - let mut table_global = Table::new(); - table_global.add_row(row![bc => - "CONNECTED", - "SIGNAL_STRENGTH", - "COUNTRY", - "ETHERNET", - ]); - table_global.add_row(row![ - &info - .connected - .as_ref() - .map_or("[N/A]".to_owned(), |c| c.0.clone()), - &info - .connected - .as_ref() - .and_then(|x| info.ssids.get(x)) - .map_or("[N/A]".to_owned(), |ss| format!("{}", ss.0)), - info.country.as_ref().map(|c| c.alpha2()).unwrap_or("00"), - &format!("{}", info.ethernet) - ]); - table_global.print_tty(false).unwrap(); - - let mut table_ssids = Table::new(); - table_ssids.add_row(row![bc => "SSID", "STRENGTH"]); - for (ssid, signal_strength) in &info.ssids { - let mut row = row![&ssid.0, format!("{}", signal_strength.0)]; - row.iter_mut() - .map(|c| { - c.style(Attr::ForegroundColor(match &signal_strength.0 { - x if x >= &90 => color::GREEN, - x if x == &50 => color::MAGENTA, - x if x == &0 => color::RED, - _ => color::YELLOW, - })) - }) - .for_each(drop); - table_ssids.add_row(row); - } - table_ssids.print_tty(false).unwrap(); - - let mut table_global = Table::new(); - table_global.add_row(row![bc => - "SSID", - "STRENGTH", - "SECURITY", - ]); - for table_info in info.available_wifi { - table_global.add_row(row![ - &table_info.ssid.0, - &format!("{}", table_info.strength.0), - &table_info.security.join(" ") - ]); - } - - table_global.print_tty(false).unwrap(); -} - -fn display_wifi_list(info: Vec, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(info, matches); - } - - let mut table_global = Table::new(); - table_global.add_row(row![bc => - "SSID", - "STRENGTH", - "SECURITY", - ]); - for table_info in info { - table_global.add_row(row![ - &table_info.ssid.0, - &format!("{}", table_info.strength.0), - &table_info.security.join(" ") - ]); - } - - table_global.print_tty(false).unwrap(); -} - -#[command(display(display_wifi_info))] -#[instrument(skip_all)] -pub async fn get( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - let wifi_manager = wifi_manager(&ctx)?; - let wpa_supplicant = wifi_manager.read().await; - let (list_networks, current_res, country_res, ethernet_res, signal_strengths) = tokio::join!( - wpa_supplicant.list_networks_low(), - wpa_supplicant.get_current_network(), - wpa_supplicant.get_country_low(), - interface_connected(&ctx.ethernet_interface), - wpa_supplicant.list_wifi_low() - ); - let signal_strengths = signal_strengths?; - let list_networks: BTreeSet<_> = list_networks?.into_iter().map(|(_, x)| x.ssid).collect(); - let available_wifi = { - let mut wifi_list: Vec = signal_strengths - .clone() - .into_iter() - .filter(|(ssid, _)| !list_networks.contains(ssid)) - .map(|(ssid, info)| WifiListOut { - ssid, - strength: info.strength, - security: info.security, - }) - .collect(); - wifi_list.sort_by_key(|x| x.strength); - wifi_list.reverse(); - wifi_list - }; - let ssids: HashMap = list_networks - .into_iter() - .map(|ssid| { - let signal_strength = signal_strengths - .get(&ssid) - .map(|x| x.strength) - .unwrap_or_default(); - (ssid, signal_strength) - }) - .collect(); - let current = current_res?; - Ok(WiFiInfo { - ssids, - connected: current, - country: country_res?, - ethernet: ethernet_res?, - available_wifi, - }) -} - -#[command(rename = "get", display(display_wifi_list))] -#[instrument(skip_all)] -pub async fn get_available( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result, Error> { - let wifi_manager = wifi_manager(&ctx)?; - let wpa_supplicant = wifi_manager.read().await; - let (wifi_list, network_list) = tokio::join!( - wpa_supplicant.list_wifi_low(), - wpa_supplicant.list_networks_low() - ); - let network_list = network_list? - .into_iter() - .map(|(_, info)| info.ssid) - .collect::>(); - let mut wifi_list: Vec = wifi_list? - .into_iter() - .filter(|(ssid, _)| !network_list.contains(ssid)) - .map(|(ssid, info)| WifiListOut { - ssid, - strength: info.strength, - security: info.security, - }) - .collect(); - wifi_list.sort_by_key(|x| x.strength); - wifi_list.reverse(); - Ok(wifi_list) -} - -#[command(rename = "set", display(display_none))] -pub async fn set_country( - #[context] ctx: RpcContext, - #[arg(parse(country_code_parse))] country: CountryCode, -) -> Result<(), Error> { - let wifi_manager = wifi_manager(&ctx)?; - if !interface_connected(&ctx.ethernet_interface).await? { - return Err(Error::new( - color_eyre::eyre::eyre!("Won't change country without hardwire connection"), - crate::ErrorKind::Wifi, - )); - } - let mut wpa_supplicant = wifi_manager.write().await; - wpa_supplicant.set_country_low(country.alpha2()).await?; - for (network_id, _wifi_info) in wpa_supplicant.list_networks_low().await? { - wpa_supplicant.remove_network_low(network_id).await?; - } - wpa_supplicant.remove_all_connections().await?; - - wpa_supplicant.save_config(ctx.db.clone()).await?; - - Ok(()) -} - -#[derive(Debug)] -pub struct WpaCli { - interface: String, -} -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] -pub struct NetworkId(String); - -/// Ssid are the names of the wifis, usually human readable. -#[derive( - Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, serde::Serialize, serde::Deserialize, -)] -pub struct Ssid(String); - -/// So a signal strength is a number between 0-100, I want the null option to be 0 since there is no signal -#[derive( - Clone, - Copy, - Debug, - Default, - PartialEq, - Eq, - PartialOrd, - Ord, - Hash, - serde::Serialize, - serde::Deserialize, -)] -pub struct SignalStrength(u8); - -impl SignalStrength { - fn new(size: Option) -> Self { - let size = match size { - None => return Self(0), - Some(x) => x, - }; - if size >= 100 { - return Self(100); - } - Self(size) - } -} - -#[derive(Debug, Clone)] -pub struct WifiInfo { - ssid: Ssid, - device: Option, -} - -#[derive(Clone, Debug)] -pub struct Psk(String); -impl WpaCli { - pub fn init(interface: String) -> Self { - WpaCli { interface } - } - - #[instrument(skip_all)] - pub async fn set_add_network_low(&mut self, ssid: &Ssid, psk: &Psk) -> Result<(), Error> { - let _ = Command::new("nmcli") - .arg("-a") - .arg("-w") - .arg("30") - .arg("d") - .arg("wifi") - .arg("con") - .arg(&ssid.0) - .arg("password") - .arg(&psk.0) - .invoke(ErrorKind::Wifi) - .await?; - Ok(()) - } - #[instrument(skip_all)] - pub async fn add_network_low(&mut self, ssid: &Ssid, psk: &Psk) -> Result<(), Error> { - if self.find_networks(ssid).await?.is_empty() { - Command::new("nmcli") - .arg("con") - .arg("add") - .arg("con-name") - .arg(&ssid.0) - .arg("type") - .arg("wifi") - .arg("ssid") - .arg(&ssid.0) - .invoke(ErrorKind::Wifi) - .await?; - } - Command::new("nmcli") - .arg("con") - .arg("modify") - .arg(&ssid.0) - .arg("wifi-sec.key-mgmt") - .arg("wpa-psk") - .invoke(ErrorKind::Wifi) - .await?; - Command::new("nmcli") - .arg("con") - .arg("modify") - .arg(&ssid.0) - .arg("ifname") - .arg(&self.interface) - .invoke(ErrorKind::Wifi) - .await - .map(|_| ()) - .unwrap_or_else(|e| { - tracing::warn!("Failed to set interface {} for {}", self.interface, ssid.0); - tracing::debug!("{:?}", e); - }); - Command::new("nmcli") - .arg("con") - .arg("modify") - .arg(&ssid.0) - .arg("wifi-sec.psk") - .arg(&psk.0) - .invoke(ErrorKind::Wifi) - .await?; - Ok(()) - } - pub async fn set_country_low(&mut self, country_code: &str) -> Result<(), Error> { - let _ = Command::new("iw") - .arg("reg") - .arg("set") - .arg(country_code) - .invoke(ErrorKind::Wifi) - .await?; - - Ok(()) - } - pub async fn get_country_low(&self) -> Result, Error> { - let r = Command::new("iw") - .arg("reg") - .arg("get") - .invoke(ErrorKind::Wifi) - .await?; - let r = String::from_utf8(r)?; - lazy_static! { - static ref RE: Regex = Regex::new("country (\\w+):").unwrap(); - } - let first_country = r.lines().find(|s| s.contains("country")).ok_or_else(|| { - Error::new( - color_eyre::eyre::eyre!("Could not find a country config lines"), - ErrorKind::Wifi, - ) - })?; - let country = &RE.captures(first_country).ok_or_else(|| { - Error::new( - color_eyre::eyre::eyre!("Could not find a country config with regex"), - ErrorKind::Wifi, - ) - })?[1]; - if country == "00" { - Ok(None) - } else { - Ok(Some(CountryCode::for_alpha2(country).map_err(|_| { - Error::new( - color_eyre::eyre::eyre!("Invalid Country Code: {}", country), - ErrorKind::Wifi, - ) - })?)) - } - } - pub async fn remove_network_low(&mut self, id: NetworkId) -> Result<(), Error> { - let _ = Command::new("nmcli") - .arg("c") - .arg("del") - .arg(&id.0) - .invoke(ErrorKind::Wifi) - .await?; - Ok(()) - } - #[instrument(skip_all)] - pub async fn list_networks_low(&self) -> Result, Error> { - let r = Command::new("nmcli") - .arg("-t") - .arg("c") - .arg("show") - .invoke(ErrorKind::Wifi) - .await?; - let r = String::from_utf8(r)?; - tracing::info!("JCWM: all the networks: {:?}", r); - Ok(r.lines() - .filter_map(|l| { - let mut cs = l.split(':'); - let name = Ssid(cs.next()?.to_owned()); - let uuid = NetworkId(cs.next()?.to_owned()); - let connection_type = cs.next()?; - let device = cs.next(); - if !connection_type.contains("wireless") { - return None; - } - let info = WifiInfo { - ssid: name, - device: device.map(|x| x.to_owned()), - }; - Some((uuid, info)) - }) - .collect::>()) - } - - #[instrument(skip_all)] - pub async fn list_wifi_low(&self) -> Result { - let r = Command::new("nmcli") - .arg("-g") - .arg("SSID,SIGNAL,security") - .arg("d") - .arg("wifi") - .arg("list") - .invoke(ErrorKind::Wifi) - .await?; - Ok(String::from_utf8(r)? - .lines() - .filter_map(|l| { - let mut values = l.split(':'); - let ssid = Ssid(values.next()?.to_owned()); - let signal = SignalStrength::new(std::str::FromStr::from_str(values.next()?).ok()); - let security: Vec = - values.next()?.split(' ').map(|x| x.to_owned()).collect(); - Some(( - ssid, - WifiListInfo { - strength: signal, - security, - }, - )) - }) - .collect::()) - } - pub async fn select_network_low(&mut self, id: &NetworkId) -> Result<(), Error> { - let _ = Command::new("nmcli") - .arg("c") - .arg("up") - .arg(&id.0) - .invoke(ErrorKind::Wifi) - .await?; - Ok(()) - } - pub async fn remove_all_connections(&mut self) -> Result<(), Error> { - let location_connections = Path::new("/etc/NetworkManager/system-connections"); - let mut connections = tokio::fs::read_dir(&location_connections).await?; - while let Some(connection) = connections.next_entry().await? { - let path = connection.path(); - if path.is_file() { - let _ = tokio::fs::remove_file(&path).await?; - } - } - - Ok(()) - } - pub async fn save_config(&mut self, db: PatchDb) -> Result<(), Error> { - let new_country = self.get_country_low().await?; - db.mutate(|d| { - d.as_server_info_mut() - .as_last_wifi_region_mut() - .ser(&new_country) - }) - .await - } - async fn check_active_network(&self, ssid: &Ssid) -> Result, Error> { - Ok(self - .list_networks_low() - .await? - .iter() - .find_map(|(network_id, wifi_info)| { - wifi_info.device.as_ref()?; - if wifi_info.ssid == *ssid { - Some(network_id.clone()) - } else { - None - } - })) - } - pub async fn find_networks(&self, ssid: &Ssid) -> Result, Error> { - Ok(self - .list_networks_low() - .await? - .iter() - .filter_map(|(network_id, wifi_info)| { - if wifi_info.ssid == *ssid { - Some(network_id.clone()) - } else { - None - } - }) - .collect()) - } - #[instrument(skip_all)] - pub async fn select_network(&mut self, db: PatchDb, ssid: &Ssid) -> Result { - let m_id = self.check_active_network(ssid).await?; - match m_id { - None => Err(Error::new( - color_eyre::eyre::eyre!("SSID Not Found"), - ErrorKind::Wifi, - )), - Some(x) => { - self.select_network_low(&x).await?; - self.save_config(db).await?; - let connect = async { - let mut current; - loop { - current = self.get_current_network().await; - if let Ok(Some(ssid)) = ¤t { - tracing::debug!("Connected to: {}", ssid.0); - break; - } - tokio::time::sleep(Duration::from_millis(500)).await; - tracing::debug!("Retrying..."); - } - current - }; - let res = match tokio::time::timeout(Duration::from_secs(20), connect).await { - Err(_) => None, - Ok(net) => net?, - }; - tracing::debug!("{:?}", res); - Ok(match res { - None => false, - Some(net) => &net == ssid, - }) - } - } - } - #[instrument(skip_all)] - pub async fn get_current_network(&self) -> Result, Error> { - let r = Command::new("iwgetid") - .arg(&self.interface) - .arg("--raw") - .invoke(ErrorKind::Wifi) - .await?; - let output = String::from_utf8(r)?; - let network = output.trim(); - tracing::debug!("Current Network: \"{}\"", network); - if network.is_empty() { - Ok(None) - } else { - Ok(Some(Ssid(network.to_owned()))) - } - } - #[instrument(skip_all)] - pub async fn remove_network(&mut self, db: PatchDb, ssid: &Ssid) -> Result { - let found_networks = self.find_networks(ssid).await?; - if found_networks.is_empty() { - return Ok(true); - } - for network_id in found_networks { - self.remove_network_low(network_id).await?; - } - self.save_config(db).await?; - Ok(true) - } - #[instrument(skip_all)] - pub async fn set_add_network( - &mut self, - db: PatchDb, - ssid: &Ssid, - psk: &Psk, - ) -> Result<(), Error> { - self.set_add_network_low(ssid, psk).await?; - self.save_config(db).await?; - Ok(()) - } - #[instrument(skip_all)] - pub async fn add_network(&mut self, db: PatchDb, ssid: &Ssid, psk: &Psk) -> Result<(), Error> { - self.add_network_low(ssid, psk).await?; - self.save_config(db).await?; - Ok(()) - } -} - -#[instrument(skip_all)] -pub async fn interface_connected(interface: &str) -> Result { - let out = Command::new("ifconfig") - .arg(interface) - .invoke(ErrorKind::Wifi) - .await?; - let v = std::str::from_utf8(&out)? - .lines() - .find(|s| s.contains("inet")); - Ok(v.is_some()) -} - -pub fn country_code_parse(code: &str, _matches: &ArgMatches) -> Result { - CountryCode::for_alpha2(code).map_err(|_| { - Error::new( - color_eyre::eyre::eyre!("Invalid Country Code: {}", code), - ErrorKind::Wifi, - ) - }) -} - -#[instrument(skip_all)] -pub async fn synchronize_wpa_supplicant_conf>( - main_datadir: P, - wifi_iface: &str, - last_country_code: &Option, -) -> Result<(), Error> { - let persistent = main_datadir.as_ref().join("system-connections"); - tracing::debug!("persistent: {:?}", persistent); - // let supplicant = Path::new("/etc/wpa_supplicant.conf"); - - if tokio::fs::metadata(&persistent).await.is_err() { - tokio::fs::create_dir_all(&persistent).await?; - } - crate::disk::mount::util::bind(&persistent, "/etc/NetworkManager/system-connections", false) - .await?; - // if tokio::fs::metadata(&supplicant).await.is_err() { - // tokio::fs::write(&supplicant, include_str!("wpa_supplicant.conf.base")).await?; - // } - - Command::new("systemctl") - .arg("restart") - .arg("NetworkManager") - .invoke(ErrorKind::Wifi) - .await?; - Command::new("ifconfig") - .arg(wifi_iface) - .arg("up") - .invoke(ErrorKind::Wifi) - .await?; - if let Some(last_country_code) = last_country_code { - tracing::info!("Setting the region"); - let _ = Command::new("iw") - .arg("reg") - .arg("set") - .arg(last_country_code.alpha2()) - .invoke(ErrorKind::Wifi) - .await?; - } else { - tracing::info!("Setting the region fallback"); - let _ = Command::new("iw") - .arg("reg") - .arg("set") - .arg("US") - .invoke(ErrorKind::Wifi) - .await?; - } - Ok(()) -} diff --git a/core/startos/src/net/wpa_supplicant.conf.base b/core/startos/src/net/wpa_supplicant.conf.base deleted file mode 100644 index 9609196a..00000000 --- a/core/startos/src/net/wpa_supplicant.conf.base +++ /dev/null @@ -1,3 +0,0 @@ -ctrl_interface=DIR=/run/wpa_supplicant GROUP=netdev -update_config=1 -country=US \ No newline at end of file diff --git a/core/startos/src/net/ws_server.rs b/core/startos/src/net/ws_server.rs deleted file mode 100644 index 16519c6c..00000000 --- a/core/startos/src/net/ws_server.rs +++ /dev/null @@ -1,94 +0,0 @@ -use crate::context::RpcContext; - -pub async fn ws_server_handle(rpc_ctx: RpcContext) { - - let ws_ctx = rpc_ctx.clone(); - let ws_server_handle = { - let builder = Server::bind(&ws_ctx.bind_ws); - - let make_svc = ::rpc_toolkit::hyper::service::make_service_fn(move |_| { - let ctx = ws_ctx.clone(); - async move { - Ok::<_, ::rpc_toolkit::hyper::Error>(::rpc_toolkit::hyper::service::service_fn( - move |req| { - let ctx = ctx.clone(); - async move { - tracing::debug!("Request to {}", req.uri().path()); - match req.uri().path() { - "/ws/db" => { - Ok(subscribe(ctx, req).await.unwrap_or_else(err_to_500)) - } - path if path.starts_with("/ws/rpc/") => { - match RequestGuid::from( - path.strip_prefix("/ws/rpc/").unwrap(), - ) { - None => { - tracing::debug!("No Guid Path"); - Response::builder() - .status(StatusCode::BAD_REQUEST) - .body(Body::empty()) - } - Some(guid) => { - match ctx.get_ws_continuation_handler(&guid).await { - Some(cont) => match cont(req).await { - Ok(r) => Ok(r), - Err(e) => Response::builder() - .status( - StatusCode::INTERNAL_SERVER_ERROR, - ) - .body(Body::from(format!("{}", e))), - }, - _ => Response::builder() - .status(StatusCode::NOT_FOUND) - .body(Body::empty()), - } - } - } - } - path if path.starts_with("/rest/rpc/") => { - match RequestGuid::from( - path.strip_prefix("/rest/rpc/").unwrap(), - ) { - None => { - tracing::debug!("No Guid Path"); - Response::builder() - .status(StatusCode::BAD_REQUEST) - .body(Body::empty()) - } - Some(guid) => { - match ctx.get_rest_continuation_handler(&guid).await - { - None => Response::builder() - .status(StatusCode::NOT_FOUND) - .body(Body::empty()), - Some(cont) => match cont(req).await { - Ok(r) => Ok(r), - Err(e) => Response::builder() - .status( - StatusCode::INTERNAL_SERVER_ERROR, - ) - .body(Body::from(format!("{}", e))), - }, - } - } - } - } - _ => Response::builder() - .status(StatusCode::NOT_FOUND) - .body(Body::empty()), - } - } - }, - )) - } - }); - builder.serve(make_svc) - } - .with_graceful_shutdown({ - let mut shutdown = rpc_ctx.shutdown.subscribe(); - async move { - shutdown.recv().await.expect("context dropped"); - } - }); - -} \ No newline at end of file diff --git a/core/startos/src/notifications.rs b/core/startos/src/notifications.rs deleted file mode 100644 index 73351471..00000000 --- a/core/startos/src/notifications.rs +++ /dev/null @@ -1,308 +0,0 @@ -use std::collections::HashMap; -use std::fmt; -use std::str::FromStr; - -use chrono::{DateTime, Utc}; -use color_eyre::eyre::eyre; -use rpc_toolkit::command; -use sqlx::PgPool; -use tokio::sync::Mutex; -use tracing::instrument; - -use crate::backup::BackupReport; -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::display_none; -use crate::util::serde::display_serializable; -use crate::{Error, ErrorKind, ResultExt}; - -#[command(subcommands(list, delete, delete_before, create))] -pub async fn notification() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_serializable))] -#[instrument(skip_all)] -pub async fn list( - #[context] ctx: RpcContext, - #[arg] before: Option, - #[arg] limit: Option, -) -> Result, Error> { - let limit = limit.unwrap_or(40); - match before { - None => { - let records = sqlx::query!( - "SELECT id, package_id, created_at, code, level, title, message, data FROM notifications ORDER BY id DESC LIMIT $1", - limit as i64 - ).fetch_all(&ctx.secret_store).await?; - let notifs = records - .into_iter() - .map(|r| { - Ok(Notification { - id: r.id as u32, - package_id: r.package_id.and_then(|p| p.parse().ok()), - created_at: DateTime::from_utc(r.created_at, Utc), - code: r.code as u32, - level: match r.level.parse::() { - Ok(a) => a, - Err(e) => return Err(e.into()), - }, - title: r.title, - message: r.message, - data: match r.data { - None => serde_json::Value::Null, - Some(v) => match v.parse::() { - Ok(a) => a, - Err(e) => { - return Err(Error::new( - eyre!("Invalid Notification Data: {}", e), - ErrorKind::ParseDbField, - )) - } - }, - }, - }) - }) - .collect::, Error>>()?; - - ctx.db - .mutate(|d| { - d.as_server_info_mut() - .as_unread_notification_count_mut() - .ser(&0) - }) - .await?; - Ok(notifs) - } - Some(before) => { - let records = sqlx::query!( - "SELECT id, package_id, created_at, code, level, title, message, data FROM notifications WHERE id < $1 ORDER BY id DESC LIMIT $2", - before, - limit as i64 - ).fetch_all(&ctx.secret_store).await?; - let res = records - .into_iter() - .map(|r| { - Ok(Notification { - id: r.id as u32, - package_id: r.package_id.and_then(|p| p.parse().ok()), - created_at: DateTime::from_utc(r.created_at, Utc), - code: r.code as u32, - level: match r.level.parse::() { - Ok(a) => a, - Err(e) => return Err(e.into()), - }, - title: r.title, - message: r.message, - data: match r.data { - None => serde_json::Value::Null, - Some(v) => match v.parse::() { - Ok(a) => a, - Err(e) => { - return Err(Error::new( - eyre!("Invalid Notification Data: {}", e), - ErrorKind::ParseDbField, - )) - } - }, - }, - }) - }) - .collect::, Error>>()?; - Ok(res) - } - } -} - -#[command(display(display_none))] -pub async fn delete(#[context] ctx: RpcContext, #[arg] id: i32) -> Result<(), Error> { - sqlx::query!("DELETE FROM notifications WHERE id = $1", id) - .execute(&ctx.secret_store) - .await?; - Ok(()) -} - -#[command(rename = "delete-before", display(display_none))] -pub async fn delete_before(#[context] ctx: RpcContext, #[arg] before: i32) -> Result<(), Error> { - sqlx::query!("DELETE FROM notifications WHERE id < $1", before) - .execute(&ctx.secret_store) - .await?; - Ok(()) -} - -#[command(display(display_none))] -pub async fn create( - #[context] ctx: RpcContext, - #[arg] package: Option, - #[arg] level: NotificationLevel, - #[arg] title: String, - #[arg] message: String, -) -> Result<(), Error> { - ctx.notification_manager - .notify(ctx.db.clone(), package, level, title, message, (), None) - .await -} - -#[derive(Debug, Clone, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)] -#[serde(rename_all = "kebab-case")] -pub enum NotificationLevel { - Success, - Info, - Warning, - Error, -} -impl fmt::Display for NotificationLevel { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - NotificationLevel::Success => write!(f, "success"), - NotificationLevel::Info => write!(f, "info"), - NotificationLevel::Warning => write!(f, "warning"), - NotificationLevel::Error => write!(f, "error"), - } - } -} -pub struct InvalidNotificationLevel(String); -impl From for crate::Error { - fn from(val: InvalidNotificationLevel) -> Self { - Error::new( - eyre!("Invalid Notification Level: {}", val.0), - ErrorKind::ParseDbField, - ) - } -} -impl FromStr for NotificationLevel { - type Err = InvalidNotificationLevel; - fn from_str(s: &str) -> Result { - match s { - s if s == "success" => Ok(NotificationLevel::Success), - s if s == "info" => Ok(NotificationLevel::Info), - s if s == "warning" => Ok(NotificationLevel::Warning), - s if s == "error" => Ok(NotificationLevel::Error), - s => Err(InvalidNotificationLevel(s.to_string())), - } - } -} -impl fmt::Display for InvalidNotificationLevel { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "Invalid Notification Level: {}", self.0) - } -} -#[derive(Debug, serde::Serialize, serde::Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct Notification { - id: u32, - package_id: Option, // TODO change for package id newtype - created_at: DateTime, - code: u32, - level: NotificationLevel, - title: String, - message: String, - data: serde_json::Value, -} - -pub trait NotificationType: - serde::Serialize + for<'de> serde::Deserialize<'de> + std::fmt::Debug -{ - const CODE: i32; -} - -impl NotificationType for () { - const CODE: i32 = 0; -} -impl NotificationType for BackupReport { - const CODE: i32 = 1; -} - -pub struct NotificationManager { - sqlite: PgPool, - cache: Mutex, NotificationLevel, String), i64>>, -} -impl NotificationManager { - pub fn new(sqlite: PgPool) -> Self { - NotificationManager { - sqlite, - cache: Mutex::new(HashMap::new()), - } - } - #[instrument(skip(db, subtype, self))] - pub async fn notify( - &self, - db: PatchDb, - package_id: Option, - level: NotificationLevel, - title: String, - message: String, - subtype: T, - debounce_interval: Option, - ) -> Result<(), Error> { - let peek = db.peek().await; - if !self - .should_notify(&package_id, &level, &title, debounce_interval) - .await - { - return Ok(()); - } - let mut count = peek.as_server_info().as_unread_notification_count().de()?; - let sql_package_id = package_id.as_ref().map(|p| &**p); - let sql_code = T::CODE; - let sql_level = format!("{}", level); - let sql_data = - serde_json::to_string(&subtype).with_kind(crate::ErrorKind::Serialization)?; - sqlx::query!( - "INSERT INTO notifications (package_id, code, level, title, message, data) VALUES ($1, $2, $3, $4, $5, $6)", - sql_package_id, - sql_code as i32, - sql_level, - title, - message, - sql_data - ).execute(&self.sqlite).await?; - count += 1; - db.mutate(|db| { - db.as_server_info_mut() - .as_unread_notification_count_mut() - .ser(&count) - }) - .await - } - async fn should_notify( - &self, - package_id: &Option, - level: &NotificationLevel, - title: &String, - debounce_interval: Option, - ) -> bool { - let mut guard = self.cache.lock().await; - let k = (package_id.clone(), level.clone(), title.clone()); - let v = (*guard).get(&k); - match v { - None => { - (*guard).insert(k, Utc::now().timestamp()); - true - } - Some(last_issued) => match debounce_interval { - None => { - (*guard).insert(k, Utc::now().timestamp()); - true - } - Some(interval) => { - if last_issued + interval as i64 > Utc::now().timestamp() { - false - } else { - (*guard).insert(k, Utc::now().timestamp()); - true - } - } - }, - } - } -} - -#[test] -fn serialization() { - println!( - "{}", - serde_json::json!({ "test": "abcdefg", "num": 32, "nested": { "inner": null, "xyz": [0,2,4]}}) - ) -} diff --git a/core/startos/src/os_install/fstab.template b/core/startos/src/os_install/fstab.template deleted file mode 100644 index c299a0aa..00000000 --- a/core/startos/src/os_install/fstab.template +++ /dev/null @@ -1,3 +0,0 @@ -{boot} /boot vfat umask=0077 0 2 -{efi} /boot/efi vfat umask=0077 0 1 -{root} / ext4 defaults 0 1 \ No newline at end of file diff --git a/core/startos/src/os_install/gpt.rs b/core/startos/src/os_install/gpt.rs deleted file mode 100644 index 4139b4cf..00000000 --- a/core/startos/src/os_install/gpt.rs +++ /dev/null @@ -1,136 +0,0 @@ -use std::path::Path; - -use color_eyre::eyre::eyre; -use gpt::disk::LogicalBlockSize; -use gpt::GptConfig; - -use crate::disk::util::DiskInfo; -use crate::disk::OsPartitionInfo; -use crate::os_install::partition_for; -use crate::Error; - -pub async fn partition(disk: &DiskInfo, overwrite: bool) -> Result { - let efi = { - let disk = disk.clone(); - tokio::task::spawn_blocking(move || { - let use_efi = Path::new("/sys/firmware/efi").exists(); - let mut device = Box::new( - std::fs::File::options() - .read(true) - .write(true) - .open(&disk.logicalname)?, - ); - let (mut gpt, guid_part) = if overwrite { - let mbr = gpt::mbr::ProtectiveMBR::with_lb_size( - u32::try_from((disk.capacity / 512) - 1).unwrap_or(0xFF_FF_FF_FF), - ); - mbr.overwrite_lba0(&mut device)?; - ( - GptConfig::new() - .writable(true) - .initialized(false) - .logical_block_size(LogicalBlockSize::Lb512) - .create_from_device(device, None)?, - None, - ) - } else { - let gpt = GptConfig::new() - .writable(true) - .initialized(true) - .logical_block_size(LogicalBlockSize::Lb512) - .open_from_device(device)?; - let mut guid_part = None; - for (idx, part_info) in disk - .partitions - .iter() - .enumerate() - .map(|(idx, x)| (idx + 1, x)) - { - if let Some(entry) = gpt.partitions().get(&(idx as u32)) { - if part_info.guid.is_some() { - if entry.first_lba < if use_efi { 33759266 } else { 33570850 } { - return Err(Error::new( - eyre!("Not enough space before embassy data"), - crate::ErrorKind::InvalidRequest, - )); - } - guid_part = Some(entry.clone()); - break; - } - } - } - (gpt, guid_part) - }; - - gpt.update_partitions(Default::default())?; - - let efi = if use_efi { - gpt.add_partition("efi", 100 * 1024 * 1024, gpt::partition_types::EFI, 0, None)?; - true - } else { - gpt.add_partition( - "bios-grub", - 8 * 1024 * 1024, - gpt::partition_types::BIOS, - 0, - None, - )?; - false - }; - gpt.add_partition( - "boot", - 1024 * 1024 * 1024, - gpt::partition_types::LINUX_FS, - 0, - None, - )?; - gpt.add_partition( - "root", - 15 * 1024 * 1024 * 1024, - match *crate::ARCH { - "x86_64" => gpt::partition_types::LINUX_ROOT_X64, - "aarch64" => gpt::partition_types::LINUX_ROOT_ARM_64, - _ => gpt::partition_types::LINUX_FS, - }, - 0, - None, - )?; - - if overwrite { - gpt.add_partition( - "data", - gpt.find_free_sectors() - .iter() - .map(|(_, size)| *size * u64::from(*gpt.logical_block_size())) - .max() - .ok_or_else(|| { - Error::new( - eyre!("No free space left on device"), - crate::ErrorKind::BlockDevice, - ) - })?, - gpt::partition_types::LINUX_LVM, - 0, - None, - )?; - } else if let Some(guid_part) = guid_part { - let mut parts = gpt.partitions().clone(); - parts.insert(gpt.find_next_partition_id(), guid_part); - gpt.update_partitions(parts)?; - } - - gpt.write()?; - - Ok(efi) - }) - .await - .unwrap()? - }; - - Ok(OsPartitionInfo { - efi: efi.then(|| partition_for(&disk.logicalname, 1)), - bios: (!efi).then(|| partition_for(&disk.logicalname, 1)), - boot: partition_for(&disk.logicalname, 2), - root: partition_for(&disk.logicalname, 3), - }) -} diff --git a/core/startos/src/os_install/mbr.rs b/core/startos/src/os_install/mbr.rs deleted file mode 100644 index 72319023..00000000 --- a/core/startos/src/os_install/mbr.rs +++ /dev/null @@ -1,88 +0,0 @@ -use color_eyre::eyre::eyre; -use mbrman::{MBRPartitionEntry, CHS, MBR}; - -use crate::disk::util::DiskInfo; -use crate::disk::OsPartitionInfo; -use crate::os_install::partition_for; -use crate::Error; - -pub async fn partition(disk: &DiskInfo, overwrite: bool) -> Result { - { - let sectors = (disk.capacity / 512) as u32; - let disk = disk.clone(); - tokio::task::spawn_blocking(move || { - let mut file = std::fs::File::options() - .read(true) - .write(true) - .open(&disk.logicalname)?; - let (mut mbr, guid_part) = if overwrite { - (MBR::new_from(&mut file, 512, rand::random())?, None) - } else { - let mut mbr = MBR::read_from(&mut file, 512)?; - let mut guid_part = None; - for (idx, part_info) in disk - .partitions - .iter() - .enumerate() - .map(|(idx, x)| (idx + 1, x)) - { - if let Some(entry) = mbr.get_mut(idx) { - if part_info.guid.is_some() { - if entry.starting_lba < 33556480 { - return Err(Error::new( - eyre!("Not enough space before embassy data"), - crate::ErrorKind::InvalidRequest, - )); - } - guid_part = Some(std::mem::replace(entry, MBRPartitionEntry::empty())); - } - *entry = MBRPartitionEntry::empty(); - } - } - (mbr, guid_part) - }; - - mbr[1] = MBRPartitionEntry { - boot: 0x80, - first_chs: CHS::empty(), - sys: 0x0b, - last_chs: CHS::empty(), - starting_lba: 2048, - sectors: 2099200 - 2048, - }; - mbr[2] = MBRPartitionEntry { - boot: 0, - first_chs: CHS::empty(), - sys: 0x83, - last_chs: CHS::empty(), - starting_lba: 2099200, - sectors: 33556480 - 2099200, - }; - - if overwrite { - mbr[3] = MBRPartitionEntry { - boot: 0, - first_chs: CHS::empty(), - sys: 0x8e, - last_chs: CHS::empty(), - starting_lba: 33556480, - sectors: sectors - 33556480, - } - } else if let Some(guid_part) = guid_part { - mbr[3] = guid_part; - } - mbr.write_into(&mut file)?; - - Ok(()) - }) - .await - .unwrap()?; - } - - Ok(OsPartitionInfo { - efi: None, - bios: None, - boot: partition_for(&disk.logicalname, 1), - root: partition_for(&disk.logicalname, 2), - }) -} diff --git a/core/startos/src/os_install/mod.rs b/core/startos/src/os_install/mod.rs deleted file mode 100644 index 9e21e9f2..00000000 --- a/core/startos/src/os_install/mod.rs +++ /dev/null @@ -1,340 +0,0 @@ -use std::path::{Path, PathBuf}; - -use color_eyre::eyre::eyre; -use models::Error; -use rpc_toolkit::command; -use serde::{Deserialize, Serialize}; -use tokio::process::Command; - -use crate::context::InstallContext; -use crate::disk::mount::filesystem::bind::Bind; -use crate::disk::mount::filesystem::block_dev::BlockDev; -use crate::disk::mount::filesystem::efivarfs::EfiVarFs; -use crate::disk::mount::filesystem::{MountType, ReadWrite}; -use crate::disk::mount::guard::{MountGuard, TmpMountGuard}; -use crate::disk::util::{DiskInfo, PartitionTable}; -use crate::disk::OsPartitionInfo; -use crate::net::utils::{find_eth_iface, find_wifi_iface}; -use crate::util::serde::IoFormat; -use crate::util::{display_none, Invoke}; -use crate::ARCH; - -mod gpt; -mod mbr; - -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct PostInstallConfig { - os_partitions: OsPartitionInfo, - ethernet_interface: String, - wifi_interface: Option, -} - -#[command(subcommands(disk, execute, reboot))] -pub fn install() -> Result<(), Error> { - Ok(()) -} - -#[command(subcommands(list))] -pub fn disk() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_none))] -pub async fn list() -> Result, Error> { - let skip = match async { - Ok::<_, Error>( - Path::new( - &String::from_utf8( - Command::new("grub-probe-default") - .arg("-t") - .arg("disk") - .arg("/run/live/medium") - .invoke(crate::ErrorKind::Grub) - .await?, - )? - .trim(), - ) - .to_owned(), - ) - } - .await - { - Ok(a) => Some(a), - Err(e) => { - tracing::error!("Could not determine live usb device: {}", e); - tracing::debug!("{:?}", e); - None - } - }; - Ok(crate::disk::util::list(&Default::default()) - .await? - .into_iter() - .filter(|i| Some(&*i.logicalname) != skip.as_deref()) - .collect()) -} - -pub fn partition_for(disk: impl AsRef, idx: usize) -> PathBuf { - let disk_path = disk.as_ref(); - let (root, leaf) = if let (Some(root), Some(leaf)) = ( - disk_path.parent(), - disk_path.file_name().and_then(|s| s.to_str()), - ) { - (root, leaf) - } else { - return Default::default(); - }; - if leaf.ends_with(|c: char| c.is_ascii_digit()) { - root.join(format!("{}p{}", leaf, idx)) - } else { - root.join(format!("{}{}", leaf, idx)) - } -} - -async fn partition(disk: &mut DiskInfo, overwrite: bool) -> Result { - let partition_type = match (overwrite, disk.partition_table) { - (true, _) | (_, None) => PartitionTable::Gpt, - (_, Some(t)) => t, - }; - disk.partition_table = Some(partition_type); - match partition_type { - PartitionTable::Gpt => gpt::partition(disk, overwrite).await, - PartitionTable::Mbr => mbr::partition(disk, overwrite).await, - } -} - -#[command(display(display_none))] -pub async fn execute( - #[arg] logicalname: PathBuf, - #[arg(short = 'o')] mut overwrite: bool, -) -> Result<(), Error> { - let mut disk = crate::disk::util::list(&Default::default()) - .await? - .into_iter() - .find(|d| &d.logicalname == &logicalname) - .ok_or_else(|| { - Error::new( - eyre!("Unknown disk {}", logicalname.display()), - crate::ErrorKind::DiskManagement, - ) - })?; - let eth_iface = find_eth_iface().await?; - let wifi_iface = find_wifi_iface().await?; - - overwrite |= disk.guid.is_none() && disk.partitions.iter().all(|p| p.guid.is_none()); - - let part_info = partition(&mut disk, overwrite).await?; - - if let Some(efi) = &part_info.efi { - Command::new("mkfs.vfat") - .arg(efi) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Command::new("fatlabel") - .arg(efi) - .arg("efi") - .invoke(crate::ErrorKind::DiskManagement) - .await?; - } - - Command::new("mkfs.vfat") - .arg(&part_info.boot) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Command::new("fatlabel") - .arg(&part_info.boot) - .arg("boot") - .invoke(crate::ErrorKind::DiskManagement) - .await?; - - if !overwrite { - if let Ok(guard) = - TmpMountGuard::mount(&BlockDev::new(part_info.root.clone()), MountType::ReadWrite).await - { - if let Err(e) = async { - // cp -r ${guard}/config /tmp/config - if tokio::fs::metadata(guard.as_ref().join("config/upgrade")) - .await - .is_ok() - { - tokio::fs::remove_file(guard.as_ref().join("config/upgrade")).await?; - } - if tokio::fs::metadata(guard.as_ref().join("config/disk.guid")) - .await - .is_ok() - { - tokio::fs::remove_file(guard.as_ref().join("config/disk.guid")).await?; - } - Command::new("cp") - .arg("-r") - .arg(guard.as_ref().join("config")) - .arg("/tmp/config.bak") - .invoke(crate::ErrorKind::Filesystem) - .await?; - Ok::<_, Error>(()) - } - .await - { - tracing::error!("Error recovering previous config: {e}"); - tracing::debug!("{e:?}"); - } - guard.unmount().await?; - } - } - - Command::new("mkfs.btrfs") - .arg("-f") - .arg(&part_info.root) - .invoke(crate::ErrorKind::DiskManagement) - .await?; - Command::new("btrfs") - .arg("property") - .arg("set") - .arg(&part_info.root) - .arg("label") - .arg("rootfs") - .invoke(crate::ErrorKind::DiskManagement) - .await?; - - let rootfs = TmpMountGuard::mount(&BlockDev::new(&part_info.root), ReadWrite).await?; - if tokio::fs::metadata("/tmp/config.bak").await.is_ok() { - Command::new("cp") - .arg("-r") - .arg("/tmp/config.bak") - .arg(rootfs.as_ref().join("config")) - .invoke(crate::ErrorKind::Filesystem) - .await?; - } else { - tokio::fs::create_dir(rootfs.as_ref().join("config")).await?; - } - tokio::fs::create_dir(rootfs.as_ref().join("next")).await?; - let current = rootfs.as_ref().join("current"); - tokio::fs::create_dir(¤t).await?; - - tokio::fs::create_dir(current.join("boot")).await?; - let boot = MountGuard::mount( - &BlockDev::new(&part_info.boot), - current.join("boot"), - ReadWrite, - ) - .await?; - - let efi = if let Some(efi) = &part_info.efi { - Some(MountGuard::mount(&BlockDev::new(efi), current.join("boot/efi"), ReadWrite).await?) - } else { - None - }; - - Command::new("unsquashfs") - .arg("-n") - .arg("-f") - .arg("-d") - .arg(¤t) - .arg("/run/live/medium/live/filesystem.squashfs") - .invoke(crate::ErrorKind::Filesystem) - .await?; - - tokio::fs::write( - rootfs.as_ref().join("config/config.yaml"), - IoFormat::Yaml.to_vec(&PostInstallConfig { - os_partitions: part_info.clone(), - ethernet_interface: eth_iface, - wifi_interface: wifi_iface, - })?, - ) - .await?; - - tokio::fs::write( - current.join("etc/fstab"), - format!( - include_str!("fstab.template"), - boot = part_info.boot.display(), - efi = part_info - .efi - .as_ref() - .map(|p| p.display().to_string()) - .unwrap_or_else(|| "# N/A".to_owned()), - root = part_info.root.display(), - ), - ) - .await?; - - Command::new("chroot") - .arg(¤t) - .arg("systemd-machine-id-setup") - .invoke(crate::ErrorKind::Systemd) - .await?; - - Command::new("chroot") - .arg(¤t) - .arg("ssh-keygen") - .arg("-A") - .invoke(crate::ErrorKind::OpenSsh) - .await?; - - let embassy_fs = MountGuard::mount( - &Bind::new(rootfs.as_ref()), - current.join("media/embassy/embassyfs"), - MountType::ReadOnly, - ) - .await?; - let dev = MountGuard::mount(&Bind::new("/dev"), current.join("dev"), ReadWrite).await?; - let proc = MountGuard::mount(&Bind::new("/proc"), current.join("proc"), ReadWrite).await?; - let sys = MountGuard::mount(&Bind::new("/sys"), current.join("sys"), ReadWrite).await?; - let efivarfs = if tokio::fs::metadata("/sys/firmware/efi").await.is_ok() { - Some( - MountGuard::mount( - &EfiVarFs, - current.join("sys/firmware/efi/efivars"), - ReadWrite, - ) - .await?, - ) - } else { - None - }; - - let mut install = Command::new("chroot"); - install.arg(¤t).arg("grub-install"); - if tokio::fs::metadata("/sys/firmware/efi").await.is_err() { - install.arg("--target=i386-pc"); - } else { - match *ARCH { - "x86_64" => install.arg("--target=x86_64-efi"), - "aarch64" => install.arg("--target=arm64-efi"), - _ => &mut install, - }; - } - install - .arg(&disk.logicalname) - .invoke(crate::ErrorKind::Grub) - .await?; - - Command::new("chroot") - .arg(¤t) - .arg("update-grub2") - .invoke(crate::ErrorKind::Grub) - .await?; - dev.unmount(false).await?; - if let Some(efivarfs) = efivarfs { - efivarfs.unmount(false).await?; - } - sys.unmount(false).await?; - proc.unmount(false).await?; - embassy_fs.unmount(false).await?; - if let Some(efi) = efi { - efi.unmount(false).await?; - } - boot.unmount(false).await?; - rootfs.unmount().await?; - Ok(()) -} - -#[command(display(display_none))] -pub async fn reboot(#[context] ctx: InstallContext) -> Result<(), Error> { - Command::new("sync") - .invoke(crate::ErrorKind::Filesystem) - .await?; - ctx.shutdown.send(()).unwrap(); - Ok(()) -} diff --git a/core/startos/src/prelude.rs b/core/startos/src/prelude.rs deleted file mode 100644 index ab5de1d3..00000000 --- a/core/startos/src/prelude.rs +++ /dev/null @@ -1,6 +0,0 @@ -pub use color_eyre::eyre::eyre; -pub use models::OptionExt; - -pub use crate::db::prelude::*; -pub use crate::ensure_code; -pub use crate::error::{Error, ErrorCollection, ErrorKind, ResultExt}; diff --git a/core/startos/src/procedure/build.rs b/core/startos/src/procedure/build.rs deleted file mode 100644 index e69de29b..00000000 diff --git a/core/startos/src/procedure/docker.rs b/core/startos/src/procedure/docker.rs deleted file mode 100644 index ad25953a..00000000 --- a/core/startos/src/procedure/docker.rs +++ /dev/null @@ -1,970 +0,0 @@ -use std::borrow::Cow; -use std::collections::{BTreeMap, BTreeSet, VecDeque}; -use std::ffi::{OsStr, OsString}; -use std::net::Ipv4Addr; -use std::os::unix::prelude::FileTypeExt; -use std::path::{Path, PathBuf}; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use futures::future::{BoxFuture, Either as EitherFuture}; -use futures::{FutureExt, TryStreamExt}; -use helpers::{NonDetachingJoinHandle, UnixRpcClient}; -use models::{Id, ImageId, SYSTEM_PACKAGE_ID}; -use nix::sys::signal; -use nix::unistd::Pid; -use serde::de::DeserializeOwned; -use serde::{Deserialize, Serialize}; -use serde_json::Value; -use tokio::io::{AsyncBufRead, AsyncBufReadExt, BufReader}; -use tokio::time::timeout; -use tracing::instrument; - -use super::ProcedureName; -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::docker::{remove_container, CONTAINER_TOOL}; -use crate::util::serde::{Duration as SerdeDuration, IoFormat}; -use crate::util::Version; -use crate::volume::{VolumeId, Volumes}; -use crate::{Error, ResultExt, HOST_IP}; - -pub const NET_TLD: &str = "embassy"; - -lazy_static::lazy_static! { - pub static ref SYSTEM_IMAGES: BTreeSet = { - let mut set = BTreeSet::new(); - - set.insert("compat".parse().unwrap()); - set.insert("utils".parse().unwrap()); - - set - }; -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct DockerContainers { - pub main: DockerContainer, - // #[serde(default)] - // pub aux: BTreeMap, -} - -/// This is like the docker procedures of the past designs, -/// but this time all the entrypoints and args are not -/// part of this struct by choice. Used for the times that we are creating our own entry points -#[derive(Clone, Debug, Deserialize, Serialize, patch_db::HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct DockerContainer { - pub image: ImageId, - #[serde(default)] - pub mounts: BTreeMap, - #[serde(default)] - pub shm_size_mb: Option, // TODO: use postfix sizing? like 1k vs 1m vs 1g - #[serde(default)] - pub sigterm_timeout: Option, - #[serde(default)] - pub system: bool, - #[serde(default)] - pub gpu_acceleration: bool, -} - -impl DockerContainer { - /// We created a new exec runner, where we are going to be passing the commands for it to run. - /// Idea is that we are going to send it command and get the inputs be filtered back from the manager. - /// Then we could in theory run commands without the cost of running the docker exec which is known to have - /// a dely of > 200ms which is not acceptable. - #[instrument(skip_all)] - pub async fn long_running_execute( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - ) -> Result<(LongRunning, UnixRpcClient), Error> { - let container_name = DockerProcedure::container_name(pkg_id, None); - - let socket_path = - Path::new("/tmp/embassy/containers").join(format!("{pkg_id}_{pkg_version}")); - if tokio::fs::metadata(&socket_path).await.is_ok() { - tokio::fs::remove_dir_all(&socket_path).await?; - } - tokio::fs::create_dir_all(&socket_path).await?; - - let mut cmd = LongRunning::setup_long_running_docker_cmd( - self, - ctx, - &container_name, - volumes, - pkg_id, - pkg_version, - &socket_path, - ) - .await?; - - let mut handle = cmd.spawn().with_kind(crate::ErrorKind::Docker)?; - - let client = UnixRpcClient::new(socket_path.join("rpc.sock")); - - let running_output = NonDetachingJoinHandle::from(tokio::spawn(async move { - if let Err(err) = handle - .wait() - .await - .map_err(|e| eyre!("Runtime error: {e:?}")) - { - tracing::error!("{}", err); - tracing::debug!("{:?}", err); - } - })); - - { - let socket = socket_path.join("rpc.sock"); - if let Err(_err) = timeout(Duration::from_secs(1), async move { - while tokio::fs::metadata(&socket).await.is_err() { - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - { - tracing::error!("Timed out waiting for init to create socket"); - } - } - - Ok((LongRunning { running_output }, client)) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct DockerProcedure { - pub image: ImageId, - #[serde(default)] - pub system: bool, - pub entrypoint: String, - #[serde(default)] - pub args: Vec, - #[serde(default)] - pub inject: bool, - #[serde(default)] - pub mounts: BTreeMap, - #[serde(default)] - pub io_format: Option, - #[serde(default)] - pub sigterm_timeout: Option, - #[serde(default)] - pub shm_size_mb: Option, // TODO: use postfix sizing? like 1k vs 1m vs 1g - #[serde(default)] - pub gpu_acceleration: bool, -} - -#[derive(Clone, Debug, Deserialize, Serialize, Default)] -#[serde(rename_all = "kebab-case")] -pub struct DockerInject { - #[serde(default)] - pub system: bool, - pub entrypoint: String, - #[serde(default)] - pub args: Vec, - #[serde(default)] - pub io_format: Option, - #[serde(default)] - pub sigterm_timeout: Option, -} -impl DockerProcedure { - pub fn main_docker_procedure( - container: &DockerContainer, - injectable: &DockerInject, - ) -> DockerProcedure { - DockerProcedure { - image: container.image.clone(), - system: injectable.system, - entrypoint: injectable.entrypoint.clone(), - args: injectable.args.clone(), - inject: false, - mounts: container.mounts.clone(), - io_format: injectable.io_format, - sigterm_timeout: injectable.sigterm_timeout, - shm_size_mb: container.shm_size_mb, - gpu_acceleration: container.gpu_acceleration, - } - } - - pub fn validate( - &self, - _eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - expected_io: bool, - ) -> Result<(), color_eyre::eyre::Report> { - for volume in self.mounts.keys() { - if !volumes.contains_key(volume) && !matches!(&volume, &VolumeId::Backup) { - color_eyre::eyre::bail!("unknown volume: {}", volume); - } - } - if self.system { - if !SYSTEM_IMAGES.contains(&self.image) { - color_eyre::eyre::bail!("unknown system image: {}", self.image); - } - } else if !image_ids.contains(&self.image) { - color_eyre::eyre::bail!("image for {} not contained in package", self.image); - } - if expected_io && self.io_format.is_none() { - color_eyre::eyre::bail!("expected io-format"); - } - Ok(()) - } - - #[instrument(skip_all)] - pub async fn execute( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - name: ProcedureName, - volumes: &Volumes, - input: Option, - timeout: Option, - ) -> Result, Error> { - let name = name.docker_name(); - let name: Option<&str> = name.as_deref(); - let mut cmd = tokio::process::Command::new(CONTAINER_TOOL); - let container_name = Self::container_name(pkg_id, name); - cmd.arg("run") - .arg("--rm") - .arg("--network=start9") - .arg(format!("--add-host=embassy:{}", Ipv4Addr::from(HOST_IP))) - .arg("--name") - .arg(&container_name) - .arg(format!("--hostname={}", &container_name)) - .arg("--no-healthcheck") - .kill_on_drop(true); - remove_container(&container_name, true).await?; - cmd.args(self.docker_args(ctx, pkg_id, pkg_version, volumes).await?); - let input_buf = if let (Some(input), Some(format)) = (&input, &self.io_format) { - cmd.stdin(std::process::Stdio::piped()); - Some(format.to_vec(input)?) - } else { - None - }; - cmd.stdout(std::process::Stdio::piped()); - cmd.stderr(std::process::Stdio::piped()); - tracing::trace!( - "{}", - format!("{:?}", cmd) - .split(r#"" ""#) - .collect::>() - .join(" ") - ); - let mut handle = cmd.spawn().with_kind(crate::ErrorKind::Docker)?; - let id = handle.id(); - let timeout_fut = if let Some(timeout) = timeout { - EitherFuture::Right(async move { - tokio::time::sleep(timeout).await; - - Ok(()) - }) - } else { - EitherFuture::Left(futures::future::pending::>()) - }; - if let (Some(input), Some(mut stdin)) = (&input_buf, handle.stdin.take()) { - use tokio::io::AsyncWriteExt; - stdin - .write_all(input) - .await - .with_kind(crate::ErrorKind::Docker)?; - stdin.flush().await?; - stdin.shutdown().await?; - drop(stdin); - } - enum Race { - Done(T), - TimedOut, - } - - let io_format = self.io_format; - let mut output = BufReader::new( - handle - .stdout - .take() - .ok_or_else(|| eyre!("Can't takeout stdout in execute")) - .with_kind(crate::ErrorKind::Docker)?, - ); - let output = NonDetachingJoinHandle::from(tokio::spawn(async move { - match async { - if let Some(format) = io_format { - return match max_by_lines(&mut output, None).await { - MaxByLines::Done(buffer) => { - Ok::( - match format.from_slice(buffer.as_bytes()) { - Ok(a) => a, - Err(e) => { - tracing::trace!( - "Failed to deserialize stdout from {}: {}, falling back to UTF-8 string.", - format, - e - ); - Value::String(buffer) - } - }, - ) - }, - MaxByLines::Error(e) => Err(e), - MaxByLines::Overflow(buffer) => Ok(Value::String(buffer)) - } - } - - let lines = buf_reader_to_lines(&mut output, 1000).await?; - if lines.is_empty() { - return Ok(Value::Null); - } - - let joined_output = lines.join("\n"); - Ok(Value::String(joined_output)) - }.await { - Ok(a) => Ok((a, output)), - Err(e) => Err((e, output)) - } - })); - let err_output = BufReader::new( - handle - .stderr - .take() - .ok_or_else(|| eyre!("Can't takeout std err")) - .with_kind(crate::ErrorKind::Docker)?, - ); - - let err_output = NonDetachingJoinHandle::from(tokio::spawn(async move { - let lines = buf_reader_to_lines(err_output, 1000).await?; - let joined_output = lines.join("\n"); - Ok::<_, Error>(joined_output) - })); - - let res = tokio::select! { - res = handle.wait() => Race::Done(res.with_kind(crate::ErrorKind::Docker)?), - res = timeout_fut => { - res?; - Race::TimedOut - }, - }; - let exit_status = match res { - Race::Done(x) => x, - Race::TimedOut => { - if let Some(id) = id { - signal::kill(Pid::from_raw(id as i32), signal::SIGKILL) - .with_kind(crate::ErrorKind::Docker)?; - } - return Ok(Err((143, "Timed out. Retrying soon...".to_owned()))); - } - }; - Ok( - if exit_status.success() || exit_status.code() == Some(143) { - Ok(serde_json::from_value( - output - .await - .with_kind(crate::ErrorKind::Unknown)? - .map(|(v, _)| v) - .map_err(|(e, _)| tracing::warn!("{}", e)) - .unwrap_or_default(), - ) - .with_kind(crate::ErrorKind::Deserialization)?) - } else { - Err(( - exit_status.code().unwrap_or_default(), - err_output.await.with_kind(crate::ErrorKind::Unknown)??, - )) - }, - ) - } - - #[instrument(skip_all)] - pub async fn inject( - &self, - _ctx: &RpcContext, - pkg_id: &PackageId, - _pkg_version: &Version, - _name: ProcedureName, - _volumes: &Volumes, - input: Option, - timeout: Option, - ) -> Result, Error> { - let mut cmd = tokio::process::Command::new(CONTAINER_TOOL); - - cmd.arg("exec"); - - cmd.args(self.docker_args_inject(pkg_id)); - let input_buf = if let (Some(input), Some(format)) = (&input, &self.io_format) { - cmd.stdin(std::process::Stdio::piped()); - Some(format.to_vec(input)?) - } else { - None - }; - cmd.stdout(std::process::Stdio::piped()); - cmd.stderr(std::process::Stdio::piped()); - tracing::trace!( - "{}", - format!("{:?}", cmd) - .split(r#"" ""#) - .collect::>() - .join(" ") - ); - let mut handle = cmd.spawn().with_kind(crate::ErrorKind::Docker)?; - let id = handle.id(); - let timeout_fut = if let Some(timeout) = timeout { - EitherFuture::Right(async move { - tokio::time::sleep(timeout).await; - - Ok(()) - }) - } else { - EitherFuture::Left(futures::future::pending::>()) - }; - if let (Some(input), Some(mut stdin)) = (&input_buf, handle.stdin.take()) { - use tokio::io::AsyncWriteExt; - stdin - .write_all(input) - .await - .with_kind(crate::ErrorKind::Docker)?; - stdin.flush().await?; - stdin.shutdown().await?; - drop(stdin); - } - enum Race { - Done(T), - TimedOut, - } - - let io_format = self.io_format; - let mut output = BufReader::new( - handle - .stdout - .take() - .ok_or_else(|| eyre!("Can't takeout stdout in inject")) - .with_kind(crate::ErrorKind::Docker)?, - ); - let output = NonDetachingJoinHandle::from(tokio::spawn(async move { - match async { - if let Some(format) = io_format { - return match max_by_lines(&mut output, None).await { - MaxByLines::Done(buffer) => { - Ok::( - match format.from_slice(buffer.as_bytes()) { - Ok(a) => a, - Err(e) => { - tracing::trace!( - "Failed to deserialize stdout from {}: {}, falling back to UTF-8 string.", - format, - e - ); - Value::String(buffer) - } - }, - ) - }, - MaxByLines::Error(e) => Err(e), - MaxByLines::Overflow(buffer) => Ok(Value::String(buffer)) - } - } - - let lines = buf_reader_to_lines(&mut output, 1000).await?; - if lines.is_empty() { - return Ok(Value::Null); - } - - let joined_output = lines.join("\n"); - Ok(Value::String(joined_output)) - }.await { - Ok(a) => Ok((a, output)), - Err(e) => Err((e, output)) - } - })); - let err_output = BufReader::new( - handle - .stderr - .take() - .ok_or_else(|| eyre!("Can't takeout std err")) - .with_kind(crate::ErrorKind::Docker)?, - ); - - let err_output = NonDetachingJoinHandle::from(tokio::spawn(async move { - let lines = buf_reader_to_lines(err_output, 1000).await?; - let joined_output = lines.join("\n"); - Ok::<_, Error>(joined_output) - })); - - let res = tokio::select! { - res = handle.wait() => Race::Done(res.with_kind(crate::ErrorKind::Docker)?), - res = timeout_fut => { - res?; - Race::TimedOut - }, - }; - let exit_status = match res { - Race::Done(x) => x, - Race::TimedOut => { - if let Some(id) = id { - signal::kill(Pid::from_raw(id as i32), signal::SIGKILL) - .with_kind(crate::ErrorKind::Docker)?; - } - return Ok(Err((143, "Timed out. Retrying soon...".to_owned()))); - } - }; - Ok( - if exit_status.success() || exit_status.code() == Some(143) { - Ok(serde_json::from_value( - output - .await - .with_kind(crate::ErrorKind::Unknown)? - .map(|(v, _)| v) - .map_err(|(e, _)| tracing::warn!("{}", e)) - .unwrap_or_default(), - ) - .with_kind(crate::ErrorKind::Deserialization)?) - } else { - Err(( - exit_status.code().unwrap_or_default(), - err_output.await.with_kind(crate::ErrorKind::Unknown)??, - )) - }, - ) - } - - #[instrument(skip_all)] - pub async fn sandboxed( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - input: Option, - timeout: Option, - ) -> Result, Error> { - let mut cmd = tokio::process::Command::new(CONTAINER_TOOL); - cmd.arg("run").arg("--rm").arg("--network=none"); - cmd.args( - self.docker_args(ctx, pkg_id, pkg_version, &volumes.to_readonly()) - .await?, - ); - let input_buf = if let (Some(input), Some(format)) = (&input, &self.io_format) { - cmd.stdin(std::process::Stdio::piped()); - Some(format.to_vec(input)?) - } else { - None - }; - cmd.stdout(std::process::Stdio::piped()); - cmd.stderr(std::process::Stdio::piped()); - let mut handle = cmd.spawn().with_kind(crate::ErrorKind::Docker)?; - if let (Some(input), Some(stdin)) = (&input_buf, &mut handle.stdin) { - use tokio::io::AsyncWriteExt; - stdin - .write_all(input) - .await - .with_kind(crate::ErrorKind::Docker)?; - } - - let err_output = BufReader::new( - handle - .stderr - .take() - .ok_or_else(|| eyre!("Can't takeout std err")) - .with_kind(crate::ErrorKind::Docker)?, - ); - let err_output = NonDetachingJoinHandle::from(tokio::spawn(async move { - let lines = buf_reader_to_lines(err_output, 1000).await?; - let joined_output = lines.join("\n"); - Ok::<_, Error>(joined_output) - })); - - let io_format = self.io_format; - let mut output = BufReader::new( - handle - .stdout - .take() - .ok_or_else(|| eyre!("Can't takeout stdout in sandboxed")) - .with_kind(crate::ErrorKind::Docker)?, - ); - let output = NonDetachingJoinHandle::from(tokio::spawn(async move { - match async { - if let Some(format) = io_format { - return match max_by_lines(&mut output, None).await { - MaxByLines::Done(buffer) => { - Ok::( - match format.from_slice(buffer.as_bytes()) { - Ok(a) => a, - Err(e) => { - tracing::trace!( - "Failed to deserialize stdout from {}: {}, falling back to UTF-8 string.", - format, - e - ); - Value::String(buffer) - } - }, - ) - }, - MaxByLines::Error(e) => Err(e), - MaxByLines::Overflow(buffer) => Ok(Value::String(buffer)) - } - } - - let lines = buf_reader_to_lines(&mut output, 1000).await?; - if lines.is_empty() { - return Ok(Value::Null); - } - - let joined_output = lines.join("\n"); - Ok(Value::String(joined_output)) - }.await { - Ok(a) => Ok((a, output)), - Err(e) => Err((e, output)) - } - })); - - let handle = if let Some(dur) = timeout { - async move { - tokio::time::timeout(dur, handle.wait()) - .await - .with_kind(crate::ErrorKind::Docker)? - .with_kind(crate::ErrorKind::Docker) - } - .boxed() - } else { - async { handle.wait().await.with_kind(crate::ErrorKind::Docker) }.boxed() - }; - let exit_status = handle.await?; - Ok( - if exit_status.success() || exit_status.code() == Some(143) { - Ok(serde_json::from_value( - output - .await - .with_kind(crate::ErrorKind::Unknown)? - .map(|(v, _)| v) - .map_err(|(e, _)| tracing::warn!("{}", e)) - .unwrap_or_default(), - ) - .with_kind(crate::ErrorKind::Deserialization)?) - } else { - Err(( - exit_status.code().unwrap_or_default(), - err_output.await.with_kind(crate::ErrorKind::Unknown)??, - )) - }, - ) - } - - pub fn container_name(pkg_id: &PackageId, name: Option<&str>) -> String { - if let Some(name) = name { - format!("{}_{}.{}", pkg_id, name, NET_TLD) - } else { - format!("{}.{}", pkg_id, NET_TLD) - } - } - - pub fn uncontainer_name(name: &str) -> Option<(PackageId, Option<&str>)> { - let (pre_tld, _) = name.split_once('.')?; - if pre_tld.contains('_') { - let (pkg, name) = name.split_once('_')?; - Some((Id::try_from(pkg).ok()?.into(), Some(name))) - } else { - Some((Id::try_from(pre_tld).ok()?.into(), None)) - } - } - - async fn docker_args( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - ) -> Result>, Error> { - let mut res = self.new_docker_args(); - for (volume_id, dst) in &self.mounts { - let volume = if let Some(v) = volumes.get(volume_id) { - v - } else { - continue; - }; - let src = volume.path_for(&ctx.datadir, pkg_id, pkg_version, volume_id); - if let Err(_e) = tokio::fs::metadata(&src).await { - tokio::fs::create_dir_all(&src).await?; - } - res.push(OsStr::new("--mount").into()); - res.push( - OsString::from(format!( - "type=bind,src={},dst={}{}", - src.display(), - dst.display(), - if volume.readonly() { ",readonly" } else { "" } - )) - .into(), - ); - } - if let Some(shm_size_mb) = self.shm_size_mb { - res.push(OsStr::new("--shm-size").into()); - res.push(OsString::from(format!("{}m", shm_size_mb)).into()); - } - if self.gpu_acceleration { - fn get_devices<'a>( - path: &'a Path, - res: &'a mut Vec, - ) -> BoxFuture<'a, Result<(), Error>> { - async move { - let mut read_dir = tokio::fs::read_dir(path).await?; - while let Some(entry) = read_dir.next_entry().await? { - let fty = entry.metadata().await?.file_type(); - if fty.is_block_device() || fty.is_char_device() { - res.push(entry.path()); - } else if fty.is_dir() { - get_devices(&entry.path(), res).await?; - } - } - Ok(()) - } - .boxed() - } - let mut devices = Vec::new(); - get_devices(Path::new("/dev/dri"), &mut devices).await?; - for device in devices { - res.push(OsStr::new("--device").into()); - res.push(OsString::from(device).into()); - } - } - res.push(OsStr::new("--interactive").into()); - res.push(OsStr::new("--log-driver=journald").into()); - res.push(OsStr::new("--entrypoint").into()); - res.push(OsStr::new(&self.entrypoint).into()); - if self.system { - res.push(OsString::from(self.image.for_package(&SYSTEM_PACKAGE_ID, None)).into()); - } else { - res.push(OsString::from(self.image.for_package(pkg_id, Some(pkg_version))).into()); - } - - res.extend(self.args.iter().map(|s| OsStr::new(s).into())); - - Ok(res) - } - - fn new_docker_args(&self) -> Vec> { - Vec::with_capacity( - (2 * self.mounts.len()) // --mount - + (2 * self.shm_size_mb.is_some() as usize) // --shm-size - + 5 // --interactive --log-driver=journald --entrypoint - + self.args.len(), // [ARG...] - ) - } - fn docker_args_inject(&self, pkg_id: &PackageId) -> Vec> { - let mut res = self.new_docker_args(); - if let Some(shm_size_mb) = self.shm_size_mb { - res.push(OsStr::new("--shm-size").into()); - res.push(OsString::from(format!("{}m", shm_size_mb)).into()); - } - res.push(OsStr::new("--interactive").into()); - - res.push(OsString::from(Self::container_name(pkg_id, None)).into()); - res.push(OsStr::new(&self.entrypoint).into()); - - res.extend(self.args.iter().map(|s| OsStr::new(s).into())); - - res - } -} - -struct RingVec { - value: VecDeque, - capacity: usize, -} -impl RingVec { - fn new(capacity: usize) -> Self { - RingVec { - value: VecDeque::with_capacity(capacity), - capacity, - } - } - fn push(&mut self, item: T) -> Option { - let popped_item = if self.value.len() == self.capacity { - self.value.pop_front() - } else { - None - }; - self.value.push_back(item); - popped_item - } -} - -/// This is created when we wanted a long running docker executor that we could send commands to and get the responses back. -/// We wanted a long running since we want to be able to have the equivelent to the docker execute without the heavy costs of 400 + ms time lag. -/// Also the long running let's us have the ability to start/ end the services quicker. -pub struct LongRunning { - pub running_output: NonDetachingJoinHandle<()>, -} - -impl LongRunning { - async fn setup_long_running_docker_cmd( - docker: &DockerContainer, - ctx: &RpcContext, - container_name: &str, - volumes: &Volumes, - pkg_id: &PackageId, - pkg_version: &Version, - socket_path: &Path, - ) -> Result { - const INIT_EXEC: &str = "/start9/bin/container-init"; - const BIND_LOCATION: &str = "/usr/lib/startos/container/"; - tracing::trace!("setup_long_running_docker_cmd"); - - remove_container(container_name, true).await?; - - let image_architecture = { - let mut cmd = tokio::process::Command::new(CONTAINER_TOOL); - cmd.arg("image") - .arg("inspect") - .arg("--format") - .arg("'{{.Architecture}}'"); - - if docker.system { - cmd.arg(docker.image.for_package(&SYSTEM_PACKAGE_ID, None)); - } else { - cmd.arg(docker.image.for_package(pkg_id, Some(pkg_version))); - } - let arch = String::from_utf8(cmd.output().await?.stdout)?; - arch.replace('\'', "").trim().to_string() - }; - - let mut cmd = tokio::process::Command::new(CONTAINER_TOOL); - cmd.arg("run") - .arg("--network=start9") - .arg(format!("--add-host=embassy:{}", Ipv4Addr::from(HOST_IP))) - .arg("--mount") - .arg(format!( - "type=bind,src={BIND_LOCATION},dst=/start9/bin/,readonly" - )) - .arg("--mount") - .arg(format!( - "type=bind,src={input},dst=/start9/sockets/", - input = socket_path.display() - )) - .arg("--name") - .arg(container_name) - .arg(format!("--hostname={}", &container_name)) - .arg("--entrypoint") - .arg(format!("{INIT_EXEC}.{image_architecture}")) - .arg("-i") - .arg("--rm") - .kill_on_drop(true); - - for (volume_id, dst) in &docker.mounts { - let volume = if let Some(v) = volumes.get(volume_id) { - v - } else { - continue; - }; - let src = volume.path_for(&ctx.datadir, pkg_id, pkg_version, volume_id); - if let Err(_e) = tokio::fs::metadata(&src).await { - tokio::fs::create_dir_all(&src).await?; - } - cmd.arg("--mount").arg(format!( - "type=bind,src={},dst={}{}", - src.display(), - dst.display(), - if volume.readonly() { ",readonly" } else { "" } - )); - } - if let Some(shm_size_mb) = docker.shm_size_mb { - cmd.arg("--shm-size").arg(format!("{}m", shm_size_mb)); - } - cmd.arg("--log-driver=journald"); - if docker.system { - cmd.arg(docker.image.for_package(&SYSTEM_PACKAGE_ID, None)); - } else { - cmd.arg(docker.image.for_package(pkg_id, Some(pkg_version))); - } - cmd.stdout(std::process::Stdio::piped()); - cmd.stderr(std::process::Stdio::inherit()); - cmd.stdin(std::process::Stdio::piped()); - Ok(cmd) - } -} -async fn buf_reader_to_lines( - reader: impl AsyncBufRead + Unpin, - limit: impl Into>, -) -> Result, Error> { - let mut lines = reader.lines(); - let mut answer = RingVec::new(limit.into().unwrap_or(1000)); - while let Some(line) = lines.next_line().await? { - answer.push(line); - } - let output: Vec = answer.value.into_iter().collect(); - Ok(output) -} - -enum MaxByLines { - Done(String), - Overflow(String), - Error(Error), -} - -async fn max_by_lines( - reader: impl AsyncBufRead + Unpin, - max_items: impl Into>, -) -> MaxByLines { - let mut answer = String::new(); - - let mut lines = reader.lines(); - let mut has_over_blown = false; - let max_items = max_items.into().unwrap_or(10_000_000); - - while let Some(line) = { - match lines.next_line().await { - Ok(a) => a, - Err(e) => return MaxByLines::Error(e.into()), - } - } { - if has_over_blown { - continue; - } - if !answer.is_empty() { - answer.push('\n'); - } - answer.push_str(&line); - if answer.len() >= max_items { - has_over_blown = true; - tracing::warn!("Reading the buffer exceeding limits of {}", max_items); - } - } - if has_over_blown { - return MaxByLines::Overflow(answer); - } - MaxByLines::Done(answer) -} - -#[cfg(test)] -mod tests { - use super::*; - /// Note, this size doesn't mean the vec will match. The vec will go to the next size, 0 -> 7 = 7 and so forth 7-15 = 15 - /// Just how the vec with capacity works. - const CAPACITY_IN: usize = 7; - #[test] - fn default_capacity_is_set() { - let ring: RingVec = RingVec::new(CAPACITY_IN); - assert_eq!(CAPACITY_IN, ring.value.capacity()); - assert_eq!(0, ring.value.len()); - } - #[test] - fn capacity_can_not_be_exceeded() { - let mut ring = RingVec::new(CAPACITY_IN); - for i in 1..100usize { - ring.push(i); - } - assert_eq!(CAPACITY_IN, ring.value.capacity()); - assert_eq!(CAPACITY_IN, ring.value.len()); - } - - #[test] - fn tests_buf_reader_to_lines() { - let mut reader = BufReader::new("hello\nworld\n".as_bytes()); - let lines = futures::executor::block_on(buf_reader_to_lines(&mut reader, None)).unwrap(); - assert_eq!(lines, vec!["hello", "world"]); - } -} diff --git a/core/startos/src/procedure/js_scripts.rs b/core/startos/src/procedure/js_scripts.rs deleted file mode 100644 index 43553cee..00000000 --- a/core/startos/src/procedure/js_scripts.rs +++ /dev/null @@ -1,806 +0,0 @@ -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use std::time::Duration; - -use container_init::ProcessGroupId; -use helpers::UnixRpcClient; -pub use js_engine::JsError; -use js_engine::{JsExecutionEnvironment, PathForVolumeId}; -use models::VolumeId; -use serde::de::DeserializeOwned; -use serde::{Deserialize, Serialize}; -use tokio::process::Command; -use tracing::instrument; - -use super::ProcedureName; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::IoFormat; -use crate::util::{Invoke, Version}; -use crate::volume::Volumes; - -#[derive(Debug, Serialize, Deserialize, Clone)] -#[serde(rename_all = "kebab-case")] - -enum ErrorValue { - Error(String), - ErrorCode((i32, String)), - Result(serde_json::Value), -} - -impl PathForVolumeId for Volumes { - fn path_for( - &self, - data_dir: &Path, - package_id: &PackageId, - version: &Version, - volume_id: &VolumeId, - ) -> Option { - let volume = self.get(volume_id)?; - Some(volume.path_for(data_dir, package_id, version, volume_id)) - } - - fn readonly(&self, volume_id: &VolumeId) -> bool { - self.get(volume_id).map(|x| x.readonly()).unwrap_or(false) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -pub struct ExecuteArgs { - pub procedure: JsProcedure, - pub directory: PathBuf, - pub pkg_id: PackageId, - pub pkg_version: Version, - pub name: ProcedureName, - pub volumes: Volumes, - pub input: Option, -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct JsProcedure { - #[serde(default)] - args: Vec, -} - -impl JsProcedure { - pub fn validate(&self, _volumes: &Volumes) -> Result<(), color_eyre::eyre::Report> { - Ok(()) - } - - #[instrument(skip_all)] - pub async fn execute( - &self, - directory: &PathBuf, - pkg_id: &PackageId, - pkg_version: &Version, - name: ProcedureName, - volumes: &Volumes, - input: Option, - timeout: Option, - _gid: ProcessGroupId, - _rpc_client: Option>, - ) -> Result, Error> { - #[cfg(not(test))] - let mut cmd = Command::new("start-deno"); - #[cfg(test)] - let mut cmd = test_start_deno_command().await?; - - cmd.arg("execute") - .input(Some(&mut std::io::Cursor::new(IoFormat::Json.to_vec( - &ExecuteArgs { - procedure: self.clone(), - directory: directory.clone(), - pkg_id: pkg_id.clone(), - pkg_version: pkg_version.clone(), - name, - volumes: volumes.clone(), - input: input.and_then(|x| serde_json::to_value(x).ok()), - }, - )?))) - .timeout(timeout) - .invoke(ErrorKind::Javascript) - .await - .and_then(|res| IoFormat::Json.from_slice(&res)) - } - - #[instrument(skip_all)] - pub async fn sandboxed( - &self, - directory: &PathBuf, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - input: Option, - timeout: Option, - name: ProcedureName, - ) -> Result, Error> { - #[cfg(not(test))] - let mut cmd = Command::new("start-deno"); - #[cfg(test)] - let mut cmd = test_start_deno_command().await?; - - cmd.arg("sandbox") - .input(Some(&mut std::io::Cursor::new(IoFormat::Json.to_vec( - &ExecuteArgs { - procedure: self.clone(), - directory: directory.clone(), - pkg_id: pkg_id.clone(), - pkg_version: pkg_version.clone(), - name, - volumes: volumes.clone(), - input: input.and_then(|x| serde_json::to_value(x).ok()), - }, - )?))) - .timeout(timeout) - .invoke(ErrorKind::Javascript) - .await - .and_then(|res| IoFormat::Json.from_slice(&res)) - } - - #[instrument(skip_all)] - pub async fn execute_impl( - &self, - directory: &PathBuf, - pkg_id: &PackageId, - pkg_version: &Version, - name: ProcedureName, - volumes: &Volumes, - input: Option, - ) -> Result, Error> { - let res = async move { - let running_action = JsExecutionEnvironment::load_from_package( - directory, - pkg_id, - pkg_version, - Box::new(volumes.clone()), - ) - .await? - .run_action(name, input, self.args.clone()); - let output: Option = running_action.await?; - let output: O = unwrap_known_error(output)?; - Ok(output) - } - .await - .map_err(|(error, message)| (error.as_code_num(), message)); - - Ok(res) - } - - #[instrument(skip_all)] - pub async fn sandboxed_impl( - &self, - directory: &PathBuf, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - input: Option, - name: ProcedureName, - ) -> Result, Error> { - Ok(async move { - let running_action = JsExecutionEnvironment::load_from_package( - directory, - pkg_id, - pkg_version, - Box::new(volumes.clone()), - ) - .await? - .read_only_effects() - .run_action(name, input, self.args.clone()); - let output: Option = running_action.await?; - let output: O = unwrap_known_error(output)?; - Ok(output) - } - .await - .map_err(|(error, message)| (error.as_code_num(), message))) - } -} - -fn unwrap_known_error( - error_value: Option, -) -> Result { - let error_value = error_value.unwrap_or_else(|| ErrorValue::Result(serde_json::Value::Null)); - match error_value { - ErrorValue::Error(error) => Err((JsError::Javascript, error)), - ErrorValue::ErrorCode((code, message)) => Err((JsError::Code(code), message)), - ErrorValue::Result(ref value) => match serde_json::from_value(value.clone()) { - Ok(a) => Ok(a), - Err(err) => { - tracing::error!("{}", err); - tracing::debug!("{:?}", err); - Err(( - JsError::BoundryLayerSerDe, - format!( - "Couldn't convert output = {:#?} to the correct type", - serde_json::to_string_pretty(&error_value).unwrap_or_default() - ), - )) - } - }, - } -} - -#[cfg(test)] -async fn test_start_deno_command() -> Result { - Command::new("cargo") - .arg("build") - .invoke(ErrorKind::Unknown) - .await?; - if tokio::fs::metadata("../target/debug/start-deno") - .await - .is_err() - { - Command::new("ln") - .arg("-rsf") - .arg("../target/debug/startbox") - .arg("../target/debug/start-deno") - .invoke(crate::ErrorKind::Filesystem) - .await?; - } - Ok(Command::new("../target/debug/start-deno")) -} - -#[tokio::test] -async fn js_action_execute() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::GetConfig; - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = Some(serde_json::json!({"test":123})); - let timeout = Some(Duration::from_secs(10)); - let _output: crate::config::action::ConfigRes = js_action - .execute( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); - assert_eq!( - &std::fs::read_to_string( - "test/js_action_execute/package-data/volumes/test-package/data/main/test.log" - ) - .unwrap(), - "This is a test" - ); - std::fs::remove_file( - "test/js_action_execute/package-data/volumes/test-package/data/main/test.log", - ) - .unwrap(); -} - -#[tokio::test] -async fn js_action_execute_error() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::SetConfig; - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - let output: Result = js_action - .execute( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap(); - assert_eq!("Err((2, \"Not setup\"))", &format!("{:?}", output)); -} - -#[tokio::test] -async fn js_action_fetch() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("fetch".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} - -#[tokio::test] -async fn js_test_slow() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("slow".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - tracing::debug!("testing start"); - tokio::select! { - a = js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) => { a.unwrap().unwrap(); }, - _ = tokio::time::sleep(Duration::from_secs(1)) => () - } - tracing::debug!("testing end should"); - tokio::time::sleep(Duration::from_secs(2)).await; - tracing::debug!("Done"); -} -#[tokio::test] -async fn js_action_var_arg() { - let js_action = JsProcedure { - args: vec![42.into()], - }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("js-action-var-arg".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} - -#[tokio::test] -async fn js_action_test_rename() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-rename".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} - -#[tokio::test] -async fn js_action_test_deep_dir() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-deep-dir".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} -#[tokio::test] -async fn js_action_test_deep_dir_escape() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-deep-dir-escape".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} -#[tokio::test] -async fn js_action_test_zero_dir() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-zero-dir".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} -#[tokio::test] -async fn js_action_test_read_dir() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-read-dir".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} - -#[tokio::test] -async fn js_rsync() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-rsync".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} - -#[tokio::test] -async fn js_disk_usage() { - let js_action = JsProcedure { args: vec![] }; - let path: PathBuf = "test/js_action_execute/" - .parse::() - .unwrap() - .canonicalize() - .unwrap(); - let package_id = "test-package".parse().unwrap(); - let package_version: Version = "0.3.0.3".parse().unwrap(); - let name = ProcedureName::Action("test-disk-usage".parse().unwrap()); - let volumes: Volumes = serde_json::from_value(serde_json::json!({ - "main": { - "type": "data" - }, - "compat": { - "type": "assets" - }, - "filebrowser" :{ - "package-id": "filebrowser", - "path": "data", - "readonly": true, - "type": "pointer", - "volume-id": "main", - } - })) - .unwrap(); - let input: Option = None; - let timeout = Some(Duration::from_secs(10)); - js_action - .execute::( - &path, - &package_id, - &package_version, - name, - &volumes, - input, - timeout, - ProcessGroupId(0), - None, - ) - .await - .unwrap() - .unwrap(); -} diff --git a/core/startos/src/procedure/mod.rs b/core/startos/src/procedure/mod.rs deleted file mode 100644 index f3a52871..00000000 --- a/core/startos/src/procedure/mod.rs +++ /dev/null @@ -1,185 +0,0 @@ -use std::collections::BTreeSet; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use models::ImageId; -use patch_db::HasModel; -use serde::de::DeserializeOwned; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use self::docker::DockerProcedure; -use crate::context::RpcContext; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::Version; -use crate::volume::Volumes; -use crate::{Error, ErrorKind}; - -pub mod docker; -#[cfg(feature = "js-engine")] -pub mod js_scripts; -pub use models::ProcedureName; - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[serde(tag = "type")] -#[model = "Model"] -pub enum PackageProcedure { - Docker(DockerProcedure), - - #[cfg(feature = "js-engine")] - Script(js_scripts::JsProcedure), -} - -impl PackageProcedure { - pub fn is_script(&self) -> bool { - match self { - #[cfg(feature = "js-engine")] - Self::Script(_) => true, - _ => false, - } - } - #[instrument(skip_all)] - pub fn validate( - &self, - eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - expected_io: bool, - ) -> Result<(), color_eyre::eyre::Report> { - match self { - PackageProcedure::Docker(action) => { - action.validate(eos_version, volumes, image_ids, expected_io) - } - #[cfg(feature = "js-engine")] - PackageProcedure::Script(action) => action.validate(volumes), - } - } - - #[instrument(skip_all)] - pub async fn execute( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - name: ProcedureName, - volumes: &Volumes, - input: Option, - timeout: Option, - ) -> Result, Error> { - tracing::trace!("Procedure execute {} {} - {:?}", self, pkg_id, name); - match self { - PackageProcedure::Docker(procedure) if procedure.inject == true => { - procedure - .inject(ctx, pkg_id, pkg_version, name, volumes, input, timeout) - .await - } - PackageProcedure::Docker(procedure) => { - procedure - .execute(ctx, pkg_id, pkg_version, name, volumes, input, timeout) - .await - } - #[cfg(feature = "js-engine")] - PackageProcedure::Script(procedure) => { - let man = ctx - .managers - .get(&(pkg_id.clone(), pkg_version.clone())) - .await - .ok_or_else(|| { - Error::new( - eyre!("No manager found for {}", pkg_id), - ErrorKind::NotFound, - ) - })?; - let rpc_client = man.rpc_client(); - let gid = if matches!(name, ProcedureName::Main) { - man.gid.new_main_gid() - } else { - man.gid.new_gid() - }; - - procedure - .execute( - &ctx.datadir, - pkg_id, - pkg_version, - name, - volumes, - input, - timeout, - gid, - rpc_client, - ) - .await - } - } - } - - #[instrument(skip_all)] - pub async fn sandboxed( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - input: Option, - timeout: Option, - name: ProcedureName, - ) -> Result, Error> { - tracing::trace!("Procedure sandboxed {} {} - {:?}", self, pkg_id, name); - match self { - PackageProcedure::Docker(procedure) => { - procedure - .sandboxed(ctx, pkg_id, pkg_version, volumes, input, timeout) - .await - } - #[cfg(feature = "js-engine")] - PackageProcedure::Script(procedure) => { - procedure - .sandboxed( - &ctx.datadir, - pkg_id, - pkg_version, - volumes, - input, - timeout, - name, - ) - .await - } - } - } -} - -impl std::fmt::Display for PackageProcedure { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - PackageProcedure::Docker(_) => write!(f, "Docker")?, - #[cfg(feature = "js-engine")] - PackageProcedure::Script(_) => write!(f, "JS")?, - } - Ok(()) - } -} - -// TODO: make this not allocate -#[derive(Debug)] -pub struct NoOutput; -impl<'de> Deserialize<'de> for NoOutput { - fn deserialize(deserializer: D) -> Result - where - D: serde::Deserializer<'de>, - { - let _ = Value::deserialize(deserializer); - Ok(NoOutput) - } -} - -#[test] -fn test_deser_no_output() { - serde_json::from_str::("").unwrap(); - serde_json::from_str::>("{\"Ok\": null}") - .unwrap() - .unwrap(); -} diff --git a/core/startos/src/properties.rs b/core/startos/src/properties.rs deleted file mode 100644 index 851033b7..00000000 --- a/core/startos/src/properties.rs +++ /dev/null @@ -1,50 +0,0 @@ -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use rpc_toolkit::command; -use serde_json::Value; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::prelude::*; -use crate::procedure::ProcedureName; -use crate::s9pk::manifest::PackageId; -use crate::{Error, ErrorKind}; - -pub fn display_properties(response: Value, _: &ArgMatches) { - println!("{}", response); -} - -#[command(display(display_properties))] -pub async fn properties(#[context] ctx: RpcContext, #[arg] id: PackageId) -> Result { - Ok(fetch_properties(ctx, id).await?) -} - -#[instrument(skip_all)] -pub async fn fetch_properties(ctx: RpcContext, id: PackageId) -> Result { - let peek = ctx.db.peek().await; - - let manifest = peek - .as_package_data() - .as_idx(&id) - .ok_or_else(|| Error::new(eyre!("{} is not installed", id), ErrorKind::NotFound))? - .expect_as_installed()? - .as_manifest() - .de()?; - if let Some(props) = manifest.properties { - props - .execute::<(), Value>( - &ctx, - &manifest.id, - &manifest.version, - ProcedureName::Properties, - &manifest.volumes, - None, - None, - ) - .await? - .map_err(|(_, e)| Error::new(eyre!("{}", e), ErrorKind::Docker)) - .and_then(|a| Ok(a)) - } else { - Ok(Value::Null) - } -} diff --git a/core/startos/src/registry/admin.rs b/core/startos/src/registry/admin.rs deleted file mode 100644 index e994b5c5..00000000 --- a/core/startos/src/registry/admin.rs +++ /dev/null @@ -1,214 +0,0 @@ -use std::path::PathBuf; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use console::style; -use futures::StreamExt; -use indicatif::{ProgressBar, ProgressStyle}; -use reqwest::{header, Body, Client, Url}; -use rpc_toolkit::command; - -use crate::s9pk::reader::S9pkReader; -use crate::util::display_none; -use crate::{Error, ErrorKind}; - -async fn registry_user_pass(location: &str) -> Result<(Url, String, String), Error> { - let mut url = Url::parse(location)?; - let user = url.username().to_string(); - let pass = url.password().map(str::to_string); - if user.is_empty() || url.path() != "/" { - return Err(Error::new( - eyre!("{location:?} is not like \"https://user@registry.example.com/\""), - ErrorKind::ParseUrl, - )); - } - let _ = url.set_username(""); - let _ = url.set_password(None); - - let pass = match pass { - Some(p) => p, - None => { - let pass_prompt = format!("{} Password for {user}: ", style("?").yellow()); - tokio::task::spawn_blocking(move || rpassword::prompt_password(pass_prompt)) - .await - .unwrap()? - } - }; - Ok((url, user.to_string(), pass.to_string())) -} - -#[derive(serde::Serialize, Debug)] -struct Package { - id: String, - version: String, - arches: Option>, -} - -async fn do_index( - httpc: &Client, - mut url: Url, - user: &str, - pass: &str, - pkg: &Package, -) -> Result<(), Error> { - url.set_path("/admin/v0/index"); - let req = httpc - .post(url) - .header(header::ACCEPT, "text/plain") - .basic_auth(user, Some(pass)) - .json(pkg) - .build()?; - let res = httpc.execute(req).await?; - if !res.status().is_success() { - let info = res.text().await?; - return Err(Error::new(eyre!("{}", info), ErrorKind::Registry)); - } - Ok(()) -} - -async fn do_upload( - httpc: &Client, - mut url: Url, - user: &str, - pass: &str, - pkg_id: &str, - body: Body, -) -> Result<(), Error> { - url.set_path("/admin/v0/upload"); - let req = httpc - .post(url) - .header(header::ACCEPT, "text/plain") - .query(&[("id", pkg_id)]) - .basic_auth(user, Some(pass)) - .body(body) - .build()?; - let res = httpc.execute(req).await?; - if !res.status().is_success() { - let info = res.text().await?; - return Err(Error::new(eyre!("{}", info), ErrorKind::Registry)); - } - Ok(()) -} - -#[command(cli_only, display(display_none))] -pub async fn publish( - #[arg] location: String, - #[arg] path: PathBuf, - #[arg(rename = "no-verify", long = "no-verify")] no_verify: bool, - #[arg(rename = "no-upload", long = "no-upload")] no_upload: bool, - #[arg(rename = "no-index", long = "no-index")] no_index: bool, -) -> Result<(), Error> { - // Prepare for progress bars. - let bytes_bar_style = - ProgressStyle::with_template("{percent}% {wide_bar} [{bytes}/{total_bytes}] [{eta}]") - .unwrap(); - let plain_line_style = - ProgressStyle::with_template("{prefix:.bold.dim} {wide_msg}...").unwrap(); - let spinner_line_style = - ProgressStyle::with_template("{prefix:.bold.dim} {spinner} {wide_msg}...").unwrap(); - - // Read the file to get manifest information and check validity.. - // Open file right away so it can not change out from under us. - let file = tokio::fs::File::open(&path).await?; - - let manifest = if no_verify { - let pb = ProgressBar::new(1) - .with_style(spinner_line_style.clone()) - .with_prefix("[1/3]") - .with_message("Querying s9pk"); - pb.enable_steady_tick(Duration::from_millis(200)); - let mut s9pk = S9pkReader::open(&path, false).await?; - let m = s9pk.manifest().await?.clone(); - pb.set_style(plain_line_style.clone()); - pb.abandon(); - m - } else { - let pb = ProgressBar::new(1) - .with_style(spinner_line_style.clone()) - .with_prefix("[1/3]") - .with_message("Verifying s9pk"); - pb.enable_steady_tick(Duration::from_millis(200)); - let mut s9pk = S9pkReader::open(&path, true).await?; - s9pk.validate().await?; - let m = s9pk.manifest().await?.clone(); - pb.set_style(plain_line_style.clone()); - pb.abandon(); - m - }; - let pkg = Package { - id: manifest.id.to_string(), - version: manifest.version.to_string(), - arches: manifest.hardware_requirements.arch.clone(), - }; - println!("{} id = {}", style(">").green(), pkg.id); - println!("{} version = {}", style(">").green(), pkg.version); - if let Some(arches) = &pkg.arches { - println!("{} arches = {:?}", style(">").green(), arches); - } else { - println!( - "{} No architecture listed in hardware_requirements", - style(">").red() - ); - } - - // Process the url and get the user's password. - let (registry, user, pass) = registry_user_pass(&location).await?; - - // Now prepare a stream of the file which will show a progress bar as it is consumed. - let file_size = file.metadata().await?.len(); - let file_stream = tokio_util::io::ReaderStream::new(file); - ProgressBar::new(0) - .with_style(plain_line_style.clone()) - .with_prefix("[2/3]") - .with_message("Uploading s9pk") - .abandon(); - let pb = ProgressBar::new(file_size).with_style(bytes_bar_style.clone()); - let stream_pb = pb.clone(); - let file_stream = file_stream.inspect(move |bytes| { - if let Ok(bytes) = bytes { - stream_pb.inc(bytes.len() as u64); - } - }); - - let httpc = Client::builder().build().unwrap(); - // And upload! - if no_upload { - println!("{} Skipping upload", style(">").yellow()); - } else { - do_upload( - &httpc, - registry.clone(), - &user, - &pass, - &pkg.id, - Body::wrap_stream(file_stream), - ) - .await?; - } - pb.finish_and_clear(); - - // Also index, so it will show up in the registry. - let pb = ProgressBar::new(0) - .with_style(spinner_line_style.clone()) - .with_prefix("[3/3]") - .with_message("Indexing registry"); - pb.enable_steady_tick(Duration::from_millis(200)); - if no_index { - println!("{} Skipping index", style(">").yellow()); - } else { - do_index(&httpc, registry.clone(), &user, &pass, &pkg).await?; - } - pb.set_style(plain_line_style.clone()); - pb.abandon(); - - // All done - if !no_index { - println!( - "{} Package {} is now published to {}", - style(">").green(), - pkg.id, - registry - ); - } - Ok(()) -} diff --git a/core/startos/src/registry/marketplace.rs b/core/startos/src/registry/marketplace.rs deleted file mode 100644 index 97973319..00000000 --- a/core/startos/src/registry/marketplace.rs +++ /dev/null @@ -1,92 +0,0 @@ -use base64::Engine; -use color_eyre::eyre::eyre; -use reqwest::{StatusCode, Url}; -use rpc_toolkit::command; -use serde_json::Value; - -use crate::context::RpcContext; -use crate::version::VersionT; -use crate::{Error, ResultExt}; - -#[command(subcommands(get))] -pub fn marketplace() -> Result<(), Error> { - Ok(()) -} - -pub fn with_query_params(ctx: RpcContext, mut url: Url) -> Url { - url.query_pairs_mut() - .append_pair( - "os.version", - &crate::version::Current::new().semver().to_string(), - ) - .append_pair( - "os.compat", - &crate::version::Current::new().compat().to_string(), - ) - .append_pair("os.arch", &*crate::PLATFORM) - .append_pair("hardware.arch", &*crate::ARCH) - .append_pair("hardware.ram", &ctx.hardware.ram.to_string()); - - for hw in &ctx.hardware.devices { - url.query_pairs_mut() - .append_pair(&format!("hardware.device.{}", hw.class()), hw.product()); - } - - url -} - -#[command] -pub async fn get(#[context] ctx: RpcContext, #[arg] url: Url) -> Result { - let mut response = ctx - .client - .get(with_query_params(ctx.clone(), url)) - .send() - .await - .with_kind(crate::ErrorKind::Network)?; - let status = response.status(); - if status.is_success() { - match response - .headers_mut() - .remove("Content-Type") - .as_ref() - .and_then(|h| h.to_str().ok()) - .and_then(|h| h.split(";").next()) - .map(|h| h.trim()) - { - Some("application/json") => response - .json() - .await - .with_kind(crate::ErrorKind::Deserialization), - Some("text/plain") => Ok(Value::String( - response - .text() - .await - .with_kind(crate::ErrorKind::Registry)?, - )), - Some(ctype) => Ok(Value::String(format!( - "data:{};base64,{}", - ctype, - base64::engine::general_purpose::URL_SAFE.encode( - &response - .bytes() - .await - .with_kind(crate::ErrorKind::Registry)? - ) - ))), - _ => Err(Error::new( - eyre!("missing Content-Type"), - crate::ErrorKind::Registry, - )), - } - } else { - let message = response.text().await.with_kind(crate::ErrorKind::Network)?; - Err(Error::new( - eyre!("{}", message), - match status { - StatusCode::BAD_REQUEST => crate::ErrorKind::InvalidRequest, - StatusCode::NOT_FOUND => crate::ErrorKind::NotFound, - _ => crate::ErrorKind::Registry, - }, - )) - } -} diff --git a/core/startos/src/registry/mod.rs b/core/startos/src/registry/mod.rs deleted file mode 100644 index 27f541f1..00000000 --- a/core/startos/src/registry/mod.rs +++ /dev/null @@ -1,2 +0,0 @@ -pub mod admin; -pub mod marketplace; diff --git a/core/startos/src/s9pk/builder.rs b/core/startos/src/s9pk/builder.rs deleted file mode 100644 index 19974243..00000000 --- a/core/startos/src/s9pk/builder.rs +++ /dev/null @@ -1,145 +0,0 @@ -use sha2::{Digest, Sha512}; -use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt, SeekFrom}; -use tracing::instrument; -use typed_builder::TypedBuilder; - -use super::header::{FileSection, Header}; -use super::manifest::Manifest; -use super::SIG_CONTEXT; -use crate::util::io::to_cbor_async_writer; -use crate::util::HashWriter; -use crate::{Error, ResultExt}; - -#[derive(TypedBuilder)] -pub struct S9pkPacker< - 'a, - W: AsyncWriteExt + AsyncSeekExt, - RLicense: AsyncReadExt + Unpin, - RInstructions: AsyncReadExt + Unpin, - RIcon: AsyncReadExt + Unpin, - RDockerImages: AsyncReadExt + Unpin, - RAssets: AsyncReadExt + Unpin, - RScripts: AsyncReadExt + Unpin, -> { - writer: W, - manifest: &'a Manifest, - license: RLicense, - instructions: RInstructions, - icon: RIcon, - docker_images: RDockerImages, - assets: RAssets, - scripts: Option, -} -impl< - 'a, - W: AsyncWriteExt + AsyncSeekExt + Unpin, - RLicense: AsyncReadExt + Unpin, - RInstructions: AsyncReadExt + Unpin, - RIcon: AsyncReadExt + Unpin, - RDockerImages: AsyncReadExt + Unpin, - RAssets: AsyncReadExt + Unpin, - RScripts: AsyncReadExt + Unpin, - > S9pkPacker<'a, W, RLicense, RInstructions, RIcon, RDockerImages, RAssets, RScripts> -{ - /// BLOCKING - #[instrument(skip_all)] - pub async fn pack(mut self, key: &ed25519_dalek::SigningKey) -> Result<(), Error> { - let header_pos = self.writer.stream_position().await?; - if header_pos != 0 { - tracing::warn!("Appending to non-empty file."); - } - let mut header = Header::placeholder(); - header.serialize(&mut self.writer).await.with_ctx(|_| { - ( - crate::ErrorKind::Serialization, - "Writing Placeholder Header", - ) - })?; - let mut position = self.writer.stream_position().await?; - - let mut writer = HashWriter::new(Sha512::new(), &mut self.writer); - // manifest - to_cbor_async_writer(&mut writer, self.manifest).await?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.manifest = FileSection { - position, - length: new_pos - position, - }; - position = new_pos; - // license - tokio::io::copy(&mut self.license, &mut writer) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "Copying License"))?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.license = FileSection { - position, - length: new_pos - position, - }; - position = new_pos; - // instructions - tokio::io::copy(&mut self.instructions, &mut writer) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "Copying Instructions"))?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.instructions = FileSection { - position, - length: new_pos - position, - }; - position = new_pos; - // icon - tokio::io::copy(&mut self.icon, &mut writer) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "Copying Icon"))?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.icon = FileSection { - position, - length: new_pos - position, - }; - position = new_pos; - // docker_images - tokio::io::copy(&mut self.docker_images, &mut writer) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "Copying Docker Images"))?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.docker_images = FileSection { - position, - length: new_pos - position, - }; - position = new_pos; - // assets - tokio::io::copy(&mut self.assets, &mut writer) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "Copying Assets"))?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.assets = FileSection { - position, - length: new_pos - position, - }; - position = new_pos; - // scripts - if let Some(mut scripts) = self.scripts { - tokio::io::copy(&mut scripts, &mut writer) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, "Copying Scripts"))?; - let new_pos = writer.inner_mut().stream_position().await?; - header.table_of_contents.scripts = Some(FileSection { - position, - length: new_pos - position, - }); - position = new_pos; - } - - // header - let (hash, _) = writer.finish(); - self.writer.seek(SeekFrom::Start(header_pos)).await?; - header.pubkey = key.into(); - header.signature = key.sign_prehashed(hash, Some(SIG_CONTEXT))?; - header - .serialize(&mut self.writer) - .await - .with_ctx(|_| (crate::ErrorKind::Serialization, "Writing Header"))?; - self.writer.seek(SeekFrom::Start(position)).await?; - - Ok(()) - } -} diff --git a/core/startos/src/s9pk/docker.rs b/core/startos/src/s9pk/docker.rs deleted file mode 100644 index be93905f..00000000 --- a/core/startos/src/s9pk/docker.rs +++ /dev/null @@ -1,95 +0,0 @@ -use std::borrow::Cow; -use std::collections::BTreeSet; -use std::io::SeekFrom; -use std::path::Path; - -use color_eyre::eyre::eyre; -use futures::{FutureExt, TryStreamExt}; -use serde::{Deserialize, Serialize}; -use tokio::io::{AsyncRead, AsyncSeek, AsyncSeekExt}; -use tokio_tar::{Archive, Entry}; - -use crate::util::io::from_cbor_async_reader; -use crate::{Error, ErrorKind, ARCH}; - -#[derive(Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct DockerMultiArch { - pub default: String, - pub available: BTreeSet, -} - -#[pin_project::pin_project(project = DockerReaderProject)] -#[derive(Debug)] -pub enum DockerReader { - SingleArch(#[pin] R), - MultiArch(#[pin] Entry>), -} -impl DockerReader { - pub async fn new(mut rdr: R) -> Result { - let arch = if let Some(multiarch) = tokio_tar::Archive::new(&mut rdr) - .entries()? - .try_filter_map(|e| { - async move { - Ok(if &*e.path()? == Path::new("multiarch.cbor") { - Some(e) - } else { - None - }) - } - .boxed() - }) - .try_next() - .await? - { - let multiarch: DockerMultiArch = from_cbor_async_reader(multiarch).await?; - Some(if multiarch.available.contains(&**ARCH) { - Cow::Borrowed(&**ARCH) - } else { - Cow::Owned(multiarch.default) - }) - } else { - None - }; - rdr.seek(SeekFrom::Start(0)).await?; - if let Some(arch) = arch { - if let Some(image) = tokio_tar::Archive::new(rdr) - .entries()? - .try_filter_map(|e| { - let arch = arch.clone(); - async move { - Ok(if &*e.path()? == Path::new(&format!("{}.tar", arch)) { - Some(e) - } else { - None - }) - } - .boxed() - }) - .try_next() - .await? - { - Ok(Self::MultiArch(image)) - } else { - Err(Error::new( - eyre!("Docker image section does not contain tarball for architecture"), - ErrorKind::ParseS9pk, - )) - } - } else { - Ok(Self::SingleArch(rdr)) - } - } -} -impl AsyncRead for DockerReader { - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut tokio::io::ReadBuf<'_>, - ) -> std::task::Poll> { - match self.project() { - DockerReaderProject::SingleArch(r) => r.poll_read(cx, buf), - DockerReaderProject::MultiArch(r) => r.poll_read(cx, buf), - } - } -} diff --git a/core/startos/src/s9pk/git_hash.rs b/core/startos/src/s9pk/git_hash.rs deleted file mode 100644 index b2990a11..00000000 --- a/core/startos/src/s9pk/git_hash.rs +++ /dev/null @@ -1,41 +0,0 @@ -use std::path::Path; - -use crate::Error; - -#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)] -pub struct GitHash(String); - -impl GitHash { - pub async fn from_path(path: impl AsRef) -> Result { - let hash = tokio::process::Command::new("git") - .args(["describe", "--always", "--abbrev=40", "--dirty=-modified"]) - .current_dir(path) - .output() - .await?; - if !hash.status.success() { - return Err(Error::new( - color_eyre::eyre::eyre!("Could not get hash: {}", String::from_utf8(hash.stderr)?), - crate::ErrorKind::Filesystem, - )); - } - Ok(GitHash(String::from_utf8(hash.stdout)?)) - } -} - -impl AsRef for GitHash { - fn as_ref(&self) -> &str { - &self.0 - } -} - -// #[tokio::test] -// async fn test_githash_for_current() { -// let answer: GitHash = GitHash::from_path(std::env::current_dir().unwrap()) -// .await -// .unwrap(); -// let answer_str: &str = answer.as_ref(); -// assert!( -// !answer_str.is_empty(), -// "Should have a hash for this current working" -// ); -// } diff --git a/core/startos/src/s9pk/header.rs b/core/startos/src/s9pk/header.rs deleted file mode 100644 index 4f77ad85..00000000 --- a/core/startos/src/s9pk/header.rs +++ /dev/null @@ -1,187 +0,0 @@ -use std::collections::BTreeMap; - -use color_eyre::eyre::eyre; -use ed25519_dalek::{Signature, VerifyingKey}; -use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt}; - -use crate::Error; - -pub const MAGIC: [u8; 2] = [59, 59]; -pub const VERSION: u8 = 1; - -#[derive(Debug)] -pub struct Header { - pub pubkey: VerifyingKey, - pub signature: Signature, - pub table_of_contents: TableOfContents, -} -impl Header { - pub fn placeholder() -> Self { - Header { - pubkey: VerifyingKey::default(), - signature: Signature::from_bytes(&[0; 64]), - table_of_contents: Default::default(), - } - } - // MUST BE SAME SIZE REGARDLESS OF DATA - pub async fn serialize(&self, mut writer: W) -> std::io::Result<()> { - writer.write_all(&MAGIC).await?; - writer.write_all(&[VERSION]).await?; - writer.write_all(self.pubkey.as_bytes()).await?; - writer.write_all(&self.signature.to_bytes()).await?; - self.table_of_contents.serialize(writer).await?; - Ok(()) - } - pub async fn deserialize(mut reader: R) -> Result { - let mut magic = [0; 2]; - reader.read_exact(&mut magic).await?; - if magic != MAGIC { - return Err(Error::new( - eyre!("Incorrect Magic: {:?}", magic), - crate::ErrorKind::ParseS9pk, - )); - } - let mut version = [0]; - reader.read_exact(&mut version).await?; - if version[0] != VERSION { - return Err(Error::new( - eyre!("Unknown Version: {}", version[0]), - crate::ErrorKind::ParseS9pk, - )); - } - let mut pubkey_bytes = [0; 32]; - reader.read_exact(&mut pubkey_bytes).await?; - let pubkey = VerifyingKey::from_bytes(&pubkey_bytes) - .map_err(|e| Error::new(e, crate::ErrorKind::ParseS9pk))?; - let mut sig_bytes = [0; 64]; - reader.read_exact(&mut sig_bytes).await?; - let signature = Signature::from_bytes(&sig_bytes); - let table_of_contents = TableOfContents::deserialize(reader).await?; - - Ok(Header { - pubkey, - signature, - table_of_contents, - }) - } -} - -#[derive(Debug, Default)] -pub struct TableOfContents { - pub manifest: FileSection, - pub license: FileSection, - pub instructions: FileSection, - pub icon: FileSection, - pub docker_images: FileSection, - pub assets: FileSection, - pub scripts: Option, -} -impl TableOfContents { - pub async fn serialize(&self, mut writer: W) -> std::io::Result<()> { - let len: u32 = ((1 + "manifest".len() + 16) - + (1 + "license".len() + 16) - + (1 + "instructions".len() + 16) - + (1 + "icon".len() + 16) - + (1 + "docker_images".len() + 16) - + (1 + "assets".len() + 16) - + (1 + "scripts".len() + 16)) as u32; - writer.write_all(&u32::to_be_bytes(len)).await?; - self.manifest - .serialize_entry("manifest", &mut writer) - .await?; - self.license.serialize_entry("license", &mut writer).await?; - self.instructions - .serialize_entry("instructions", &mut writer) - .await?; - self.icon.serialize_entry("icon", &mut writer).await?; - self.docker_images - .serialize_entry("docker_images", &mut writer) - .await?; - self.assets.serialize_entry("assets", &mut writer).await?; - self.scripts - .unwrap_or_default() - .serialize_entry("scripts", &mut writer) - .await?; - Ok(()) - } - pub async fn deserialize(mut reader: R) -> std::io::Result { - let mut toc_len = [0; 4]; - reader.read_exact(&mut toc_len).await?; - let toc_len = u32::from_be_bytes(toc_len); - let mut reader = reader.take(toc_len as u64); - let mut table = BTreeMap::new(); - while let Some((label, section)) = FileSection::deserialize_entry(&mut reader).await? { - table.insert(label, section); - } - fn from_table( - table: &BTreeMap, FileSection>, - label: &str, - ) -> std::io::Result { - table.get(label.as_bytes()).copied().ok_or_else(|| { - std::io::Error::new( - std::io::ErrorKind::UnexpectedEof, - format!("Missing Required Label: {}", label), - ) - }) - } - #[allow(dead_code)] - fn as_opt(fs: FileSection) -> Option { - if fs.position | fs.length == 0 { - // 0/0 is not a valid file section - None - } else { - Some(fs) - } - } - Ok(TableOfContents { - manifest: from_table(&table, "manifest")?, - license: from_table(&table, "license")?, - instructions: from_table(&table, "instructions")?, - icon: from_table(&table, "icon")?, - docker_images: from_table(&table, "docker_images")?, - assets: from_table(&table, "assets")?, - scripts: table.get("scripts".as_bytes()).cloned(), - }) - } -} - -#[derive(Clone, Copy, Debug, Default)] -pub struct FileSection { - pub position: u64, - pub length: u64, -} -impl FileSection { - pub async fn serialize_entry( - self, - label: &str, - mut writer: W, - ) -> std::io::Result<()> { - writer.write_all(&[label.len() as u8]).await?; - writer.write_all(label.as_bytes()).await?; - writer.write_all(&u64::to_be_bytes(self.position)).await?; - writer.write_all(&u64::to_be_bytes(self.length)).await?; - Ok(()) - } - pub async fn deserialize_entry( - mut reader: R, - ) -> std::io::Result, Self)>> { - let mut label_len = [0]; - let read = reader.read(&mut label_len).await?; - if read == 0 { - return Ok(None); - } - let mut label = vec![0; label_len[0] as usize]; - reader.read_exact(&mut label).await?; - let mut pos = [0; 8]; - reader.read_exact(&mut pos).await?; - let mut len = [0; 8]; - reader.read_exact(&mut len).await?; - Ok(Some(( - label, - FileSection { - position: u64::from_be_bytes(pos), - length: u64::from_be_bytes(len), - }, - ))) - } -} diff --git a/core/startos/src/s9pk/manifest.rs b/core/startos/src/s9pk/manifest.rs deleted file mode 100644 index 3eee540e..00000000 --- a/core/startos/src/s9pk/manifest.rs +++ /dev/null @@ -1,211 +0,0 @@ -use std::collections::BTreeMap; -use std::path::{Path, PathBuf}; - -use color_eyre::eyre::eyre; -pub use models::PackageId; -use serde::{Deserialize, Serialize}; -use url::Url; - -use super::git_hash::GitHash; -use crate::action::Actions; -use crate::backup::BackupActions; -use crate::config::action::ConfigActions; -use crate::dependencies::Dependencies; -use crate::migration::Migrations; -use crate::net::interface::Interfaces; -use crate::prelude::*; -use crate::procedure::docker::DockerContainers; -use crate::procedure::PackageProcedure; -use crate::status::health_check::HealthChecks; -use crate::util::serde::Regex; -use crate::util::Version; -use crate::version::{Current, VersionT}; -use crate::volume::Volumes; -use crate::Error; - -fn current_version() -> Version { - Current::new().semver().into() -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct Manifest { - #[serde(default = "current_version")] - pub eos_version: Version, - pub id: PackageId, - #[serde(default)] - pub git_hash: Option, - pub title: String, - pub version: Version, - pub description: Description, - #[serde(default)] - pub assets: Assets, - #[serde(default)] - pub build: Option>, - pub release_notes: String, - pub license: String, // type of license - pub wrapper_repo: Url, - pub upstream_repo: Url, - pub support_site: Option, - pub marketing_site: Option, - pub donation_url: Option, - #[serde(default)] - pub alerts: Alerts, - pub main: PackageProcedure, - pub health_checks: HealthChecks, - pub config: Option, - pub properties: Option, - pub volumes: Volumes, - // #[serde(default)] - pub interfaces: Interfaces, - // #[serde(default)] - pub backup: BackupActions, - #[serde(default)] - pub migrations: Migrations, - #[serde(default)] - pub actions: Actions, - // #[serde(default)] - // pub permissions: Permissions, - #[serde(default)] - pub dependencies: Dependencies, - pub containers: Option, - - #[serde(default)] - pub replaces: Vec, - - #[serde(default)] - pub hardware_requirements: HardwareRequirements, -} - -impl Manifest { - pub fn package_procedures(&self) -> impl Iterator { - use std::iter::once; - let main = once(&self.main); - let cfg_get = self.config.as_ref().map(|a| &a.get).into_iter(); - let cfg_set = self.config.as_ref().map(|a| &a.set).into_iter(); - let props = self.properties.iter(); - let backups = vec![&self.backup.create, &self.backup.restore].into_iter(); - let migrations = self - .migrations - .to - .values() - .chain(self.migrations.from.values()); - let actions = self.actions.0.values().map(|a| &a.implementation); - main.chain(cfg_get) - .chain(cfg_set) - .chain(props) - .chain(backups) - .chain(migrations) - .chain(actions) - } - - pub fn with_git_hash(mut self, git_hash: GitHash) -> Self { - self.git_hash = Some(git_hash); - self - } -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct HardwareRequirements { - #[serde(default)] - device: BTreeMap, - ram: Option, - pub arch: Option>, -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Assets { - #[serde(default)] - pub license: Option, - #[serde(default)] - pub instructions: Option, - #[serde(default)] - pub icon: Option, - #[serde(default)] - pub docker_images: Option, - #[serde(default)] - pub assets: Option, - #[serde(default)] - pub scripts: Option, -} -impl Assets { - pub fn license_path(&self) -> &Path { - self.license - .as_ref() - .map(|a| a.as_path()) - .unwrap_or(Path::new("LICENSE.md")) - } - pub fn instructions_path(&self) -> &Path { - self.instructions - .as_ref() - .map(|a| a.as_path()) - .unwrap_or(Path::new("INSTRUCTIONS.md")) - } - pub fn icon_path(&self) -> &Path { - self.icon - .as_ref() - .map(|a| a.as_path()) - .unwrap_or(Path::new("icon.png")) - } - pub fn icon_type(&self) -> &str { - self.icon - .as_ref() - .and_then(|icon| icon.extension()) - .and_then(|ext| ext.to_str()) - .unwrap_or("png") - } - pub fn docker_images_path(&self) -> &Path { - self.docker_images - .as_ref() - .map(|a| a.as_path()) - .unwrap_or(Path::new("docker-images")) - } - pub fn assets_path(&self) -> &Path { - self.assets - .as_ref() - .map(|a| a.as_path()) - .unwrap_or(Path::new("assets")) - } - pub fn scripts_path(&self) -> &Path { - self.scripts - .as_ref() - .map(|a| a.as_path()) - .unwrap_or(Path::new("scripts")) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -pub struct Description { - pub short: String, - pub long: String, -} -impl Description { - pub fn validate(&self) -> Result<(), Error> { - if self.short.chars().skip(160).next().is_some() { - return Err(Error::new( - eyre!("Short description must be 160 characters or less."), - crate::ErrorKind::ValidateS9pk, - )); - } - if self.long.chars().skip(5000).next().is_some() { - return Err(Error::new( - eyre!("Long description must be 5000 characters or less."), - crate::ErrorKind::ValidateS9pk, - )); - } - Ok(()) - } -} - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct Alerts { - pub install: Option, - pub uninstall: Option, - pub restore: Option, - pub start: Option, - pub stop: Option, -} diff --git a/core/startos/src/s9pk/mod.rs b/core/startos/src/s9pk/mod.rs deleted file mode 100644 index e1bf4cab..00000000 --- a/core/startos/src/s9pk/mod.rs +++ /dev/null @@ -1,246 +0,0 @@ -use std::ffi::OsStr; -use std::path::PathBuf; - -use color_eyre::eyre::eyre; -use futures::TryStreamExt; -use imbl::OrdMap; -use rpc_toolkit::command; -use serde_json::Value; -use tokio::io::AsyncRead; -use tracing::instrument; - -use crate::context::SdkContext; -use crate::s9pk::builder::S9pkPacker; -use crate::s9pk::docker::DockerMultiArch; -use crate::s9pk::git_hash::GitHash; -use crate::s9pk::manifest::Manifest; -use crate::s9pk::reader::S9pkReader; -use crate::util::display_none; -use crate::util::io::BufferedWriteReader; -use crate::util::serde::IoFormat; -use crate::volume::Volume; -use crate::{Error, ErrorKind, ResultExt}; - -pub mod builder; -pub mod docker; -pub mod git_hash; -pub mod header; -pub mod manifest; -pub mod reader; - -pub const SIG_CONTEXT: &[u8] = b"s9pk"; - -#[command(cli_only, display(display_none))] -#[instrument(skip_all)] -pub async fn pack(#[context] ctx: SdkContext, #[arg] path: Option) -> Result<(), Error> { - use tokio::fs::File; - - let path = if let Some(path) = path { - path - } else { - std::env::current_dir()? - }; - let manifest_value: Value = if path.join("manifest.toml").exists() { - IoFormat::Toml - .from_async_reader(File::open(path.join("manifest.toml")).await?) - .await? - } else if path.join("manifest.yaml").exists() { - IoFormat::Yaml - .from_async_reader(File::open(path.join("manifest.yaml")).await?) - .await? - } else if path.join("manifest.json").exists() { - IoFormat::Json - .from_async_reader(File::open(path.join("manifest.json")).await?) - .await? - } else { - return Err(Error::new( - eyre!("manifest not found"), - crate::ErrorKind::Pack, - )); - }; - - let manifest: Manifest = serde_json::from_value::(manifest_value.clone()) - .with_kind(crate::ErrorKind::Deserialization)? - .with_git_hash(GitHash::from_path(&path).await?); - let extra_keys = - enumerate_extra_keys(&serde_json::to_value(&manifest).unwrap(), &manifest_value); - for k in extra_keys { - tracing::warn!("Unrecognized Manifest Key: {}", k); - } - - let outfile_path = path.join(format!("{}.s9pk", manifest.id)); - let mut outfile = File::create(outfile_path).await?; - S9pkPacker::builder() - .manifest(&manifest) - .writer(&mut outfile) - .license( - File::open(path.join(manifest.assets.license_path())) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - manifest.assets.license_path().display().to_string(), - ) - })?, - ) - .icon( - File::open(path.join(manifest.assets.icon_path())) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - manifest.assets.icon_path().display().to_string(), - ) - })?, - ) - .instructions( - File::open(path.join(manifest.assets.instructions_path())) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - manifest.assets.instructions_path().display().to_string(), - ) - })?, - ) - .docker_images({ - let docker_images_path = path.join(manifest.assets.docker_images_path()); - let res: Box = if tokio::fs::metadata(&docker_images_path).await?.is_dir() { - let tars: Vec<_> = tokio_stream::wrappers::ReadDirStream::new(tokio::fs::read_dir(&docker_images_path).await?).try_collect().await?; - let mut arch_info = DockerMultiArch::default(); - for tar in &tars { - if tar.path().extension() == Some(OsStr::new("tar")) { - arch_info.available.insert(tar.path().file_stem().unwrap_or_default().to_str().unwrap_or_default().to_owned()); - } - } - if arch_info.available.contains("aarch64") { - arch_info.default = "aarch64".to_owned(); - } else { - arch_info.default = arch_info.available.iter().next().cloned().unwrap_or_default(); - } - let arch_info_cbor = IoFormat::Cbor.to_vec(&arch_info)?; - Box::new(BufferedWriteReader::new(|w| async move { - let mut docker_images = tokio_tar::Builder::new(w); - let mut multiarch_header = tokio_tar::Header::new_gnu(); - multiarch_header.set_path("multiarch.cbor")?; - multiarch_header.set_size(arch_info_cbor.len() as u64); - multiarch_header.set_cksum(); - docker_images.append(&multiarch_header, std::io::Cursor::new(arch_info_cbor)).await?; - for tar in tars - { - docker_images - .append_path_with_name( - tar.path(), - tar.file_name(), - ) - .await?; - } - Ok::<_, std::io::Error>(()) - }, 1024 * 1024)) - } else { - Box::new(File::open(docker_images_path) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - manifest.assets.docker_images_path().display().to_string(), - ) - })?) - }; - res - }) - .assets({ - let asset_volumes = manifest - .volumes - .iter() - .filter(|(_, v)| matches!(v, &&Volume::Assets {})).map(|(id, _)| id.clone()).collect::>(); - let assets_path = manifest.assets.assets_path().to_owned(); - let path = path.clone(); - - BufferedWriteReader::new(|w| async move { - let mut assets = tokio_tar::Builder::new(w); - for asset_volume in asset_volumes - { - assets - .append_dir_all( - &asset_volume, - path.join(&assets_path).join(&asset_volume), - ) - .await?; - } - Ok::<_, std::io::Error>(()) - }, 1024 * 1024) - }) - .scripts({ - let script_path = path.join(manifest.assets.scripts_path()).join("embassy.js"); - let needs_script = manifest.package_procedures().any(|a| a.is_script()); - let has_script = script_path.exists(); - match (needs_script, has_script) { - (true, true) => Some(File::open(script_path).await?), - (true, false) => { - return Err(Error::new(eyre!("Script is declared in manifest, but no such script exists at ./scripts/embassy.js"), ErrorKind::Pack).into()) - } - (false, true) => { - tracing::warn!("Manifest does not declare any actions that use scripts, but a script exists at ./scripts/embassy.js"); - None - } - (false, false) => None - } - }) - .build() - .pack(&ctx.developer_key()?) - .await?; - outfile.sync_all().await?; - - Ok(()) -} - -#[command(rename = "s9pk", cli_only, display(display_none))] -pub async fn verify(#[arg] path: PathBuf) -> Result<(), Error> { - let mut s9pk = S9pkReader::open(path, true).await?; - s9pk.validate().await?; - - Ok(()) -} - -fn enumerate_extra_keys(reference: &Value, candidate: &Value) -> Vec { - match (reference, candidate) { - (Value::Object(m_r), Value::Object(m_c)) => { - let om_r: OrdMap = m_r.clone().into_iter().collect(); - let om_c: OrdMap = m_c.clone().into_iter().collect(); - let common = om_r.clone().intersection(om_c.clone()); - let top_extra = common.clone().symmetric_difference(om_c.clone()); - let mut all_extra = top_extra - .keys() - .map(|s| format!(".{}", s)) - .collect::>(); - for (k, v) in common { - all_extra.extend( - enumerate_extra_keys(&v, om_c.get(&k).unwrap()) - .into_iter() - .map(|s| format!(".{}{}", k, s)), - ) - } - all_extra - } - (_, Value::Object(m1)) => m1.clone().keys().map(|s| format!(".{}", s)).collect(), - _ => Vec::new(), - } -} - -#[test] -fn test_enumerate_extra_keys() { - use serde_json::json; - let extras = enumerate_extra_keys( - &json!({ - "test": 1, - "test2": null, - }), - &json!({ - "test": 1, - "test2": { "test3": null }, - "test4": null - }), - ); - println!("{:?}", extras) -} diff --git a/core/startos/src/s9pk/reader.rs b/core/startos/src/s9pk/reader.rs deleted file mode 100644 index 61b5e46a..00000000 --- a/core/startos/src/s9pk/reader.rs +++ /dev/null @@ -1,406 +0,0 @@ -use std::collections::BTreeSet; -use std::io::SeekFrom; -use std::ops::Range; -use std::path::Path; -use std::pin::Pin; -use std::str::FromStr; -use std::task::{Context, Poll}; - -use color_eyre::eyre::eyre; -use digest::Output; -use ed25519_dalek::VerifyingKey; -use futures::TryStreamExt; -use models::ImageId; -use sha2::{Digest, Sha512}; -use tokio::fs::File; -use tokio::io::{AsyncRead, AsyncReadExt, AsyncSeek, AsyncSeekExt, ReadBuf}; -use tracing::instrument; - -use super::header::{FileSection, Header, TableOfContents}; -use super::manifest::{Manifest, PackageId}; -use super::SIG_CONTEXT; -use crate::install::progress::InstallProgressTracker; -use crate::s9pk::docker::DockerReader; -use crate::util::Version; -use crate::{Error, ResultExt}; - -const MAX_REPLACES: usize = 10; -const MAX_TITLE_LEN: usize = 30; - -#[pin_project::pin_project] -#[derive(Debug)] -pub struct ReadHandle<'a, R = File> { - pos: &'a mut u64, - range: Range, - #[pin] - rdr: &'a mut R, -} -impl<'a, R: AsyncRead + Unpin> ReadHandle<'a, R> { - pub async fn to_vec(mut self) -> std::io::Result> { - let mut buf = vec![0; (self.range.end - self.range.start) as usize]; - self.read_exact(&mut buf).await?; - Ok(buf) - } -} -impl<'a, R: AsyncRead + Unpin> AsyncRead for ReadHandle<'a, R> { - fn poll_read( - self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &mut ReadBuf<'_>, - ) -> Poll> { - let this = self.project(); - let start = buf.filled().len(); - let mut take_buf = buf.take(this.range.end.saturating_sub(**this.pos) as usize); - let res = AsyncRead::poll_read(this.rdr, cx, &mut take_buf); - let n = take_buf.filled().len(); - unsafe { buf.assume_init(start + n) }; - buf.advance(n); - **this.pos += n as u64; - res - } -} -impl<'a, R: AsyncSeek + Unpin> AsyncSeek for ReadHandle<'a, R> { - fn start_seek(self: Pin<&mut Self>, position: SeekFrom) -> std::io::Result<()> { - let this = self.project(); - AsyncSeek::start_seek( - this.rdr, - match position { - SeekFrom::Current(n) => SeekFrom::Current(n), - SeekFrom::End(n) => SeekFrom::Start((this.range.end as i64 + n) as u64), - SeekFrom::Start(n) => SeekFrom::Start(this.range.start + n), - }, - ) - } - fn poll_complete(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - let this = self.project(); - match AsyncSeek::poll_complete(this.rdr, cx) { - Poll::Ready(Ok(n)) => { - let res = n.saturating_sub(this.range.start); - **this.pos = this.range.start + res; - Poll::Ready(Ok(res)) - } - a => a, - } - } -} - -#[derive(Debug)] -pub struct ImageTag { - pub package_id: PackageId, - pub image_id: ImageId, - pub version: Version, -} -impl ImageTag { - #[instrument(skip_all)] - pub fn validate(&self, id: &PackageId, version: &Version) -> Result<(), Error> { - if id != &self.package_id { - return Err(Error::new( - eyre!( - "Contains image for incorrect package: id {}", - self.package_id, - ), - crate::ErrorKind::ValidateS9pk, - )); - } - if version != &self.version { - return Err(Error::new( - eyre!( - "Contains image with incorrect version: expected {} received {}", - version, - self.version, - ), - crate::ErrorKind::ValidateS9pk, - )); - } - Ok(()) - } -} -impl FromStr for ImageTag { - type Err = Error; - fn from_str(s: &str) -> Result { - let rest = s.strip_prefix("start9/").ok_or_else(|| { - Error::new( - eyre!("Invalid image tag prefix: expected start9/"), - crate::ErrorKind::ValidateS9pk, - ) - })?; - let (package, rest) = rest.split_once("/").ok_or_else(|| { - Error::new( - eyre!("Image tag missing image id"), - crate::ErrorKind::ValidateS9pk, - ) - })?; - let (image, version) = rest.split_once(":").ok_or_else(|| { - Error::new( - eyre!("Image tag missing version"), - crate::ErrorKind::ValidateS9pk, - ) - })?; - Ok(ImageTag { - package_id: package.parse()?, - image_id: image.parse()?, - version: version.parse()?, - }) - } -} - -pub struct S9pkReader { - hash: Option>, - hash_string: Option, - developer_key: VerifyingKey, - toc: TableOfContents, - pos: u64, - rdr: R, -} -impl S9pkReader { - pub async fn open>(path: P, check_sig: bool) -> Result { - let p = path.as_ref(); - let rdr = File::open(p) - .await - .with_ctx(|_| (crate::error::ErrorKind::Filesystem, p.display().to_string()))?; - - Self::from_reader(rdr, check_sig).await - } -} -impl S9pkReader> { - pub fn validated(&mut self) { - self.rdr.validated() - } -} -impl S9pkReader { - #[instrument(skip_all)] - pub async fn validate(&mut self) -> Result<(), Error> { - if self.toc.icon.length > 102_400 { - // 100 KiB - return Err(Error::new( - eyre!("icon must be less than 100KiB"), - crate::ErrorKind::ValidateS9pk, - )); - } - let image_tags = self.image_tags().await?; - let man = self.manifest().await?; - let containers = &man.containers; - let validated_image_ids = image_tags - .into_iter() - .map(|i| i.validate(&man.id, &man.version).map(|_| i.image_id)) - .collect::, _>>()?; - man.description.validate()?; - man.actions.0.iter().try_for_each(|(_, action)| { - action.validate( - containers, - &man.eos_version, - &man.volumes, - &validated_image_ids, - ) - })?; - man.backup.validate( - containers, - &man.eos_version, - &man.volumes, - &validated_image_ids, - )?; - if let Some(cfg) = &man.config { - cfg.validate( - containers, - &man.eos_version, - &man.volumes, - &validated_image_ids, - )?; - } - man.health_checks - .validate(&man.eos_version, &man.volumes, &validated_image_ids)?; - man.interfaces.validate()?; - man.main - .validate(&man.eos_version, &man.volumes, &validated_image_ids, false) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Main"))?; - man.migrations.validate( - containers, - &man.eos_version, - &man.volumes, - &validated_image_ids, - )?; - - #[cfg(feature = "js-engine")] - if man.containers.is_some() - || matches!(man.main, crate::procedure::PackageProcedure::Script(_)) - { - return Err(Error::new( - eyre!("Right now we don't support the containers and the long running main"), - crate::ErrorKind::ValidateS9pk, - )); - } - - if man.replaces.len() >= MAX_REPLACES { - return Err(Error::new( - eyre!("Cannot have more than {MAX_REPLACES} replaces"), - crate::ErrorKind::ValidateS9pk, - )); - } - if let Some(too_big) = man.replaces.iter().find(|x| x.len() >= MAX_REPLACES) { - return Err(Error::new( - eyre!("We have found a replaces of ({too_big}) that exceeds the max length of {MAX_TITLE_LEN} "), - crate::ErrorKind::ValidateS9pk, - )); - } - if man.title.len() >= MAX_TITLE_LEN { - return Err(Error::new( - eyre!("Cannot have more than a length of {MAX_TITLE_LEN} for title"), - crate::ErrorKind::ValidateS9pk, - )); - } - - if man.containers.is_some() - && matches!(man.main, crate::procedure::PackageProcedure::Docker(_)) - { - return Err(Error::new( - eyre!("Cannot have a main docker and a main in containers"), - crate::ErrorKind::ValidateS9pk, - )); - } - if let Some(props) = &man.properties { - props - .validate(&man.eos_version, &man.volumes, &validated_image_ids, true) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Properties"))?; - } - man.volumes.validate(&man.interfaces)?; - - Ok(()) - } - #[instrument(skip_all)] - pub async fn image_tags(&mut self) -> Result, Error> { - let mut tar = tokio_tar::Archive::new(self.docker_images().await?); - let mut entries = tar.entries()?; - while let Some(mut entry) = entries.try_next().await? { - if &*entry.path()? != Path::new("manifest.json") { - continue; - } - let mut buf = Vec::with_capacity(entry.header().size()? as usize); - entry.read_to_end(&mut buf).await?; - #[derive(serde::Deserialize)] - struct ManEntry { - #[serde(rename = "RepoTags")] - tags: Vec, - } - let man_entries = serde_json::from_slice::>(&buf) - .with_ctx(|_| (crate::ErrorKind::Deserialization, "manifest.json"))?; - return man_entries - .iter() - .flat_map(|e| &e.tags) - .map(|t| t.parse()) - .collect(); - } - Err(Error::new( - eyre!("image.tar missing manifest.json"), - crate::ErrorKind::ParseS9pk, - )) - } - #[instrument(skip_all)] - pub async fn from_reader(mut rdr: R, check_sig: bool) -> Result { - let header = Header::deserialize(&mut rdr).await?; - - let (hash, hash_string) = if check_sig { - let mut hasher = Sha512::new(); - let mut buf = [0; 1024]; - let mut read; - while { - read = rdr.read(&mut buf).await?; - read != 0 - } { - hasher.update(&buf[0..read]); - } - let hash = hasher.clone().finalize(); - header - .pubkey - .verify_prehashed(hasher, Some(SIG_CONTEXT), &header.signature)?; - ( - Some(hash), - Some(base32::encode( - base32::Alphabet::RFC4648 { padding: false }, - hash.as_slice(), - )), - ) - } else { - (None, None) - }; - - let pos = rdr.stream_position().await?; - - Ok(S9pkReader { - hash_string, - hash, - developer_key: header.pubkey, - toc: header.table_of_contents, - pos, - rdr, - }) - } - - pub fn hash(&self) -> Option<&Output> { - self.hash.as_ref() - } - - pub fn hash_str(&self) -> Option<&str> { - self.hash_string.as_ref().map(|s| s.as_str()) - } - - pub fn developer_key(&self) -> &VerifyingKey { - &self.developer_key - } - - pub async fn reset(&mut self) -> Result<(), Error> { - self.rdr.seek(SeekFrom::Start(0)).await?; - Ok(()) - } - - async fn read_handle<'a>( - &'a mut self, - section: FileSection, - ) -> Result, Error> { - if self.pos != section.position { - self.rdr.seek(SeekFrom::Start(section.position)).await?; - self.pos = section.position; - } - Ok(ReadHandle { - range: self.pos..(self.pos + section.length), - pos: &mut self.pos, - rdr: &mut self.rdr, - }) - } - - pub async fn manifest_raw(&mut self) -> Result, Error> { - self.read_handle(self.toc.manifest).await - } - - pub async fn manifest(&mut self) -> Result { - let slice = self.manifest_raw().await?.to_vec().await?; - serde_cbor::de::from_reader(slice.as_slice()) - .with_ctx(|_| (crate::ErrorKind::ParseS9pk, "Deserializing Manifest (CBOR)")) - } - - pub async fn license(&mut self) -> Result, Error> { - self.read_handle(self.toc.license).await - } - - pub async fn instructions(&mut self) -> Result, Error> { - self.read_handle(self.toc.instructions).await - } - - pub async fn icon(&mut self) -> Result, Error> { - self.read_handle(self.toc.icon).await - } - - pub async fn docker_images(&mut self) -> Result>, Error> { - DockerReader::new(self.read_handle(self.toc.docker_images).await?).await - } - - pub async fn assets(&mut self) -> Result, Error> { - self.read_handle(self.toc.assets).await - } - - pub async fn scripts(&mut self) -> Result>, Error> { - Ok(match self.toc.scripts { - None => None, - Some(a) => Some(self.read_handle(a).await?), - }) - } -} diff --git a/core/startos/src/s9pk/specv2.md b/core/startos/src/s9pk/specv2.md deleted file mode 100644 index 9bf99346..00000000 --- a/core/startos/src/s9pk/specv2.md +++ /dev/null @@ -1,28 +0,0 @@ -## Header - -### Magic - -2B: `0x3b3b` - -### Version - -varint: `0x02` - -### Pubkey - -32B: ed25519 pubkey - -### TOC - -- number of sections (varint) -- FOREACH section - - sig (32B: ed25519 signature of BLAKE-3 of rest of section) - - name (varstring) - - TYPE (varint) - - TYPE=FILE (`0x01`) - - mime (varstring) - - pos (32B: u64 BE) - - len (32B: u64 BE) - - hash (32B: BLAKE-3 of file contents) - - TYPE=TOC (`0x02`) - - recursively defined diff --git a/core/startos/src/setup.rs b/core/startos/src/setup.rs deleted file mode 100644 index 64c32409..00000000 --- a/core/startos/src/setup.rs +++ /dev/null @@ -1,510 +0,0 @@ -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use std::time::Duration; - -use color_eyre::eyre::eyre; -use josekit::jwk::Jwk; -use openssl::x509::X509; -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; -use serde::{Deserialize, Serialize}; -use sqlx::Connection; -use tokio::fs::File; -use tokio::io::AsyncWriteExt; -use tokio::try_join; -use torut::onion::OnionAddressV3; -use tracing::instrument; - -use crate::account::AccountInfo; -use crate::backup::restore::recover_full_embassy; -use crate::backup::target::BackupTargetFS; -use crate::context::rpc::RpcContextConfig; -use crate::context::setup::SetupResult; -use crate::context::SetupContext; -use crate::disk::fsck::RepairStrategy; -use crate::disk::main::DEFAULT_PASSWORD; -use crate::disk::mount::filesystem::cifs::Cifs; -use crate::disk::mount::filesystem::ReadWrite; -use crate::disk::mount::guard::TmpMountGuard; -use crate::disk::util::{pvscan, recovery_info, DiskInfo, EmbassyOsRecoveryInfo}; -use crate::disk::REPAIR_DISK_PATH; -use crate::hostname::Hostname; -use crate::init::{init, InitResult}; -use crate::middleware::encrypt::EncryptedWire; -use crate::net::ssl::root_ca_start_time; -use crate::prelude::*; -use crate::util::io::{dir_copy, dir_size, Counter}; -use crate::{Error, ErrorKind, ResultExt}; - -#[command(subcommands(status, disk, attach, execute, cifs, complete, get_pubkey, exit))] -pub fn setup() -> Result<(), Error> { - Ok(()) -} - -#[command(subcommands(list_disks))] -pub fn disk() -> Result<(), Error> { - Ok(()) -} - -#[command(rename = "list", rpc_only, metadata(authenticated = false))] -pub async fn list_disks(#[context] ctx: SetupContext) -> Result, Error> { - crate::disk::util::list(&ctx.os_partitions).await -} - -async fn setup_init( - ctx: &SetupContext, - password: Option, -) -> Result<(Hostname, OnionAddressV3, X509), Error> { - let InitResult { secret_store, db } = - init(&RpcContextConfig::load(ctx.config_path.clone()).await?).await?; - let mut secrets_handle = secret_store.acquire().await?; - let mut secrets_tx = secrets_handle.begin().await?; - - let mut account = AccountInfo::load(secrets_tx.as_mut()).await?; - - if let Some(password) = password { - account.set_password(&password)?; - account.save(secrets_tx.as_mut()).await?; - db.mutate(|m| { - m.as_server_info_mut() - .as_password_hash_mut() - .ser(&account.password) - }) - .await?; - } - - secrets_tx.commit().await?; - - Ok(( - account.hostname, - account.key.tor_address(), - account.root_ca_cert, - )) -} - -#[command(rpc_only)] -pub async fn attach( - #[context] ctx: SetupContext, - #[arg] guid: Arc, - #[arg(rename = "embassy-password")] password: Option, -) -> Result<(), Error> { - let mut status = ctx.setup_status.write().await; - if status.is_some() { - return Err(Error::new( - eyre!("Setup already in progress"), - ErrorKind::InvalidRequest, - )); - } - *status = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: None, - complete: false, - })); - drop(status); - tokio::task::spawn(async move { - if let Err(e) = async { - let password: Option = match password { - Some(a) => match a.decrypt(&*ctx) { - a @ Some(_) => a, - None => { - return Err(Error::new( - color_eyre::eyre::eyre!("Couldn't decode password"), - crate::ErrorKind::Unknown, - )); - } - }, - None => None, - }; - let requires_reboot = crate::disk::main::import( - &*guid, - &ctx.datadir, - if tokio::fs::metadata(REPAIR_DISK_PATH).await.is_ok() { - RepairStrategy::Aggressive - } else { - RepairStrategy::Preen - }, - if guid.ends_with("_UNENC") { None } else { Some(DEFAULT_PASSWORD) }, - ) - .await?; - if tokio::fs::metadata(REPAIR_DISK_PATH).await.is_ok() { - tokio::fs::remove_file(REPAIR_DISK_PATH) - .await - .with_ctx(|_| (ErrorKind::Filesystem, REPAIR_DISK_PATH))?; - } - if requires_reboot.0 { - crate::disk::main::export(&*guid, &ctx.datadir).await?; - return Err(Error::new( - eyre!( - "Errors were corrected with your disk, but the server must be restarted in order to proceed" - ), - ErrorKind::DiskManagement, - )); - } - let (hostname, tor_addr, root_ca) = setup_init(&ctx, password).await?; - *ctx.setup_result.write().await = Some((guid, SetupResult { - tor_address: format!("https://{}", tor_addr), - lan_address: hostname.lan_address(), - root_ca: String::from_utf8(root_ca.to_pem()?)?, - })); - *ctx.setup_status.write().await = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: None, - complete: true, - })); - Ok(()) - }.await { - tracing::error!("Error Setting Up Embassy: {}", e); - tracing::debug!("{:?}", e); - *ctx.setup_status.write().await = Some(Err(e.into())); - } - }); - Ok(()) -} - -#[derive(Debug, Clone, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct SetupStatus { - pub bytes_transferred: u64, - pub total_bytes: Option, - pub complete: bool, -} - -#[command(rpc_only, metadata(authenticated = false))] -pub async fn status(#[context] ctx: SetupContext) -> Result, RpcError> { - ctx.setup_status.read().await.clone().transpose() -} - -/// We want to be able to get a secret, a shared private key with the frontend -/// This way the frontend can send a secret, like the password for the setup/ recovory -/// without knowing the password over clearnet. We use the public key shared across the network -/// since it is fine to share the public, and encrypt against the public. -#[command(rename = "get-pubkey", rpc_only, metadata(authenticated = false))] -pub async fn get_pubkey(#[context] ctx: SetupContext) -> Result { - let secret = ctx.as_ref().clone(); - let pub_key = secret.to_public_key()?; - Ok(pub_key) -} - -#[command(subcommands(verify_cifs))] -pub fn cifs() -> Result<(), Error> { - Ok(()) -} - -#[command(rename = "verify", rpc_only)] -pub async fn verify_cifs( - #[context] ctx: SetupContext, - #[arg] hostname: String, - #[arg] path: PathBuf, - #[arg] username: String, - #[arg] password: Option, -) -> Result { - let password: Option = password.map(|x| x.decrypt(&*ctx)).flatten(); - let guard = TmpMountGuard::mount( - &Cifs { - hostname, - path, - username, - password, - }, - ReadWrite, - ) - .await?; - let embassy_os = recovery_info(&guard).await?; - guard.unmount().await?; - embassy_os.ok_or_else(|| Error::new(eyre!("No Backup Found"), crate::ErrorKind::NotFound)) -} - -#[derive(Debug, Deserialize)] -#[serde(tag = "type")] -#[serde(rename_all = "kebab-case")] -pub enum RecoverySource { - Migrate { guid: String }, - Backup { target: BackupTargetFS }, -} - -#[command(rpc_only)] -pub async fn execute( - #[context] ctx: SetupContext, - #[arg(rename = "embassy-logicalname")] embassy_logicalname: PathBuf, - #[arg(rename = "embassy-password")] embassy_password: EncryptedWire, - #[arg(rename = "recovery-source")] recovery_source: Option, - #[arg(rename = "recovery-password")] recovery_password: Option, -) -> Result<(), Error> { - let embassy_password = match embassy_password.decrypt(&*ctx) { - Some(a) => a, - None => { - return Err(Error::new( - color_eyre::eyre::eyre!("Couldn't decode embassy-password"), - crate::ErrorKind::Unknown, - )) - } - }; - let recovery_password: Option = match recovery_password { - Some(a) => match a.decrypt(&*ctx) { - Some(a) => Some(a), - None => { - return Err(Error::new( - color_eyre::eyre::eyre!("Couldn't decode recovery-password"), - crate::ErrorKind::Unknown, - )) - } - }, - None => None, - }; - let mut status = ctx.setup_status.write().await; - if status.is_some() { - return Err(Error::new( - eyre!("Setup already in progress"), - ErrorKind::InvalidRequest, - )); - } - *status = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: None, - complete: false, - })); - drop(status); - tokio::task::spawn({ - async move { - let ctx = ctx.clone(); - let recovery_source = recovery_source; - - let embassy_password = embassy_password; - let recovery_source = recovery_source; - let recovery_password = recovery_password; - match execute_inner( - ctx.clone(), - embassy_logicalname, - embassy_password, - recovery_source, - recovery_password, - ) - .await - { - Ok((guid, hostname, tor_addr, root_ca)) => { - tracing::info!("Setup Complete!"); - *ctx.setup_result.write().await = Some(( - guid, - SetupResult { - tor_address: format!("https://{}", tor_addr), - lan_address: hostname.lan_address(), - root_ca: String::from_utf8( - root_ca.to_pem().expect("failed to serialize root ca"), - ) - .expect("invalid pem string"), - }, - )); - *ctx.setup_status.write().await = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: None, - complete: true, - })); - } - Err(e) => { - tracing::error!("Error Setting Up Server: {}", e); - tracing::debug!("{:?}", e); - *ctx.setup_status.write().await = Some(Err(e.into())); - } - } - } - }); - Ok(()) -} - -#[instrument(skip_all)] -#[command(rpc_only)] -pub async fn complete(#[context] ctx: SetupContext) -> Result { - let (guid, setup_result) = if let Some((guid, setup_result)) = &*ctx.setup_result.read().await { - (guid.clone(), setup_result.clone()) - } else { - return Err(Error::new( - eyre!("setup.execute has not completed successfully"), - crate::ErrorKind::InvalidRequest, - )); - }; - let mut guid_file = File::create("/media/embassy/config/disk.guid").await?; - guid_file.write_all(guid.as_bytes()).await?; - guid_file.sync_all().await?; - Ok(setup_result) -} - -#[instrument(skip_all)] -#[command(rpc_only)] -pub async fn exit(#[context] ctx: SetupContext) -> Result<(), Error> { - ctx.shutdown.send(()).expect("failed to shutdown"); - Ok(()) -} - -#[instrument(skip_all)] -pub async fn execute_inner( - ctx: SetupContext, - embassy_logicalname: PathBuf, - embassy_password: String, - recovery_source: Option, - recovery_password: Option, -) -> Result<(Arc, Hostname, OnionAddressV3, X509), Error> { - let encryption_password = if ctx.disable_encryption { - None - } else { - Some(DEFAULT_PASSWORD) - }; - let guid = Arc::new( - crate::disk::main::create( - &[embassy_logicalname], - &pvscan().await?, - &ctx.datadir, - encryption_password, - ) - .await?, - ); - let _ = crate::disk::main::import( - &*guid, - &ctx.datadir, - RepairStrategy::Preen, - encryption_password, - ) - .await?; - - if let Some(RecoverySource::Backup { target }) = recovery_source { - recover(ctx, guid, embassy_password, target, recovery_password).await - } else if let Some(RecoverySource::Migrate { guid: old_guid }) = recovery_source { - migrate(ctx, guid, &old_guid, embassy_password).await - } else { - let (hostname, tor_addr, root_ca) = fresh_setup(&ctx, &embassy_password).await?; - Ok((guid, hostname, tor_addr, root_ca)) - } -} - -async fn fresh_setup( - ctx: &SetupContext, - embassy_password: &str, -) -> Result<(Hostname, OnionAddressV3, X509), Error> { - let account = AccountInfo::new(embassy_password, root_ca_start_time().await?)?; - let sqlite_pool = ctx.secret_store().await?; - account.save(&sqlite_pool).await?; - sqlite_pool.close().await; - let InitResult { secret_store, .. } = - init(&RpcContextConfig::load(ctx.config_path.clone()).await?).await?; - secret_store.close().await; - Ok(( - account.hostname.clone(), - account.key.tor_address(), - account.root_ca_cert.clone(), - )) -} - -#[instrument(skip_all)] -async fn recover( - ctx: SetupContext, - guid: Arc, - embassy_password: String, - recovery_source: BackupTargetFS, - recovery_password: Option, -) -> Result<(Arc, Hostname, OnionAddressV3, X509), Error> { - let recovery_source = TmpMountGuard::mount(&recovery_source, ReadWrite).await?; - recover_full_embassy( - ctx, - guid.clone(), - embassy_password, - recovery_source, - recovery_password, - ) - .await -} - -#[instrument(skip_all)] -async fn migrate( - ctx: SetupContext, - guid: Arc, - old_guid: &str, - embassy_password: String, -) -> Result<(Arc, Hostname, OnionAddressV3, X509), Error> { - *ctx.setup_status.write().await = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: None, - complete: false, - })); - - let _ = crate::disk::main::import( - &old_guid, - "/media/embassy/migrate", - RepairStrategy::Preen, - if guid.ends_with("_UNENC") { - None - } else { - Some(DEFAULT_PASSWORD) - }, - ) - .await?; - - let main_transfer_args = ("/media/embassy/migrate/main/", "/embassy-data/main/"); - let package_data_transfer_args = ( - "/media/embassy/migrate/package-data/", - "/embassy-data/package-data/", - ); - - let tmpdir = Path::new(package_data_transfer_args.0).join("tmp"); - if tokio::fs::metadata(&tmpdir).await.is_ok() { - tokio::fs::remove_dir_all(&tmpdir).await?; - } - - let ordering = std::sync::atomic::Ordering::Relaxed; - - let main_transfer_size = Counter::new(0, ordering); - let package_data_transfer_size = Counter::new(0, ordering); - - let size = tokio::select! { - res = async { - let (main_size, package_data_size) = try_join!( - dir_size(main_transfer_args.0, Some(&main_transfer_size)), - dir_size(package_data_transfer_args.0, Some(&package_data_transfer_size)) - )?; - Ok::<_, Error>(main_size + package_data_size) - } => { res? }, - res = async { - loop { - tokio::time::sleep(Duration::from_secs(1)).await; - *ctx.setup_status.write().await = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: Some(main_transfer_size.load() + package_data_transfer_size.load()), - complete: false, - })); - } - } => res, - }; - - *ctx.setup_status.write().await = Some(Ok(SetupStatus { - bytes_transferred: 0, - total_bytes: Some(size), - complete: false, - })); - - let main_transfer_progress = Counter::new(0, ordering); - let package_data_transfer_progress = Counter::new(0, ordering); - - tokio::select! { - res = async { - try_join!( - dir_copy(main_transfer_args.0, main_transfer_args.1, Some(&main_transfer_progress)), - dir_copy(package_data_transfer_args.0, package_data_transfer_args.1, Some(&package_data_transfer_progress)) - )?; - Ok::<_, Error>(()) - } => { res? }, - res = async { - loop { - tokio::time::sleep(Duration::from_secs(1)).await; - *ctx.setup_status.write().await = Some(Ok(SetupStatus { - bytes_transferred: main_transfer_progress.load() + package_data_transfer_progress.load(), - total_bytes: Some(size), - complete: false, - })); - } - } => res, - } - - let (hostname, tor_addr, root_ca) = setup_init(&ctx, Some(embassy_password)).await?; - - crate::disk::main::export(&old_guid, "/media/embassy/migrate").await?; - - Ok((guid, hostname, tor_addr, root_ca)) -} diff --git a/core/startos/src/shutdown.rs b/core/startos/src/shutdown.rs deleted file mode 100644 index e5ff969b..00000000 --- a/core/startos/src/shutdown.rs +++ /dev/null @@ -1,147 +0,0 @@ -use std::path::PathBuf; -use std::sync::Arc; - -use rpc_toolkit::command; - -use crate::context::RpcContext; -use crate::disk::main::export; -use crate::init::{STANDBY_MODE_PATH, SYSTEM_REBUILD_PATH}; -use crate::prelude::*; -use crate::sound::SHUTDOWN; -use crate::util::docker::CONTAINER_TOOL; -use crate::util::{display_none, Invoke}; -use crate::PLATFORM; - -#[derive(Debug, Clone)] -pub struct Shutdown { - pub export_args: Option<(Arc, PathBuf)>, - pub restart: bool, -} -impl Shutdown { - /// BLOCKING - pub fn execute(&self) { - use std::process::Command; - - if self.restart { - tracing::info!("Beginning server restart"); - } else { - tracing::info!("Beginning server shutdown"); - } - - let rt = tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .unwrap(); - rt.block_on(async { - use tokio::process::Command; - - if let Err(e) = Command::new("systemctl") - .arg("stop") - .arg("systemd-journald") - .invoke(crate::ErrorKind::Journald) - .await - { - tracing::error!("Error Stopping Journald: {}", e); - tracing::debug!("{:?}", e); - } - if CONTAINER_TOOL == "docker" { - if let Err(e) = Command::new("systemctl") - .arg("stop") - .arg("docker") - .invoke(crate::ErrorKind::Docker) - .await - { - tracing::error!("Error Stopping Docker: {}", e); - tracing::debug!("{:?}", e); - } - } else if CONTAINER_TOOL == "podman" { - if let Err(e) = Command::new("podman") - .arg("rm") - .arg("-f") - .arg("netdummy") - .invoke(crate::ErrorKind::Docker) - .await - { - tracing::error!("Error Stopping Podman: {}", e); - tracing::debug!("{:?}", e); - } - } - if let Some((guid, datadir)) = &self.export_args { - if let Err(e) = export(guid, datadir).await { - tracing::error!("Error Exporting Volume Group: {}", e); - tracing::debug!("{:?}", e); - } - } - if &*PLATFORM != "raspberrypi" || self.restart { - if let Err(e) = SHUTDOWN.play().await { - tracing::error!("Error Playing Shutdown Song: {}", e); - tracing::debug!("{:?}", e); - } - } - }); - drop(rt); - if &*PLATFORM == "raspberrypi" { - if !self.restart { - std::fs::write(STANDBY_MODE_PATH, "").unwrap(); - Command::new("sync").spawn().unwrap().wait().unwrap(); - } - Command::new("reboot").spawn().unwrap().wait().unwrap(); - } else if self.restart { - Command::new("reboot").spawn().unwrap().wait().unwrap(); - } else { - Command::new("shutdown") - .arg("-h") - .arg("now") - .spawn() - .unwrap() - .wait() - .unwrap(); - } - } -} - -#[command(display(display_none))] -pub async fn shutdown(#[context] ctx: RpcContext) -> Result<(), Error> { - ctx.db - .mutate(|db| { - db.as_server_info_mut() - .as_status_info_mut() - .as_shutting_down_mut() - .ser(&true) - }) - .await?; - ctx.shutdown - .send(Some(Shutdown { - export_args: Some((ctx.disk_guid.clone(), ctx.datadir.clone())), - restart: false, - })) - .map_err(|_| ()) - .expect("receiver dropped"); - Ok(()) -} - -#[command(display(display_none))] -pub async fn restart(#[context] ctx: RpcContext) -> Result<(), Error> { - ctx.db - .mutate(|db| { - db.as_server_info_mut() - .as_status_info_mut() - .as_restarting_mut() - .ser(&true) - }) - .await?; - ctx.shutdown - .send(Some(Shutdown { - export_args: Some((ctx.disk_guid.clone(), ctx.datadir.clone())), - restart: true, - })) - .map_err(|_| ()) - .expect("receiver dropped"); - Ok(()) -} - -#[command(display(display_none))] -pub async fn rebuild(#[context] ctx: RpcContext) -> Result<(), Error> { - tokio::fs::write(SYSTEM_REBUILD_PATH, b"").await?; - restart(ctx).await -} diff --git a/core/startos/src/sound.rs b/core/startos/src/sound.rs deleted file mode 100644 index 8dc78357..00000000 --- a/core/startos/src/sound.rs +++ /dev/null @@ -1,418 +0,0 @@ -use std::cmp::Ordering; -use std::time::Duration; - -use divrem::DivRem; -use proptest_derive::Arbitrary; -use tokio::process::Command; -use tracing::instrument; - -use crate::util::{FileLock, Invoke}; -use crate::{Error, ErrorKind}; - -lazy_static::lazy_static! { - static ref SEMITONE_K: f64 = 2f64.powf(1f64 / 12f64); - static ref A_4: f64 = 440f64; - static ref C_0: f64 = *A_4 / SEMITONE_K.powf(9f64) / 2f64.powf(4f64); -} - -pub const SOUND_LOCK_FILE: &str = "/etc/embassy/sound.lock"; - -struct SoundInterface { - guard: Option, -} -impl SoundInterface { - #[instrument(skip_all)] - pub async fn lease() -> Result { - let guard = FileLock::new(SOUND_LOCK_FILE, true).await?; - Ok(SoundInterface { guard: Some(guard) }) - } - #[instrument(skip_all)] - pub async fn close(mut self) -> Result<(), Error> { - if let Some(lock) = self.guard.take() { - lock.unlock().await?; - } - Ok(()) - } - #[instrument(skip_all)] - pub async fn play_for_time_slice( - &mut self, - tempo_qpm: u16, - note: &Note, - time_slice: &TimeSlice, - ) -> Result<(), Error> { - Command::new("beep") - .arg("-f") - .arg(note.frequency().to_string()) - .arg("-l") - .arg(time_slice.to_duration(tempo_qpm).as_millis().to_string()) - .invoke(ErrorKind::SoundError) - .await?; - Ok(()) - } -} - -pub struct Song { - pub tempo_qpm: u16, - pub note_sequence: Notes, -} -impl<'a, T> Song -where - T: IntoIterator, TimeSlice)> + Clone, -{ - #[instrument(skip_all)] - pub async fn play(&self) -> Result<(), Error> { - let mut sound = SoundInterface::lease().await?; - for (note, slice) in self.note_sequence.clone() { - match note { - None => tokio::time::sleep(slice.to_duration(self.tempo_qpm)).await, - Some(n) => { - sound - .play_for_time_slice(self.tempo_qpm, &n, &slice) - .await? - } - }; - } - sound.close().await?; - Ok(()) - } -} - -impl Drop for SoundInterface { - fn drop(&mut self) { - let guard = self.guard.take(); - tokio::spawn(async move { - if let Some(guard) = guard { - if let Err(e) = guard.unlock().await { - tracing::error!("Failed to drop Sound Interface File Lock: {}", e); - tracing::debug!("{:?}", e); - } - } - }); - } -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq, Arbitrary)] -pub struct Note { - pub semitone: Semitone, - pub octave: i8, -} -impl Note { - pub fn frequency(&self) -> f64 { - SEMITONE_K.powf((self.semitone as isize) as f64) * (*C_0) * (2f64.powf(self.octave as f64)) - } -} -impl PartialOrd for Note { - fn partial_cmp(&self, other: &Self) -> Option { - Some(self.cmp(other)) - } -} -impl Ord for Note { - fn cmp(&self, other: &Self) -> Ordering { - if self.octave == other.octave { - self.semitone.cmp(&other.semitone) - } else { - self.octave.cmp(&other.octave) - } - } -} -#[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Arbitrary)] -pub enum Semitone { - C = 0, - Db = 1, - D = 2, - Eb = 3, - E = 4, - F = 5, - Gb = 6, - G = 7, - Ab = 8, - A = 9, - Bb = 10, - B = 11, -} - -impl Semitone { - pub fn rotate(&self, n: isize) -> Semitone { - let temp = (*self as isize) + n; - - match temp.rem_euclid(12) { - 0 => Semitone::C, - 1 => Semitone::Db, - 2 => Semitone::D, - 3 => Semitone::Eb, - 4 => Semitone::E, - 5 => Semitone::F, - 6 => Semitone::Gb, - 7 => Semitone::G, - 8 => Semitone::Ab, - 9 => Semitone::A, - 10 => Semitone::Bb, - 11 => Semitone::B, - _ => panic!("crate::sound::Semitone::rotate: Unreachable"), - } - } -} - -#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Arbitrary)] -pub struct Interval(isize); - -#[derive(Debug, Clone, Copy)] -pub enum TimeSlice { - Sixteenth, - Eighth, - Quarter, - Half, - Whole, - Triplet(&'static TimeSlice), - Dot(&'static TimeSlice), - Tie(&'static TimeSlice, &'static TimeSlice), -} -impl TimeSlice { - pub fn to_duration(&self, tempo_qpm: u16) -> Duration { - let micros_per_quarter = 1_000_000f64 * 60f64 / tempo_qpm as f64; - match &self { - &Self::Sixteenth => Duration::from_micros((micros_per_quarter / 4.0) as u64), - &Self::Eighth => Duration::from_micros((micros_per_quarter / 2.0) as u64), - &Self::Quarter => Duration::from_micros(micros_per_quarter as u64), - &Self::Half => Duration::from_micros((micros_per_quarter * 2.0) as u64), - &Self::Whole => Duration::from_micros((micros_per_quarter * 4.0) as u64), - &Self::Triplet(ts) => ts.to_duration(tempo_qpm) * 2 / 3, - &Self::Dot(ts) => ts.to_duration(tempo_qpm) * 3 / 2, - &Self::Tie(ts0, ts1) => ts0.to_duration(tempo_qpm) + ts1.to_duration(tempo_qpm), - } - } -} - -pub fn interval(i: &Interval, note: &Note) -> Note { - match (i, note) { - (Interval(n), Note { semitone, octave }) => { - use std::cmp::Ordering::*; - let (o_t, s_t) = n.div_rem(12); - let new_semitone = semitone.rotate(s_t); - let new_octave = match (new_semitone.cmp(semitone), s_t.cmp(&0)) { - (Greater, Less) => octave.clone() as isize + o_t - 1, - (Less, Greater) => octave.clone() as isize + o_t + 1, - _ => octave.clone() as isize + o_t, - }; - Note { - semitone: new_semitone, - octave: new_octave as i8, - } - } - } -} - -pub const MINOR_THIRD: Interval = Interval(3); -pub const MAJOR_THIRD: Interval = Interval(4); -pub const FOURTH: Interval = Interval(5); -pub const FIFTH: Interval = Interval(7); - -fn iterate T>(f: F, init: &T) -> impl Iterator { - let mut temp = init.clone(); - let ff = move || { - let next = f(&temp); - let now = std::mem::replace(&mut temp, next); - now - }; - std::iter::repeat_with(ff) -} - -pub fn circle_of_fifths(note: &Note) -> impl Iterator { - iterate(|n| interval(&FIFTH, n), note) -} - -pub fn circle_of_fourths(note: &Note) -> impl Iterator { - iterate(|n| interval(&FOURTH, n), note) -} - -#[derive(Clone, Debug)] -pub struct CircleOf<'a> { - current: Note, - duration: TimeSlice, - interval: &'a Interval, -} -impl<'a> CircleOf<'a> { - pub const fn new(interval: &'a Interval, start: Note, duration: TimeSlice) -> Self { - CircleOf { - current: start, - duration, - interval, - } - } -} -impl<'a> Iterator for CircleOf<'a> { - type Item = (Option, TimeSlice); - fn next(&mut self) -> Option { - let current = self.current; - let prev = std::mem::replace(&mut self.current, interval(&self.interval, ¤t)); - Some((Some(prev), self.duration.clone())) - } -} - -macro_rules! song { - ($tempo:expr, [$($note:expr;)*]) => { - { - #[allow(dead_code)] - const fn note(semi: Semitone, octave: i8, duration: TimeSlice) -> (Option, TimeSlice) { - ( - Some(Note { - semitone: semi, - octave, - }), - duration, - ) - } - #[allow(dead_code)] - const fn rest(duration: TimeSlice) -> (Option, TimeSlice) { - (None, duration) - } - - use crate::sound::Semitone::*; - use crate::sound::TimeSlice::*; - Song { - tempo_qpm: $tempo as u16, - note_sequence: [ - $( - $note, - )* - ] - } - } - }; -} - -pub const BEP: Song<[(Option, TimeSlice); 1]> = song!(150, [note(A, 4, Sixteenth);]); - -pub const CHIME: Song<[(Option, TimeSlice); 2]> = song!(400, [ - note(B, 5, Eighth); - note(E, 6, Tie(&Dot(&Quarter), &Half)); -]); - -pub const SHUTDOWN: Song<[(Option, TimeSlice); 12]> = song!(200, [ - note(C, 5, Eighth); - rest(Eighth); - note(G, 4, Triplet(&Eighth)); - note(Gb, 4, Triplet(&Eighth)); - note(G, 4, Triplet(&Eighth)); - note(Ab, 4, Quarter); - note(G, 4, Eighth); - rest(Eighth); - rest(Quarter); - note(B, 4, Eighth); - rest(Eighth); - note(C, 5, Eighth); -]); - -pub const UPDATE_FAILED_1: Song<[(Option, TimeSlice); 5]> = song!(120, [ - note(C, 4, Triplet(&Sixteenth)); - note(Eb, 4, Triplet(&Sixteenth)); - note(Gb, 4, Triplet(&Sixteenth)); - note(A, 4, Quarter); - rest(Eighth); -]); -pub const UPDATE_FAILED_2: Song<[(Option, TimeSlice); 5]> = song!(110, [ - note(B, 3, Triplet(&Sixteenth)); - note(D, 4, Triplet(&Sixteenth)); - note(F, 4, Triplet(&Sixteenth)); - note(Ab, 4, Quarter); - rest(Eighth); -]); -pub const UPDATE_FAILED_3: Song<[(Option, TimeSlice); 5]> = song!(100, [ - note(Bb, 3, Triplet(&Sixteenth)); - note(Db, 4, Triplet(&Sixteenth)); - note(E, 4, Triplet(&Sixteenth)); - note(G, 4, Quarter); - rest(Eighth); -]); -pub const UPDATE_FAILED_4: Song<[(Option, TimeSlice); 5]> = song!(90, [ - note(A, 3, Triplet(&Sixteenth)); - note(C, 4, Triplet(&Sixteenth)); - note(Eb, 4, Triplet(&Sixteenth)); - note(Gb, 4, Tie(&Dot(&Quarter), &Quarter)); - rest(Quarter); -]); - -pub const BEETHOVEN: Song<[(Option, TimeSlice); 9]> = song!(216, [ - note(G, 4, Eighth); - note(G, 4, Eighth); - note(G, 4, Eighth); - note(Eb, 4, Half); - rest(Half); - note(F, 4, Eighth); - note(F, 4, Eighth); - note(F, 4, Eighth); - note(D, 4, Half); -]); - -lazy_static::lazy_static! { - pub static ref CIRCLE_OF_5THS_SHORT: Song>> = Song { - tempo_qpm: 300, - note_sequence: CircleOf::new( - &FIFTH, - Note { - semitone: Semitone::A, - octave: 3, - }, - TimeSlice::Triplet(&TimeSlice::Eighth), - ) - .take(6), - }; - pub static ref CIRCLE_OF_4THS_SHORT: Song>> = Song { - tempo_qpm: 300, - note_sequence: CircleOf::new( - &FOURTH, - Note { - semitone: Semitone::C, - octave: 4, - }, - TimeSlice::Triplet(&TimeSlice::Eighth) - ).take(6) - }; -} - -proptest::prop_compose! { - fn arb_interval() (i in -88isize..88isize) -> Interval { - Interval(i) - } -} -proptest::prop_compose! { - fn arb_note() (o in 0..8i8, s: Semitone) -> Note { - Note { - semitone: s, - octave: o, - } - } -} - -proptest::proptest! { - #[test] - fn positive_interval_greater(a in arb_note(), i in arb_interval()) { - proptest::prop_assume!(i > Interval(0)); - proptest::prop_assert!(interval(&i, &a) > a) - } - - #[test] - fn negative_interval_less(a in arb_note(), i in arb_interval()) { - proptest::prop_assume!(i < Interval(0)); - proptest::prop_assert!(interval(&i, &a) < a) - } - - #[test] - fn zero_interval_equal(a in arb_note()) { - proptest::prop_assert!(interval(&Interval(0), &a) == a) - } - - #[test] - fn positive_negative_cancellation(a in arb_note(), i in arb_interval()) { - let neg_i = match i { - Interval(n) => Interval(0-n) - }; - proptest::prop_assert_eq!(interval(&neg_i, &interval(&i, &a)), a) - } - - #[test] - fn freq_conversion_preserves_ordering(a in arb_note(), b in arb_note()) { - proptest::prop_assert_eq!(Some(a.cmp(&b)), a.frequency().partial_cmp(&b.frequency())) - } - -} diff --git a/core/startos/src/ssh.rs b/core/startos/src/ssh.rs deleted file mode 100644 index 697e0572..00000000 --- a/core/startos/src/ssh.rs +++ /dev/null @@ -1,197 +0,0 @@ -use std::path::Path; - -use chrono::Utc; -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use rpc_toolkit::command; -use sqlx::{Pool, Postgres}; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::util::display_none; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::{Error, ErrorKind}; - -static SSH_AUTHORIZED_KEYS_FILE: &str = "/home/start9/.ssh/authorized_keys"; - -#[derive(Debug, serde::Deserialize, serde::Serialize)] -pub struct PubKey( - #[serde(serialize_with = "crate::util::serde::serialize_display")] - #[serde(deserialize_with = "crate::util::serde::deserialize_from_str")] - openssh_keys::PublicKey, -); - -#[derive(serde::Serialize, serde::Deserialize)] -#[serde(rename_all = "kebab-case")] -pub struct SshKeyResponse { - pub alg: String, - pub fingerprint: String, - pub hostname: String, - pub created_at: String, -} -impl std::fmt::Display for SshKeyResponse { - fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { - write!( - f, - "{} {} {} {}", - self.created_at, self.alg, self.fingerprint, self.hostname - ) - } -} - -impl std::str::FromStr for PubKey { - type Err = Error; - fn from_str(s: &str) -> Result { - s.parse().map(|pk| PubKey(pk)).map_err(|e| Error { - source: e.into(), - kind: crate::ErrorKind::ParseSshKey, - revision: None, - }) - } -} - -#[command(subcommands(add, delete, list,))] -pub fn ssh() -> Result<(), Error> { - Ok(()) -} - -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn add(#[context] ctx: RpcContext, #[arg] key: PubKey) -> Result { - let pool = &ctx.secret_store; - // check fingerprint for duplicates - let fp = key.0.fingerprint_md5(); - match sqlx::query!("SELECT * FROM ssh_keys WHERE fingerprint = $1", fp) - .fetch_optional(pool) - .await? - { - None => { - // if no duplicates, insert into DB - let raw_key = format!("{}", key.0); - let created_at = Utc::now().to_rfc3339(); - sqlx::query!( - "INSERT INTO ssh_keys (fingerprint, openssh_pubkey, created_at) VALUES ($1, $2, $3)", - fp, - raw_key, - created_at - ) - .execute(pool) - .await?; - // insert into live key file, for now we actually do a wholesale replacement of the keys file, for maximum - // consistency - sync_keys_from_db(pool, Path::new(SSH_AUTHORIZED_KEYS_FILE)).await?; - Ok(SshKeyResponse { - alg: key.0.keytype().to_owned(), - fingerprint: fp, - hostname: key.0.comment.unwrap_or(String::new()).to_owned(), - created_at, - }) - } - Some(_) => Err(Error::new(eyre!("Duplicate ssh key"), ErrorKind::Duplicate)), - } -} -#[command(display(display_none))] -#[instrument(skip_all)] -pub async fn delete(#[context] ctx: RpcContext, #[arg] fingerprint: String) -> Result<(), Error> { - let pool = &ctx.secret_store; - // check if fingerprint is in DB - // if in DB, remove it from DB - let n = sqlx::query!("DELETE FROM ssh_keys WHERE fingerprint = $1", fingerprint) - .execute(pool) - .await? - .rows_affected(); - // if not in DB, Err404 - if n == 0 { - Err(Error { - source: color_eyre::eyre::eyre!("SSH Key Not Found"), - kind: crate::error::ErrorKind::NotFound, - revision: None, - }) - } else { - // AND overlay key file - sync_keys_from_db(pool, Path::new(SSH_AUTHORIZED_KEYS_FILE)).await?; - Ok(()) - } -} - -fn display_all_ssh_keys(all: Vec, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(all, matches); - } - - let mut table = Table::new(); - table.add_row(row![bc => - "CREATED AT", - "ALGORITHM", - "FINGERPRINT", - "HOSTNAME", - ]); - for key in all { - let row = row![ - &format!("{}", key.created_at), - &key.alg, - &key.fingerprint, - &key.hostname, - ]; - table.add_row(row); - } - table.print_tty(false).unwrap(); -} - -#[command(display(display_all_ssh_keys))] -#[instrument(skip_all)] -pub async fn list( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result, Error> { - let pool = &ctx.secret_store; - // list keys in DB and return them - let entries = sqlx::query!("SELECT fingerprint, openssh_pubkey, created_at FROM ssh_keys") - .fetch_all(pool) - .await?; - Ok(entries - .into_iter() - .map(|r| { - let k = PubKey(r.openssh_pubkey.parse().unwrap()).0; - let alg = k.keytype().to_owned(); - let fingerprint = k.fingerprint_md5(); - let hostname = k.comment.unwrap_or("".to_owned()); - let created_at = r.created_at; - SshKeyResponse { - alg, - fingerprint, - hostname, - created_at, - } - }) - .collect()) -} - -#[instrument(skip_all)] -pub async fn sync_keys_from_db>( - pool: &Pool, - dest: P, -) -> Result<(), Error> { - let dest = dest.as_ref(); - let keys = sqlx::query!("SELECT openssh_pubkey FROM ssh_keys") - .fetch_all(pool) - .await?; - let contents: String = keys - .into_iter() - .map(|k| format!("{}\n", k.openssh_pubkey)) - .collect(); - let ssh_dir = dest.parent().ok_or_else(|| { - Error::new( - eyre!("SSH Key File cannot be \"/\""), - crate::ErrorKind::Filesystem, - ) - })?; - if tokio::fs::metadata(ssh_dir).await.is_err() { - tokio::fs::create_dir_all(ssh_dir).await?; - } - std::fs::write(dest, contents).map_err(|e| e.into()) -} diff --git a/core/startos/src/status/health_check.rs b/core/startos/src/status/health_check.rs deleted file mode 100644 index 1b3e8f6b..00000000 --- a/core/startos/src/status/health_check.rs +++ /dev/null @@ -1,126 +0,0 @@ -use std::collections::{BTreeMap, BTreeSet}; - -use chrono::{DateTime, Utc}; -pub use models::HealthCheckId; -use models::ImageId; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::procedure::{NoOutput, PackageProcedure, ProcedureName}; -use crate::s9pk::manifest::PackageId; -use crate::util::serde::Duration; -use crate::util::Version; -use crate::volume::Volumes; -use crate::{Error, ResultExt}; - -#[derive(Clone, Debug, Deserialize, Serialize)] -pub struct HealthChecks(pub BTreeMap); -impl HealthChecks { - #[instrument(skip_all)] - pub fn validate( - &self, - eos_version: &Version, - volumes: &Volumes, - image_ids: &BTreeSet, - ) -> Result<(), Error> { - for check in self.0.values() { - check - .implementation - .validate(eos_version, volumes, image_ids, false) - .with_ctx(|_| { - ( - crate::ErrorKind::ValidateS9pk, - format!("Health Check {}", check.name), - ) - })?; - } - Ok(()) - } - pub async fn check_all( - &self, - ctx: &RpcContext, - started: DateTime, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - ) -> Result, Error> { - let res = futures::future::try_join_all(self.0.iter().map(|(id, check)| async move { - Ok::<_, Error>(( - id.clone(), - check - .check(ctx, id, started, pkg_id, pkg_version, volumes) - .await?, - )) - })) - .await?; - Ok(res.into_iter().collect()) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct HealthCheck { - pub name: String, - pub success_message: Option, - #[serde(flatten)] - implementation: PackageProcedure, - pub timeout: Option, -} -impl HealthCheck { - #[instrument(skip_all)] - pub async fn check( - &self, - ctx: &RpcContext, - id: &HealthCheckId, - started: DateTime, - pkg_id: &PackageId, - pkg_version: &Version, - volumes: &Volumes, - ) -> Result { - let res = self - .implementation - .execute( - ctx, - pkg_id, - pkg_version, - ProcedureName::Health(id.clone()), - volumes, - Some(Utc::now().signed_duration_since(started).num_milliseconds()), - Some( - self.timeout - .map_or(std::time::Duration::from_secs(30), |d| *d), - ), - ) - .await?; - Ok(match res { - Ok(NoOutput) => HealthCheckResult::Success, - Err((59, _)) => HealthCheckResult::Disabled, - Err((60, _)) => HealthCheckResult::Starting, - Err((61, message)) => HealthCheckResult::Loading { message }, - Err((_, error)) => HealthCheckResult::Failure { error }, - }) - } -} - -#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)] -#[serde(rename_all = "kebab-case")] -#[serde(tag = "result")] -pub enum HealthCheckResult { - Success, - Disabled, - Starting, - Loading { message: String }, - Failure { error: String }, -} -impl std::fmt::Display for HealthCheckResult { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - HealthCheckResult::Success => write!(f, "Succeeded"), - HealthCheckResult::Disabled => write!(f, "Disabled"), - HealthCheckResult::Starting => write!(f, "Starting"), - HealthCheckResult::Loading { message } => write!(f, "Loading ({})", message), - HealthCheckResult::Failure { error } => write!(f, "Failed ({})", error), - } - } -} diff --git a/core/startos/src/status/mod.rs b/core/startos/src/status/mod.rs deleted file mode 100644 index 2a5a9391..00000000 --- a/core/startos/src/status/mod.rs +++ /dev/null @@ -1,94 +0,0 @@ -use std::collections::BTreeMap; - -use chrono::{DateTime, Utc}; -use models::PackageId; -use serde::{Deserialize, Serialize}; - -use self::health_check::HealthCheckId; -use crate::prelude::*; -use crate::status::health_check::HealthCheckResult; - -pub mod health_check; -#[derive(Clone, Debug, Deserialize, Serialize, HasModel)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct Status { - pub configured: bool, - pub main: MainStatus, - #[serde(default)] - pub dependency_config_errors: DependencyConfigErrors, -} - -#[derive(Clone, Debug, Deserialize, Serialize, HasModel, Default)] -#[serde(rename_all = "kebab-case")] -#[model = "Model"] -pub struct DependencyConfigErrors(pub BTreeMap); -impl Map for DependencyConfigErrors { - type Key = PackageId; - type Value = String; -} - -#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] -#[serde(tag = "status")] -#[serde(rename_all = "kebab-case")] -pub enum MainStatus { - Stopped, - Restarting, - Stopping, - Starting, - Running { - started: DateTime, - health: BTreeMap, - }, - BackingUp { - started: Option>, - health: BTreeMap, - }, -} -impl MainStatus { - pub fn running(&self) -> bool { - match self { - MainStatus::Starting { .. } - | MainStatus::Running { .. } - | MainStatus::BackingUp { - started: Some(_), .. - } => true, - MainStatus::Stopped - | MainStatus::Stopping - | MainStatus::Restarting - | MainStatus::BackingUp { started: None, .. } => false, - } - } - pub fn stop(&mut self) { - match self { - MainStatus::Starting { .. } | MainStatus::Running { .. } => { - *self = MainStatus::Stopping; - } - MainStatus::BackingUp { started, .. } => { - *started = None; - } - MainStatus::Stopped | MainStatus::Stopping | MainStatus::Restarting => (), - } - } - pub fn started(&self) -> Option> { - match self { - MainStatus::Running { started, .. } => Some(*started), - MainStatus::BackingUp { started, .. } => *started, - MainStatus::Stopped => None, - MainStatus::Restarting => None, - MainStatus::Stopping => None, - MainStatus::Starting { .. } => None, - } - } - pub fn backing_up(&self) -> Self { - let (started, health) = match self { - MainStatus::Starting { .. } => (Some(Utc::now()), Default::default()), - MainStatus::Running { started, health } => (Some(started.clone()), health.clone()), - MainStatus::Stopped | MainStatus::Stopping | MainStatus::Restarting => { - (None, Default::default()) - } - MainStatus::BackingUp { .. } => return self.clone(), - }; - MainStatus::BackingUp { started, health } - } -} diff --git a/core/startos/src/system.rs b/core/startos/src/system.rs deleted file mode 100644 index b5cd4284..00000000 --- a/core/startos/src/system.rs +++ /dev/null @@ -1,915 +0,0 @@ -use std::collections::BTreeSet; -use std::fmt; - -use chrono::Utc; -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use futures::FutureExt; -use rpc_toolkit::command; -use rpc_toolkit::yajrc::RpcError; -use serde::{Deserialize, Deserializer, Serialize, Serializer}; -use tokio::process::Command; -use tokio::sync::broadcast::Receiver; -use tokio::sync::RwLock; -use tracing::instrument; - -use crate::context::{CliContext, RpcContext}; -use crate::disk::util::{get_available, get_used}; -use crate::logs::{ - cli_logs_generic_follow, cli_logs_generic_nofollow, fetch_logs, follow_logs, LogFollowResponse, - LogResponse, LogSource, -}; -use crate::prelude::*; -use crate::shutdown::Shutdown; -use crate::util::cpupower::{get_available_governors, set_governor, Governor}; -use crate::util::serde::{display_serializable, IoFormat}; -use crate::util::{display_none, Invoke}; -use crate::{Error, ErrorKind, ResultExt}; - -#[command(subcommands(zram, governor))] -pub async fn experimental() -> Result<(), Error> { - Ok(()) -} - -pub async fn enable_zram() -> Result<(), Error> { - let mem_info = get_mem_info().await?; - Command::new("modprobe") - .arg("zram") - .invoke(ErrorKind::Zram) - .await?; - tokio::fs::write("/sys/block/zram0/comp_algorithm", "lz4") - .await - .with_kind(ErrorKind::Zram)?; - tokio::fs::write( - "/sys/block/zram0/disksize", - format!("{}M", mem_info.total.0 as u64 / 4), - ) - .await - .with_kind(ErrorKind::Zram)?; - Command::new("mkswap") - .arg("/dev/zram0") - .invoke(ErrorKind::Zram) - .await?; - Command::new("swapon") - .arg("-p") - .arg("5") - .arg("/dev/zram0") - .invoke(ErrorKind::Zram) - .await?; - Ok(()) -} - -#[command(display(display_none))] -pub async fn zram(#[context] ctx: RpcContext, #[arg] enable: bool) -> Result<(), Error> { - let db = ctx.db.peek().await; - - let zram = db.as_server_info().as_zram().de()?; - if enable == zram { - return Ok(()); - } - if enable { - enable_zram().await?; - } else { - Command::new("swapoff") - .arg("/dev/zram0") - .invoke(ErrorKind::Zram) - .await?; - tokio::fs::write("/sys/block/zram0/reset", "1") - .await - .with_kind(ErrorKind::Zram)?; - } - ctx.db - .mutate(|v| { - v.as_server_info_mut().as_zram_mut().ser(&enable)?; - Ok(()) - }) - .await?; - Ok(()) -} - -#[derive(Debug, Deserialize, Serialize)] -pub struct GovernorInfo { - current: Option, - available: BTreeSet, -} - -fn display_governor_info(arg: GovernorInfo, matches: &ArgMatches) { - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(arg, matches); - } - - let mut table = Table::new(); - table.add_row(row![bc -> "GOVERNORS"]); - for entry in arg.available { - if Some(&entry) == arg.current.as_ref() { - table.add_row(row![g -> format!("* {entry} (current)")]); - } else { - table.add_row(row![entry]); - } - } - table.print_tty(false).unwrap(); -} - -#[command(display(display_governor_info))] -pub async fn governor( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, - #[arg] set: Option, -) -> Result { - let available = get_available_governors().await?; - if let Some(set) = set { - if !available.contains(&set) { - return Err(Error::new( - eyre!("Governor {set} not available"), - ErrorKind::InvalidRequest, - )); - } - set_governor(&set).await?; - ctx.db - .mutate(|d| d.as_server_info_mut().as_governor_mut().ser(&Some(set))) - .await?; - } - let current = ctx.db.peek().await.as_server_info().as_governor().de()?; - Ok(GovernorInfo { current, available }) -} - -#[derive(Serialize, Deserialize)] -pub struct TimeInfo { - now: String, - uptime: u64, -} - -fn display_time(arg: TimeInfo, matches: &ArgMatches) { - use std::fmt::Write; - - use prettytable::*; - - if matches.is_present("format") { - return display_serializable(arg, matches); - } - - let days = arg.uptime / (24 * 60 * 60); - let days_s = arg.uptime % (24 * 60 * 60); - let hours = days_s / (60 * 60); - let hours_s = arg.uptime % (60 * 60); - let minutes = hours_s / 60; - let seconds = arg.uptime % 60; - let mut uptime_string = String::new(); - if days > 0 { - write!(&mut uptime_string, "{days} days").unwrap(); - } - if hours > 0 { - if !uptime_string.is_empty() { - uptime_string += ", "; - } - write!(&mut uptime_string, "{hours} hours").unwrap(); - } - if minutes > 0 { - if !uptime_string.is_empty() { - uptime_string += ", "; - } - write!(&mut uptime_string, "{minutes} minutes").unwrap(); - } - if !uptime_string.is_empty() { - uptime_string += ", "; - } - write!(&mut uptime_string, "{seconds} seconds").unwrap(); - - let mut table = Table::new(); - table.add_row(row![bc -> "NOW", &arg.now]); - table.add_row(row![bc -> "UPTIME", &uptime_string]); - table.print_tty(false).unwrap(); -} - -#[command(display(display_time))] -pub async fn time( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - Ok(TimeInfo { - now: Utc::now().to_rfc3339(), - uptime: ctx.start_time.elapsed().as_secs(), - }) -} - -#[command( - custom_cli(cli_logs(async, context(CliContext))), - subcommands(self(logs_nofollow(async)), logs_follow), - display(display_none) -)] -pub async fn logs( - #[arg(short = 'l', long = "limit")] limit: Option, - #[arg(short = 'c', long = "cursor")] cursor: Option, - #[arg(short = 'B', long = "before", default)] before: bool, - #[arg(short = 'f', long = "follow", default)] follow: bool, -) -> Result<(Option, Option, bool, bool), Error> { - Ok((limit, cursor, before, follow)) -} -pub async fn cli_logs( - ctx: CliContext, - (limit, cursor, before, follow): (Option, Option, bool, bool), -) -> Result<(), RpcError> { - if follow { - if cursor.is_some() { - return Err(RpcError::from(Error::new( - eyre!("The argument '--cursor ' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - if before { - return Err(RpcError::from(Error::new( - eyre!("The argument '--before' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - cli_logs_generic_follow(ctx, "server.logs.follow", None, limit).await - } else { - cli_logs_generic_nofollow(ctx, "server.logs", None, limit, cursor, before).await - } -} -pub async fn logs_nofollow( - _ctx: (), - (limit, cursor, before, _): (Option, Option, bool, bool), -) -> Result { - fetch_logs(LogSource::System, limit, cursor, before).await -} - -#[command(rpc_only, rename = "follow", display(display_none))] -pub async fn logs_follow( - #[context] ctx: RpcContext, - #[parent_data] (limit, _, _, _): (Option, Option, bool, bool), -) -> Result { - follow_logs(ctx, LogSource::System, limit).await -} - -#[command( - rename = "kernel-logs", - custom_cli(cli_kernel_logs(async, context(CliContext))), - subcommands(self(kernel_logs_nofollow(async)), kernel_logs_follow), - display(display_none) -)] -pub async fn kernel_logs( - #[arg(short = 'l', long = "limit")] limit: Option, - #[arg(short = 'c', long = "cursor")] cursor: Option, - #[arg(short = 'B', long = "before", default)] before: bool, - #[arg(short = 'f', long = "follow", default)] follow: bool, -) -> Result<(Option, Option, bool, bool), Error> { - Ok((limit, cursor, before, follow)) -} -pub async fn cli_kernel_logs( - ctx: CliContext, - (limit, cursor, before, follow): (Option, Option, bool, bool), -) -> Result<(), RpcError> { - if follow { - if cursor.is_some() { - return Err(RpcError::from(Error::new( - eyre!("The argument '--cursor ' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - if before { - return Err(RpcError::from(Error::new( - eyre!("The argument '--before' cannot be used with '--follow'"), - crate::ErrorKind::InvalidRequest, - ))); - } - cli_logs_generic_follow(ctx, "server.kernel-logs.follow", None, limit).await - } else { - cli_logs_generic_nofollow(ctx, "server.kernel-logs", None, limit, cursor, before).await - } -} -pub async fn kernel_logs_nofollow( - _ctx: (), - (limit, cursor, before, _): (Option, Option, bool, bool), -) -> Result { - fetch_logs(LogSource::Kernel, limit, cursor, before).await -} - -#[command(rpc_only, rename = "follow", display(display_none))] -pub async fn kernel_logs_follow( - #[context] ctx: RpcContext, - #[parent_data] (limit, _, _, _): (Option, Option, bool, bool), -) -> Result { - follow_logs(ctx, LogSource::Kernel, limit).await -} - -#[derive(Serialize, Deserialize)] -pub struct MetricLeaf { - value: T, - unit: Option, -} - -#[derive(Clone, Debug)] -pub struct Celsius(f64); -impl fmt::Display for Celsius { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{:.1}°C", self.0) - } -} -impl Serialize for Celsius { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - MetricLeaf { - value: format!("{:.1}", self.0), - unit: Some(String::from("°C")), - } - .serialize(serializer) - } -} -impl<'de> Deserialize<'de> for Celsius { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = MetricLeaf::::deserialize(deserializer)?; - Ok(Celsius(s.value.parse().map_err(serde::de::Error::custom)?)) - } -} -#[derive(Clone, Debug)] -pub struct Percentage(f64); -impl Serialize for Percentage { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - MetricLeaf { - value: format!("{:.1}", self.0), - unit: Some(String::from("%")), - } - .serialize(serializer) - } -} -impl<'de> Deserialize<'de> for Percentage { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = MetricLeaf::::deserialize(deserializer)?; - Ok(Percentage( - s.value.parse().map_err(serde::de::Error::custom)?, - )) - } -} - -#[derive(Clone, Debug)] -pub struct MebiBytes(pub f64); -impl Serialize for MebiBytes { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - MetricLeaf { - value: format!("{:.2}", self.0), - unit: Some(String::from("MiB")), - } - .serialize(serializer) - } -} -impl<'de> Deserialize<'de> for MebiBytes { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = MetricLeaf::::deserialize(deserializer)?; - Ok(MebiBytes( - s.value.parse().map_err(serde::de::Error::custom)?, - )) - } -} - -#[derive(Clone, Debug)] -pub struct GigaBytes(f64); -impl Serialize for GigaBytes { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - MetricLeaf { - value: format!("{:.2}", self.0), - unit: Some(String::from("GB")), - } - .serialize(serializer) - } -} -impl<'de> Deserialize<'de> for GigaBytes { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = MetricLeaf::::deserialize(deserializer)?; - Ok(GigaBytes( - s.value.parse().map_err(serde::de::Error::custom)?, - )) - } -} - -#[derive(Deserialize, Serialize, Clone, Debug)] -#[serde(rename_all = "kebab-case")] -pub struct MetricsGeneral { - pub temperature: Option, -} -#[derive(Deserialize, Serialize, Clone, Debug)] -#[serde(rename_all = "kebab-case")] -pub struct MetricsMemory { - pub percentage_used: Percentage, - pub total: MebiBytes, - pub available: MebiBytes, - pub used: MebiBytes, - pub zram_total: MebiBytes, - pub zram_available: MebiBytes, - pub zram_used: MebiBytes, -} -#[derive(Deserialize, Serialize, Clone, Debug)] -#[serde(rename_all = "kebab-case")] -pub struct MetricsCpu { - percentage_used: Percentage, - idle: Percentage, - user_space: Percentage, - kernel_space: Percentage, - wait: Percentage, -} -#[derive(Deserialize, Serialize, Clone, Debug)] -#[serde(rename_all = "kebab-case")] -pub struct MetricsDisk { - percentage_used: Percentage, - used: GigaBytes, - available: GigaBytes, - capacity: GigaBytes, -} -#[derive(Deserialize, Serialize, Clone, Debug)] -#[serde(rename_all = "kebab-case")] -pub struct Metrics { - general: MetricsGeneral, - memory: MetricsMemory, - cpu: MetricsCpu, - disk: MetricsDisk, -} - -#[command(display(display_serializable))] -pub async fn metrics( - #[context] ctx: RpcContext, - #[allow(unused_variables)] - #[arg(long = "format")] - format: Option, -) -> Result { - match ctx.metrics_cache.read().await.clone() { - None => Err(Error { - source: color_eyre::eyre::eyre!("No Metrics Found"), - kind: ErrorKind::NotFound, - revision: None, - }), - Some(metrics_val) => Ok(metrics_val), - } -} - -pub async fn launch_metrics_task Receiver>>( - cache: &RwLock>, - mut mk_shutdown: F, -) { - // fetch init temp - let init_temp = match get_temp().await { - Ok(a) => Some(a), - Err(e) => { - tracing::error!("Could not get initial temperature: {}", e); - tracing::debug!("{:?}", e); - None - } - }; - // fetch init cpu - let init_cpu; - let proc_stat; - loop { - match get_proc_stat().await { - Ok(mut ps) => match get_cpu_info(&mut ps).await { - Ok(mc) => { - proc_stat = ps; - init_cpu = mc; - break; - } - Err(e) => { - tracing::error!("Could not get initial cpu info: {}", e); - tracing::debug!("{:?}", e); - } - }, - Err(e) => { - tracing::error!("Could not get initial proc stat: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::time::sleep(tokio::time::Duration::from_secs(1)).await; - } - // fetch init memory - let init_mem; - loop { - match get_mem_info().await { - Ok(a) => { - init_mem = a; - break; - } - Err(e) => { - tracing::error!("Could not get initial mem info: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::time::sleep(tokio::time::Duration::from_secs(1)).await; - } - // fetch init disk usage - let init_disk; - loop { - match get_disk_info().await { - Ok(a) => { - init_disk = a; - break; - } - Err(e) => { - tracing::error!("Could not get initial disk info: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::time::sleep(tokio::time::Duration::from_secs(1)).await; - } - { - // lock for writing - let mut guard = cache.write().await; - // write - *guard = Some(Metrics { - general: MetricsGeneral { - temperature: init_temp, - }, - memory: init_mem, - cpu: init_cpu, - disk: init_disk, - }) - } - - let mut task_vec = Vec::new(); - // launch persistent temp task - if cache - .read() - .await - .as_ref() - .unwrap() - .general - .temperature - .is_some() - { - task_vec.push(launch_temp_task(cache, mk_shutdown()).boxed()); - } - // launch persistent cpu task - task_vec.push(launch_cpu_task(cache, proc_stat, mk_shutdown()).boxed()); - // launch persistent mem task - task_vec.push(launch_mem_task(cache, mk_shutdown()).boxed()); - // launch persistent disk task - task_vec.push(launch_disk_task(cache, mk_shutdown()).boxed()); - - futures::future::join_all(task_vec).await; -} - -async fn launch_temp_task( - cache: &RwLock>, - mut shutdown: Receiver>, -) { - loop { - match get_temp().await { - Ok(a) => { - let mut lock = cache.write().await; - (*lock).as_mut().unwrap().general.temperature = Some(a) - } - Err(e) => { - tracing::error!("Could not get new temperature: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::select! { - _ = shutdown.recv() => return, - _ = tokio::time::sleep(tokio::time::Duration::from_secs(4)) => (), - } - } -} - -async fn launch_cpu_task( - cache: &RwLock>, - mut init: ProcStat, - mut shutdown: Receiver>, -) { - loop { - // read /proc/stat, diff against previous metrics, compute cpu load - match get_cpu_info(&mut init).await { - Ok(info) => { - let mut lock = cache.write().await; - (*lock).as_mut().unwrap().cpu = info; - } - Err(e) => { - tracing::error!("Could not get new CPU Metrics: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::select! { - _ = shutdown.recv() => return, - _ = tokio::time::sleep(tokio::time::Duration::from_secs(4)) => (), - } - } -} - -async fn launch_mem_task( - cache: &RwLock>, - mut shutdown: Receiver>, -) { - loop { - // read /proc/meminfo - match get_mem_info().await { - Ok(a) => { - let mut lock = cache.write().await; - (*lock).as_mut().unwrap().memory = a; - } - Err(e) => { - tracing::error!("Could not get new Memory Metrics: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::select! { - _ = shutdown.recv() => return, - _ = tokio::time::sleep(tokio::time::Duration::from_secs(4)) => (), - } - } -} -async fn launch_disk_task( - cache: &RwLock>, - mut shutdown: Receiver>, -) { - loop { - // run df and capture output - match get_disk_info().await { - Ok(a) => { - let mut lock = cache.write().await; - (*lock).as_mut().unwrap().disk = a; - } - Err(e) => { - tracing::error!("Could not get new Disk Metrics: {}", e); - tracing::debug!("{:?}", e); - } - } - tokio::select! { - _ = shutdown.recv() => return, - _ = tokio::time::sleep(tokio::time::Duration::from_secs(60)) => (), - } - } -} - -#[instrument(skip_all)] -async fn get_temp() -> Result { - let temp = serde_json::from_slice::( - &Command::new("sensors") - .arg("-j") - .invoke(ErrorKind::Filesystem) - .await?, - ) - .with_kind(ErrorKind::Deserialization)? - .as_object() - .into_iter() - .flatten() - .flat_map(|(_, v)| v.as_object()) - .flatten() - .flat_map(|(_, v)| v.as_object()) - .flatten() - .filter_map(|(k, v)| { - // we have seen so far that `temp1` is always a composite reading of some sort, so we should just use that for each chip - if k.trim() == "temp1_input" { - v.as_f64() - } else { - None - } - }) - .reduce(f64::max) - .ok_or_else(|| Error::new(eyre!("No temperatures available"), ErrorKind::Filesystem))?; - Ok(Celsius(temp)) -} - -#[derive(Debug, Clone)] -pub struct ProcStat { - user: u64, - nice: u64, - system: u64, - idle: u64, - iowait: u64, - irq: u64, - softirq: u64, - // below are only applicable to virtualized environments - // steal: u64, - // guest: u64, - // guest_nice: u64, -} -impl ProcStat { - fn total(&self) -> u64 { - self.user + self.nice + self.system + self.idle + self.iowait + self.irq + self.softirq - } - fn user(&self) -> u64 { - self.user + self.nice - } - fn system(&self) -> u64 { - self.system + self.irq + self.softirq - } - fn used(&self) -> u64 { - self.user() + self.system() - } -} - -#[instrument(skip_all)] -async fn get_proc_stat() -> Result { - use tokio::io::AsyncBufReadExt; - let mut cpu_line = String::new(); - let _n = tokio::io::BufReader::new(tokio::fs::File::open("/proc/stat").await?) - .read_line(&mut cpu_line) - .await?; - let stats: Vec = cpu_line - .split_whitespace() - .skip(1) - .map(|s| { - s.parse::().map_err(|e| { - Error::new( - color_eyre::eyre::eyre!("Invalid /proc/stat column value: {}", e), - ErrorKind::ParseSysInfo, - ) - }) - }) - .collect::, Error>>()?; - - if stats.len() < 10 { - Err(Error { - source: color_eyre::eyre::eyre!( - "Columns missing from /proc/stat. Need 10, found {}", - stats.len() - ), - kind: ErrorKind::ParseSysInfo, - revision: None, - }) - } else { - Ok(ProcStat { - user: stats[0], - nice: stats[1], - system: stats[2], - idle: stats[3], - iowait: stats[4], - irq: stats[5], - softirq: stats[6], - }) - } -} - -#[instrument(skip_all)] -async fn get_cpu_info(last: &mut ProcStat) -> Result { - let new = get_proc_stat().await?; - let total_old = last.total(); - let total_new = new.total(); - let total_diff = total_new - total_old; - let res = MetricsCpu { - user_space: Percentage((new.user() - last.user()) as f64 * 100.0 / total_diff as f64), - kernel_space: Percentage((new.system() - last.system()) as f64 * 100.0 / total_diff as f64), - idle: Percentage((new.idle - last.idle) as f64 * 100.0 / total_diff as f64), - wait: Percentage((new.iowait - last.iowait) as f64 * 100.0 / total_diff as f64), - percentage_used: Percentage((new.used() - last.used()) as f64 * 100.0 / total_diff as f64), - }; - *last = new; - Ok(res) -} - -pub struct MemInfo { - mem_total: Option, - mem_free: Option, - mem_available: Option, - buffers: Option, - cached: Option, - slab: Option, - zram_total: Option, - zram_free: Option, -} -#[instrument(skip_all)] -pub async fn get_mem_info() -> Result { - let contents = tokio::fs::read_to_string("/proc/meminfo").await?; - let mut mem_info = MemInfo { - mem_total: None, - mem_free: None, - mem_available: None, - buffers: None, - cached: None, - slab: None, - zram_total: None, - zram_free: None, - }; - fn get_num_kb(l: &str) -> Result { - let e = Error::new( - color_eyre::eyre::eyre!("Invalid meminfo line: {}", l), - ErrorKind::ParseSysInfo, - ); - match l.split_whitespace().skip(1).next() { - Some(x) => match x.parse() { - Ok(y) => Ok(y), - Err(_) => Err(e), - }, - None => Err(e), - } - } - for entry in contents.lines() { - match entry { - _ if entry.starts_with("MemTotal") => mem_info.mem_total = Some(get_num_kb(entry)?), - _ if entry.starts_with("MemFree") => mem_info.mem_free = Some(get_num_kb(entry)?), - _ if entry.starts_with("MemAvailable") => { - mem_info.mem_available = Some(get_num_kb(entry)?) - } - _ if entry.starts_with("Buffers") => mem_info.buffers = Some(get_num_kb(entry)?), - _ if entry.starts_with("Cached") => mem_info.cached = Some(get_num_kb(entry)?), - _ if entry.starts_with("Slab") => mem_info.slab = Some(get_num_kb(entry)?), - _ if entry.starts_with("SwapTotal") => mem_info.zram_total = Some(get_num_kb(entry)?), - _ if entry.starts_with("SwapFree") => mem_info.zram_free = Some(get_num_kb(entry)?), - _ => (), - } - } - fn ensure_present(a: Option, field: &str) -> Result { - a.ok_or(Error::new( - color_eyre::eyre::eyre!("{} missing from /proc/meminfo", field), - ErrorKind::ParseSysInfo, - )) - } - let mem_total = ensure_present(mem_info.mem_total, "MemTotal")?; - let mem_free = ensure_present(mem_info.mem_free, "MemFree")?; - let mem_available = ensure_present(mem_info.mem_available, "MemAvailable")?; - let buffers = ensure_present(mem_info.buffers, "Buffers")?; - let cached = ensure_present(mem_info.cached, "Cached")?; - let slab = ensure_present(mem_info.slab, "Slab")?; - let zram_total_k = ensure_present(mem_info.zram_total, "SwapTotal")?; - let zram_free_k = ensure_present(mem_info.zram_free, "SwapFree")?; - - let total = MebiBytes(mem_total as f64 / 1024.0); - let available = MebiBytes(mem_available as f64 / 1024.0); - let used = MebiBytes((mem_total - mem_free - buffers - cached - slab) as f64 / 1024.0); - let zram_total = MebiBytes(zram_total_k as f64 / 1024.0); - let zram_available = MebiBytes(zram_free_k as f64 / 1024.0); - let zram_used = MebiBytes((zram_total_k - zram_free_k) as f64 / 1024.0); - let percentage_used = Percentage((total.0 - available.0) / total.0 * 100.0); - Ok(MetricsMemory { - percentage_used, - total, - available, - used, - zram_total, - zram_available, - zram_used, - }) -} - -#[instrument(skip_all)] -async fn get_disk_info() -> Result { - let package_used_task = get_used("/embassy-data/package-data"); - let package_available_task = get_available("/embassy-data/package-data"); - let os_used_task = get_used("/embassy-data/main"); - let os_available_task = get_available("/embassy-data/main"); - - let (package_used, package_available, os_used, os_available) = futures::try_join!( - package_used_task, - package_available_task, - os_used_task, - os_available_task, - )?; - - let total_used = package_used + os_used; - let total_available = package_available + os_available; - let total_size = total_used + total_available; - let total_percentage = total_used as f64 / total_size as f64 * 100.0f64; - - Ok(MetricsDisk { - capacity: GigaBytes(total_size as f64 / 1_000_000_000.0), - used: GigaBytes(total_used as f64 / 1_000_000_000.0), - available: GigaBytes(total_available as f64 / 1_000_000_000.0), - percentage_used: Percentage(total_percentage as f64), - }) -} - -#[tokio::test] -#[ignore] -pub async fn test_get_temp() { - println!("{}", get_temp().await.unwrap()) -} - -#[tokio::test] -pub async fn test_get_proc_stat() { - println!("{:?}", get_proc_stat().await.unwrap()) -} - -#[tokio::test] -pub async fn test_get_mem_info() { - println!("{:?}", get_mem_info().await.unwrap()) -} - -#[tokio::test] -#[ignore] -pub async fn test_get_disk_usage() { - println!("{:?}", get_disk_info().await.unwrap()) -} diff --git a/core/startos/src/update/latest_information.rs b/core/startos/src/update/latest_information.rs deleted file mode 100644 index e897dd40..00000000 --- a/core/startos/src/update/latest_information.rs +++ /dev/null @@ -1,23 +0,0 @@ -use std::collections::HashMap; - -use emver::Version; -use serde::{Deserialize, Serialize}; -use serde_with::{serde_as, DisplayFromStr}; - -#[serde_as] -#[derive(Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub struct LatestInformation { - release_notes: HashMap, - headline: String, - #[serde_as(as = "DisplayFromStr")] - pub version: Version, -} - -/// Captured from https://beta-registry-0-3.start9labs.com/eos/latest 2021-09-24 -#[test] -fn latest_information_from_server() { - let data_from_server = r#"{"release-notes":{"0.3.0":"This major software release encapsulates the optimal performance, security, and management enhancments to the embassyOS experience."},"headline":"Major embassyOS release","version":"0.3.0"}"#; - let latest_information: LatestInformation = serde_json::from_str(data_from_server).unwrap(); - assert_eq!(latest_information.version.minor(), 3); -} diff --git a/core/startos/src/update/mod.rs b/core/startos/src/update/mod.rs deleted file mode 100644 index 4ce57a8d..00000000 --- a/core/startos/src/update/mod.rs +++ /dev/null @@ -1,326 +0,0 @@ -use std::path::PathBuf; -use std::sync::atomic::{AtomicBool, Ordering}; - -use clap::ArgMatches; -use color_eyre::eyre::{eyre, Result}; -use emver::Version; -use helpers::{Rsync, RsyncOptions}; -use lazy_static::lazy_static; -use reqwest::Url; -use rpc_toolkit::command; -use tokio::process::Command; -use tokio_stream::StreamExt; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::db::model::UpdateProgress; -use crate::disk::mount::filesystem::bind::Bind; -use crate::disk::mount::filesystem::ReadWrite; -use crate::disk::mount::guard::MountGuard; -use crate::notifications::NotificationLevel; -use crate::prelude::*; -use crate::registry::marketplace::with_query_params; -use crate::sound::{ - CIRCLE_OF_5THS_SHORT, UPDATE_FAILED_1, UPDATE_FAILED_2, UPDATE_FAILED_3, UPDATE_FAILED_4, -}; -use crate::update::latest_information::LatestInformation; -use crate::util::Invoke; -use crate::{Error, ErrorKind, ResultExt, PLATFORM}; - -mod latest_information; - -lazy_static! { - static ref UPDATED: AtomicBool = AtomicBool::new(false); -} - -/// An user/ daemon would call this to update the system to the latest version and do the updates available, -/// and this will return something if there is an update, and in that case there will need to be a restart. -#[command( - rename = "update", - display(display_update_result), - metadata(sync_db = true) -)] -#[instrument(skip_all)] -pub async fn update_system( - #[context] ctx: RpcContext, - #[arg(rename = "marketplace-url")] marketplace_url: Url, -) -> Result { - if UPDATED.load(Ordering::SeqCst) { - return Ok(UpdateResult::NoUpdates); - } - Ok(if maybe_do_update(ctx, marketplace_url).await?.is_some() { - UpdateResult::Updating - } else { - UpdateResult::NoUpdates - }) -} - -/// What is the status of the updates? -#[derive(serde::Serialize, serde::Deserialize, Clone, Debug)] -#[serde(rename_all = "kebab-case")] -pub enum UpdateResult { - NoUpdates, - Updating, -} - -fn display_update_result(status: UpdateResult, _: &ArgMatches) { - match status { - UpdateResult::Updating => { - println!("Updating..."); - } - UpdateResult::NoUpdates => { - println!("No updates available"); - } - } -} - -#[instrument(skip_all)] -async fn maybe_do_update(ctx: RpcContext, marketplace_url: Url) -> Result, Error> { - let peeked = ctx.db.peek().await; - let latest_version: Version = ctx - .client - .get(with_query_params( - ctx.clone(), - format!("{}/eos/v0/latest", marketplace_url,).parse()?, - )) - .send() - .await - .with_kind(ErrorKind::Network)? - .json::() - .await - .with_kind(ErrorKind::Network)? - .version; - let current_version = peeked.as_server_info().as_version().de()?; - if latest_version < *current_version { - return Ok(None); - } - - let eos_url = EosUrl { - base: marketplace_url, - version: latest_version, - }; - let status = ctx - .db - .mutate(|db| { - let mut status = peeked.as_server_info().as_status_info().de()?; - if status.update_progress.is_some() { - return Err(Error::new( - eyre!("Server is already updating!"), - crate::ErrorKind::InvalidRequest, - )); - } - - status.update_progress = Some(UpdateProgress { - size: None, - downloaded: 0, - }); - db.as_server_info_mut().as_status_info_mut().ser(&status)?; - Ok(status) - }) - .await?; - - if status.updated { - return Ok(None); - } - - tokio::spawn(async move { - let res = do_update(ctx.clone(), eos_url).await; - ctx.db - .mutate(|db| { - db.as_server_info_mut() - .as_status_info_mut() - .as_update_progress_mut() - .ser(&None) - }) - .await?; - match res { - Ok(()) => { - ctx.db - .mutate(|db| { - db.as_server_info_mut() - .as_status_info_mut() - .as_updated_mut() - .ser(&true) - }) - .await?; - CIRCLE_OF_5THS_SHORT - .play() - .await - .expect("could not play sound"); - } - Err(e) => { - ctx.notification_manager - .notify( - ctx.db.clone(), - None, - NotificationLevel::Error, - "StartOS Update Failed".to_owned(), - format!("Update was not successful because of {}", e), - (), - None, - ) - .await - .expect(""); - // TODO: refactor sound lib to make compound tempos easier to deal with - UPDATE_FAILED_1 - .play() - .await - .expect("could not play song: update failed 1"); - UPDATE_FAILED_2 - .play() - .await - .expect("could not play song: update failed 2"); - UPDATE_FAILED_3 - .play() - .await - .expect("could not play song: update failed 3"); - UPDATE_FAILED_4 - .play() - .await - .expect("could not play song: update failed 4"); - } - } - Ok::<(), Error>(()) - }); - Ok(Some(())) -} - -#[instrument(skip_all)] -async fn do_update(ctx: RpcContext, eos_url: EosUrl) -> Result<(), Error> { - let mut rsync = Rsync::new( - eos_url.rsync_path()?, - "/media/embassy/next/", - Default::default(), - ) - .await?; - while let Some(progress) = rsync.progress.next().await { - ctx.db - .mutate(|db| { - db.as_server_info_mut() - .as_status_info_mut() - .as_update_progress_mut() - .ser(&Some(UpdateProgress { - size: Some(100), - downloaded: (100.0 * progress) as u64, - })) - }) - .await?; - } - rsync.wait().await?; - - copy_fstab().await?; - copy_machine_id().await?; - copy_ssh_host_keys().await?; - sync_boot().await?; - swap_boot_label().await?; - - Ok(()) -} - -#[derive(Debug)] -struct EosUrl { - base: Url, - version: Version, -} - -impl EosUrl { - #[instrument()] - pub fn rsync_path(&self) -> Result { - let host = self - .base - .host_str() - .ok_or_else(|| Error::new(eyre!("Could not get host of base"), ErrorKind::ParseUrl))?; - let version: &Version = &self.version; - Ok(format!("{host}::{version}/{}/", &*PLATFORM) - .parse() - .map_err(|_| Error::new(eyre!("Could not parse path"), ErrorKind::ParseUrl))?) - } -} - -async fn copy_fstab() -> Result<(), Error> { - tokio::fs::copy("/etc/fstab", "/media/embassy/next/etc/fstab").await?; - Ok(()) -} - -async fn copy_machine_id() -> Result<(), Error> { - tokio::fs::copy("/etc/machine-id", "/media/embassy/next/etc/machine-id").await?; - Ok(()) -} - -async fn copy_ssh_host_keys() -> Result<(), Error> { - tokio::fs::copy( - "/etc/ssh/ssh_host_rsa_key", - "/media/embassy/next/etc/ssh/ssh_host_rsa_key", - ) - .await?; - tokio::fs::copy( - "/etc/ssh/ssh_host_rsa_key.pub", - "/media/embassy/next/etc/ssh/ssh_host_rsa_key.pub", - ) - .await?; - tokio::fs::copy( - "/etc/ssh/ssh_host_ecdsa_key", - "/media/embassy/next/etc/ssh/ssh_host_ecdsa_key", - ) - .await?; - tokio::fs::copy( - "/etc/ssh/ssh_host_ecdsa_key.pub", - "/media/embassy/next/etc/ssh/ssh_host_ecdsa_key.pub", - ) - .await?; - tokio::fs::copy( - "/etc/ssh/ssh_host_ed25519_key", - "/media/embassy/next/etc/ssh/ssh_host_ed25519_key", - ) - .await?; - tokio::fs::copy( - "/etc/ssh/ssh_host_ed25519_key.pub", - "/media/embassy/next/etc/ssh/ssh_host_ed25519_key.pub", - ) - .await?; - Ok(()) -} - -async fn sync_boot() -> Result<(), Error> { - Rsync::new( - "/media/embassy/next/boot/", - "/boot/", - RsyncOptions { - delete: false, - force: false, - ignore_existing: false, - exclude: Vec::new(), - no_permissions: false, - no_owner: false, - }, - ) - .await? - .wait() - .await?; - if &*PLATFORM != "raspberrypi" { - let dev_mnt = - MountGuard::mount(&Bind::new("/dev"), "/media/embassy/next/dev", ReadWrite).await?; - let sys_mnt = - MountGuard::mount(&Bind::new("/sys"), "/media/embassy/next/sys", ReadWrite).await?; - let proc_mnt = - MountGuard::mount(&Bind::new("/proc"), "/media/embassy/next/proc", ReadWrite).await?; - let boot_mnt = - MountGuard::mount(&Bind::new("/boot"), "/media/embassy/next/boot", ReadWrite).await?; - Command::new("chroot") - .arg("/media/embassy/next") - .arg("update-grub2") - .invoke(ErrorKind::MigrationFailed) - .await?; - boot_mnt.unmount(false).await?; - proc_mnt.unmount(false).await?; - sys_mnt.unmount(false).await?; - dev_mnt.unmount(false).await?; - } - Ok(()) -} - -#[instrument(skip_all)] -async fn swap_boot_label() -> Result<(), Error> { - tokio::fs::write("/media/embassy/config/upgrade", b"").await?; - Ok(()) -} diff --git a/core/startos/src/util/config.rs b/core/startos/src/util/config.rs deleted file mode 100644 index f719f563..00000000 --- a/core/startos/src/util/config.rs +++ /dev/null @@ -1,58 +0,0 @@ -use std::fs::File; -use std::path::{Path, PathBuf}; - -use patch_db::Value; -use serde::Deserialize; - -use crate::prelude::*; -use crate::util::serde::IoFormat; -use crate::{Config, Error}; - -pub const DEVICE_CONFIG_PATH: &str = "/media/embassy/config/config.yaml"; -pub const CONFIG_PATH: &str = "/etc/embassy/config.yaml"; -pub const CONFIG_PATH_LOCAL: &str = ".embassy/config.yaml"; - -pub fn local_config_path() -> Option { - if let Ok(home) = std::env::var("HOME") { - Some(Path::new(&home).join(CONFIG_PATH_LOCAL)) - } else { - None - } -} - -/// BLOCKING -pub fn load_config_from_paths<'a, T: for<'de> Deserialize<'de>>( - paths: impl IntoIterator>, -) -> Result { - let mut config = Default::default(); - for path in paths { - if path.as_ref().exists() { - let format: IoFormat = path - .as_ref() - .extension() - .and_then(|s| s.to_str()) - .map(|f| f.parse()) - .transpose()? - .unwrap_or_default(); - let new = format.from_reader(File::open(path)?)?; - config = merge_configs(config, new); - } - } - from_value(Value::Object(config)) -} - -pub fn merge_configs(mut first: Config, second: Config) -> Config { - for (k, v) in second.into_iter() { - let new = match first.remove(&k) { - None => v, - Some(old) => match (old, v) { - (Value::Object(first), Value::Object(second)) => { - Value::Object(merge_configs(first, second)) - } - (first, _) => first, - }, - }; - first.insert(k, new); - } - first -} diff --git a/core/startos/src/util/cpupower.rs b/core/startos/src/util/cpupower.rs deleted file mode 100644 index cc4ac5ef..00000000 --- a/core/startos/src/util/cpupower.rs +++ /dev/null @@ -1,144 +0,0 @@ -use std::borrow::Cow; -use std::collections::BTreeSet; - -use imbl::OrdMap; -use tokio::process::Command; - -use crate::prelude::*; -use crate::util::Invoke; - -pub const GOVERNOR_HEIRARCHY: &[Governor] = &[ - Governor(Cow::Borrowed("ondemand")), - Governor(Cow::Borrowed("schedutil")), - Governor(Cow::Borrowed("conservative")), -]; - -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize, serde::Deserialize)] -pub struct Governor(Cow<'static, str>); -impl std::str::FromStr for Governor { - type Err = std::convert::Infallible; - fn from_str(s: &str) -> Result { - Ok(Self(s.to_owned().into())) - } -} -impl std::fmt::Display for Governor { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - self.0.fmt(f) - } -} -impl std::ops::Deref for Governor { - type Target = str; - fn deref(&self) -> &Self::Target { - &*self.0 - } -} -impl std::borrow::Borrow for Governor { - fn borrow(&self) -> &str { - &**self - } -} - -pub async fn get_available_governors() -> Result, Error> { - let raw = Command::new("cpupower") - .arg("frequency-info") - .arg("-g") - .invoke(ErrorKind::CpuSettings) - .await - .map_or_else(|e| Ok(e.source.to_string()), String::from_utf8)?; - let mut for_cpu: OrdMap> = OrdMap::new(); - let mut current_cpu = None; - for line in raw.lines() { - if line.starts_with("analyzing") { - current_cpu = Some( - sscanf::sscanf!(line, "analyzing CPU {u32}:") - .map_err(|e| eyre!("{e}")) - .with_kind(ErrorKind::ParseSysInfo)?, - ); - } else if let Some(rest) = line - .trim() - .strip_prefix("available cpufreq governors:") - .map(|s| s.trim()) - { - if rest != "Not Available" { - for_cpu - .entry(current_cpu.ok_or_else(|| { - Error::new( - eyre!("governors listed before cpu"), - ErrorKind::ParseSysInfo, - ) - })?) - .or_default() - .extend( - rest.split_ascii_whitespace() - .map(|g| Governor(Cow::Owned(g.to_owned()))), - ); - } - } - } - Ok(for_cpu - .into_iter() - .fold(None, |acc: Option>, (_, x)| { - if let Some(acc) = acc { - Some(acc.intersection(&x).cloned().collect()) - } else { - Some(x) - } - }) - .unwrap_or_default()) // include only governors available for ALL cpus -} - -pub async fn current_governor() -> Result, Error> { - let Some(raw) = Command::new("cpupower") - .arg("frequency-info") - .arg("-p") - .invoke(ErrorKind::CpuSettings) - .await - .and_then(|s| Ok(Some(String::from_utf8(s)?))) - .or_else(|e| { - if e.source - .to_string() - .contains("Unable to determine current policy") - { - Ok(None) - } else { - Err(e) - } - })? - else { - return Ok(None); - }; - - for line in raw.lines() { - if let Some(governor) = line - .trim() - .strip_prefix("The governor \"") - .and_then(|s| s.strip_suffix("\" may decide which speed to use")) - { - return Ok(Some(Governor(Cow::Owned(governor.to_owned())))); - } - } - Err(Error::new( - eyre!("Failed to parse cpupower output:\n{raw}"), - ErrorKind::ParseSysInfo, - )) -} - -pub async fn get_preferred_governor() -> Result, Error> { - let governors = get_available_governors().await?; - for governor in GOVERNOR_HEIRARCHY { - if governors.contains(governor) { - return Ok(Some(governor)); - } - } - Ok(None) -} - -pub async fn set_governor(governor: &Governor) -> Result<(), Error> { - Command::new("cpupower") - .arg("frequency-set") - .arg("-g") - .arg(&*governor.0) - .invoke(ErrorKind::CpuSettings) - .await?; - Ok(()) -} diff --git a/core/startos/src/util/crypto.rs b/core/startos/src/util/crypto.rs deleted file mode 100644 index 5c1aed01..00000000 --- a/core/startos/src/util/crypto.rs +++ /dev/null @@ -1,9 +0,0 @@ -use ed25519_dalek::{SecretKey, EXPANDED_SECRET_KEY_LENGTH}; - -#[inline] -pub fn ed25519_expand_key(key: &SecretKey) -> [u8; EXPANDED_SECRET_KEY_LENGTH] { - ed25519_dalek_v1::ExpandedSecretKey::from( - &ed25519_dalek_v1::SecretKey::from_bytes(key).unwrap(), - ) - .to_bytes() -} diff --git a/core/startos/src/util/docker.rs b/core/startos/src/util/docker.rs deleted file mode 100644 index fb6bc15f..00000000 --- a/core/startos/src/util/docker.rs +++ /dev/null @@ -1,239 +0,0 @@ -use std::net::Ipv4Addr; -use std::time::Duration; - -use models::{Error, ErrorKind, PackageId, ResultExt, Version}; -use nix::sys::signal::Signal; -use tokio::process::Command; - -use crate::util::Invoke; - -#[cfg(feature = "docker")] -pub const CONTAINER_TOOL: &str = "docker"; -#[cfg(not(feature = "docker"))] -pub const CONTAINER_TOOL: &str = "podman"; - -#[cfg(feature = "docker")] -pub const CONTAINER_DATADIR: &str = "/var/lib/docker"; -#[cfg(not(feature = "docker"))] -pub const CONTAINER_DATADIR: &str = "/var/lib/containers"; - -pub struct DockerImageSha(String); - -// docker images start9/${package}/*:${version} -q --no-trunc -pub async fn images_for( - package: &PackageId, - version: &Version, -) -> Result, Error> { - Ok(String::from_utf8( - Command::new(CONTAINER_TOOL) - .arg("images") - .arg(format!("start9/{package}/*:{version}")) - .arg("--no-trunc") - .arg("-q") - .invoke(ErrorKind::Docker) - .await?, - )? - .lines() - .map(|l| DockerImageSha(l.trim().to_owned())) - .collect()) -} - -// docker rmi -f ${sha} -pub async fn remove_image(sha: &DockerImageSha) -> Result<(), Error> { - match Command::new(CONTAINER_TOOL) - .arg("rmi") - .arg("-f") - .arg(&sha.0) - .invoke(ErrorKind::Docker) - .await - .map(|_| ()) - { - Err(e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such image") => - { - Ok(()) - } - a => a, - }?; - Ok(()) -} - -// docker image prune -f -pub async fn prune_images() -> Result<(), Error> { - Command::new(CONTAINER_TOOL) - .arg("image") - .arg("prune") - .arg("-f") - .invoke(ErrorKind::Docker) - .await?; - Ok(()) -} - -// docker container inspect ${name} --format '{{.NetworkSettings.Networks.start9.IPAddress}}' -pub async fn get_container_ip(name: &str) -> Result, Error> { - match Command::new(CONTAINER_TOOL) - .arg("container") - .arg("inspect") - .arg(name) - .arg("--format") - .arg("{{.NetworkSettings.Networks.start9.IPAddress}}") - .invoke(ErrorKind::Docker) - .await - { - Err(e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such container") => - { - Ok(None) - } - Err(e) => Err(e), - Ok(a) => { - let out = std::str::from_utf8(&a)?.trim(); - if out.is_empty() { - Ok(None) - } else { - Ok(Some({ - out.parse() - .with_ctx(|_| (ErrorKind::ParseNetAddress, out.to_string()))? - })) - } - } - } -} - -// docker stop -t ${timeout} -s ${signal} ${name} -pub async fn stop_container( - name: &str, - timeout: Option, - signal: Option, -) -> Result<(), Error> { - let mut cmd = Command::new(CONTAINER_TOOL); - cmd.arg("stop"); - if let Some(dur) = timeout { - cmd.arg("-t").arg(dur.as_secs().to_string()); - } - if let Some(sig) = signal { - cmd.arg("-s").arg(sig.to_string()); - } - cmd.arg(name); - match cmd.invoke(ErrorKind::Docker).await { - Ok(_) => Ok(()), - Err(mut e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such container") => - { - e.kind = ErrorKind::NotFound; - Err(e) - } - Err(e) => Err(e), - } -} - -// docker kill -s ${signal} ${name} -pub async fn kill_container(name: &str, signal: Option) -> Result<(), Error> { - let mut cmd = Command::new(CONTAINER_TOOL); - cmd.arg("kill"); - if let Some(sig) = signal { - cmd.arg("-s").arg(sig.to_string()); - } - cmd.arg(name); - match cmd.invoke(ErrorKind::Docker).await { - Ok(_) => Ok(()), - Err(mut e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such container") => - { - e.kind = ErrorKind::NotFound; - Err(e) - } - Err(e) => Err(e), - } -} - -// docker pause ${name} -pub async fn pause_container(name: &str) -> Result<(), Error> { - let mut cmd = Command::new(CONTAINER_TOOL); - cmd.arg("pause"); - cmd.arg(name); - match cmd.invoke(ErrorKind::Docker).await { - Ok(_) => Ok(()), - Err(mut e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such container") => - { - e.kind = ErrorKind::NotFound; - Err(e) - } - Err(e) => Err(e), - } -} - -// docker unpause ${name} -pub async fn unpause_container(name: &str) -> Result<(), Error> { - let mut cmd = Command::new(CONTAINER_TOOL); - cmd.arg("unpause"); - cmd.arg(name); - match cmd.invoke(ErrorKind::Docker).await { - Ok(_) => Ok(()), - Err(mut e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such container") => - { - e.kind = ErrorKind::NotFound; - Err(e) - } - Err(e) => Err(e), - } -} - -// docker rm -f ${name} -pub async fn remove_container(name: &str, force: bool) -> Result<(), Error> { - let mut cmd = Command::new(CONTAINER_TOOL); - cmd.arg("rm"); - if force { - cmd.arg("-f"); - } - cmd.arg(name); - match cmd.invoke(ErrorKind::Docker).await { - Ok(_) => Ok(()), - Err(e) - if e.source - .to_string() - .to_ascii_lowercase() - .contains("no such container") => - { - Ok(()) - } - Err(e) => Err(e), - } -} - -// docker network create -d bridge --subnet ${subnet} --opt com.podman.network.bridge.name=${bridge_name} -pub async fn create_bridge_network( - name: &str, - subnet: &str, - bridge_name: &str, -) -> Result<(), Error> { - let mut cmd = Command::new(CONTAINER_TOOL); - cmd.arg("network").arg("create"); - cmd.arg("-d").arg("bridge"); - cmd.arg("--subnet").arg(subnet); - cmd.arg("--opt") - .arg(format!("com.docker.network.bridge.name={bridge_name}")); - cmd.arg(name); - cmd.invoke(ErrorKind::Docker).await?; - Ok(()) -} diff --git a/core/startos/src/util/http_reader.rs b/core/startos/src/util/http_reader.rs deleted file mode 100644 index 87e8c114..00000000 --- a/core/startos/src/util/http_reader.rs +++ /dev/null @@ -1,380 +0,0 @@ -use std::cmp::min; -use std::convert::TryFrom; -use std::fmt::Display; -use std::future::Future; -use std::io::Error as StdIOError; -use std::pin::Pin; -use std::task::{Context, Poll}; - -use color_eyre::eyre::eyre; -use futures::Stream; -use http::header::{ACCEPT_RANGES, CONTENT_LENGTH, RANGE}; -use hyper::body::Bytes; -use pin_project::pin_project; -use reqwest::{Client, Url}; -use tokio::io::{AsyncRead, AsyncSeek}; - -use crate::{Error, ResultExt}; - -#[pin_project] -pub struct HttpReader { - http_url: Url, - cursor_pos: usize, - http_client: Client, - total_bytes: usize, - range_unit: Option, - read_in_progress: ReadInProgress, -} - -type InProgress = Pin< - Box< - dyn Future< - Output = Result< - Pin< - Box< - dyn Stream> - + Send - + Sync - + 'static, - >, - >, - Error, - >, - > + Send - + Sync - + 'static, - >, ->; - -enum ReadInProgress { - None, - InProgress(InProgress), - Complete(Pin> + Send + Sync + 'static>>), -} -impl ReadInProgress { - fn take(&mut self) -> Self { - std::mem::replace(self, Self::None) - } -} - -// If we want to add support for units other than Accept-Ranges: bytes, we can use this enum -#[derive(Clone, Copy)] -enum RangeUnit { - Bytes, -} -impl Default for RangeUnit { - fn default() -> Self { - RangeUnit::Bytes - } -} - -impl Display for RangeUnit { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - RangeUnit::Bytes => write!(f, "bytes"), - } - } -} - -impl HttpReader { - pub async fn new(http_url: Url) -> Result { - let http_client = Client::builder() - // .proxy(reqwest::Proxy::all("socks5h://127.0.0.1:9050").unwrap()) - .build() - .with_kind(crate::ErrorKind::TLSInit)?; - - // Make a head request so that we can get the file size and check for http range support. - let head_request = http_client - .head(http_url.clone()) - .send() - .await - .with_kind(crate::ErrorKind::InvalidRequest)?; - - let accept_ranges = head_request.headers().get(ACCEPT_RANGES); - - let range_unit = match accept_ranges { - Some(range_type) => { - // as per rfc, header will contain data but not always UTF8 characters. - - let value = range_type - .to_str() - .map_err(|err| Error::new(err, crate::ErrorKind::Utf8))?; - - match value { - "bytes" => Some(RangeUnit::Bytes), - _ => { - return Err(Error::new( - eyre!( - "{} HTTP range downloading not supported with this unit {value}", - http_url - ), - crate::ErrorKind::MissingHeader, - )); - } - } - } - - // None can mean just get entire contents, but we currently error out. - None => { - return Err(Error::new( - eyre!( - "{} HTTP range downloading not supported with this url", - http_url - ), - crate::ErrorKind::MissingHeader, - )) - } - }; - - let total_bytes_option = head_request.headers().get(CONTENT_LENGTH); - - let total_bytes = match total_bytes_option { - Some(bytes) => bytes - .to_str() - .map_err(|err| Error::new(err, crate::ErrorKind::Utf8))? - .parse::()?, - None => { - return Err(Error::new( - eyre!("No content length headers for {}", http_url), - crate::ErrorKind::MissingHeader, - )) - } - }; - - Ok(HttpReader { - http_url, - cursor_pos: 0, - http_client, - total_bytes, - range_unit, - read_in_progress: ReadInProgress::None, - }) - } - - // https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests - async fn get_range( - range_unit: Option, - http_client: Client, - http_url: Url, - start: usize, - len: usize, - total_bytes: usize, - ) -> Result< - Pin> + Send + Sync + 'static>>, - Error, - > { - let end = min(start + len, total_bytes) - 1; - - if start > end { - return Ok(Box::pin(futures::stream::empty())); - } - - let data_range = format!("{}={}-{} ", range_unit.unwrap_or_default(), start, end); - - let data_resp = http_client - .get(http_url) - .header(RANGE, data_range) - .send() - .await - .with_kind(crate::ErrorKind::Network)? - .error_for_status() - .with_kind(crate::ErrorKind::Network)?; - - Ok(Box::pin(data_resp.bytes_stream())) - } -} - -impl AsyncRead for HttpReader { - fn poll_read( - self: Pin<&mut Self>, - cx: &mut Context<'_>, - buf: &mut tokio::io::ReadBuf<'_>, - ) -> Poll> { - fn poll_complete( - body: &mut Pin< - Box> + Send + Sync + 'static>, - >, - cx: &mut Context<'_>, - buf: &mut tokio::io::ReadBuf<'_>, - ) -> Poll>> { - Poll::Ready(match futures::ready!(body.as_mut().poll_next(cx)) { - Some(Ok(bytes)) => { - if buf.remaining() < bytes.len() { - Some(Err(StdIOError::new( - std::io::ErrorKind::InvalidInput, - format!("more bytes returned than expected"), - ))) - } else { - buf.put_slice(&*bytes); - Some(Ok(bytes.len())) - } - } - Some(Err(e)) => Some(Err(StdIOError::new(std::io::ErrorKind::Interrupted, e))), - None => None, - }) - } - let this = self.project(); - - loop { - let mut in_progress = match this.read_in_progress.take() { - ReadInProgress::Complete(mut body) => match poll_complete(&mut body, cx, buf) { - Poll::Pending => { - *this.read_in_progress = ReadInProgress::Complete(body); - return Poll::Pending; - } - Poll::Ready(Some(Ok(len))) => { - *this.read_in_progress = ReadInProgress::Complete(body); - *this.cursor_pos += len; - - return Poll::Ready(Ok(())); - } - Poll::Ready(res) => { - if let Some(Err(e)) = res { - tracing::error!( - "Error reading bytes from {}: {}, attempting to resume download", - this.http_url, - e - ); - tracing::debug!("{:?}", e); - } - if *this.cursor_pos == *this.total_bytes { - return Poll::Ready(Ok(())); - } - continue; - } - }, - ReadInProgress::None => Box::pin(HttpReader::get_range( - *this.range_unit, - this.http_client.clone(), - this.http_url.clone(), - *this.cursor_pos, - buf.remaining(), - *this.total_bytes, - )), - ReadInProgress::InProgress(fut) => fut, - }; - - let res_poll = in_progress.as_mut().poll(cx); - - match res_poll { - Poll::Ready(result) => match result { - Ok(body) => { - *this.read_in_progress = ReadInProgress::Complete(body); - } - Err(err) => { - break Poll::Ready(Err(StdIOError::new( - std::io::ErrorKind::Interrupted, - Box::::from(err.source), - ))); - } - }, - Poll::Pending => { - *this.read_in_progress = ReadInProgress::InProgress(in_progress); - - break Poll::Pending; - } - } - } - } -} - -impl AsyncSeek for HttpReader { - fn start_seek(self: Pin<&mut Self>, position: std::io::SeekFrom) -> std::io::Result<()> { - let this = self.project(); - - this.read_in_progress.take(); // invalidate any existing reads - - match position { - std::io::SeekFrom::Start(offset) => { - let pos_res = usize::try_from(offset); - - match pos_res { - Ok(pos) => { - if pos > *this.total_bytes { - StdIOError::new( - std::io::ErrorKind::InvalidInput, - format!( - "The offset: {} cannot be greater than {} bytes", - pos, *this.total_bytes - ), - ); - } - *this.cursor_pos = pos; - } - Err(err) => return Err(StdIOError::new(std::io::ErrorKind::InvalidInput, err)), - } - Ok(()) - } - std::io::SeekFrom::Current(offset) => { - // We explicitly check if we read before byte 0. - let new_pos = i64::try_from(*this.cursor_pos) - .map_err(|err| StdIOError::new(std::io::ErrorKind::InvalidInput, err))? - + offset; - - if new_pos < 0 { - return Err(StdIOError::new( - std::io::ErrorKind::InvalidInput, - "Can't read before byte 0", - )); - } - - *this.cursor_pos = new_pos as usize; - Ok(()) - } - - std::io::SeekFrom::End(offset) => { - // We explicitly check if we read before byte 0. - let new_pos = i64::try_from(*this.total_bytes) - .map_err(|err| StdIOError::new(std::io::ErrorKind::InvalidInput, err))? - + offset; - - if new_pos < 0 { - return Err(StdIOError::new( - std::io::ErrorKind::InvalidInput, - "Can't read before byte 0", - )); - } - - *this.cursor_pos = new_pos as usize; - Ok(()) - } - } - } - - fn poll_complete(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll> { - Poll::Ready(Ok(self.cursor_pos as u64)) - } -} - -#[tokio::test] -async fn main_test() { - let http_url = Url::parse("https://start9.com/latest/_static/css/main.css").unwrap(); - - println!("Getting this resource: {}", http_url); - let mut test_reader = HttpReader::new(http_url).await.unwrap(); - - let mut buf = Vec::new(); - - tokio::io::copy(&mut test_reader, &mut buf).await.unwrap(); - - assert_eq!(buf.len(), test_reader.total_bytes) -} - -#[tokio::test] -#[ignore] -async fn s9pk_test() { - use tokio::io::BufReader; - - let http_url = Url::parse("http://qhc6ac47cytstejcepk2ia3ipadzjhlkc5qsktsbl4e7u2krfmfuaqqd.onion/content/files/2022/09/ghost.s9pk").unwrap(); - - println!("Getting this resource: {}", http_url); - let test_reader = - BufReader::with_capacity(1024 * 1024, HttpReader::new(http_url).await.unwrap()); - - let mut s9pk = crate::s9pk::reader::S9pkReader::from_reader(test_reader, false) - .await - .unwrap(); - - let manifest = s9pk.manifest().await.unwrap(); - assert_eq!(&manifest.id.to_string(), "ghost"); -} diff --git a/core/startos/src/util/io.rs b/core/startos/src/util/io.rs deleted file mode 100644 index 282a2db8..00000000 --- a/core/startos/src/util/io.rs +++ /dev/null @@ -1,671 +0,0 @@ -use std::future::Future; -use std::io::Cursor; -use std::os::unix::prelude::MetadataExt; -use std::path::Path; -use std::sync::atomic::AtomicU64; -use std::task::Poll; -use std::time::Duration; - -use futures::future::{BoxFuture, Fuse}; -use futures::{AsyncSeek, FutureExt, TryStreamExt}; -use helpers::NonDetachingJoinHandle; -use nix::unistd::{Gid, Uid}; -use tokio::io::{ - duplex, AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, DuplexStream, ReadBuf, WriteHalf, -}; -use tokio::net::TcpStream; -use tokio::time::{Instant, Sleep}; - -use crate::ResultExt; - -pub trait AsyncReadSeek: AsyncRead + AsyncSeek {} -impl AsyncReadSeek for T {} - -#[derive(Clone, Debug)] -pub struct AsyncCompat(pub T); -impl futures::io::AsyncRead for AsyncCompat -where - T: tokio::io::AsyncRead, -{ - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut [u8], - ) -> std::task::Poll> { - let mut read_buf = ReadBuf::new(buf); - tokio::io::AsyncRead::poll_read( - unsafe { self.map_unchecked_mut(|a| &mut a.0) }, - cx, - &mut read_buf, - ) - .map(|res| res.map(|_| read_buf.filled().len())) - } -} -impl tokio::io::AsyncRead for AsyncCompat -where - T: futures::io::AsyncRead, -{ - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut ReadBuf, - ) -> std::task::Poll> { - futures::io::AsyncRead::poll_read( - unsafe { self.map_unchecked_mut(|a| &mut a.0) }, - cx, - buf.initialize_unfilled(), - ) - .map(|res| res.map(|len| buf.set_filled(len))) - } -} -impl futures::io::AsyncWrite for AsyncCompat -where - T: tokio::io::AsyncWrite, -{ - fn poll_write( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &[u8], - ) -> std::task::Poll> { - tokio::io::AsyncWrite::poll_write(unsafe { self.map_unchecked_mut(|a| &mut a.0) }, cx, buf) - } - fn poll_flush( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - tokio::io::AsyncWrite::poll_flush(unsafe { self.map_unchecked_mut(|a| &mut a.0) }, cx) - } - fn poll_close( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - tokio::io::AsyncWrite::poll_shutdown(unsafe { self.map_unchecked_mut(|a| &mut a.0) }, cx) - } -} -impl tokio::io::AsyncWrite for AsyncCompat -where - T: futures::io::AsyncWrite, -{ - fn poll_write( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &[u8], - ) -> std::task::Poll> { - futures::io::AsyncWrite::poll_write( - unsafe { self.map_unchecked_mut(|a| &mut a.0) }, - cx, - buf, - ) - } - fn poll_flush( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - futures::io::AsyncWrite::poll_flush(unsafe { self.map_unchecked_mut(|a| &mut a.0) }, cx) - } - fn poll_shutdown( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - futures::io::AsyncWrite::poll_close(unsafe { self.map_unchecked_mut(|a| &mut a.0) }, cx) - } -} - -pub async fn from_yaml_async_reader(mut reader: R) -> Result -where - T: for<'de> serde::Deserialize<'de>, - R: AsyncRead + Unpin, -{ - let mut buffer = Vec::new(); - reader.read_to_end(&mut buffer).await?; - serde_yaml::from_slice(&buffer) - .map_err(color_eyre::eyre::Error::from) - .with_kind(crate::ErrorKind::Deserialization) -} - -pub async fn to_yaml_async_writer(mut writer: W, value: &T) -> Result<(), crate::Error> -where - T: serde::Serialize, - W: AsyncWrite + Unpin, -{ - let mut buffer = serde_yaml::to_string(value) - .with_kind(crate::ErrorKind::Serialization)? - .into_bytes(); - buffer.extend_from_slice(b"\n"); - writer.write_all(&buffer).await?; - Ok(()) -} - -pub async fn from_toml_async_reader(mut reader: R) -> Result -where - T: for<'de> serde::Deserialize<'de>, - R: AsyncRead + Unpin, -{ - let mut buffer = Vec::new(); - reader.read_to_end(&mut buffer).await?; - serde_toml::from_str(std::str::from_utf8(&buffer)?) - .map_err(color_eyre::eyre::Error::from) - .with_kind(crate::ErrorKind::Deserialization) -} - -pub async fn to_toml_async_writer(mut writer: W, value: &T) -> Result<(), crate::Error> -where - T: serde::Serialize, - W: AsyncWrite + Unpin, -{ - let mut buffer = serde_toml::to_string(value) - .with_kind(crate::ErrorKind::Serialization)? - .into_bytes(); - buffer.extend_from_slice(b"\n"); - writer.write_all(&buffer).await?; - Ok(()) -} - -pub async fn from_cbor_async_reader(mut reader: R) -> Result -where - T: for<'de> serde::Deserialize<'de>, - R: AsyncRead + Unpin, -{ - let mut buffer = Vec::new(); - reader.read_to_end(&mut buffer).await?; - serde_cbor::de::from_reader(buffer.as_slice()) - .map_err(color_eyre::eyre::Error::from) - .with_kind(crate::ErrorKind::Deserialization) -} -pub async fn to_cbor_async_writer(mut writer: W, value: &T) -> Result<(), crate::Error> -where - T: serde::Serialize, - W: AsyncWrite + Unpin, -{ - let mut buffer = Vec::new(); - serde_cbor::ser::into_writer(value, &mut buffer).with_kind(crate::ErrorKind::Serialization)?; - buffer.extend_from_slice(b"\n"); - writer.write_all(&buffer).await?; - Ok(()) -} - -pub async fn from_json_async_reader(mut reader: R) -> Result -where - T: for<'de> serde::Deserialize<'de>, - R: AsyncRead + Unpin, -{ - let mut buffer = Vec::new(); - reader.read_to_end(&mut buffer).await?; - serde_json::from_slice(&buffer) - .map_err(color_eyre::eyre::Error::from) - .with_kind(crate::ErrorKind::Deserialization) -} - -pub async fn to_json_async_writer(mut writer: W, value: &T) -> Result<(), crate::Error> -where - T: serde::Serialize, - W: AsyncWrite + Unpin, -{ - let buffer = serde_json::to_string(value).with_kind(crate::ErrorKind::Serialization)?; - writer.write_all(&buffer.as_bytes()).await?; - Ok(()) -} - -pub async fn to_json_pretty_async_writer(mut writer: W, value: &T) -> Result<(), crate::Error> -where - T: serde::Serialize, - W: AsyncWrite + Unpin, -{ - let mut buffer = - serde_json::to_string_pretty(value).with_kind(crate::ErrorKind::Serialization)?; - buffer.push_str("\n"); - writer.write_all(&buffer.as_bytes()).await?; - Ok(()) -} - -pub async fn copy_and_shutdown( - r: &mut R, - mut w: W, -) -> Result<(), std::io::Error> { - tokio::io::copy(r, &mut w).await?; - w.flush().await?; - w.shutdown().await?; - Ok(()) -} - -pub fn dir_size<'a, P: AsRef + 'a + Send + Sync>( - path: P, - ctr: Option<&'a Counter>, -) -> BoxFuture<'a, Result> { - async move { - tokio_stream::wrappers::ReadDirStream::new(tokio::fs::read_dir(path.as_ref()).await?) - .try_fold(0, |acc, e| async move { - let m = e.metadata().await?; - Ok(acc - + if m.is_file() { - if let Some(ctr) = ctr { - ctr.add(m.len()); - } - m.len() - } else if m.is_dir() { - dir_size(e.path(), ctr).await? - } else { - 0 - }) - }) - .await - } - .boxed() -} - -pub fn response_to_reader(response: reqwest::Response) -> impl AsyncRead + Unpin { - tokio_util::io::StreamReader::new(response.bytes_stream().map_err(|e| { - std::io::Error::new( - if e.is_connect() { - std::io::ErrorKind::ConnectionRefused - } else if e.is_timeout() { - std::io::ErrorKind::TimedOut - } else { - std::io::ErrorKind::Other - }, - e, - ) - })) -} - -#[pin_project::pin_project] -pub struct BufferedWriteReader { - #[pin] - hdl: Fuse>>, - #[pin] - rdr: DuplexStream, -} -impl BufferedWriteReader { - pub fn new< - W: FnOnce(WriteHalf) -> Fut, - Fut: Future> + Send + Sync + 'static, - >( - write_fn: W, - max_buf_size: usize, - ) -> Self { - let (w, rdr) = duplex(max_buf_size); - let (_, w) = tokio::io::split(w); - BufferedWriteReader { - hdl: NonDetachingJoinHandle::from(tokio::spawn(write_fn(w))).fuse(), - rdr, - } - } -} -impl AsyncRead for BufferedWriteReader { - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut ReadBuf<'_>, - ) -> std::task::Poll> { - let this = self.project(); - let res = this.rdr.poll_read(cx, buf); - match this.hdl.poll(cx) { - Poll::Ready(Ok(Err(e))) => return Poll::Ready(Err(e)), - Poll::Ready(Err(e)) => { - return Poll::Ready(Err(std::io::Error::new(std::io::ErrorKind::BrokenPipe, e))) - } - _ => res, - } - } -} - -pub trait CursorExt { - fn pure_read(&mut self, buf: &mut ReadBuf<'_>); -} - -impl> CursorExt for Cursor { - fn pure_read(&mut self, buf: &mut ReadBuf<'_>) { - let end = self.position() as usize - + std::cmp::min( - buf.remaining(), - self.get_ref().as_ref().len() - self.position() as usize, - ); - buf.put_slice(&self.get_ref().as_ref()[self.position() as usize..end]); - self.set_position(end as u64); - } -} - -#[pin_project::pin_project] -#[derive(Debug)] -pub struct BackTrackingReader { - #[pin] - reader: T, - buffer: Cursor>, - buffering: bool, -} -impl BackTrackingReader { - pub fn new(reader: T) -> Self { - Self { - reader, - buffer: Cursor::new(Vec::new()), - buffering: false, - } - } - pub fn start_buffering(&mut self) { - self.buffer.set_position(0); - self.buffer.get_mut().truncate(0); - self.buffering = true; - } - pub fn stop_buffering(&mut self) { - self.buffer.set_position(0); - self.buffer.get_mut().truncate(0); - self.buffering = false; - } - pub fn rewind(&mut self) { - self.buffering = false; - } - pub fn unwrap(self) -> T { - self.reader - } -} - -impl AsyncRead for BackTrackingReader { - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut ReadBuf<'_>, - ) -> Poll> { - let this = self.project(); - if *this.buffering { - let filled = buf.filled().len(); - let res = this.reader.poll_read(cx, buf); - this.buffer - .get_mut() - .extend_from_slice(&buf.filled()[filled..]); - res - } else { - let mut ready = false; - if (this.buffer.position() as usize) < this.buffer.get_ref().len() { - this.buffer.pure_read(buf); - ready = true; - } - if buf.remaining() > 0 { - match this.reader.poll_read(cx, buf) { - Poll::Pending => { - if ready { - Poll::Ready(Ok(())) - } else { - Poll::Pending - } - } - a => a, - } - } else { - Poll::Ready(Ok(())) - } - } - } -} - -impl AsyncWrite for BackTrackingReader { - fn is_write_vectored(&self) -> bool { - self.reader.is_write_vectored() - } - fn poll_flush( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> Poll> { - self.project().reader.poll_flush(cx) - } - fn poll_shutdown( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> Poll> { - self.project().reader.poll_shutdown(cx) - } - fn poll_write( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &[u8], - ) -> Poll> { - self.project().reader.poll_write(cx, buf) - } - fn poll_write_vectored( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - bufs: &[std::io::IoSlice<'_>], - ) -> Poll> { - self.project().reader.poll_write_vectored(cx, bufs) - } -} - -pub struct Counter { - atomic: AtomicU64, - ordering: std::sync::atomic::Ordering, -} -impl Counter { - pub fn new(init: u64, ordering: std::sync::atomic::Ordering) -> Self { - Self { - atomic: AtomicU64::new(init), - ordering, - } - } - pub fn load(&self) -> u64 { - self.atomic.load(self.ordering) - } - pub fn add(&self, value: u64) { - self.atomic.fetch_add(value, self.ordering); - } -} - -#[pin_project::pin_project] -pub struct CountingReader<'a, R> { - ctr: &'a Counter, - #[pin] - rdr: R, -} -impl<'a, R> CountingReader<'a, R> { - pub fn new(rdr: R, ctr: &'a Counter) -> Self { - Self { ctr, rdr } - } - pub fn into_inner(self) -> R { - self.rdr - } -} -impl<'a, R: AsyncRead> AsyncRead for CountingReader<'a, R> { - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut ReadBuf<'_>, - ) -> Poll> { - let this = self.project(); - let start = buf.filled().len(); - let res = this.rdr.poll_read(cx, buf); - let len = buf.filled().len() - start; - if len > 0 { - this.ctr.add(len as u64); - } - res - } -} - -pub fn dir_copy<'a, P0: AsRef + 'a + Send + Sync, P1: AsRef + 'a + Send + Sync>( - src: P0, - dst: P1, - ctr: Option<&'a Counter>, -) -> BoxFuture<'a, Result<(), crate::Error>> { - async move { - let m = tokio::fs::metadata(&src).await?; - let dst_path = dst.as_ref(); - tokio::fs::create_dir_all(&dst_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("mkdir {}", dst_path.display()), - ) - })?; - tokio::fs::set_permissions(&dst_path, m.permissions()) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("chmod {}", dst_path.display()), - ) - })?; - let tmp_dst_path = dst_path.to_owned(); - tokio::task::spawn_blocking(move || { - nix::unistd::chown( - &tmp_dst_path, - Some(Uid::from_raw(m.uid())), - Some(Gid::from_raw(m.gid())), - ) - }) - .await - .with_kind(crate::ErrorKind::Unknown)? - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("chown {}", dst_path.display()), - ) - })?; - tokio_stream::wrappers::ReadDirStream::new(tokio::fs::read_dir(src.as_ref()).await?) - .map_err(|e| crate::Error::new(e, crate::ErrorKind::Filesystem)) - .try_for_each(|e| async move { - let m = e.metadata().await?; - let src_path = e.path(); - let dst_path = dst_path.join(e.file_name()); - if m.is_file() { - let mut dst_file = tokio::fs::File::create(&dst_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("create {}", dst_path.display()), - ) - })?; - let mut rdr = tokio::fs::File::open(&src_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("open {}", src_path.display()), - ) - })?; - if let Some(ctr) = ctr { - tokio::io::copy(&mut CountingReader::new(rdr, ctr), &mut dst_file).await - } else { - tokio::io::copy(&mut rdr, &mut dst_file).await - } - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("cp {} -> {}", src_path.display(), dst_path.display()), - ) - })?; - dst_file.flush().await?; - dst_file.shutdown().await?; - dst_file.sync_all().await?; - drop(dst_file); - let tmp_dst_path = dst_path.clone(); - tokio::task::spawn_blocking(move || { - nix::unistd::chown( - &tmp_dst_path, - Some(Uid::from_raw(m.uid())), - Some(Gid::from_raw(m.gid())), - ) - }) - .await - .with_kind(crate::ErrorKind::Unknown)? - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("chown {}", dst_path.display()), - ) - })?; - } else if m.is_dir() { - dir_copy(src_path, dst_path, ctr).await?; - } else if m.file_type().is_symlink() { - tokio::fs::symlink( - tokio::fs::read_link(&src_path).await.with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("readlink {}", src_path.display()), - ) - })?, - &dst_path, - ) - .await - .with_ctx(|_| { - ( - crate::ErrorKind::Filesystem, - format!("cp -P {} -> {}", src_path.display(), dst_path.display()), - ) - })?; - // Do not set permissions (see https://unix.stackexchange.com/questions/87200/change-permissions-for-a-symbolic-link) - } - Ok(()) - }) - .await?; - Ok(()) - } - .boxed() -} - -#[pin_project::pin_project] -pub struct TimeoutStream { - timeout: Duration, - #[pin] - sleep: Sleep, - #[pin] - stream: S, -} -impl TimeoutStream { - pub fn new(stream: S, timeout: Duration) -> Self { - Self { - timeout, - sleep: tokio::time::sleep(timeout), - stream, - } - } -} -impl AsyncRead for TimeoutStream { - fn poll_read( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &mut tokio::io::ReadBuf<'_>, - ) -> std::task::Poll> { - let mut this = self.project(); - if let std::task::Poll::Ready(_) = this.sleep.as_mut().poll(cx) { - return std::task::Poll::Ready(Err(std::io::Error::new( - std::io::ErrorKind::TimedOut, - "timed out", - ))); - } - let res = this.stream.poll_read(cx, buf); - if res.is_ready() { - this.sleep.reset(Instant::now() + *this.timeout); - } - res - } -} -impl AsyncWrite for TimeoutStream { - fn poll_write( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - buf: &[u8], - ) -> std::task::Poll> { - let this = self.project(); - let res = this.stream.poll_write(cx, buf); - if res.is_ready() { - this.sleep.reset(Instant::now() + *this.timeout); - } - res - } - fn poll_flush( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - let this = self.project(); - let res = this.stream.poll_flush(cx); - if res.is_ready() { - this.sleep.reset(Instant::now() + *this.timeout); - } - res - } - fn poll_shutdown( - self: std::pin::Pin<&mut Self>, - cx: &mut std::task::Context<'_>, - ) -> std::task::Poll> { - let this = self.project(); - let res = this.stream.poll_shutdown(cx); - if res.is_ready() { - this.sleep.reset(Instant::now() + *this.timeout); - } - res - } -} diff --git a/core/startos/src/util/logger.rs b/core/startos/src/util/logger.rs deleted file mode 100644 index c7ab41ba..00000000 --- a/core/startos/src/util/logger.rs +++ /dev/null @@ -1,52 +0,0 @@ -use tracing::Subscriber; -use tracing_subscriber::util::SubscriberInitExt; - -#[derive(Clone)] -pub struct EmbassyLogger {} - -impl EmbassyLogger { - fn base_subscriber() -> impl Subscriber { - use tracing_error::ErrorLayer; - use tracing_subscriber::prelude::*; - use tracing_subscriber::{fmt, EnvFilter}; - - let filter_layer = EnvFilter::builder() - .with_default_directive( - format!("{}=info", std::module_path!().split("::").next().unwrap()) - .parse() - .unwrap(), - ) - .from_env_lossy(); - #[cfg(feature = "unstable")] - let filter_layer = filter_layer - .add_directive("tokio=trace".parse().unwrap()) - .add_directive("runtime=trace".parse().unwrap()); - let fmt_layer = fmt::layer().with_target(true); - - let sub = tracing_subscriber::registry() - .with(filter_layer) - .with(fmt_layer) - .with(ErrorLayer::default()); - - #[cfg(feature = "unstable")] - let sub = sub.with(console_subscriber::spawn()); - - sub - } - pub fn init() -> Self { - Self::base_subscriber().init(); - color_eyre::install().unwrap_or_else(|_| tracing::warn!("tracing too many times")); - - EmbassyLogger {} - } -} - -#[tokio::test] -pub async fn order_level() { - assert!(tracing::Level::WARN > tracing::Level::ERROR) -} - -#[test] -pub fn module() { - println!("{}", module_path!()) -} diff --git a/core/startos/src/util/lshw.rs b/core/startos/src/util/lshw.rs deleted file mode 100644 index dd260f64..00000000 --- a/core/startos/src/util/lshw.rs +++ /dev/null @@ -1,63 +0,0 @@ -use models::{Error, ResultExt}; -use serde::{Deserialize, Serialize}; -use tokio::process::Command; - -use crate::util::Invoke; - -const KNOWN_CLASSES: &[&str] = &["processor", "display"]; - -#[derive(Debug, Deserialize, Serialize)] -#[serde(tag = "class")] -#[serde(rename_all = "kebab-case")] -pub enum LshwDevice { - Processor(LshwProcessor), - Display(LshwDisplay), -} -impl LshwDevice { - pub fn class(&self) -> &'static str { - match self { - Self::Processor(_) => "processor", - Self::Display(_) => "display", - } - } - pub fn product(&self) -> &str { - match self { - Self::Processor(hw) => hw.product.as_str(), - Self::Display(hw) => hw.product.as_str(), - } - } -} - -#[derive(Debug, Deserialize, Serialize)] -pub struct LshwProcessor { - pub product: String, -} - -#[derive(Debug, Deserialize, Serialize)] -pub struct LshwDisplay { - pub product: String, -} - -pub async fn lshw() -> Result, Error> { - let mut cmd = Command::new("lshw"); - cmd.arg("-json"); - for class in KNOWN_CLASSES { - cmd.arg("-class").arg(*class); - } - Ok( - serde_json::from_slice::>( - &cmd.invoke(crate::ErrorKind::Lshw).await?, - ) - .with_kind(crate::ErrorKind::Deserialization)? - .into_iter() - .filter_map(|v| match serde_json::from_value(v) { - Ok(a) => Some(a), - Err(e) => { - tracing::error!("Failed to parse lshw output: {e}"); - tracing::debug!("{e:?}"); - None - } - }) - .collect(), - ) -} diff --git a/core/startos/src/util/mod.rs b/core/startos/src/util/mod.rs deleted file mode 100644 index 2683f23c..00000000 --- a/core/startos/src/util/mod.rs +++ /dev/null @@ -1,468 +0,0 @@ -use std::collections::BTreeMap; -use std::future::Future; -use std::marker::PhantomData; -use std::path::{Path, PathBuf}; -use std::pin::Pin; -use std::process::Stdio; -use std::sync::Arc; -use std::task::{Context, Poll}; -use std::time::Duration; - -use async_trait::async_trait; -use clap::ArgMatches; -use color_eyre::eyre::{self, eyre}; -use fd_lock_rs::FdLock; -use helpers::canonicalize; -pub use helpers::NonDetachingJoinHandle; -use lazy_static::lazy_static; -pub use models::Version; -use pin_project::pin_project; -use sha2::Digest; -use tokio::fs::File; -use tokio::sync::{Mutex, OwnedMutexGuard, RwLock}; -use tracing::instrument; - -use crate::shutdown::Shutdown; -use crate::{Error, ErrorKind, ResultExt as _}; -pub mod config; -pub mod cpupower; -pub mod crypto; -pub mod docker; -pub mod http_reader; -pub mod io; -pub mod logger; -pub mod lshw; -pub mod serde; - -#[derive(Clone, Copy, Debug, ::serde::Deserialize, ::serde::Serialize)] -pub enum Never {} -impl Never {} -impl Never { - pub fn absurd(self) -> T { - match self {} - } -} -impl std::fmt::Display for Never { - fn fmt(&self, _f: &mut std::fmt::Formatter) -> std::fmt::Result { - self.absurd() - } -} -impl std::error::Error for Never {} - -#[async_trait::async_trait] -pub trait Invoke<'a> { - type Extended<'ext> - where - Self: 'ext, - 'ext: 'a; - fn timeout<'ext: 'a>(&'ext mut self, timeout: Option) -> Self::Extended<'ext>; - fn input<'ext: 'a, Input: tokio::io::AsyncRead + Unpin + Send>( - &'ext mut self, - input: Option<&'ext mut Input>, - ) -> Self::Extended<'ext>; - async fn invoke(&mut self, error_kind: crate::ErrorKind) -> Result, Error>; -} - -pub struct ExtendedCommand<'a> { - cmd: &'a mut tokio::process::Command, - timeout: Option, - input: Option<&'a mut (dyn tokio::io::AsyncRead + Unpin + Send)>, -} -impl<'a> std::ops::Deref for ExtendedCommand<'a> { - type Target = tokio::process::Command; - fn deref(&self) -> &Self::Target { - &*self.cmd - } -} -impl<'a> std::ops::DerefMut for ExtendedCommand<'a> { - fn deref_mut(&mut self) -> &mut Self::Target { - self.cmd - } -} - -#[async_trait::async_trait] -impl<'a> Invoke<'a> for tokio::process::Command { - type Extended<'ext> = ExtendedCommand<'ext> - where - Self: 'ext, - 'ext: 'a; - fn timeout<'ext: 'a>(&'ext mut self, timeout: Option) -> Self::Extended<'ext> { - ExtendedCommand { - cmd: self, - timeout, - input: None, - } - } - fn input<'ext: 'a, Input: tokio::io::AsyncRead + Unpin + Send>( - &'ext mut self, - input: Option<&'ext mut Input>, - ) -> Self::Extended<'ext> { - ExtendedCommand { - cmd: self, - timeout: None, - input: if let Some(input) = input { - Some(&mut *input) - } else { - None - }, - } - } - async fn invoke(&mut self, error_kind: crate::ErrorKind) -> Result, Error> { - ExtendedCommand { - cmd: self, - timeout: None, - input: None, - } - .invoke(error_kind) - .await - } -} - -#[async_trait::async_trait] -impl<'a> Invoke<'a> for ExtendedCommand<'a> { - type Extended<'ext> = &'ext mut ExtendedCommand<'ext> - where - Self: 'ext, - 'ext: 'a; - fn timeout<'ext: 'a>(&'ext mut self, timeout: Option) -> Self::Extended<'ext> { - self.timeout = timeout; - self - } - fn input<'ext: 'a, Input: tokio::io::AsyncRead + Unpin + Send>( - &'ext mut self, - input: Option<&'ext mut Input>, - ) -> Self::Extended<'ext> { - self.input = if let Some(input) = input { - Some(&mut *input) - } else { - None - }; - self - } - async fn invoke(&mut self, error_kind: crate::ErrorKind) -> Result, Error> { - self.cmd.kill_on_drop(true); - if self.input.is_some() { - self.cmd.stdin(Stdio::piped()); - } - self.cmd.stdout(Stdio::piped()); - self.cmd.stderr(Stdio::piped()); - let mut child = self.cmd.spawn()?; - if let (Some(mut stdin), Some(input)) = (child.stdin.take(), self.input.take()) { - use tokio::io::AsyncWriteExt; - tokio::io::copy(input, &mut stdin).await?; - stdin.flush().await?; - stdin.shutdown().await?; - drop(stdin); - } - let res = match self.timeout { - None => child.wait_with_output().await?, - Some(t) => tokio::time::timeout(t, child.wait_with_output()) - .await - .with_kind(ErrorKind::Timeout)??, - }; - crate::ensure_code!( - res.status.success(), - error_kind, - "{}", - Some(&res.stderr) - .filter(|a| !a.is_empty()) - .or(Some(&res.stdout)) - .filter(|a| !a.is_empty()) - .and_then(|a| std::str::from_utf8(a).ok()) - .unwrap_or(&format!("Unknown Error ({})", res.status)) - ); - Ok(res.stdout) - } -} - -pub trait Apply: Sized { - fn apply O>(self, func: F) -> O { - func(self) - } -} - -pub trait ApplyRef { - fn apply_ref O>(&self, func: F) -> O { - func(&self) - } - - fn apply_mut O>(&mut self, func: F) -> O { - func(self) - } -} - -impl Apply for T {} -impl ApplyRef for T {} - -pub async fn daemon Fut, Fut: Future + Send + 'static>( - mut f: F, - cooldown: std::time::Duration, - mut shutdown: tokio::sync::broadcast::Receiver>, -) -> Result<(), eyre::Error> { - loop { - tokio::select! { - _ = shutdown.recv() => return Ok(()), - _ = tokio::time::sleep(cooldown) => (), - } - match tokio::spawn(f()).await { - Err(e) if e.is_panic() => return Err(eyre!("daemon panicked!")), - _ => (), - } - } -} - -pub trait SOption {} -pub struct SSome(T); -impl SSome { - pub fn into(self) -> T { - self.0 - } -} -impl From for SSome { - fn from(t: T) -> Self { - SSome(t) - } -} -impl SOption for SSome {} -pub struct SNone(PhantomData); -impl SNone { - pub fn new() -> Self { - SNone(PhantomData) - } -} -impl SOption for SNone {} - -#[async_trait] -pub trait AsyncFileExt: Sized { - async fn maybe_open + Send + Sync>(path: P) -> std::io::Result>; - async fn delete + Send + Sync>(path: P) -> std::io::Result<()>; -} -#[async_trait] -impl AsyncFileExt for File { - async fn maybe_open + Send + Sync>(path: P) -> std::io::Result> { - match File::open(path).await { - Ok(f) => Ok(Some(f)), - Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), - Err(e) => Err(e), - } - } - async fn delete + Send + Sync>(path: P) -> std::io::Result<()> { - if let Ok(m) = tokio::fs::metadata(path.as_ref()).await { - if m.is_dir() { - tokio::fs::remove_dir_all(path).await - } else { - tokio::fs::remove_file(path).await - } - } else { - Ok(()) - } - } -} - -pub struct FmtWriter(W); -impl std::io::Write for FmtWriter { - fn write(&mut self, buf: &[u8]) -> std::io::Result { - self.0 - .write_str( - std::str::from_utf8(buf) - .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?, - ) - .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?; - Ok(buf.len()) - } - fn flush(&mut self) -> std::io::Result<()> { - Ok(()) - } -} - -pub fn display_none(_: T, _: &ArgMatches) {} - -pub struct Container(RwLock>); -impl Container { - pub fn new(value: Option) -> Self { - Container(RwLock::new(value)) - } - pub async fn set(&self, value: T) -> Option { - std::mem::replace(&mut *self.0.write().await, Some(value)) - } - pub async fn take(&self) -> Option { - self.0.write().await.take() - } - pub async fn is_empty(&self) -> bool { - self.0.read().await.is_none() - } - pub async fn drop(&self) { - *self.0.write().await = None; - } -} - -#[pin_project] -pub struct HashWriter { - hasher: H, - #[pin] - writer: W, -} -impl HashWriter { - pub fn new(hasher: H, writer: W) -> Self { - HashWriter { hasher, writer } - } - pub fn finish(self) -> (H, W) { - (self.hasher, self.writer) - } - pub fn inner(&self) -> &W { - &self.writer - } - pub fn inner_mut(&mut self) -> &mut W { - &mut self.writer - } -} -impl tokio::io::AsyncWrite for HashWriter { - fn poll_write( - self: Pin<&mut Self>, - cx: &mut Context, - buf: &[u8], - ) -> Poll> { - let this = self.project(); - let written = tokio::io::AsyncWrite::poll_write(this.writer, cx, buf); - match written { - // only update the hasher once - Poll::Ready(res) => { - if let Ok(n) = res { - this.hasher.update(&buf[..n]); - } - Poll::Ready(res) - } - Poll::Pending => Poll::Pending, - } - } - fn poll_flush(self: Pin<&mut Self>, cx: &mut Context) -> Poll> { - self.project().writer.poll_flush(cx) - } - - fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context) -> Poll> { - self.project().writer.poll_shutdown(cx) - } -} - -pub trait IntoDoubleEndedIterator: IntoIterator { - type IntoIter: Iterator + DoubleEndedIterator; - fn into_iter(self) -> >::IntoIter; -} -impl IntoDoubleEndedIterator for T -where - T: IntoIterator, - ::IntoIter: DoubleEndedIterator, -{ - type IntoIter = ::IntoIter; - fn into_iter(self) -> >::IntoIter { - IntoIterator::into_iter(self) - } -} - -pub struct GeneralBoxedGuard(Option>); -impl GeneralBoxedGuard { - pub fn new(f: impl FnOnce() + 'static + Send + Sync) -> Self { - GeneralBoxedGuard(Some(Box::new(f))) - } - - pub fn drop(mut self) { - self.0.take().unwrap()() - } - - pub fn drop_without_action(mut self) { - self.0 = None; - } -} - -impl Drop for GeneralBoxedGuard { - fn drop(&mut self) { - if let Some(destroy) = self.0.take() { - destroy(); - } - } -} - -pub struct GeneralGuard T, T = ()>(Option); -impl T, T> GeneralGuard { - pub fn new(f: F) -> Self { - GeneralGuard(Some(f)) - } - - pub fn drop(mut self) -> T { - self.0.take().unwrap()() - } - - pub fn drop_without_action(mut self) { - self.0 = None; - } -} - -impl T, T> Drop for GeneralGuard { - fn drop(&mut self) { - if let Some(destroy) = self.0.take() { - destroy(); - } - } -} - -pub struct FileLock(OwnedMutexGuard<()>, Option>); -impl Drop for FileLock { - fn drop(&mut self) { - if let Some(fd_lock) = self.1.take() { - tokio::task::spawn_blocking(|| fd_lock.unlock(true).map_err(|(_, e)| e).unwrap()); - } - } -} -impl FileLock { - #[instrument(skip_all)] - pub async fn new(path: impl AsRef + Send + Sync, blocking: bool) -> Result { - lazy_static! { - static ref INTERNAL_LOCKS: Mutex>>> = - Mutex::new(BTreeMap::new()); - } - let path = canonicalize(path.as_ref(), true) - .await - .with_kind(ErrorKind::Filesystem)?; - let mut internal_locks = INTERNAL_LOCKS.lock().await; - if !internal_locks.contains_key(&path) { - internal_locks.insert(path.clone(), Arc::new(Mutex::new(()))); - } - let tex = internal_locks.get(&path).unwrap().clone(); - drop(internal_locks); - let tex_guard = if blocking { - tex.lock_owned().await - } else { - tex.try_lock_owned() - .with_kind(crate::ErrorKind::Filesystem)? - }; - let parent = path.parent().unwrap_or(Path::new("/")); - if tokio::fs::metadata(parent).await.is_err() { - tokio::fs::create_dir_all(parent) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, parent.display().to_string()))?; - } - let f = File::create(&path) - .await - .with_ctx(|_| (crate::ErrorKind::Filesystem, path.display().to_string()))?; - let file_guard = tokio::task::spawn_blocking(move || { - fd_lock_rs::FdLock::lock(f, fd_lock_rs::LockType::Exclusive, blocking) - }) - .await - .with_kind(crate::ErrorKind::Unknown)? - .with_kind(crate::ErrorKind::Filesystem)?; - Ok(FileLock(tex_guard, Some(file_guard))) - } - pub async fn unlock(mut self) -> Result<(), Error> { - if let Some(fd_lock) = self.1.take() { - tokio::task::spawn_blocking(|| fd_lock.unlock(true).map_err(|(_, e)| e)) - .await - .with_kind(crate::ErrorKind::Unknown)? - .with_kind(crate::ErrorKind::Filesystem)?; - } - Ok(()) - } -} - -pub fn assure_send(x: T) -> T { - x -} diff --git a/core/startos/src/util/serde.rs b/core/startos/src/util/serde.rs deleted file mode 100644 index 4a6f7551..00000000 --- a/core/startos/src/util/serde.rs +++ /dev/null @@ -1,845 +0,0 @@ -use std::marker::PhantomData; -use std::ops::Deref; -use std::process::exit; -use std::str::FromStr; - -use clap::ArgMatches; -use color_eyre::eyre::eyre; -use serde::ser::{SerializeMap, SerializeSeq}; -use serde::{Deserialize, Deserializer, Serialize, Serializer}; -use serde_json::Value; - -use super::IntoDoubleEndedIterator; -use crate::{Error, ResultExt}; - -pub fn deserialize_from_str< - 'de, - D: serde::de::Deserializer<'de>, - T: FromStr, - E: std::fmt::Display, ->( - deserializer: D, -) -> std::result::Result { - struct Visitor, E>(std::marker::PhantomData); - impl<'de, T: FromStr, Err: std::fmt::Display> serde::de::Visitor<'de> - for Visitor - { - type Value = T; - fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(formatter, "a parsable string") - } - fn visit_str(self, v: &str) -> Result - where - E: serde::de::Error, - { - v.parse().map_err(|e| serde::de::Error::custom(e)) - } - } - deserializer.deserialize_str(Visitor(std::marker::PhantomData)) -} - -pub fn deserialize_from_str_opt< - 'de, - D: serde::de::Deserializer<'de>, - T: FromStr, - E: std::fmt::Display, ->( - deserializer: D, -) -> std::result::Result, D::Error> { - struct Visitor, E>(std::marker::PhantomData); - impl<'de, T: FromStr, Err: std::fmt::Display> serde::de::Visitor<'de> - for Visitor - { - type Value = Option; - fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(formatter, "a parsable string") - } - fn visit_str(self, v: &str) -> Result - where - E: serde::de::Error, - { - v.parse().map(Some).map_err(|e| serde::de::Error::custom(e)) - } - fn visit_some(self, deserializer: D) -> Result - where - D: serde::de::Deserializer<'de>, - { - deserializer.deserialize_str(Visitor(std::marker::PhantomData)) - } - fn visit_none(self) -> Result - where - E: serde::de::Error, - { - Ok(None) - } - fn visit_unit(self) -> Result - where - E: serde::de::Error, - { - Ok(None) - } - } - deserializer.deserialize_any(Visitor(std::marker::PhantomData)) -} - -pub fn serialize_display( - t: &T, - serializer: S, -) -> Result { - String::serialize(&t.to_string(), serializer) -} - -pub fn serialize_display_opt( - t: &Option, - serializer: S, -) -> Result { - Option::::serialize(&t.as_ref().map(|t| t.to_string()), serializer) -} - -pub mod ed25519_pubkey { - use ed25519_dalek::VerifyingKey; - use serde::de::{Error, Unexpected, Visitor}; - use serde::{Deserializer, Serializer}; - - pub fn serialize( - pubkey: &VerifyingKey, - serializer: S, - ) -> Result { - serializer.serialize_str(&base32::encode( - base32::Alphabet::RFC4648 { padding: true }, - pubkey.as_bytes(), - )) - } - pub fn deserialize<'de, D: Deserializer<'de>>( - deserializer: D, - ) -> Result { - struct PubkeyVisitor; - impl<'de> Visitor<'de> for PubkeyVisitor { - type Value = ed25519_dalek::VerifyingKey; - fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(formatter, "an RFC4648 encoded string") - } - fn visit_str(self, v: &str) -> Result - where - E: Error, - { - VerifyingKey::from_bytes( - &<[u8; 32]>::try_from( - base32::decode(base32::Alphabet::RFC4648 { padding: true }, v).ok_or( - Error::invalid_value(Unexpected::Str(v), &"an RFC4648 encoded string"), - )?, - ) - .map_err(|e| Error::invalid_length(e.len(), &"32 bytes"))?, - ) - .map_err(Error::custom) - } - } - deserializer.deserialize_str(PubkeyVisitor) - } -} - -#[derive(Debug, Serialize)] -#[serde(untagged)] -pub enum ValuePrimative { - Null, - Boolean(bool), - String(String), - Number(serde_json::Number), -} -impl<'de> serde::de::Deserialize<'de> for ValuePrimative { - fn deserialize(deserializer: D) -> Result - where - D: serde::de::Deserializer<'de>, - { - struct Visitor; - impl<'de> serde::de::Visitor<'de> for Visitor { - type Value = ValuePrimative; - fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { - write!(formatter, "a JSON primative value") - } - fn visit_unit(self) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Null) - } - fn visit_none(self) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Null) - } - fn visit_bool(self, v: bool) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Boolean(v)) - } - fn visit_str(self, v: &str) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::String(v.to_owned())) - } - fn visit_string(self, v: String) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::String(v)) - } - fn visit_f32(self, v: f32) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number( - serde_json::Number::from_f64(v as f64).ok_or_else(|| { - serde::de::Error::invalid_value( - serde::de::Unexpected::Float(v as f64), - &"a finite number", - ) - })?, - )) - } - fn visit_f64(self, v: f64) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number( - serde_json::Number::from_f64(v).ok_or_else(|| { - serde::de::Error::invalid_value( - serde::de::Unexpected::Float(v), - &"a finite number", - ) - })?, - )) - } - fn visit_u8(self, v: u8) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_u16(self, v: u16) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_u32(self, v: u32) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_u64(self, v: u64) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_i8(self, v: i8) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_i16(self, v: i16) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_i32(self, v: i32) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - fn visit_i64(self, v: i64) -> Result - where - E: serde::de::Error, - { - Ok(ValuePrimative::Number(v.into())) - } - } - deserializer.deserialize_any(Visitor) - } -} - -#[derive(Clone, Copy, Debug, Deserialize, Serialize)] -#[serde(rename_all = "kebab-case")] -pub enum IoFormat { - Json, - JsonPretty, - Yaml, - Cbor, - Toml, - TomlPretty, -} -impl Default for IoFormat { - fn default() -> Self { - IoFormat::JsonPretty - } -} -impl std::fmt::Display for IoFormat { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - use IoFormat::*; - match self { - Json => write!(f, "JSON"), - JsonPretty => write!(f, "JSON (pretty)"), - Yaml => write!(f, "YAML"), - Cbor => write!(f, "CBOR"), - Toml => write!(f, "TOML"), - TomlPretty => write!(f, "TOML (pretty)"), - } - } -} -impl std::str::FromStr for IoFormat { - type Err = Error; - fn from_str(s: &str) -> Result { - serde_json::from_value(Value::String(s.to_owned())) - .with_kind(crate::ErrorKind::Deserialization) - } -} -impl IoFormat { - pub fn to_writer( - &self, - mut writer: W, - value: &T, - ) -> Result<(), Error> { - match self { - IoFormat::Json => { - serde_json::to_writer(writer, value).with_kind(crate::ErrorKind::Serialization) - } - IoFormat::JsonPretty => serde_json::to_writer_pretty(writer, value) - .with_kind(crate::ErrorKind::Serialization), - IoFormat::Yaml => { - serde_yaml::to_writer(writer, value).with_kind(crate::ErrorKind::Serialization) - } - IoFormat::Cbor => serde_cbor::ser::into_writer(value, writer) - .with_kind(crate::ErrorKind::Serialization), - IoFormat::Toml => writer - .write_all( - serde_toml::to_string( - &serde_toml::Value::try_from(value) - .with_kind(crate::ErrorKind::Serialization)?, - ) - .with_kind(crate::ErrorKind::Serialization)? - .as_bytes(), - ) - .with_kind(crate::ErrorKind::Serialization), - IoFormat::TomlPretty => writer - .write_all( - serde_toml::to_string_pretty( - &serde_toml::Value::try_from(value) - .with_kind(crate::ErrorKind::Serialization)?, - ) - .with_kind(crate::ErrorKind::Serialization)? - .as_bytes(), - ) - .with_kind(crate::ErrorKind::Serialization), - } - } - pub fn to_vec(&self, value: &T) -> Result, Error> { - match self { - IoFormat::Json => serde_json::to_vec(value).with_kind(crate::ErrorKind::Serialization), - IoFormat::JsonPretty => { - serde_json::to_vec_pretty(value).with_kind(crate::ErrorKind::Serialization) - } - IoFormat::Yaml => serde_yaml::to_string(value) - .with_kind(crate::ErrorKind::Serialization) - .map(|s| s.into_bytes()), - IoFormat::Cbor => { - let mut res = Vec::new(); - serde_cbor::ser::into_writer(value, &mut res) - .with_kind(crate::ErrorKind::Serialization)?; - Ok(res) - } - IoFormat::Toml => serde_toml::to_string( - &serde_toml::Value::try_from(value).with_kind(crate::ErrorKind::Serialization)?, - ) - .with_kind(crate::ErrorKind::Serialization) - .map(|s| s.into_bytes()), - IoFormat::TomlPretty => serde_toml::to_string_pretty( - &serde_toml::Value::try_from(value).with_kind(crate::ErrorKind::Serialization)?, - ) - .map(|s| s.into_bytes()) - .with_kind(crate::ErrorKind::Serialization), - } - } - /// BLOCKING - pub fn from_reader Deserialize<'de>>( - &self, - mut reader: R, - ) -> Result { - match self { - IoFormat::Json | IoFormat::JsonPretty => { - serde_json::from_reader(reader).with_kind(crate::ErrorKind::Deserialization) - } - IoFormat::Yaml => { - serde_yaml::from_reader(reader).with_kind(crate::ErrorKind::Deserialization) - } - IoFormat::Cbor => { - serde_cbor::de::from_reader(reader).with_kind(crate::ErrorKind::Deserialization) - } - IoFormat::Toml | IoFormat::TomlPretty => { - let mut s = String::new(); - reader - .read_to_string(&mut s) - .with_kind(crate::ErrorKind::Deserialization)?; - serde_toml::from_str(&s).with_kind(crate::ErrorKind::Deserialization) - } - } - } - pub async fn from_async_reader< - R: tokio::io::AsyncRead + Unpin, - T: for<'de> Deserialize<'de>, - >( - &self, - reader: R, - ) -> Result { - use crate::util::io::*; - match self { - IoFormat::Json | IoFormat::JsonPretty => from_json_async_reader(reader).await, - IoFormat::Yaml => from_yaml_async_reader(reader).await, - IoFormat::Cbor => from_cbor_async_reader(reader).await, - IoFormat::Toml | IoFormat::TomlPretty => from_toml_async_reader(reader).await, - } - } - pub fn from_slice Deserialize<'de>>(&self, slice: &[u8]) -> Result { - match self { - IoFormat::Json | IoFormat::JsonPretty => { - serde_json::from_slice(slice).with_kind(crate::ErrorKind::Deserialization) - } - IoFormat::Yaml => { - serde_yaml::from_slice(slice).with_kind(crate::ErrorKind::Deserialization) - } - IoFormat::Cbor => { - serde_cbor::de::from_reader(slice).with_kind(crate::ErrorKind::Deserialization) - } - IoFormat::Toml | IoFormat::TomlPretty => { - serde_toml::from_str(std::str::from_utf8(slice)?) - .with_kind(crate::ErrorKind::Deserialization) - } - } - } -} - -pub fn display_serializable(t: T, matches: &ArgMatches) { - let format = match matches.value_of("format").map(|f| f.parse()) { - Some(Ok(f)) => f, - Some(Err(_)) => { - eprintln!("unrecognized formatter"); - exit(1) - } - None => IoFormat::default(), - }; - format - .to_writer(std::io::stdout(), &t) - .expect("Error serializing result to stdout") -} - -pub fn parse_stdin_deserializable Deserialize<'de>>( - stdin: &mut std::io::Stdin, - matches: &ArgMatches, -) -> Result { - let format = match matches.value_of("format").map(|f| f.parse()) { - Some(Ok(f)) => f, - Some(Err(_)) => { - eprintln!("unrecognized formatter"); - exit(1) - } - None => IoFormat::default(), - }; - format.from_reader(stdin) -} - -#[derive(Debug, Clone, Copy)] -pub struct Duration(std::time::Duration); -impl Deref for Duration { - type Target = std::time::Duration; - fn deref(&self) -> &Self::Target { - &self.0 - } -} -impl From for Duration { - fn from(t: std::time::Duration) -> Self { - Duration(t) - } -} -impl std::str::FromStr for Duration { - type Err = Error; - fn from_str(s: &str) -> Result { - let units_idx = s.find(|c: char| c.is_alphabetic()).ok_or_else(|| { - Error::new( - eyre!("Must specify units for duration"), - crate::ErrorKind::Deserialization, - ) - })?; - let (num, units) = s.split_at(units_idx); - use std::time::Duration; - Ok(Duration(match units { - "d" if num.contains(".") => Duration::from_secs_f64(num.parse::()? * 86_400_f64), - "d" => Duration::from_secs(num.parse::()? * 86_400), - "h" if num.contains(".") => Duration::from_secs_f64(num.parse::()? * 3_600_f64), - "h" => Duration::from_secs(num.parse::()? * 3_600), - "m" if num.contains(".") => Duration::from_secs_f64(num.parse::()? * 60_f64), - "m" => Duration::from_secs(num.parse::()? * 60), - "s" if num.contains(".") => Duration::from_secs_f64(num.parse()?), - "s" => Duration::from_secs(num.parse()?), - "ms" if num.contains(".") => Duration::from_secs_f64(num.parse::()? / 1_000_f64), - "ms" => { - let millis: u128 = num.parse()?; - Duration::new((millis / 1_000) as u64, (millis % 1_000) as u32) - } - "us" | "µs" if num.contains(".") => { - Duration::from_secs_f64(num.parse::()? / 1_000_000_f64) - } - "us" | "µs" => { - let micros: u128 = num.parse()?; - Duration::new((micros / 1_000_000) as u64, (micros % 1_000_000) as u32) - } - "ns" if num.contains(".") => { - Duration::from_secs_f64(num.parse::()? / 1_000_000_000_f64) - } - "ns" => { - let nanos: u128 = num.parse()?; - Duration::new( - (nanos / 1_000_000_000) as u64, - (nanos % 1_000_000_000) as u32, - ) - } - _ => { - return Err(Error::new( - eyre!("Invalid units for duration"), - crate::ErrorKind::Deserialization, - )) - } - })) - } -} -impl std::fmt::Display for Duration { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - let nanos = self.as_nanos(); - match () { - _ if nanos % 86_400_000_000_000 == 0 => write!(f, "{}d", nanos / 86_400_000_000_000), - _ if nanos % 3_600_000_000_000 == 0 => write!(f, "{}h", nanos / 3_600_000_000_000), - _ if nanos % 60_000_000_000 == 0 => write!(f, "{}m", nanos / 60_000_000_000), - _ if nanos % 1_000_000_000 == 0 => write!(f, "{}s", nanos / 1_000_000_000), - _ if nanos % 1_000_000 == 0 => write!(f, "{}ms", nanos / 1_000_000), - _ if nanos % 1_000 == 0 => write!(f, "{}µs", nanos / 1_000), - _ => write!(f, "{}ns", nanos), - } - } -} -impl<'de> Deserialize<'de> for Duration { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - deserialize_from_str(deserializer) - } -} -impl Serialize for Duration { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serialize_display(self, serializer) - } -} - -pub fn deserialize_number_permissive< - 'de, - D: serde::de::Deserializer<'de>, - T: FromStr + num::cast::FromPrimitive, - E: std::fmt::Display, ->( - deserializer: D, -) -> std::result::Result { - struct Visitor + num::cast::FromPrimitive, E>(std::marker::PhantomData); - impl<'de, T: FromStr + num::cast::FromPrimitive, Err: std::fmt::Display> - serde::de::Visitor<'de> for Visitor - { - type Value = T; - fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(formatter, "a parsable string") - } - fn visit_str(self, v: &str) -> Result - where - E: serde::de::Error, - { - v.parse().map_err(|e| serde::de::Error::custom(e)) - } - fn visit_f64(self, v: f64) -> Result - where - E: serde::de::Error, - { - T::from_f64(v).ok_or_else(|| { - serde::de::Error::custom(format!( - "{} cannot be represented by the requested type", - v - )) - }) - } - fn visit_u64(self, v: u64) -> Result - where - E: serde::de::Error, - { - T::from_u64(v).ok_or_else(|| { - serde::de::Error::custom(format!( - "{} cannot be represented by the requested type", - v - )) - }) - } - fn visit_i64(self, v: i64) -> Result - where - E: serde::de::Error, - { - T::from_i64(v).ok_or_else(|| { - serde::de::Error::custom(format!( - "{} cannot be represented by the requested type", - v - )) - }) - } - } - deserializer.deserialize_str(Visitor(std::marker::PhantomData)) -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct Port(pub u16); -impl<'de> Deserialize<'de> for Port { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - //TODO: if number, be permissive - deserialize_number_permissive(deserializer).map(Port) - } -} -impl Serialize for Port { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serialize_display(&self.0, serializer) - } -} - -#[derive(Debug, Clone)] -pub struct Reversible> -where - for<'a> &'a Container: IntoDoubleEndedIterator<&'a T>, -{ - reversed: bool, - data: Container, - phantom: PhantomData, -} -impl Reversible -where - for<'a> &'a Container: IntoDoubleEndedIterator<&'a T>, -{ - pub fn new(data: Container) -> Self { - Reversible { - reversed: false, - data, - phantom: PhantomData, - } - } - - pub fn reverse(&mut self) { - self.reversed = !self.reversed - } - - pub fn iter( - &self, - ) -> itertools::Either< - <&Container as IntoDoubleEndedIterator<&T>>::IntoIter, - std::iter::Rev<<&Container as IntoDoubleEndedIterator<&T>>::IntoIter>, - > { - let iter = IntoDoubleEndedIterator::into_iter(&self.data); - if self.reversed { - itertools::Either::Right(iter.rev()) - } else { - itertools::Either::Left(iter) - } - } -} -impl Serialize for Reversible -where - for<'a> &'a Container: IntoDoubleEndedIterator<&'a T>, - T: Serialize, -{ - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - let iter = IntoDoubleEndedIterator::into_iter(&self.data); - let mut seq_ser = serializer.serialize_seq(match iter.size_hint() { - (lower, Some(upper)) if lower == upper => Some(upper), - _ => None, - })?; - if self.reversed { - for elem in iter.rev() { - seq_ser.serialize_element(elem)?; - } - } else { - for elem in iter { - seq_ser.serialize_element(elem)?; - } - } - seq_ser.end() - } -} -impl<'de, T, Container> Deserialize<'de> for Reversible -where - for<'a> &'a Container: IntoDoubleEndedIterator<&'a T>, - Container: Deserialize<'de>, -{ - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - Ok(Reversible::new(Deserialize::deserialize(deserializer)?)) - } - fn deserialize_in_place(deserializer: D, place: &mut Self) -> Result<(), D::Error> - where - D: Deserializer<'de>, - { - Deserialize::deserialize_in_place(deserializer, &mut place.data) - } -} - -pub struct KeyVal { - pub key: K, - pub value: V, -} -impl Serialize for KeyVal { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - let mut map = serializer.serialize_map(Some(1))?; - map.serialize_entry(&self.key, &self.value)?; - map.end() - } -} -impl<'de, K: Deserialize<'de>, V: Deserialize<'de>> Deserialize<'de> for KeyVal { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - struct Visitor(PhantomData<(K, V)>); - impl<'de, K: Deserialize<'de>, V: Deserialize<'de>> serde::de::Visitor<'de> for Visitor { - type Value = KeyVal; - fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result { - write!(formatter, "A map with a single element") - } - fn visit_map(self, mut map: A) -> Result - where - A: serde::de::MapAccess<'de>, - { - let (key, value) = map - .next_entry()? - .ok_or_else(|| serde::de::Error::invalid_length(0, &"1"))?; - Ok(KeyVal { key, value }) - } - } - deserializer.deserialize_map(Visitor(PhantomData)) - } -} - -pub struct Base32(pub T); -impl<'de, T: TryFrom>> Deserialize<'de> for Base32 { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = String::deserialize(deserializer)?; - base32::decode(base32::Alphabet::RFC4648 { padding: true }, &s) - .ok_or_else(|| { - serde::de::Error::invalid_value( - serde::de::Unexpected::Str(&s), - &"a valid base32 string", - ) - })? - .try_into() - .map_err(|_| serde::de::Error::custom("invalid length")) - .map(Self) - } -} -impl> Serialize for Base32 { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serializer.serialize_str(&base32::encode( - base32::Alphabet::RFC4648 { padding: true }, - self.0.as_ref(), - )) - } -} - -pub struct Base64(pub T); -impl<'de, T: TryFrom>> Deserialize<'de> for Base64 { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - let s = String::deserialize(deserializer)?; - base64::decode(&s) - .map_err(serde::de::Error::custom)? - .try_into() - .map_err(|_| serde::de::Error::custom("invalid length")) - .map(Self) - } -} -impl> Serialize for Base64 { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serializer.serialize_str(&base64::encode(self.0.as_ref())) - } -} - -#[derive(Clone, Debug)] -pub struct Regex(regex::Regex); -impl From for regex::Regex { - fn from(value: Regex) -> Self { - value.0 - } -} -impl From for Regex { - fn from(value: regex::Regex) -> Self { - Regex(value) - } -} -impl AsRef for Regex { - fn as_ref(&self) -> ®ex::Regex { - &self.0 - } -} -impl AsMut for Regex { - fn as_mut(&mut self) -> &mut regex::Regex { - &mut self.0 - } -} -impl<'de> Deserialize<'de> for Regex { - fn deserialize(deserializer: D) -> Result - where - D: Deserializer<'de>, - { - deserialize_from_str(deserializer).map(Self) - } -} -impl Serialize for Regex { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serialize_display(&self.0, serializer) - } -} diff --git a/core/startos/src/version/mod.rs b/core/startos/src/version/mod.rs deleted file mode 100644 index 4c6f157a..00000000 --- a/core/startos/src/version/mod.rs +++ /dev/null @@ -1,236 +0,0 @@ -use std::cmp::Ordering; - -use async_trait::async_trait; -use color_eyre::eyre::eyre; -use rpc_toolkit::command; -use sqlx::PgPool; - -use crate::prelude::*; -use crate::Error; - -mod v0_3_4; -mod v0_3_4_1; -mod v0_3_4_2; -mod v0_3_4_3; -mod v0_3_4_4; -mod v0_3_5; -mod v0_3_5_1; - -pub type Current = v0_3_5_1::Version; - -#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)] -#[serde(untagged)] -enum Version { - V0_3_4(Wrapper), - V0_3_4_1(Wrapper), - V0_3_4_2(Wrapper), - V0_3_4_3(Wrapper), - V0_3_4_4(Wrapper), - V0_3_5(Wrapper), - V0_3_5_1(Wrapper), - Other(emver::Version), -} - -impl Version { - fn from_util_version(version: crate::util::Version) -> Self { - serde_json::to_value(version.clone()) - .and_then(serde_json::from_value) - .unwrap_or_else(|_e| { - tracing::warn!("Can't deserialize: {:?} and falling back to other", version); - Version::Other(version.into_version()) - }) - } - #[cfg(test)] - fn as_sem_ver(&self) -> emver::Version { - match self { - Version::V0_3_4(Wrapper(x)) => x.semver(), - Version::V0_3_4_1(Wrapper(x)) => x.semver(), - Version::V0_3_4_2(Wrapper(x)) => x.semver(), - Version::V0_3_4_3(Wrapper(x)) => x.semver(), - Version::V0_3_4_4(Wrapper(x)) => x.semver(), - Version::V0_3_5(Wrapper(x)) => x.semver(), - Version::V0_3_5_1(Wrapper(x)) => x.semver(), - Version::Other(x) => x.clone(), - } - } -} - -#[async_trait] -pub trait VersionT -where - Self: Sized + Send + Sync, -{ - type Previous: VersionT; - fn new() -> Self; - fn semver(&self) -> emver::Version; - fn compat(&self) -> &'static emver::VersionRange; - async fn up(&self, db: PatchDb, secrets: &PgPool) -> Result<(), Error>; - async fn down(&self, db: PatchDb, secrets: &PgPool) -> Result<(), Error>; - async fn commit(&self, db: PatchDb) -> Result<(), Error> { - let semver = self.semver().into(); - let compat = self.compat().clone(); - db.mutate(|d| { - d.as_server_info_mut().as_version_mut().ser(&semver)?; - d.as_server_info_mut() - .as_eos_version_compat_mut() - .ser(&compat)?; - Ok(()) - }) - .await?; - Ok(()) - } - async fn migrate_to( - &self, - version: &V, - db: PatchDb, - secrets: &PgPool, - ) -> Result<(), Error> { - match self.semver().cmp(&version.semver()) { - Ordering::Greater => self.rollback_to_unchecked(version, db, secrets).await, - Ordering::Less => version.migrate_from_unchecked(self, db, secrets).await, - Ordering::Equal => Ok(()), - } - } - async fn migrate_from_unchecked( - &self, - version: &V, - db: PatchDb, - secrets: &PgPool, - ) -> Result<(), Error> { - let previous = Self::Previous::new(); - if version.semver() < previous.semver() { - previous - .migrate_from_unchecked(version, db.clone(), secrets) - .await?; - } else if version.semver() > previous.semver() { - return Err(Error::new( - eyre!( - "NO PATH FROM {}, THIS IS LIKELY A MISTAKE IN THE VERSION DEFINITION", - version.semver() - ), - crate::ErrorKind::MigrationFailed, - )); - } - tracing::info!("{} -> {}", previous.semver(), self.semver(),); - self.up(db.clone(), secrets).await?; - self.commit(db).await?; - Ok(()) - } - async fn rollback_to_unchecked( - &self, - version: &V, - db: PatchDb, - secrets: &PgPool, - ) -> Result<(), Error> { - let previous = Self::Previous::new(); - tracing::info!("{} -> {}", self.semver(), previous.semver(),); - self.down(db.clone(), secrets).await?; - previous.commit(db.clone()).await?; - if version.semver() < previous.semver() { - previous.rollback_to_unchecked(version, db, secrets).await?; - } else if version.semver() > previous.semver() { - return Err(Error::new( - eyre!( - "NO PATH TO {}, THIS IS LIKELY A MISTAKE IN THE VERSION DEFINITION", - version.semver() - ), - crate::ErrorKind::MigrationFailed, - )); - } - Ok(()) - } -} -#[derive(Debug, Clone)] -struct Wrapper(T); -impl serde::Serialize for Wrapper -where - T: VersionT, -{ - fn serialize(&self, serializer: S) -> Result { - self.0.semver().serialize(serializer) - } -} -impl<'de, T> serde::Deserialize<'de> for Wrapper -where - T: VersionT, -{ - fn deserialize>(deserializer: D) -> Result { - let v = crate::util::Version::deserialize(deserializer)?; - let version = T::new(); - if *v == version.semver() { - Ok(Wrapper(version)) - } else { - Err(serde::de::Error::custom("Mismatched Version")) - } - } -} - -pub async fn init(db: &PatchDb, secrets: &PgPool) -> Result<(), Error> { - let version = Version::from_util_version(db.peek().await.as_server_info().as_version().de()?); - - match version { - Version::V0_3_4(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::V0_3_4_1(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::V0_3_4_2(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::V0_3_4_3(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::V0_3_4_4(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::V0_3_5(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::V0_3_5_1(v) => v.0.migrate_to(&Current::new(), db.clone(), secrets).await?, - Version::Other(_) => { - return Err(Error::new( - eyre!("Cannot downgrade"), - crate::ErrorKind::InvalidRequest, - )) - } - } - Ok(()) -} - -pub const COMMIT_HASH: &str = - include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/../../GIT_HASH.txt")); - -#[command(rename = "git-info", local, metadata(authenticated = false))] -pub fn git_info() -> Result<&'static str, Error> { - Ok(COMMIT_HASH) -} - -#[cfg(test)] -mod tests { - use proptest::prelude::*; - - use super::*; - - fn em_version() -> impl Strategy { - any::<(usize, usize, usize, usize)>().prop_map(|(major, minor, patch, super_minor)| { - emver::Version::new(major, minor, patch, super_minor) - }) - } - - fn versions() -> impl Strategy { - prop_oneof![ - Just(Version::V0_3_4(Wrapper(v0_3_4::Version::new()))), - Just(Version::V0_3_4_1(Wrapper(v0_3_4_1::Version::new()))), - Just(Version::V0_3_4_2(Wrapper(v0_3_4_2::Version::new()))), - Just(Version::V0_3_4_3(Wrapper(v0_3_4_3::Version::new()))), - Just(Version::V0_3_4_4(Wrapper(v0_3_4_4::Version::new()))), - Just(Version::V0_3_5(Wrapper(v0_3_5::Version::new()))), - Just(Version::V0_3_5_1(Wrapper(v0_3_5_1::Version::new()))), - em_version().prop_map(Version::Other), - ] - } - - proptest! { - #[test] - fn emversion_isomorphic_version(original in em_version()) { - let version = Version::from_util_version(original.clone().into()); - let back = version.as_sem_ver(); - prop_assert_eq!(original, back, "All versions should round trip"); - } - #[test] - fn version_isomorphic_em_version(version in versions()) { - let sem_ver = version.as_sem_ver(); - let back = Version::from_util_version(sem_ver.into()); - prop_assert_eq!(format!("{:?}",version), format!("{:?}", back), "All versions should round trip"); - } - } -} diff --git a/core/startos/src/version/v0_3_4.rs b/core/startos/src/version/v0_3_4.rs deleted file mode 100644 index 944da4c9..00000000 --- a/core/startos/src/version/v0_3_4.rs +++ /dev/null @@ -1,135 +0,0 @@ -use async_trait::async_trait; -use emver::VersionRange; -use itertools::Itertools; -use openssl::hash::MessageDigest; -use ssh_key::public::Ed25519PublicKey; - -use super::*; -use crate::account::AccountInfo; -use crate::hostname::{sync_hostname, Hostname}; -use crate::prelude::*; - -const V0_3_4: emver::Version = emver::Version::new(0, 3, 4, 0); - -lazy_static::lazy_static! { - pub static ref V0_3_0_COMPAT: VersionRange = VersionRange::Conj( - Box::new(VersionRange::Anchor( - emver::GTE, - emver::Version::new(0, 3, 0, 0), - )), - Box::new(VersionRange::Anchor(emver::LTE, Current::new().semver())), - ); -} - -const COMMUNITY_URL: &str = "https://community-registry.start9.com/"; -const MAIN_REGISTRY: &str = "https://registry.start9.com/"; -const COMMUNITY_SERVICES: &[&str] = &[ - "ipfs", - "agora", - "lightning-jet", - "balanceofsatoshis", - "mastodon", - "lndg", - "robosats", - "thunderhub", - "syncthing", - "sphinx-relay", -]; - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = Self; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_4 - } - fn compat(&self) -> &'static VersionRange { - &*V0_3_0_COMPAT - } - async fn up(&self, db: PatchDb, secrets: &PgPool) -> Result<(), Error> { - let mut account = AccountInfo::load(secrets).await?; - let account = db - .mutate(|d| { - d.as_server_info_mut().as_pubkey_mut().ser( - &ssh_key::PublicKey::from(Ed25519PublicKey::from(&account.key.ssh_key())) - .to_openssh()?, - )?; - d.as_server_info_mut().as_ca_fingerprint_mut().ser( - &account - .root_ca_cert - .digest(MessageDigest::sha256()) - .unwrap() - .iter() - .map(|x| format!("{x:X}")) - .join(":"), - )?; - let server_info = d.as_server_info(); - account.hostname = server_info.as_hostname().de().map(Hostname)?; - account.server_id = server_info.as_id().de()?; - - Ok(account) - }) - .await?; - account.save(secrets).await?; - sync_hostname(&account.hostname).await?; - - let parsed_url = Some(COMMUNITY_URL.parse().unwrap()); - db.mutate(|d| { - let mut ui = d.as_ui().de()?; - use patch_db::value::{json, Value}; - ui["marketplace"]["known-hosts"][COMMUNITY_URL] = json!({}); - ui["marketplace"]["known-hosts"][MAIN_REGISTRY] = json!({}); - for package_id in d.as_package_data().keys()? { - if !COMMUNITY_SERVICES.contains(&&*package_id.to_string()) { - continue; - } - d.as_package_data_mut() - .as_idx_mut(&package_id) - .or_not_found(&package_id)? - .as_installed_mut() - .or_not_found(&package_id)? - .as_marketplace_url_mut() - .ser(&parsed_url)?; - } - ui["theme"] = json!("Dark"); - ui["widgets"] = json!([]); - - d.as_ui_mut().ser(&ui) - }) - .await - } - async fn down(&self, db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - db.mutate(|d| { - let mut ui = d.as_ui().de()?; - let parsed_url = Some(MAIN_REGISTRY.parse().unwrap()); - for package_id in d.as_package_data().keys()? { - if !COMMUNITY_SERVICES.contains(&&*package_id.to_string()) { - continue; - } - d.as_package_data_mut() - .as_idx_mut(&package_id) - .or_not_found(&package_id)? - .as_installed_mut() - .or_not_found(&package_id)? - .as_marketplace_url_mut() - .ser(&parsed_url)?; - } - - use patch_db::Value; - if let Value::Object(ref mut obj) = ui { - obj.remove("theme"); - obj.remove("widgets"); - } - - ui["marketplace"]["known-hosts"][COMMUNITY_URL].take(); - ui["marketplace"]["known-hosts"][MAIN_REGISTRY].take(); - d.as_ui_mut().ser(&ui) - }) - .await - } -} diff --git a/core/startos/src/version/v0_3_4_1.rs b/core/startos/src/version/v0_3_4_1.rs deleted file mode 100644 index 915a4723..00000000 --- a/core/startos/src/version/v0_3_4_1.rs +++ /dev/null @@ -1,31 +0,0 @@ -use async_trait::async_trait; -use emver::VersionRange; - -use super::v0_3_4::V0_3_0_COMPAT; -use super::*; -use crate::prelude::*; - -const V0_3_4_1: emver::Version = emver::Version::new(0, 3, 4, 1); - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = v0_3_4::Version; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_4_1 - } - fn compat(&self) -> &'static VersionRange { - &*V0_3_0_COMPAT - } - async fn up(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } - async fn down(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } -} diff --git a/core/startos/src/version/v0_3_4_2.rs b/core/startos/src/version/v0_3_4_2.rs deleted file mode 100644 index 5931b287..00000000 --- a/core/startos/src/version/v0_3_4_2.rs +++ /dev/null @@ -1,31 +0,0 @@ -use async_trait::async_trait; -use emver::VersionRange; - -use super::v0_3_4::V0_3_0_COMPAT; -use super::*; -use crate::prelude::*; - -const V0_3_4_2: emver::Version = emver::Version::new(0, 3, 4, 2); - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = v0_3_4_1::Version; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_4_2 - } - fn compat(&self) -> &'static VersionRange { - &*V0_3_0_COMPAT - } - async fn up(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } - async fn down(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } -} diff --git a/core/startos/src/version/v0_3_4_3.rs b/core/startos/src/version/v0_3_4_3.rs deleted file mode 100644 index d3199e91..00000000 --- a/core/startos/src/version/v0_3_4_3.rs +++ /dev/null @@ -1,31 +0,0 @@ -use async_trait::async_trait; -use emver::VersionRange; - -use super::v0_3_4::V0_3_0_COMPAT; -use super::*; -use crate::prelude::*; - -const V0_3_4_3: emver::Version = emver::Version::new(0, 3, 4, 3); - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = v0_3_4_2::Version; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_4_3 - } - fn compat(&self) -> &'static VersionRange { - &V0_3_0_COMPAT - } - async fn up(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } - async fn down(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } -} diff --git a/core/startos/src/version/v0_3_4_4.rs b/core/startos/src/version/v0_3_4_4.rs deleted file mode 100644 index b6345ca4..00000000 --- a/core/startos/src/version/v0_3_4_4.rs +++ /dev/null @@ -1,43 +0,0 @@ -use async_trait::async_trait; -use emver::VersionRange; -use models::ResultExt; -use sqlx::PgPool; - -use super::v0_3_4::V0_3_0_COMPAT; -use super::{v0_3_4_3, VersionT}; -use crate::prelude::*; - -const V0_3_4_4: emver::Version = emver::Version::new(0, 3, 4, 4); - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = v0_3_4_3::Version; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_4_4 - } - fn compat(&self) -> &'static VersionRange { - &V0_3_0_COMPAT - } - async fn up(&self, db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - db.mutate(|v| { - let tor_address_lens = v.as_server_info_mut().as_tor_address_mut(); - let mut tor_addr = tor_address_lens.de()?; - tor_addr - .set_scheme("https") - .map_err(|_| eyre!("unable to update url scheme to https")) - .with_kind(crate::ErrorKind::ParseUrl)?; - tor_address_lens.ser(&tor_addr) - }) - .await?; - Ok(()) - } - async fn down(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } -} diff --git a/core/startos/src/version/v0_3_5.rs b/core/startos/src/version/v0_3_5.rs deleted file mode 100644 index ba28cd46..00000000 --- a/core/startos/src/version/v0_3_5.rs +++ /dev/null @@ -1,109 +0,0 @@ -use std::collections::BTreeMap; -use std::path::Path; - -use async_trait::async_trait; -use emver::VersionRange; -use models::DataUrl; -use sqlx::PgPool; - -use super::v0_3_4::V0_3_0_COMPAT; -use super::{v0_3_4_4, VersionT}; -use crate::prelude::*; - -const V0_3_5: emver::Version = emver::Version::new(0, 3, 5, 0); - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = v0_3_4_4::Version; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_5 - } - fn compat(&self) -> &'static VersionRange { - &V0_3_0_COMPAT - } - async fn up(&self, db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - let peek = db.peek().await; - let mut url_replacements = BTreeMap::new(); - for (_, pde) in peek.as_package_data().as_entries()? { - for (dependency, info) in pde - .as_installed() - .map(|i| i.as_dependency_info().as_entries()) - .transpose()? - .into_iter() - .flatten() - { - if !url_replacements.contains_key(&dependency) { - url_replacements.insert( - dependency, - DataUrl::from_path( - <&Value>::from(info.as_icon()) - .as_str() - .and_then(|i| i.strip_prefix("/public/package-data/")) - .map(|path| { - Path::new("/embassy-data/package-data/public").join(path) - }) - .unwrap_or_default(), - ) - .await - .unwrap_or_else(|_| { - DataUrl::from_slice( - "image/png", - include_bytes!("../install/package-icon.png"), - ) - }), - ); - } - } - } - let prev_zram = db - .mutate(|v| { - for (_, pde) in v.as_package_data_mut().as_entries_mut()? { - for (dependency, info) in pde - .as_installed_mut() - .map(|i| i.as_dependency_info_mut().as_entries_mut()) - .transpose()? - .into_iter() - .flatten() - { - if let Some(url) = url_replacements.get(&dependency) { - info.as_icon_mut().ser(url)?; - } else { - info.as_icon_mut().ser(&DataUrl::from_slice( - "image/png", - include_bytes!("../install/package-icon.png"), - ))?; - } - let manifest = <&mut Value>::from(&mut *info) - .as_object_mut() - .and_then(|o| o.remove("manifest")); - if let Some(title) = manifest - .as_ref() - .and_then(|m| m.as_object()) - .and_then(|m| m.get("title")) - .and_then(|t| t.as_str()) - .map(|s| s.to_owned()) - { - info.as_title_mut().ser(&title)?; - } else { - info.as_title_mut().ser(&dependency.to_string())?; - } - } - } - v.as_server_info_mut().as_zram_mut().replace(&true) - }) - .await?; - if !prev_zram { - crate::system::enable_zram().await?; - } - Ok(()) - } - async fn down(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } -} diff --git a/core/startos/src/version/v0_3_5_1.rs b/core/startos/src/version/v0_3_5_1.rs deleted file mode 100644 index c004dc8b..00000000 --- a/core/startos/src/version/v0_3_5_1.rs +++ /dev/null @@ -1,32 +0,0 @@ -use async_trait::async_trait; -use emver::VersionRange; -use sqlx::PgPool; - -use super::v0_3_4::V0_3_0_COMPAT; -use super::{v0_3_5, VersionT}; -use crate::prelude::*; - -const V0_3_5_1: emver::Version = emver::Version::new(0, 3, 5, 1); - -#[derive(Clone, Debug)] -pub struct Version; - -#[async_trait] -impl VersionT for Version { - type Previous = v0_3_5::Version; - fn new() -> Self { - Version - } - fn semver(&self) -> emver::Version { - V0_3_5_1 - } - fn compat(&self) -> &'static VersionRange { - &V0_3_0_COMPAT - } - async fn up(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } - async fn down(&self, _db: PatchDb, _secrets: &PgPool) -> Result<(), Error> { - Ok(()) - } -} diff --git a/core/startos/src/volume.rs b/core/startos/src/volume.rs deleted file mode 100644 index 1633b7d1..00000000 --- a/core/startos/src/volume.rs +++ /dev/null @@ -1,240 +0,0 @@ -use std::collections::BTreeMap; -use std::ops::{Deref, DerefMut}; -use std::path::{Path, PathBuf}; - -pub use helpers::script_dir; -pub use models::VolumeId; -use serde::{Deserialize, Serialize}; -use tracing::instrument; - -use crate::context::RpcContext; -use crate::net::interface::{InterfaceId, Interfaces}; -use crate::net::PACKAGE_CERT_PATH; -use crate::prelude::*; -use crate::s9pk::manifest::PackageId; -use crate::util::Version; -use crate::{Error, ResultExt}; - -pub const PKG_VOLUME_DIR: &str = "package-data/volumes"; -pub const BACKUP_DIR: &str = "/media/embassy/backups"; - -#[derive(Clone, Debug, Default, Deserialize, Serialize)] -pub struct Volumes(BTreeMap); -impl Volumes { - #[instrument(skip_all)] - pub fn validate(&self, interfaces: &Interfaces) -> Result<(), Error> { - for (id, volume) in &self.0 { - volume - .validate(interfaces) - .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, format!("Volume {}", id)))?; - if let Volume::Backup { .. } = volume { - return Err(Error::new( - eyre!("Invalid volume type \"backup\""), - ErrorKind::ParseS9pk, - )); // Volume::Backup is for internal use and shouldn't be declared in manifest - } - } - Ok(()) - } - #[instrument(skip_all)] - pub async fn install( - &self, - ctx: &RpcContext, - pkg_id: &PackageId, - version: &Version, - ) -> Result<(), Error> { - for (volume_id, volume) in &self.0 { - volume - .install(&ctx.datadir, pkg_id, version, volume_id) - .await?; // TODO: concurrent? - } - Ok(()) - } - pub fn get_path_for( - &self, - path: &PathBuf, - pkg_id: &PackageId, - version: &Version, - volume_id: &VolumeId, - ) -> Option { - self.0 - .get(volume_id) - .map(|volume| volume.path_for(path, pkg_id, version, volume_id)) - } - pub fn to_readonly(&self) -> Self { - Volumes( - self.0 - .iter() - .map(|(id, volume)| { - let mut volume = volume.clone(); - volume.set_readonly(); - (id.clone(), volume) - }) - .collect(), - ) - } -} -impl Deref for Volumes { - type Target = BTreeMap; - fn deref(&self) -> &Self::Target { - &self.0 - } -} -impl DerefMut for Volumes { - fn deref_mut(&mut self) -> &mut Self::Target { - &mut self.0 - } -} -impl Map for Volumes { - type Key = VolumeId; - type Value = Volume; -} - -pub fn data_dir>(datadir: P, pkg_id: &PackageId, volume_id: &VolumeId) -> PathBuf { - datadir - .as_ref() - .join(PKG_VOLUME_DIR) - .join(pkg_id) - .join("data") - .join(volume_id) -} - -pub fn asset_dir>(datadir: P, pkg_id: &PackageId, version: &Version) -> PathBuf { - datadir - .as_ref() - .join(PKG_VOLUME_DIR) - .join(pkg_id) - .join("assets") - .join(version.as_str()) -} - -pub fn backup_dir(pkg_id: &PackageId) -> PathBuf { - Path::new(BACKUP_DIR).join(pkg_id).join("data") -} - -pub fn cert_dir(pkg_id: &PackageId, interface_id: &InterfaceId) -> PathBuf { - Path::new(PACKAGE_CERT_PATH).join(pkg_id).join(interface_id) -} - -#[derive(Clone, Debug, Deserialize, Serialize)] -#[serde(tag = "type")] -#[serde(rename_all = "kebab-case")] -pub enum Volume { - #[serde(rename_all = "kebab-case")] - Data { - #[serde(skip)] - readonly: bool, - }, - #[serde(rename_all = "kebab-case")] - Assets {}, - #[serde(rename_all = "kebab-case")] - Pointer { - package_id: PackageId, - volume_id: VolumeId, - path: PathBuf, - readonly: bool, - }, - #[serde(rename_all = "kebab-case")] - Certificate { interface_id: InterfaceId }, - #[serde(rename_all = "kebab-case")] - Backup { readonly: bool }, -} -impl Volume { - #[instrument(skip_all)] - pub fn validate(&self, interfaces: &Interfaces) -> Result<(), color_eyre::eyre::Report> { - match self { - Volume::Certificate { interface_id } => { - if !interfaces.0.contains_key(interface_id) { - color_eyre::eyre::bail!("unknown interface: {}", interface_id); - } - } - _ => (), - } - Ok(()) - } - pub async fn install( - &self, - path: &PathBuf, - pkg_id: &PackageId, - version: &Version, - volume_id: &VolumeId, - ) -> Result<(), Error> { - match self { - Volume::Data { .. } => { - tokio::fs::create_dir_all(self.path_for(path, pkg_id, version, volume_id)).await?; - } - _ => (), - } - Ok(()) - } - pub fn path_for( - &self, - data_dir_path: impl AsRef, - pkg_id: &PackageId, - version: &Version, - volume_id: &VolumeId, - ) -> PathBuf { - match self { - Volume::Data { .. } => data_dir(&data_dir_path, pkg_id, volume_id), - Volume::Assets {} => asset_dir(&data_dir_path, pkg_id, version).join(volume_id), - Volume::Pointer { - package_id, - volume_id, - path, - .. - } => data_dir(&data_dir_path, package_id, volume_id).join(if path.is_absolute() { - path.strip_prefix("/").unwrap() - } else { - path.as_ref() - }), - Volume::Certificate { interface_id } => cert_dir(pkg_id, &interface_id), - Volume::Backup { .. } => backup_dir(pkg_id), - } - } - - pub fn pointer_path(&self, data_dir_path: impl AsRef) -> Option { - if let Volume::Pointer { - path, - package_id, - volume_id, - .. - } = self - { - Some( - data_dir(data_dir_path.as_ref(), package_id, volume_id).join( - if path.is_absolute() { - path.strip_prefix("/").unwrap() - } else { - path.as_ref() - }, - ), - ) - } else { - None - } - } - - pub fn set_readonly(&mut self) { - match self { - Volume::Data { readonly } => { - *readonly = true; - } - Volume::Pointer { readonly, .. } => { - *readonly = true; - } - Volume::Backup { readonly } => { - *readonly = true; - } - _ => (), - } - } - pub fn readonly(&self) -> bool { - match self { - Volume::Data { readonly } => *readonly, - Volume::Assets {} => true, - Volume::Pointer { readonly, .. } => *readonly, - Volume::Certificate { .. } => true, - Volume::Backup { readonly } => *readonly, - } - } -} diff --git a/core/startos/startd.service b/core/startos/startd.service deleted file mode 100644 index 894298e5..00000000 --- a/core/startos/startd.service +++ /dev/null @@ -1,19 +0,0 @@ -[Unit] -Description=StartOS Daemon -After=network-online.target -Requires=network-online.target -Wants=avahi-daemon.service - -[Service] -Type=simple -Environment=RUST_LOG=startos=debug,js_engine=debug,patch_db=warn -ExecStart=/usr/bin/startd -Restart=always -RestartSec=3 -ManagedOOMPreference=avoid -CPUAccounting=true -CPUWeight=1000 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target diff --git a/core/startos/test/config-spec/lnd-correct.yaml b/core/startos/test/config-spec/lnd-correct.yaml deleted file mode 100644 index d9acba2e..00000000 --- a/core/startos/test/config-spec/lnd-correct.yaml +++ /dev/null @@ -1,547 +0,0 @@ -control-tor-address: - name: Control Tor Address - description: The Tor address for the control interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: control -peer-tor-address: - name: Peer Tor Address - description: The Tor address for the peer interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: peer -watchtower-tor-address: - name: Watchtower Tor Address - description: The Tor address for the watchtower interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: watchtower -alias: - type: string - name: Alias - description: The public, human-readable name of your Lightning node - nullable: true - placeholder: Enter a value - pattern: ".{1,32}" - pattern-description: Must be at least 1 character and no more than 32 characters -color: - type: string - name: Color - description: The public color dot of your Lightning node - nullable: false - pattern: "[0-9a-fA-F]{6}" - pattern-description: | - Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green, and final two are - blue - default: - charset: "a-f,0-9" - len: 6 -accept-keysend: - type: boolean - name: Accept Keysend - description: | - Allow others to send payments directly to your public key through keysend instead of having to get a new invoice - default: false -accept-amp: - type: boolean - name: Accept Spontaneous AMPs - description: | - If enabled, spontaneous payments through AMP will be accepted. Payments to AMP - invoices will be accepted regardless of this setting. - default: false -reject-htlc: - type: boolean - name: Reject Routing Requests - description: | - If true, LND will not forward any HTLCs that are meant as onward payments. This option will still allow LND to send - HTLCs and receive HTLCs but lnd won't be used as a hop. - default: false -min-chan-size: - type: number - name: Minimum Channel Size - description: | - The smallest channel size that we should accept. Incoming channels smaller than this will be rejected. - nullable: true - range: "[1,16777215]" - integral: true - units: satoshis -max-chan-size: - type: number - name: Maximum Channel Size - description: | - The largest channel size that we should accept. Incoming channels larger than this will be rejected. - For non-Wumbo channels this limit remains 16777215 satoshis by default as specified in BOLT-0002. For wumbo - channels this limit is 1,000,000,000 satoshis (10 BTC). Set this config option explicitly to restrict your maximum - channel size to better align with your risk tolerance. Don't forget to enable Wumbo channels under 'Advanced,' if desired. - nullable: true - range: "[1,1000000000]" - integral: true - units: satoshis -tor: - type: object - name: Tor Config - nullable: false - spec: - use-tor-only: - type: boolean - name: Use Tor for all traffic - description: >- - Use the tor proxy even for connections that are reachable on clearnet. This will hide your node's public IP - address, but will slow down your node's performance - nullable: false - default: false - stream-isolation: - type: boolean - name: Stream Isolation - description: >- - Enable Tor stream isolation by randomizing user credentials for each - connection. With this mode active, each connection will use a new circuit. - This means that multiple applications (other than lnd) using Tor won't be mixed - in with lnd's traffic. - - This option may not be used when 'Use Tor for all traffic' is disabled, since direct - connections compromise source IP privacy by default. - nullable: false - default: false -bitcoind: - type: union - name: Bitcoin Core - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - tag: - id: type - name: Type - variant-names: - internal: Internal (Bitcoin Core) - internal-proxy: Internal (Bitcoin Proxy) - external: External - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - default: internal - variants: - internal: - user: - type: pointer - name: RPC Username - description: The username for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.username" - password: - type: pointer - name: RPC Password - description: The password for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.password" - internal-proxy: - user: - type: pointer - name: RPC Username - description: The username for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].name' - # index: 'users.[first(item => ''item.name = "lnd")].name' - password: - type: pointer - name: RPC Password - description: The password for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].password' - # index: 'users.[first(item => ''item.name = "lnd")].password' - external: - connection-settings: - type: union - name: Connection Settings - description: Information to connect to an external unpruned Bitcoin Core node - tag: - id: type - name: Type - description: | - - Manual: Raw information for finding a Bitcoin Core node - - Quick Connect: A Quick Connect URL for a Bitcoin Core node - variant-names: - manual: Manual - quick-connect: Quick Connect - default: quick-connect - variants: - manual: - host: - type: string - name: Public Address - description: The public address of your Bitcoin Core server - nullable: false - rpc-user: - type: string - name: RPC Username - description: The username for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-password: - type: string - name: RPC Password - description: The password for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-port: - type: number - name: RPC Port - description: The port that your Bitcoin Core RPC server is bound to - nullable: false - range: "[0,65535]" - integral: true - default: 8332 - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 - quick-connect: - quick-connect-url: - type: string - name: Quick Connect URL - description: | - The Quick Connect URL for your Bitcoin Core RPC server - NOTE: LND will not accept a .onion url for this option - nullable: false - pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?' - pattern-description: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 -autopilot: - type: object - name: Autopilot - description: Autopilot Settings - nullable: false - spec: - enabled: - type: boolean - name: Enabled - description: | - If the autopilot agent should be active or not. The autopilot agent will - attempt to AUTOMATICALLY OPEN CHANNELS to put your node in an advantageous - position within the network graph. DO NOT ENABLE THIS IF YOU WANT TO MANAGE - CHANNELS MANUALLY OR DO NOT UNDERSTAND IT. - default: false - private: - type: boolean - name: Private - description: | - Whether the channels created by the autopilot agent should be private or not. - Private channels won't be announced to the network. - default: false - maxchannels: - type: number - name: Maximum Channels - description: The maximum number of channels that should be created. - nullable: false - range: "[1,*)" - integral: true - default: 5 - allocation: - type: number - name: Allocation - description: | - The fraction of total funds that should be committed to automatic channel - establishment. For example 60% means that 60% of the total funds available - within the wallet should be used to automatically establish channels. The total - amount of attempted channels will still respect the "Maximum Channels" parameter. - nullable: false - range: "[0,100]" - integral: false - default: 60 - units: "%" - min-channel-size: - type: number - name: Minimum Channel Size - description: The smallest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 20000 - units: "satoshis" - max-channel-size: - type: number - name: Maximum Channel Size - description: The largest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 16777215 - units: "satoshis" - advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - min-confirmations: - type: number - name: Minimum Confirmations - description: | - The minimum number of confirmations each of your inputs in funding transactions - created by the autopilot agent must have. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks - confirmation-target: - type: number - name: Confirmation Target - description: The confirmation target (in blocks) for channels opened by autopilot. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks -advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - debug-level: - type: enum - name: Log Verbosity - values: - - trace - - debug - - info - - warn - - error - - critical - description: | - Sets the level of log filtration. Trace is the most verbose, Critical is the least. - default: info - db-bolt-no-freelist-sync: - type: boolean - name: Disallow Bolt DB Freelist Sync - description: | - If true, prevents the database from syncing its freelist to disk. - default: false - db-bolt-auto-compact: - type: boolean - name: Compact Database on Startup - description: | - Performs database compaction on startup. This is necessary to keep disk usage down over time at the cost of - having longer startup times. - default: true - db-bolt-auto-compact-min-age: - type: number - name: Minimum Autocompaction Age for Bolt DB - description: | - How long ago (in hours) the last compaction of a database file must be for it to be considered for auto - compaction again. Can be set to 0 to compact on every startup. - nullable: false - range: "[0, *)" - integral: true - default: 168 - units: hours - db-bolt-db-timeout: - type: number - name: Bolt DB Timeout - description: How long should LND try to open the database before giving up? - nullable: false - range: "[1, 86400]" - integral: true - default: 60 - units: seconds - recovery-window: - type: number - name: Recovery Window - description: Number of blocks in the past that LND should scan for unknown transactions - nullable: true - range: "[1,*)" - integral: true - units: "blocks" - payments-expiration-grace-period: - type: number - name: Payments Expiration Grace Period - description: | - A period to wait before for closing channels with outgoing htlcs that have timed out and are a result of this - nodes instead payment. In addition to our current block based deadline, is specified this grace period will - also be taken into account. - nullable: false - range: "[1,*)" - integral: true - default: 30 - units: "seconds" - default-remote-max-htlcs: - type: number - name: Maximum Remote HTLCs - description: | - The default max_htlc applied when opening or accepting channels. This value limits the number of concurrent - HTLCs that the remote party can add to the commitment. The maximum possible value is 483. - nullable: false - range: "[1,483]" - integral: true - default: 483 - units: htlcs - max-channel-fee-allocation: - type: number - name: Maximum Channel Fee Allocation - description: | - The maximum percentage of total funds that can be allocated to a channel's commitment fee. This only applies for - the initiator of the channel. - nullable: false - range: "[0.1, 1]" - integral: false - default: 0.5 - max-commit-fee-rate-anchors: - type: number - name: Maximum Commitment Fee for Anchor Channels - description: | - The maximum fee rate in sat/vbyte that will be used for commitments of channels of the anchors type. Must be - large enough to ensure transaction propagation. - nullable: false - range: "[1,*)" - integral: true - default: 10 - protocol-wumbo-channels: - type: boolean - name: Enable Wumbo Channels - description: | - If set, then lnd will create and accept requests for channels larger than 0.16 BTC - nullable: false - default: false - protocol-no-anchors: - type: boolean - name: Disable Anchor Channels - description: | - Set to disable support for anchor commitments. Anchor channels allow you to determine your fees at close time by - using a Child Pays For Parent transaction. - nullable: false - default: false - protocol-disable-script-enforced-lease: - type: boolean - name: Disable Script Enforced Channel Leases - description: >- - Set to disable support for script enforced lease channel commitments. If not - set, lnd will accept these channels by default if the remote channel party - proposes them. Note that lnd will require 1 UTXO to be reserved for this - channel type if it is enabled. - - Note: This may cause you to be unable to close a channel and your wallets may not understand why - nullable: false - default: false - gc-canceled-invoices-on-startup: - type: boolean - name: Cleanup Canceled Invoices on Startup - description: | - If true, LND will attempt to garbage collect canceled invoices upon start. - nullable: false - default: false - bitcoin: - type: object - name: Bitcoin Channel Configuration - description: Configuration options for lightning network channel management operating over the Bitcoin network - nullable: false - spec: - default-channel-confirmations: - type: number - name: Default Channel Confirmations - description: | - The default number of confirmations a channel must have before it's considered - open. LND will require any incoming channel requests to wait this many - confirmations before it considers the channel active. - nullable: false - range: "[1,6]" - integral: true - default: 3 - units: "blocks" - min-htlc: - type: number - name: Minimum Incoming HTLC Size - description: | - The smallest HTLC LND will to accept on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1 - units: "millisatoshis" - min-htlc-out: - type: number - name: Minimum Outgoing HTLC Size - description: | - The smallest HTLC LND will send out on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1000 - units: "millisatoshis" - base-fee: - type: number - name: Routing Base Fee - description: | - The base fee in millisatoshi you will charge for forwarding payments on your - channels. - nullable: false - range: "[0,*)" - integral: true - default: 1000 - units: "millisatoshi" - fee-rate: - type: number - name: Routing Fee Rate - description: | - The fee rate used when forwarding payments on your channels. The total fee - charged is the Base Fee + (amount * Fee Rate / 1000000), where amount is the - forwarded amount. Measured in sats per million - nullable: false - range: "[1,1000000)" - integral: true - default: 1 - units: "sats per million" - time-lock-delta: - type: number - name: Time Lock Delta - description: The CLTV delta we will subtract from a forwarded HTLC's timelock value. - nullable: false - range: "[6, 144]" - integral: true - default: 40 - units: "blocks" diff --git a/core/startos/test/config-spec/lnd-invalid-regex.yaml b/core/startos/test/config-spec/lnd-invalid-regex.yaml deleted file mode 100644 index 7bd6d677..00000000 --- a/core/startos/test/config-spec/lnd-invalid-regex.yaml +++ /dev/null @@ -1,546 +0,0 @@ -control-tor-address: - name: Control Tor Address - description: The Tor address for the control interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: control -peer-tor-address: - name: Peer Tor Address - description: The Tor address for the peer interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: peer -watchtower-tor-address: - name: Watchtower Tor Address - description: The Tor address for the watchtower interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: watchtower -alias: - type: string - name: Alias - description: The public, human-readable name of your Lightning node - nullable: true - pattern: ".{1,32}" - pattern-description: Must be at least 1 character and no more than 32 characters -color: - type: string - name: Color - description: The public color dot of your Lightning node - nullable: false - pattern: "[0-9a-fA-F]{6" - pattern-description: | - Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green, and final two are - blue - default: - charset: "a-f,0-9" - len: 6 -accept-keysend: - type: boolean - name: Accept Keysend - description: | - Allow others to send payments directly to your public key through keysend instead of having to get a new invoice - default: false -accept-amp: - type: boolean - name: Accept Spontaneous AMPs - description: | - If enabled, spontaneous payments through AMP will be accepted. Payments to AMP - invoices will be accepted regardless of this setting. - default: false -reject-htlc: - type: boolean - name: Reject Routing Requests - description: | - If true, LND will not forward any HTLCs that are meant as onward payments. This option will still allow LND to send - HTLCs and receive HTLCs but lnd won't be used as a hop. - default: false -min-chan-size: - type: number - name: Minimum Channel Size - description: | - The smallest channel size that we should accept. Incoming channels smaller than this will be rejected. - nullable: true - range: "[1,16777215]" - integral: true - units: satoshis -max-chan-size: - type: number - name: Maximum Channel Size - description: | - The largest channel size that we should accept. Incoming channels larger than this will be rejected. - For non-Wumbo channels this limit remains 16777215 satoshis by default as specified in BOLT-0002. For wumbo - channels this limit is 1,000,000,000 satoshis (10 BTC). Set this config option explicitly to restrict your maximum - channel size to better align with your risk tolerance. Don't forget to enable Wumbo channels under 'Advanced,' if desired. - nullable: true - range: "[1,1000000000]" - integral: true - units: satoshis -tor: - type: object - name: Tor Config - nullable: false - spec: - use-tor-only: - type: boolean - name: Use Tor for all traffic - description: >- - Use the tor proxy even for connections that are reachable on clearnet. This will hide your node's public IP - address, but will slow down your node's performance - nullable: false - default: false - stream-isolation: - type: boolean - name: Stream Isolation - description: >- - Enable Tor stream isolation by randomizing user credentials for each - connection. With this mode active, each connection will use a new circuit. - This means that multiple applications (other than lnd) using Tor won't be mixed - in with lnd's traffic. - - This option may not be used when 'Use Tor for all traffic' is disabled, since direct - connections compromise source IP privacy by default. - nullable: false - default: false -bitcoind: - type: union - name: Bitcoin Core - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - tag: - id: type - name: Type - variant-names: - internal: Internal (Bitcoin Core) - internal-proxy: Internal (Bitcoin Proxy) - external: External - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - default: internal - variants: - internal: - user: - type: pointer - name: RPC Username - description: The username for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.username" - password: - type: pointer - name: RPC Password - description: The password for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.password" - internal-proxy: - user: - type: pointer - name: RPC Username - description: The username for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].name' - # index: 'users.[first(item => ''item.name = "lnd")].name' - password: - type: pointer - name: RPC Password - description: The password for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].password' - # index: 'users.[first(item => ''item.name = "lnd")].password' - external: - connection-settings: - type: union - name: Connection Settings - description: Information to connect to an external unpruned Bitcoin Core node - tag: - id: type - name: Type - description: | - - Manual: Raw information for finding a Bitcoin Core node - - Quick Connect: A Quick Connect URL for a Bitcoin Core node - variant-names: - manual: Manual - quick-connect: Quick Connect - default: quick-connect - variants: - manual: - host: - type: string - name: Public Address - description: The public address of your Bitcoin Core server - nullable: false - rpc-user: - type: string - name: RPC Username - description: The username for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-password: - type: string - name: RPC Password - description: The password for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-port: - type: number - name: RPC Port - description: The port that your Bitcoin Core RPC server is bound to - nullable: false - range: "[0,65535]" - integral: true - default: 8332 - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 - quick-connect: - quick-connect-url: - type: string - name: Quick Connect URL - description: | - The Quick Connect URL for your Bitcoin Core RPC server - NOTE: LND will not accept a .onion url for this option - nullable: false - pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?' - pattern-description: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 -autopilot: - type: object - name: Autopilot - description: Autopilot Settings - nullable: false - spec: - enabled: - type: boolean - name: Enabled - description: | - If the autopilot agent should be active or not. The autopilot agent will - attempt to AUTOMATICALLY OPEN CHANNELS to put your node in an advantageous - position within the network graph. DO NOT ENABLE THIS IF YOU WANT TO MANAGE - CHANNELS MANUALLY OR DO NOT UNDERSTAND IT. - default: false - private: - type: boolean - name: Private - description: | - Whether the channels created by the autopilot agent should be private or not. - Private channels won't be announced to the network. - default: false - maxchannels: - type: number - name: Maximum Channels - description: The maximum number of channels that should be created. - nullable: false - range: "[1,*)" - integral: true - default: 5 - allocation: - type: number - name: Allocation - description: | - The fraction of total funds that should be committed to automatic channel - establishment. For example 60% means that 60% of the total funds available - within the wallet should be used to automatically establish channels. The total - amount of attempted channels will still respect the "Maximum Channels" parameter. - nullable: false - range: "[0,100]" - integral: false - default: 60 - units: "%" - min-channel-size: - type: number - name: Minimum Channel Size - description: The smallest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 20000 - units: "satoshis" - max-channel-size: - type: number - name: Maximum Channel Size - description: The largest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 16777215 - units: "satoshis" - advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - min-confirmations: - type: number - name: Minimum Confirmations - description: | - The minimum number of confirmations each of your inputs in funding transactions - created by the autopilot agent must have. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks - confirmation-target: - type: number - name: Confirmation Target - description: The confirmation target (in blocks) for channels opened by autopilot. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks -advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - debug-level: - type: enum - name: Log Verbosity - values: - - trace - - debug - - info - - warn - - error - - critical - description: | - Sets the level of log filtration. Trace is the most verbose, Critical is the least. - default: info - db-bolt-no-freelist-sync: - type: boolean - name: Disallow Bolt DB Freelist Sync - description: | - If true, prevents the database from syncing its freelist to disk. - default: false - db-bolt-auto-compact: - type: boolean - name: Compact Database on Startup - description: | - Performs database compaction on startup. This is necessary to keep disk usage down over time at the cost of - having longer startup times. - default: true - db-bolt-auto-compact-min-age: - type: number - name: Minimum Autocompaction Age for Bolt DB - description: | - How long ago (in hours) the last compaction of a database file must be for it to be considered for auto - compaction again. Can be set to 0 to compact on every startup. - nullable: false - range: "[0, *)" - integral: true - default: 168 - units: hours - db-bolt-db-timeout: - type: number - name: Bolt DB Timeout - description: How long should LND try to open the database before giving up? - nullable: false - range: "[1, 86400]" - integral: true - default: 60 - units: seconds - recovery-window: - type: number - name: Recovery Window - description: Number of blocks in the past that LND should scan for unknown transactions - nullable: true - range: "[1,*)" - integral: true - units: "blocks" - payments-expiration-grace-period: - type: number - name: Payments Expiration Grace Period - description: | - A period to wait before for closing channels with outgoing htlcs that have timed out and are a result of this - nodes instead payment. In addition to our current block based deadline, is specified this grace period will - also be taken into account. - nullable: false - range: "[1,*)" - integral: true - default: 30 - units: "seconds" - default-remote-max-htlcs: - type: number - name: Maximum Remote HTLCs - description: | - The default max_htlc applied when opening or accepting channels. This value limits the number of concurrent - HTLCs that the remote party can add to the commitment. The maximum possible value is 483. - nullable: false - range: "[1,483]" - integral: true - default: 483 - units: htlcs - max-channel-fee-allocation: - type: number - name: Maximum Channel Fee Allocation - description: | - The maximum percentage of total funds that can be allocated to a channel's commitment fee. This only applies for - the initiator of the channel. - nullable: false - range: "[0.1, 1]" - integral: false - default: 0.5 - max-commit-fee-rate-anchors: - type: number - name: Maximum Commitment Fee for Anchor Channels - description: | - The maximum fee rate in sat/vbyte that will be used for commitments of channels of the anchors type. Must be - large enough to ensure transaction propagation. - nullable: false - range: "[1,*)" - integral: true - default: 10 - protocol-wumbo-channels: - type: boolean - name: Enable Wumbo Channels - description: | - If set, then lnd will create and accept requests for channels larger than 0.16 BTC - nullable: false - default: false - protocol-no-anchors: - type: boolean - name: Disable Anchor Channels - description: | - Set to disable support for anchor commitments. Anchor channels allow you to determine your fees at close time by - using a Child Pays For Parent transaction. - nullable: false - default: false - protocol-disable-script-enforced-lease: - type: boolean - name: Disable Script Enforced Channel Leases - description: >- - Set to disable support for script enforced lease channel commitments. If not - set, lnd will accept these channels by default if the remote channel party - proposes them. Note that lnd will require 1 UTXO to be reserved for this - channel type if it is enabled. - - Note: This may cause you to be unable to close a channel and your wallets may not understand why - nullable: false - default: false - gc-canceled-invoices-on-startup: - type: boolean - name: Cleanup Canceled Invoices on Startup - description: | - If true, LND will attempt to garbage collect canceled invoices upon start. - nullable: false - default: false - bitcoin: - type: object - name: Bitcoin Channel Configuration - description: Configuration options for lightning network channel management operating over the Bitcoin network - nullable: false - spec: - default-channel-confirmations: - type: number - name: Default Channel Confirmations - description: | - The default number of confirmations a channel must have before it's considered - open. LND will require any incoming channel requests to wait this many - confirmations before it considers the channel active. - nullable: false - range: "[1,6]" - integral: true - default: 3 - units: "blocks" - min-htlc: - type: number - name: Minimum Incoming HTLC Size - description: | - The smallest HTLC LND will to accept on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1 - units: "millisatoshis" - min-htlc-out: - type: number - name: Minimum Outgoing HTLC Size - description: | - The smallest HTLC LND will send out on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1000 - units: "millisatoshis" - base-fee: - type: number - name: Routing Base Fee - description: | - The base fee in millisatoshi you will charge for forwarding payments on your - channels. - nullable: false - range: "[0,*)" - integral: true - default: 1000 - units: "millisatoshi" - fee-rate: - type: number - name: Routing Fee Rate - description: | - The fee rate used when forwarding payments on your channels. The total fee - charged is the Base Fee + (amount * Fee Rate / 1000000), where amount is the - forwarded amount. Measured in sats per million - nullable: false - range: "[1,1000000)" - integral: true - default: 1 - units: "sats per million" - time-lock-delta: - type: number - name: Time Lock Delta - description: The CLTV delta we will subtract from a forwarded HTLC's timelock value. - nullable: false - range: "[6, 144]" - integral: true - default: 40 - units: "blocks" diff --git a/core/startos/test/config-spec/lnd-missing-pattern-description.yaml b/core/startos/test/config-spec/lnd-missing-pattern-description.yaml deleted file mode 100644 index 50e8b9d0..00000000 --- a/core/startos/test/config-spec/lnd-missing-pattern-description.yaml +++ /dev/null @@ -1,545 +0,0 @@ -control-tor-address: - name: Control Tor Address - description: The Tor address for the control interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: control -peer-tor-address: - name: Peer Tor Address - description: The Tor address for the peer interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: peer -watchtower-tor-address: - name: Watchtower Tor Address - description: The Tor address for the watchtower interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: watchtower -alias: - type: string - name: Alias - description: The public, human-readable name of your Lightning node - nullable: true - pattern: ".{1,32}" - pattern-description: Must be at least 1 character and no more than 32 characters -color: - type: string - name: Color - description: The public color dot of your Lightning node - nullable: false - pattern: "[0-9a-fA-F]{6" - pattern-description: | - Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green, and final two are - blue - default: - charset: "a-f,0-9" - len: 6 -accept-keysend: - type: boolean - name: Accept Keysend - description: | - Allow others to send payments directly to your public key through keysend instead of having to get a new invoice - default: false -accept-amp: - type: boolean - name: Accept Spontaneous AMPs - description: | - If enabled, spontaneous payments through AMP will be accepted. Payments to AMP - invoices will be accepted regardless of this setting. - default: false -reject-htlc: - type: boolean - name: Reject Routing Requests - description: | - If true, LND will not forward any HTLCs that are meant as onward payments. This option will still allow LND to send - HTLCs and receive HTLCs but lnd won't be used as a hop. - default: false -min-chan-size: - type: number - name: Minimum Channel Size - description: | - The smallest channel size that we should accept. Incoming channels smaller than this will be rejected. - nullable: true - range: "[1,16777215]" - integral: true - units: satoshis -max-chan-size: - type: number - name: Maximum Channel Size - description: | - The largest channel size that we should accept. Incoming channels larger than this will be rejected. - For non-Wumbo channels this limit remains 16777215 satoshis by default as specified in BOLT-0002. For wumbo - channels this limit is 1,000,000,000 satoshis (10 BTC). Set this config option explicitly to restrict your maximum - channel size to better align with your risk tolerance. Don't forget to enable Wumbo channels under 'Advanced,' if desired. - nullable: true - range: "[1,1000000000]" - integral: true - units: satoshis -tor: - type: object - name: Tor Config - nullable: false - spec: - use-tor-only: - type: boolean - name: Use Tor for all traffic - description: >- - Use the tor proxy even for connections that are reachable on clearnet. This will hide your node's public IP - address, but will slow down your node's performance - nullable: false - default: false - stream-isolation: - type: boolean - name: Stream Isolation - description: >- - Enable Tor stream isolation by randomizing user credentials for each - connection. With this mode active, each connection will use a new circuit. - This means that multiple applications (other than lnd) using Tor won't be mixed - in with lnd's traffic. - - This option may not be used when 'Use Tor for all traffic' is disabled, since direct - connections compromise source IP privacy by default. - nullable: false - default: false -bitcoind: - type: union - name: Bitcoin Core - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - tag: - id: type - name: Type - variant-names: - internal: Internal (Bitcoin Core) - internal-proxy: Internal (Bitcoin Proxy) - external: External - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - default: internal - variants: - internal: - user: - type: pointer - name: RPC Username - description: The username for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.username" - password: - type: pointer - name: RPC Password - description: The password for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.password" - internal-proxy: - user: - type: pointer - name: RPC Username - description: The username for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].name' - # index: 'users.[first(item => ''item.name = "lnd")].name' - password: - type: pointer - name: RPC Password - description: The password for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].password' - # index: 'users.[first(item => ''item.name = "lnd")].password' - external: - connection-settings: - type: union - name: Connection Settings - description: Information to connect to an external unpruned Bitcoin Core node - tag: - id: type - name: Type - description: | - - Manual: Raw information for finding a Bitcoin Core node - - Quick Connect: A Quick Connect URL for a Bitcoin Core node - variant-names: - manual: Manual - quick-connect: Quick Connect - default: quick-connect - variants: - manual: - host: - type: string - name: Public Address - description: The public address of your Bitcoin Core server - nullable: false - rpc-user: - type: string - name: RPC Username - description: The username for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-password: - type: string - name: RPC Password - description: The password for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-port: - type: number - name: RPC Port - description: The port that your Bitcoin Core RPC server is bound to - nullable: false - range: "[0,65535]" - integral: true - default: 8332 - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 - quick-connect: - quick-connect-url: - type: string - name: Quick Connect URL - description: | - The Quick Connect URL for your Bitcoin Core RPC server - NOTE: LND will not accept a .onion url for this option - nullable: false - pattern: 'btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\?(label=.+)?)?)?' - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 -autopilot: - type: object - name: Autopilot - description: Autopilot Settings - nullable: false - spec: - enabled: - type: boolean - name: Enabled - description: | - If the autopilot agent should be active or not. The autopilot agent will - attempt to AUTOMATICALLY OPEN CHANNELS to put your node in an advantageous - position within the network graph. DO NOT ENABLE THIS IF YOU WANT TO MANAGE - CHANNELS MANUALLY OR DO NOT UNDERSTAND IT. - default: false - private: - type: boolean - name: Private - description: | - Whether the channels created by the autopilot agent should be private or not. - Private channels won't be announced to the network. - default: false - maxchannels: - type: number - name: Maximum Channels - description: The maximum number of channels that should be created. - nullable: false - range: "[1,*)" - integral: true - default: 5 - allocation: - type: number - name: Allocation - description: | - The fraction of total funds that should be committed to automatic channel - establishment. For example 60% means that 60% of the total funds available - within the wallet should be used to automatically establish channels. The total - amount of attempted channels will still respect the "Maximum Channels" parameter. - nullable: false - range: "[0,100]" - integral: false - default: 60 - units: "%" - min-channel-size: - type: number - name: Minimum Channel Size - description: The smallest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 20000 - units: "satoshis" - max-channel-size: - type: number - name: Maximum Channel Size - description: The largest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 16777215 - units: "satoshis" - advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - min-confirmations: - type: number - name: Minimum Confirmations - description: | - The minimum number of confirmations each of your inputs in funding transactions - created by the autopilot agent must have. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks - confirmation-target: - type: number - name: Confirmation Target - description: The confirmation target (in blocks) for channels opened by autopilot. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks -advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - debug-level: - type: enum - name: Log Verbosity - values: - - trace - - debug - - info - - warn - - error - - critical - description: | - Sets the level of log filtration. Trace is the most verbose, Critical is the least. - default: info - db-bolt-no-freelist-sync: - type: boolean - name: Disallow Bolt DB Freelist Sync - description: | - If true, prevents the database from syncing its freelist to disk. - default: false - db-bolt-auto-compact: - type: boolean - name: Compact Database on Startup - description: | - Performs database compaction on startup. This is necessary to keep disk usage down over time at the cost of - having longer startup times. - default: true - db-bolt-auto-compact-min-age: - type: number - name: Minimum Autocompaction Age for Bolt DB - description: | - How long ago (in hours) the last compaction of a database file must be for it to be considered for auto - compaction again. Can be set to 0 to compact on every startup. - nullable: false - range: "[0, *)" - integral: true - default: 168 - units: hours - db-bolt-db-timeout: - type: number - name: Bolt DB Timeout - description: How long should LND try to open the database before giving up? - nullable: false - range: "[1, 86400]" - integral: true - default: 60 - units: seconds - recovery-window: - type: number - name: Recovery Window - description: Number of blocks in the past that LND should scan for unknown transactions - nullable: true - range: "[1,*)" - integral: true - units: "blocks" - payments-expiration-grace-period: - type: number - name: Payments Expiration Grace Period - description: | - A period to wait before for closing channels with outgoing htlcs that have timed out and are a result of this - nodes instead payment. In addition to our current block based deadline, is specified this grace period will - also be taken into account. - nullable: false - range: "[1,*)" - integral: true - default: 30 - units: "seconds" - default-remote-max-htlcs: - type: number - name: Maximum Remote HTLCs - description: | - The default max_htlc applied when opening or accepting channels. This value limits the number of concurrent - HTLCs that the remote party can add to the commitment. The maximum possible value is 483. - nullable: false - range: "[1,483]" - integral: true - default: 483 - units: htlcs - max-channel-fee-allocation: - type: number - name: Maximum Channel Fee Allocation - description: | - The maximum percentage of total funds that can be allocated to a channel's commitment fee. This only applies for - the initiator of the channel. - nullable: false - range: "[0.1, 1]" - integral: false - default: 0.5 - max-commit-fee-rate-anchors: - type: number - name: Maximum Commitment Fee for Anchor Channels - description: | - The maximum fee rate in sat/vbyte that will be used for commitments of channels of the anchors type. Must be - large enough to ensure transaction propagation. - nullable: false - range: "[1,*)" - integral: true - default: 10 - protocol-wumbo-channels: - type: boolean - name: Enable Wumbo Channels - description: | - If set, then lnd will create and accept requests for channels larger than 0.16 BTC - nullable: false - default: false - protocol-no-anchors: - type: boolean - name: Disable Anchor Channels - description: | - Set to disable support for anchor commitments. Anchor channels allow you to determine your fees at close time by - using a Child Pays For Parent transaction. - nullable: false - default: false - protocol-disable-script-enforced-lease: - type: boolean - name: Disable Script Enforced Channel Leases - description: >- - Set to disable support for script enforced lease channel commitments. If not - set, lnd will accept these channels by default if the remote channel party - proposes them. Note that lnd will require 1 UTXO to be reserved for this - channel type if it is enabled. - - Note: This may cause you to be unable to close a channel and your wallets may not understand why - nullable: false - default: false - gc-canceled-invoices-on-startup: - type: boolean - name: Cleanup Canceled Invoices on Startup - description: | - If true, LND will attempt to garbage collect canceled invoices upon start. - nullable: false - default: false - bitcoin: - type: object - name: Bitcoin Channel Configuration - description: Configuration options for lightning network channel management operating over the Bitcoin network - nullable: false - spec: - default-channel-confirmations: - type: number - name: Default Channel Confirmations - description: | - The default number of confirmations a channel must have before it's considered - open. LND will require any incoming channel requests to wait this many - confirmations before it considers the channel active. - nullable: false - range: "[1,6]" - integral: true - default: 3 - units: "blocks" - min-htlc: - type: number - name: Minimum Incoming HTLC Size - description: | - The smallest HTLC LND will to accept on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1 - units: "millisatoshis" - min-htlc-out: - type: number - name: Minimum Outgoing HTLC Size - description: | - The smallest HTLC LND will send out on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1000 - units: "millisatoshis" - base-fee: - type: number - name: Routing Base Fee - description: | - The base fee in millisatoshi you will charge for forwarding payments on your - channels. - nullable: false - range: "[0,*)" - integral: true - default: 1000 - units: "millisatoshi" - fee-rate: - type: number - name: Routing Fee Rate - description: | - The fee rate used when forwarding payments on your channels. The total fee - charged is the Base Fee + (amount * Fee Rate / 1000000), where amount is the - forwarded amount. Measured in sats per million - nullable: false - range: "[1,1000000)" - integral: true - default: 1 - units: "sats per million" - time-lock-delta: - type: number - name: Time Lock Delta - description: The CLTV delta we will subtract from a forwarded HTLC's timelock value. - nullable: false - range: "[6, 144]" - integral: true - default: 40 - units: "blocks" diff --git a/core/startos/test/config-spec/lnd-missing-pattern.yaml b/core/startos/test/config-spec/lnd-missing-pattern.yaml deleted file mode 100644 index dce276b0..00000000 --- a/core/startos/test/config-spec/lnd-missing-pattern.yaml +++ /dev/null @@ -1,545 +0,0 @@ -control-tor-address: - name: Control Tor Address - description: The Tor address for the control interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: control -peer-tor-address: - name: Peer Tor Address - description: The Tor address for the peer interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: peer -watchtower-tor-address: - name: Watchtower Tor Address - description: The Tor address for the watchtower interface. - type: pointer - subtype: package - package-id: lnd - target: tor-address - interface: watchtower -alias: - type: string - name: Alias - description: The public, human-readable name of your Lightning node - nullable: true - pattern: ".{1,32}" - pattern-description: Must be at least 1 character and no more than 32 characters -color: - type: string - name: Color - description: The public color dot of your Lightning node - nullable: false - pattern: "[0-9a-fA-F]{6" - pattern-description: | - Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green, and final two are - blue - default: - charset: "a-f,0-9" - len: 6 -accept-keysend: - type: boolean - name: Accept Keysend - description: | - Allow others to send payments directly to your public key through keysend instead of having to get a new invoice - default: false -accept-amp: - type: boolean - name: Accept Spontaneous AMPs - description: | - If enabled, spontaneous payments through AMP will be accepted. Payments to AMP - invoices will be accepted regardless of this setting. - default: false -reject-htlc: - type: boolean - name: Reject Routing Requests - description: | - If true, LND will not forward any HTLCs that are meant as onward payments. This option will still allow LND to send - HTLCs and receive HTLCs but lnd won't be used as a hop. - default: false -min-chan-size: - type: number - name: Minimum Channel Size - description: | - The smallest channel size that we should accept. Incoming channels smaller than this will be rejected. - nullable: true - range: "[1,16777215]" - integral: true - units: satoshis -max-chan-size: - type: number - name: Maximum Channel Size - description: | - The largest channel size that we should accept. Incoming channels larger than this will be rejected. - For non-Wumbo channels this limit remains 16777215 satoshis by default as specified in BOLT-0002. For wumbo - channels this limit is 1,000,000,000 satoshis (10 BTC). Set this config option explicitly to restrict your maximum - channel size to better align with your risk tolerance. Don't forget to enable Wumbo channels under 'Advanced,' if desired. - nullable: true - range: "[1,1000000000]" - integral: true - units: satoshis -tor: - type: object - name: Tor Config - nullable: false - spec: - use-tor-only: - type: boolean - name: Use Tor for all traffic - description: >- - Use the tor proxy even for connections that are reachable on clearnet. This will hide your node's public IP - address, but will slow down your node's performance - nullable: false - default: false - stream-isolation: - type: boolean - name: Stream Isolation - description: >- - Enable Tor stream isolation by randomizing user credentials for each - connection. With this mode active, each connection will use a new circuit. - This means that multiple applications (other than lnd) using Tor won't be mixed - in with lnd's traffic. - - This option may not be used when 'Use Tor for all traffic' is disabled, since direct - connections compromise source IP privacy by default. - nullable: false - default: false -bitcoind: - type: union - name: Bitcoin Core - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - tag: - id: type - name: Type - variant-names: - internal: Internal (Bitcoin Core) - internal-proxy: Internal (Bitcoin Proxy) - external: External - description: | - The Bitcoin Core node to connect to: - - internal: The Bitcoin Core and Proxy services installed to your Embassy - - external: An unpruned Bitcoin Core node running on a different device - default: internal - variants: - internal: - user: - type: pointer - name: RPC Username - description: The username for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.username" - password: - type: pointer - name: RPC Password - description: The password for Bitcoin Core's RPC interface - subtype: package - package-id: bitcoind - target: config - multi: false - selector: "$.rpc.password" - internal-proxy: - user: - type: pointer - name: RPC Username - description: The username for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].name' - # index: 'users.[first(item => ''item.name = "lnd")].name' - password: - type: pointer - name: RPC Password - description: The password for the RPC user allocated to lnd - subtype: package - package-id: btc-rpc-proxy - target: config - multi: false - selector: '$.users[?(@.name == "lnd")].password' - # index: 'users.[first(item => ''item.name = "lnd")].password' - external: - connection-settings: - type: union - name: Connection Settings - description: Information to connect to an external unpruned Bitcoin Core node - tag: - id: type - name: Type - description: | - - Manual: Raw information for finding a Bitcoin Core node - - Quick Connect: A Quick Connect URL for a Bitcoin Core node - variant-names: - manual: Manual - quick-connect: Quick Connect - default: quick-connect - variants: - manual: - host: - type: string - name: Public Address - description: The public address of your Bitcoin Core server - nullable: false - rpc-user: - type: string - name: RPC Username - description: The username for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-password: - type: string - name: RPC Password - description: The password for the RPC user on your Bitcoin Core RPC server - nullable: false - rpc-port: - type: number - name: RPC Port - description: The port that your Bitcoin Core RPC server is bound to - nullable: false - range: "[0,65535]" - integral: true - default: 8332 - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 - quick-connect: - quick-connect-url: - type: string - name: Quick Connect URL - description: | - The Quick Connect URL for your Bitcoin Core RPC server - NOTE: LND will not accept a .onion url for this option - nullable: false - pattern-description: Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md - zmq-block-port: - type: number - name: ZeroMQ Block Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks - nullable: false - range: "[0,65535]" - integral: true - default: 28332 - zmq-tx-port: - type: number - name: ZeroMQ Transaction Port - description: The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions - nullable: false - range: "[0,65535]" - integral: true - default: 28333 -autopilot: - type: object - name: Autopilot - description: Autopilot Settings - nullable: false - spec: - enabled: - type: boolean - name: Enabled - description: | - If the autopilot agent should be active or not. The autopilot agent will - attempt to AUTOMATICALLY OPEN CHANNELS to put your node in an advantageous - position within the network graph. DO NOT ENABLE THIS IF YOU WANT TO MANAGE - CHANNELS MANUALLY OR DO NOT UNDERSTAND IT. - default: false - private: - type: boolean - name: Private - description: | - Whether the channels created by the autopilot agent should be private or not. - Private channels won't be announced to the network. - default: false - maxchannels: - type: number - name: Maximum Channels - description: The maximum number of channels that should be created. - nullable: false - range: "[1,*)" - integral: true - default: 5 - allocation: - type: number - name: Allocation - description: | - The fraction of total funds that should be committed to automatic channel - establishment. For example 60% means that 60% of the total funds available - within the wallet should be used to automatically establish channels. The total - amount of attempted channels will still respect the "Maximum Channels" parameter. - nullable: false - range: "[0,100]" - integral: false - default: 60 - units: "%" - min-channel-size: - type: number - name: Minimum Channel Size - description: The smallest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 20000 - units: "satoshis" - max-channel-size: - type: number - name: Maximum Channel Size - description: The largest channel that the autopilot agent should create. - nullable: false - range: "[0,*)" - integral: true - default: 16777215 - units: "satoshis" - advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - min-confirmations: - type: number - name: Minimum Confirmations - description: | - The minimum number of confirmations each of your inputs in funding transactions - created by the autopilot agent must have. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks - confirmation-target: - type: number - name: Confirmation Target - description: The confirmation target (in blocks) for channels opened by autopilot. - nullable: false - range: "[0,*)" - integral: true - default: 1 - units: blocks -advanced: - type: object - name: Advanced - description: Advanced Options - nullable: false - spec: - debug-level: - type: enum - name: Log Verbosity - values: - - trace - - debug - - info - - warn - - error - - critical - description: | - Sets the level of log filtration. Trace is the most verbose, Critical is the least. - default: info - db-bolt-no-freelist-sync: - type: boolean - name: Disallow Bolt DB Freelist Sync - description: | - If true, prevents the database from syncing its freelist to disk. - default: false - db-bolt-auto-compact: - type: boolean - name: Compact Database on Startup - description: | - Performs database compaction on startup. This is necessary to keep disk usage down over time at the cost of - having longer startup times. - default: true - db-bolt-auto-compact-min-age: - type: number - name: Minimum Autocompaction Age for Bolt DB - description: | - How long ago (in hours) the last compaction of a database file must be for it to be considered for auto - compaction again. Can be set to 0 to compact on every startup. - nullable: false - range: "[0, *)" - integral: true - default: 168 - units: hours - db-bolt-db-timeout: - type: number - name: Bolt DB Timeout - description: How long should LND try to open the database before giving up? - nullable: false - range: "[1, 86400]" - integral: true - default: 60 - units: seconds - recovery-window: - type: number - name: Recovery Window - description: Number of blocks in the past that LND should scan for unknown transactions - nullable: true - range: "[1,*)" - integral: true - units: "blocks" - payments-expiration-grace-period: - type: number - name: Payments Expiration Grace Period - description: | - A period to wait before for closing channels with outgoing htlcs that have timed out and are a result of this - nodes instead payment. In addition to our current block based deadline, is specified this grace period will - also be taken into account. - nullable: false - range: "[1,*)" - integral: true - default: 30 - units: "seconds" - default-remote-max-htlcs: - type: number - name: Maximum Remote HTLCs - description: | - The default max_htlc applied when opening or accepting channels. This value limits the number of concurrent - HTLCs that the remote party can add to the commitment. The maximum possible value is 483. - nullable: false - range: "[1,483]" - integral: true - default: 483 - units: htlcs - max-channel-fee-allocation: - type: number - name: Maximum Channel Fee Allocation - description: | - The maximum percentage of total funds that can be allocated to a channel's commitment fee. This only applies for - the initiator of the channel. - nullable: false - range: "[0.1, 1]" - integral: false - default: 0.5 - max-commit-fee-rate-anchors: - type: number - name: Maximum Commitment Fee for Anchor Channels - description: | - The maximum fee rate in sat/vbyte that will be used for commitments of channels of the anchors type. Must be - large enough to ensure transaction propagation. - nullable: false - range: "[1,*)" - integral: true - default: 10 - protocol-wumbo-channels: - type: boolean - name: Enable Wumbo Channels - description: | - If set, then lnd will create and accept requests for channels larger than 0.16 BTC - nullable: false - default: false - protocol-no-anchors: - type: boolean - name: Disable Anchor Channels - description: | - Set to disable support for anchor commitments. Anchor channels allow you to determine your fees at close time by - using a Child Pays For Parent transaction. - nullable: false - default: false - protocol-disable-script-enforced-lease: - type: boolean - name: Disable Script Enforced Channel Leases - description: >- - Set to disable support for script enforced lease channel commitments. If not - set, lnd will accept these channels by default if the remote channel party - proposes them. Note that lnd will require 1 UTXO to be reserved for this - channel type if it is enabled. - - Note: This may cause you to be unable to close a channel and your wallets may not understand why - nullable: false - default: false - gc-canceled-invoices-on-startup: - type: boolean - name: Cleanup Canceled Invoices on Startup - description: | - If true, LND will attempt to garbage collect canceled invoices upon start. - nullable: false - default: false - bitcoin: - type: object - name: Bitcoin Channel Configuration - description: Configuration options for lightning network channel management operating over the Bitcoin network - nullable: false - spec: - default-channel-confirmations: - type: number - name: Default Channel Confirmations - description: | - The default number of confirmations a channel must have before it's considered - open. LND will require any incoming channel requests to wait this many - confirmations before it considers the channel active. - nullable: false - range: "[1,6]" - integral: true - default: 3 - units: "blocks" - min-htlc: - type: number - name: Minimum Incoming HTLC Size - description: | - The smallest HTLC LND will to accept on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1 - units: "millisatoshis" - min-htlc-out: - type: number - name: Minimum Outgoing HTLC Size - description: | - The smallest HTLC LND will send out on your channels, in millisatoshis. - nullable: false - range: "[1,*)" - integral: true - default: 1000 - units: "millisatoshis" - base-fee: - type: number - name: Routing Base Fee - description: | - The base fee in millisatoshi you will charge for forwarding payments on your - channels. - nullable: false - range: "[0,*)" - integral: true - default: 1000 - units: "millisatoshi" - fee-rate: - type: number - name: Routing Fee Rate - description: | - The fee rate used when forwarding payments on your channels. The total fee - charged is the Base Fee + (amount * Fee Rate / 1000000), where amount is the - forwarded amount. Measured in sats per million - nullable: false - range: "[1,1000000)" - integral: true - default: 1 - units: "sats per million" - time-lock-delta: - type: number - name: Time Lock Delta - description: The CLTV delta we will subtract from a forwarded HTLC's timelock value. - nullable: false - range: "[6, 144]" - integral: true - default: 40 - units: "blocks" diff --git a/core/startos/test/js_action_execute/package-data/scripts/test-package/0.3.0.3/embassy.js b/core/startos/test/js_action_execute/package-data/scripts/test-package/0.3.0.3/embassy.js deleted file mode 100644 index 86565053..00000000 --- a/core/startos/test/js_action_execute/package-data/scripts/test-package/0.3.0.3/embassy.js +++ /dev/null @@ -1,1054 +0,0 @@ -export function properties() { - return "Anything here"; -} - -export async function getConfig(effects) { - try { - await effects.writeFile({ - path: "../test.log", - toWrite: "This is a test", - volumeId: "main", - }); - throw new Error( - "Expecting that the ../test.log should not be a valid path since we are breaking out of the parent" - ); - } catch (e) {} - try { - await effects.writeFile({ - path: "./hack_back/broken.log", - toWrite: "This is a test", - volumeId: "main", - }); - throw new Error( - "Expecting that using a symlink to break out of parent still fails for writing" - ); - } catch (e) {} - try { - await effects.createDir({ - path: "./hack_back/broken_dir", - volumeId: "main", - }); - throw new Error( - "Expecting that using a symlink to break out of parent still fails for writing dir" - ); - } catch (e) {} - try { - await effects.readFile({ - path: "./hack_back/data/bad_file.txt", - volumeId: "main", - }); - throw new Error( - "Expecting that using a symlink to break out of parent still fails for reading" - ); - } catch (e) {} - - // Testing dir, create + delete - await effects.createDir({ - path: "./testing", - volumeId: "main", - }); - await effects.writeJsonFile({ - path: "./testing/test2.log", - toWrite: { value: "This is a test" }, - volumeId: "main", - }); - - ( - await effects.readJsonFile({ - path: "./testing/test2.log", - volumeId: "main", - // @ts-ignore - }) - ).value; - - await effects.removeFile({ - path: "./testing/test2.log", - volumeId: "main", - }); - await effects.removeDir({ - path: "./testing", - volumeId: "main", - }); - - // Testing reading + writing - await effects.writeFile({ - path: "./test.log", - toWrite: "This is a test", - volumeId: "main", - }); - - effects.debug( - `Read results are ${await effects.readFile({ - path: "./test.log", - volumeId: "main", - })}` - ); - // Testing loging - effects.trace("trace"); - effects.debug("debug"); - effects.warn("warn"); - effects.error("error"); - effects.info("info"); - - { - const metadata = await effects.metadata({ - path: "./test.log", - volumeId: "main", - }); - - if (typeof metadata.fileType !== "string") { - throw new TypeError("File type is not a string"); - } - if (typeof metadata.isDir !== "boolean") { - throw new TypeError("isDir is not a boolean"); - } - if (typeof metadata.isFile !== "boolean") { - throw new TypeError("isFile is not a boolean"); - } - if (typeof metadata.isSymlink !== "boolean") { - throw new TypeError("isSymlink is not a boolean"); - } - if (typeof metadata.len !== "number") { - throw new TypeError("len is not a number"); - } - if (typeof metadata.gid !== "number") { - throw new TypeError("gid is not a number"); - } - if (typeof metadata.uid !== "number") { - throw new TypeError("uid is not a number"); - } - if (typeof metadata.mode !== "number") { - throw new TypeError("mode is not a number"); - } - if (!(metadata.modified instanceof Date)) { - throw new TypeError("modified is not a Date"); - } - if (!(metadata.accessed instanceof Date)) { - throw new TypeError("accessed is not a Date"); - } - if (!(metadata.created instanceof Date)) { - throw new TypeError("created is not a Date"); - } - if (typeof metadata.readonly !== "boolean") { - throw new TypeError("readonly is not a boolean"); - } - effects.error(JSON.stringify(metadata)); - } - return { - result: { - spec: { - "control-tor-address": { - name: "Control Tor Address", - description: "The Tor address for the control interface.", - type: "pointer", - subtype: "package", - "package-id": "lnd", - target: "tor-address", - interface: "control", - }, - "peer-tor-address": { - name: "Peer Tor Address", - description: "The Tor address for the peer interface.", - type: "pointer", - subtype: "package", - "package-id": "lnd", - target: "tor-address", - interface: "peer", - }, - "watchtower-tor-address": { - name: "Watchtower Tor Address", - description: "The Tor address for the watchtower interface.", - type: "pointer", - subtype: "package", - "package-id": "lnd", - target: "tor-address", - interface: "watchtower", - }, - alias: { - type: "string", - name: "Alias", - description: "The public, human-readable name of your Lightning node", - nullable: true, - pattern: ".{1,32}", - "pattern-description": - "Must be at least 1 character and no more than 32 characters", - }, - color: { - type: "string", - name: "Color", - description: "The public color dot of your Lightning node", - nullable: false, - pattern: "[0-9a-fA-F]{6}", - "pattern-description": - "Must be a valid 6 digit hexadecimal RGB value. The first two digits are red, middle two are green, and final two are\nblue\n", - default: { - charset: "a-f,0-9", - len: 6, - }, - }, - "accept-keysend": { - type: "boolean", - name: "Accept Keysend", - description: - "Allow others to send payments directly to your public key through keysend instead of having to get a new invoice\n", - default: false, - }, - "accept-amp": { - type: "boolean", - name: "Accept Spontaneous AMPs", - description: - "If enabled, spontaneous payments through AMP will be accepted. Payments to AMP\ninvoices will be accepted regardless of this setting.\n", - default: false, - }, - "reject-htlc": { - type: "boolean", - name: "Reject Routing Requests", - description: - "If true, LND will not forward any HTLCs that are meant as onward payments. This option will still allow LND to send\nHTLCs and receive HTLCs but lnd won't be used as a hop.\n", - default: false, - }, - "min-chan-size": { - type: "number", - name: "Minimum Channel Size", - description: - "The smallest channel size that we should accept. Incoming channels smaller than this will be rejected.\n", - nullable: true, - range: "[1,16777215]", - integral: true, - units: "satoshis", - }, - "max-chan-size": { - type: "number", - name: "Maximum Channel Size", - description: - "The largest channel size that we should accept. Incoming channels larger than this will be rejected.\nFor non-Wumbo channels this limit remains 16777215 satoshis by default as specified in BOLT-0002. For wumbo\nchannels this limit is 1,000,000,000 satoshis (10 BTC). Set this config option explicitly to restrict your maximum\nchannel size to better align with your risk tolerance. Don't forget to enable Wumbo channels under 'Advanced,' if desired.\n", - nullable: true, - range: "[1,1000000000]", - integral: true, - units: "satoshis", - }, - tor: { - type: "object", - name: "Tor Config", - spec: { - "use-tor-only": { - type: "boolean", - name: "Use Tor for all traffic", - description: - "Use the tor proxy even for connections that are reachable on clearnet. This will hide your node's public IP address, but will slow down your node's performance", - default: false, - }, - "stream-isolation": { - type: "boolean", - name: "Stream Isolation", - description: - "Enable Tor stream isolation by randomizing user credentials for each connection. With this mode active, each connection will use a new circuit. This means that multiple applications (other than lnd) using Tor won't be mixed in with lnd's traffic.\nThis option may not be used when 'Use Tor for all traffic' is disabled, since direct connections compromise source IP privacy by default.", - default: false, - }, - }, - }, - bitcoind: { - type: "union", - name: "Bitcoin Core", - description: - "The Bitcoin Core node to connect to:\n - internal: The Bitcoin Core and Proxy services installed to your Embassy\n - external: An unpruned Bitcoin Core node running on a different device\n", - tag: { - id: "type", - name: "Type", - "variant-names": { - internal: "Internal (Bitcoin Core)", - "internal-proxy": "Internal (Bitcoin Proxy)", - external: "External", - }, - description: - "The Bitcoin Core node to connect to:\n - internal: The Bitcoin Core and Proxy services installed to your Embassy\n - external: An unpruned Bitcoin Core node running on a different device\n", - }, - default: "internal", - variants: { - internal: { - user: { - type: "pointer", - name: "RPC Username", - description: "The username for Bitcoin Core's RPC interface", - subtype: "package", - "package-id": "bitcoind", - target: "config", - multi: false, - selector: "$.rpc.username", - }, - password: { - type: "pointer", - name: "RPC Password", - description: "The password for Bitcoin Core's RPC interface", - subtype: "package", - "package-id": "bitcoind", - target: "config", - multi: false, - selector: "$.rpc.password", - }, - }, - "internal-proxy": { - user: { - type: "pointer", - name: "RPC Username", - description: "The username for the RPC user allocated to lnd", - subtype: "package", - "package-id": "btc-rpc-proxy", - target: "config", - multi: false, - selector: '$.users[?(@.name == "lnd")].name', - }, - password: { - type: "pointer", - name: "RPC Password", - description: "The password for the RPC user allocated to lnd", - subtype: "package", - "package-id": "btc-rpc-proxy", - target: "config", - multi: false, - selector: '$.users[?(@.name == "lnd")].password', - }, - }, - external: { - "connection-settings": { - type: "union", - name: "Connection Settings", - description: - "Information to connect to an external unpruned Bitcoin Core node", - tag: { - id: "type", - name: "Type", - description: - "- Manual: Raw information for finding a Bitcoin Core node\n- Quick Connect: A Quick Connect URL for a Bitcoin Core node\n", - "variant-names": { - manual: "Manual", - "quick-connect": "Quick Connect", - }, - }, - default: "quick-connect", - variants: { - manual: { - host: { - type: "string", - name: "Public Address", - description: - "The public address of your Bitcoin Core server", - nullable: false, - }, - "rpc-user": { - type: "string", - name: "RPC Username", - description: - "The username for the RPC user on your Bitcoin Core RPC server", - nullable: false, - }, - "rpc-password": { - type: "string", - name: "RPC Password", - description: - "The password for the RPC user on your Bitcoin Core RPC server", - nullable: false, - }, - "rpc-port": { - type: "number", - name: "RPC Port", - description: - "The port that your Bitcoin Core RPC server is bound to", - nullable: false, - range: "[0,65535]", - integral: true, - default: 8332, - }, - "zmq-block-port": { - type: "number", - name: "ZeroMQ Block Port", - description: - "The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks", - nullable: false, - range: "[0,65535]", - integral: true, - default: 28332, - }, - "zmq-tx-port": { - type: "number", - name: "ZeroMQ Transaction Port", - description: - "The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions", - nullable: false, - range: "[0,65535]", - integral: true, - default: 28333, - }, - }, - "quick-connect": { - "quick-connect-url": { - type: "string", - name: "Quick Connect URL", - description: - "The Quick Connect URL for your Bitcoin Core RPC server\nNOTE: LND will not accept a .onion url for this option\n", - nullable: false, - pattern: - "btcstandup://[^:]*:[^@]*@[a-zA-Z0-9.-]+:[0-9]+(/(\\?(label=.+)?)?)?", - "pattern-description": - "Must be a valid Quick Connect URL. For help, check out https://github.com/BlockchainCommons/Gordian/blob/master/Docs/Quick-Connect-API.md", - }, - "zmq-block-port": { - type: "number", - name: "ZeroMQ Block Port", - description: - "The port that your Bitcoin Core ZeroMQ server is bound to for raw blocks", - nullable: false, - range: "[0,65535]", - integral: true, - default: 28332, - }, - "zmq-tx-port": { - type: "number", - name: "ZeroMQ Transaction Port", - description: - "The port that your Bitcoin Core ZeroMQ server is bound to for raw transactions", - nullable: false, - range: "[0,65535]", - integral: true, - default: 28333, - }, - }, - }, - }, - }, - }, - }, - autopilot: { - type: "object", - name: "Autopilot", - description: "Autopilot Settings", - spec: { - enabled: { - type: "boolean", - name: "Enabled", - description: - "If the autopilot agent should be active or not. The autopilot agent will\nattempt to AUTOMATICALLY OPEN CHANNELS to put your node in an advantageous\nposition within the network graph. DO NOT ENABLE THIS IF YOU WANT TO MANAGE \nCHANNELS MANUALLY OR DO NOT UNDERSTAND IT.\n", - default: false, - }, - private: { - type: "boolean", - name: "Private", - description: - "Whether the channels created by the autopilot agent should be private or not.\nPrivate channels won't be announced to the network.\n", - default: false, - }, - maxchannels: { - type: "number", - name: "Maximum Channels", - description: - "The maximum number of channels that should be created.", - nullable: false, - range: "[1,*)", - integral: true, - default: 5, - }, - allocation: { - type: "number", - name: "Allocation", - description: - 'The fraction of total funds that should be committed to automatic channel\nestablishment. For example 60% means that 60% of the total funds available\nwithin the wallet should be used to automatically establish channels. The total\namount of attempted channels will still respect the "Maximum Channels" parameter.\n', - nullable: false, - range: "[0,100]", - integral: false, - default: 60, - units: "%", - }, - "min-channel-size": { - type: "number", - name: "Minimum Channel Size", - description: - "The smallest channel that the autopilot agent should create.", - nullable: false, - range: "[0,*)", - integral: true, - default: 20000, - units: "satoshis", - }, - "max-channel-size": { - type: "number", - name: "Maximum Channel Size", - description: - "The largest channel that the autopilot agent should create.", - nullable: false, - range: "[0,*)", - integral: true, - default: 16777215, - units: "satoshis", - }, - advanced: { - type: "object", - name: "Advanced", - description: "Advanced Options", - spec: { - "min-confirmations": { - type: "number", - name: "Minimum Confirmations", - description: - "The minimum number of confirmations each of your inputs in funding transactions\ncreated by the autopilot agent must have.\n", - nullable: false, - range: "[0,*)", - integral: true, - default: 1, - units: "blocks", - }, - "confirmation-target": { - type: "number", - name: "Confirmation Target", - description: - "The confirmation target (in blocks) for channels opened by autopilot.", - nullable: false, - range: "[0,*)", - integral: true, - default: 1, - units: "blocks", - }, - }, - }, - }, - }, - advanced: { - type: "object", - name: "Advanced", - description: "Advanced Options", - spec: { - "debug-level": { - type: "enum", - name: "Log Verbosity", - values: ["trace", "debug", "info", "warn", "error", "critical"], - description: - "Sets the level of log filtration. Trace is the most verbose, Critical is the least.\n", - default: "info", - "value-names": {}, - }, - "db-bolt-no-freelist-sync": { - type: "boolean", - name: "Disallow Bolt DB Freelist Sync", - description: - "If true, prevents the database from syncing its freelist to disk.\n", - default: false, - }, - "db-bolt-auto-compact": { - type: "boolean", - name: "Compact Database on Startup", - description: - "Performs database compaction on startup. This is necessary to keep disk usage down over time at the cost of\nhaving longer startup times.\n", - default: true, - }, - "db-bolt-auto-compact-min-age": { - type: "number", - name: "Minimum Autocompaction Age for Bolt DB", - description: - "How long ago (in hours) the last compaction of a database file must be for it to be considered for auto\ncompaction again. Can be set to 0 to compact on every startup.\n", - nullable: false, - range: "[0, *)", - integral: true, - default: 168, - units: "hours", - }, - "db-bolt-db-timeout": { - type: "number", - name: "Bolt DB Timeout", - description: - "How long should LND try to open the database before giving up?", - nullable: false, - range: "[1, 86400]", - integral: true, - default: 60, - units: "seconds", - }, - "recovery-window": { - type: "number", - name: "Recovery Window", - description: - "Number of blocks in the past that LND should scan for unknown transactions", - nullable: true, - range: "[1,*)", - integral: true, - units: "blocks", - }, - "payments-expiration-grace-period": { - type: "number", - name: "Payments Expiration Grace Period", - description: - "A period to wait before for closing channels with outgoing htlcs that have timed out and are a result of this\nnodes instead payment. In addition to our current block based deadline, is specified this grace period will\nalso be taken into account.\n", - nullable: false, - range: "[1,*)", - integral: true, - default: 30, - units: "seconds", - }, - "default-remote-max-htlcs": { - type: "number", - name: "Maximum Remote HTLCs", - description: - "The default max_htlc applied when opening or accepting channels. This value limits the number of concurrent\nHTLCs that the remote party can add to the commitment. The maximum possible value is 483.\n", - nullable: false, - range: "[1,483]", - integral: true, - default: 483, - units: "htlcs", - }, - "max-channel-fee-allocation": { - type: "number", - name: "Maximum Channel Fee Allocation", - description: - "The maximum percentage of total funds that can be allocated to a channel's commitment fee. This only applies for\nthe initiator of the channel.\n", - nullable: false, - range: "[0.1, 1]", - integral: false, - default: 0.5, - }, - "max-commit-fee-rate-anchors": { - type: "number", - name: "Maximum Commitment Fee for Anchor Channels", - description: - "The maximum fee rate in sat/vbyte that will be used for commitments of channels of the anchors type. Must be\nlarge enough to ensure transaction propagation.\n", - nullable: false, - range: "[1,*)", - integral: true, - default: 10, - }, - "protocol-wumbo-channels": { - type: "boolean", - name: "Enable Wumbo Channels", - description: - "If set, then lnd will create and accept requests for channels larger than 0.16 BTC\n", - default: false, - }, - "protocol-no-anchors": { - type: "boolean", - name: "Disable Anchor Channels", - description: - "Set to disable support for anchor commitments. Anchor channels allow you to determine your fees at close time by\nusing a Child Pays For Parent transaction.\n", - default: false, - }, - "protocol-disable-script-enforced-lease": { - type: "boolean", - name: "Disable Script Enforced Channel Leases", - description: - "Set to disable support for script enforced lease channel commitments. If not set, lnd will accept these channels by default if the remote channel party proposes them. Note that lnd will require 1 UTXO to be reserved for this channel type if it is enabled.\nNote: This may cause you to be unable to close a channel and your wallets may not understand why", - default: false, - }, - "gc-canceled-invoices-on-startup": { - type: "boolean", - name: "Cleanup Canceled Invoices on Startup", - description: - "If true, LND will attempt to garbage collect canceled invoices upon start.\n", - default: false, - }, - bitcoin: { - type: "object", - name: "Bitcoin Channel Configuration", - description: - "Configuration options for lightning network channel management operating over the Bitcoin network", - spec: { - "default-channel-confirmations": { - type: "number", - name: "Default Channel Confirmations", - description: - "The default number of confirmations a channel must have before it's considered\nopen. LND will require any incoming channel requests to wait this many\nconfirmations before it considers the channel active.\n", - nullable: false, - range: "[1,6]", - integral: true, - default: 3, - units: "blocks", - }, - "min-htlc": { - type: "number", - name: "Minimum Incoming HTLC Size", - description: - "The smallest HTLC LND will to accept on your channels, in millisatoshis.\n", - nullable: false, - range: "[1,*)", - integral: true, - default: 1, - units: "millisatoshis", - }, - "min-htlc-out": { - type: "number", - name: "Minimum Outgoing HTLC Size", - description: - "The smallest HTLC LND will send out on your channels, in millisatoshis.\n", - nullable: false, - range: "[1,*)", - integral: true, - default: 1000, - units: "millisatoshis", - }, - "base-fee": { - type: "number", - name: "Routing Base Fee", - description: - "The base fee in millisatoshi you will charge for forwarding payments on your\nchannels.\n", - nullable: false, - range: "[0,*)", - integral: true, - default: 1000, - units: "millisatoshi", - }, - "fee-rate": { - type: "number", - name: "Routing Fee Rate", - description: - "The fee rate used when forwarding payments on your channels. The total fee\ncharged is the Base Fee + (amount * Fee Rate / 1000000), where amount is the\nforwarded amount. Measured in sats per million\n", - nullable: false, - range: "[1,1000000)", - integral: true, - default: 1, - units: "sats per million", - }, - "time-lock-delta": { - type: "number", - name: "Time Lock Delta", - description: - "The CLTV delta we will subtract from a forwarded HTLC's timelock value.", - nullable: false, - range: "[6, 144]", - integral: true, - default: 40, - units: "blocks", - }, - }, - }, - }, - }, - }, - }, - }; -} - -export async function setConfig(effects) { - return { - error: "Not setup", - }; -} - -const assert = (condition, message) => { - if (!condition) { - throw new Error(message); - } -}; -const ackermann = (m, n) => { - if (m === 0) { - return n+1 - } - if (n === 0) { - return ackermann((m - 1), 1); - } - if (m !== 0 && n !== 0) { - return ackermann((m-1), ackermann(m, (n-1))) - } -} - -export const action = { - async slow(effects, _input) { - while(true) { - effects.error("A"); - await ackermann(3,10); - // await effects.sleep(100); - - } - }, - - async fetch(effects, _input) { - const example = await effects.fetch( - "https://postman-echo.com/get?foo1=bar1&foo2=bar2" - ); - assert( - Number(example.headers["content-length"]) > 0 && - Number(example.headers["content-length"]) <= 1000000, - "Should have content length" - ); - assert( - example.text() instanceof Promise, - "example.text() should be a promise" - ); - assert(example.body === undefined, "example.body should not be defined"); - assert( - JSON.parse(await example.text()).args.foo1 === "bar1", - "Body should be parsed" - ); - const message = `This worked @ ${new Date().toISOString()}`; - const secondResponse = await effects.fetch( - "https://postman-echo.com/post", - { - method: "POST", - body: JSON.stringify({ message }), - headers: { - test: "1234", - }, - } - ); - assert( - (await secondResponse.json()).json.message === message, - "Body should be parsed from response" - ); - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - - async "js-action-var-arg"(_effects, _input, testInput) { - assert(testInput == 42, "Input should be passed in"); - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - async "test-rename"(effects, _input) { - let failed = false; - await effects.writeFile({ - volumeId: "main", - path: "test-rename.txt", - toWrite: "Hello World", - }); - await effects.rename({ - srcVolume: "main", - srcPath: "test-rename.txt", - dstVolume: "main", - dstPath: "test-rename-2.txt", - }); - - const readIn = await effects.readFile({ - volumeId: "main", - path: "test-rename-2.txt", - }); - assert(readIn === "Hello World", "Contents should be the same"); - - await effects.removeFile({ - path: "test-rename-2.txt", - volumeId: "main", - }); - - failed = false; - try { - await effects.removeFile({ - path: "test-rename.txt", - volumeId: "main", - }); - } catch (_) { - failed = true; - } - assert(failed, "Should not be able to remove file that doesn't exist"); - - - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - /** - * Created this test because of issue - * https://github.com/Start9Labs/start-os/issues/1737 - * which that we couldn't create a dir that was deeply nested, and the parents where - * not created yet. Found this out during the migrations, where the parent would die. - * @param {*} effects - * @param {*} _input - * @returns - */ - async "test-deep-dir"(effects, _input) { - await effects - .removeDir({ - volumeId: "main", - path: "test-deep-dir", - }) - .catch(() => {}); - await effects.createDir({ - volumeId: "main", - path: "test-deep-dir/deep/123", - }); - await effects.removeDir({ - volumeId: "main", - path: "test-deep-dir", - }); - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - - /** - * Test is for the feature of listing what's in a dir - * @param {*} effects - * @param {*} _input - * @returns - */ - async "test-read-dir"(effects, _input) { - await effects - .removeDir({ - volumeId: "main", - path: "test-read-dir", - }) - .catch(() => {}); - await effects.createDir({ - volumeId: "main", - path: "test-read-dir/deep/123", - }); - await effects.writeFile({ - path: "./test-read-dir/broken.log", - toWrite: "This is a test", - volumeId: "main", - }) - let readDir = JSON.stringify(await effects.readDir({ - volumeId: "main", - path: "test-read-dir", - })) - const expected = '["broken.log","deep"]' - assert(readDir === expected, `Failed to match the input (${readDir}) === (${expected}) of readDir`) - - await effects.removeDir({ - volumeId: "main", - path: "test-read-dir", - }); - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - /** - * Created this test because of issue - * https://github.com/Start9Labs/start-os/issues/2121 - * That the empty in the create dies - * @param {*} effects - * @param {*} _input - * @returns - */ - async "test-zero-dir"(effects, _input) { - await effects.createDir({ - volumeId: "main", - path: "./", - }); - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - /** - * Found case where we could escape with the new deeper dir fix. - * @param {*} effects - * @param {*} _input - * @returns - */ - async "test-deep-dir-escape"(effects, _input) { - await effects - .removeDir({ - volumeId: "main", - path: "test-deep-dir", - }) - .catch(() => {}); - await effects.createDir({ - volumeId: "main", - path: "test-deep-dir/../../test", - }).then(_ => {throw new Error("Should not be able to create sub")}, _ => {}); - - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - }, - - - /** - * Want to test that rsync works - * @param {*} effects - * @param {*} _input - * @returns - */ - async "test-rsync"(effects, _input) { - try { - await effects - .removeDir({ - volumeId: "main", - path: "test-rsync-out", - }) - .catch(() => {}); - const runningRsync = effects.runRsync({ - srcVolume: "main", - srcPath: "testing-rsync", - dstVolume: "main", - dstPath: "test-rsync-out", - options: { - delete: true, - force: true, - ignoreExisting: false, - } - }); - assert(await runningRsync.id() >= 1, "Expect that we have an id"); - const progress = await runningRsync.progress() - assert(progress >= 0 && progress <= 1, `Expect progress to be 0 <= progress(${progress}) <= 1`); - await runningRsync.wait(); - assert((await effects.readFile({ - volumeId: "main", - path: "test-rsync-out/testing-rsync/someFile.txt", - })).length > 0, 'Asserting that we read in the file "test_rsync/test-package/0.3.0.3/embassy.js"'); - - - return { - result: { - copyable: false, - message: "Done", - version: "0", - qr: false, - }, - }; - } - catch (e) { - throw e; - } - finally { - await effects - .removeDir({ - volumeId: "main", - path: "test-rsync-out", - }) - .catch(() => {}); - } - }, - - async "test-disk-usage"(effects, _input) { - const usage = await effects.diskUsage() - return { - result: { - copyable: false, - message: `${usage.used} / ${usage.total}`, - version: "0", - qr: false, - }, - }; - } - -}; diff --git a/core/startos/test/js_action_execute/package-data/volumes/test-package/data/bad_file.txt b/core/startos/test/js_action_execute/package-data/volumes/test-package/data/bad_file.txt deleted file mode 100644 index 7bd6da0b..00000000 --- a/core/startos/test/js_action_execute/package-data/volumes/test-package/data/bad_file.txt +++ /dev/null @@ -1 +0,0 @@ -Out of volumes diff --git a/core/startos/test/js_action_execute/package-data/volumes/test-package/data/main/testing-rsync/someFile.txt b/core/startos/test/js_action_execute/package-data/volumes/test-package/data/main/testing-rsync/someFile.txt deleted file mode 100644 index ac0bb024..00000000 --- a/core/startos/test/js_action_execute/package-data/volumes/test-package/data/main/testing-rsync/someFile.txt +++ /dev/null @@ -1 +0,0 @@ -Here's something in this file for the rsync \ No newline at end of file diff --git a/core/startos/update-sqlx-data.sh b/core/startos/update-sqlx-data.sh deleted file mode 100755 index 83ec111f..00000000 --- a/core/startos/update-sqlx-data.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/bash - -TMP_DIR=$(mktemp -d) -mkdir $TMP_DIR/pgdata -docker run -d --rm --name=tmp_postgres -e POSTGRES_PASSWORD=password -v $TMP_DIR/pgdata:/var/lib/postgresql/data postgres - -( - set -e - ctr=0 - while ! docker exec tmp_postgres psql -U postgres || [ $ctr -lt 5 ]; do - ctr=$[ctr + 1] - sleep 1; - done - - PG_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' tmp_postgres) - - DATABASE_URL=postgres://postgres:password@$PG_IP/postgres cargo sqlx migrate run - DATABASE_URL=postgres://postgres:password@$PG_IP/postgres PLATFORM=$(uname -m) cargo sqlx prepare -- --lib --profile=test -) - -docker stop tmp_postgres -sudo rm -rf $TMP_DIR diff --git a/docs/BETA-RELEASE-CHECKLIST.md b/docs/BETA-RELEASE-CHECKLIST.md index 65afeba6..85563aad 100644 --- a/docs/BETA-RELEASE-CHECKLIST.md +++ b/docs/BETA-RELEASE-CHECKLIST.md @@ -1,188 +1,270 @@ -# CRITICAL CHANGES FOR BETA ISO BUILD +# Beta Release Checklist (v0.5.0-beta) -## ⚠️ MUST-HAVE CHANGES - Without these, the beta will NOT work! +## Pre-Build Verification -### 1. Backend: Podman Detection Fix -**File:** `core/container/src/podman_client.rs` +### Source Code -```rust -fn podman_async(&self) -> TokioCommand { - // Always use sudo podman to access system-wide containers - let mut cmd = TokioCommand::new("sudo"); - cmd.arg("podman"); - cmd -} -``` +- [ ] All changes committed and pushed to `main` +- [ ] `cargo clippy --all-targets --all-features` passes (zero warnings) +- [ ] `cargo fmt --all` applied +- [ ] `cd neode-ui && npm run type-check` passes (zero errors) +- [ ] `cd neode-ui && npm test` passes (all tests green) +- [ ] `cargo test --all-features` passes on dev server -**System Config:** `/etc/sudoers.d/archipelago-podman` -``` -archipelago ALL=(ALL) NOPASSWD: /usr/bin/podman -``` +### Critical Files -### 2. Backend: Bitcoin UI Container Mapping -**File:** `core/archipelago/src/container/docker_packages.rs` - -Add special case for bitcoin-knots (line ~95): -```rust -} else if app_id == "bitcoin-knots" { - // Check if bitcoin-ui exists (maps to "bitcoin" but serves bitcoin-knots) - if let Some(ui_address) = ui_containers.get("bitcoin") { - debug!("Using bitcoin-ui for bitcoin-knots: {}", ui_address); - Some(ui_address.clone()) - } else { - extract_lan_address(&container.ports) - } -``` - -### 3. Backend: IndeedHub Metadata -**File:** `core/archipelago/src/container/docker_packages.rs` - -Add to `get_app_metadata()` function (line ~327): -```rust -"indeedhub" => AppMetadata { - title: "IndeedHub".to_string(), - description: "Decentralized media streaming platform".to_string(), - icon: "/assets/img/app-icons/indeedhub.png".to_string(), - repo: "https://github.com/indeedhub/indeedhub".to_string(), -}, -``` - -### 4. Frontend: Marketplace Bitcoin Knots -**File:** `neode-ui/src/views/Marketplace.vue` - -```javascript -{ - id: 'bitcoin-knots', // CHANGED from 'bitcoin' - title: 'Bitcoin Knots', - version: '28.1.0', // UPDATED version - dockerImage: 'docker.io/bitcoinknots/bitcoin:latest', // CHANGED image - // ... rest of config -} -``` - -### 5. Auto-Installer: Profile Script Fix -**File:** `image-recipe/build-auto-installer-iso.sh` (line ~850) - -Remove `|| [ ! -t 0 ]` check: -```bash -# CORRECT: -if [ -n "$INSTALLER_STARTED" ]; then - return 0 -fi - -# WRONG (will break auto-login): -# if [ -n "$INSTALLER_STARTED" ] || [ ! -t 0 ]; then -``` - -### 6. Nginx Configuration -**File:** Captured from `/etc/nginx/sites-available/default` - -MUST include: -- HTTPS on port 443 -- HTTP redirect to HTTPS -- Backend proxy: `/rpc/`, `/ws/`, `/health` -- Root: `/opt/archipelago/web-ui` -- SSL certificates in `/etc/nginx/ssl/` - -### 7. Bitcoin UI Files -**Files:** `docker/bitcoin-ui/index.html` and `Dockerfile` - -MUST be included in ISO or downloadable, so users can deploy the web UI container. +- [ ] `core/container/src/podman_client.rs` — sudo podman +- [ ] `core/archipelago/src/container/docker_packages.rs` — app metadata + UI mapping +- [ ] `core/archipelago/src/api/rpc/package.rs` — app configs, capabilities, dependencies +- [ ] `core/archipelago/src/session.rs` — session security hardening +- [ ] `core/security/src/secrets_manager.rs` — encryption + rotation +- [ ] `neode-ui/src/views/Marketplace.vue` — all app entries with pinned image versions +- [ ] `neode-ui/src/api/websocket.ts` — heartbeat + reconnection +- [ ] `image-recipe/build-auto-installer-iso.sh` — all container images captured +- [ ] `image-recipe/configs/nginx-archipelago.conf` — all app proxies + path traversal blocks +- [ ] All app icons present in `neode-ui/public/assets/img/app-icons/` --- -## Build Verification Before Beta Release +## App Integration Matrix -Run these checks: +Every app must be tested for install, launch, and uninstall on a fresh system. + +### Core Bitcoin Stack + +| App | Image | Version | Install | Launch | UI Loads | Uninstall | +|-----|-------|---------|---------|--------|----------|-----------| +| Bitcoin Knots | `bitcoinknots/bitcoin` | `v28.1` | [ ] | [ ] | [ ] | [ ] | +| Electrs | `mempool/electrs` | `v0.4.1` | [ ] | [ ] | [ ] | [ ] | +| LND | `lightninglabs/lnd` | `v0.18.4` | [ ] | [ ] | [ ] | [ ] | +| BTCPay Server | `btcpayserver/btcpayserver` | `2.0.6` | [ ] | [ ] | [ ] | [ ] | +| Mempool | `mempool/frontend` | `v3.0.0` | [ ] | [ ] | [ ] | [ ] | +| Fedimint | `fedimintui/fedimint` | `0.5.0` | [ ] | [ ] | [ ] | [ ] | +| Fedimint Gateway | `fedimintui/gateway-ui` | `0.5.0` | [ ] | [ ] | [ ] | [ ] | + +### Storage & Media + +| App | Image | Version | Install | Launch | UI Loads | Uninstall | +|-----|-------|---------|---------|--------|----------|-----------| +| File Browser | `filebrowser/filebrowser` | `v2` | [ ] | [ ] | [ ] | [ ] | +| Immich | `ghcr.io/immich-app/immich-server` | `v1.121.0` | [ ] | [ ] | [ ] | [ ] | +| PhotoPrism | `photoprism/photoprism` | `240915` | [ ] | [ ] | [ ] | [ ] | + +### Productivity & Privacy + +| App | Image | Version | Install | Launch | UI Loads | Uninstall | +|-----|-------|---------|---------|--------|----------|-----------| +| Penpot | `penpotapp/frontend` | `2.4` | [ ] | [ ] | [ ] | [ ] | +| SearXNG | `searxng/searxng` | `2024.11.17-e2554de75` | [ ] | [ ] | [ ] | [ ] | +| Ollama | `ollama/ollama` | `0.5.4` | [ ] | [ ] | [ ] | [ ] | + +### Network & Infrastructure + +| App | Image | Version | Install | Launch | UI Loads | Uninstall | +|-----|-------|---------|---------|--------|----------|-----------| +| Nostr Relay | `scsiblade/nostr-rs-relay` | `0.9.0` | [ ] | [ ] | [ ] | [ ] | +| Nginx Proxy Manager | `jc21/nginx-proxy-manager` | `2.12.1` | [ ] | [ ] | [ ] | [ ] | +| Tailscale | `tailscale/tailscale` | pinned | [ ] | [ ] | [ ] | [ ] | +| Home Assistant | `homeassistant/home-assistant` | pinned | [ ] | [ ] | [ ] | [ ] | + +### Virtual Apps (No Container) + +| App | Behavior | Works | +|-----|----------|-------| +| IndeedHub | Opens external URL | [ ] | + +--- + +## Dependency Chain Tests + +These must be tested in order on a fresh install: + +- [ ] Install Bitcoin Knots → starts and begins syncing +- [ ] Install Electrs while Bitcoin running → connects to Bitcoin automatically +- [ ] Install LND while Bitcoin running → connects to Bitcoin automatically +- [ ] Install BTCPay while Bitcoin running → connects; Lightning available if LND present +- [ ] Install Mempool while Bitcoin + Electrs running → shows blockchain data +- [ ] Try installing Electrs without Bitcoin → shows clear error message +- [ ] Try installing LND without Bitcoin → shows clear error message +- [ ] Try installing Mempool without Bitcoin + Electrs → shows missing deps error +- [ ] Fedimint Gateway auto-detects LND credentials when available + +--- + +## Security Hardening Verification + +### Session Security + +- [ ] Sessions expire after 24 hours of inactivity +- [ ] Password change invalidates all other sessions +- [ ] Maximum 5 concurrent sessions (oldest evicted when exceeded) +- [ ] Session tokens are SHA-256 hashed in memory (never stored as plaintext) +- [ ] Login rate limiting: 5 failures per 60 seconds per IP + +### Container Security + +- [ ] All container images use pinned versions (no `:latest`) +- [ ] Read-only root filesystem enabled for compatible apps +- [ ] `--cap-drop=ALL` applied to all containers +- [ ] `--security-opt=no-new-privileges:true` applied to all containers +- [ ] Required capabilities added explicitly per app (e.g., CHOWN for File Browser) + +### Secrets Management + +- [ ] Secrets encrypted with AES-256-GCM on disk +- [ ] Secret metadata tracked (creation date, rotation count) +- [ ] Secret rotation generates new random values and re-encrypts +- [ ] `security.list-expiring` RPC returns secrets older than threshold + +### Path Traversal Prevention + +- [ ] Nginx blocks `..` in filebrowser API paths (403 response) +- [ ] Frontend `sanitizePath()` strips `..` and resolves paths +- [ ] File Browser token not exposed in URLs + +### Authentication + +- [ ] TOTP 2FA enrollment and verification works +- [ ] TOTP backup codes work for recovery +- [ ] Maximum 5 TOTP attempts before session invalidation +- [ ] Pending TOTP sessions expire after 5 minutes +- [ ] Cookie-based auth (no tokens in query strings) + +--- + +## WebSocket & Connectivity + +- [ ] WebSocket connects on login and receives initial data dump +- [ ] WebSocket reconnects after network interruption (exponential backoff, max 30s) +- [ ] Server sends ping every 30s; client responds with pong +- [ ] Client sends JSON ping every 30s; server responds with JSON pong +- [ ] Server closes inactive connections after 5 minutes +- [ ] Connection state shown in UI (connected/reconnecting/disconnected) +- [ ] Install progress updates delivered in real-time via WebSocket + +--- + +## Fresh Install Testing Matrix + +### ISO Build + +- [ ] ISO builds successfully on dev server +- [ ] ISO size is reasonable (< 10 GB) +- [ ] All container images captured in ISO + +### Installation + +- [ ] Boot from USB on x86_64 hardware +- [ ] Auto-installer partitions disk correctly +- [ ] Debian 12 installs without errors +- [ ] Archipelago services start on first boot +- [ ] Web UI accessible at server IP within 3 minutes of first boot + +### Onboarding Flow + +- [ ] Welcome screen displays with intro video +- [ ] Password creation enforces minimum requirements +- [ ] Path selection shows all 6 options +- [ ] DID generation completes within 60 seconds +- [ ] Identity naming is optional and skippable +- [ ] Backup download produces valid JSON file +- [ ] Onboarding completes and reaches Dashboard + +### Post-Onboarding + +- [ ] Dashboard shows all overview cards +- [ ] App Store loads with all curated apps +- [ ] Settings shows server name, version, DID, Tor address +- [ ] Logout and re-login works +- [ ] Password change works and invalidates other sessions + +--- + +## Performance Targets + +- [ ] Backend startup: < 3 seconds +- [ ] Frontend initial load: < 500 KB gzipped +- [ ] WebSocket initial data: < 1 second after connection +- [ ] App install progress visible in UI within 5 seconds of starting + +--- + +## Nginx Proxy Verification + +All app proxies must work in both HTTP and HTTPS blocks: + +- [ ] `/rpc/` → backend:5678 +- [ ] `/ws/` → backend:5678 (WebSocket upgrade) +- [ ] `/health` → backend:5678 +- [ ] `/app/filebrowser/` → filebrowser:80 +- [ ] `/app/searxng/` → searxng:8080 +- [ ] `/app/immich/` → immich:2283 +- [ ] `/app/penpot/` → penpot-frontend:80 +- [ ] `/app/ollama/` → ollama:11434 +- [ ] `/app/photoprism/` → photoprism:2342 +- [ ] `/app/nginx-proxy-manager/` → npm:81 +- [ ] `/app/tailscale/` → tailscale:8240 +- [ ] BTCPay (port 23000) opens in new tab +- [ ] Home Assistant (port 8123) opens in new tab +- [ ] Tor hidden services resolve for all configured apps + +--- + +## Rollback Procedures + +### If Backend Fails to Start ```bash -# 1. Verify all source changes are committed -cd /Users/dorian/Projects/archy -git status # Should show all critical files committed +# Check logs +sudo journalctl -u archipelago -n 50 --no-pager -# 2. Build ISO from live server -cd image-recipe -DEV_SERVER=archipelago@192.168.1.228 ./build-auto-installer-iso.sh - -# 3. Test ISO on clean VM -# - Boot ISO -# - Verify auto-installer runs -# - System should boot to login -# - Access http://SERVER-IP -# - Complete onboarding -# - Install Bitcoin Knots from App Store -# - Verify "Already Installed" shows after install -# - Verify "Launch" button works -# - Verify web UI loads on port 8334 - -# 4. Test all critical features -# - Bitcoin node syncing -# - RPC accessible -# - Web UI functional -# - Backend detects container -# - App Store shows proper status +# Restore previous binary +sudo cp /usr/local/bin/archipelago.bak /usr/local/bin/archipelago +sudo systemctl restart archipelago ``` ---- - -## Critical Files Checklist - -Before building beta ISO, ensure these files have the latest changes: - -- [ ] `core/container/src/podman_client.rs` - sudo podman -- [ ] `core/archipelago/src/container/docker_packages.rs` - app metadata + UI mapping -- [ ] `neode-ui/src/views/Marketplace.vue` - bitcoin-knots ID -- [ ] `neode-ui/src/utils/dummyApps.ts` - IndeedHub data -- [ ] `image-recipe/build-auto-installer-iso.sh` - auto-start fix -- [ ] `docker/bitcoin-ui/` - UI files present -- [ ] `scripts/deploy-bitcoin-knots.sh` - deployment script -- [ ] All assets: `neode-ui/public/assets/img/app-icons/*.png` - ---- - -## Testing Matrix - -| Feature | Expected | Status | -|---------|----------|--------| -| Bitcoin Knots container runs | Running | ✅ | -| Bitcoin UI container runs | Running | ✅ | -| Backend detects bitcoin-knots | Detected | ✅ | -| Backend maps bitcoin-ui → bitcoin-knots | Port 8334 | ✅ | -| App shows in My Apps | Listed | ✅ | -| App Store shows "Already Installed" | Badge shown | ✅ (after ID fix) | -| Launch button visible | Clickable | ✅ | -| Launch opens web UI | Port 8334 | ✅ | -| RPC accessible | Port 8332 | ✅ | -| Blockchain syncing | Active | ✅ | - ---- - -## Roll-Back Plan - -If beta ISO fails: - -1. Check `/var/log/archipelago.log` on installed system -2. Verify containers with `sudo podman ps -a` -3. Check nginx status: `sudo systemctl status nginx` -4. Test backend: `curl http://localhost:5678/health` -5. Rebuild ISO with `BUILD_FROM_SOURCE=1` if server state is corrupt - ---- - -## Support Commands for Users +### If Frontend is Broken ```bash -# Check Bitcoin status -sudo podman logs -f bitcoin-knots - -# Check blockchain sync progress -curl --user archipelago:archipelago123 \ - --data-binary '{"jsonrpc":"1.0","id":"test","method":"getblockchaininfo","params":[]}' \ - -H 'content-type: text/plain;' http://localhost:8332/ | grep blocks - -# Restart if needed -sudo podman restart bitcoin-knots bitcoin-ui - -# View Archipelago backend logs -sudo journalctl -u archipelago -f +# Restore previous frontend build +sudo cp -r /opt/archipelago/web-ui.bak/* /opt/archipelago/web-ui/ +sudo systemctl reload nginx ``` + +### If Container Won't Start + +```bash +# Check container logs +sudo podman logs + +# Remove and recreate +sudo podman rm -f +# Reinstall from App Store +``` + +### If ISO Install Fails + +1. Boot into rescue mode from USB +2. Check `/var/log/installer.log` on target disk +3. Verify disk partitioning with `lsblk` +4. Re-run installer with `INSTALLER_STARTED= /opt/installer.sh` + +### Full System Rollback + +If the beta is unusable: +1. Re-flash the ISO from the last known good build +2. Restore user data from `/var/lib/archipelago/` backup +3. Re-import DID from backup JSON file + +--- + +## Sign-Off + +| Reviewer | Area | Date | Pass/Fail | +|----------|------|------|-----------| +| | Backend | | | +| | Frontend | | | +| | Security | | | +| | ISO Build | | | +| | Fresh Install | | | +| | App Integrations | | | diff --git a/docs/adr/001-podman-over-docker.md b/docs/adr/001-podman-over-docker.md new file mode 100644 index 00000000..17e0ce99 --- /dev/null +++ b/docs/adr/001-podman-over-docker.md @@ -0,0 +1,32 @@ +# ADR-001: Podman Over Docker + +**Status**: Accepted +**Date**: 2026-03 + +## Context + +Archipelago needs a container runtime for running applications. Docker and Podman are the two main options. + +## Decision + +Use Podman as the container runtime instead of Docker. + +## Consequences + +### Positive +- **Rootless by default**: Containers run without root privileges, reducing attack surface +- **Daemonless**: No persistent daemon process; containers are managed as individual processes under systemd +- **Docker-compatible**: Supports Docker images and most Docker CLI commands +- **Systemd integration**: Podman containers can be managed as systemd services natively +- **No vendor lock-in**: OCI-compliant, works with any container registry + +### Negative +- **Smaller ecosystem**: Some Docker-specific tools and compose features require adaptation +- **Docker Compose differences**: Podman Compose exists but has occasional compatibility gaps +- **Documentation**: Most container documentation assumes Docker; developers need to translate +- **Networking**: Podman networking (CNI/netavark) differs from Docker's bridge networking + +### Mitigation +- Use `podman` CLI wrapper that provides Docker-compatible interface +- Document Podman-specific commands in developer guide +- Use `archy-net` custom network for inter-container DNS diff --git a/docs/adr/002-did-key-method.md b/docs/adr/002-did-key-method.md new file mode 100644 index 00000000..3b046cc9 --- /dev/null +++ b/docs/adr/002-did-key-method.md @@ -0,0 +1,31 @@ +# ADR-002: DID Key Method for Node Identity + +**Status**: Accepted +**Date**: 2026-03 + +## Context + +Each Archipelago node needs a cryptographic identity for peer authentication, federation, and verifiable credentials. Multiple DID methods exist (did:web, did:ion, did:key, did:peer). + +## Decision + +Use `did:key` as the primary DID method. + +## Consequences + +### Positive +- **Self-contained**: The DID document is derived entirely from the public key — no external resolution needed +- **Offline-capable**: Works without internet, aligning with sovereignty principles +- **Simple**: No registration, no blockchain, no web server required +- **Fast**: DID resolution is a local computation, not a network request +- **Ed25519**: Uses Ed25519 keys which are fast, compact, and well-supported + +### Negative +- **No key rotation**: The DID is bound to a single key; rotating requires a new DID +- **No service endpoints in DID**: Cannot embed service URLs in the DID document itself +- **No revocation**: Cannot revoke a did:key without out-of-band mechanisms + +### Mitigation +- Use federation trust lists for key management and revocation +- Store service endpoints (onion address, pubkey) separately in federation state +- Support migration to did:peer or did:web in future versions if key rotation is needed diff --git a/docs/adr/003-nostr-for-discovery.md b/docs/adr/003-nostr-for-discovery.md new file mode 100644 index 00000000..b5bb8c06 --- /dev/null +++ b/docs/adr/003-nostr-for-discovery.md @@ -0,0 +1,35 @@ +# ADR-003: Nostr Relays for Node and App Discovery + +**Status**: Accepted +**Date**: 2026-03 + +## Context + +Archipelago nodes need to discover peers and community apps without a central registry. Options: custom P2P protocol, DHT, BitTorrent tracker, Nostr relays, IPFS. + +## Decision + +Use Nostr relays (NIP-78, kind 30078) for both node discovery and marketplace app manifests. + +## Consequences + +### Positive +- **Decentralized**: Multiple independent relays; no single point of failure +- **Existing infrastructure**: Thousands of Nostr relays already running globally +- **Censorship-resistant**: If one relay censors, others still serve events +- **Simple protocol**: WebSocket + JSON — easy to implement without heavy dependencies +- **Key management**: Nostr uses secp256k1, same curve as Bitcoin — natural fit +- **NIP-33 replaceable events**: Latest event replaces previous — clean update model +- **Tor-compatible**: WebSocket over Tor SOCKS proxy works natively + +### Negative +- **Relay availability varies**: Some relays may be down or rate-limited +- **No guaranteed persistence**: Relays may prune old events +- **Spam potential**: Open publishing means anyone can publish junk manifests +- **Latency**: Querying multiple relays adds latency to discovery + +### Mitigation +- Query multiple relays in parallel; deduplicate results +- Cache results locally with 15-minute TTL +- Use trust scoring to rank manifests (DID verification, relay consensus, federation trust) +- Use hashtag filtering (`archipelago-marketplace`) to narrow queries diff --git a/docs/adr/004-tor-for-peer-communication.md b/docs/adr/004-tor-for-peer-communication.md new file mode 100644 index 00000000..b007e846 --- /dev/null +++ b/docs/adr/004-tor-for-peer-communication.md @@ -0,0 +1,35 @@ +# ADR-004: Tor Hidden Services for Peer Communication + +**Status**: Accepted +**Date**: 2026-03 + +## Context + +Federated nodes need to communicate directly for state sync, app deployment, and peer verification. Options: direct IP, VPN tunnel, Tor hidden services, I2P. + +## Decision + +Use Tor hidden services (.onion addresses) for all inter-node communication. + +## Consequences + +### Positive +- **NAT traversal**: Works behind any firewall or NAT without port forwarding +- **IP privacy**: Nodes never expose their real IP addresses to each other +- **End-to-end encryption**: Tor provides encryption without additional TLS setup +- **Censorship resistance**: Onion routing makes traffic analysis difficult +- **Stable addressing**: .onion addresses persist across IP changes and network migrations +- **No central infrastructure**: No VPN server, STUN/TURN server, or relay needed + +### Negative +- **Latency**: Tor adds 200-500ms per hop; 3 hops per direction = noticeable delay +- **Bandwidth**: Tor network has limited bandwidth; not suitable for bulk data transfer +- **Reliability**: Tor circuits can break; connections may need retry logic +- **Setup complexity**: Requires running a Tor daemon (`archy-tor` container) +- **Blocked networks**: Some networks block Tor; bridges can help but add complexity + +### Mitigation +- Use Tor only for RPC/control plane; bulk data (container images) pulled from registries +- Implement retry with backoff for Tor connections +- Container `archy-tor` runs automatically with host networking for hidden service access +- Federation sync interval (5 min) tolerates occasional connection failures diff --git a/docs/adr/005-chacha20-backup-encryption.md b/docs/adr/005-chacha20-backup-encryption.md new file mode 100644 index 00000000..8cddfdc6 --- /dev/null +++ b/docs/adr/005-chacha20-backup-encryption.md @@ -0,0 +1,32 @@ +# ADR-005: ChaCha20-Poly1305 for Backup Encryption + +**Status**: Accepted +**Date**: 2026-03 + +## Context + +Backups contain sensitive data (keys, credentials, app state) and must be encrypted at rest. Options: AES-256-GCM, ChaCha20-Poly1305, XChaCha20-Poly1305. + +## Decision + +Use ChaCha20-Poly1305 (AEAD) with Argon2id key derivation for backup encryption. + +## Consequences + +### Positive +- **Software performance**: ChaCha20 is faster than AES on hardware without AES-NI (common on ARM/SBCs) +- **Constant-time**: No timing side channels, unlike some AES implementations +- **AEAD**: Authenticated encryption ensures both confidentiality and integrity +- **Widely audited**: Used in TLS 1.3, WireGuard, and Signal Protocol +- **Simple implementation**: No padding, no CBC/CTR mode complexity +- **Argon2id KDF**: Memory-hard key derivation resists GPU/ASIC brute force attacks + +### Negative +- **96-bit nonce**: Must ensure nonce uniqueness per encryption (random generation with collision check) +- **Not FIPS-certified**: Some enterprise environments require AES (not relevant for personal nodes) +- **Less hardware acceleration**: AES-NI on x86 can make AES faster on desktop CPUs + +### Mitigation +- Generate random nonce per backup; store nonce alongside ciphertext +- Argon2id with high memory cost (64MB) and iterations (3) for password-to-key derivation +- Target hardware is mixed x86/ARM; ChaCha20's consistent performance is an advantage diff --git a/docs/api-reference.md b/docs/api-reference.md new file mode 100644 index 00000000..50bc437f --- /dev/null +++ b/docs/api-reference.md @@ -0,0 +1,397 @@ +# Archipelago API Reference + +All endpoints use JSON-RPC over HTTP POST to `/rpc/v1`. + +**Request format:** +```json +{ + "method": "namespace.action", + "params": { ... } +} +``` + +**Response format:** +```json +{ + "result": { ... } +} +``` + +**Error format:** +```json +{ + "error": { "message": "Error description" } +} +``` + +**Authentication:** All endpoints require a valid session cookie (`archipelago_session`) except those marked "No Auth". + +--- + +## Authentication + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `auth.login` | `{ password: string }` | `{ ok: bool, totp_required?: bool }` | No Auth | +| `auth.logout` | — | `{ ok: bool }` | Yes | +| `auth.changePassword` | `{ current: string, new: string }` | `{ ok: bool }` | Yes | +| `auth.isOnboardingComplete` | — | `{ complete: bool }` | No Auth | +| `auth.onboardingComplete` | — | `{ ok: bool }` | Yes | +| `auth.resetOnboarding` | — | `{ ok: bool }` | Yes | + +### TOTP 2FA + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `auth.totp.setup.begin` | `{ password: string }` | `{ secret: string, qr_uri: string, backup_codes: string[] }` | Yes | +| `auth.totp.setup.confirm` | `{ code: string }` | `{ ok: bool }` | Yes | +| `auth.totp.disable` | `{ password: string }` | `{ ok: bool }` | Yes | +| `auth.totp.status` | — | `{ enabled: bool }` | Yes | +| `auth.login.totp` | `{ code: string }` | `{ ok: bool }` | No Auth | +| `auth.login.backup` | `{ code: string }` | `{ ok: bool }` | No Auth | + +--- + +## Container Orchestration + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `container-install` | `{ image: string, name?: string }` | `{ ok: bool, container_id: string }` | Yes | +| `container-start` | `{ id: string }` | `{ ok: bool }` | Yes | +| `container-stop` | `{ id: string }` | `{ ok: bool }` | Yes | +| `container-remove` | `{ id: string }` | `{ ok: bool }` | Yes | +| `container-list` | — | `{ containers: Container[] }` | Yes | +| `container-status` | `{ id: string }` | `{ status: string, ... }` | Yes | +| `container-logs` | `{ id: string, lines?: number }` | `{ logs: string }` | Yes | +| `container-health` | `{ id: string }` | `{ healthy: bool }` | Yes | + +## Package Management + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `package.install` | `{ id: string, dockerImage?: string, url?: string, version?: string }` | `{ ok: bool }` | Yes | +| `package.start` | `{ id: string }` | `{ ok: bool }` | Yes | +| `package.stop` | `{ id: string }` | `{ ok: bool }` | Yes | +| `package.restart` | `{ id: string }` | `{ ok: bool }` | Yes | +| `package.uninstall` | `{ id: string }` | `{ ok: bool }` | Yes | + +## Bundled Apps + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `bundled-app-start` | `{ id: string }` | `{ ok: bool }` | Yes | +| `bundled-app-stop` | `{ id: string }` | `{ ok: bool }` | Yes | + +--- + +## Node Identity & P2P + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `node.did` | — | `{ did: string }` | Yes | +| `node.signChallenge` | `{ challenge: string }` | `{ signature: string }` | Yes | +| `node.tor-address` | — | `{ address: string }` | Yes | +| `node.nostr-publish` | — | `{ ok: bool, event_id: string }` | Yes | +| `node.nostr-pubkey` | — | `{ pubkey: string }` | Yes | +| `node-nostr-verify-revoked` | — | `{ revoked: bool, nostr_pubkey: string }` | Yes | +| `node-nostr-discover` | — | `{ nodes: DiscoveredNode[] }` | Yes | +| `node-add-peer` | `{ did: string, address: string }` | `{ ok: bool }` | Yes | +| `node-list-peers` | — | `{ peers: Peer[] }` | Yes | +| `node-remove-peer` | `{ did: string }` | `{ ok: bool }` | Yes | +| `node-send-message` | `{ to: string, message: string }` | `{ ok: bool }` | Yes | +| `node-check-peer` | `{ did: string }` | `{ online: bool }` | Yes | +| `node-messages-received` | — | `{ messages: Message[] }` | Yes | +| `node.createBackup` | `{ password: string }` | `{ path: string }` | Yes | + +--- + +## Identity Management + +### Multi-Identity + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `identity.list` | `{}` | `{ identities: Identity[] }` | Yes | +| `identity.create` | `{ label: string }` | `{ identity: Identity }` | Yes | +| `identity.get` | `{ id: string }` | `{ identity: Identity }` | Yes | +| `identity.delete` | `{ id: string }` | `{ ok: bool }` | Yes | +| `identity.set-default` | `{ id: string }` | `{ ok: bool }` | Yes | +| `identity.sign` | `{ id: string, data: string }` | `{ signature: string }` | Yes | +| `identity.verify` | `{ id: string, data: string, signature: string }` | `{ valid: bool }` | Yes | +| `identity.resolve-did` | `{ did: string }` | `{ document: DIDDocument }` | Yes | +| `identity.resolve-remote-did` | `{ did: string }` | `{ document: DIDDocument }` | Yes | +| `identity.verify-did-document` | `{ document: object }` | `{ valid: bool }` | Yes | + +### Nostr Keys + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `identity.create-nostr-key` | `{ id: string }` | `{ pubkey: string, npub: string }` | Yes | +| `identity.nostr-sign` | `{ id: string, event: object }` | `{ signed_event: object }` | Yes | + +### Bitcoin Names (NIP-05) + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `identity.register-name` | `{ name: string, pubkey: string }` | `{ ok: bool }` | Yes | +| `identity.remove-name` | `{ name: string }` | `{ ok: bool }` | Yes | +| `identity.resolve-name` | `{ name: string }` | `{ pubkey: string }` | Yes | +| `identity.list-names` | `{}` | `{ names: NameEntry[] }` | Yes | +| `identity.link-name` | `{ name: string, identity_id: string }` | `{ ok: bool }` | Yes | + +### Verifiable Credentials + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `identity.issue-credential` | `{ subject: string, claims: object }` | `{ credential: VC }` | Yes | +| `identity.verify-credential` | `{ credential: object }` | `{ valid: bool }` | Yes | +| `identity.list-credentials` | `{ id?: string }` | `{ credentials: VC[] }` | Yes | +| `identity.revoke-credential` | `{ credential_id: string }` | `{ ok: bool }` | Yes | +| `identity.create-presentation` | `{ credentials: string[] }` | `{ presentation: VP }` | Yes | +| `identity.verify-presentation` | `{ presentation: object }` | `{ valid: bool }` | Yes | + +--- + +## Bitcoin & Lightning + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `bitcoin.getinfo` | — | `{ blocks: number, connections: number, ... }` | Yes | +| `lnd.getinfo` | — | `{ identity_pubkey: string, num_active_channels: number, ... }` | Yes | +| `lnd.listchannels` | — | `{ channels: Channel[] }` | Yes | +| `lnd.openchannel` | `{ pubkey: string, amount: number }` | `{ funding_txid: string }` | Yes | +| `lnd.closechannel` | `{ channel_point: string }` | `{ closing_txid: string }` | Yes | +| `lnd.newaddress` | — | `{ address: string }` | Yes | +| `lnd.sendcoins` | `{ addr: string, amount: number }` | `{ txid: string }` | Yes | +| `lnd.createinvoice` | `{ amount: number, memo?: string }` | `{ payment_request: string }` | Yes | +| `lnd.payinvoice` | `{ payment_request: string }` | `{ preimage: string }` | Yes | +| `lnd.create-psbt` | `{ outputs: object, ... }` | `{ psbt: string }` | Yes | +| `lnd.finalize-psbt` | `{ psbt: string }` | `{ signed_psbt: string }` | Yes | + +--- + +## Ecash Wallet + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `wallet.ecash-balance` | — | `{ balance: number, mint_url: string }` | Yes | +| `wallet.ecash-mint` | `{ amount: number }` | `{ ok: bool }` | Yes | +| `wallet.ecash-melt` | `{ amount: number, invoice: string }` | `{ ok: bool }` | Yes | +| `wallet.ecash-send` | `{ amount: number }` | `{ token: string }` | Yes | +| `wallet.ecash-receive` | `{ token: string }` | `{ amount: number }` | Yes | +| `wallet.ecash-history` | — | `{ transactions: EcashTx[] }` | Yes | +| `wallet.networking-profits` | — | `{ total_sats: number, ... }` | Yes | + +--- + +## Network + +### Interfaces & WiFi + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `network.list-interfaces` | — | `{ interfaces: Interface[] }` | Yes | +| `network.scan-wifi` | — | `{ networks: WifiNetwork[] }` | Yes | +| `network.configure-wifi` | `{ ssid: string, password: string }` | `{ ok: bool }` | Yes | +| `network.configure-ethernet` | `{ interface: string, mode: "dhcp"\|"static", ip?: string, gateway?: string, dns?: string }` | `{ ok: bool }` | Yes | +| `network.diagnostics` | — | `{ wan_ip: string, nat_type: string, upnp_available: bool, tor_connected: bool, wifi_count: number }` | Yes | + +### DNS + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `network.dns-status` | — | `{ provider: string, servers: string[], doh_enabled: bool, doh_url: string?, resolv_conf_servers: string[] }` | Yes | +| `network.configure-dns` | `{ provider: "system"\|"cloudflare"\|"google"\|"quad9"\|"mullvad"\|"custom", servers?: string[] }` | `{ ok: bool, provider: string, servers: string[], doh_enabled: bool }` | Yes | + +### Network Overlay + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `network.get-visibility` | — | `{ visibility: string }` | Yes | +| `network.set-visibility` | `{ visibility: string }` | `{ ok: bool }` | Yes | +| `network.request-connection` | `{ target_did: string }` | `{ request_id: string }` | Yes | +| `network.list-requests` | — | `{ requests: ConnectionRequest[] }` | Yes | +| `network.accept-request` | `{ request_id: string }` | `{ ok: bool }` | Yes | +| `network.reject-request` | `{ request_id: string }` | `{ ok: bool }` | Yes | + +### Router / UPnP + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `router.discover` | — | `{ router: RouterInfo }` | Yes | +| `router.list-forwards` | — | `{ forwards: PortForward[] }` | Yes | +| `router.add-forward` | `{ port: number, protocol: string, description: string }` | `{ ok: bool }` | Yes | +| `router.remove-forward` | `{ port: number, protocol: string }` | `{ ok: bool }` | Yes | +| `router.detect` | `{ ... }` | `{ detected: bool, ... }` | Yes | +| `router.info` | — | `{ ... }` | Yes | +| `router.configure` | `{ ... }` | `{ ok: bool }` | Yes | + +--- + +## Tor + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `tor.list-services` | — | `{ services: TorService[] }` | Yes | +| `tor.create-service` | `{ name: string, port: number }` | `{ onion_address: string }` | Yes | +| `tor.delete-service` | `{ name: string }` | `{ ok: bool }` | Yes | +| `tor.get-onion-address` | `{ name: string }` | `{ address: string }` | Yes | + +## Nostr Relays + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `nostr.list-relays` | — | `{ relays: RelayConfig[] }` | Yes | +| `nostr.add-relay` | `{ url: string }` | `{ ok: bool }` | Yes | +| `nostr.remove-relay` | `{ url: string }` | `{ ok: bool }` | Yes | +| `nostr.toggle-relay` | `{ url: string }` | `{ ok: bool }` | Yes | +| `nostr.get-stats` | — | `{ total_relays: number, connected: number, enabled: number }` | Yes | + +--- + +## VPN + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `vpn.status` | — | `{ connected: bool, provider?: string, ip_address?: string, hostname?: string, peers_connected: number }` | Yes | +| `vpn.configure` | `{ provider: "tailscale"\|"wireguard", auth_key?: string, address?: string, dns?: string, peer?: object }` | `{ ok: bool }` | Yes | +| `vpn.disconnect` | — | `{ disconnected: bool }` | Yes | + +## Mesh Networking + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `mesh.status` | — | `{ enabled: bool, device: string?, nodes: MeshNode[] }` | Yes | +| `mesh.discover` | `{ timeout_secs?: number }` | `{ nodes: MeshNode[] }` | Yes | +| `mesh.broadcast` | — | `{ ok: bool }` | Yes | +| `mesh.configure` | `{ enabled: bool, device?: string }` | `{ ok: bool }` | Yes | + +--- + +## Federation + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `federation.invite` | — | `{ code: string }` | Yes | +| `federation.join` | `{ code: string }` | `{ ok: bool, node: FederatedNode }` | Yes | +| `federation.list-nodes` | — | `{ nodes: FederatedNode[] }` | Yes | +| `federation.remove-node` | `{ did: string }` | `{ ok: bool }` | Yes | +| `federation.set-trust` | `{ did: string, trust: "trusted"\|"observer"\|"untrusted" }` | `{ ok: bool }` | Yes | +| `federation.sync-state` | — | `{ results: SyncResult[] }` | Yes | +| `federation.get-state` | — | `{ state: NodeStateSnapshot }` | Federation peer | +| `federation.peer-joined` | `{ did: string, onion: string, pubkey: string }` | `{ accepted: bool }` | Federation peer | +| `federation.deploy-app` | `{ target_did: string, app_id: string, version?: string }` | `{ ok: bool }` | Yes | + +--- + +## Marketplace + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `marketplace.discover` | — | `{ apps: DiscoveredApp[], relay_count: number }` | Yes | +| `marketplace.publish` | `{ app_id, name, version, description, author, container, category, ... }` | `{ ok: bool, event_id: string }` | Yes | +| `marketplace.get-manifest` | `{ app_id: string }` | `DiscoveredApp \| { error: string }` | Yes | +| `marketplace.list-published` | — | `{ manifests: AppManifest[] }` | Yes | +| `marketplace.verify` | `{ ... manifest fields ... }` | `{ valid: bool, issues: string[], trust_score: number }` | Yes | + +--- + +## DWN (Decentralized Web Node) + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `dwn.status` | — | `{ running: bool, message_count: number, protocol_count: number }` | Yes | +| `dwn.sync` | — | `{ synced: number }` | Yes | +| `dwn.register-protocol` | `{ uri: string, definition: object }` | `{ ok: bool }` | Yes | +| `dwn.list-protocols` | — | `{ protocols: Protocol[] }` | Yes | +| `dwn.remove-protocol` | `{ uri: string }` | `{ ok: bool }` | Yes | +| `dwn.query-messages` | `{ protocol?: string, limit?: number }` | `{ messages: DwnMessage[] }` | Yes | +| `dwn.write-message` | `{ protocol: string, data: object }` | `{ ok: bool, message_id: string }` | Yes | + +--- + +## Content Catalog + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `content.list-mine` | — | `{ items: ContentItem[] }` | Yes | +| `content.add` | `{ title: string, type: string, data: object }` | `{ ok: bool, id: string }` | Yes | +| `content.remove` | `{ id: string }` | `{ ok: bool }` | Yes | +| `content.set-pricing` | `{ id: string, price_sats: number }` | `{ ok: bool }` | Yes | +| `content.set-availability` | `{ id: string, available: bool }` | `{ ok: bool }` | Yes | +| `content.browse-peer` | `{ peer_did: string }` | `{ items: ContentItem[] }` | Yes | + +--- + +## System + +### Monitoring + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `system.stats` | — | `{ cpu_percent: number, ram_used: number, ram_total: number, disk_used: number, disk_total: number, uptime_secs: number, load_avg: number[] }` | Yes | +| `system.processes` | — | `{ processes: Process[] }` | Yes | +| `system.temperature` | — | `{ celsius: number? }` | Yes | +| `system.detect-usb-devices` | — | `{ devices: UsbDevice[] }` | Yes | + +### Updates + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `update.check` | — | `{ available: bool, version?: string, changelog?: string }` | Yes | +| `update.status` | — | `{ state: string, progress?: number }` | Yes | +| `update.dismiss` | — | `{ ok: bool }` | Yes | +| `update.download` | — | `{ ok: bool }` | Yes | +| `update.apply` | — | `{ ok: bool }` | Yes | +| `update.rollback` | — | `{ ok: bool }` | Yes | +| `update.get-schedule` | — | `{ auto_check: bool, auto_install: bool, schedule: string }` | Yes | +| `update.set-schedule` | `{ auto_check?: bool, auto_install?: bool, schedule?: string }` | `{ ok: bool }` | Yes | + +### Backup & Restore + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `backup.create` | `{ password: string, include?: string[] }` | `{ path: string, size: number }` | Yes | +| `backup.list` | — | `{ backups: BackupEntry[] }` | Yes | +| `backup.verify` | `{ path: string, password: string }` | `{ valid: bool }` | Yes | +| `backup.restore` | `{ path: string, password: string }` | `{ ok: bool }` | Yes | +| `backup.delete` | `{ path: string }` | `{ ok: bool }` | Yes | +| `backup.list-drives` | — | `{ drives: UsbDrive[] }` | Yes | +| `backup.to-usb` | `{ drive: string, password: string }` | `{ ok: bool }` | Yes | + +### Security + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `security.rotate-secrets` | `{ app_id?: string }` | `{ rotated: string[] }` | Yes | +| `security.list-expiring` | `{ days?: number }` | `{ secrets: ExpiringSecret[] }` | Yes | + +--- + +## Utility + +| Method | Params | Returns | Auth | +|--------|--------|---------|------| +| `echo` | `{ message: string }` | `{ message: string }` | No Auth | +| `server.echo` | `{ message: string }` | `{ message: string }` | No Auth | + +--- + +## Example: cURL + +```bash +# Login +curl -c cookies.txt -X POST http://192.168.1.228/rpc/v1 \ + -H "Content-Type: application/json" \ + -d '{"method":"auth.login","params":{"password":"password123"}}' + +# Get system stats (authenticated) +curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \ + -H "Content-Type: application/json" \ + -d '{"method":"system.stats"}' + +# Get DID +curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \ + -H "Content-Type: application/json" \ + -d '{"method":"node.did"}' +``` diff --git a/docs/app-developer-guide.md b/docs/app-developer-guide.md new file mode 100644 index 00000000..4104251a --- /dev/null +++ b/docs/app-developer-guide.md @@ -0,0 +1,277 @@ +# Archipelago App Developer Guide + +Build and publish containerized apps for the Archipelago ecosystem. + +## Overview + +Apps run as Podman containers on user nodes. You publish app manifests to Nostr relays, where nodes discover and install them through the community marketplace. + +## App Manifest + +Every app needs a manifest (YAML for local apps, JSON for marketplace publishing). + +### Template Manifest + +```yaml +# apps/my-app/manifest.yml +app: + id: my-app # Unique, lowercase kebab-case + name: My App + version: 1.0.0 # Semantic versioning + +container: + image: docker.io/myorg/my-app:1.0.0 # Never use :latest + ports: + - container: 8080 + host: 8180 + protocol: tcp + volumes: + - name: data + path: /data + env: + APP_MODE: production + capabilities: [] # Only add if absolutely necessary + readonly_root: true # Required + no_new_privileges: true # Required + run_as_user: 1000 # Must be >= 1000 + +metadata: + description: + short: "One-line description (max 120 chars)" + long: "Detailed description of what this app does and why." + author: + name: "Your Name" + did: "did:key:z6Mk..." # Your Archipelago node DID + category: money # money | commerce | data | networking | home | community | other + icon_url: "https://example.com/icon.png" + repo_url: "https://github.com/myorg/my-app" + license: MIT + min_archipelago_version: "0.1.0" + dependencies: [] # e.g., ["bitcoin-knots"] if this app needs Bitcoin +``` + +### Required Fields + +| Field | Description | +|-------|-------------| +| `app.id` | Unique identifier, lowercase, kebab-case only | +| `app.name` | Human-readable name | +| `app.version` | Semantic version (major.minor.patch) | +| `container.image` | Full image reference with pinned version tag | +| `metadata.description.short` | One-line description, max 120 characters | +| `metadata.author.did` | Your node's DID (get via `node.did` RPC) | + +## Security Requirements + +These are enforced by the marketplace and the node. Non-compliant apps are flagged. + +### Mandatory + +1. **No `:latest` tag** — Pin a specific version: `myapp:1.0.0` +2. **Read-only root filesystem** — `readonly_root: true` (use volumes for writable data) +3. **Non-root user** — `run_as_user: 1000` or higher +4. **No privilege escalation** — `no_new_privileges: true` +5. **Minimal capabilities** — Drop all caps, only add required ones + +### Allowed Capabilities + +Only these Linux capabilities may be requested: + +| Capability | When Needed | +|-----------|-------------| +| `CHOWN` | App needs to change file ownership | +| `NET_BIND_SERVICE` | App binds to ports below 1024 | +| `DAC_OVERRIDE` | App needs to bypass file permissions | +| `SETUID`, `SETGID` | App manages user switching (e.g., nginx) | + +### Forbidden + +- `--network host` — Apps cannot share the host network +- Mounting system paths: `/`, `/etc`, `/var`, `/usr`, `/proc`, `/sys` +- `SYS_ADMIN`, `SYS_PTRACE`, or any privileged capability +- Hardcoded secrets in environment variables or images + +## Container Best Practices + +### Volumes + +```yaml +volumes: + - name: data # App data persists across restarts + path: /data + - name: config # Configuration files + path: /config +``` + +Data is stored at `/var/lib/archipelago/{app-id}/` on the host. + +### Health Checks + +Define a health check endpoint in your container: + +```dockerfile +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD curl -f http://localhost:8080/health || exit 1 +``` + +### Logging + +- Log to stdout/stderr (Podman captures container logs) +- Never log secrets, passwords, or keys +- Use structured logging (JSON) for machine parsing + +### Networking + +Apps get their own network namespace. To connect to other Archipelago apps: + +```yaml +# If your app needs to talk to Bitcoin +dependencies: + - bitcoin-knots + +container: + env: + BITCOIN_RPC_HOST: bitcoin-knots # Container DNS name on archy-net + BITCOIN_RPC_PORT: "8332" +``` + +The `archy-net` Podman network provides DNS resolution between containers. + +## Publishing to the Marketplace + +### 1. Build and Push Your Image + +```bash +podman build -t docker.io/myorg/my-app:1.0.0 . +podman push docker.io/myorg/my-app:1.0.0 +``` + +### 2. Get Your Node's DID + +```bash +curl -b cookies.txt -X POST http://localhost/rpc/v1 \ + -d '{"method":"node.did"}' +# Returns: {"result":{"did":"did:key:z6Mk..."}} +``` + +### 3. Publish via RPC + +```bash +curl -b cookies.txt -X POST http://localhost/rpc/v1 \ + -H "Content-Type: application/json" \ + -d '{ + "method": "marketplace.publish", + "params": { + "app_id": "my-app", + "name": "My App", + "version": "1.0.0", + "description": {"short": "A useful tool", "long": "Detailed description..."}, + "author": {"name": "Dev Name", "did": "did:key:z6Mk...", "nostr_pubkey": ""}, + "container": { + "image": "docker.io/myorg/my-app:1.0.0", + "ports": [{"container": 8080, "host": 8180, "protocol": "tcp"}], + "volumes": [], + "env": {}, + "capabilities": [], + "readonly_root": true, + "no_new_privileges": true, + "run_as_user": 1000 + }, + "category": "other", + "icon_url": "", + "repo_url": "https://github.com/myorg/my-app", + "license": "MIT", + "min_archipelago_version": "0.1.0", + "dependencies": [] + } + }' +``` + +The manifest is published to all configured Nostr relays as a NIP-78 event (kind 30078). + +### 4. Verify Discovery + +```bash +curl -b cookies.txt -X POST http://localhost/rpc/v1 \ + -d '{"method":"marketplace.discover"}' +# Your app should appear in the results +``` + +## Trust Model + +Published apps receive trust scores (0-100) based on: + +| Factor | Points | How to Maximize | +|--------|--------|-----------------| +| Valid DID in author | 30 | Always include your node's DID | +| Found on multiple relays | 5-20 | Configure many relays in your node | +| Developer in federation | 20 | Have federated peers who trust you | +| Proper semver version | 10 | Use `major.minor.patch` format | +| Repository URL present | 5 | Include your repo URL | +| Security compliance | 15 | Meet all security requirements | + +### Trust Tiers + +| Score | Tier | User Experience | +|-------|------|----------------| +| 80-100 | Verified | One-click install | +| 50-79 | Community | Install with confirmation | +| 20-49 | Unverified | Install with warning | +| 0-19 | Untrusted | Requires explicit override | + +## Testing Your App + +### Local Testing + +```bash +# Run your container locally +podman run -d --name my-app \ + -p 8180:8080 \ + --read-only \ + --security-opt no-new-privileges \ + --user 1000:1000 \ + docker.io/myorg/my-app:1.0.0 + +# Verify it works +curl http://localhost:8180/health + +# Check logs +podman logs my-app +``` + +### On an Archipelago Node + +1. Install via the marketplace UI or RPC: + ```bash + curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \ + -d '{"method":"package.install","params":{"id":"my-app","dockerImage":"docker.io/myorg/my-app:1.0.0"}}' + ``` +2. Verify the container is running: + ```bash + curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \ + -d '{"method":"container-list"}' + ``` +3. Check the UI at `http://192.168.1.228/app/my-app/` + +### Validate Manifest + +```bash +curl -b cookies.txt -X POST http://localhost/rpc/v1 \ + -H "Content-Type: application/json" \ + -d '{"method":"marketplace.verify","params":{...your manifest...}}' +# Returns: {"result":{"valid":true,"issues":[],"trust_score":65,"trust_tier":"community"}} +``` + +## Updating Your App + +1. Build and push the new version: `docker.io/myorg/my-app:1.1.0` +2. Publish an updated manifest with the new version +3. NIP-33 replaceable events: the latest publish overwrites the previous one on relays +4. Nodes running your app can see the update in their marketplace + +## App Icon + +- Provide a URL to your app icon (PNG, WebP, or SVG) +- Recommended size: 256x256 pixels +- Square aspect ratio +- If no icon URL, a generic placeholder is shown in the marketplace diff --git a/docs/arm64-build.md b/docs/arm64-build.md new file mode 100644 index 00000000..6428f19a --- /dev/null +++ b/docs/arm64-build.md @@ -0,0 +1,114 @@ +# ARM64 (aarch64) Cross-Compilation Guide + +## Overview + +Archipelago supports both x86_64 and ARM64 (aarch64) platforms. The backend is compiled natively on x86_64 and cross-compiled for ARM64 targets like Raspberry Pi 5. + +## Prerequisites + +### On the Build Server (Debian 12) + +```bash +# 1. Add the ARM64 Rust target +rustup target add aarch64-unknown-linux-gnu + +# 2. Install the cross-linker and C library +sudo apt update +sudo apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross + +# 3. Install cross-compilation OpenSSL headers (for reqwest/hyper TLS) +sudo apt install -y libssl-dev:arm64 +# If the above fails (no multiarch), use vendored OpenSSL instead: +# Set OPENSSL_STATIC=1 and add openssl = { version = "0.10", features = ["vendored"] } +``` + +### Cargo Configuration + +The cross-compilation config is at `core/.cargo/config.toml`: + +```toml +[target.aarch64-unknown-linux-gnu] +linker = "aarch64-linux-gnu-gcc" +``` + +## Building + +### Native (x86_64) + +```bash +cd core +cargo build --release -p archipelago +# Output: core/target/release/archipelago +``` + +### ARM64 Cross-Compilation + +```bash +cd core + +# Option A: System OpenSSL (requires libssl-dev:arm64) +PKG_CONFIG_ALLOW_CROSS=1 \ +PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig \ +cargo build --release --target aarch64-unknown-linux-gnu -p archipelago + +# Output: core/target/aarch64-unknown-linux-gnu/release/archipelago + +# Option B: Vendored OpenSSL (no system packages needed) +OPENSSL_STATIC=1 \ +cargo build --release --target aarch64-unknown-linux-gnu -p archipelago +``` + +### Verify the Binary + +```bash +file core/target/aarch64-unknown-linux-gnu/release/archipelago +# Should show: ELF 64-bit LSB pie executable, ARM aarch64, ... +``` + +## Using `cross` (Alternative) + +The `cross` tool uses Docker containers for hermetic cross-compilation: + +```bash +cargo install cross + +# Build for ARM64 (downloads a Docker image with all dependencies) +cross build --release --target aarch64-unknown-linux-gnu -p archipelago +``` + +This is the simplest approach and avoids installing system cross-compilation packages. + +## Troubleshooting + +### `cannot find -lssl` / `cannot find -lcrypto` + +OpenSSL headers for ARM64 are missing. Either: +- Install `libssl-dev:arm64` (requires multiarch support) +- Use vendored OpenSSL: set `OPENSSL_STATIC=1` +- Add `openssl = { version = "0.10", features = ["vendored"] }` to Cargo.toml + +### `cc: error: unrecognized command-line option` + +The wrong linker is being used. Verify `aarch64-linux-gnu-gcc` is installed: +```bash +which aarch64-linux-gnu-gcc +aarch64-linux-gnu-gcc --version +``` + +### `Exec format error` when running + +You're trying to run an ARM64 binary on x86_64. Use `qemu-aarch64-static` for testing: +```bash +sudo apt install qemu-user-static +qemu-aarch64-static ./archipelago +``` + +## Target Hardware + +| Device | Arch | Status | +|--------|------|--------| +| Raspberry Pi 5 | aarch64 | Primary ARM target | +| Raspberry Pi 4 | aarch64 | Supported | +| Rock Pi 4 | aarch64 | Untested | +| Orange Pi 5 | aarch64 | Untested | +| x86_64 NUC/Mini PC | x86_64 | Primary platform | diff --git a/docs/arm64-container-images.md b/docs/arm64-container-images.md new file mode 100644 index 00000000..33f9cb32 --- /dev/null +++ b/docs/arm64-container-images.md @@ -0,0 +1,21 @@ +# ARM64 Container Image Compatibility + +All core Archipelago marketplace apps have multi-arch Docker images with ARM64 (linux/arm64) support. + +## Core Apps + +| App | Image | Tag | ARM64 | ARMv7 | +|-----|-------|-----|-------|-------| +| Bitcoin Knots | `bitcoinknots/bitcoin` | `latest` | Yes | Yes | +| Electrs | `mempool/electrs` | `latest` | Yes | No | +| BTCPay Server | `btcpayserver/btcpayserver` | `1.13.5` | Yes | Yes | +| LND | `lightninglabs/lnd` | `v0.17.4-beta` | Yes | No | +| Mempool | `mempool/frontend` | `v2.5.0` | Yes | Yes | +| FileBrowser | `filebrowser/filebrowser` | `v2.27.0` | Yes | Yes | + +## Notes + +- All images use multi-arch manifest lists — Podman/Docker will automatically pull the correct architecture +- No changes needed to `Marketplace.vue` image references — the same tags work on both x86_64 and ARM64 +- Three apps also support ARMv7 (32-bit ARM), but Archipelago targets ARM64 only +- Verified 2026-03-11 via Docker Hub registry API manifest inspection diff --git a/docs/arm64-rpi5-testing.md b/docs/arm64-rpi5-testing.md new file mode 100644 index 00000000..6953ebd4 --- /dev/null +++ b/docs/arm64-rpi5-testing.md @@ -0,0 +1,78 @@ +# ARM64 Raspberry Pi 5 Testing Guide + +## Prerequisites + +- Raspberry Pi 5 (4GB+ RAM recommended) +- USB flash drive (16GB+) for the installer +- MicroSD card or NVMe SSD for the target install +- Monitor + keyboard (or serial console for headless setup) +- Ethernet connection (WiFi can be configured after install) + +## Building the ARM64 ISO + +On the build server (192.168.1.228): + +```bash +cd ~/archy/image-recipe +sudo ARCH=arm64 ./build-auto-installer-iso.sh +# Output: results/archipelago-installer-arm64.iso +``` + +## Flashing to USB + +```bash +# On macOS +diskutil list # identify USB drive +diskutil unmountDisk /dev/diskN +sudo dd if=results/archipelago-installer-arm64.iso of=/dev/rdiskN bs=4m + +# On Linux +sudo dd if=results/archipelago-installer-arm64.iso of=/dev/sdX bs=4M status=progress +``` + +Or use Balena Etcher for a GUI approach. + +## Testing Checklist + +### Boot & Install +- [ ] RPi 5 boots from USB drive (may need to enable USB boot in EEPROM) +- [ ] Auto-installer detects target disk (NVMe/SD) +- [ ] Installation completes without errors +- [ ] System reboots into installed OS + +### First Boot +- [ ] Archipelago service starts (`systemctl status archipelago`) +- [ ] Nginx starts and serves UI +- [ ] Web UI loads in browser at `http://` +- [ ] Onboarding flow completes +- [ ] Login works with default password + +### Container Stack +- [ ] Podman runs on ARM64 (`podman version`) +- [ ] Bitcoin Knots installs and syncs +- [ ] LND installs and connects to Bitcoin +- [ ] Electrs installs and indexes +- [ ] Mempool installs and shows data +- [ ] FileBrowser installs and serves files + +### Performance +- [ ] Backend response time < 200ms for RPC calls +- [ ] UI renders smoothly (no jank) +- [ ] Container startup time reasonable (< 30s per app) +- [ ] Memory usage stable (no leaks over 24h) + +## Known RPi 5 Considerations + +1. **USB Boot**: RPi 5 needs EEPROM update for USB boot. Run `sudo rpi-eeprom-update` on a stock Raspberry Pi OS first. +2. **NVMe**: RPi 5 supports NVMe via the M.2 HAT. Recommended for performance. +3. **Power**: Use the official 27W USB-C power supply. Underpowered supplies cause throttling. +4. **Thermals**: Consider a heatsink or active cooling case for sustained Bitcoin node operation. +5. **Storage**: Bitcoin blockchain requires ~600GB+. Use NVMe or external SSD. + +## Reporting Issues + +Document any ARM64-specific issues found during testing: +- Architecture-specific container failures +- Performance differences vs x86_64 +- Hardware compatibility problems +- Missing kernel modules or firmware diff --git a/docs/canary-deploy.md b/docs/canary-deploy.md new file mode 100644 index 00000000..f3264a27 --- /dev/null +++ b/docs/canary-deploy.md @@ -0,0 +1,69 @@ +# Canary Deploy Process + +## Overview + +Deploy changes to the secondary server first (192.168.1.198), verify health, then deploy to the primary server (192.168.1.228). This reduces risk by catching issues before they affect the main system. + +## Steps + +### 1. Deploy to Secondary (Canary) + +```bash +# Deploy to secondary server only +TARGET_HOST=archipelago@192.168.1.198 ./scripts/deploy-to-target.sh --live +``` + +### 2. Verify Health + +```bash +# Check health endpoint +curl -s http://192.168.1.198/health + +# Check backend service +ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.198 "sudo systemctl status archipelago" + +# Spot-check the UI +# Open http://192.168.1.198 in browser, verify pages load +``` + +### 3. Deploy to Primary + +Once the secondary is healthy and verified: + +```bash +./scripts/deploy-to-target.sh --live +``` + +### 4. Verify Primary + +```bash +curl -s http://192.168.1.228/health +``` + +## Quick Deploy to Both (Non-Canary) + +If you're confident and want to deploy to both at once: + +```bash +./scripts/deploy-to-target.sh --both +``` + +This deploys to 228 first, then copies the built artifacts to 198. Not a true canary — use the step-by-step process above for safer rollouts. + +## Rollback + +If the canary (198) shows issues, do NOT deploy to primary. Fix the issue first. + +If primary (228) shows issues after deploy: + +```bash +# Check logs +ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228 "sudo journalctl -u archipelago -n 50" + +# Restart services +ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228 "sudo systemctl restart archipelago && sudo systemctl restart nginx" +``` + +## Post-Deploy Health Check + +The deploy script automatically waits up to 60 seconds for the health endpoint to return 200 after deploying. If it fails, check the backend logs for errors. diff --git a/docs/developer-guide.md b/docs/developer-guide.md new file mode 100644 index 00000000..6629806c --- /dev/null +++ b/docs/developer-guide.md @@ -0,0 +1,311 @@ +# Archipelago Developer Guide + +## Project Structure + +``` +archy/ +├── core/ # Rust backend +│ └── archipelago/ +│ ├── src/ +│ │ ├── main.rs # Entry point, module declarations +│ │ ├── api/rpc/ # RPC endpoint handlers +│ │ │ ├── mod.rs # Route dispatcher +│ │ │ ├── auth.rs # Login, session, TOTP +│ │ │ ├── container.rs # Container lifecycle +│ │ │ ├── package.rs # Package install/remove +│ │ │ ├── interfaces.rs # Network interfaces, WiFi, DNS +│ │ │ ├── federation.rs # Federation management +│ │ │ ├── marketplace.rs # Community marketplace +│ │ │ └── ... # Other endpoint groups +│ │ ├── auth.rs # Password hashing, sessions +│ │ ├── config.rs # Configuration loading +│ │ ├── server.rs # HTTP/WS server (axum) +│ │ ├── container/ # Podman integration +│ │ ├── network/ # Network management +│ │ │ ├── dns.rs # DNS configuration +│ │ │ ├── router.rs # UPnP, diagnostics +│ │ │ └── dwn_*.rs # DWN protocol +│ │ ├── federation.rs # Federation protocol +│ │ ├── marketplace.rs # Marketplace discovery +│ │ ├── identity.rs # DID key management +│ │ ├── vpn.rs # VPN (Tailscale/WireGuard) +│ │ ├── mesh.rs # Meshtastic mesh networking +│ │ └── ... +│ ├── Cargo.toml +│ └── tests/ # Integration tests +├── neode-ui/ # Vue 3 frontend +│ ├── src/ +│ │ ├── api/ # RPC client, WebSocket, container client +│ │ │ └── rpc-client.ts # Central RPC client (all backend calls) +│ │ ├── views/ # Page components +│ │ │ ├── Home.vue # Dashboard with system stats +│ │ │ ├── Marketplace.vue # App store (curated + community) +│ │ │ ├── Server.vue # Network, VPN, DNS management +│ │ │ ├── Federation.vue # Federation dashboard +│ │ │ ├── Settings.vue # User settings +│ │ │ ├── Web5.vue # DID, DWN, Nostr +│ │ │ └── ... +│ │ ├── stores/ # Pinia state management +│ │ ├── components/ # Reusable UI components +│ │ ├── composables/ # Vue composables +│ │ ├── router/ # Vue Router with guards +│ │ ├── types/ # TypeScript type definitions +│ │ └── style.css # Global styles + Tailwind utilities +│ ├── vite.config.ts +│ └── package.json +├── scripts/ # Deployment and utility scripts +│ ├── deploy-to-target.sh # Main deploy script +│ ├── first-boot-containers.sh # ISO first-boot setup +│ └── run-tests.sh # CI test runner +├── image-recipe/ # ISO build configuration +│ ├── build-auto-installer-iso.sh +│ └── configs/ # Nginx, systemd configs +├── docs/ # Documentation +│ ├── architecture.md +│ ├── app-manifest-spec.md +│ ├── marketplace-protocol.md +│ └── multi-node-architecture.md +├── apps/ # App manifests (YAML) +├── CLAUDE.md # AI development instructions +└── loop/plan.md # Project roadmap +``` + +## Development Setup + +### Prerequisites + +- **macOS** (development machine): Node.js 20+, npm +- **Linux server** (`192.168.1.228`): Rust toolchain, Podman, Nginx, Debian 12 +- SSH key: `~/.ssh/archipelago-deploy` + +### Local Frontend Development + +```bash +cd neode-ui +npm install +npm start # Vite dev server on :8100, mock backend on :5959 +``` + +The dev server at `http://localhost:8100` uses a mock backend. Login with `password123`. + +### Deploying Changes + +**Never build Rust on macOS.** The deploy script rsyncs source to the Linux server and builds there. + +```bash +# Deploy to live server (builds backend + frontend, restarts services) +./scripts/deploy-to-target.sh --live + +# Deploy to both servers +./scripts/deploy-to-target.sh --both +``` + +The deploy script: +1. Rsyncs source to the server +2. Builds Rust backend on the server (`cargo build --release`) +3. Builds Vue frontend (`npm run build`) +4. Copies artifacts to production paths +5. Restarts the `archipelago` systemd service +6. Runs a health check + +### Running Tests + +```bash +# Frontend tests +cd neode-ui && npm test + +# Backend tests (on dev server via SSH) +ssh -i ~/.ssh/archipelago-deploy archipelago@192.168.1.228 \ + "cd ~/archy/core && cargo test --all-features" + +# Both +./scripts/run-tests.sh +``` + +## Adding a New RPC Endpoint + +### 1. Create the Handler + +Add a handler method in the appropriate file under `core/archipelago/src/api/rpc/`. If no existing file fits, create a new one. + +```rust +// core/archipelago/src/api/rpc/mymodule.rs +use super::RpcHandler; +use anyhow::Result; + +impl RpcHandler { + /// mymodule.action — description of what it does. + pub(super) async fn handle_mymodule_action( + &self, + params: Option, + ) -> Result { + let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; + let name = params + .get("name") + .and_then(|v| v.as_str()) + .ok_or_else(|| anyhow::anyhow!("Missing required parameter: name"))?; + + // Your logic here + let result = do_something(name).await?; + + Ok(serde_json::json!({ "ok": true, "result": result })) + } +} +``` + +**Key patterns:** +- Handlers are `pub(super)` — visible only to the RPC router +- Accept `Option` for params (omit for parameterless endpoints) +- Return `Result` +- Use `self.config.data_dir` for data persistence +- Use `anyhow::bail!()` for error responses + +### 2. Register the Route + +Add the module declaration and route in `core/archipelago/src/api/rpc/mod.rs`: + +```rust +// At the top: +mod mymodule; + +// In the match statement (handle_rpc_call): +"mymodule.action" => self.handle_mymodule_action(params).await, +``` + +### 3. Add Module (if new) + +If your logic warrants a separate module: + +```rust +// core/archipelago/src/main.rs +mod mymodule; // Add to module declarations +``` + +### 4. Frontend Client + +Add a convenience method to `neode-ui/src/api/rpc-client.ts`: + +```typescript +async myAction(params: { name: string }): Promise<{ ok: boolean; result: string }> { + return this.call({ + method: 'mymodule.action', + params, + }) +} +``` + +### 5. Deploy and Test + +```bash +./scripts/deploy-to-target.sh --live +curl -X POST http://192.168.1.228/rpc/v1 \ + -H "Content-Type: application/json" \ + -b "archipelago_session=YOUR_SESSION" \ + -d '{"method":"mymodule.action","params":{"name":"test"}}' +``` + +## Adding a New Vue Page + +### 1. Create the Component + +```vue + + + + +``` + +### 2. Add the Route + +In `neode-ui/src/router/index.ts`, add inside the dashboard children: + +```typescript +{ + path: 'my-page', + name: 'my-page', + component: () => import('@/views/MyPage.vue'), +}, +``` + +### 3. Standards + +- Always use ` diff --git a/neode-ui/src/components/PWAInstallPrompt.vue b/neode-ui/src/components/PWAInstallPrompt.vue index f44c182c..d2598d33 100644 --- a/neode-ui/src/components/PWAInstallPrompt.vue +++ b/neode-ui/src/components/PWAInstallPrompt.vue @@ -46,7 +46,7 @@ onMounted(() => { // Don't show if already dismissed this session or if already installed if (sessionStorage.getItem(DISMISS_KEY) === '1') return if (window.matchMedia('(display-mode: standalone)').matches) return - if ((window.navigator as any).standalone) return + if ((window.navigator as Navigator & { standalone?: boolean }).standalone) return const handler = (e: Event) => { e.preventDefault() @@ -55,11 +55,11 @@ onMounted(() => { } window.addEventListener('beforeinstallprompt', handler) - ;(window as any).__beforeinstallpromptHandler = handler + ;(window as Window & { __beforeinstallpromptHandler?: EventListener }).__beforeinstallpromptHandler = handler }) onBeforeUnmount(() => { - window.removeEventListener('beforeinstallprompt', (window as any).__beforeinstallpromptHandler) + window.removeEventListener('beforeinstallprompt', (window as Window & { __beforeinstallpromptHandler?: EventListener }).__beforeinstallpromptHandler as EventListener) }) function dismiss() { diff --git a/neode-ui/src/components/SplashScreen.vue b/neode-ui/src/components/SplashScreen.vue index 86f9674f..1c07f2ae 100644 --- a/neode-ui/src/components/SplashScreen.vue +++ b/neode-ui/src/components/SplashScreen.vue @@ -205,7 +205,7 @@ watch([showWelcome, showLogo], ([welcome, logo]) => { if ((welcome || logo) && videoElement.value) { if (videoElement.value.paused) { videoElement.value.play().catch(err => { - console.warn('Video autoplay failed:', err) + if (import.meta.env.DEV) console.warn('Video autoplay failed:', err) }) } // Add pause prevention handler once, remove when no longer needed @@ -228,7 +228,7 @@ watch(showWelcome, (isShowing) => { if (isShowing && videoElement.value) { // Start video immediately when welcome appears videoElement.value.play().catch(err => { - console.warn('Video autoplay failed on welcome:', err) + if (import.meta.env.DEV) console.warn('Video autoplay failed on welcome:', err) }) } }) @@ -414,7 +414,7 @@ function startAlienIntro() { playWelcomeNoderunnerSpeech() if (videoElement.value) { videoElement.value.play().catch(err => { - console.warn('Video autoplay failed on welcome:', err) + if (import.meta.env.DEV) console.warn('Video autoplay failed on welcome:', err) }) } backgroundOpacity.value = 0.3 diff --git a/neode-ui/src/components/__tests__/LineChart.test.ts b/neode-ui/src/components/__tests__/LineChart.test.ts new file mode 100644 index 00000000..80f5a5f1 --- /dev/null +++ b/neode-ui/src/components/__tests__/LineChart.test.ts @@ -0,0 +1,117 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { shallowMount } from '@vue/test-utils' +import LineChart from '../LineChart.vue' + +// Mock canvas context +const mockContext = { + clearRect: vi.fn(), + beginPath: vi.fn(), + moveTo: vi.fn(), + lineTo: vi.fn(), + stroke: vi.fn(), + fill: vi.fn(), + fillRect: vi.fn(), + fillText: vi.fn(), + closePath: vi.fn(), + setLineDash: vi.fn(), + save: vi.fn(), + restore: vi.fn(), + scale: vi.fn(), + createLinearGradient: vi.fn().mockReturnValue({ + addColorStop: vi.fn(), + }), + canvas: { width: 600, height: 200 }, + strokeStyle: '', + fillStyle: '', + lineWidth: 0, + font: '', + textAlign: '', + textBaseline: '', + globalAlpha: 1, +} + +beforeEach(() => { + vi.clearAllMocks() + HTMLCanvasElement.prototype.getContext = vi.fn().mockReturnValue(mockContext) +}) + +describe('LineChart', () => { + const sampleDatasets = [ + { label: 'CPU', data: [10, 20, 30, 40, 50], color: '#fb923c' }, + ] + + it('renders a canvas element', () => { + const wrapper = shallowMount(LineChart, { + props: { datasets: sampleDatasets }, + }) + expect(wrapper.find('canvas').exists()).toBe(true) + }) + + it('accepts datasets prop', () => { + const wrapper = shallowMount(LineChart, { + props: { datasets: sampleDatasets }, + }) + expect(wrapper.exists()).toBe(true) + }) + + it('renders with empty datasets', () => { + const wrapper = shallowMount(LineChart, { + props: { datasets: [] }, + }) + expect(wrapper.find('canvas').exists()).toBe(true) + }) + + it('renders with multiple datasets', () => { + const wrapper = shallowMount(LineChart, { + props: { + datasets: [ + { label: 'CPU', data: [10, 20, 30], color: '#fb923c' }, + { label: 'Memory', data: [50, 60, 70], color: '#4ade80' }, + ], + }, + }) + expect(wrapper.exists()).toBe(true) + }) + + it('accepts optional height and width props', () => { + const wrapper = shallowMount(LineChart, { + props: { + datasets: sampleDatasets, + height: 300, + width: 600, + }, + }) + const canvas = wrapper.find('canvas') + expect(canvas.attributes('width')).toBe('600') + expect(canvas.attributes('height')).toBe('300') + }) + + it('uses default width of 400 and height of 180', () => { + const wrapper = shallowMount(LineChart, { + props: { datasets: sampleDatasets }, + }) + const canvas = wrapper.find('canvas') + expect(canvas.attributes('width')).toBe('400') + expect(canvas.attributes('height')).toBe('180') + }) + + it('renders with dataset containing single data point', () => { + const wrapper = shallowMount(LineChart, { + props: { + datasets: [{ label: 'Test', data: [42], color: '#3b82f6' }], + }, + }) + expect(wrapper.exists()).toBe(true) + }) + + it('accepts yMax and yLabel props', () => { + const wrapper = shallowMount(LineChart, { + props: { + datasets: sampleDatasets, + yMax: 100, + yLabel: 'Percent', + }, + }) + expect(wrapper.exists()).toBe(true) + }) +}) diff --git a/neode-ui/src/components/__tests__/PWAInstallPrompt.test.ts b/neode-ui/src/components/__tests__/PWAInstallPrompt.test.ts new file mode 100644 index 00000000..87b76eef --- /dev/null +++ b/neode-ui/src/components/__tests__/PWAInstallPrompt.test.ts @@ -0,0 +1,103 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { shallowMount } from '@vue/test-utils' +import PWAInstallPrompt from '../PWAInstallPrompt.vue' + +describe('PWAInstallPrompt', () => { + beforeEach(() => { + vi.clearAllMocks() + sessionStorage.clear() + // Mock matchMedia to return non-standalone + Object.defineProperty(window, 'matchMedia', { + writable: true, + value: vi.fn().mockImplementation((query: string) => ({ + matches: false, + media: query, + onchange: null, + addEventListener: vi.fn(), + removeEventListener: vi.fn(), + addListener: vi.fn(), + removeListener: vi.fn(), + dispatchEvent: vi.fn(), + })), + }) + }) + + it('renders without errors', () => { + const wrapper = shallowMount(PWAInstallPrompt, { + global: { stubs: { Teleport: true, Transition: true } }, + }) + expect(wrapper.exists()).toBe(true) + }) + + it('does not show prompt initially', () => { + const wrapper = shallowMount(PWAInstallPrompt, { + global: { stubs: { Teleport: true, Transition: true } }, + }) + expect(wrapper.text()).not.toContain('Install Archipelago') + }) + + it('shows prompt after beforeinstallprompt event', async () => { + const wrapper = shallowMount(PWAInstallPrompt, { + global: { stubs: { Teleport: true, Transition: true } }, + }) + + // Fire the beforeinstallprompt event + const event = new Event('beforeinstallprompt') + Object.defineProperty(event, 'preventDefault', { value: vi.fn() }) + window.dispatchEvent(event) + + await wrapper.vm.$nextTick() + expect(wrapper.text()).toContain('Install Archipelago') + }) + + it('hides prompt when dismissed', async () => { + const wrapper = shallowMount(PWAInstallPrompt, { + global: { stubs: { Teleport: true, Transition: true } }, + }) + + // Show prompt + const event = new Event('beforeinstallprompt') + Object.defineProperty(event, 'preventDefault', { value: vi.fn() }) + window.dispatchEvent(event) + await wrapper.vm.$nextTick() + + // Click dismiss button + const dismissBtn = wrapper.findAll('button').find(b => b.text().includes('Not now')) + expect(dismissBtn).toBeDefined() + await dismissBtn!.trigger('click') + expect(sessionStorage.getItem('archipelago_pwa_install_dismissed')).toBe('1') + }) + + it('does not show if already dismissed this session', async () => { + sessionStorage.setItem('archipelago_pwa_install_dismissed', '1') + const wrapper = shallowMount(PWAInstallPrompt, { + global: { stubs: { Teleport: true, Transition: true } }, + }) + + // Fire beforeinstallprompt — should not show + const event = new Event('beforeinstallprompt') + Object.defineProperty(event, 'preventDefault', { value: vi.fn() }) + window.dispatchEvent(event) + await wrapper.vm.$nextTick() + + expect(wrapper.text()).not.toContain('Install Archipelago') + }) + + it('does not show in standalone mode', async () => { + Object.defineProperty(window, 'matchMedia', { + writable: true, + value: vi.fn().mockReturnValue({ matches: true }), + }) + + const wrapper = shallowMount(PWAInstallPrompt, { + global: { stubs: { Teleport: true, Transition: true } }, + }) + + const event = new Event('beforeinstallprompt') + Object.defineProperty(event, 'preventDefault', { value: vi.fn() }) + window.dispatchEvent(event) + await wrapper.vm.$nextTick() + + expect(wrapper.text()).not.toContain('Install Archipelago') + }) +}) diff --git a/neode-ui/src/composables/__tests__/useAudioPlayer.test.ts b/neode-ui/src/composables/__tests__/useAudioPlayer.test.ts new file mode 100644 index 00000000..e13a3fdb --- /dev/null +++ b/neode-ui/src/composables/__tests__/useAudioPlayer.test.ts @@ -0,0 +1,138 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { useAudioPlayer } from '../useAudioPlayer' + +// Mock HTMLAudioElement +class MockAudio { + src = '' + currentTime = 0 + duration = 120 + paused = true + private listeners: Record void>> = {} + + addEventListener(event: string, handler: () => void) { + if (!this.listeners[event]) this.listeners[event] = [] + this.listeners[event].push(handler) + } + + removeEventListener() { + // no-op for tests + } + + play() { + this.paused = false + this.emit('play') + return Promise.resolve() + } + + pause() { + this.paused = true + this.emit('pause') + } + + private emit(event: string) { + const handlers = this.listeners[event] || [] + handlers.forEach(h => h()) + } + + // Helper to simulate events in tests + simulateEvent(event: string) { + this.emit(event) + } +} + +vi.stubGlobal('Audio', MockAudio) + +describe('useAudioPlayer', () => { + beforeEach(() => { + // Reset singleton state by stopping any active playback + const player = useAudioPlayer() + player.stop() + }) + + it('returns all expected properties', () => { + const player = useAudioPlayer() + expect(player.play).toBeTypeOf('function') + expect(player.pause).toBeTypeOf('function') + expect(player.seek).toBeTypeOf('function') + expect(player.stop).toBeTypeOf('function') + expect(player.playing).toBeDefined() + expect(player.currentName).toBeDefined() + expect(player.currentTime).toBeDefined() + expect(player.duration).toBeDefined() + expect(player.progress).toBeDefined() + expect(player.currentSrc).toBeDefined() + expect(player.error).toBeDefined() + }) + + it('starts in stopped state', () => { + const player = useAudioPlayer() + expect(player.playing.value).toBe(false) + expect(player.currentSrc.value).toBeNull() + expect(player.currentName.value).toBe('') + }) + + it('play sets playing state and current source', () => { + const player = useAudioPlayer() + player.play('/audio/test.mp3', 'Test Track') + expect(player.playing.value).toBe(true) + expect(player.currentSrc.value).toBe('/audio/test.mp3') + expect(player.currentName.value).toBe('Test Track') + }) + + it('play toggles pause when same source is playing', () => { + const player = useAudioPlayer() + player.play('/audio/test.mp3', 'Test') + expect(player.playing.value).toBe(true) + // Play same source again — should pause + player.play('/audio/test.mp3', 'Test') + expect(player.playing.value).toBe(false) + }) + + it('play switches to new source', () => { + const player = useAudioPlayer() + player.play('/audio/first.mp3', 'First') + player.play('/audio/second.mp3', 'Second') + expect(player.currentSrc.value).toBe('/audio/second.mp3') + expect(player.currentName.value).toBe('Second') + }) + + it('pause pauses playback', () => { + const player = useAudioPlayer() + player.play('/audio/test.mp3', 'Test') + player.pause() + expect(player.playing.value).toBe(false) + }) + + it('stop resets all state', () => { + const player = useAudioPlayer() + player.play('/audio/test.mp3', 'Test') + player.stop() + expect(player.playing.value).toBe(false) + expect(player.currentSrc.value).toBeNull() + expect(player.currentName.value).toBe('') + }) + + it('progress computes correctly', () => { + const player = useAudioPlayer() + expect(player.progress.value).toBe(0) // duration is 0 + + player.currentTime.value = 30 + player.duration.value = 120 + expect(player.progress.value).toBe(25) // 30/120 * 100 + }) + + it('progress is 0 when duration is 0', () => { + const player = useAudioPlayer() + player.duration.value = 0 + player.currentTime.value = 10 + expect(player.progress.value).toBe(0) + }) + + it('shared state across multiple useAudioPlayer calls', () => { + const p1 = useAudioPlayer() + const p2 = useAudioPlayer() + p1.play('/audio/shared.mp3', 'Shared') + expect(p2.currentSrc.value).toBe('/audio/shared.mp3') + expect(p2.playing.value).toBe(true) + }) +}) diff --git a/neode-ui/src/composables/__tests__/useFileType.test.ts b/neode-ui/src/composables/__tests__/useFileType.test.ts new file mode 100644 index 00000000..8768acb9 --- /dev/null +++ b/neode-ui/src/composables/__tests__/useFileType.test.ts @@ -0,0 +1,202 @@ +import { describe, it, expect } from 'vitest' +import { ref } from 'vue' +import { getFileCategory, useFileType, formatSize, formatDate } from '../useFileType' + +describe('getFileCategory', () => { + it('returns folder for directories', () => { + expect(getFileCategory('', true)).toBe('folder') + expect(getFileCategory('jpg', true)).toBe('folder') + }) + + it('identifies image extensions', () => { + expect(getFileCategory('jpg', false)).toBe('image') + expect(getFileCategory('jpeg', false)).toBe('image') + expect(getFileCategory('png', false)).toBe('image') + expect(getFileCategory('gif', false)).toBe('image') + expect(getFileCategory('webp', false)).toBe('image') + expect(getFileCategory('svg', false)).toBe('image') + expect(getFileCategory('bmp', false)).toBe('image') + expect(getFileCategory('ico', false)).toBe('image') + }) + + it('identifies audio extensions', () => { + expect(getFileCategory('mp3', false)).toBe('audio') + expect(getFileCategory('flac', false)).toBe('audio') + expect(getFileCategory('wav', false)).toBe('audio') + expect(getFileCategory('ogg', false)).toBe('audio') + expect(getFileCategory('aac', false)).toBe('audio') + expect(getFileCategory('m4a', false)).toBe('audio') + }) + + it('identifies video extensions', () => { + expect(getFileCategory('mp4', false)).toBe('video') + expect(getFileCategory('mkv', false)).toBe('video') + expect(getFileCategory('avi', false)).toBe('video') + expect(getFileCategory('mov', false)).toBe('video') + expect(getFileCategory('webm', false)).toBe('video') + }) + + it('identifies document extensions', () => { + expect(getFileCategory('pdf', false)).toBe('document') + expect(getFileCategory('doc', false)).toBe('document') + expect(getFileCategory('docx', false)).toBe('document') + expect(getFileCategory('txt', false)).toBe('document') + expect(getFileCategory('md', false)).toBe('document') + }) + + it('identifies spreadsheet extensions', () => { + expect(getFileCategory('xls', false)).toBe('spreadsheet') + expect(getFileCategory('xlsx', false)).toBe('spreadsheet') + expect(getFileCategory('csv', false)).toBe('spreadsheet') + expect(getFileCategory('ods', false)).toBe('spreadsheet') + }) + + it('identifies archive extensions', () => { + expect(getFileCategory('zip', false)).toBe('archive') + expect(getFileCategory('tar', false)).toBe('archive') + expect(getFileCategory('gz', false)).toBe('archive') + expect(getFileCategory('rar', false)).toBe('archive') + expect(getFileCategory('7z', false)).toBe('archive') + }) + + it('returns file for unknown extensions', () => { + expect(getFileCategory('xyz', false)).toBe('file') + expect(getFileCategory('', false)).toBe('file') + expect(getFileCategory('bin', false)).toBe('file') + }) +}) + +describe('useFileType', () => { + it('returns correct category and computed values for an image', () => { + const ext = ref('jpg') + const isDir = ref(false) + const result = useFileType(ext, isDir) + + expect(result.category.value).toBe('image') + expect(result.isImage.value).toBe(true) + expect(result.isAudio.value).toBe(false) + expect(result.isVideo.value).toBe(false) + expect(result.iconColor.value).toBe('text-blue-400') + expect(result.badgeLabel.value).toBe('Image') + }) + + it('returns correct values for audio', () => { + const ext = ref('mp3') + const isDir = ref(false) + const result = useFileType(ext, isDir) + + expect(result.category.value).toBe('audio') + expect(result.isAudio.value).toBe(true) + expect(result.isImage.value).toBe(false) + expect(result.iconColor.value).toBe('text-orange-400') + expect(result.badgeLabel.value).toBe('Audio') + }) + + it('returns correct values for video', () => { + const ext = ref('mp4') + const isDir = ref(false) + const result = useFileType(ext, isDir) + + expect(result.category.value).toBe('video') + expect(result.isVideo.value).toBe(true) + expect(result.iconColor.value).toBe('text-purple-400') + }) + + it('returns folder when isDir is true', () => { + const ext = ref('jpg') + const isDir = ref(true) + const result = useFileType(ext, isDir) + + expect(result.category.value).toBe('folder') + expect(result.isImage.value).toBe(false) + expect(result.iconColor.value).toBe('text-amber-400') + expect(result.badgeLabel.value).toBe('Folder') + }) + + it('reacts to ref changes', () => { + const ext = ref('jpg') + const isDir = ref(false) + const result = useFileType(ext, isDir) + + expect(result.category.value).toBe('image') + + ext.value = 'mp3' + expect(result.category.value).toBe('audio') + expect(result.isAudio.value).toBe(true) + expect(result.isImage.value).toBe(false) + }) + + it('provides icon paths for each category', () => { + const ext = ref('pdf') + const isDir = ref(false) + const result = useFileType(ext, isDir) + + expect(result.iconPaths.value).toBeDefined() + expect(result.iconPaths.value.length).toBeGreaterThan(0) + }) + + it('provides badge class for each category', () => { + const ext = ref('zip') + const isDir = ref(false) + const result = useFileType(ext, isDir) + + expect(result.badgeClass.value).toContain('bg-yellow') + }) +}) + +describe('formatSize', () => { + it('formats 0 bytes', () => { + expect(formatSize(0)).toBe('0 B') + }) + + it('formats bytes', () => { + expect(formatSize(500)).toBe('500 B') + }) + + it('formats kilobytes', () => { + expect(formatSize(1024)).toBe('1.0 KB') + expect(formatSize(1536)).toBe('1.5 KB') + }) + + it('formats megabytes', () => { + expect(formatSize(1048576)).toBe('1.0 MB') + }) + + it('formats gigabytes', () => { + expect(formatSize(1073741824)).toBe('1.0 GB') + }) + + it('formats terabytes', () => { + expect(formatSize(1099511627776)).toBe('1.0 TB') + }) +}) + +describe('formatDate', () => { + it('returns "Just now" for very recent dates', () => { + const now = new Date().toISOString() + expect(formatDate(now)).toBe('Just now') + }) + + it('returns minutes ago for recent dates', () => { + const fiveMinAgo = new Date(Date.now() - 5 * 60 * 1000).toISOString() + expect(formatDate(fiveMinAgo)).toBe('5m ago') + }) + + it('returns hours ago for dates within 24h', () => { + const threeHoursAgo = new Date(Date.now() - 3 * 60 * 60 * 1000).toISOString() + expect(formatDate(threeHoursAgo)).toBe('3h ago') + }) + + it('returns days ago for dates within a week', () => { + const twoDaysAgo = new Date(Date.now() - 2 * 24 * 60 * 60 * 1000).toISOString() + expect(formatDate(twoDaysAgo)).toBe('2d ago') + }) + + it('returns formatted date for older dates', () => { + const oldDate = new Date('2025-01-15').toISOString() + const result = formatDate(oldDate) + // Should be a locale date string, not a relative time + expect(result).toMatch(/\d/) + expect(result).not.toContain('ago') + }) +}) diff --git a/neode-ui/src/composables/__tests__/useMarketplaceApp.test.ts b/neode-ui/src/composables/__tests__/useMarketplaceApp.test.ts new file mode 100644 index 00000000..aba926af --- /dev/null +++ b/neode-ui/src/composables/__tests__/useMarketplaceApp.test.ts @@ -0,0 +1,97 @@ +import { describe, it, expect, beforeEach } from 'vitest' +import { useMarketplaceApp } from '../useMarketplaceApp' + +describe('useMarketplaceApp', () => { + beforeEach(() => { + const { clearCurrentApp } = useMarketplaceApp() + clearCurrentApp() + }) + + it('getCurrentApp returns null initially', () => { + const { getCurrentApp } = useMarketplaceApp() + expect(getCurrentApp()).toBeNull() + }) + + it('setCurrentApp stores a full app', () => { + const { setCurrentApp, getCurrentApp } = useMarketplaceApp() + setCurrentApp({ + id: 'bitcoin', + title: 'Bitcoin Core', + version: '25.0', + icon: '/icons/btc.png', + category: 'Finance', + description: 'Bitcoin node', + author: 'Satoshi', + source: 'github', + manifestUrl: 'https://example.com/manifest', + url: 'https://example.com', + repoUrl: 'https://github.com/bitcoin/bitcoin', + s9pkUrl: '', + dockerImage: 'bitcoin:25.0', + }) + + const app = getCurrentApp() + expect(app).not.toBeNull() + expect(app!.id).toBe('bitcoin') + expect(app!.title).toBe('Bitcoin Core') + expect(app!.version).toBe('25.0') + }) + + it('setCurrentApp with partial app fills defaults', () => { + const { setCurrentApp, getCurrentApp } = useMarketplaceApp() + setCurrentApp({ id: 'lnd' }) + + const app = getCurrentApp() + expect(app).not.toBeNull() + expect(app!.id).toBe('lnd') + expect(app!.title).toBe('') + expect(app!.version).toBe('') + expect(app!.icon).toBe('') + expect(app!.dockerImage).toBe('') + }) + + it('manifestUrl falls back to s9pkUrl then url', () => { + const { setCurrentApp, getCurrentApp } = useMarketplaceApp() + setCurrentApp({ id: 'test', s9pkUrl: 'https://s9pk.example.com/app.s9pk' }) + + const app = getCurrentApp() + expect(app!.manifestUrl).toBe('https://s9pk.example.com/app.s9pk') + expect(app!.url).toBe('https://s9pk.example.com/app.s9pk') + }) + + it('url falls back to s9pkUrl then manifestUrl', () => { + const { setCurrentApp, getCurrentApp } = useMarketplaceApp() + setCurrentApp({ id: 'test', manifestUrl: 'https://manifest.example.com' }) + + const app = getCurrentApp() + expect(app!.url).toBe('https://manifest.example.com') + }) + + it('clearCurrentApp sets app to null', () => { + const { setCurrentApp, clearCurrentApp, getCurrentApp } = useMarketplaceApp() + setCurrentApp({ id: 'bitcoin' }) + expect(getCurrentApp()).not.toBeNull() + clearCurrentApp() + expect(getCurrentApp()).toBeNull() + }) + + it('shared state across multiple useMarketplaceApp calls', () => { + const instance1 = useMarketplaceApp() + const instance2 = useMarketplaceApp() + + instance1.setCurrentApp({ id: 'mempool', title: 'Mempool' }) + const app = instance2.getCurrentApp() + expect(app!.id).toBe('mempool') + expect(app!.title).toBe('Mempool') + }) + + it('handles description as object', () => { + const { setCurrentApp, getCurrentApp } = useMarketplaceApp() + setCurrentApp({ + id: 'test', + description: { short: 'Short desc', long: 'Long description' }, + }) + const app = getCurrentApp() + expect(app!.description).toEqual({ short: 'Short desc', long: 'Long description' }) + }) +}) diff --git a/neode-ui/src/composables/__tests__/useMessageToast.test.ts b/neode-ui/src/composables/__tests__/useMessageToast.test.ts new file mode 100644 index 00000000..93a2ed3d --- /dev/null +++ b/neode-ui/src/composables/__tests__/useMessageToast.test.ts @@ -0,0 +1,178 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' + +const mockPush = vi.fn() + +vi.mock('vue-router', () => ({ + useRouter: () => ({ push: mockPush }), +})) + +vi.mock('@/api/rpc-client', () => ({ + rpcClient: { + getReceivedMessages: vi.fn(), + }, +})) + +import { useMessageToast } from '../useMessageToast' +import { rpcClient } from '@/api/rpc-client' + +const mockedRpc = vi.mocked(rpcClient) + +describe('useMessageToast', () => { + beforeEach(() => { + vi.clearAllMocks() + vi.useFakeTimers() + // Reset shared singleton state + const toast = useMessageToast() + toast.stopPolling() + toast.receivedMessages.value = [] + toast.lastMessageCount.value = 0 + toast.loadingMessages.value = false + toast.toastMessage.value = { show: false, text: '' } + }) + + afterEach(() => { + const toast = useMessageToast() + toast.stopPolling() + vi.useRealTimers() + }) + + it('starts with empty state', () => { + const toast = useMessageToast() + expect(toast.receivedMessages.value).toEqual([]) + expect(toast.lastMessageCount.value).toBe(0) + expect(toast.loadingMessages.value).toBe(false) + expect(toast.toastMessage.value.show).toBe(false) + expect(toast.unreadCount.value).toBe(0) + }) + + it('loadReceivedMessages fetches and stores messages', async () => { + mockedRpc.getReceivedMessages.mockResolvedValue({ + messages: [ + { from_pubkey: 'abc', message: 'Hello', timestamp: '2026-01-01' }, + ], + }) + const toast = useMessageToast() + await toast.loadReceivedMessages() + + expect(toast.receivedMessages.value.length).toBe(1) + expect(toast.lastMessageCount.value).toBe(1) + expect(toast.loadingMessages.value).toBe(false) + }) + + it('does not show toast on initial load', async () => { + mockedRpc.getReceivedMessages.mockResolvedValue({ + messages: [{ from_pubkey: 'a', message: 'Hi', timestamp: '2026-01-01' }], + }) + const toast = useMessageToast() + await toast.loadReceivedMessages() + + expect(toast.toastMessage.value.show).toBe(false) + }) + + it('shows toast when new messages arrive after initial load', async () => { + const toast = useMessageToast() + + // Initial load + mockedRpc.getReceivedMessages.mockResolvedValue({ + messages: [{ from_pubkey: 'a', message: 'First', timestamp: '2026-01-01' }], + }) + await toast.loadReceivedMessages() + + // New message arrives + mockedRpc.getReceivedMessages.mockResolvedValue({ + messages: [ + { from_pubkey: 'a', message: 'First', timestamp: '2026-01-01' }, + { from_pubkey: 'b', message: 'Second', timestamp: '2026-01-02' }, + ], + }) + await toast.loadReceivedMessages() + + expect(toast.toastMessage.value.show).toBe(true) + expect(toast.toastMessage.value.text).toBe('Second') + }) + + it('shows count for multiple new messages', async () => { + const toast = useMessageToast() + + // Initial load + mockedRpc.getReceivedMessages.mockResolvedValue({ + messages: [{ from_pubkey: 'a', message: 'One', timestamp: '2026-01-01' }], + }) + await toast.loadReceivedMessages() + + // Multiple new messages + mockedRpc.getReceivedMessages.mockResolvedValue({ + messages: [ + { from_pubkey: 'a', message: 'One', timestamp: '2026-01-01' }, + { from_pubkey: 'b', message: 'Two', timestamp: '2026-01-02' }, + { from_pubkey: 'c', message: 'Three', timestamp: '2026-01-03' }, + ], + }) + await toast.loadReceivedMessages() + + expect(toast.toastMessage.value.show).toBe(true) + expect(toast.toastMessage.value.text).toBe('2 new messages') + }) + + it('unreadCount reflects difference', async () => { + const toast = useMessageToast() + toast.receivedMessages.value = [ + { from_pubkey: 'a', message: 'Hi', timestamp: '2026-01-01' }, + { from_pubkey: 'b', message: 'Hey', timestamp: '2026-01-02' }, + ] + toast.lastMessageCount.value = 1 + expect(toast.unreadCount.value).toBe(1) + }) + + it('unreadCount is never negative', () => { + const toast = useMessageToast() + toast.receivedMessages.value = [] + toast.lastMessageCount.value = 5 + expect(toast.unreadCount.value).toBe(0) + }) + + it('markAsRead syncs lastMessageCount', () => { + const toast = useMessageToast() + toast.receivedMessages.value = [ + { from_pubkey: 'a', message: 'Hi', timestamp: '2026-01-01' }, + { from_pubkey: 'b', message: 'Hey', timestamp: '2026-01-02' }, + ] + toast.lastMessageCount.value = 0 + toast.markAsRead() + expect(toast.lastMessageCount.value).toBe(2) + expect(toast.unreadCount.value).toBe(0) + }) + + it('dismissToastAndOpenMessages clears toast and navigates', () => { + const toast = useMessageToast() + toast.toastMessage.value = { show: true, text: 'New message' } + toast.dismissToastAndOpenMessages() + + expect(toast.toastMessage.value.show).toBe(false) + expect(mockPush).toHaveBeenCalledWith({ path: '/dashboard/web5', query: { tab: 'messages' } }) + }) + + it('stops polling on 401 error', async () => { + const toast = useMessageToast() + mockedRpc.getReceivedMessages.mockRejectedValue(new Error('401 Unauthorized')) + toast.startPolling() + + // Wait for initial load triggered by startPolling + await vi.advanceTimersByTimeAsync(0) + + // Polling should have stopped, so advancing time should NOT call again + vi.clearAllMocks() + await vi.advanceTimersByTimeAsync(60000) + expect(mockedRpc.getReceivedMessages).not.toHaveBeenCalled() + }) + + it('startPolling does not create duplicate timers', () => { + const toast = useMessageToast() + mockedRpc.getReceivedMessages.mockResolvedValue({ messages: [] }) + toast.startPolling() + toast.startPolling() + toast.startPolling() + // Should only have one timer — verify by stopping and checking no more calls + toast.stopPolling() + }) +}) diff --git a/neode-ui/src/composables/__tests__/useToast.test.ts b/neode-ui/src/composables/__tests__/useToast.test.ts new file mode 100644 index 00000000..74b1c5df --- /dev/null +++ b/neode-ui/src/composables/__tests__/useToast.test.ts @@ -0,0 +1,131 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { useToast } from '../useToast' + +describe('useToast', () => { + beforeEach(() => { + vi.useFakeTimers() + // Get a fresh toast instance and clear any leftover state + const { toasts, dismiss } = useToast() + // Dismiss all existing toasts + for (const t of [...toasts.value]) { + dismiss(t.id) + } + vi.advanceTimersByTime(500) + }) + + afterEach(() => { + vi.useRealTimers() + }) + + it('creates a success toast', () => { + const { success, toasts } = useToast() + + success('Operation complete') + + expect(toasts.value.length).toBeGreaterThanOrEqual(1) + const toast = toasts.value[toasts.value.length - 1]! + expect(toast.message).toBe('Operation complete') + expect(toast.variant).toBe('success') + expect(toast.dismissing).toBe(false) + }) + + it('creates an error toast', () => { + const { error, toasts } = useToast() + + error('Something went wrong') + + const toast = toasts.value[toasts.value.length - 1]! + expect(toast.message).toBe('Something went wrong') + expect(toast.variant).toBe('error') + }) + + it('creates an info toast', () => { + const { info, toasts } = useToast() + + info('FYI: Node syncing') + + const toast = toasts.value[toasts.value.length - 1]! + expect(toast.message).toBe('FYI: Node syncing') + expect(toast.variant).toBe('info') + }) + + it('auto-dismisses toast after duration', () => { + const { success, toasts } = useToast() + + success('Will auto-dismiss') + const toast = toasts.value[toasts.value.length - 1]! + const toastId = toast.id + + expect(toasts.value.some((t) => t.id === toastId)).toBe(true) + + // After 3000ms, the toast should start dismissing + vi.advanceTimersByTime(3000) + + const dismissingToast = toasts.value.find((t) => t.id === toastId) + if (dismissingToast) { + expect(dismissingToast.dismissing).toBe(true) + } + + // After another 300ms, the toast should be fully removed + vi.advanceTimersByTime(300) + + expect(toasts.value.some((t) => t.id === toastId)).toBe(false) + }) + + it('dismiss marks toast as dismissing then removes it', () => { + const { info, toasts, dismiss } = useToast() + + info('Dismissable') + const toast = toasts.value[toasts.value.length - 1]! + + dismiss(toast.id) + + // Should be marked as dismissing + const found = toasts.value.find((t) => t.id === toast.id) + if (found) { + expect(found.dismissing).toBe(true) + } + + // After 300ms animation delay, should be removed + vi.advanceTimersByTime(300) + + expect(toasts.value.some((t) => t.id === toast.id)).toBe(false) + }) + + it('dismiss is a no-op for nonexistent toast ID', () => { + const { dismiss, toasts } = useToast() + const countBefore = toasts.value.length + + dismiss(999999) + + expect(toasts.value.length).toBe(countBefore) + }) + + it('each toast gets a unique ID', () => { + const { info, toasts } = useToast() + + info('First') + info('Second') + info('Third') + + const ids = toasts.value.slice(-3).map((t) => t.id) + const uniqueIds = new Set(ids) + expect(uniqueIds.size).toBe(3) + }) + + it('caps visible toasts at 5', () => { + const { info, toasts } = useToast() + + for (let i = 0; i < 7; i++) { + info(`Toast ${i}`) + } + + expect(toasts.value.length).toBeLessThanOrEqual(5) + }) + + it('toasts ref is readonly', () => { + const { toasts } = useToast() + // The readonly wrapper prevents direct mutation + expect(typeof toasts.value).toBe('object') + }) +}) diff --git a/neode-ui/src/composables/useLoginSounds.ts b/neode-ui/src/composables/useLoginSounds.ts index c1896a28..8638e32e 100644 --- a/neode-ui/src/composables/useLoginSounds.ts +++ b/neode-ui/src/composables/useLoginSounds.ts @@ -93,7 +93,7 @@ export function stopSynthwave() { if (introAudio) { if (introGain && audioContext) { const t = audioContext.currentTime - introGain.gain.setValueAtTime(1, t) + introGain.gain.setValueAtTime(introGain.gain.value, t) introGain.gain.linearRampToValueAtTime(0.001, t + 0.2) } setTimeout(() => { @@ -104,6 +104,23 @@ export function stopSynthwave() { } } +/** Stop ALL login audio and close AudioContext. Call on route change to dashboard. */ +export function stopAllAudio() { + // Stop synthwave loop + if (introAudio) { + introAudio.pause() + introAudio = null + introGain = null + } + // Stop intro typing + stopIntroTyping() + // Close AudioContext to kill any lingering BufferSource nodes (playLoopStart) + if (audioContext) { + audioContext.close().catch(() => {}) + audioContext = null + } +} + /** Pop sound - plays when intro initiator (tap to start) is pressed */ export function playPop() { const audio = new Audio('/assets/audio/pop.mp3') diff --git a/neode-ui/src/composables/useMarketplaceApp.ts b/neode-ui/src/composables/useMarketplaceApp.ts index ff4acd67..4e0016ab 100644 --- a/neode-ui/src/composables/useMarketplaceApp.ts +++ b/neode-ui/src/composables/useMarketplaceApp.ts @@ -14,6 +14,8 @@ export interface MarketplaceAppInfo { repoUrl: string s9pkUrl: string dockerImage: string + /** External web URL for iframe-based web apps (no container needed) */ + webUrl?: string } // Simple in-memory store for the current marketplace app @@ -36,6 +38,7 @@ export function useMarketplaceApp() { repoUrl: app.repoUrl ?? '', s9pkUrl: app.s9pkUrl ?? '', dockerImage: app.dockerImage ?? '', + webUrl: (app as Record).webUrl as string | undefined, } } diff --git a/neode-ui/src/composables/useMessageToast.ts b/neode-ui/src/composables/useMessageToast.ts index 6126bfcc..de0a4ac3 100644 --- a/neode-ui/src/composables/useMessageToast.ts +++ b/neode-ui/src/composables/useMessageToast.ts @@ -48,7 +48,7 @@ export function useMessageToast() { stopPolling() return } - console.error('Failed to load messages:', e) + if (import.meta.env.DEV) console.error('Failed to load messages:', e) } finally { loadingMessages.value = false } diff --git a/neode-ui/src/locales/en.json b/neode-ui/src/locales/en.json index 50bee153..3f14d2fb 100644 --- a/neode-ui/src/locales/en.json +++ b/neode-ui/src/locales/en.json @@ -314,7 +314,7 @@ "modeEasyDesc": "Goal-based interface. Choose what you want to do, and the system handles the rest.", "modePro": "Pro", "modeProDesc": "Full control over all services. Configure everything manually with all technical details.", - "modeChat": "Chat", + "modeChat": "AIUI", "modeChatDesc": "Conversational AI interface. Manage your node through natural language. Coming soon." }, "marketplace": { @@ -353,8 +353,8 @@ "closeAssistant": "Close AI Assistant", "loadingAssistant": "Loading AI assistant...", "aiAssistant": "AI Assistant", - "notConfigured": "AI Assistant is not yet configured on this node.", - "deployCta": "Deploy the AIUI app from the App Store to enable this feature." + "notConfigured": "AI Assistant needs to be enabled before use.", + "deployCta": "Go to Settings to configure your AI provider API key, then return here to start chatting." }, "web5": { "title": "Web5", @@ -682,6 +682,13 @@ "rollbackSuccess": "Rolled back to previous version. Service will restart.", "rollbackFailed": "Rollback failed." }, + "kiosk": { + "pressEsc": "Press ESC to exit", + "online": "Online", + "offline": "Offline", + "escHint": "Press ESC to exit apps", + "navHint": "Use arrow keys to navigate" + }, "kioskRecovery": { "title": "Archipelago Recovery", "subtitle": "Kiosk failsafe — no authentication required", diff --git a/neode-ui/src/router/__tests__/onboarding.test.ts b/neode-ui/src/router/__tests__/onboarding.test.ts new file mode 100644 index 00000000..9e6318a8 --- /dev/null +++ b/neode-ui/src/router/__tests__/onboarding.test.ts @@ -0,0 +1,243 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { createRouter, createWebHistory } from 'vue-router' +import { setActivePinia, createPinia } from 'pinia' +import { defineComponent } from 'vue' + +// Mock the app store module +const mockStore = { + isAuthenticated: false, + isConnected: false, + isReconnecting: false, + needsSessionValidation: vi.fn().mockReturnValue(false), + checkSession: vi.fn().mockResolvedValue(false), + connectWebSocket: vi.fn().mockResolvedValue(undefined), +} + +vi.mock('@/stores/app', () => ({ + useAppStore: () => mockStore, +})) + +const Stub = defineComponent({ template: '

' }) + +function createTestRouter() { + return createRouter({ + history: createWebHistory(), + routes: [ + { + path: '/', + component: Stub, + meta: { public: true }, + children: [ + { path: '', component: Stub }, + { path: 'login', name: 'login', component: Stub }, + { path: 'onboarding/intro', name: 'onboarding-intro', component: Stub }, + { path: 'onboarding/options', name: 'onboarding-options', component: Stub }, + { path: 'onboarding/path', name: 'onboarding-path', component: Stub }, + { path: 'onboarding/did', name: 'onboarding-did', component: Stub }, + { path: 'onboarding/identity', name: 'onboarding-identity', component: Stub }, + { path: 'onboarding/backup', name: 'onboarding-backup', component: Stub }, + { path: 'onboarding/verify', name: 'onboarding-verify', component: Stub }, + { path: 'onboarding/done', name: 'onboarding-done', component: Stub }, + ], + }, + { + path: '/dashboard', + component: Stub, + children: [ + { path: '', name: 'home', component: Stub }, + { path: 'apps', name: 'apps', component: Stub }, + { path: 'settings', name: 'settings', component: Stub }, + ], + }, + ], + }) +} + +describe('Onboarding Routing Flow', () => { + let router: ReturnType + + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + mockStore.isAuthenticated = false + mockStore.isConnected = false + mockStore.isReconnecting = false + mockStore.needsSessionValidation.mockReturnValue(false) + mockStore.checkSession.mockResolvedValue(false) + router = createTestRouter() + + // Add the same beforeEach guard as the real router + router.beforeEach(async (to) => { + const isPublic = to.meta.public + + if (isPublic) { + if (to.path === '/login' && mockStore.isAuthenticated) { + if (mockStore.needsSessionValidation()) { + return true + } + return { name: 'home' } + } + return true + } + + if (mockStore.needsSessionValidation()) { + mockStore.checkSession() + return true + } + + if (!mockStore.isAuthenticated) { + const hasSession = await mockStore.checkSession() + if (hasSession) return true + return '/login' + } + + if (!mockStore.isConnected && !mockStore.isReconnecting) { + mockStore.connectWebSocket() + } + + return true + }) + }) + + describe('unauthenticated users', () => { + it('redirects to /login when accessing /dashboard, not to /onboarding', async () => { + mockStore.checkSession.mockResolvedValue(false) + await router.push('/dashboard') + expect(router.currentRoute.value.path).toBe('/login') + expect(router.currentRoute.value.path).not.toContain('/onboarding') + }) + + it('redirects to /login when accessing /dashboard/apps', async () => { + mockStore.checkSession.mockResolvedValue(false) + await router.push('/dashboard/apps') + expect(router.currentRoute.value.path).toBe('/login') + }) + + it('redirects to /login when accessing /dashboard/settings', async () => { + mockStore.checkSession.mockResolvedValue(false) + await router.push('/dashboard/settings') + expect(router.currentRoute.value.path).toBe('/login') + }) + }) + + describe('onboarding routes are accessible when authenticated', () => { + beforeEach(() => { + mockStore.isAuthenticated = true + }) + + it('allows access to /onboarding/intro', async () => { + await router.push('/onboarding/intro') + expect(router.currentRoute.value.path).toBe('/onboarding/intro') + expect(router.currentRoute.value.name).toBe('onboarding-intro') + }) + + it('allows access to /onboarding/options', async () => { + await router.push('/onboarding/options') + expect(router.currentRoute.value.path).toBe('/onboarding/options') + expect(router.currentRoute.value.name).toBe('onboarding-options') + }) + + it('allows access to /onboarding/path', async () => { + await router.push('/onboarding/path') + expect(router.currentRoute.value.path).toBe('/onboarding/path') + expect(router.currentRoute.value.name).toBe('onboarding-path') + }) + }) + + describe('OnboardingDid route', () => { + it('is accessible when authenticated', async () => { + mockStore.isAuthenticated = true + await router.push('/onboarding/did') + expect(router.currentRoute.value.path).toBe('/onboarding/did') + expect(router.currentRoute.value.name).toBe('onboarding-did') + }) + + it('is accessible when unauthenticated (public route)', async () => { + mockStore.isAuthenticated = false + await router.push('/onboarding/did') + expect(router.currentRoute.value.path).toBe('/onboarding/did') + expect(router.currentRoute.value.name).toBe('onboarding-did') + }) + }) + + describe('post-onboarding navigation to dashboard', () => { + it('allows authenticated users to navigate from onboarding to /dashboard', async () => { + mockStore.isAuthenticated = true + + // Start at onboarding done page + await router.push('/onboarding/done') + expect(router.currentRoute.value.path).toBe('/onboarding/done') + + // Navigate to dashboard (simulating post-onboarding completion) + await router.push('/dashboard') + expect(router.currentRoute.value.path).toBe('/dashboard') + expect(router.currentRoute.value.name).toBe('home') + }) + + it('allows authenticated users to navigate from onboarding/did to /dashboard', async () => { + mockStore.isAuthenticated = true + + await router.push('/onboarding/did') + expect(router.currentRoute.value.path).toBe('/onboarding/did') + + await router.push('/dashboard') + expect(router.currentRoute.value.path).toBe('/dashboard') + expect(router.currentRoute.value.name).toBe('home') + }) + + it('allows navigation through the full onboarding sequence', async () => { + mockStore.isAuthenticated = true + + await router.push('/onboarding/intro') + expect(router.currentRoute.value.name).toBe('onboarding-intro') + + await router.push('/onboarding/path') + expect(router.currentRoute.value.name).toBe('onboarding-path') + + await router.push('/onboarding/did') + expect(router.currentRoute.value.name).toBe('onboarding-did') + + await router.push('/onboarding/identity') + expect(router.currentRoute.value.name).toBe('onboarding-identity') + + await router.push('/onboarding/backup') + expect(router.currentRoute.value.name).toBe('onboarding-backup') + + await router.push('/onboarding/verify') + expect(router.currentRoute.value.name).toBe('onboarding-verify') + + await router.push('/onboarding/done') + expect(router.currentRoute.value.name).toBe('onboarding-done') + }) + }) + + describe('dashboard guard blocks unauthenticated access after onboarding', () => { + it('prevents unauthenticated navigation from onboarding/done to /dashboard', async () => { + // Start at onboarding done (public route, accessible without auth) + await router.push('/onboarding/done') + expect(router.currentRoute.value.path).toBe('/onboarding/done') + + // Try to navigate to dashboard without auth + mockStore.checkSession.mockResolvedValue(false) + await router.push('/dashboard') + expect(router.currentRoute.value.path).toBe('/login') + }) + + it('prevents unauthenticated access to /dashboard even with stale session', async () => { + mockStore.isAuthenticated = false + mockStore.needsSessionValidation.mockReturnValue(false) + mockStore.checkSession.mockResolvedValue(false) + + await router.push('/dashboard/apps') + expect(router.currentRoute.value.path).toBe('/login') + }) + + it('allows dashboard access when session check succeeds', async () => { + mockStore.isAuthenticated = false + mockStore.checkSession.mockResolvedValue(true) + + await router.push('/dashboard') + expect(router.currentRoute.value.path).toBe('/dashboard') + }) + }) +}) diff --git a/neode-ui/src/router/index.ts b/neode-ui/src/router/index.ts index fd8cd139..71c0f0ea 100644 --- a/neode-ui/src/router/index.ts +++ b/neode-ui/src/router/index.ts @@ -1,6 +1,7 @@ import { createRouter, createWebHistory } from 'vue-router' import { nextTick } from 'vue' import { useAppStore } from '../stores/app' +import { stopAllAudio } from '../composables/useLoginSounds' const router = createRouter({ history: createWebHistory(), @@ -61,6 +62,18 @@ const router = createRouter({ }, ], }, + { + path: '/recovery', + name: 'recovery', + component: () => import('../views/KioskRecovery.vue'), + meta: { public: true }, + }, + { + path: '/kiosk', + name: 'kiosk', + component: () => import('../views/Kiosk.vue'), + meta: { public: true }, + }, { path: '/dashboard', component: () => import('../views/Dashboard.vue'), @@ -110,16 +123,36 @@ const router = createRouter({ name: 'server', component: () => import('../views/Server.vue'), }, + { + path: 'monitoring', + name: 'monitoring', + component: () => import('../views/Monitoring.vue'), + }, + { + path: 'server/federation', + name: 'federation', + component: () => import('../views/Federation.vue'), + }, { path: 'web5', name: 'web5', component: () => import('../views/Web5.vue'), }, + { + path: 'web5/credentials', + name: 'credentials', + component: () => import('../views/Credentials.vue'), + }, { path: 'settings', name: 'settings', component: () => import('../views/Settings.vue'), }, + { + path: 'settings/update', + name: 'system-update', + component: () => import('../views/SystemUpdate.vue'), + }, { path: 'goals/:goalId', name: 'goal-detail', @@ -216,13 +249,20 @@ router.beforeEach(async (to, _from, next) => { // Validated and authenticated - ensure WebSocket is connected if (!store.isConnected && !store.isReconnecting) { store.connectWebSocket().catch((err) => { - console.warn('[Router] WebSocket connection failed:', err) + if (import.meta.env.DEV) console.warn('[Router] WebSocket connection failed:', err) }) } next() }) +// Stop all login/splash audio when entering the dashboard +router.afterEach((to, from) => { + if (to.path.startsWith('/dashboard') && !from.path.startsWith('/dashboard')) { + stopAllAudio() + } +}) + // Focus Home nav item for gamepad when landing on dashboard home (e.g. after login) router.afterEach((to) => { if (to.path === '/dashboard' || to.path === '/dashboard/') { diff --git a/neode-ui/src/stores/__tests__/aiPermissions.test.ts b/neode-ui/src/stores/__tests__/aiPermissions.test.ts new file mode 100644 index 00000000..12cf3abb --- /dev/null +++ b/neode-ui/src/stores/__tests__/aiPermissions.test.ts @@ -0,0 +1,106 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useAIPermissionsStore, AI_PERMISSION_CATEGORIES } from '../aiPermissions' + +const STORAGE_KEY = 'archipelago-ai-permissions' + +describe('useAIPermissionsStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + localStorage.clear() + vi.clearAllMocks() + }) + + it('starts with empty permissions when no localStorage', () => { + const store = useAIPermissionsStore() + expect(store.enabled.size).toBe(0) + expect(store.noneEnabled).toBe(true) + expect(store.allEnabled).toBe(false) + }) + + it('loads valid categories from localStorage', () => { + localStorage.setItem(STORAGE_KEY, JSON.stringify(['apps', 'system'])) + setActivePinia(createPinia()) + const store = useAIPermissionsStore() + expect(store.isEnabled('apps')).toBe(true) + expect(store.isEnabled('system')).toBe(true) + expect(store.enabled.size).toBe(2) + }) + + it('filters invalid categories from localStorage', () => { + localStorage.setItem(STORAGE_KEY, JSON.stringify(['apps', 'invalid-category', 'system'])) + setActivePinia(createPinia()) + const store = useAIPermissionsStore() + expect(store.enabled.size).toBe(2) + expect(store.isEnabled('apps')).toBe(true) + expect(store.isEnabled('system')).toBe(true) + }) + + it('handles corrupt localStorage gracefully', () => { + localStorage.setItem(STORAGE_KEY, 'not-valid-json{') + setActivePinia(createPinia()) + const store = useAIPermissionsStore() + expect(store.enabled.size).toBe(0) + }) + + it('toggle adds a category', () => { + const store = useAIPermissionsStore() + store.toggle('bitcoin') + expect(store.isEnabled('bitcoin')).toBe(true) + }) + + it('toggle removes an enabled category', () => { + const store = useAIPermissionsStore() + store.toggle('bitcoin') + store.toggle('bitcoin') + expect(store.isEnabled('bitcoin')).toBe(false) + }) + + it('toggle persists to localStorage', () => { + const store = useAIPermissionsStore() + store.toggle('apps') + const stored = JSON.parse(localStorage.getItem(STORAGE_KEY) || '[]') + expect(stored).toContain('apps') + }) + + it('enableAll enables all categories', () => { + const store = useAIPermissionsStore() + store.enableAll() + expect(store.allEnabled).toBe(true) + expect(store.enabled.size).toBe(AI_PERMISSION_CATEGORIES.length) + for (const cat of AI_PERMISSION_CATEGORIES) { + expect(store.isEnabled(cat.id)).toBe(true) + } + }) + + it('disableAll disables all categories', () => { + const store = useAIPermissionsStore() + store.enableAll() + store.disableAll() + expect(store.noneEnabled).toBe(true) + expect(store.enabled.size).toBe(0) + }) + + it('enabledCategories returns array of enabled IDs', () => { + const store = useAIPermissionsStore() + store.toggle('apps') + store.toggle('network') + expect(store.enabledCategories).toContain('apps') + expect(store.enabledCategories).toContain('network') + expect(store.enabledCategories.length).toBe(2) + }) + + it('AI_PERMISSION_CATEGORIES has 10 categories', () => { + expect(AI_PERMISSION_CATEGORIES.length).toBe(10) + }) + + it('all categories have required fields', () => { + for (const cat of AI_PERMISSION_CATEGORIES) { + expect(cat.id).toBeTruthy() + expect(cat.label).toBeTruthy() + expect(cat.description).toBeTruthy() + expect(cat.icon).toBeTruthy() + expect(cat.group).toBeTruthy() + } + }) +}) diff --git a/neode-ui/src/stores/__tests__/appLauncher.test.ts b/neode-ui/src/stores/__tests__/appLauncher.test.ts new file mode 100644 index 00000000..14ddb888 --- /dev/null +++ b/neode-ui/src/stores/__tests__/appLauncher.test.ts @@ -0,0 +1,147 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' + +import { useAppLauncherStore } from '../appLauncher' + +// Mock window.open for new-tab tests +const mockWindowOpen = vi.fn() +vi.stubGlobal('open', mockWindowOpen) + +describe('useAppLauncherStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + // Default to HTTP to avoid proxy rewriting + Object.defineProperty(window, 'location', { + value: { origin: 'http://192.168.1.228', protocol: 'http:', hostname: '192.168.1.228' }, + writable: true, + configurable: true, + }) + }) + + it('starts closed with empty state', () => { + const store = useAppLauncherStore() + expect(store.isOpen).toBe(false) + expect(store.url).toBe('') + expect(store.title).toBe('') + }) + + it('opens an app in the iframe overlay', () => { + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:8080', title: 'Mempool' }) + + expect(store.isOpen).toBe(true) + expect(store.url).toBe('http://192.168.1.228:8080') + expect(store.title).toBe('Mempool') + expect(mockWindowOpen).not.toHaveBeenCalled() + }) + + it('opens BTCPay (port 23000) in a new tab due to X-Frame-Options', () => { + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:23000', title: 'BTCPay' }) + + expect(store.isOpen).toBe(false) + expect(mockWindowOpen).toHaveBeenCalledWith( + 'http://192.168.1.228:23000', + '_blank', + 'noopener,noreferrer', + ) + }) + + it('opens Home Assistant (port 8123) in a new tab', () => { + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:8123', title: 'Home Assistant' }) + + expect(store.isOpen).toBe(false) + expect(mockWindowOpen).toHaveBeenCalled() + }) + + it('opens Grafana (port 3000) in a new tab', () => { + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:3000', title: 'Grafana' }) + + expect(store.isOpen).toBe(false) + expect(mockWindowOpen).toHaveBeenCalled() + }) + + it('opens in new tab when openInNewTab flag is set', () => { + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:8080', title: 'Mempool', openInNewTab: true }) + + expect(store.isOpen).toBe(false) + expect(mockWindowOpen).toHaveBeenCalledWith( + 'http://192.168.1.228:8080', + '_blank', + 'noopener,noreferrer', + ) + }) + + it('rewrites URL to proxy path on HTTPS for same-host apps', () => { + Object.defineProperty(window, 'location', { + value: { origin: 'https://192.168.1.228', protocol: 'https:', hostname: '192.168.1.228' }, + writable: true, + configurable: true, + }) + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:8083', title: 'FileBrowser' }) + + expect(store.isOpen).toBe(true) + expect(store.url).toBe('https://192.168.1.228/app/filebrowser/') + }) + + it('does not rewrite URL on HTTP (no mixed content)', () => { + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.228:8083', title: 'FileBrowser' }) + + expect(store.url).toBe('http://192.168.1.228:8083') + }) + + it('does not rewrite URL for different hosts', () => { + Object.defineProperty(window, 'location', { + value: { origin: 'https://192.168.1.228', protocol: 'https:', hostname: '192.168.1.228' }, + writable: true, + configurable: true, + }) + const store = useAppLauncherStore() + + store.open({ url: 'http://192.168.1.100:8083', title: 'Remote FileBrowser' }) + + // Different host — no proxy rewriting + expect(store.url).toBe('http://192.168.1.100:8083') + }) + + it('close resets state', () => { + const store = useAppLauncherStore() + store.open({ url: 'http://192.168.1.228:8080', title: 'Mempool' }) + + store.close() + + expect(store.isOpen).toBe(false) + expect(store.url).toBe('') + expect(store.title).toBe('') + }) + + it('close restores focus to previous element', async () => { + vi.useFakeTimers() + const store = useAppLauncherStore() + const mockButton = { focus: vi.fn() } as unknown as HTMLElement + Object.defineProperty(document, 'activeElement', { value: mockButton, configurable: true }) + + store.open({ url: 'http://192.168.1.228:8080', title: 'Mempool' }) + store.close() + + expect(store.isOpen).toBe(false) + expect(store.url).toBe('') + + // requestAnimationFrame fires the focus restore callback + vi.runAllTimers() + vi.useRealTimers() + }) +}) diff --git a/neode-ui/src/stores/__tests__/cli.test.ts b/neode-ui/src/stores/__tests__/cli.test.ts new file mode 100644 index 00000000..19c17893 --- /dev/null +++ b/neode-ui/src/stores/__tests__/cli.test.ts @@ -0,0 +1,65 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' + +vi.mock('@/composables/useNavSounds', () => ({ + playNavSound: vi.fn(), +})) + +import { useCLIStore } from '../cli' +import { playNavSound } from '@/composables/useNavSounds' + +const mockedPlayNavSound = vi.mocked(playNavSound) + +describe('useCLIStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + }) + + it('starts closed', () => { + const store = useCLIStore() + expect(store.isOpen).toBe(false) + }) + + it('open sets isOpen to true and plays sound', () => { + const store = useCLIStore() + store.open() + expect(store.isOpen).toBe(true) + expect(mockedPlayNavSound).toHaveBeenCalledWith('action') + }) + + it('close sets isOpen to false without sound', () => { + const store = useCLIStore() + store.open() + vi.clearAllMocks() + store.close() + expect(store.isOpen).toBe(false) + expect(mockedPlayNavSound).not.toHaveBeenCalled() + }) + + it('toggle opens and plays sound when closed', () => { + const store = useCLIStore() + store.toggle() + expect(store.isOpen).toBe(true) + expect(mockedPlayNavSound).toHaveBeenCalledWith('action') + }) + + it('toggle closes without sound when open', () => { + const store = useCLIStore() + store.open() + vi.clearAllMocks() + store.toggle() + expect(store.isOpen).toBe(false) + expect(mockedPlayNavSound).not.toHaveBeenCalled() + }) + + it('multiple toggles alternate state', () => { + const store = useCLIStore() + store.toggle() + expect(store.isOpen).toBe(true) + store.toggle() + expect(store.isOpen).toBe(false) + store.toggle() + expect(store.isOpen).toBe(true) + }) +}) diff --git a/neode-ui/src/stores/__tests__/cloud.test.ts b/neode-ui/src/stores/__tests__/cloud.test.ts new file mode 100644 index 00000000..c66381f6 --- /dev/null +++ b/neode-ui/src/stores/__tests__/cloud.test.ts @@ -0,0 +1,233 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' + +vi.mock('@/api/filebrowser-client', () => ({ + fileBrowserClient: { + login: vi.fn(), + listDirectory: vi.fn(), + upload: vi.fn(), + deleteItem: vi.fn(), + downloadUrl: vi.fn(), + createFolder: vi.fn(), + }, +})) + +import { useCloudStore } from '../cloud' +import { fileBrowserClient } from '@/api/filebrowser-client' + +const mockedClient = vi.mocked(fileBrowserClient) + +const mockItems = [ + { name: 'photos', path: '/photos', size: 0, modified: '2026-01-01', isDir: true, type: '', extension: '' }, + { name: 'readme.md', path: '/readme.md', size: 256, modified: '2026-01-02', isDir: false, type: '', extension: 'md' }, + { name: 'archive.zip', path: '/archive.zip', size: 4096, modified: '2026-01-03', isDir: false, type: '', extension: 'zip' }, +] + +describe('useCloudStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + }) + + it('starts with default state', () => { + const store = useCloudStore() + expect(store.currentPath).toBe('/') + expect(store.items).toEqual([]) + expect(store.loading).toBe(false) + expect(store.error).toBeNull() + expect(store.authenticated).toBe(false) + }) + + it('init authenticates with filebrowser', async () => { + mockedClient.login.mockResolvedValue(true) + const store = useCloudStore() + + const result = await store.init() + + expect(result).toBe(true) + expect(store.authenticated).toBe(true) + expect(mockedClient.login).toHaveBeenCalledOnce() + }) + + it('init returns false on auth failure', async () => { + mockedClient.login.mockResolvedValue(false) + const store = useCloudStore() + + const result = await store.init() + + expect(result).toBe(false) + expect(store.authenticated).toBe(false) + }) + + it('init skips login if already authenticated', async () => { + mockedClient.login.mockResolvedValue(true) + const store = useCloudStore() + + await store.init() + await store.init() + + expect(mockedClient.login).toHaveBeenCalledOnce() + }) + + it('navigate loads items and updates path', async () => { + mockedClient.login.mockResolvedValue(true) + mockedClient.listDirectory.mockResolvedValue(mockItems) + const store = useCloudStore() + store.authenticated = true + + await store.navigate('/photos') + + expect(store.items).toEqual(mockItems) + expect(store.currentPath).toBe('/photos') + expect(store.loading).toBe(false) + expect(store.error).toBeNull() + }) + + it('navigate authenticates if not authenticated', async () => { + mockedClient.login.mockResolvedValue(true) + mockedClient.listDirectory.mockResolvedValue(mockItems) + const store = useCloudStore() + + await store.navigate('/') + + expect(mockedClient.login).toHaveBeenCalled() + expect(store.items).toEqual(mockItems) + }) + + it('navigate sets error on auth failure', async () => { + mockedClient.login.mockResolvedValue(false) + const store = useCloudStore() + + await store.navigate('/') + + expect(store.error).toBe('Failed to authenticate with File Browser') + expect(store.loading).toBe(false) + }) + + it('navigate falls back to creating directory on list failure', async () => { + const store = useCloudStore() + store.authenticated = true + + // First listDirectory fails, then createFolder succeeds, then retry succeeds + mockedClient.listDirectory + .mockRejectedValueOnce(new Error('Not found')) + .mockResolvedValueOnce([]) + mockedClient.createFolder.mockResolvedValue(undefined) + + await store.navigate('/new-folder') + + expect(mockedClient.createFolder).toHaveBeenCalledWith('/', 'new-folder') + expect(store.currentPath).toBe('/new-folder') + }) + + it('navigate falls back to root when directory creation also fails', async () => { + const store = useCloudStore() + store.authenticated = true + + // Call 1: listDirectory('/deep/nested') rejects + // Call 2: listDirectory('/') in the fallback catch resolves + mockedClient.listDirectory + .mockRejectedValueOnce(new Error('Not found')) + .mockResolvedValueOnce(mockItems) + + mockedClient.createFolder.mockRejectedValueOnce(new Error('Create failed')) + + await store.navigate('/deep/nested') + + expect(store.currentPath).toBe('/') + expect(store.items).toEqual(mockItems) + }) + + it('navigate sets error when root listing fails', async () => { + const store = useCloudStore() + store.authenticated = true + + mockedClient.listDirectory.mockRejectedValueOnce(new Error('Server error')) + + await store.navigate('/') + + expect(store.error).toBe('Failed to list root directory') + }) + + it('breadcrumbs computes from path', () => { + const store = useCloudStore() + store.currentPath = '/photos/vacation/2026' + + expect(store.breadcrumbs).toEqual([ + { name: 'Home', path: '/' }, + { name: 'photos', path: '/photos' }, + { name: 'vacation', path: '/photos/vacation' }, + { name: '2026', path: '/photos/vacation/2026' }, + ]) + }) + + it('breadcrumbs at root only shows Home', () => { + const store = useCloudStore() + expect(store.breadcrumbs).toEqual([{ name: 'Home', path: '/' }]) + }) + + it('sortedItems puts directories first, sorted alphabetically', () => { + const store = useCloudStore() + store.items = [ + { name: 'readme.md', path: '/readme.md', size: 256, modified: '2026-01-01', isDir: false, type: '', extension: 'md' }, + { name: 'docs', path: '/docs', size: 0, modified: '2026-01-01', isDir: true, type: '', extension: '' }, + { name: 'archive.zip', path: '/archive.zip', size: 4096, modified: '2026-01-01', isDir: false, type: '', extension: 'zip' }, + { name: 'assets', path: '/assets', size: 0, modified: '2026-01-01', isDir: true, type: '', extension: '' }, + ] + + const sorted = store.sortedItems + expect(sorted.map((i) => i.name)).toEqual(['assets', 'docs', 'archive.zip', 'readme.md']) + }) + + it('uploadFile uploads and refreshes', async () => { + const store = useCloudStore() + store.authenticated = true + store.currentPath = '/uploads' + mockedClient.upload.mockResolvedValue(undefined) + mockedClient.listDirectory.mockResolvedValue([]) + + const file = new File(['test'], 'test.txt') + await store.uploadFile(file) + + expect(mockedClient.upload).toHaveBeenCalledWith('/uploads', file) + expect(mockedClient.listDirectory).toHaveBeenCalled() + }) + + it('deleteItem deletes and refreshes', async () => { + const store = useCloudStore() + store.authenticated = true + store.currentPath = '/' + mockedClient.deleteItem.mockResolvedValue(undefined) + mockedClient.listDirectory.mockResolvedValue([]) + + await store.deleteItem('/old-file.txt') + + expect(mockedClient.deleteItem).toHaveBeenCalledWith('/old-file.txt') + expect(mockedClient.listDirectory).toHaveBeenCalled() + }) + + it('downloadUrl delegates to filebrowser client', () => { + mockedClient.downloadUrl.mockReturnValue('http://localhost/api/raw/file.txt?auth=token') + const store = useCloudStore() + + const url = store.downloadUrl('/file.txt') + + expect(url).toBe('http://localhost/api/raw/file.txt?auth=token') + expect(mockedClient.downloadUrl).toHaveBeenCalledWith('/file.txt') + }) + + it('reset clears all state', () => { + const store = useCloudStore() + store.currentPath = '/deep/path' + store.items = mockItems + store.loading = true + store.error = 'something' + + store.reset() + + expect(store.currentPath).toBe('/') + expect(store.items).toEqual([]) + expect(store.loading).toBe(false) + expect(store.error).toBeNull() + }) +}) diff --git a/neode-ui/src/stores/__tests__/controller.test.ts b/neode-ui/src/stores/__tests__/controller.test.ts new file mode 100644 index 00000000..07b4a26a --- /dev/null +++ b/neode-ui/src/stores/__tests__/controller.test.ts @@ -0,0 +1,52 @@ +import { describe, it, expect, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useControllerStore } from '../controller' + +describe('useControllerStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + }) + + it('starts with default state', () => { + const store = useControllerStore() + expect(store.isActive).toBe(false) + expect(store.gamepadCount).toBe(0) + }) + + it('setActive sets isActive to true', () => { + const store = useControllerStore() + store.setActive(true) + expect(store.isActive).toBe(true) + }) + + it('setActive sets isActive to false', () => { + const store = useControllerStore() + store.setActive(true) + store.setActive(false) + expect(store.isActive).toBe(false) + }) + + it('setGamepadCount updates count and activates when > 0', () => { + const store = useControllerStore() + store.setGamepadCount(2) + expect(store.gamepadCount).toBe(2) + expect(store.isActive).toBe(true) + }) + + it('setGamepadCount deactivates when count is 0', () => { + const store = useControllerStore() + store.setGamepadCount(1) + expect(store.isActive).toBe(true) + store.setGamepadCount(0) + expect(store.gamepadCount).toBe(0) + expect(store.isActive).toBe(false) + }) + + it('setActive does not affect gamepadCount', () => { + const store = useControllerStore() + store.setGamepadCount(3) + store.setActive(false) + expect(store.isActive).toBe(false) + expect(store.gamepadCount).toBe(3) + }) +}) diff --git a/neode-ui/src/stores/__tests__/goals.test.ts b/neode-ui/src/stores/__tests__/goals.test.ts new file mode 100644 index 00000000..90c99bd6 --- /dev/null +++ b/neode-ui/src/stores/__tests__/goals.test.ts @@ -0,0 +1,227 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' + +// Mock the app store +const mockPackages: Record = {} +vi.mock('@/stores/app', () => ({ + useAppStore: () => ({ + packages: mockPackages, + }), +})) + +// Mock the goals data with a controlled set +vi.mock('@/data/goals', () => ({ + GOALS: [ + { + id: 'accept-payments', + title: 'Accept Payments', + subtitle: 'Receive Bitcoin and Lightning payments', + icon: 'payments', + category: 'payments', + requiredApps: ['bitcoin-knots', 'lnd'], + steps: [ + { id: 'install-bitcoin', title: 'Install Bitcoin', description: '', appId: 'bitcoin-knots', action: 'install', isAutomatic: true }, + { id: 'install-lnd', title: 'Install LND', description: '', appId: 'lnd', action: 'install', isAutomatic: true }, + { id: 'open-channel', title: 'Open Channel', description: '', action: 'configure', isAutomatic: false }, + ], + estimatedTime: '~30 min', + difficulty: 'beginner', + }, + { + id: 'create-identity', + title: 'Create Identity', + subtitle: 'Sovereign digital identity', + icon: 'identity', + category: 'identity', + requiredApps: [], + steps: [ + { id: 'generate-did', title: 'Generate DID', description: '', action: 'verify', isAutomatic: true }, + { id: 'setup-nostr', title: 'Setup Nostr', description: '', action: 'configure', isAutomatic: false }, + ], + estimatedTime: '~5 min', + difficulty: 'beginner', + }, + { + id: 'store-photos', + title: 'Store Photos', + subtitle: 'Private photo backup', + icon: 'photos', + category: 'storage', + requiredApps: ['immich'], + steps: [ + { id: 'install-immich', title: 'Install Immich', description: '', appId: 'immich', action: 'install', isAutomatic: true }, + { id: 'configure-immich', title: 'Configure', description: '', action: 'configure', isAutomatic: false }, + ], + estimatedTime: '~15 min', + difficulty: 'beginner', + }, + ], +})) + +import { useGoalStore } from '../goals' + +describe('useGoalStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + localStorage.clear() + // Clear mock packages + Object.keys(mockPackages).forEach((k) => delete mockPackages[k]) + }) + + it('starts with empty progress', () => { + const store = useGoalStore() + expect(store.progress).toEqual({}) + }) + + it('loads progress from localStorage', () => { + const savedProgress = { + 'accept-payments': { + goalId: 'accept-payments', + status: 'in-progress', + currentStepIndex: 1, + completedSteps: ['install-bitcoin'], + startedAt: 1000, + }, + } + localStorage.setItem('archipelago-goal-progress', JSON.stringify(savedProgress)) + + const store = useGoalStore() + expect(store.progress['accept-payments']).toBeDefined() + expect(store.progress['accept-payments']!.completedSteps).toContain('install-bitcoin') + }) + + it('handles corrupt localStorage data', () => { + localStorage.setItem('archipelago-goal-progress', 'not-valid-json{{{') + + const store = useGoalStore() + expect(store.progress).toEqual({}) + }) + + it('startGoal creates progress entry and saves', () => { + const store = useGoalStore() + + store.startGoal('accept-payments') + + expect(store.progress['accept-payments']).toBeDefined() + expect(store.progress['accept-payments']!.status).toBe('in-progress') + expect(store.progress['accept-payments']!.currentStepIndex).toBe(0) + expect(store.progress['accept-payments']!.completedSteps).toEqual([]) + expect(localStorage.getItem('archipelago-goal-progress')).not.toBeNull() + }) + + it('completeStep adds step to completedSteps', () => { + const store = useGoalStore() + store.startGoal('accept-payments') + + store.completeStep('accept-payments', 'install-bitcoin') + + expect(store.progress['accept-payments']!.completedSteps).toContain('install-bitcoin') + }) + + it('completeStep does not duplicate step IDs', () => { + const store = useGoalStore() + store.startGoal('accept-payments') + + store.completeStep('accept-payments', 'install-bitcoin') + store.completeStep('accept-payments', 'install-bitcoin') + + expect(store.progress['accept-payments']!.completedSteps.filter((s) => s === 'install-bitcoin')).toHaveLength(1) + }) + + it('completeStep marks goal completed when all steps done', () => { + const store = useGoalStore() + store.startGoal('accept-payments') + + store.completeStep('accept-payments', 'install-bitcoin') + store.completeStep('accept-payments', 'install-lnd') + store.completeStep('accept-payments', 'open-channel') + + expect(store.progress['accept-payments']!.status).toBe('completed') + }) + + it('completeStep is a no-op when goal not started', () => { + const store = useGoalStore() + + store.completeStep('accept-payments', 'install-bitcoin') + + expect(store.progress['accept-payments']).toBeUndefined() + }) + + it('resetGoal removes progress entry', () => { + const store = useGoalStore() + store.startGoal('accept-payments') + expect(store.progress['accept-payments']).toBeDefined() + + store.resetGoal('accept-payments') + + expect(store.progress['accept-payments']).toBeUndefined() + }) + + describe('getGoalStatus', () => { + it('returns not-started for unknown goal', () => { + const store = useGoalStore() + expect(store.getGoalStatus('nonexistent')).toBe('not-started') + }) + + it('returns not-started when no apps installed and no progress', () => { + const store = useGoalStore() + expect(store.getGoalStatus('accept-payments')).toBe('not-started') + }) + + it('returns completed when all required apps are running', () => { + mockPackages['bitcoin-knots'] = { state: 'running' } + mockPackages['lnd'] = { state: 'running' } + + const store = useGoalStore() + expect(store.getGoalStatus('accept-payments')).toBe('completed') + }) + + it('returns in-progress when some required apps are installed', () => { + mockPackages['bitcoin-knots'] = { state: 'running' } + + const store = useGoalStore() + expect(store.getGoalStatus('accept-payments')).toBe('in-progress') + }) + + it('uses manual progress for goals without required apps', () => { + const store = useGoalStore() + + // create-identity has no required apps + expect(store.getGoalStatus('create-identity')).toBe('not-started') + + store.startGoal('create-identity') + expect(store.getGoalStatus('create-identity')).toBe('in-progress') + }) + + it('recognizes app aliases (immich-server matches immich)', () => { + mockPackages['immich-server'] = { state: 'running' } + + const store = useGoalStore() + expect(store.getGoalStatus('store-photos')).toBe('completed') + }) + + it('auto-syncs install steps from actual package state', () => { + mockPackages['bitcoin-knots'] = { state: 'stopped' } + + const store = useGoalStore() + store.getGoalStatus('accept-payments') + + // Should have auto-created progress and marked install-bitcoin as completed + expect(store.progress['accept-payments']).toBeDefined() + expect(store.progress['accept-payments']!.completedSteps).toContain('install-bitcoin') + }) + }) + + it('goalStatuses computes status for all goals', () => { + mockPackages['bitcoin-knots'] = { state: 'running' } + mockPackages['lnd'] = { state: 'running' } + + const store = useGoalStore() + const statuses = store.goalStatuses + + expect(statuses['accept-payments']).toBe('completed') + expect(statuses['create-identity']).toBe('not-started') + expect(statuses['store-photos']).toBe('not-started') + }) +}) diff --git a/neode-ui/src/stores/__tests__/loginTransition.test.ts b/neode-ui/src/stores/__tests__/loginTransition.test.ts new file mode 100644 index 00000000..089986e3 --- /dev/null +++ b/neode-ui/src/stores/__tests__/loginTransition.test.ts @@ -0,0 +1,52 @@ +import { describe, it, expect, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useLoginTransitionStore } from '../loginTransition' + +describe('useLoginTransitionStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + }) + + it('starts with all flags false', () => { + const store = useLoginTransitionStore() + expect(store.justLoggedIn).toBe(false) + expect(store.pendingWelcomeTyping).toBe(false) + expect(store.startWelcomeTyping).toBe(false) + }) + + it('setJustLoggedIn updates justLoggedIn', () => { + const store = useLoginTransitionStore() + store.setJustLoggedIn(true) + expect(store.justLoggedIn).toBe(true) + store.setJustLoggedIn(false) + expect(store.justLoggedIn).toBe(false) + }) + + it('setPendingWelcomeTyping updates pendingWelcomeTyping', () => { + const store = useLoginTransitionStore() + store.setPendingWelcomeTyping(true) + expect(store.pendingWelcomeTyping).toBe(true) + store.setPendingWelcomeTyping(false) + expect(store.pendingWelcomeTyping).toBe(false) + }) + + it('setStartWelcomeTyping updates startWelcomeTyping', () => { + const store = useLoginTransitionStore() + store.setStartWelcomeTyping(true) + expect(store.startWelcomeTyping).toBe(true) + store.setStartWelcomeTyping(false) + expect(store.startWelcomeTyping).toBe(false) + }) + + it('flags are independent of each other', () => { + const store = useLoginTransitionStore() + store.setJustLoggedIn(true) + store.setPendingWelcomeTyping(true) + expect(store.startWelcomeTyping).toBe(false) + + store.setStartWelcomeTyping(true) + store.setJustLoggedIn(false) + expect(store.pendingWelcomeTyping).toBe(true) + expect(store.startWelcomeTyping).toBe(true) + }) +}) diff --git a/neode-ui/src/stores/__tests__/screensaver.test.ts b/neode-ui/src/stores/__tests__/screensaver.test.ts new file mode 100644 index 00000000..22b0c784 --- /dev/null +++ b/neode-ui/src/stores/__tests__/screensaver.test.ts @@ -0,0 +1,81 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useScreensaverStore } from '../screensaver' + +describe('useScreensaverStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.useFakeTimers() + }) + + afterEach(() => { + vi.useRealTimers() + }) + + it('starts inactive', () => { + const store = useScreensaverStore() + expect(store.isActive).toBe(false) + }) + + it('activate sets isActive to true', () => { + const store = useScreensaverStore() + store.activate() + expect(store.isActive).toBe(true) + }) + + it('deactivate sets isActive to false', () => { + const store = useScreensaverStore() + store.activate() + store.deactivate() + expect(store.isActive).toBe(false) + }) + + it('deactivate starts inactivity timer that activates after 3 minutes', () => { + const store = useScreensaverStore() + store.deactivate() + expect(store.isActive).toBe(false) + + vi.advanceTimersByTime(3 * 60 * 1000) + expect(store.isActive).toBe(true) + }) + + it('resetInactivityTimer restarts the 3-minute countdown', () => { + const store = useScreensaverStore() + store.deactivate() + + // Advance 2 minutes + vi.advanceTimersByTime(2 * 60 * 1000) + expect(store.isActive).toBe(false) + + // Reset timer + store.resetInactivityTimer() + + // Advance another 2 minutes (would have triggered without reset) + vi.advanceTimersByTime(2 * 60 * 1000) + expect(store.isActive).toBe(false) + + // Full 3 minutes from reset + vi.advanceTimersByTime(1 * 60 * 1000) + expect(store.isActive).toBe(true) + }) + + it('clearInactivityTimer prevents activation', () => { + const store = useScreensaverStore() + store.deactivate() + store.clearInactivityTimer() + + vi.advanceTimersByTime(5 * 60 * 1000) + expect(store.isActive).toBe(false) + }) + + it('activate clears any pending timer', () => { + const store = useScreensaverStore() + store.deactivate() + store.activate() + + // If timer wasn't cleared, deactivating and waiting would trigger twice + store.deactivate() + vi.advanceTimersByTime(3 * 60 * 1000) + expect(store.isActive).toBe(true) + }) +}) diff --git a/neode-ui/src/stores/__tests__/spotlight.test.ts b/neode-ui/src/stores/__tests__/spotlight.test.ts new file mode 100644 index 00000000..2f2be129 --- /dev/null +++ b/neode-ui/src/stores/__tests__/spotlight.test.ts @@ -0,0 +1,194 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' + +// Mock the nav sounds module +vi.mock('@/composables/useNavSounds', () => ({ + playNavSound: vi.fn(), +})) + +import { useSpotlightStore } from '../spotlight' +import { playNavSound } from '@/composables/useNavSounds' + +const mockedPlayNavSound = vi.mocked(playNavSound) + +describe('useSpotlightStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + localStorage.clear() + }) + + it('starts closed with default state', () => { + const store = useSpotlightStore() + expect(store.isOpen).toBe(false) + expect(store.selectedIndex).toBe(0) + expect(store.recentItems).toEqual([]) + }) + + it('open sets isOpen to true and plays sound', () => { + const store = useSpotlightStore() + + store.open() + + expect(store.isOpen).toBe(true) + expect(store.selectedIndex).toBe(0) + expect(mockedPlayNavSound).toHaveBeenCalledWith('action') + }) + + it('close sets isOpen to false and resets index', () => { + const store = useSpotlightStore() + store.open() + + store.close() + + expect(store.isOpen).toBe(false) + expect(store.selectedIndex).toBe(0) + }) + + it('toggle opens when closed', () => { + const store = useSpotlightStore() + + store.toggle() + + expect(store.isOpen).toBe(true) + }) + + it('toggle closes when open', () => { + const store = useSpotlightStore() + store.open() + + store.toggle() + + expect(store.isOpen).toBe(false) + }) + + it('setSelectedIndex updates the selected index', () => { + const store = useSpotlightStore() + + store.setSelectedIndex(3) + + expect(store.selectedIndex).toBe(3) + }) + + describe('recent items', () => { + it('addRecentItem adds item with timestamp', () => { + const store = useSpotlightStore() + + store.addRecentItem({ id: 'home', label: 'Home', path: '/dashboard', type: 'navigate' }) + + expect(store.recentItems).toHaveLength(1) + expect(store.recentItems[0]!.id).toBe('home') + expect(store.recentItems[0]!.label).toBe('Home') + expect(store.recentItems[0]!.timestamp).toBeGreaterThan(0) + }) + + it('addRecentItem deduplicates by id and type', () => { + const store = useSpotlightStore() + + store.addRecentItem({ id: 'home', label: 'Home', path: '/dashboard', type: 'navigate' }) + store.addRecentItem({ id: 'home', label: 'Home Updated', path: '/dashboard', type: 'navigate' }) + + expect(store.recentItems).toHaveLength(1) + expect(store.recentItems[0]!.label).toBe('Home Updated') + }) + + it('addRecentItem keeps different types with same id', () => { + const store = useSpotlightStore() + + store.addRecentItem({ id: 'bitcoin', label: 'Bitcoin (navigate)', type: 'navigate' }) + store.addRecentItem({ id: 'bitcoin', label: 'Bitcoin (action)', type: 'action' }) + + expect(store.recentItems).toHaveLength(2) + }) + + it('addRecentItem caps at 8 items', () => { + const store = useSpotlightStore() + + for (let i = 0; i < 10; i++) { + store.addRecentItem({ id: `item-${i}`, label: `Item ${i}`, type: 'navigate' }) + } + + expect(store.recentItems).toHaveLength(8) + // Most recent should be first + expect(store.recentItems[0]!.id).toBe('item-9') + }) + + it('addRecentItem persists to localStorage', () => { + const store = useSpotlightStore() + + store.addRecentItem({ id: 'apps', label: 'Apps', path: '/apps', type: 'navigate' }) + + const stored = JSON.parse(localStorage.getItem('archipelago-spotlight-recent')!) + expect(stored).toHaveLength(1) + expect(stored[0].id).toBe('apps') + }) + + it('loadRecentItems reads from localStorage', () => { + const saved = [ + { id: 'home', label: 'Home', path: '/dashboard', type: 'navigate', timestamp: 1000 }, + { id: 'apps', label: 'Apps', path: '/apps', type: 'navigate', timestamp: 2000 }, + ] + localStorage.setItem('archipelago-spotlight-recent', JSON.stringify(saved)) + + const store = useSpotlightStore() + store.loadRecentItems() + + expect(store.recentItems).toHaveLength(2) + expect(store.recentItems[0]!.id).toBe('home') + }) + + it('loadRecentItems handles corrupt localStorage', () => { + localStorage.setItem('archipelago-spotlight-recent', 'not-json{{{') + + const store = useSpotlightStore() + store.loadRecentItems() + + expect(store.recentItems).toEqual([]) + }) + + it('loadRecentItems handles empty localStorage', () => { + const store = useSpotlightStore() + store.loadRecentItems() + + expect(store.recentItems).toEqual([]) + }) + + it('open calls loadRecentItems', () => { + const saved = [ + { id: 'test', label: 'Test', type: 'navigate', timestamp: 1000 }, + ] + localStorage.setItem('archipelago-spotlight-recent', JSON.stringify(saved)) + + const store = useSpotlightStore() + store.open() + + expect(store.recentItems).toHaveLength(1) + }) + }) + + describe('help modal', () => { + it('showHelpModal opens the modal with content', () => { + const store = useSpotlightStore() + + store.showHelpModal({ + title: 'What is Bitcoin?', + content: 'A peer-to-peer electronic cash system.', + relatedPath: '/apps/bitcoin', + }) + + expect(store.helpModal.show).toBe(true) + expect(store.helpModal.title).toBe('What is Bitcoin?') + expect(store.helpModal.content).toBe('A peer-to-peer electronic cash system.') + expect(store.helpModal.relatedPath).toBe('/apps/bitcoin') + }) + + it('closeHelpModal closes the modal', () => { + const store = useSpotlightStore() + store.showHelpModal({ title: 'Test', content: 'Content' }) + + store.closeHelpModal() + + expect(store.helpModal.show).toBe(false) + }) + }) +}) diff --git a/neode-ui/src/stores/__tests__/uiMode.test.ts b/neode-ui/src/stores/__tests__/uiMode.test.ts new file mode 100644 index 00000000..5f33ece1 --- /dev/null +++ b/neode-ui/src/stores/__tests__/uiMode.test.ts @@ -0,0 +1,90 @@ +import { describe, it, expect, beforeEach } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useUIModeStore } from '../uiMode' + +describe('useUIModeStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + localStorage.clear() + }) + + it('defaults to gamer mode when no stored value', () => { + const store = useUIModeStore() + expect(store.mode).toBe('gamer') + expect(store.isGamer).toBe(true) + expect(store.isEasy).toBe(false) + expect(store.isChat).toBe(false) + }) + + it('loads stored mode from localStorage', () => { + localStorage.setItem('archipelago-ui-mode', 'easy') + const store = useUIModeStore() + expect(store.mode).toBe('easy') + expect(store.isEasy).toBe(true) + }) + + it('ignores invalid localStorage values', () => { + localStorage.setItem('archipelago-ui-mode', 'invalid-mode') + const store = useUIModeStore() + expect(store.mode).toBe('gamer') + }) + + it('setMode updates mode and persists', () => { + const store = useUIModeStore() + store.setMode('easy') + expect(store.mode).toBe('easy') + expect(store.isEasy).toBe(true) + expect(localStorage.getItem('archipelago-ui-mode')).toBe('easy') + }) + + it('setMode to chat mode', () => { + const store = useUIModeStore() + store.setMode('chat') + expect(store.mode).toBe('chat') + expect(store.isChat).toBe(true) + expect(store.isGamer).toBe(false) + }) + + it('cycleMode cycles between easy and gamer', () => { + const store = useUIModeStore() + // Start at gamer + expect(store.mode).toBe('gamer') + + // Cycle to easy + const next1 = store.cycleMode() + expect(next1).toBe('easy') + expect(store.mode).toBe('easy') + + // Cycle back to gamer (wraps after easy since order is [easy, gamer]) + const next2 = store.cycleMode() + expect(next2).toBe('gamer') + expect(store.mode).toBe('gamer') + }) + + it('cycleMode from chat wraps to easy', () => { + const store = useUIModeStore() + store.setMode('chat') + const next = store.cycleMode() + // chat is not in the order array, so idx=-1, next = order[0] = easy + expect(next).toBe('easy') + }) + + it('syncFromBackend updates mode from backend', () => { + const store = useUIModeStore() + store.syncFromBackend('easy') + expect(store.mode).toBe('easy') + expect(localStorage.getItem('archipelago-ui-mode')).toBe('easy') + }) + + it('syncFromBackend ignores invalid modes', () => { + const store = useUIModeStore() + store.syncFromBackend('invalid' as 'gamer') + expect(store.mode).toBe('gamer') // unchanged + }) + + it('syncFromBackend ignores undefined', () => { + const store = useUIModeStore() + store.syncFromBackend(undefined) + expect(store.mode).toBe('gamer') // unchanged + }) +}) diff --git a/neode-ui/src/stores/__tests__/web5Badge.test.ts b/neode-ui/src/stores/__tests__/web5Badge.test.ts new file mode 100644 index 00000000..071b78cf --- /dev/null +++ b/neode-ui/src/stores/__tests__/web5Badge.test.ts @@ -0,0 +1,78 @@ +import { describe, it, expect, beforeEach, vi } from 'vitest' +import { setActivePinia, createPinia } from 'pinia' +import { useWeb5BadgeStore } from '../web5Badge' + +// Mock rpcClient +vi.mock('@/api/rpc-client', () => ({ + rpcClient: { + call: vi.fn(), + }, +})) + +import { rpcClient } from '@/api/rpc-client' + +describe('useWeb5BadgeStore', () => { + beforeEach(() => { + setActivePinia(createPinia()) + vi.clearAllMocks() + }) + + it('starts with zero pending requests', () => { + const store = useWeb5BadgeStore() + expect(store.pendingRequestCount).toBe(0) + }) + + it('refresh updates count from API', async () => { + vi.mocked(rpcClient.call).mockResolvedValueOnce({ + requests: [{ id: '1' }, { id: '2' }, { id: '3' }], + }) + + const store = useWeb5BadgeStore() + await store.refresh() + + expect(store.pendingRequestCount).toBe(3) + expect(rpcClient.call).toHaveBeenCalledWith({ method: 'network.list-requests' }) + }) + + it('refresh handles empty requests', async () => { + vi.mocked(rpcClient.call).mockResolvedValueOnce({ requests: [] }) + + const store = useWeb5BadgeStore() + await store.refresh() + + expect(store.pendingRequestCount).toBe(0) + }) + + it('refresh handles null requests gracefully', async () => { + vi.mocked(rpcClient.call).mockResolvedValueOnce({ requests: null }) + + const store = useWeb5BadgeStore() + await store.refresh() + + expect(store.pendingRequestCount).toBe(0) + }) + + it('refresh handles API error gracefully', async () => { + vi.mocked(rpcClient.call).mockRejectedValueOnce(new Error('Network error')) + + const store = useWeb5BadgeStore() + store.pendingRequestCount = 5 // pre-existing value + await store.refresh() + + // Should not throw, count stays at pre-existing value (error swallowed) + expect(store.pendingRequestCount).toBe(5) + }) + + it('refresh updates count on subsequent calls', async () => { + vi.mocked(rpcClient.call) + .mockResolvedValueOnce({ requests: [{ id: '1' }] }) + .mockResolvedValueOnce({ requests: [{ id: '1' }, { id: '2' }] }) + + const store = useWeb5BadgeStore() + await store.refresh() + expect(store.pendingRequestCount).toBe(1) + + await store.refresh() + expect(store.pendingRequestCount).toBe(2) + }) +}) diff --git a/neode-ui/src/stores/app.ts b/neode-ui/src/stores/app.ts index 881cfb49..ec7f501e 100644 --- a/neode-ui/src/stores/app.ts +++ b/neode-ui/src/stores/app.ts @@ -47,7 +47,7 @@ export const useAppStore = defineStore('app', () => { // Connect WebSocket in background - don't block login flow connectWebSocket().catch((err) => { - console.warn('[Store] WebSocket connection failed after login, will retry:', err) + if (import.meta.env.DEV) console.warn('[Store] WebSocket connection failed after login, will retry:', err) }) return {} } catch (err) { @@ -64,7 +64,7 @@ export const useAppStore = defineStore('app', () => { localStorage.setItem('neode-auth', 'true') await initializeData() connectWebSocket().catch((err) => { - console.warn('[Store] WebSocket connection failed after TOTP login, will retry:', err) + if (import.meta.env.DEV) console.warn('[Store] WebSocket connection failed after TOTP login, will retry:', err) }) } @@ -72,7 +72,7 @@ export const useAppStore = defineStore('app', () => { try { await rpcClient.logout() } catch (err) { - console.error('Logout error:', err) + if (import.meta.env.DEV) console.error('Logout error:', err) } finally { isAuthenticated.value = false sessionValidated = false @@ -87,7 +87,7 @@ export const useAppStore = defineStore('app', () => { async function connectWebSocket(): Promise { try { - console.log('[Store] Connecting WebSocket...') + if (import.meta.env.DEV) console.log('[Store] Connecting WebSocket...') isReconnecting.value = true // Don't create multiple subscriptions - check if already subscribed @@ -96,20 +96,16 @@ export const useAppStore = defineStore('app', () => { isWsSubscribed = true // Listen for connection state changes - wsClient.onConnectionStateChange((connected) => { - console.log('[Store] WebSocket connection state changed:', connected) - isConnected.value = connected - if (!connected) { - isReconnecting.value = true - } else { - isReconnecting.value = false - } + wsClient.onConnectionStateChange((state) => { + if (import.meta.env.DEV) console.log('[Store] WebSocket connection state changed:', state) + isConnected.value = state === 'connected' + isReconnecting.value = state === 'connecting' }) wsClient.subscribe((update: { type?: string; data?: DataModel; rev?: number; patch?: import('@/types/api').PatchOperation[] }) => { // Handle mock backend format: {type: 'initial', data: {...}} if (update?.type === 'initial' && update?.data) { - console.log('[Store] Received initial data from mock backend') + if (import.meta.env.DEV) console.log('[Store] Received initial data from mock backend') data.value = update.data isConnected.value = true isReconnecting.value = false @@ -123,7 +119,7 @@ export const useAppStore = defineStore('app', () => { // Handle patch updates (both backends) else if (data.value && update?.patch) { try { - console.log('[Store] Applying patch at revision', update.rev || 'unknown') + if (import.meta.env.DEV) console.log('[Store] Applying patch at revision', update.rev || 'unknown') data.value = applyDataPatch(data.value, update.patch) // Mark as connected once we receive any valid patch if (!isConnected.value) { @@ -131,7 +127,7 @@ export const useAppStore = defineStore('app', () => { isReconnecting.value = false } } catch (err) { - console.error('[Store] Failed to apply WebSocket patch:', err) + if (import.meta.env.DEV) console.error('[Store] Failed to apply WebSocket patch:', err) } } }) @@ -140,14 +136,14 @@ export const useAppStore = defineStore('app', () => { // Now connect (or reconnect if already connected) // Only attempt to connect if not already connected if (wsClient.isConnected()) { - console.log('[Store] WebSocket already connected') + if (import.meta.env.DEV) console.log('[Store] WebSocket already connected') isConnected.value = true isReconnecting.value = false return } await wsClient.connect() - console.log('[Store] WebSocket connected') + if (import.meta.env.DEV) console.log('[Store] WebSocket connected') // Connection state will be updated via the callback if (wsClient.isConnected()) { @@ -156,7 +152,7 @@ export const useAppStore = defineStore('app', () => { } } catch (err) { - console.error('[Store] WebSocket connection failed:', err) + if (import.meta.env.DEV) console.error('[Store] WebSocket connection failed:', err) // Don't mark as disconnected immediately - let reconnection logic handle it // The WebSocket client will retry automatically isReconnecting.value = true @@ -215,13 +211,13 @@ export const useAppStore = defineStore('app', () => { await initializeData() connectWebSocket().catch((err) => { - console.warn('[Store] WebSocket reconnection failed, will retry:', err) + if (import.meta.env.DEV) console.warn('[Store] WebSocket reconnection failed, will retry:', err) isReconnecting.value = true }) return true } catch (err) { - console.error('[Store] Session check failed:', err) + if (import.meta.env.DEV) console.error('[Store] Session check failed:', err) localStorage.removeItem('neode-auth') isAuthenticated.value = false sessionValidated = false diff --git a/neode-ui/src/stores/appLauncher.ts b/neode-ui/src/stores/appLauncher.ts index 0e001470..79ea3feb 100644 --- a/neode-ui/src/stores/appLauncher.ts +++ b/neode-ui/src/stores/appLauncher.ts @@ -5,11 +5,25 @@ import { ref } from 'vue' * Verified by checking response headers from each app container. * These always open in a new tab. Other apps load in the iframe overlay. */ +/** Hostnames of external sites that block iframes via X-Frame-Options or CSP. + * Sites listed here that also appear in EXTERNAL_PROXY will be proxied (not blocked). + */ +const IFRAME_BLOCKED_HOSTS: string[] = [] + +/** External sites proxied through nginx to strip X-Frame-Options for iframe embedding */ +const EXTERNAL_PROXY: Record = { + 'botfights.net': '/ext/botfights/', + '484.kitchen': '/ext/484-kitchen/', + 'present.l484.com': '/ext/arch-presentation/', +} + function mustOpenInNewTab(url: string): boolean { try { const u = new URL(url) - // External sites — third-party cookie/iframe restrictions - if (u.hostname.includes('indeehub')) return true + // External sites that block iframes + if (IFRAME_BLOCKED_HOSTS.some(h => u.hostname === h || u.hostname.endsWith(`.${h}`))) { + return true + } // Local apps with X-Frame-Options or CSP frame-ancestors blocking iframes if ( u.port === '23000' || // BTCPay — X-Frame-Options: DENY @@ -67,6 +81,13 @@ function toEmbeddableUrl(url: string): string { try { const u = new URL(url) const origin = window.location.origin + + // External sites proxied through nginx to strip X-Frame-Options + const extProxy = EXTERNAL_PROXY[u.hostname] + if (extProxy) { + return `${origin}${extProxy}` + } + const proxyPath = PORT_TO_PROXY[u.port] const sameHost = u.hostname === window.location.hostname const needsProxy = window.location.protocol === 'https:' && u.protocol === 'http:' diff --git a/neode-ui/src/stores/cloud.ts b/neode-ui/src/stores/cloud.ts index 17ece69b..a2567778 100644 --- a/neode-ui/src/stores/cloud.ts +++ b/neode-ui/src/stores/cloud.ts @@ -95,6 +95,14 @@ export const useCloudStore = defineStore('cloud', () => { return fileBrowserClient.downloadUrl(path) } + async function fetchBlobUrl(path: string): Promise { + return fileBrowserClient.fetchBlobUrl(path) + } + + async function downloadFile(path: string): Promise { + return fileBrowserClient.downloadFile(path) + } + function reset(): void { currentPath.value = '/' items.value = [] @@ -116,6 +124,8 @@ export const useCloudStore = defineStore('cloud', () => { uploadFile, deleteItem, downloadUrl, + fetchBlobUrl, + downloadFile, reset, } }) diff --git a/neode-ui/src/stores/container.ts b/neode-ui/src/stores/container.ts index bd4bfd25..617f381b 100644 --- a/neode-ui/src/stores/container.ts +++ b/neode-ui/src/stores/container.ts @@ -165,7 +165,7 @@ export const useContainerStore = defineStore('container', () => { containers.value = await containerClient.listContainers() } catch (e) { error.value = e instanceof Error ? e.message : 'Failed to fetch containers' - console.error('Failed to fetch containers:', e) + if (import.meta.env.DEV) console.error('Failed to fetch containers:', e) } finally { loading.value = false } @@ -175,7 +175,7 @@ export const useContainerStore = defineStore('container', () => { try { healthStatus.value = await containerClient.getHealthStatus() } catch (e) { - console.error('Failed to fetch health status:', e) + if (import.meta.env.DEV) console.error('Failed to fetch health status:', e) } } diff --git a/neode-ui/src/types/api.ts b/neode-ui/src/types/api.ts index 8b9de08b..8e86250c 100644 --- a/neode-ui/src/types/api.ts +++ b/neode-ui/src/types/api.ts @@ -4,9 +4,19 @@ export interface DataModel { 'server-info': ServerInfo 'package-data': { [id: string]: PackageDataEntry } 'peer-health'?: { [onion: string]: boolean } + notifications?: AppNotification[] ui: UIData } +export interface AppNotification { + id: string + level: 'info' | 'warning' | 'error' + title: string + message: string + timestamp: string + app_id?: string +} + export interface ServerInfo { id: string version: string diff --git a/neode-ui/src/utils/__tests__/githubAppInfo.test.ts b/neode-ui/src/utils/__tests__/githubAppInfo.test.ts new file mode 100644 index 00000000..72330447 --- /dev/null +++ b/neode-ui/src/utils/__tests__/githubAppInfo.test.ts @@ -0,0 +1,140 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { fetchGitHubAppInfo, fetchMultipleAppInfo } from '../githubAppInfo' + +const mockFetch = vi.fn() +vi.stubGlobal('fetch', mockFetch) + +describe('fetchGitHubAppInfo', () => { + beforeEach(() => { + vi.clearAllMocks() + }) + + it('returns empty object for invalid repo URL', async () => { + const result = await fetchGitHubAppInfo('not-a-url', 'test') + expect(result).toEqual({}) + expect(mockFetch).not.toHaveBeenCalled() + }) + + it('returns empty object when repo API returns non-OK', async () => { + // Start9 repo check + mockFetch.mockResolvedValueOnce({ ok: false }) + // Original repo fetch fails + mockFetch.mockResolvedValueOnce({ ok: false, status: 404 }) + + const result = await fetchGitHubAppInfo('https://github.com/owner/repo', 'app') + expect(result).toEqual({}) + }) + + it('fetches repo info successfully', async () => { + // Start9 wrapper check — not found + mockFetch.mockResolvedValueOnce({ ok: false }) + // Repo API call + mockFetch.mockResolvedValueOnce({ + ok: true, + json: () => Promise.resolve({ + description: 'A Bitcoin node', + homepage: 'https://bitcoin.org', + html_url: 'https://github.com/owner/repo', + }), + }) + // README fetch + mockFetch.mockResolvedValueOnce({ + ok: true, + json: () => Promise.resolve({ content: btoa('# README') }), + }) + // Icon path checks — all fail + for (let i = 0; i < 6; i++) { + mockFetch.mockResolvedValueOnce({ ok: false }) + } + // Releases check — no icon + mockFetch.mockResolvedValueOnce({ ok: false }) + // Raw icon URL HEAD checks — all fail + for (let i = 0; i < 6; i++) { + mockFetch.mockResolvedValueOnce({ ok: false }) + } + + const result = await fetchGitHubAppInfo('https://github.com/owner/repo', 'app') + expect(result.description).toBe('A Bitcoin node') + expect(result.readme).toBe('# README') + expect(result.homepage).toBe('https://bitcoin.org') + }) + + it('finds icon from repository contents', async () => { + // Start9 wrapper — not found + mockFetch.mockResolvedValueOnce({ ok: false }) + // Repo API + mockFetch.mockResolvedValueOnce({ + ok: true, + json: () => Promise.resolve({ description: '', html_url: 'https://github.com/o/r' }), + }) + // README + mockFetch.mockResolvedValueOnce({ ok: false }) + // First icon path (icon.png) — found! + mockFetch.mockResolvedValueOnce({ + ok: true, + json: () => Promise.resolve({ download_url: 'https://raw.github.com/o/r/main/icon.png' }), + }) + + const result = await fetchGitHubAppInfo('https://github.com/o/r', 'test') + expect(result.icon).toBe('https://raw.github.com/o/r/main/icon.png') + }) + + it('tries Start9Labs wrapper repo first', async () => { + // Start9 wrapper check — found! + mockFetch.mockResolvedValueOnce({ ok: true }) + // Now fetches Start9Labs/app-startos repo + mockFetch.mockResolvedValueOnce({ + ok: true, + json: () => Promise.resolve({ + description: 'Start9 wrapper', + html_url: 'https://github.com/Start9Labs/bitcoin-startos', + }), + }) + // README + mockFetch.mockResolvedValueOnce({ ok: false }) + // Icon paths — all fail + for (let i = 0; i < 6; i++) { + mockFetch.mockResolvedValueOnce({ ok: false }) + } + // Releases + mockFetch.mockResolvedValueOnce({ ok: false }) + // Raw URLs + for (let i = 0; i < 6; i++) { + mockFetch.mockResolvedValueOnce({ ok: false }) + } + + const result = await fetchGitHubAppInfo('https://github.com/bitcoin/bitcoin', 'bitcoin') + expect(result.description).toBe('Start9 wrapper') + }) + + it('handles fetch errors gracefully', async () => { + mockFetch.mockRejectedValue(new Error('Network error')) + const result = await fetchGitHubAppInfo('https://github.com/owner/repo', 'app') + expect(result).toEqual({}) + }) +}) + +describe('fetchMultipleAppInfo', () => { + beforeEach(() => { + vi.clearAllMocks() + vi.useFakeTimers() + }) + + it('returns empty record for empty input', async () => { + const result = await fetchMultipleAppInfo([]) + expect(result).toEqual({}) + }) + + it('fetches info for multiple apps', async () => { + // Each app triggers multiple fetch calls, but they all return non-matching URLs + mockFetch.mockResolvedValue({ ok: false }) + + const apps = [ + { id: 'app1', 'wrapper-repo': 'not-a-github-url' }, + { id: 'app2', 'wrapper-repo': 'also-invalid' }, + ] + const result = await fetchMultipleAppInfo(apps) + expect(result.app1).toEqual({}) + expect(result.app2).toEqual({}) + }) +}) diff --git a/neode-ui/src/utils/githubAppInfo.ts b/neode-ui/src/utils/githubAppInfo.ts index 639ee0f4..4ba72f27 100644 --- a/neode-ui/src/utils/githubAppInfo.ts +++ b/neode-ui/src/utils/githubAppInfo.ts @@ -18,7 +18,7 @@ export async function fetchGitHubAppInfo(repoUrl: string, appId: string): Promis // Extract owner and repo from URL const match = repoUrl.match(/github\.com\/([^\/]+)\/([^\/]+)/) if (!match) { - console.warn(`[GitHub] Invalid repo URL: ${repoUrl}`) + if (import.meta.env.DEV) console.warn(`[GitHub] Invalid repo URL: ${repoUrl}`) return {} } @@ -49,7 +49,7 @@ export async function fetchGitHubAppInfo(repoUrl: string, appId: string): Promis const repoResponse = await fetch(repoApiUrl) if (!repoResponse.ok) { - console.warn(`[GitHub] Failed to fetch repo ${targetOwner}/${targetRepo}: ${repoResponse.status}`) + if (import.meta.env.DEV) console.warn(`[GitHub] Failed to fetch repo ${targetOwner}/${targetRepo}: ${repoResponse.status}`) return {} } @@ -64,7 +64,7 @@ export async function fetchGitHubAppInfo(repoUrl: string, appId: string): Promis readme = atob(readmeData.content) // Base64 decode } } catch (e) { - console.warn(`[GitHub] Failed to fetch README for ${targetOwner}/${targetRepo}`) + if (import.meta.env.DEV) console.warn(`[GitHub] Failed to fetch README for ${targetOwner}/${targetRepo}`) } // Try to find icon in repository @@ -144,7 +144,7 @@ export async function fetchGitHubAppInfo(repoUrl: string, appId: string): Promis homepage: repoData.homepage || repoData.html_url } } catch (error) { - console.error(`[GitHub] Error fetching app info for ${repoUrl}:`, error) + if (import.meta.env.DEV) console.error(`[GitHub] Error fetching app info for ${repoUrl}:`, error) return {} } } diff --git a/neode-ui/src/views/AppDetails.vue b/neode-ui/src/views/AppDetails.vue index b7686b2c..c7bae10e 100644 --- a/neode-ui/src/views/AppDetails.vue +++ b/neode-ui/src/views/AppDetails.vue @@ -784,7 +784,14 @@ function launchApp() { 'uptime-kuma': { dev: 'http://localhost:3001', prod: 'http://localhost:3001' }, 'tailscale': { dev: 'http://localhost:8240', prod: 'http://localhost:8240' }, 'lnd': { dev: 'http://localhost:8081', prod: 'http://localhost:8081' }, - 'bitcoin-knots': { dev: 'http://localhost:8334', prod: 'http://localhost:8334' } + 'bitcoin-knots': { dev: 'http://localhost:8334', prod: 'http://localhost:8334' }, + 'botfights': { dev: 'https://botfights.net', prod: 'https://botfights.net' }, + 'nwnn': { dev: 'https://nwnn.l484.com', prod: 'https://nwnn.l484.com' }, + '484-kitchen': { dev: 'https://484.kitchen', prod: 'https://484.kitchen' }, + 'call-the-operator': { dev: 'https://cta.tx1138.com', prod: 'https://cta.tx1138.com' }, + 'arch-presentation': { dev: 'https://present.l484.com', prod: 'https://present.l484.com' }, + 'syntropy-institute': { dev: 'https://syntropy.institute', prod: 'https://syntropy.institute' }, + 't-zero': { dev: 'https://teeminuszero.net', prod: 'https://teeminuszero.net' } } if (appUrls[id]) { diff --git a/neode-ui/src/views/Apps.vue b/neode-ui/src/views/Apps.vue index 177ff631..4fd2f213 100644 --- a/neode-ui/src/views/Apps.vue +++ b/neode-ui/src/views/Apps.vue @@ -69,8 +69,9 @@ @click="goToApp(id as string)" @keydown.enter="goToApp(id as string)" > - + - + -
+
@@ -78,13 +78,35 @@ const aiuiFrame = ref(null) const aiuiConnected = ref(false) let broker: ContextBroker | null = null +const aiuiAvailable = ref(null) // null = checking, true/false = result + const aiuiUrl = computed(() => { const envUrl = import.meta.env.VITE_AIUI_URL if (envUrl) return `${envUrl}?embedded=true` - if (import.meta.env.PROD) return '/aiui/?embedded=true' + // In production, only return the URL if we've confirmed AIUI files exist + if (import.meta.env.PROD && aiuiAvailable.value === true) return '/aiui/?embedded=true' return '' }) +/** Check if AIUI is actually deployed by fetching its index.html */ +async function checkAiuiAvailable() { + if (import.meta.env.VITE_AIUI_URL) { + aiuiAvailable.value = true + return + } + if (!import.meta.env.PROD) { + aiuiAvailable.value = false + return + } + try { + const res = await fetch('/aiui/', { method: 'HEAD' }) + // If we get HTML back (200), AIUI is deployed. If 404/403, it's not. + aiuiAvailable.value = res.ok + } catch { + aiuiAvailable.value = false + } +} + function closeChat() { if (window.history.length > 1) { router.back() @@ -106,8 +128,9 @@ function onAiuiMessage(event: MessageEvent) { } } -onMounted(() => { +onMounted(async () => { window.addEventListener('message', onAiuiMessage) + await checkAiuiAvailable() if (aiuiUrl.value) { broker = new ContextBroker(aiuiFrame, aiuiUrl.value) broker.start() diff --git a/neode-ui/src/views/Dashboard.vue b/neode-ui/src/views/Dashboard.vue index 9d6e23a0..5dd512a7 100644 --- a/neode-ui/src/views/Dashboard.vue +++ b/neode-ui/src/views/Dashboard.vue @@ -110,7 +110,7 @@ - Chat + AIUI @@ -336,7 +336,7 @@ - Chat + AIUI
diff --git a/neode-ui/src/views/Kiosk.vue b/neode-ui/src/views/Kiosk.vue new file mode 100644 index 00000000..c0b08d60 --- /dev/null +++ b/neode-ui/src/views/Kiosk.vue @@ -0,0 +1,284 @@ + + + + + diff --git a/neode-ui/src/views/Marketplace.vue b/neode-ui/src/views/Marketplace.vue index 938a1e83..359ffd13 100644 --- a/neode-ui/src/views/Marketplace.vue +++ b/neode-ui/src/views/Marketplace.vue @@ -397,6 +397,7 @@ const categories = computed(() => [ { id: 'home', name: t('marketplace.homeCategory') }, { id: 'car', name: t('marketplace.auto') }, { id: 'networking', name: t('marketplace.networking') }, + { id: 'l484', name: 'L484' }, { id: 'other', name: t('marketplace.other') } ]) @@ -522,11 +523,14 @@ const installedPackages = computed(() => { // Function to categorize community apps based on their ID and description function categorizeCommunityApp(app: MarketplaceApp): string { + // If app already has a category set, use it + if (app.category) return app.category + const id = app.id.toLowerCase() const title = app.title?.toLowerCase() || '' const description = (typeof app.description === 'string' ? app.description : app.description?.short ?? '').toLowerCase() const combined = `${id} ${title} ${description}` - + // Money category if (id.includes('bitcoin') || id.includes('btc') || id.includes('lightning') || id.includes('lnd') || id.includes('cln') || id.includes('electr') || @@ -949,6 +953,96 @@ function getCuratedAppList() { dockerImage: 'docker.io/scsiblade/nostr-rs-relay:0.9.0', manifestUrl: undefined, repoUrl: 'https://sr.ht/~gheartsfield/nostr-rs-relay/' + }, + { + id: 'botfights', + title: 'BotFights', + version: '1.0.0', + description: 'AI bot arena — build, train, and battle autonomous agents. Compete in strategy tournaments with your own coded bots.', + icon: '/assets/img/app-icons/botfights.svg', + author: 'BotFights', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://botfights.net', + webUrl: 'https://botfights.net' + }, + { + id: 'nwnn', + title: 'Next Web News Network', + version: '1.0.0', + category: 'l484', + description: 'Decentralized news and link aggregator, synchronized from Telegram. Community-curated content on Bitcoin, sovereignty, and decentralized tech.', + icon: '/assets/img/app-icons/nwnn.png', + author: 'L484', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://nwnn.l484.com', + webUrl: 'https://nwnn.l484.com' + }, + { + id: '484-kitchen', + title: '484 Kitchen', + version: '1.0.0', + category: 'l484', + description: 'K484 application platform — an internal tool for the L484 network.', + icon: '/assets/img/app-icons/484-kitchen.png', + author: 'L484', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://484.kitchen', + webUrl: 'https://484.kitchen' + }, + { + id: 'call-the-operator', + title: 'Call the Operator', + version: '1.0.0', + category: 'l484', + description: 'Escape the Matrix — a portal for exploring decentralized alternatives and reclaiming digital sovereignty.', + icon: '/assets/img/app-icons/call-the-operator.png', + author: 'TX1138', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://cta.tx1138.com', + webUrl: 'https://cta.tx1138.com' + }, + { + id: 'arch-presentation', + title: 'Arch Presentation', + version: '1.0.0', + category: 'l484', + description: 'Archipelago: The Future of Decentralized Infrastructure — an interactive presentation about the Archipelago project vision.', + icon: '/assets/img/app-icons/arch-presentation.png', + author: 'L484', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://present.l484.com', + webUrl: 'https://present.l484.com' + }, + { + id: 'syntropy-institute', + title: 'Syntropy Institute', + version: '1.0.0', + category: 'l484', + description: 'Medicine Reimagined — Manual Kinetics, Syntropy Frequency analysis-therapy, digital homeopathy, and concierge protocols.', + icon: '/assets/img/app-icons/syntropy-institute.png', + author: 'Syntropy Institute', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://syntropy.institute', + webUrl: 'https://syntropy.institute' + }, + { + id: 't-zero', + title: 'T-0', + version: '1.0.0', + category: 'l484', + description: 'Documentary series exploring decentralization, Bitcoin, and the mavericks building the ungovernable future. Conversations with the builders, powered by Nostr.', + icon: '/assets/img/app-icons/t-zero.png', + author: 'T-0', + dockerImage: '', + manifestUrl: undefined, + repoUrl: 'https://teeminuszero.net', + webUrl: 'https://teeminuszero.net' } ] } diff --git a/neode-ui/src/views/MarketplaceAppDetails.vue b/neode-ui/src/views/MarketplaceAppDetails.vue index 4478bbf9..b6edd25b 100644 --- a/neode-ui/src/views/MarketplaceAppDetails.vue +++ b/neode-ui/src/views/MarketplaceAppDetails.vue @@ -374,6 +374,7 @@ import { useAppStore } from '../stores/app' import { rpcClient } from '../api/rpc-client' import { useMarketplaceApp, type MarketplaceAppInfo } from '../composables/useMarketplaceApp' import { useMobileBackButton } from '../composables/useMobileBackButton' +import { useAppLauncherStore } from '../stores/appLauncher' const { t } = useI18n() const { bottomPosition } = useMobileBackButton() @@ -391,8 +392,14 @@ const loading = ref(true) const appId = computed(() => route.params.id as string) +// Web-only apps (no container, just a URL) — always treated as "installed" +const isWebOnly = computed(() => { + return !!(app.value?.webUrl && !app.value?.dockerImage) +}) + // Check if app is already installed const isInstalled = computed(() => { + if (isWebOnly.value) return true return !!store.packages[appId.value] }) @@ -503,6 +510,14 @@ function goBack() { } function goToInstalledApp() { + // Web-only apps: launch directly via appLauncher + if (isWebOnly.value && app.value?.webUrl) { + useAppLauncherStore().open({ + url: app.value.webUrl, + title: app.value.title || appId.value, + }) + return + } router.push({ path: `/dashboard/apps/${appId.value}`, query: { from: 'marketplace' } diff --git a/neode-ui/src/views/__tests__/settings.test.ts b/neode-ui/src/views/__tests__/settings.test.ts new file mode 100644 index 00000000..d734ea23 --- /dev/null +++ b/neode-ui/src/views/__tests__/settings.test.ts @@ -0,0 +1,317 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { shallowMount, VueWrapper } from '@vue/test-utils' +import { createPinia, setActivePinia } from 'pinia' +import { defineComponent, h } from 'vue' + +// Mock rpc-client before importing anything that uses it +vi.mock('@/api/rpc-client', () => ({ + rpcClient: { + call: vi.fn().mockResolvedValue({ backups: [] }), + login: vi.fn(), + logout: vi.fn(), + changePassword: vi.fn(), + totpStatus: vi.fn().mockResolvedValue({ enabled: false }), + totpSetupBegin: vi.fn(), + totpSetupConfirm: vi.fn(), + totpDisable: vi.fn(), + getTorAddress: vi.fn().mockResolvedValue({ tor_address: null }), + }, +})) + +// Mock websocket module +vi.mock('@/api/websocket', () => ({ + wsClient: { + connect: vi.fn().mockResolvedValue(undefined), + disconnect: vi.fn(), + subscribe: vi.fn(), + isConnected: vi.fn().mockReturnValue(false), + onConnectionStateChange: vi.fn(), + }, + applyDataPatch: vi.fn(), +})) + +// Stub the ControllerIndicator component +vi.mock('@/components/ControllerIndicator.vue', () => ({ + default: defineComponent({ name: 'ControllerIndicator', render: () => h('div') }), +})) + +// Mock useModalKeyboard composable +vi.mock('@/composables/useModalKeyboard', () => ({ + useModalKeyboard: vi.fn(), +})) + +// Stub vue-router +const pushMock = vi.fn() +vi.mock('vue-router', () => ({ + useRouter: () => ({ + push: pushMock, + }), + RouterLink: defineComponent({ + name: 'RouterLink', + props: { to: { type: String, default: '' } }, + setup(_, { slots }) { + return () => h('a', {}, slots.default?.()) + }, + }), +})) + +// Stub global fetch for the Claude status check in onMounted +vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error('not available'))) + +import { createI18n } from 'vue-i18n' +import en from '@/locales/en.json' +import Settings from '../Settings.vue' +import { rpcClient } from '@/api/rpc-client' +import { useAppStore } from '@/stores/app' + +const i18n = createI18n({ legacy: false, locale: 'en', messages: { en } }) + +const mockedRpc = vi.mocked(rpcClient) + +function mountSettings(storeOverrides?: Partial>): VueWrapper { + const pinia = createPinia() + setActivePinia(pinia) + + const store = useAppStore() + // Set default store state for tests + store.isAuthenticated = true + store.$patch({ + data: { + 'server-info': { + id: 'test-node', + version: '0.1.0', + name: 'Test Node', + pubkey: 'test-pubkey', + 'status-info': { restarting: false, 'shutting-down': false, updated: false, 'backup-progress': null, 'update-progress': null }, + 'lan-address': '192.168.1.100', + 'tor-address': null, + unread: 0, + 'wifi-ssids': [], + 'zram-enabled': false, + }, + 'package-data': {}, + ui: { name: null, 'ack-welcome': '', marketplace: { 'selected-hosts': [], 'known-hosts': {} }, theme: 'dark' }, + }, + }) + + if (storeOverrides) { + store.$patch(storeOverrides as Record) + } + + return shallowMount(Settings, { + global: { + plugins: [pinia, i18n], + stubs: { + Teleport: true, + RouterLink: defineComponent({ + name: 'RouterLink', + props: { to: { type: String, default: '' } }, + setup(_, { slots }) { + return () => h('a', {}, slots.default?.()) + }, + }), + }, + }, + }) +} + +describe('Settings View', () => { + beforeEach(() => { + vi.clearAllMocks() + localStorage.clear() + mockedRpc.totpStatus.mockResolvedValue({ enabled: false }) + mockedRpc.call.mockResolvedValue({ backups: [] }) + mockedRpc.getTorAddress.mockResolvedValue({ tor_address: null }) + pushMock.mockResolvedValue(undefined) + }) + + it('renders without errors', () => { + const wrapper = mountSettings() + expect(wrapper.exists()).toBe(true) + }) + + it('displays the Settings heading', () => { + const wrapper = mountSettings() + const heading = wrapper.find('h1') + expect(heading.exists()).toBe(true) + expect(heading.text()).toBe('Settings') + }) + + it('displays the Account section with server name and version', () => { + const wrapper = mountSettings() + const html = wrapper.html() + + // Account section heading + const sectionHeadings = wrapper.findAll('h2') + const accountHeading = sectionHeadings.find((h) => h.text() === 'Account') + expect(accountHeading).toBeDefined() + + // Server name rendered + expect(html).toContain('Test Node') + + // Version rendered + expect(html).toContain('0.1.0') + }) + + it('displays the version from server info', () => { + const wrapper = mountSettings() + const html = wrapper.html() + expect(html).toContain('0.1.0') + expect(html).toContain('Version') + }) + + it('displays the Interface Mode section', () => { + const wrapper = mountSettings() + const sectionHeadings = wrapper.findAll('h2') + const modeHeading = sectionHeadings.find((h) => h.text() === 'Interface Mode') + expect(modeHeading).toBeDefined() + }) + + it('displays the Claude Authentication section', () => { + const wrapper = mountSettings() + const sectionHeadings = wrapper.findAll('h2') + const claudeHeading = sectionHeadings.find((h) => h.text() === 'Claude Authentication') + expect(claudeHeading).toBeDefined() + }) + + it('displays the AI Data Access section', () => { + const wrapper = mountSettings() + const sectionHeadings = wrapper.findAll('h2') + const aiHeading = sectionHeadings.find((h) => h.text() === 'AI Data Access') + expect(aiHeading).toBeDefined() + }) + + it('displays the System Updates section', () => { + const wrapper = mountSettings() + const sectionHeadings = wrapper.findAll('h2') + const updatesHeading = sectionHeadings.find((h) => h.text() === 'System Updates') + expect(updatesHeading).toBeDefined() + }) + + it('displays the Backup & Restore section', () => { + const wrapper = mountSettings() + const sectionHeadings = wrapper.findAll('h2') + const backupHeading = sectionHeadings.find((h) => h.text().includes('Backup')) + expect(backupHeading).toBeDefined() + }) + + it('displays the Network section', () => { + const wrapper = mountSettings() + const sectionHeadings = wrapper.findAll('h2') + const networkHeading = sectionHeadings.find((h) => h.text() === 'Network') + expect(networkHeading).toBeDefined() + }) + + it('displays a Logout button', () => { + const wrapper = mountSettings() + const buttons = wrapper.findAll('button') + const logoutButton = buttons.find((b) => b.text().includes('Logout')) + expect(logoutButton).toBeDefined() + expect(logoutButton!.exists()).toBe(true) + }) + + it('logout button triggers store logout and navigates to login', async () => { + const wrapper = mountSettings() + const store = useAppStore() + const logoutSpy = vi.spyOn(store, 'logout').mockResolvedValue() + + const buttons = wrapper.findAll('button') + const logoutButton = buttons.find((b) => b.text().includes('Logout')) + expect(logoutButton).toBeDefined() + + await logoutButton!.trigger('click') + // Allow async handlers to settle + await vi.dynamicImportSettled() + + expect(logoutSpy).toHaveBeenCalled() + expect(pushMock).toHaveBeenCalledWith('/login') + }) + + it('displays a Change Password button', () => { + const wrapper = mountSettings() + const buttons = wrapper.findAll('button') + const changePasswordButton = buttons.find((b) => b.text().includes('Change Password')) + expect(changePasswordButton).toBeDefined() + expect(changePasswordButton!.exists()).toBe(true) + }) + + it('displays Two-Factor Authentication section with status', () => { + const wrapper = mountSettings() + const html = wrapper.html() + expect(html).toContain('Two-Factor Authentication') + }) + + it('shows Enable 2FA button when TOTP is not enabled', () => { + const wrapper = mountSettings() + const buttons = wrapper.findAll('button') + const enable2faButton = buttons.find((b) => b.text().includes('Enable 2FA')) + expect(enable2faButton).toBeDefined() + }) + + it('displays session status as currently logged in', () => { + const wrapper = mountSettings() + expect(wrapper.html()).toContain('Currently logged in') + }) + + it('shows server name from the store', () => { + const wrapper = mountSettings() + expect(wrapper.html()).toContain('Server Name') + expect(wrapper.html()).toContain('Test Node') + }) + + it('defaults version to 0.0.0 when server info has no version', () => { + const pinia = createPinia() + setActivePinia(pinia) + const store = useAppStore() + store.$patch({ + isAuthenticated: true, + data: { + 'server-info': { + id: 'test', + version: '', + name: null, + pubkey: '', + 'status-info': { restarting: false, 'shutting-down': false, updated: false, 'backup-progress': null, 'update-progress': null }, + 'lan-address': null, + 'tor-address': null, + unread: 0, + 'wifi-ssids': [], + }, + 'package-data': {}, + ui: { name: null, 'ack-welcome': '', marketplace: { 'selected-hosts': [], 'known-hosts': {} }, theme: 'dark' }, + }, + }) + + const wrapper = shallowMount(Settings, { + global: { + plugins: [pinia, i18n], + stubs: { + Teleport: true, + RouterLink: defineComponent({ + name: 'RouterLink', + props: { to: { type: String, default: '' } }, + setup(_, { slots }) { + return () => h('a', {}, slots.default?.()) + }, + }), + }, + }, + }) + + // When version is empty string, computed returns '0.0.0' from the fallback + const html = wrapper.html() + expect(html).toContain('0.0.0') + }) + + it('calls totpStatus on mount to check 2FA state', async () => { + mountSettings() + // onMounted calls loadTotpStatus which calls rpcClient.totpStatus + expect(mockedRpc.totpStatus).toHaveBeenCalled() + }) + + it('calls backup.list on mount to load backups', async () => { + mountSettings() + // onMounted calls loadBackups which calls rpcClient.call with backup.list + expect(mockedRpc.call).toHaveBeenCalledWith({ method: 'backup.list' }) + }) +}) diff --git a/scripts/audit-secrets.sh b/scripts/audit-secrets.sh index 3fb37c1f..15a6603f 100755 --- a/scripts/audit-secrets.sh +++ b/scripts/audit-secrets.sh @@ -25,7 +25,7 @@ PATTERNS=( ) # Allowed files (config templates, docs, test fixtures) -ALLOW_PATTERNS="test|mock|example|template|CLAUDE.md|deploy-config|\.md$|node_modules|dist|target" +ALLOW_PATTERNS="test|mock|example|template|CLAUDE.md|deploy-config|\.md$|node_modules|dist|target|default\)|grep.*rpc|audit-secrets" main() { log "=== Secrets Audit ===" @@ -96,7 +96,7 @@ main() { # 5. Check for credential files in repo log "5. Checking for credential files..." local cred_files - cred_files=$(cd "$REPO_ROOT" && git ls-files '*.pem' '*.key' '*credentials*' '*macaroon*' 2>/dev/null || echo "") + cred_files=$(cd "$REPO_ROOT" && git ls-files '*.pem' '*.key' '*macaroon*' 2>/dev/null | grep -v '\.rs$' | grep -v '\.ts$' || echo "") if [ -z "$cred_files" ]; then pass "No credential files tracked in git" else diff --git a/scripts/chaos-test.sh b/scripts/chaos-test.sh new file mode 100755 index 00000000..dce0ab19 --- /dev/null +++ b/scripts/chaos-test.sh @@ -0,0 +1,339 @@ +#!/usr/bin/env bash +# chaos-test.sh — Chaos/resilience test for Archipelago server. +# +# Tests the server's ability to survive adverse conditions: +# - Process kills (verify systemd restart) +# - Container stop/start cycling +# - Concurrent RPC requests (verify no crashes) +# - High disk usage warnings +# - Network interruption recovery +# +# Usage: +# ssh archipelago@192.168.1.228 "cd ~/archy && bash scripts/chaos-test.sh" +# +# Duration: ~30 minutes by default (set CHAOS_DURATION_HOURS for longer) + +set -uo pipefail + +CHAOS_DURATION_HOURS="${CHAOS_DURATION_HOURS:-0.5}" +RPC_URL="http://localhost:5678/rpc/v1" +HEALTH_URL="http://localhost/health" +MAX_RECOVERY_WAIT=60 + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +PASS=0 +FAIL=0 +TESTS=() + +log() { echo -e "${GREEN}[CHAOS]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +fail() { echo -e "${RED}[FAIL]${NC} $*"; } + +record() { + local name="$1" result="$2" + if [ "$result" = "PASS" ]; then + PASS=$((PASS + 1)) + TESTS+=("PASS $name") + else + FAIL=$((FAIL + 1)) + TESTS+=("FAIL $name") + fi +} + +# Authenticate +COOKIE_FILE=$(mktemp) +authenticate() { + curl -s -c "$COOKIE_FILE" -X POST "$RPC_URL" \ + -H "Content-Type: application/json" \ + -d '{"method":"auth.login","params":{"password":"password123"}}' > /dev/null 2>&1 +} + +rpc() { + local method="$1" + local params="${2:-null}" + local csrf + csrf=$(grep csrf_token "$COOKIE_FILE" 2>/dev/null | awk '{print $NF}' || echo "") + curl -s -b "$COOKIE_FILE" -X POST "$RPC_URL" \ + -H "Content-Type: application/json" \ + -H "X-CSRF-Token: $csrf" \ + -d "{\"method\":\"$method\",\"params\":$params}" 2>/dev/null +} + +wait_for_health() { + local timeout="${1:-$MAX_RECOVERY_WAIT}" + local elapsed=0 + while [ "$elapsed" -lt "$timeout" ]; do + if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then + return 0 + fi + sleep 2 + elapsed=$((elapsed + 2)) + done + return 1 +} + +echo "" +echo "============================================" +echo " Archipelago Chaos Test Suite" +echo "============================================" +echo " Duration: ${CHAOS_DURATION_HOURS}h" +echo "" + +# Pre-check +if ! curl -sf "$HEALTH_URL" > /dev/null 2>&1; then + fail "Server not healthy at $HEALTH_URL — aborting" + exit 1 +fi +log "Server is healthy" +authenticate + +# ============================================================================= +# Test 1: Process Kill Recovery +# ============================================================================= +log "=== Test 1: Process Kill Recovery ===" +log "Killing archipelago process..." + +sudo systemctl kill --signal=SIGKILL archipelago 2>/dev/null || \ + sudo kill -9 $(pgrep -f "/usr/local/bin/archipelago" | head -1) 2>/dev/null + +sleep 2 + +if wait_for_health 30; then + log "Backend recovered after SIGKILL in <30s" + record "Process kill recovery" "PASS" +else + fail "Backend did not recover after SIGKILL within 30s" + record "Process kill recovery" "FAIL" + # Try to restart manually + sudo systemctl start archipelago + sleep 5 +fi + +authenticate + +# ============================================================================= +# Test 2: Graceful Restart +# ============================================================================= +log "=== Test 2: Graceful Restart ===" +log "Restarting archipelago service..." + +sudo systemctl restart archipelago +sleep 2 + +if wait_for_health 20; then + log "Backend restarted gracefully" + record "Graceful restart" "PASS" +else + fail "Backend did not come up after restart" + record "Graceful restart" "FAIL" +fi + +authenticate + +# ============================================================================= +# Test 3: Concurrent RPC Requests +# ============================================================================= +log "=== Test 3: Concurrent RPC Load (100 requests) ===" + +CONCURRENT_PASS=0 +CONCURRENT_FAIL=0 + +for i in $(seq 1 100); do + ( + result=$(curl -sf -X POST "$RPC_URL" \ + -H "Content-Type: application/json" \ + -d '{"method":"system.stats"}' 2>/dev/null) + if echo "$result" | grep -q "cpu_usage_percent"; then + echo "OK" >> /tmp/chaos-concurrent-ok + else + echo "FAIL" >> /tmp/chaos-concurrent-fail + fi + ) & +done + +wait +rm -f /tmp/chaos-concurrent-ok /tmp/chaos-concurrent-fail 2>/dev/null + +# Re-authenticate in case cookies expired during load +authenticate + +# Check server still healthy +if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then + log "Server survived 100 concurrent requests" + record "Concurrent RPC load" "PASS" +else + fail "Server crashed under concurrent load" + record "Concurrent RPC load" "FAIL" + sudo systemctl restart archipelago + sleep 5 + authenticate +fi + +# ============================================================================= +# Test 4: Container Stop/Start Cycling +# ============================================================================= +log "=== Test 4: Container Stop/Start Cycling ===" + +# Use filebrowser as test container (lightweight, quick to restart) +CONTAINER_ID="filebrowser" +if [ -n "$CONTAINER_ID" ]; then + log "Testing with container: $CONTAINER_ID" + + # Stop + rpc "package.stop" "{\"id\":\"$CONTAINER_ID\"}" > /dev/null + sleep 3 + + # Verify stopped + status=$(rpc "container-status" "{\"id\":\"$CONTAINER_ID\"}") + + # Start + rpc "package.start" "{\"id\":\"$CONTAINER_ID\"}" > /dev/null + sleep 10 + + # Verify running (check both container-status and podman directly) + status=$(rpc "container-status" "{\"id\":\"$CONTAINER_ID\"}") + podman_running=$(podman ps --filter "name=^${CONTAINER_ID}$" --format "{{.Status}}" 2>/dev/null | head -1 | grep -ci "up" || echo "0") + if echo "$status" | grep -qi "running" || [ "$podman_running" -gt 0 ]; then + log "Container $CONTAINER_ID stop/start cycle OK" + record "Container cycling" "PASS" + else + warn "Container $CONTAINER_ID may not have restarted" + record "Container cycling" "FAIL" + fi +else + warn "No running containers found, skipping container test" + TESTS+=("SKIP Container cycling (no containers)") +fi + +# ============================================================================= +# Test 5: RPC Error Handling +# ============================================================================= +log "=== Test 5: RPC Error Handling ===" + +# Invalid method +result=$(rpc "nonexistent.method") +if echo "$result" | grep -qi "error\|unknown"; then + log "Invalid method correctly returns error" + err_pass=true +else + fail "Invalid method did not return error" + err_pass=false +fi + +# Malformed JSON — server should not crash (any response is acceptable) +http_code=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$RPC_URL" -H "Content-Type: application/json" -d '{broken}' 2>/dev/null || echo "000") +if [ "$http_code" != "000" ]; then + log "Malformed JSON handled without crash (HTTP $http_code)" +else + # Server may have been restarting from previous test, wait and retry + sleep 3 + http_code=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$RPC_URL" -H "Content-Type: application/json" -d '{broken}' 2>/dev/null | tail -c 3 || echo "000") + if [ -n "$http_code" ] && [ "$http_code" != "000" ]; then + log "Malformed JSON handled without crash (HTTP $http_code, retry)" + else + warn "Server unreachable for malformed JSON test" + err_pass=false + fi +fi + +# Missing params +result=$(rpc "backup.create") +if echo "$result" | grep -qi "error\|missing"; then + log "Missing params correctly returns error" +else + err_pass=false +fi + +if [ "$err_pass" = true ]; then + record "RPC error handling" "PASS" +else + record "RPC error handling" "FAIL" +fi + +# ============================================================================= +# Test 6: Rapid Reconnection +# ============================================================================= +log "=== Test 6: Rapid Restart Cycling ===" + +for i in 1 2 3; do + sudo systemctl restart archipelago + sleep 3 + if ! wait_for_health 15; then + fail "Failed to recover on cycle $i" + record "Rapid restart cycling" "FAIL" + break + fi +done + +if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then + log "Server survived 3 rapid restarts" + record "Rapid restart cycling" "PASS" +fi + +authenticate + +# ============================================================================= +# Test 7: Data Integrity After Chaos +# ============================================================================= +log "=== Test 7: Data Integrity Check ===" + +# Check system stats still work +stats=$(rpc "system.stats") +if echo "$stats" | grep -q "cpu_usage_percent"; then + log "System stats OK" + data_ok=true +else + fail "System stats broken" + data_ok=false +fi + +# Check update status +update=$(rpc "update.status") +if echo "$update" | grep -q "current_version"; then + log "Update status OK" +else + data_ok=false +fi + +# Check backup list +backups=$(rpc "backup.list") +if echo "$backups" | grep -q "backups"; then + log "Backup list OK" +else + data_ok=false +fi + +if [ "$data_ok" = true ]; then + record "Data integrity" "PASS" +else + record "Data integrity" "FAIL" +fi + +# ============================================================================= +# Summary +# ============================================================================= +rm -f "$COOKIE_FILE" + +echo "" +echo "============================================" +echo " Chaos Test Results" +echo "============================================" +for r in "${TESTS[@]}"; do + case "$r" in + PASS*) echo -e " ${GREEN}$r${NC}" ;; + FAIL*) echo -e " ${RED}$r${NC}" ;; + SKIP*) echo -e " ${YELLOW}$r${NC}" ;; + esac +done +echo "" +echo " Passed: $PASS Failed: $FAIL" +echo "============================================" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi diff --git a/scripts/create-release-manifest.sh b/scripts/create-release-manifest.sh new file mode 100755 index 00000000..d3459189 --- /dev/null +++ b/scripts/create-release-manifest.sh @@ -0,0 +1,201 @@ +#!/usr/bin/env bash +# create-release-manifest.sh — Build a release manifest for the Archipelago update system. +# +# Generates a JSON manifest with version info, changelog, and SHA256 hashes for +# each component, matching the format expected by core/archipelago/src/update.rs. +# +# Usage: +# ./scripts/create-release-manifest.sh --version 0.2.0 --date 2026-04-01 +# +# The script reads built artifacts from the build output directories and produces +# a manifest.json file suitable for hosting at the UPDATE_MANIFEST_URL. + +set -euo pipefail + +# Defaults +VERSION="" +RELEASE_DATE="" +OUTPUT_FILE="manifest.json" +BACKEND_BINARY="" +FRONTEND_ARCHIVE="" +BASE_URL="https://github.com/archipelago-os/releases/releases/download" + +usage() { + echo "Usage: $0 --version VERSION [--date DATE] [--output FILE]" + echo "" + echo "Options:" + echo " --version VERSION Release version (e.g., 0.2.0) [required]" + echo " --date DATE Release date (YYYY-MM-DD) [default: today]" + echo " --output FILE Output manifest path [default: manifest.json]" + echo " --backend PATH Path to backend binary [default: auto-detect]" + echo " --frontend PATH Path to frontend archive [default: auto-detect]" + echo " --base-url URL Base download URL [default: GitHub releases]" + exit 1 +} + +# Parse arguments +while [[ $# -gt 0 ]]; do + case "$1" in + --version) VERSION="$2"; shift 2 ;; + --date) RELEASE_DATE="$2"; shift 2 ;; + --output) OUTPUT_FILE="$2"; shift 2 ;; + --backend) BACKEND_BINARY="$2"; shift 2 ;; + --frontend) FRONTEND_ARCHIVE="$2"; shift 2 ;; + --base-url) BASE_URL="$2"; shift 2 ;; + -h|--help) usage ;; + *) echo "Unknown option: $1"; usage ;; + esac +done + +if [ -z "$VERSION" ]; then + echo "Error: --version is required" + usage +fi + +if [ -z "$RELEASE_DATE" ]; then + RELEASE_DATE=$(date +%Y-%m-%d) +fi + +# Find project root +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + +# Auto-detect backend binary +if [ -z "$BACKEND_BINARY" ]; then + BACKEND_BINARY="$PROJECT_ROOT/core/target/release/archipelago" +fi + +# Auto-detect frontend archive +if [ -z "$FRONTEND_ARCHIVE" ]; then + FRONTEND_DIST="$PROJECT_ROOT/web/dist/neode-ui" + if [ -d "$FRONTEND_DIST" ]; then + FRONTEND_ARCHIVE="/tmp/archipelago-frontend-${VERSION}.tar.gz" + echo "Creating frontend archive from $FRONTEND_DIST..." + tar -czf "$FRONTEND_ARCHIVE" -C "$PROJECT_ROOT/web/dist" neode-ui + fi +fi + +# Compute SHA256 hash +sha256_of() { + if command -v sha256sum &>/dev/null; then + sha256sum "$1" | awk '{print $1}' + else + shasum -a 256 "$1" | awk '{print $1}' + fi +} + +# File size in bytes +size_of() { + if [[ "$(uname)" == "Darwin" ]]; then + stat -f%z "$1" + else + stat -c%s "$1" + fi +} + +# Get current version from Cargo.toml +CURRENT_VERSION=$(grep '^version' "$PROJECT_ROOT/core/archipelago/Cargo.toml" | head -1 | sed 's/.*"\(.*\)".*/\1/') + +echo "Building release manifest v${VERSION}" +echo " Current version: ${CURRENT_VERSION}" +echo " Release date: ${RELEASE_DATE}" +echo " Output: ${OUTPUT_FILE}" + +# Build components array +COMPONENTS="[]" + +if [ -f "$BACKEND_BINARY" ]; then + HASH=$(sha256_of "$BACKEND_BINARY") + SIZE=$(size_of "$BACKEND_BINARY") + echo " Backend binary: ${BACKEND_BINARY} (${SIZE} bytes, sha256: ${HASH})" + COMPONENTS=$(echo "$COMPONENTS" | python3 -c " +import sys, json +c = json.load(sys.stdin) +c.append({ + 'name': 'archipelago', + 'current_version': '$CURRENT_VERSION', + 'new_version': '$VERSION', + 'download_url': '$BASE_URL/v$VERSION/archipelago', + 'sha256': '$HASH', + 'size_bytes': $SIZE +}) +print(json.dumps(c)) +") +else + echo " Warning: Backend binary not found at $BACKEND_BINARY" +fi + +if [ -n "$FRONTEND_ARCHIVE" ] && [ -f "$FRONTEND_ARCHIVE" ]; then + HASH=$(sha256_of "$FRONTEND_ARCHIVE") + SIZE=$(size_of "$FRONTEND_ARCHIVE") + ARCHIVE_NAME=$(basename "$FRONTEND_ARCHIVE") + echo " Frontend archive: ${FRONTEND_ARCHIVE} (${SIZE} bytes, sha256: ${HASH})" + COMPONENTS=$(echo "$COMPONENTS" | python3 -c " +import sys, json +c = json.load(sys.stdin) +c.append({ + 'name': '$ARCHIVE_NAME', + 'current_version': '$CURRENT_VERSION', + 'new_version': '$VERSION', + 'download_url': '$BASE_URL/v$VERSION/$ARCHIVE_NAME', + 'sha256': '$HASH', + 'size_bytes': $SIZE +}) +print(json.dumps(c)) +") +else + echo " Warning: Frontend archive not found" +fi + +# Read changelog from CHANGELOG.md if available +CHANGELOG="[]" +CHANGELOG_FILE="$PROJECT_ROOT/CHANGELOG.md" +if [ -f "$CHANGELOG_FILE" ]; then + # Extract entries for this version (lines between ## vVERSION and next ##) + ENTRIES=$(python3 -c " +import re, sys +content = open('$CHANGELOG_FILE').read() +pattern = r'## .*?${VERSION}.*?\n(.*?)(?=\n## |\Z)' +m = re.search(pattern, content, re.DOTALL) +if m: + for line in m.group(1).strip().split('\n')[:10]: + line = line.strip() + if line: + print(line) +" 2>/dev/null || echo "") + if [ -n "$ENTRIES" ]; then + CHANGELOG=$(echo "$ENTRIES" | python3 -c " +import sys, json +lines = [l.strip().lstrip('- ') for l in sys.stdin if l.strip()] +print(json.dumps(lines)) +") + fi +fi + +# If no changelog entries found, add a default +if [ "$CHANGELOG" = "[]" ]; then + CHANGELOG="[\"Update to version ${VERSION}\"]" +fi + +# Generate manifest +python3 -c " +import json +manifest = { + 'version': '$VERSION', + 'release_date': '$RELEASE_DATE', + 'changelog': $CHANGELOG, + 'components': $COMPONENTS +} +print(json.dumps(manifest, indent=2)) +" > "$OUTPUT_FILE" + +echo "" +echo "Manifest written to: $OUTPUT_FILE" +echo "" +cat "$OUTPUT_FILE" +echo "" +echo "Next steps:" +echo " 1. Review the manifest above" +echo " 2. Upload artifacts to: $BASE_URL/v$VERSION/" +echo " 3. Upload manifest.json to the releases repo main branch" +echo " 4. Tag the release: git tag v$VERSION && git push --tags" diff --git a/scripts/deploy-to-target.sh b/scripts/deploy-to-target.sh index 82de7519..f1b88a2c 100755 --- a/scripts/deploy-to-target.sh +++ b/scripts/deploy-to-target.sh @@ -63,6 +63,15 @@ if ! ssh $SSH_OPTS -o ConnectTimeout=5 "$TARGET_HOST" "echo ok" >/dev/null 2>&1; exit 1 fi echo " Connected." + +# Pre-deploy health check (informational — warns but does not block) +TARGET_IP_ONLY="$(echo "$TARGET_HOST" | cut -d@ -f2)" +PRE_HEALTH=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 "http://$TARGET_IP_ONLY/health" 2>/dev/null || echo "000") +if [ "$PRE_HEALTH" = "200" ]; then + echo " Server health: OK (200)" +else + echo " ⚠️ Server health: $PRE_HEALTH (may be down or unhealthy — deploying anyway)" +fi echo "" # When --both: deploy to 228 first, then copy to 198 @@ -634,13 +643,32 @@ PYEOF fi # end FRONTEND_ONLY guard + # Post-deploy health check — wait up to 60s for server to come healthy + echo "" + echo "$(timestamp) 🩺 Post-deploy health check..." + HEALTH_OK=false + for i in $(seq 1 12); do + POST_HEALTH=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 "http://$TARGET_IP_ONLY/health" 2>/dev/null || echo "000") + if [ "$POST_HEALTH" = "200" ]; then + echo " Health: OK (200) after $((i * 5))s" + HEALTH_OK=true + break + fi + echo " Health: $POST_HEALTH (waiting... ${i}/12)" + sleep 5 + done + if [ "$HEALTH_OK" = false ]; then + echo " ⚠️ Server did not become healthy within 60s (last: $POST_HEALTH)" + echo " Rollback: ssh $TARGET_HOST and check 'sudo journalctl -u archipelago -n 50'" + fi + DEPLOY_END=$(date +%s) DEPLOY_ELAPSED=$((DEPLOY_END - DEPLOY_START)) echo "" echo "$(timestamp) ✅ Deployed to live system! (${DEPLOY_ELAPSED}s total)" echo " Backend: $(ssh $SSH_OPTS "$TARGET_HOST" 'sudo systemctl is-active archipelago')" - echo " Web UI: http://$(echo $TARGET_HOST | cut -d@ -f2)" - echo " PWA install: https://$(echo $TARGET_HOST | cut -d@ -f2) (use HTTPS, accept cert once, then Install app)" + echo " Web UI: http://$TARGET_IP_ONLY" + echo " PWA install: https://$TARGET_IP_ONLY (use HTTPS, accept cert once, then Install app)" else echo "" echo "✅ Build complete!" diff --git a/scripts/kiosk-watchdog.sh b/scripts/kiosk-watchdog.sh new file mode 100755 index 00000000..9f920b7e --- /dev/null +++ b/scripts/kiosk-watchdog.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# kiosk-watchdog.sh — Monitors Archipelago backend health. +# Restarts the backend if it's unresponsive for 60 seconds. +# Shows server IP on text console if X is not running. +# +# Designed to run as a systemd service or standalone. + +set -euo pipefail + +HEALTH_URL="http://localhost/health" +CHECK_INTERVAL=5 # seconds between checks +MAX_FAILS=12 # 12 x 5s = 60s before restart + +FAIL_COUNT=0 + +echo "Archipelago kiosk watchdog started" + +while true; do + # Check backend health + if curl -sf "$HEALTH_URL" > /dev/null 2>&1; then + if [ "$FAIL_COUNT" -gt 0 ]; then + echo "Backend recovered after $FAIL_COUNT failed checks" + fi + FAIL_COUNT=0 + else + FAIL_COUNT=$((FAIL_COUNT + 1)) + echo "Health check failed ($FAIL_COUNT/$MAX_FAILS)" + + if [ "$FAIL_COUNT" -ge "$MAX_FAILS" ]; then + echo "Backend unresponsive for 60s, restarting archipelago service..." + systemctl restart archipelago || true + FAIL_COUNT=0 + sleep 10 + continue + fi + fi + + # If X is not running, display IP on text console as fallback + if ! pgrep -x Xorg > /dev/null 2>&1 && ! pgrep -x chromium > /dev/null 2>&1; then + IP=$(hostname -I 2>/dev/null | awk '{print $1}') + if [ -n "$IP" ]; then + printf '\n\n Archipelago Server\n IP: %s\n Web UI: http://%s\n\n' "$IP" "$IP" > /dev/tty1 2>/dev/null || true + fi + fi + + sleep "$CHECK_INTERVAL" +done diff --git a/scripts/setup-kiosk.sh b/scripts/setup-kiosk.sh old mode 100644 new mode 100755 index a03cc1a2..7fb0a5a0 --- a/scripts/setup-kiosk.sh +++ b/scripts/setup-kiosk.sh @@ -22,16 +22,32 @@ XINITRC="$HOMEDIR/.xinitrc" cat > "$XINITRC" << 'XINITRC_EOF' #!/bin/bash -# Archipelago kiosk - Chromium fullscreen -exec chromium --kiosk \ - --app=http://localhost \ - --noerrdialogs \ - --disable-infobars \ - --disable-translate \ - --no-first-run \ - --check-for-update-interval=31536000 \ - --disable-features=TranslateUI \ - --disable-session-crashed-bubble +# Archipelago kiosk — Chromium fullscreen with auto-restart on crash + +# Disable screen blanking +xset s off +xset -dpms +xset s noblank + +# Hide cursor after inactivity +unclutter -idle 3 -root & + +# Run Chromium in a restart loop (recovers from crashes within ~3s) +while true; do + chromium --kiosk \ + --app=http://localhost/kiosk \ + --noerrdialogs \ + --disable-infobars \ + --disable-translate \ + --no-first-run \ + --check-for-update-interval=31536000 \ + --disable-features=TranslateUI \ + --disable-session-crashed-bubble \ + --disable-save-password-bubble \ + --disable-suggestions-service \ + --disable-component-update + sleep 3 +done XINITRC_EOF # Replace localhost with actual URL if different @@ -59,18 +75,81 @@ cat >> "$BASHPROFILE" << 'BASHPROFILE_EOF' # ARCHIPELAGO_KIOSK - Start X/kiosk when logging in at physical console if [ -z "$DISPLAY" ] && [ "$(tty)" = "/dev/tty1" ]; then - exec startx + startx 2>/dev/null + # If X fails, show IP on text console as fallback + if [ $? -ne 0 ]; then + IP=$(hostname -I | awk '{print $1}') + echo "" + echo " ============================================= " + echo " Archipelago Server (kiosk display failed) " + echo " IP: $IP " + echo " Web UI: http://$IP " + echo " ============================================= " + echo "" + fi fi # END ARCHIPELAGO_KIOSK BASHPROFILE_EOF chown "$KIOSK_USER:$KIOSK_USER" "$BASHPROFILE" -echo "✅ Kiosk installed!" +# Install kiosk X11 launcher script (used by systemd service) +KIOSK_X11="/usr/local/bin/archipelago-kiosk-x11" +cat > "$KIOSK_X11" << 'X11_EOF' +#!/bin/bash +# Archipelago kiosk X11 session — launched by systemd or startx + +# Disable screen blanking +xset s off +xset -dpms +xset s noblank + +# Hide cursor after inactivity +unclutter -idle 3 -root & + +# Run Chromium in a restart loop (recovers from crashes within ~3s) +while true; do + chromium --kiosk \ + --app=http://localhost/kiosk \ + --noerrdialogs \ + --disable-infobars \ + --disable-translate \ + --no-first-run \ + --check-for-update-interval=31536000 \ + --disable-features=TranslateUI \ + --disable-session-crashed-bubble \ + --disable-save-password-bubble \ + --disable-suggestions-service \ + --disable-component-update + sleep 3 +done +X11_EOF + +# Replace localhost with actual URL if different +if [ "$ARCHIPELAGO_URL" != "http://localhost" ]; then + sed -i "s|http://localhost|$ARCHIPELAGO_URL|g" "$KIOSK_X11" +fi +chmod +x "$KIOSK_X11" + +# Install kiosk watchdog script +install -m 755 "$(dirname "$0")/kiosk-watchdog.sh" /usr/local/bin/archipelago-kiosk-watchdog 2>/dev/null || true + +# Install systemd services +SCRIPT_DIR="$(cd "$(dirname "$0")/.." && pwd)" +if [ -f "$SCRIPT_DIR/image-recipe/configs/archipelago-kiosk.service" ]; then + cp "$SCRIPT_DIR/image-recipe/configs/archipelago-kiosk.service" /etc/systemd/system/ + cp "$SCRIPT_DIR/image-recipe/configs/archipelago-kiosk-watchdog.service" /etc/systemd/system/ + systemctl daemon-reload + systemctl enable archipelago-kiosk-watchdog + echo " Systemd services installed (enable archipelago-kiosk.service to auto-start)" +fi + +echo "Kiosk installed!" echo "" echo " When you log in at the physical console (monitor + keyboard):" echo " - X will start automatically" -echo " - Chromium will open in kiosk mode" +echo " - Chromium opens in kiosk mode with crash auto-restart" +echo " - If X fails, IP address is displayed on text console" echo " - Your keyboard/touchpad will control the Archipelago UI" echo "" echo " To use: Connect a display, plug in keyboard, reboot (or log in at tty1)" diff --git a/scripts/test-all-apps.sh b/scripts/test-all-apps.sh new file mode 100755 index 00000000..6f9062af --- /dev/null +++ b/scripts/test-all-apps.sh @@ -0,0 +1,240 @@ +#!/usr/bin/env bash +# test-all-apps.sh — End-to-end integration test for all marketplace apps. +# Tests each app through: install → health check → UI access → stop → restart → uninstall. +# +# Usage: SSH to the server and run: +# ./scripts/test-all-apps.sh +# +# Or run remotely: +# ssh archipelago@192.168.1.228 "cd ~/archy && bash scripts/test-all-apps.sh" + +set -euo pipefail + +# Configuration +RPC_URL="http://localhost:5678/rpc/" +HEALTH_URL="http://localhost/health" +MAX_WAIT=120 # Max seconds to wait for container healthy +COOKIE_FILE=$(mktemp) + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +# Counters +PASS=0 +FAIL=0 +SKIP=0 +RESULTS=() + +# Apps to test with their docker images +declare -A APP_IMAGES=( + ["filebrowser"]="docker.io/filebrowser/filebrowser:v2-s6" +) + +# Apps that need archy-net dependencies (skip if dependency not running) +declare -A APP_DEPS=( + ["electrs"]="bitcoin-knots" + ["mempool"]="bitcoin-knots electrs" + ["btcpay"]="bitcoin-knots" + ["lnd"]="bitcoin-knots" +) + +log() { echo -e "${GREEN}[TEST]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +fail() { echo -e "${RED}[FAIL]${NC} $*"; } + +# Authenticate and get session cookie +authenticate() { + log "Authenticating..." + local response + response=$(curl -s -c "$COOKIE_FILE" -X POST "$RPC_URL" \ + -H "Content-Type: application/json" \ + -d '{"method":"auth.login","params":{"password":"password123"}}') + + if echo "$response" | grep -q '"error"'; then + # Try to get CSRF token from cookies + local csrf + csrf=$(grep csrf_token "$COOKIE_FILE" 2>/dev/null | awk '{print $NF}') + if [ -z "$csrf" ]; then + fail "Authentication failed: $response" + exit 1 + fi + fi + log "Authenticated" +} + +# RPC call helper +rpc() { + local method="$1" + local params="${2:-null}" + local csrf + csrf=$(grep csrf_token "$COOKIE_FILE" 2>/dev/null | awk '{print $NF}' || echo "") + + curl -s -b "$COOKIE_FILE" -X POST "$RPC_URL" \ + -H "Content-Type: application/json" \ + -H "X-CSRF-Token: $csrf" \ + -d "{\"method\":\"$method\",\"params\":$params}" +} + +# Wait for a container to be running +wait_for_container() { + local app_id="$1" + local timeout="$2" + local elapsed=0 + + while [ "$elapsed" -lt "$timeout" ]; do + local status + status=$(rpc "container-status" "{\"id\":\"$app_id\"}" 2>/dev/null || echo "") + if echo "$status" | grep -qi '"running"'; then + return 0 + fi + + # Also check via package list + local packages + packages=$(rpc "container-list" 2>/dev/null || echo "") + if echo "$packages" | grep -qi "\"$app_id\".*running"; then + return 0 + fi + + sleep 5 + elapsed=$((elapsed + 5)) + done + return 1 +} + +# Test a single app lifecycle +test_app() { + local app_id="$1" + local docker_image="$2" + local result="PASS" + + echo "" + log "==========================================" + log "Testing: $app_id" + log "==========================================" + + # Check dependencies + if [ -n "${APP_DEPS[$app_id]:-}" ]; then + for dep in ${APP_DEPS[$app_id]}; do + local dep_status + dep_status=$(rpc "container-status" "{\"id\":\"$dep\"}" 2>/dev/null || echo "") + if ! echo "$dep_status" | grep -qi '"running"'; then + warn "Skipping $app_id — dependency $dep not running" + SKIP=$((SKIP + 1)) + RESULTS+=("SKIP $app_id (needs $dep)") + return + fi + done + fi + + # Step 1: Install + log "[$app_id] Installing..." + local install_result + install_result=$(rpc "package.install" "{\"id\":\"$app_id\",\"dockerImage\":\"$docker_image\"}" 2>/dev/null || echo "error") + + if echo "$install_result" | grep -qi '"error"'; then + # May already be installed + if echo "$install_result" | grep -qi "already exists"; then + warn "[$app_id] Already installed, continuing..." + else + fail "[$app_id] Install failed: $install_result" + FAIL=$((FAIL + 1)) + RESULTS+=("FAIL $app_id (install failed)") + return + fi + fi + + # Step 2: Wait for healthy + log "[$app_id] Waiting for container to be running (max ${MAX_WAIT}s)..." + if ! wait_for_container "$app_id" "$MAX_WAIT"; then + fail "[$app_id] Container did not start within ${MAX_WAIT}s" + result="FAIL" + else + log "[$app_id] Container is running" + fi + + # Step 3: Stop + log "[$app_id] Stopping..." + rpc "package.stop" "{\"id\":\"$app_id\"}" > /dev/null 2>&1 + sleep 3 + + # Step 4: Restart + log "[$app_id] Restarting..." + rpc "package.start" "{\"id\":\"$app_id\"}" > /dev/null 2>&1 + sleep 5 + + if ! wait_for_container "$app_id" 60; then + fail "[$app_id] Container did not restart" + result="FAIL" + else + log "[$app_id] Restart successful" + fi + + # Step 5: Uninstall + log "[$app_id] Uninstalling..." + rpc "package.uninstall" "{\"id\":\"$app_id\"}" > /dev/null 2>&1 + sleep 3 + + # Verify removed + local check + check=$(rpc "container-status" "{\"id\":\"$app_id\"}" 2>/dev/null || echo "") + if echo "$check" | grep -qi '"running"'; then + fail "[$app_id] Container still running after uninstall" + result="FAIL" + fi + + if [ "$result" = "PASS" ]; then + log "[$app_id] PASSED" + PASS=$((PASS + 1)) + RESULTS+=("PASS $app_id") + else + FAIL=$((FAIL + 1)) + RESULTS+=("FAIL $app_id") + fi +} + +# Main +echo "" +echo "============================================" +echo " Archipelago App Integration Test Suite" +echo "============================================" +echo "" + +# Check backend health +if ! curl -sf "$HEALTH_URL" > /dev/null 2>&1; then + fail "Backend not healthy at $HEALTH_URL" + exit 1 +fi +log "Backend is healthy" + +authenticate + +# Test each app +for app_id in "${!APP_IMAGES[@]}"; do + test_app "$app_id" "${APP_IMAGES[$app_id]}" +done + +# Cleanup +rm -f "$COOKIE_FILE" + +# Summary +echo "" +echo "============================================" +echo " Test Results" +echo "============================================" +for r in "${RESULTS[@]}"; do + case "$r" in + PASS*) echo -e " ${GREEN}$r${NC}" ;; + FAIL*) echo -e " ${RED}$r${NC}" ;; + SKIP*) echo -e " ${YELLOW}$r${NC}" ;; + esac +done +echo "" +echo " Passed: $PASS Failed: $FAIL Skipped: $SKIP" +echo "============================================" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi diff --git a/scripts/test-stability-72h.sh b/scripts/test-stability-72h.sh index 5dbcc48c..e4e24fc9 100755 --- a/scripts/test-stability-72h.sh +++ b/scripts/test-stability-72h.sh @@ -60,13 +60,13 @@ check_health() { # 3. RPC responds local rpc_resp - rpc_resp=$(rpc "system.info" 2>/dev/null) + rpc_resp=$(rpc "server.echo" '{"message":"stability-check"}' 2>/dev/null) if ! echo "$rpc_resp" | grep -q '"result"'; then # Try re-login login - rpc_resp=$(rpc "system.info" 2>/dev/null) + rpc_resp=$(rpc "server.echo" '{"message":"stability-check"}' 2>/dev/null) if ! echo "$rpc_resp" | grep -q '"result"'; then - fail_log "RPC system.info failed after re-login" + fail_log "RPC server.echo failed after re-login" failures=$((failures + 1)) fi fi diff --git a/scripts/uptime-monitor.sh b/scripts/uptime-monitor.sh new file mode 100755 index 00000000..498302c2 --- /dev/null +++ b/scripts/uptime-monitor.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Uptime Monitor for REL-05 +# Runs every 5 minutes via cron, records metrics to a CSV file. +# Install: */5 * * * * /opt/archipelago/scripts/uptime-monitor.sh +# +# Tracks: timestamp, http_status, response_time_ms, cpu_percent, +# mem_used_mb, mem_total_mb, disk_used_gb, disk_total_gb, +# container_count, uptime_secs, restart_count + +set -euo pipefail + +LOG_DIR="/var/lib/archipelago/uptime-monitor" +LOG_FILE="$LOG_DIR/metrics.csv" +RESTART_FILE="$LOG_DIR/restart-count" +BACKEND_URL="http://localhost:5678/health" +RPC_URL="http://localhost:5678/rpc/v1" + +mkdir -p "$LOG_DIR" + +# Write CSV header if file doesn't exist +if [ ! -f "$LOG_FILE" ]; then + echo "timestamp,http_status,response_ms,cpu_percent,mem_used_mb,mem_total_mb,disk_used_gb,disk_total_gb,containers,uptime_secs,restart_count" > "$LOG_FILE" +fi + +# Track restart count +if [ ! -f "$RESTART_FILE" ]; then + echo "0" > "$RESTART_FILE" +fi +RESTART_COUNT=$(cat "$RESTART_FILE" 2>/dev/null || echo "0") + +TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + +# Check HTTP health +HTTP_START=$(date +%s%N) +HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$BACKEND_URL" 2>/dev/null || echo "000") +HTTP_END=$(date +%s%N) +RESPONSE_MS=$(( (HTTP_END - HTTP_START) / 1000000 )) + +# Get system stats from RPC +STATS=$(curl -s --max-time 10 -X POST "$RPC_URL" \ + -H "Content-Type: application/json" \ + -d '{"method":"system.stats"}' 2>/dev/null || echo '{"result":{}}') + +CPU=$(echo "$STATS" | python3 -c "import sys,json; d=json.load(sys.stdin).get('result',{}); print(d.get('cpu_usage_percent',0))" 2>/dev/null || echo "0") +MEM_USED=$(echo "$STATS" | python3 -c "import sys,json; d=json.load(sys.stdin).get('result',{}); print(round(d.get('mem_used_bytes',0)/1048576))" 2>/dev/null || echo "0") +MEM_TOTAL=$(echo "$STATS" | python3 -c "import sys,json; d=json.load(sys.stdin).get('result',{}); print(round(d.get('mem_total_bytes',0)/1048576))" 2>/dev/null || echo "0") +DISK_USED=$(echo "$STATS" | python3 -c "import sys,json; d=json.load(sys.stdin).get('result',{}); print(round(d.get('disk_used_bytes',0)/1073741824,1))" 2>/dev/null || echo "0") +DISK_TOTAL=$(echo "$STATS" | python3 -c "import sys,json; d=json.load(sys.stdin).get('result',{}); print(round(d.get('disk_total_bytes',0)/1073741824,1))" 2>/dev/null || echo "0") +UPTIME=$(echo "$STATS" | python3 -c "import sys,json; d=json.load(sys.stdin).get('result',{}); print(d.get('uptime_secs',0))" 2>/dev/null || echo "0") + +# Count running containers +CONTAINERS=$(sudo podman ps --format "{{.Names}}" 2>/dev/null | wc -l || echo "0") + +# Detect restart (uptime < 300s = likely just restarted) +if [ "$UPTIME" -lt 300 ] 2>/dev/null; then + # Check if we already counted this restart + LAST_UPTIME_FILE="$LOG_DIR/last-uptime" + LAST_UPTIME=$(cat "$LAST_UPTIME_FILE" 2>/dev/null || echo "99999") + if [ "$LAST_UPTIME" -gt 300 ] 2>/dev/null; then + RESTART_COUNT=$((RESTART_COUNT + 1)) + echo "$RESTART_COUNT" > "$RESTART_FILE" + fi + echo "$UPTIME" > "$LAST_UPTIME_FILE" +else + echo "$UPTIME" > "$LOG_DIR/last-uptime" +fi + +# Append metrics +echo "$TIMESTAMP,$HTTP_STATUS,$RESPONSE_MS,$CPU,$MEM_USED,$MEM_TOTAL,$DISK_USED,$DISK_TOTAL,$CONTAINERS,$UPTIME,$RESTART_COUNT" >> "$LOG_FILE" + +# Generate summary report +TOTAL_CHECKS=$(wc -l < "$LOG_FILE") +TOTAL_CHECKS=$((TOTAL_CHECKS - 1)) # exclude header +if [ "$TOTAL_CHECKS" -gt 0 ]; then + OK_CHECKS=$(grep -c ",200," "$LOG_FILE" || echo "0") + UPTIME_PCT=$(python3 -c "print(round($OK_CHECKS / $TOTAL_CHECKS * 100, 3))" 2>/dev/null || echo "0") + + cat > "$LOG_DIR/summary.json" << EOF +{ + "start": "$(head -2 "$LOG_FILE" | tail -1 | cut -d',' -f1)", + "last_check": "$TIMESTAMP", + "total_checks": $TOTAL_CHECKS, + "ok_checks": $OK_CHECKS, + "uptime_percent": $UPTIME_PCT, + "restart_count": $RESTART_COUNT, + "current_status": "$HTTP_STATUS" +} +EOF +fi