mid coding commit

.cursor/rules/coding-rules.mdc

@@ -0,0 +1,271 @@
+# Archipelago Development Rules
+
+## CRITICAL: Project Structure & Location
+
+### NEVER Reference External Directories
+- ❌ **NEVER** reference `/Users/tx1138/Code/Archipelago/` in code, scripts, or documentation
+- ✅ **ALWAYS** use workspace-relative paths: `./`, `../`, or `$PROJECT_ROOT`
+- ✅ The workspace at `/Users/tx1138/Archipelago` is the ONLY project location
+- ✅ All files must be created in the workspace, never in external directories
+
+### File Creation Rules
+- ✅ Create files directly in the workspace using relative paths
+- ❌ Never assume files exist elsewhere - check first, create if missing
+- ✅ When copying from external sources, copy TO workspace, then update references
+
+## Design System & Styling
+
+### Tailwind CSS Rules
+- ✅ **ALWAYS** create global utility classes in `neode-ui/src/style.css` or a dedicated `tailwind.css`
+- ❌ **NEVER** use inline Tailwind classes directly in components
+- ✅ Create semantic class names: `.glass-card`, `.glass-button`, `.nav-tab-active`
+- ✅ Use CSS variables for design tokens: `--color-primary`, `--spacing-base`
+
+### Design Standards (From Memory)
+- **Font**: Avenir Next font family (preferred)
+- **Padding**: 4px grid system, 16px default padding
+- **Containers**: iOS-style glassmorphism
+  - Background: `rgba(255,255,255,0.15)`
+  - Backdrop blur: `20px`
+  - Subtle white borders
+- **Backgrounds**: Persistent background images (not dark themes)
+- **Animations**: Smooth 2s splash screens with logo draw/glitch animations
+
+### Example Global Classes
+```css
+.glass-card {
+  background: rgba(255, 255, 255, 0.15);
+  backdrop-filter: blur(20px);
+  -webkit-backdrop-filter: blur(20px);
+  border: 1px solid rgba(255, 255, 255, 0.2);
+  border-radius: 16px;
+  padding: 16px; /* 4px grid */
+}
+
+.glass-button {
+  background: rgba(255, 255, 255, 0.1);
+  backdrop-filter: blur(10px);
+  border: 1px solid rgba(255, 255, 255, 0.15);
+  transition: all 0.2s ease;
+}
+
+.glass-button:hover {
+  background: rgba(255, 255, 255, 0.2);
+}
+```
+
+## StartOS Independence
+
+### Zero StartOS Dependencies
+- ❌ **NEVER** import or reference StartOS-specific code
+- ❌ **NEVER** copy StartOS patterns without refactoring
+- ✅ **ALWAYS** create Archipelago-native implementations
+- ✅ Use our own container orchestration (`core/container/`)
+- ✅ Use our own security modules (`core/security/`)
+- ✅ Use our own performance modules (`core/performance/`)
+
+### Backend Architecture
+- ✅ Use `archipelago-container` crate, not StartOS container code
+- ✅ Use our RPC endpoints in `core/startos/src/container/`
+- ⚠️ **TEMPORARY**: Using StartOS backend (`startbox`) as base - this is temporary
+- ✅ **GOAL**: Build our own Archipelago backend binary that uses ONLY our modules
+- ✅ Mark all StartOS-derived code with `// TODO: Refactor to Archipelago-native`
+- ✅ For development: Use mock backend for UI work, avoid StartOS backend when possible
+- ✅ All new features must use our modules (`archipelago-container`, `archipelago-security`, etc.)
+
+## Container & App Development
+
+### App Manifest Rules
+- ✅ **ALWAYS** create manifests in `apps/{app-id}/manifest.yml`
+- ✅ Follow the manifest specification in `docs/app-manifest-spec.md`
+- ✅ Use semantic versioning: `MAJOR.MINOR.PATCH`
+- ✅ Include security policies, resource limits, health checks
+
+### Container Orchestration
+- ✅ Use `archipelago_container::PodmanClient` for all container operations
+- ✅ Use `archipelago_container::AppManifest` for manifest parsing
+- ✅ Use `archipelago_container::DependencyResolver` for dependency management
+- ❌ Never use Docker directly - always use Podman via our client
+
+### Security First
+- ✅ **ALWAYS** set `readonly_root: true` unless explicitly needed
+- ✅ **ALWAYS** drop all capabilities, add only required ones
+- ✅ **ALWAYS** use isolated networks unless host network is required
+- ✅ **ALWAYS** verify container images with Cosign signatures
+- ✅ Use AppArmor profiles from `core/security/`
+
+## Frontend Development
+
+### Vue.js Component Rules
+- ✅ Use Composition API (`<script setup lang="ts">`)
+- ✅ Use Pinia stores for state management
+- ✅ Use TypeScript for all components
+- ✅ Create reusable components in `neode-ui/src/components/`
+- ✅ Use global Tailwind classes, not inline utilities
+
+### API Client Rules
+- ✅ Use `neode-ui/src/api/rpc-client.ts` for RPC calls
+- ✅ Use `neode-ui/src/api/container-client.ts` for container operations
+- ✅ **NEVER** hardcode API endpoints - use environment variables
+- ✅ Handle errors gracefully with user-friendly messages
+
+### State Management
+- ✅ Use Pinia stores for all application state
+- ✅ Keep stores focused and single-purpose
+- ✅ Use TypeScript interfaces for store state
+- ✅ Don't duplicate state - use computed properties
+
+## Backend Development
+
+### Rust Code Organization
+- ✅ New modules go in `core/{module-name}/`
+- ✅ Use workspace structure: add to `core/Cargo.toml` members
+- ✅ Follow Rust naming conventions: `snake_case` for modules/files
+- ✅ Use `thiserror` for error types, `anyhow` for error handling
+
+### RPC Endpoint Rules
+- ✅ Use `rpc_toolkit::command` macro for all endpoints
+- ✅ Use `#[context] ctx: RpcContext` for context
+- ✅ Use `#[arg]` for parameters
+- ✅ Return `Result<T, Error>` for all endpoints
+- ✅ Add endpoints to `core/startos/src/lib.rs` subcommands
+
+### Error Handling
+- ✅ Use `crate::Error` and `crate::ErrorKind` for errors
+- ✅ Provide context with `.context()` or `.with_kind()`
+- ✅ Log errors with `tracing::error!` or `log::error!`
+- ✅ Return user-friendly error messages
+
+## Development Workflow
+
+### Scripts & Automation
+- ✅ All scripts in `scripts/` directory
+- ✅ Use `#!/bin/bash` with `set +e` (don't exit on first error)
+- ✅ Check for prerequisites before running
+- ✅ Provide clear error messages
+- ✅ Use workspace-relative paths
+
+### Node.js & Dependencies
+- ⚠️ **Node.js Version**: Requires Node.js 20.19+ or 22.12+ for Vite 7
+- ✅ If dependencies are broken, delete `node_modules` and `package-lock.json`, then `npm install`
+- ✅ Always verify `node_modules/.bin/` executables work after install
+- ✅ Use `npm ci` for CI/CD (clean install from lock file)
+
+### Testing
+- ✅ Write tests for all Rust modules
+- ✅ Test container operations with mock Podman
+- ✅ Test UI components with Vitest
+- ✅ Test API endpoints with integration tests
+
+### Documentation
+- ✅ Update `docs/` when adding features
+- ✅ Document all public APIs
+- ✅ Include examples in documentation
+- ✅ Keep README.md up to date
+
+## Common Mistakes to Avoid
+
+### ❌ DON'T:
+1. Reference `/Users/tx1138/Code/Archipelago/` anywhere
+2. Create files outside the workspace
+3. Use inline Tailwind classes
+4. Import StartOS code directly
+5. Skip security policies in manifests
+6. Hardcode paths or URLs
+7. Forget to add new modules to Cargo.toml
+8. Create components without global styles
+9. Use Docker instead of Podman
+10. Skip error handling
+
+### ✅ DO:
+1. Always use workspace-relative paths
+2. Create global Tailwind utility classes
+3. Build Archipelago-native solutions
+4. Include security in all containers
+5. Use environment variables for configuration
+6. Add modules to workspace Cargo.toml
+7. Create reusable styled components
+8. Use Podman via our client wrapper
+9. Handle all errors gracefully
+10. Follow the architecture plan
+
+## Architecture Adherence
+
+### Stick to the Plan
+- ✅ Follow `docs/architecture.md` for system design
+- ✅ Use Alpine Linux base (not Ubuntu/Debian)
+- ✅ Use Podman (not Docker)
+- ✅ Use rootless containers
+- ✅ Implement security hardening
+- ✅ Support multi-arch (ARM64, x86_64)
+
+### Container Orchestration
+- ✅ Use manifest-based app definitions
+- ✅ Implement dependency resolution
+- ✅ Monitor container health
+- ✅ Support Parmanode compatibility
+- ✅ Enable secrets management
+
+### Future-Proofing
+- ✅ Design for time-travel snapshots
+- ✅ Plan for decentralized marketplace
+- ✅ Support multi-node clustering
+- ✅ Enable hardware attestation
+- ✅ Keep protocol-agnostic design
+
+## Code Quality
+
+### TypeScript
+- ✅ Use strict mode
+- ✅ Define interfaces for all data structures
+- ✅ Use type guards for runtime checks
+- ✅ Avoid `any` - use `unknown` if needed
+
+### Rust
+- ✅ Use `clippy` for linting
+- ✅ Use `rustfmt` for formatting
+- ✅ Document public APIs with `///`
+- ✅ Use `#[derive(Debug)]` for error types
+
+### General
+- ✅ Keep functions small and focused
+- ✅ Use descriptive variable names
+- ✅ Comment complex logic
+- ✅ Remove dead code
+- ✅ Follow existing code style
+
+## Performance
+
+### Optimization Rules
+- ✅ Use resource limits in all containers
+- ✅ Implement caching where appropriate
+- ✅ Lazy load components when possible
+- ✅ Optimize images and assets
+- ✅ Use connection pooling for databases
+
+### Monitoring
+- ✅ Log important events
+- ✅ Track container resource usage
+- ✅ Monitor health checks
+- ✅ Alert on failures
+
+## Security
+
+### Always Implement
+- ✅ Image signature verification
+- ✅ Secrets encryption
+- ✅ AppArmor/SELinux profiles
+- ✅ Network isolation
+- ✅ Capability dropping
+- ✅ Read-only root filesystems
+
+### Never Skip
+- ❌ Security policies in manifests
+- ❌ Image verification
+- ❌ Secrets management
+- ❌ Network isolation
+- ❌ Resource limits
+
+---
+
+**Remember**: This is Archipelago, not StartOS. Build it right, build it secure, build it our way.

README.md

@@ -0,0 +1,99 @@
+# Archipelago Bitcoin Node OS
+
+Next-generation Bitcoin Node OS built on Alpine Linux with Podman containerization.
+
+## Quick Start
+
+### Development Setup
+
+1. **Run setup script (optional):**
+   ```bash
+   ./scripts/dev-setup.sh
+   ```
+
+2. **Start development servers:**
+
+   **Quick start (mock backend for UI development):**
+   ```bash
+   ./scripts/dev.sh
+   ```
+   
+   **Or use the interactive starter:**
+   ```bash
+   ./scripts/dev-start.sh
+   ```
+
+   **Or manually:**
+   ```bash
+   # Terminal 1: Backend
+   cd /Users/tx1138/Code/Archipelago/core
+   cargo run --bin startbox
+   
+   # Terminal 2: Frontend
+   cd /Users/tx1138/Code/Archipelago/neode-ui
+   npm run dev
+   ```
+
+3. **Open in browser:**
+   - Frontend: http://localhost:8100
+   - Backend API: http://localhost:5959
+
+### Mock Backend (UI Development Only)
+
+For frontend-only development:
+
+```bash
+cd /Users/tx1138/Code/Archipelago/neode-ui
+npm run dev:mock
+```
+
+## Project Structure
+
+```
+Archipelago/
+├── core/                    # Rust backend
+│   ├── container/          # Container orchestration (NEW)
+│   ├── parmanode/          # Parmanode compatibility (NEW)
+│   ├── security/           # Security modules (NEW)
+│   ├── performance/        # Performance optimization (NEW)
+│   └── startos/            # Main backend (in Code/Archipelago)
+├── neode-ui/               # Vue.js frontend (in Code/Archipelago)
+├── apps/                   # App manifests (NEW)
+├── image-recipe/           # Alpine Linux build files
+├── scripts/                # Development and build scripts
+└── docs/                   # Documentation
+```
+
+## Development
+
+See [Development Setup Guide](./docs/development-setup.md) for detailed instructions.
+
+## Architecture
+
+See [Architecture Documentation](./docs/architecture.md) for system design details.
+
+## App Manifests
+
+See [App Manifest Specification](./docs/app-manifest-spec.md) for creating containerized apps.
+
+## Features
+
+- 🐧 **Alpine Linux Base** - Minimal 130MB OS
+- 🐳 **Podman Containers** - Rootless, secure containerization
+- 🔒 **Security Hardened** - AppArmor, secrets management, image verification
+- ⚡ **High Performance** - Resource management, optimization
+- 🔌 **Parmanode Compatible** - Run existing Parmanode modules
+- 📱 **Modern UI** - Vue.js 3 with TypeScript
+- 🌐 **Web5 & Nostr** - Decentralized protocols support
+- 📡 **Mesh Networking** - Meshtastic and router support
+
+## Requirements
+
+- Rust (latest stable)
+- Node.js 18+
+- Podman (for containers)
+- PostgreSQL (for backend)
+
+## License
+
+MIT

core/Cargo.toml

@@ -0,0 +1,29 @@
+[workspace]
+resolver = "2"
+
+members = [
+    "archipelago",
+    "container",
+    "parmanode",
+    "performance",
+    "security",
+]
+
+# Profiles at workspace root (members' [profile] are ignored in virtual workspaces)
+[profile.release]
+opt-level = 3
+
+[profile.dev]
+opt-level = 0
+
+[profile.test]
+opt-level = 3
+
+[profile.dev.package.backtrace]
+opt-level = 3
+
+[profile.dev.package.sqlx-macros]
+opt-level = 3
+
+# Archipelago workspace - no StartOS dependencies
+# All patches removed - we use standard crates.io dependencies

core/archipelago/Cargo.toml

@@ -0,0 +1,48 @@
+[package]
+name = "archipelago"
+version = "0.1.0"
+edition = "2021"
+description = "Archipelago Bitcoin Node OS - Native backend"
+authors = ["Archipelago Team"]
+
+[[bin]]
+name = "archipelago"
+path = "src/main.rs"
+
+[dependencies]
+# Core dependencies
+tokio = { version = "1", features = ["full"] }
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+anyhow = "1.0"
+thiserror = "1.0"
+tracing = "0.1"
+tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+
+# HTTP and WebSocket
+hyper = { version = "0.14", features = ["full", "http1"] }
+hyper-util = { version = "0.1", features = ["full", "http1"] }
+http-body-util = "0.1"
+tower = "0.5"
+tower-http = { version = "0.6", features = ["cors", "trace"] }
+tokio-tungstenite = { version = "0.20", features = ["native-tls"] }
+
+# Our modules
+archipelago-container = { path = "../container" }
+archipelago-security = { path = "../security" }
+archipelago-performance = { path = "../performance" }
+archipelago-parmanode = { path = "../parmanode" }
+
+# Database (optional for now - can use SQLite or skip)
+# sqlx = { version = "0.7", features = ["sqlite", "runtime-tokio-rustls"] }
+
+# Authentication
+bcrypt = "0.15"
+uuid = { version = "1.0", features = ["v4"] }
+
+# Configuration
+toml = "0.8"
+serde_yaml = "0.9"
+
+[dev-dependencies]
+tokio-test = "0.4"

core/archipelago/src/api/handler.rs

@@ -0,0 +1,64 @@
+use crate::api::rpc::RpcHandler;
+use crate::config::Config;
+use anyhow::Result;
+use http_body_util::{BodyExt, Full};
+use hyper::body::Bytes;
+use hyper::{Method, Request, Response, StatusCode};
+use hyper_util::rt::TokioIo;
+use std::sync::Arc;
+use tracing::debug;
+
+pub struct ApiHandler {
+    config: Config,
+    rpc_handler: Arc<RpcHandler>,
+    // Add other handlers here (websocket, static files, etc.)
+}
+
+impl ApiHandler {
+    pub async fn new(config: Config) -> Result<Self> {
+        let rpc_handler = Arc::new(RpcHandler::new(config.clone()).await?);
+
+        Ok(Self {
+            config,
+            rpc_handler,
+        })
+    }
+
+    pub async fn handle_request(
+        &self,
+        req: Request<http_body_util::Body<Bytes>>,
+    ) -> Result<Response<Full<Bytes>>> {
+        let path = req.uri().path();
+        let method = req.method();
+        
+        // Convert Incoming body to bytes
+        let (parts, body) = req.into_parts();
+        let collected = body.collect().await
+            .map_err(|e| anyhow::anyhow!("Failed to read body: {}", e))?;
+        let body_bytes = collected.to_bytes();
+        
+        // Reconstruct request with Full<Bytes> body for RPC handler
+        let req_with_bytes = Request::from_parts(parts, Full::new(body_bytes));
+
+        debug!("{} {}", method, path);
+
+        // Route requests
+        match (method, path) {
+            (&Method::POST, "/rpc/v1") => {
+                self.rpc_handler.handle(req_with_bytes).await
+            }
+            (&Method::GET, "/health") => {
+                Ok(Response::builder()
+                    .status(StatusCode::OK)
+                    .body(Full::new(Bytes::from("OK")))
+                    .unwrap())
+            }
+            _ => {
+                Ok(Response::builder()
+                    .status(StatusCode::NOT_FOUND)
+                    .body(Full::new(Bytes::from("Not Found")))
+                    .unwrap())
+            }
+        }
+    }
+}

core/archipelago/src/api/mod.rs

@@ -0,0 +1,5 @@
+mod handler;
+mod rpc;
+
+pub use handler::ApiHandler;
+pub use rpc::RpcHandler;

core/archipelago/src/api/rpc.rs

@@ -0,0 +1,336 @@
+use crate::config::Config;
+use crate::container::DevContainerOrchestrator;
+use anyhow::{Context, Result};
+use http_body_util::{BodyExt, Full};
+use hyper::body::Bytes;
+use hyper::{Request, Response, StatusCode};
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+use tokio::sync::RwLock;
+use tracing::{debug, error};
+
+#[derive(Debug, Deserialize)]
+struct RpcRequest {
+    method: String,
+    params: Option<serde_json::Value>,
+}
+
+#[derive(Debug, Serialize)]
+struct RpcResponse {
+    result: Option<serde_json::Value>,
+    error: Option<RpcError>,
+}
+
+#[derive(Debug, Serialize)]
+struct RpcError {
+    code: i32,
+    message: String,
+    data: Option<serde_json::Value>,
+}
+
+pub struct RpcHandler {
+    config: Config,
+    orchestrator: Option<Arc<DevContainerOrchestrator>>,
+}
+
+impl RpcHandler {
+    pub async fn new(config: Config) -> Result<Self> {
+        let orchestrator = if config.dev_mode {
+            Some(Arc::new(
+                DevContainerOrchestrator::new(config.clone()).await?,
+            ))
+        } else {
+            None
+        };
+
+        Ok(Self {
+            config,
+            orchestrator,
+        })
+    }
+
+    pub async fn handle(
+        &self,
+        req: Request<Full<Bytes>>,
+    ) -> Result<Response<Full<Bytes>>> {
+        // Read request body - Full<Bytes> is already collected
+        let (_, body) = req.into_parts();
+        // Full<Bytes> implements Body, collect it to get the bytes
+        use http_body_util::BodyExt;
+        let collected = body.collect().await
+            .context("Failed to collect body")?;
+        let body_bytes = collected.to_bytes();
+
+        let rpc_req: RpcRequest = serde_json::from_slice(&body_bytes)
+            .context("Invalid RPC request")?;
+
+        debug!("RPC method: {}", rpc_req.method);
+
+        // Route to handler
+        let result = match rpc_req.method.as_str() {
+            "echo" => self.handle_echo(rpc_req.params).await,
+            "server.echo" => self.handle_echo(rpc_req.params).await,
+            "container-install" => self.handle_container_install(rpc_req.params).await,
+            "container-start" => self.handle_container_start(rpc_req.params).await,
+            "container-stop" => self.handle_container_stop(rpc_req.params).await,
+            "container-remove" => self.handle_container_remove(rpc_req.params).await,
+            "container-list" => self.handle_container_list().await,
+            "container-status" => self.handle_container_status(rpc_req.params).await,
+            "container-logs" => self.handle_container_logs(rpc_req.params).await,
+            "container-health" => self.handle_container_health(rpc_req.params).await,
+            _ => {
+                Err(anyhow::anyhow!("Unknown method: {}", rpc_req.method))
+            }
+        };
+
+        // Build response
+        let rpc_resp = match result {
+            Ok(data) => RpcResponse {
+                result: Some(data),
+                error: None,
+            },
+            Err(e) => {
+                error!("RPC error: {}", e);
+                RpcResponse {
+                    result: None,
+                    error: Some(RpcError {
+                        code: -1,
+                        message: e.to_string(),
+                        data: None,
+                    }),
+                }
+            }
+        };
+
+        let body = serde_json::to_vec(&rpc_resp)
+            .context("Failed to serialize response")?;
+
+        Ok(Response::builder()
+            .status(StatusCode::OK)
+            .header("Content-Type", "application/json")
+            .body(Full::new(Bytes::from(body)))
+            .unwrap())
+    }
+
+    async fn handle_echo(&self, params: Option<serde_json::Value>) -> Result<serde_json::Value> {
+        if let Some(p) = params {
+            if let Some(msg) = p.get("message").and_then(|v| v.as_str()) {
+                return Ok(serde_json::json!({ "message": msg }));
+            }
+        }
+        Ok(serde_json::json!({ "message": "Hello from Archipelago!" }))
+    }
+
+    async fn handle_container_install(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
+        let manifest_path = params
+            .get("manifest_path")
+            .and_then(|v| v.as_str())
+            .ok_or_else(|| anyhow::anyhow!("Missing manifest_path"))?;
+
+        // Load manifest
+        let manifest_content = tokio::fs::read_to_string(manifest_path)
+            .await
+            .context("Failed to read manifest file")?;
+        let manifest: archipelago_container::AppManifest = serde_yaml::from_str(&manifest_content)
+            .context("Failed to parse manifest")?;
+
+        let container_name = orchestrator
+            .install_container(&manifest, manifest_path)
+            .await
+            .context("Failed to install container")?;
+
+        Ok(serde_json::json!(container_name))
+    }
+
+    async fn handle_container_start(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
+        let app_id = params
+            .get("app_id")
+            .and_then(|v| v.as_str())
+            .ok_or_else(|| anyhow::anyhow!("Missing app_id"))?;
+
+        orchestrator
+            .start_container(app_id)
+            .await
+            .context("Failed to start container")?;
+
+        Ok(serde_json::json!({ "status": "started" }))
+    }
+
+    async fn handle_container_stop(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
+        let app_id = params
+            .get("app_id")
+            .and_then(|v| v.as_str())
+            .ok_or_else(|| anyhow::anyhow!("Missing app_id"))?;
+
+        orchestrator
+            .stop_container(app_id)
+            .await
+            .context("Failed to stop container")?;
+
+        Ok(serde_json::json!({ "status": "stopped" }))
+    }
+
+    async fn handle_container_remove(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
+        let app_id = params
+            .get("app_id")
+            .and_then(|v| v.as_str())
+            .ok_or_else(|| anyhow::anyhow!("Missing app_id"))?;
+        let preserve_data = params
+            .get("preserve_data")
+            .and_then(|v| v.as_bool())
+            .unwrap_or(false);
+
+        orchestrator
+            .remove_container(app_id, preserve_data)
+            .await
+            .context("Failed to remove container")?;
+
+        Ok(serde_json::json!({ "status": "removed" }))
+    }
+
+    async fn handle_container_list(&self) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let containers = orchestrator
+            .list_containers()
+            .await
+            .context("Failed to list containers")?;
+
+        Ok(serde_json::to_value(containers)?)
+    }
+
+    async fn handle_container_status(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
+        let app_id = params
+            .get("app_id")
+            .and_then(|v| v.as_str())
+            .ok_or_else(|| anyhow::anyhow!("Missing app_id"))?;
+
+        let status = orchestrator
+            .get_container_status(app_id)
+            .await
+            .context("Failed to get container status")?;
+
+        Ok(serde_json::to_value(status)?)
+    }
+
+    async fn handle_container_logs(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
+        let app_id = params
+            .get("app_id")
+            .and_then(|v| v.as_str())
+            .ok_or_else(|| anyhow::anyhow!("Missing app_id"))?;
+        let lines = params
+            .get("lines")
+            .and_then(|v| v.as_u64())
+            .unwrap_or(100) as u32;
+
+        let logs = orchestrator
+            .get_container_logs(app_id, lines)
+            .await
+            .context("Failed to get container logs")?;
+
+        Ok(serde_json::to_value(logs)?)
+    }
+
+    async fn handle_container_health(
+        &self,
+        params: Option<serde_json::Value>,
+    ) -> Result<serde_json::Value> {
+        let orchestrator = self
+            .orchestrator
+            .as_ref()
+            .ok_or_else(|| anyhow::anyhow!("Container orchestrator not available (dev mode required)"))?;
+
+        // If app_id is provided, get health for that app
+        if let Some(params) = params {
+            if let Some(app_id) = params.get("app_id").and_then(|v| v.as_str()) {
+                let health = orchestrator
+                    .get_health_status(app_id)
+                    .await
+                    .context("Failed to get container health")?;
+                return Ok(serde_json::json!({ app_id: health }));
+            }
+        }
+
+        // Otherwise, get health for all containers
+        let containers = orchestrator
+            .list_containers()
+            .await
+            .context("Failed to list containers")?;
+
+        let mut health_map = serde_json::Map::new();
+        for container in containers {
+            // Extract app_id from container name
+            if let Some(app_id) = container.name.strip_prefix("archipelago-") {
+                if let Some(app_id) = app_id.strip_suffix("-dev") {
+                    match orchestrator.get_health_status(app_id).await {
+                        Ok(health) => {
+                            health_map.insert(app_id.to_string(), serde_json::Value::String(health));
+                        }
+                        Err(_) => {
+                            health_map.insert(app_id.to_string(), serde_json::Value::String("unknown".to_string()));
+                        }
+                    }
+                }
+            }
+        }
+
+        Ok(serde_json::Value::Object(health_map))
+    }
+}

core/archipelago/src/auth.rs

@@ -0,0 +1,78 @@
+// Authentication module for Archipelago
+// Handles user setup, onboarding, and login
+
+use anyhow::Result;
+use serde::{Deserialize, Serialize};
+use std::path::PathBuf;
+use tokio::fs;
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct User {
+    pub password_hash: String,
+    pub setup_complete: bool,
+    pub onboarding_complete: bool,
+}
+
+pub struct AuthManager {
+    data_dir: PathBuf,
+}
+
+impl AuthManager {
+    pub fn new(data_dir: PathBuf) -> Self {
+        Self { data_dir }
+    }
+
+    pub async fn is_setup(&self) -> Result<bool> {
+        let user_file = self.data_dir.join("user.json");
+        Ok(user_file.exists())
+    }
+
+    pub async fn get_user(&self) -> Result<Option<User>> {
+        let user_file = self.data_dir.join("user.json");
+        if !user_file.exists() {
+            return Ok(None);
+        }
+
+        let content = fs::read_to_string(&user_file).await?;
+        let user: User = serde_json::from_str(&content)?;
+        Ok(Some(user))
+    }
+
+    pub async fn setup_user(&self, password: &str) -> Result<()> {
+        use bcrypt::{hash, DEFAULT_COST};
+        
+        let password_hash = hash(password, DEFAULT_COST)?;
+        
+        let user = User {
+            password_hash,
+            setup_complete: true,
+            onboarding_complete: false,
+        };
+
+        let user_file = self.data_dir.join("user.json");
+        let content = serde_json::to_string_pretty(&user)?;
+        fs::write(&user_file, content).await?;
+
+        Ok(())
+    }
+
+    pub async fn complete_onboarding(&self) -> Result<()> {
+        if let Some(mut user) = self.get_user().await? {
+            user.onboarding_complete = true;
+            let user_file = self.data_dir.join("user.json");
+            let content = serde_json::to_string_pretty(&user)?;
+            fs::write(&user_file, content).await?;
+        }
+        Ok(())
+    }
+
+    pub async fn verify_password(&self, password: &str) -> Result<bool> {
+        use bcrypt::verify;
+        
+        if let Some(user) = self.get_user().await? {
+            Ok(verify(password, &user.password_hash)?)
+        } else {
+            Ok(false)
+        }
+    }
+}

core/archipelago/src/config.rs

@@ -0,0 +1,139 @@
+use anyhow::{Context, Result};
+use serde::{Deserialize, Serialize};
+use std::path::{Path, PathBuf};
+use tokio::fs;
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum ContainerRuntime {
+    Podman,
+    Docker,
+    Auto,
+}
+
+impl ContainerRuntime {
+    pub fn from_str(s: &str) -> Self {
+        match s.to_lowercase().as_str() {
+            "podman" => ContainerRuntime::Podman,
+            "docker" => ContainerRuntime::Docker,
+            "auto" | _ => ContainerRuntime::Auto,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum BitcoinSimulation {
+    Mock,
+    Testnet,
+    Mainnet,
+    None,
+}
+
+impl BitcoinSimulation {
+    pub fn from_str(s: &str) -> Self {
+        match s.to_lowercase().as_str() {
+            "mock" => BitcoinSimulation::Mock,
+            "testnet" => BitcoinSimulation::Testnet,
+            "mainnet" => BitcoinSimulation::Mainnet,
+            "none" | _ => BitcoinSimulation::None,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct Config {
+    pub data_dir: PathBuf,
+    pub bind_host: String,
+    pub bind_port: u16,
+    pub log_level: String,
+    // Dev mode configuration
+    pub dev_mode: bool,
+    pub container_runtime: ContainerRuntime,
+    pub port_offset: u16,
+    pub bitcoin_simulation: BitcoinSimulation,
+    pub dev_data_dir: PathBuf,
+}
+
+impl Config {
+    pub async fn load() -> Result<Self> {
+        // Default configuration
+        let mut config = Self::default();
+
+        // Try to load from config file
+        let config_path = Path::new("/etc/archipelago/config.toml");
+        if config_path.exists() {
+            let content = fs::read_to_string(config_path).await
+                .context("Failed to read config file")?;
+            let file_config: Config = toml::de::from_str(&content)
+                .context("Failed to parse config file")?;
+            config = file_config;
+        }
+
+        // Override with environment variables
+        if let Ok(data_dir) = std::env::var("ARCHIPELAGO_DATA_DIR") {
+            config.data_dir = PathBuf::from(data_dir);
+        }
+
+        if let Ok(bind) = std::env::var("ARCHIPELAGO_BIND") {
+            let parts: Vec<&str> = bind.split(':').collect();
+            if parts.len() == 2 {
+                config.bind_host = parts[0].to_string();
+                config.bind_port = parts[1].parse()
+                    .context("Invalid port in ARCHIPELAGO_BIND")?;
+            }
+        }
+
+        if let Ok(level) = std::env::var("ARCHIPELAGO_LOG_LEVEL") {
+            config.log_level = level;
+        }
+
+        // Dev mode configuration
+        if let Ok(dev_mode) = std::env::var("ARCHIPELAGO_DEV_MODE") {
+            config.dev_mode = dev_mode.parse().unwrap_or(false);
+        }
+
+        if let Ok(runtime) = std::env::var("ARCHIPELAGO_CONTAINER_RUNTIME") {
+            config.container_runtime = ContainerRuntime::from_str(&runtime);
+        }
+
+        if let Ok(offset) = std::env::var("ARCHIPELAGO_PORT_OFFSET") {
+            config.port_offset = offset.parse()
+                .context("Invalid port offset in ARCHIPELAGO_PORT_OFFSET")?;
+        }
+
+        if let Ok(sim) = std::env::var("ARCHIPELAGO_BITCOIN_SIMULATION") {
+            config.bitcoin_simulation = BitcoinSimulation::from_str(&sim);
+        }
+
+        if let Ok(dev_data_dir) = std::env::var("ARCHIPELAGO_DEV_DATA_DIR") {
+            config.dev_data_dir = PathBuf::from(dev_data_dir);
+        }
+
+        // Ensure data directory exists
+        fs::create_dir_all(&config.data_dir).await
+            .context("Failed to create data directory")?;
+
+        // Ensure dev data directory exists if in dev mode
+        if config.dev_mode {
+            fs::create_dir_all(&config.dev_data_dir).await
+                .context("Failed to create dev data directory")?;
+        }
+
+        Ok(config)
+    }
+}
+
+impl Default for Config {
+    fn default() -> Self {
+        Self {
+            data_dir: PathBuf::from("/var/lib/archipelago"),
+            bind_host: "127.0.0.1".to_string(),
+            bind_port: 5959,
+            log_level: "info".to_string(),
+            dev_mode: false,
+            container_runtime: ContainerRuntime::Auto,
+            port_offset: 10000,
+            bitcoin_simulation: BitcoinSimulation::Mock,
+            dev_data_dir: PathBuf::from("/tmp/archipelago-dev"),
+        }
+    }
+}

core/archipelago/src/container/data_manager.rs

@@ -0,0 +1,119 @@
+use anyhow::{Context, Result};
+use std::path::{Path, PathBuf};
+use tokio::fs;
+
+pub struct DevDataManager {
+    dev_data_dir: PathBuf,
+}
+
+impl DevDataManager {
+    pub fn new(dev_data_dir: PathBuf) -> Self {
+        Self { dev_data_dir }
+    }
+
+    /// Get the dev data directory for an app
+    pub fn get_app_data_dir(&self, app_id: &str) -> PathBuf {
+        self.dev_data_dir.join(app_id)
+    }
+
+    /// Create data directory for an app
+    pub async fn create_app_data_dir(&self, app_id: &str) -> Result<PathBuf> {
+        let app_dir = self.get_app_data_dir(app_id);
+        fs::create_dir_all(&app_dir)
+            .await
+            .with_context(|| format!("Failed to create app data directory: {:?}", app_dir))?;
+        Ok(app_dir)
+    }
+
+    /// Map a volume source path to dev path
+    pub fn map_volume_path(&self, app_id: &str, volume_source: &str) -> PathBuf {
+        // If volume source is already in dev_data_dir, use it as-is
+        if volume_source.starts_with(self.dev_data_dir.to_str().unwrap_or("")) {
+            PathBuf::from(volume_source)
+        } else {
+            // Map production path to dev path
+            // e.g., /var/lib/archipelago/bitcoin -> /tmp/archipelago-dev/bitcoin
+            let app_dir = self.get_app_data_dir(app_id);
+            
+            // Extract the relative path from the production path
+            if let Some(relative) = volume_source.strip_prefix("/var/lib/archipelago/") {
+                app_dir.join(relative)
+            } else if let Some(relative) = volume_source.strip_prefix("/var/lib/archipelago") {
+                app_dir.join(relative)
+            } else {
+                // If it doesn't match expected pattern, use app_id as base
+                app_dir.join(volume_source.trim_start_matches('/'))
+            }
+        }
+    }
+
+    /// Clean up app data directory
+    pub async fn cleanup_app_data(&self, app_id: &str) -> Result<()> {
+        let app_dir = self.get_app_data_dir(app_id);
+        if app_dir.exists() {
+            fs::remove_dir_all(&app_dir)
+                .await
+                .with_context(|| format!("Failed to remove app data directory: {:?}", app_dir))?;
+        }
+        Ok(())
+    }
+
+    /// Preserve app data (no-op for cleanup, used when removing container)
+    pub async fn preserve_app_data(&self, _app_id: &str) -> Result<()> {
+        // In dev mode, we might want to preserve data between container removals
+        // This is a no-op by default, but can be extended
+        Ok(())
+    }
+
+    /// Get all app data directories
+    pub async fn list_app_data_dirs(&self) -> Result<Vec<String>> {
+        if !self.dev_data_dir.exists() {
+            return Ok(vec![]);
+        }
+
+        let mut entries = fs::read_dir(&self.dev_data_dir)
+            .await
+            .with_context(|| format!("Failed to read dev data directory: {:?}", self.dev_data_dir))?;
+
+        let mut app_ids = Vec::new();
+        while let Some(entry) = entries.next_entry().await? {
+            if entry.file_type().await?.is_dir() {
+                if let Some(name) = entry.file_name().to_str() {
+                    app_ids.push(name.to_string());
+                }
+            }
+        }
+
+        Ok(app_ids)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::PathBuf;
+
+    #[tokio::test]
+    async fn test_map_volume_path() {
+        let temp_dir = std::env::temp_dir().join("test-archipelago");
+        let manager = DevDataManager::new(temp_dir.clone());
+        
+        let dev_path = manager.map_volume_path("bitcoin-core", "/var/lib/archipelago/bitcoin");
+        assert!(dev_path.to_string_lossy().contains("bitcoin-core"));
+        
+        // Cleanup
+        let _ = tokio::fs::remove_dir_all(&temp_dir).await;
+    }
+
+    #[tokio::test]
+    async fn test_create_app_data_dir() {
+        let temp_dir = std::env::temp_dir().join("test-archipelago-2");
+        let manager = DevDataManager::new(temp_dir.clone());
+
+        let app_dir = manager.create_app_data_dir("test-app").await.unwrap();
+        assert!(app_dir.exists());
+        
+        // Cleanup
+        let _ = tokio::fs::remove_dir_all(&temp_dir).await;
+    }
+}

core/archipelago/src/container/dev_orchestrator.rs

@@ -0,0 +1,262 @@
+use archipelago_container::{
+    AppManifest, BitcoinSimulator, BitcoinSimulationMode, ContainerRuntime as ContainerRuntimeTrait,
+    ContainerStatus, PortManager,
+};
+use anyhow::{Context, Result};
+use std::path::PathBuf;
+use std::sync::Arc;
+use tokio::sync::RwLock;
+
+use crate::config::{Config, ContainerRuntime, BitcoinSimulation};
+use crate::container::data_manager::DevDataManager;
+
+pub struct DevContainerOrchestrator {
+    runtime: Arc<dyn ContainerRuntimeTrait>,
+    port_manager: Arc<PortManager>,
+    bitcoin_simulator: Arc<BitcoinSimulator>,
+    data_manager: Arc<DevDataManager>,
+    config: Config,
+}
+
+impl DevContainerOrchestrator {
+    pub async fn new(config: Config) -> Result<Self> {
+        let user = std::env::var("USER").unwrap_or_else(|_| "archipelago".to_string());
+
+        // Create runtime based on config
+        let runtime: Arc<dyn ContainerRuntimeTrait> = match &config.container_runtime {
+            ContainerRuntime::Podman => {
+                Arc::new(archipelago_container::PodmanRuntime::new(user.clone()))
+            }
+            ContainerRuntime::Docker => {
+                Arc::new(archipelago_container::DockerRuntime::new(user.clone()))
+            }
+            ContainerRuntime::Auto => {
+                Arc::new(
+                    archipelago_container::AutoRuntime::new(user.clone())
+                        .await
+                        .context("Failed to create auto runtime")?,
+                )
+            }
+        };
+
+        let port_manager = Arc::new(PortManager::new(config.port_offset));
+        let bitcoin_simulator = Arc::new(BitcoinSimulator::new(
+            BitcoinSimulationMode::from(
+                match &config.bitcoin_simulation {
+                    BitcoinSimulation::Mock => "mock",
+                    BitcoinSimulation::Testnet => "testnet",
+                    BitcoinSimulation::Mainnet => "mainnet",
+                    BitcoinSimulation::None => "none",
+                }
+            ),
+        ));
+        let data_manager = Arc::new(DevDataManager::new(config.dev_data_dir.clone()));
+
+        Ok(Self {
+            runtime,
+            port_manager,
+            bitcoin_simulator,
+            data_manager,
+            config,
+        })
+    }
+
+    /// Install a container from a manifest
+    pub async fn install_container(
+        &self,
+        manifest: &AppManifest,
+        manifest_path: &str,
+    ) -> Result<String> {
+        let app_id = &manifest.app.id;
+        let container_name = format!("archipelago-{}-dev", app_id);
+
+        // Check dependencies
+        if self.config.dev_mode {
+            // In dev mode, check if Bitcoin dependency can be satisfied
+            for dep in &manifest.app.dependencies {
+                if let archipelago_container::Dependency::App {
+                    app_id: dep_id,
+                    version: _,
+                } = dep
+                {
+                    if dep_id == "bitcoin-core" {
+                        if !self.bitcoin_simulator.is_bitcoin_available() {
+                            return Err(anyhow::anyhow!(
+                                "Bitcoin Core dependency not satisfied (simulation: {:?})",
+                                self.bitcoin_simulator.mode()
+                            ));
+                        }
+                    }
+                }
+            }
+        }
+
+        // Allocate ports
+        let base_ports: Vec<u16> = manifest.app.ports.iter().map(|p| p.host).collect();
+        let _dev_ports = self
+            .port_manager
+            .allocate_ports(app_id, &base_ports)
+            .context("Failed to allocate ports")?;
+
+        // Create app data directory
+        self.data_manager
+            .create_app_data_dir(app_id)
+            .await
+            .context("Failed to create app data directory")?;
+
+        // Map volumes to dev paths
+        let mut dev_manifest = manifest.clone();
+        for volume in &mut dev_manifest.app.volumes {
+            let dev_path = self.data_manager.map_volume_path(app_id, &volume.source);
+            volume.source = dev_path.to_string_lossy().to_string();
+        }
+
+        // Pull image
+        self.runtime
+            .pull_image(
+                &manifest.app.container.image,
+                manifest.app.container.image_signature.as_deref(),
+            )
+            .await
+            .context("Failed to pull image")?;
+
+        // Create container with port offset
+        let port_offset = if self.config.dev_mode {
+            self.config.port_offset
+        } else {
+            0
+        };
+
+        self.runtime
+            .create_container(&dev_manifest, &container_name, port_offset)
+            .await
+            .context("Failed to create container")?;
+
+        Ok(container_name)
+    }
+
+    /// Start a container
+    pub async fn start_container(&self, app_id: &str) -> Result<()> {
+        let container_name = format!("archipelago-{}-dev", app_id);
+        self.runtime
+            .start_container(&container_name)
+            .await
+            .context("Failed to start container")?;
+        Ok(())
+    }
+
+    /// Stop a container
+    pub async fn stop_container(&self, app_id: &str) -> Result<()> {
+        let container_name = format!("archipelago-{}-dev", app_id);
+        self.runtime
+            .stop_container(&container_name)
+            .await
+            .context("Failed to stop container")?;
+        Ok(())
+    }
+
+    /// Remove a container
+    pub async fn remove_container(&self, app_id: &str, preserve_data: bool) -> Result<()> {
+        let container_name = format!("archipelago-{}-dev", app_id);
+
+        // Stop container first
+        let _ = self.runtime.stop_container(&container_name).await;
+
+        // Remove container
+        self.runtime
+            .remove_container(&container_name)
+            .await
+            .context("Failed to remove container")?;
+
+        // Release ports
+        let _ = self.port_manager.release_ports(app_id);
+
+        // Clean up or preserve data
+        if preserve_data {
+            self.data_manager.preserve_app_data(app_id).await?;
+        } else {
+            self.data_manager.cleanup_app_data(app_id).await?;
+        }
+
+        Ok(())
+    }
+
+    /// Get container status with dev port info
+    pub async fn get_container_status(&self, app_id: &str) -> Result<ContainerStatus> {
+        let container_name = format!("archipelago-{}-dev", app_id);
+        let mut status = self
+            .runtime
+            .get_container_status(&container_name)
+            .await
+            .context("Failed to get container status")?;
+
+        // Add dev port information
+        if let Some(ports) = self.port_manager.get_port_mapping(app_id) {
+            status.ports = ports.iter().map(|p| p.to_string()).collect();
+        }
+
+        Ok(status)
+    }
+
+    /// List all containers with dev info
+    pub async fn list_containers(&self) -> Result<Vec<ContainerStatus>> {
+        let containers = self
+            .runtime
+            .list_containers()
+            .await
+            .context("Failed to list containers")?;
+
+        // Filter to only archipelago containers and add port info
+        let mut result = Vec::new();
+        for container in containers {
+            if container.name.contains("archipelago-") {
+                // Extract app_id from container name
+                if let Some(app_id) = container.name.strip_prefix("archipelago-") {
+                    if let Some(app_id) = app_id.strip_suffix("-dev") {
+                        if let Some(ports) = self.port_manager.get_port_mapping(app_id) {
+                            let mut container_with_ports = container.clone();
+                            container_with_ports.ports = ports.iter().map(|p| p.to_string()).collect();
+                            result.push(container_with_ports);
+                        } else {
+                            result.push(container);
+                        }
+                    }
+                }
+            }
+        }
+
+        Ok(result)
+    }
+
+    /// Get container logs
+    pub async fn get_container_logs(&self, app_id: &str, lines: u32) -> Result<Vec<String>> {
+        let container_name = format!("archipelago-{}-dev", app_id);
+        self.runtime
+            .get_container_logs(&container_name, lines)
+            .await
+            .context("Failed to get container logs")
+    }
+
+    /// Get health status
+    pub async fn get_health_status(&self, app_id: &str) -> Result<String> {
+        let status = self.get_container_status(app_id).await?;
+        match status.state {
+            archipelago_container::ContainerState::Running => Ok("healthy".to_string()),
+            archipelago_container::ContainerState::Stopped
+            | archipelago_container::ContainerState::Exited => Ok("unhealthy".to_string()),
+            archipelago_container::ContainerState::Created => Ok("starting".to_string()),
+            archipelago_container::ContainerState::Paused => Ok("paused".to_string()),
+            archipelago_container::ContainerState::Unknown(_) => Ok("unknown".to_string()),
+        }
+    }
+
+    /// Get port mapping for an app
+    pub fn get_port_mapping(&self, app_id: &str) -> Option<Vec<u16>> {
+        self.port_manager.get_port_mapping(app_id)
+    }
+
+    /// Get Bitcoin simulator
+    pub fn bitcoin_simulator(&self) -> &Arc<BitcoinSimulator> {
+        &self.bitcoin_simulator
+    }
+}

core/archipelago/src/container/mod.rs

@@ -0,0 +1,5 @@
+pub mod data_manager;
+pub mod dev_orchestrator;
+
+pub use data_manager::DevDataManager;
+pub use dev_orchestrator::DevContainerOrchestrator;

core/archipelago/src/main.rs

@@ -0,0 +1,48 @@
+// Archipelago Bitcoin Node OS - Native Backend
+// Pure Archipelago implementation, no StartOS dependencies
+
+use anyhow::Result;
+use std::net::SocketAddr;
+use tracing::{info, error};
+
+mod api;
+mod auth;
+mod config;
+mod container;
+mod server;
+
+use config::Config;
+use server::Server;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    // Initialize tracing
+    tracing_subscriber::fmt()
+        .with_env_filter(
+            tracing_subscriber::EnvFilter::try_from_default_env()
+                .unwrap_or_else(|_| "archipelago=debug,info".into()),
+        )
+        .init();
+
+    info!("🚀 Starting Archipelago Bitcoin Node OS");
+
+    // Load configuration
+    let config = Config::load().await?;
+    info!("📁 Data directory: {}", config.data_dir.display());
+
+    // Create server
+    let server = Server::new(config.clone()).await?;
+
+    // Start server
+    let addr: SocketAddr = format!("{}:{}", config.bind_host, config.bind_port)
+        .parse()
+        .expect("Invalid bind address");
+
+    info!("🌐 Server listening on http://{}", addr);
+    info!("📡 RPC API: http://{}/rpc/v1", addr);
+    info!("🔌 WebSocket: ws://{}/ws", addr);
+
+    server.serve(addr).await?;
+
+    Ok(())
+}

core/archipelago/src/server.rs

@@ -0,0 +1,63 @@
+use crate::api::ApiHandler;
+use crate::config::Config;
+use anyhow::Result;
+use hyper_util::rt::TokioIo;
+use hyper_util::server::conn::auto::Builder as AutoBuilder;
+use std::net::SocketAddr;
+use std::sync::Arc;
+use tokio::net::TcpListener;
+use hyper::service::service_fn;
+use tracing::{error, info};
+
+pub struct Server {
+    config: Config,
+    api_handler: Arc<ApiHandler>,
+}
+
+impl Server {
+    pub async fn new(config: Config) -> Result<Self> {
+        let api_handler = Arc::new(ApiHandler::new(config.clone()).await?);
+
+        Ok(Self {
+            config,
+            api_handler,
+        })
+    }
+
+    pub async fn serve(&self, addr: SocketAddr) -> Result<()> {
+        let listener = TcpListener::bind(addr).await?;
+
+        loop {
+            let (stream, peer_addr) = match listener.accept().await {
+                Ok(conn) => conn,
+                Err(e) => {
+                    error!("Failed to accept connection: {}", e);
+                    continue;
+                }
+            };
+
+            let io = TokioIo::new(stream);
+            let handler = self.api_handler.clone();
+
+            tokio::spawn(async move {
+                let service = service_fn(move |req| {
+                    let handler = handler.clone();
+                    async move {
+                        handler.handle_request(req).await
+                            .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))
+                    }
+                });
+
+                let builder = AutoBuilder::new(
+                    hyper_util::rt::TokioExecutor::new()
+                );
+                if let Err(e) = builder
+                    .serve_connection(io, service)
+                    .await
+                {
+                    error!("Error serving connection from {}: {}", peer_addr, e);
+                }
+            });
+        }
+    }
+}

core/container-init/Cargo.toml

@@ -0,0 +1,32 @@
+[package]
+name = "container-init"
+version = "0.1.0"
+edition = "2021"
+
+[features]
+dev = []
+metal = []
+sound = []
+unstable = []
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+[dependencies]
+async-stream = "0.3"
+# cgroups-rs = "0.2"
+color-eyre = "0.6"
+futures = "0.3"
+serde = { version = "1", features = ["derive", "rc"] }
+serde_json = "1"
+helpers = { path = "../helpers" }
+imbl = "2"
+nix = { version = "0.27", features = ["process", "signal"] }
+tokio = { version = "1", features = ["full"] }
+tokio-stream = { version = "0.1", features = ["io-util", "sync", "net"] }
+tracing = "0.1"
+tracing-error = "0.2"
+tracing-futures = "0.2"
+tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+yajrc = { version = "*", git = "https://github.com/dr-bonez/yajrc.git", branch = "develop" }
+
+[target.'cfg(target_os = "linux")'.dependencies]
+procfs = "0.15"

core/container-init/src/lib.rs

@@ -0,0 +1,214 @@
+use nix::unistd::Pid;
+use serde::{Deserialize, Serialize, Serializer};
+use yajrc::RpcMethod;
+
+/// Know what the process is called
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct ProcessId(pub u32);
+impl From<ProcessId> for Pid {
+    fn from(pid: ProcessId) -> Self {
+        Pid::from_raw(pid.0 as i32)
+    }
+}
+impl From<Pid> for ProcessId {
+    fn from(pid: Pid) -> Self {
+        ProcessId(pid.as_raw() as u32)
+    }
+}
+impl From<i32> for ProcessId {
+    fn from(pid: i32) -> Self {
+        ProcessId(pid as u32)
+    }
+}
+
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct ProcessGroupId(pub u32);
+
+#[derive(Debug, Serialize, Deserialize, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
+#[serde(rename_all = "kebab-case")]
+pub enum OutputStrategy {
+    Inherit,
+    Collect,
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct RunCommand;
+impl Serialize for RunCommand {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct RunCommandParams {
+    pub gid: Option<ProcessGroupId>,
+    pub command: String,
+    pub args: Vec<String>,
+    pub output: OutputStrategy,
+}
+impl RpcMethod for RunCommand {
+    type Params = RunCommandParams;
+    type Response = ProcessId;
+    fn as_str<'a>(&'a self) -> &'a str {
+        "command"
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum LogLevel {
+    Trace(String),
+    Warn(String),
+    Error(String),
+    Info(String),
+    Debug(String),
+}
+impl LogLevel {
+    pub fn trace(&self) {
+        match self {
+            LogLevel::Trace(x) => tracing::trace!("{}", x),
+            LogLevel::Warn(x) => tracing::warn!("{}", x),
+            LogLevel::Error(x) => tracing::error!("{}", x),
+            LogLevel::Info(x) => tracing::info!("{}", x),
+            LogLevel::Debug(x) => tracing::debug!("{}", x),
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct Log;
+impl Serialize for Log {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct LogParams {
+    pub gid: Option<ProcessGroupId>,
+    pub level: LogLevel,
+}
+impl RpcMethod for Log {
+    type Params = LogParams;
+    type Response = ();
+    fn as_str<'a>(&'a self) -> &'a str {
+        "log"
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct ReadLineStdout;
+impl Serialize for ReadLineStdout {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ReadLineStdoutParams {
+    pub pid: ProcessId,
+}
+impl RpcMethod for ReadLineStdout {
+    type Params = ReadLineStdoutParams;
+    type Response = String;
+    fn as_str<'a>(&'a self) -> &'a str {
+        "read-line-stdout"
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct ReadLineStderr;
+impl Serialize for ReadLineStderr {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ReadLineStderrParams {
+    pub pid: ProcessId,
+}
+impl RpcMethod for ReadLineStderr {
+    type Params = ReadLineStderrParams;
+    type Response = String;
+    fn as_str<'a>(&'a self) -> &'a str {
+        "read-line-stderr"
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct Output;
+impl Serialize for Output {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct OutputParams {
+    pub pid: ProcessId,
+}
+impl RpcMethod for Output {
+    type Params = OutputParams;
+    type Response = String;
+    fn as_str<'a>(&'a self) -> &'a str {
+        "output"
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct SendSignal;
+impl Serialize for SendSignal {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SendSignalParams {
+    pub pid: ProcessId,
+    pub signal: u32,
+}
+impl RpcMethod for SendSignal {
+    type Params = SendSignalParams;
+    type Response = ();
+    fn as_str<'a>(&'a self) -> &'a str {
+        "signal"
+    }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct SignalGroup;
+impl Serialize for SignalGroup {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        Serialize::serialize(Self.as_str(), serializer)
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct SignalGroupParams {
+    pub gid: ProcessGroupId,
+    pub signal: u32,
+}
+impl RpcMethod for SignalGroup {
+    type Params = SignalGroupParams;
+    type Response = ();
+    fn as_str<'a>(&'a self) -> &'a str {
+        "signal-group"
+    }
+}

core/container-init/src/main.rs

@@ -0,0 +1,432 @@
+use std::collections::BTreeMap;
+use std::ops::DerefMut;
+use std::os::unix::process::ExitStatusExt;
+use std::process::Stdio;
+use std::sync::Arc;
+
+use container_init::{
+    LogParams, OutputParams, OutputStrategy, ProcessGroupId, ProcessId, RunCommandParams,
+    SendSignalParams, SignalGroupParams,
+};
+use futures::StreamExt;
+use helpers::NonDetachingJoinHandle;
+use nix::errno::Errno;
+use nix::sys::signal::Signal;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
+use tokio::process::{Child, Command};
+use tokio::select;
+use tokio::sync::{watch, Mutex};
+use yajrc::{Id, RpcError};
+
+/// Outputs embedded in the JSONRpc output of the executable.
+#[derive(Debug, Clone, Serialize)]
+#[serde(untagged)]
+#[allow(dead_code)]
+enum Output {
+    Command(ProcessId),
+    ReadLineStdout(String),
+    ReadLineStderr(String),
+    Output(String),
+    Log,
+    Signal,
+    SignalGroup,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(tag = "method", content = "params", rename_all = "kebab-case")]
+enum Input {
+    /// Run a new command, with the args
+    Command(RunCommandParams),
+    /// Want to log locall on the service rather than the eos
+    Log(LogParams),
+    // /// Get a line of stdout from the command
+    // ReadLineStdout(ReadLineStdoutParams),
+    // /// Get a line of stderr from the command
+    // ReadLineStderr(ReadLineStderrParams),
+    /// Get output of command
+    Output(OutputParams),
+    /// Send the sigterm to the process
+    Signal(SendSignalParams),
+    /// Signal a group of processes
+    SignalGroup(SignalGroupParams),
+}
+
+#[derive(Deserialize)]
+struct IncomingRpc {
+    id: Id,
+    #[serde(flatten)]
+    input: Input,
+}
+
+struct ChildInfo {
+    gid: Option<ProcessGroupId>,
+    child: Arc<Mutex<Option<Child>>>,
+    #[allow(dead_code)]
+    output: Option<InheritOutput>,
+}
+
+struct InheritOutput {
+    _thread: NonDetachingJoinHandle<()>,
+    #[allow(dead_code)]
+    stdout: watch::Receiver<String>,
+    #[allow(dead_code)]
+    stderr: watch::Receiver<String>,
+}
+
+struct HandlerMut {
+    processes: BTreeMap<ProcessId, ChildInfo>,
+    // groups: BTreeMap<ProcessGroupId, Cgroup>,
+}
+
+#[derive(Clone)]
+struct Handler {
+    children: Arc<Mutex<HandlerMut>>,
+}
+impl Handler {
+    fn new() -> Self {
+        Handler {
+            children: Arc::new(Mutex::new(HandlerMut {
+                processes: BTreeMap::new(),
+                // groups: BTreeMap::new(),
+            })),
+        }
+    }
+    async fn handle(&self, req: Input) -> Result<Output, RpcError> {
+        Ok(match req {
+            Input::Command(RunCommandParams {
+                gid,
+                command,
+                args,
+                output,
+            }) => Output::Command(self.command(gid, command, args, output).await?),
+            // Input::ReadLineStdout(ReadLineStdoutParams { pid }) => {
+            //     Output::ReadLineStdout(self.read_line_stdout(pid).await?)
+            // }
+            // Input::ReadLineStderr(ReadLineStderrParams { pid }) => {
+            //     Output::ReadLineStderr(self.read_line_stderr(pid).await?)
+            // }
+            Input::Log(LogParams { gid: _, level }) => {
+                level.trace();
+                Output::Log
+            }
+            Input::Output(OutputParams { pid }) => Output::Output(self.output(pid).await?),
+            Input::Signal(SendSignalParams { pid, signal }) => {
+                self.signal(pid, signal).await?;
+                Output::Signal
+            }
+            Input::SignalGroup(SignalGroupParams { gid, signal }) => {
+                self.signal_group(gid, signal).await?;
+                Output::SignalGroup
+            }
+        })
+    }
+
+    async fn command(
+        &self,
+        gid: Option<ProcessGroupId>,
+        command: String,
+        args: Vec<String>,
+        output: OutputStrategy,
+    ) -> Result<ProcessId, RpcError> {
+        let mut cmd = Command::new(command);
+        cmd.args(args);
+        cmd.kill_on_drop(true);
+        cmd.stdout(Stdio::piped());
+        cmd.stderr(Stdio::piped());
+        let mut child = cmd.spawn().map_err(|e| {
+            let mut err = yajrc::INTERNAL_ERROR.clone();
+            err.data = Some(json!(e.to_string()));
+            err
+        })?;
+        let pid = ProcessId(child.id().ok_or_else(|| {
+            let mut err = yajrc::INTERNAL_ERROR.clone();
+            err.data = Some(json!("Child has no pid"));
+            err
+        })?);
+        let output = match output {
+            OutputStrategy::Inherit => {
+                let (stdout_send, stdout) = watch::channel(String::new());
+                let (stderr_send, stderr) = watch::channel(String::new());
+                if let (Some(child_stdout), Some(child_stderr)) =
+                    (child.stdout.take(), child.stderr.take())
+                {
+                    Some(InheritOutput {
+                        _thread: tokio::spawn(async move {
+                            tokio::join!(
+                                async {
+                                    if let Err(e) = async {
+                                        let mut lines = BufReader::new(child_stdout).lines();
+                                        while let Some(line) = lines.next_line().await? {
+                                            tracing::info!("({}): {}", pid.0, line);
+                                            let _ = stdout_send.send(line);
+                                        }
+                                        Ok::<_, std::io::Error>(())
+                                    }
+                                    .await
+                                    {
+                                        tracing::error!(
+                                            "Error reading stdout of pid {}: {}",
+                                            pid.0,
+                                            e
+                                        );
+                                    }
+                                },
+                                async {
+                                    if let Err(e) = async {
+                                        let mut lines = BufReader::new(child_stderr).lines();
+                                        while let Some(line) = lines.next_line().await? {
+                                            tracing::warn!("({}): {}", pid.0, line);
+                                            let _ = stderr_send.send(line);
+                                        }
+                                        Ok::<_, std::io::Error>(())
+                                    }
+                                    .await
+                                    {
+                                        tracing::error!(
+                                            "Error reading stdout of pid {}: {}",
+                                            pid.0,
+                                            e
+                                        );
+                                    }
+                                }
+                            );
+                        })
+                        .into(),
+                        stdout,
+                        stderr,
+                    })
+                } else {
+                    None
+                }
+            }
+            OutputStrategy::Collect => None,
+        };
+        self.children.lock().await.processes.insert(
+            pid,
+            ChildInfo {
+                gid,
+                child: Arc::new(Mutex::new(Some(child))),
+                output,
+            },
+        );
+        Ok(pid)
+    }
+
+    async fn output(&self, pid: ProcessId) -> Result<String, RpcError> {
+        let not_found = || {
+            let mut err = yajrc::INTERNAL_ERROR.clone();
+            err.data = Some(json!(format!("Child with pid {} not found", pid.0)));
+            err
+        };
+        let mut child = {
+            self.children
+                .lock()
+                .await
+                .processes
+                .get(&pid)
+                .ok_or_else(not_found)?
+                .child
+                .clone()
+        }
+        .lock_owned()
+        .await;
+        if let Some(child) = child.take() {
+            let output = child.wait_with_output().await?;
+            if output.status.success() {
+                Ok(String::from_utf8(output.stdout).map_err(|_| yajrc::PARSE_ERROR)?)
+            } else {
+                Err(RpcError {
+                    code: output
+                        .status
+                        .code()
+                        .or_else(|| output.status.signal().map(|s| 128 + s))
+                        .unwrap_or(0),
+                    message: "Command failed".into(),
+                    data: Some(json!(String::from_utf8(if output.stderr.is_empty() {
+                        output.stdout
+                    } else {
+                        output.stderr
+                    })
+                    .map_err(|_| yajrc::PARSE_ERROR)?)),
+                })
+            }
+        } else {
+            Err(not_found())
+        }
+    }
+
+    async fn signal(&self, pid: ProcessId, signal: u32) -> Result<(), RpcError> {
+        let not_found = || {
+            let mut err = yajrc::INTERNAL_ERROR.clone();
+            err.data = Some(json!(format!("Child with pid {} not found", pid.0)));
+            err
+        };
+
+        Self::killall(pid, Signal::try_from(signal as i32)?)?;
+
+        if signal == 9 {
+            self.children
+                .lock()
+                .await
+                .processes
+                .remove(&pid)
+                .ok_or_else(not_found)?;
+        }
+        Ok(())
+    }
+
+    async fn signal_group(&self, gid: ProcessGroupId, signal: u32) -> Result<(), RpcError> {
+        let mut to_kill = Vec::new();
+        {
+            let mut children_ref = self.children.lock().await;
+            let children = std::mem::take(&mut children_ref.deref_mut().processes);
+            for (pid, child_info) in children {
+                if child_info.gid == Some(gid) {
+                    to_kill.push(pid);
+                } else {
+                    children_ref.processes.insert(pid, child_info);
+                }
+            }
+        }
+        for pid in to_kill {
+            tracing::info!("Killing pid {}", pid.0);
+            Self::killall(pid, Signal::try_from(signal as i32)?)?;
+        }
+
+        Ok(())
+    }
+
+    fn killall(pid: ProcessId, signal: Signal) -> Result<(), RpcError> {
+        for proc in procfs::process::all_processes()? {
+            let stat = proc?.stat()?;
+            if ProcessId::from(stat.ppid) == pid {
+                Self::killall(stat.pid.into(), signal)?;
+            }
+        }
+        if let Err(e) = nix::sys::signal::kill(pid.into(), Some(signal)) {
+            if e != Errno::ESRCH {
+                tracing::error!("Failed to kill pid {}: {}", pid.0, e);
+            }
+        }
+        Ok(())
+    }
+
+    async fn graceful_exit(self) {
+        let kill_all = futures::stream::iter(
+            std::mem::take(&mut self.children.lock().await.deref_mut().processes).into_iter(),
+        )
+        .for_each_concurrent(None, |(pid, child)| async move {
+            let _ = Self::killall(pid, Signal::SIGTERM);
+            if let Some(child) = child.child.lock().await.take() {
+                let _ = child.wait_with_output().await;
+            }
+        });
+        kill_all.await
+    }
+}
+
+#[tokio::main]
+async fn main() {
+    use tokio::signal::unix::{signal, SignalKind};
+    let mut sigint = signal(SignalKind::interrupt()).unwrap();
+    let mut sigterm = signal(SignalKind::terminate()).unwrap();
+    let mut sigquit = signal(SignalKind::quit()).unwrap();
+    let mut sighangup = signal(SignalKind::hangup()).unwrap();
+
+    use tracing_error::ErrorLayer;
+    use tracing_subscriber::prelude::*;
+    use tracing_subscriber::{fmt, EnvFilter};
+
+    let filter_layer = EnvFilter::new("container_init=debug");
+    let fmt_layer = fmt::layer().with_target(true);
+
+    tracing_subscriber::registry()
+        .with(filter_layer)
+        .with(fmt_layer)
+        .with(ErrorLayer::default())
+        .init();
+    color_eyre::install().unwrap();
+
+    let handler = Handler::new();
+    let handler_thread = async {
+        let listener = tokio::net::UnixListener::bind("/start9/sockets/rpc.sock")?;
+        loop {
+            let (stream, _) = listener.accept().await?;
+            let (r, w) = stream.into_split();
+            let mut lines = BufReader::new(r).lines();
+            let handler = handler.clone();
+            tokio::spawn(async move {
+                let w = Arc::new(Mutex::new(w));
+                while let Some(line) = lines.next_line().await.transpose() {
+                    let handler = handler.clone();
+                    let w = w.clone();
+                    tokio::spawn(async move {
+                        if let Err(e) = async {
+                            let req = serde_json::from_str::<IncomingRpc>(&line?)?;
+                            match handler.handle(req.input).await {
+                                Ok(output) => {
+                                    if  w.lock().await.write_all(
+                                        format!("{}\n", json!({ "id": req.id, "jsonrpc": "2.0", "result": output }))
+                                            .as_bytes(),
+                                    )
+                                    .await.is_err() {
+                                        tracing::error!("Error sending to {id:?}", id = req.id);
+                                    }
+                                }
+                                Err(e) =>
+                                if  w
+                                    .lock()
+                                    .await
+                                    .write_all(
+                                        format!("{}\n", json!({ "id": req.id, "jsonrpc": "2.0", "error": e }))
+                                            .as_bytes(),
+                                    )
+                                    .await.is_err() {
+
+                                        tracing::error!("Handle + Error sending to {id:?}", id = req.id);
+                                    },
+                            }
+                            Ok::<_, color_eyre::Report>(())
+                        }
+                        .await
+                        {
+                            tracing::error!("Error parsing RPC request: {}", e);
+                            tracing::debug!("{:?}", e);
+                        }
+                    });
+                }
+                Ok::<_, std::io::Error>(())
+            });
+        }
+        #[allow(unreachable_code)]
+        Ok::<_, std::io::Error>(())
+    };
+
+    select! {
+        res = handler_thread => {
+            match res {
+                Ok(()) => tracing::debug!("Done with inputs/outputs"),
+                Err(e) => {
+                    tracing::error!("Error reading RPC input: {}", e);
+                    tracing::debug!("{:?}", e);
+                }
+            }
+        },
+        _ = sigint.recv() => {
+            tracing::debug!("SIGINT");
+        },
+        _ = sigterm.recv() => {
+            tracing::debug!("SIGTERM");
+        },
+        _ = sigquit.recv() => {
+            tracing::debug!("SIGQUIT");
+        },
+        _ = sighangup.recv() => {
+            tracing::debug!("SIGHUP");
+        }
+    }
+    handler.graceful_exit().await;
+    ::std::process::exit(0)
+}

core/container/src/bitcoin_simulator.rs

@@ -0,0 +1,234 @@
+use anyhow::{Context, Result};
+use serde_json::{json, Value};
+use std::sync::Arc;
+use tokio::sync::RwLock;
+
+#[derive(Debug, Clone)]
+pub enum BitcoinSimulationMode {
+    Mock,
+    Testnet,
+    Mainnet,
+    None,
+}
+
+pub struct BitcoinSimulator {
+    mode: BitcoinSimulationMode,
+    rpc_url: Option<String>,
+    mock_blockchain_info: Arc<RwLock<Value>>,
+}
+
+impl BitcoinSimulator {
+    pub fn new(mode: BitcoinSimulationMode) -> Self {
+        let mock_blockchain_info = json!({
+            "chain": "main",
+            "blocks": 800000,
+            "headers": 800000,
+            "bestblockhash": "0000000000000000000123456789abcdef0123456789abcdef0123456789abcdef",
+            "difficulty": 50000000000.0,
+            "mediantime": 1700000000,
+            "verificationprogress": 1.0,
+            "initialblockdownload": false,
+            "chainwork": "0000000000000000000000000000000000000000000000000000000000000000",
+            "size_on_disk": 500000000000i64,
+            "pruned": false,
+            "softforks": {},
+            "warnings": ""
+        });
+
+        let rpc_url = match mode {
+            BitcoinSimulationMode::Mock => None,
+            BitcoinSimulationMode::Testnet => Some("http://localhost:18332".to_string()),
+            BitcoinSimulationMode::Mainnet => Some("http://localhost:8332".to_string()),
+            BitcoinSimulationMode::None => None,
+        };
+
+        Self {
+            mode,
+            rpc_url,
+            mock_blockchain_info: Arc::new(RwLock::new(mock_blockchain_info)),
+        }
+    }
+
+    pub fn is_bitcoin_available(&self) -> bool {
+        match self.mode {
+            BitcoinSimulationMode::Mock => true,
+            BitcoinSimulationMode::Testnet | BitcoinSimulationMode::Mainnet => {
+                // In real mode, we'd check if the container is running
+                // For now, assume it's available if we have an RPC URL
+                self.rpc_url.is_some()
+            }
+            BitcoinSimulationMode::None => false,
+        }
+    }
+
+    pub fn get_bitcoin_rpc_url(&self) -> Option<String> {
+        self.rpc_url.clone()
+    }
+
+    pub async fn simulate_rpc_call(&self, method: &str, params: &[Value]) -> Result<Value> {
+        match self.mode {
+            BitcoinSimulationMode::Mock => {
+                self.mock_rpc_call(method, params).await
+            }
+            BitcoinSimulationMode::Testnet | BitcoinSimulationMode::Mainnet => {
+                // Make actual RPC call to Bitcoin node
+                self.real_rpc_call(method, params).await
+            }
+            BitcoinSimulationMode::None => {
+                Err(anyhow::anyhow!("Bitcoin simulation is disabled"))
+            }
+        }
+    }
+
+    async fn mock_rpc_call(&self, method: &str, _params: &[Value]) -> Result<Value> {
+        match method {
+            "getblockchaininfo" => {
+                let info = self.mock_blockchain_info.read().await;
+                Ok(info.clone())
+            }
+            "getnetworkinfo" => {
+                Ok(json!({
+                    "version": 260000,
+                    "subversion": "/Bitcoin Core:26.0.0/",
+                    "protocolversion": 70016,
+                    "localservices": "000000000000040d",
+                    "localservicesnames": ["NETWORK", "WITNESS", "NETWORK_LIMITED"],
+                    "connections": 8,
+                    "connections_in": 4,
+                    "connections_out": 4,
+                    "networkactive": true,
+                    "networks": [],
+                    "relayfee": 0.00001000,
+                    "incrementalfee": 0.00001000,
+                    "localaddresses": [],
+                    "warnings": ""
+                }))
+            }
+            "getwalletinfo" => {
+                Ok(json!({
+                    "walletname": "wallet.dat",
+                    "walletversion": 169900,
+                    "balance": 0.0,
+                    "unconfirmed_balance": 0.0,
+                    "immature_balance": 0.0,
+                    "txcount": 0,
+                    "keypoololdest": 1700000000,
+                    "keypoolsize": 1000,
+                    "keypoolsize_hd_internal": 1000,
+                    "paytxfee": 0.00000000,
+                    "hdseedid": "0000000000000000000000000000000000000000",
+                    "private_keys_enabled": true,
+                    "avoid_reuse": false,
+                    "scanning": false
+                }))
+            }
+            "getblockcount" => {
+                Ok(json!(800000))
+            }
+            "getblockhash" => {
+                Ok(json!("0000000000000000000123456789abcdef0123456789abcdef0123456789abcdef"))
+            }
+            "getmempoolinfo" => {
+                Ok(json!({
+                    "loaded": true,
+                    "size": 100,
+                    "bytes": 100000,
+                    "usage": 200000,
+                    "total_fee": 0.00001000,
+                    "maxmempool": 300000000,
+                    "mempoolminfee": 0.00001000,
+                    "minrelaytxfee": 0.00001000
+                }))
+            }
+            "getpeerinfo" => {
+                Ok(json!([]))
+            }
+            "getrawmempool" => {
+                Ok(json!([]))
+            }
+            "estimatesmartfee" => {
+                Ok(json!({
+                    "feerate": 0.00001000,
+                    "blocks": 6
+                }))
+            }
+            _ => {
+                // Default response for unknown methods
+                Ok(json!(null))
+            }
+        }
+    }
+
+    async fn real_rpc_call(&self, method: &str, params: &[Value]) -> Result<Value> {
+        let url = self.rpc_url.as_ref()
+            .ok_or_else(|| anyhow::anyhow!("No RPC URL configured"))?;
+
+        let client = reqwest::Client::new();
+        let request_body = json!({
+            "jsonrpc": "2.0",
+            "id": 1,
+            "method": method,
+            "params": params
+        });
+
+        // TODO: Get RPC credentials from config/secrets
+        let response = client
+            .post(url)
+            .json(&request_body)
+            .send()
+            .await
+            .context("Failed to send RPC request")?;
+
+        let response_json: Value = response
+            .json::<Value>()
+            .await
+            .context("Failed to parse RPC response")?;
+
+        if let Some(error) = response_json.get("error") {
+            return Err(anyhow::anyhow!("Bitcoin RPC error: {}", error));
+        }
+
+        Ok(response_json.get("result")
+            .cloned()
+            .unwrap_or(Value::Null))
+    }
+
+    pub fn mode(&self) -> &BitcoinSimulationMode {
+        &self.mode
+    }
+}
+
+impl From<&str> for BitcoinSimulationMode {
+    fn from(s: &str) -> Self {
+        match s.to_lowercase().as_str() {
+            "mock" => BitcoinSimulationMode::Mock,
+            "testnet" => BitcoinSimulationMode::Testnet,
+            "mainnet" => BitcoinSimulationMode::Mainnet,
+            "none" | _ => BitcoinSimulationMode::None,
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[tokio::test]
+    async fn test_mock_bitcoin_available() {
+        let simulator = BitcoinSimulator::new(BitcoinSimulationMode::Mock);
+        assert!(simulator.is_bitcoin_available());
+    }
+
+    #[tokio::test]
+    async fn test_mock_getblockchaininfo() {
+        let simulator = BitcoinSimulator::new(BitcoinSimulationMode::Mock);
+        let result = simulator.simulate_rpc_call("getblockchaininfo", &[]).await.unwrap();
+        assert!(result.get("blocks").is_some());
+    }
+
+    #[tokio::test]
+    async fn test_none_bitcoin_not_available() {
+        let simulator = BitcoinSimulator::new(BitcoinSimulationMode::None);
+        assert!(!simulator.is_bitcoin_available());
+    }
+}

core/container/src/lib.rs

@@ -2,8 +2,14 @@ pub mod manifest;
 pub mod podman_client;
 pub mod dependency_resolver;
 pub mod health_monitor;
+pub mod runtime;
+pub mod port_manager;
+pub mod bitcoin_simulator;
 
 pub use manifest::{AppManifest, Dependency, ResourceLimits, SecurityPolicy, HealthCheck};
-pub use podman_client::PodmanClient;
+pub use podman_client::{PodmanClient, ContainerStatus, ContainerState};
 pub use dependency_resolver::DependencyResolver;
 pub use health_monitor::HealthMonitor;
+pub use runtime::{ContainerRuntime, PodmanRuntime, DockerRuntime, AutoRuntime};
+pub use port_manager::{PortManager, PortError};
+pub use bitcoin_simulator::{BitcoinSimulator, BitcoinSimulationMode};

core/container/src/port_manager.rs

@@ -0,0 +1,151 @@
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+use thiserror::Error;
+
+#[derive(Debug, Error)]
+pub enum PortError {
+    #[error("Port {0} is already allocated to app {1}")]
+    PortConflict(u16, String),
+    #[error("App {0} has no allocated ports")]
+    NoPortsAllocated(String),
+}
+
+pub struct PortManager {
+    allocations: Arc<RwLock<HashMap<String, Vec<u16>>>>,
+    port_to_app: Arc<RwLock<HashMap<u16, String>>>,
+    port_offset: u16,
+}
+
+impl PortManager {
+    pub fn new(port_offset: u16) -> Self {
+        Self {
+            allocations: Arc::new(RwLock::new(HashMap::new())),
+            port_to_app: Arc::new(RwLock::new(HashMap::new())),
+            port_offset,
+        }
+    }
+
+    /// Allocate ports for an app, applying the port offset
+    pub fn allocate_ports(&self, app_id: &str, base_ports: &[u16]) -> Result<Vec<u16>, PortError> {
+        let mut allocations = self.allocations.write().unwrap();
+        let mut port_to_app = self.port_to_app.write().unwrap();
+        let mut allocated_ports = Vec::new();
+
+        // Check for conflicts and allocate ports
+        for &base_port in base_ports {
+            let dev_port = base_port + self.port_offset;
+
+            // Check if port is already allocated
+            if let Some(existing_app) = port_to_app.get(&dev_port) {
+                if existing_app != app_id {
+                    return Err(PortError::PortConflict(dev_port, existing_app.clone()));
+                }
+            }
+
+            allocated_ports.push(dev_port);
+            port_to_app.insert(dev_port, app_id.to_string());
+        }
+
+        // Store allocation for this app
+        allocations.insert(app_id.to_string(), allocated_ports.clone());
+
+        Ok(allocated_ports)
+    }
+
+    /// Get allocated ports for an app
+    pub fn get_port_mapping(&self, app_id: &str) -> Option<Vec<u16>> {
+        let allocations = self.allocations.read().unwrap();
+        allocations.get(app_id).cloned()
+    }
+
+    /// Get the dev port for a specific base port of an app
+    pub fn get_dev_port(&self, app_id: &str, base_port: u16) -> Option<u16> {
+        self.get_port_mapping(app_id)
+            .and_then(|ports| {
+                // Find the port that corresponds to this base port
+                ports.iter().find(|&&p| p == base_port + self.port_offset).copied()
+            })
+    }
+
+    /// Release all ports allocated to an app
+    pub fn release_ports(&self, app_id: &str) -> Result<(), PortError> {
+        let mut allocations = self.allocations.write().unwrap();
+        let mut port_to_app = self.port_to_app.write().unwrap();
+
+        if let Some(ports) = allocations.remove(app_id) {
+            for port in ports {
+                port_to_app.remove(&port);
+            }
+            Ok(())
+        } else {
+            Err(PortError::NoPortsAllocated(app_id.to_string()))
+        }
+    }
+
+    /// Check if a port is available
+    pub fn is_port_available(&self, base_port: u16) -> bool {
+        let dev_port = base_port + self.port_offset;
+        let port_to_app = self.port_to_app.read().unwrap();
+        !port_to_app.contains_key(&dev_port)
+    }
+
+    /// Get all allocated ports
+    pub fn get_all_allocations(&self) -> HashMap<String, Vec<u16>> {
+        let allocations = self.allocations.read().unwrap();
+        allocations.clone()
+    }
+
+    /// Get port offset
+    pub fn port_offset(&self) -> u16 {
+        self.port_offset
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_port_allocation() {
+        let manager = PortManager::new(10000);
+        
+        let ports = manager.allocate_ports("app1", &[8332, 8333]).unwrap();
+        assert_eq!(ports, vec![18332, 18333]);
+        
+        let mapping = manager.get_port_mapping("app1").unwrap();
+        assert_eq!(mapping, vec![18332, 18333]);
+    }
+
+    #[test]
+    fn test_port_conflict() {
+        let manager = PortManager::new(10000);
+        
+        manager.allocate_ports("app1", &[8332]).unwrap();
+        
+        // Try to allocate the same port to another app
+        let result = manager.allocate_ports("app2", &[8332]);
+        assert!(result.is_err());
+    }
+
+    #[test]
+    fn test_port_release() {
+        let manager = PortManager::new(10000);
+        
+        manager.allocate_ports("app1", &[8332]).unwrap();
+        manager.release_ports("app1").unwrap();
+        
+        // Port should now be available
+        assert!(manager.is_port_available(8332));
+    }
+
+    #[test]
+    fn test_get_dev_port() {
+        let manager = PortManager::new(10000);
+        
+        manager.allocate_ports("app1", &[8332, 8333]).unwrap();
+        
+        assert_eq!(manager.get_dev_port("app1", 8332), Some(18332));
+        assert_eq!(manager.get_dev_port("app1", 8333), Some(18333));
+        assert_eq!(manager.get_dev_port("app1", 9999), None);
+    }
+}

core/container/src/runtime.rs

@@ -0,0 +1,434 @@
+use crate::manifest::AppManifest;
+use crate::podman_client::{ContainerStatus, ContainerState, PodmanClient};
+use anyhow::{Context, Result};
+use async_trait::async_trait;
+use std::process::Command;
+use tokio::process::Command as TokioCommand;
+
+#[async_trait]
+pub trait ContainerRuntime: Send + Sync {
+    async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()>;
+    async fn create_container(
+        &self,
+        manifest: &AppManifest,
+        name: &str,
+        port_offset: u16,
+    ) -> Result<String>;
+    async fn start_container(&self, name: &str) -> Result<()>;
+    async fn stop_container(&self, name: &str) -> Result<()>;
+    async fn remove_container(&self, name: &str) -> Result<()>;
+    async fn get_container_status(&self, name: &str) -> Result<ContainerStatus>;
+    async fn get_container_logs(&self, name: &str, lines: u32) -> Result<Vec<String>>;
+    async fn list_containers(&self) -> Result<Vec<ContainerStatus>>;
+}
+
+pub struct PodmanRuntime {
+    client: PodmanClient,
+}
+
+impl PodmanRuntime {
+    pub fn new(user: String) -> Self {
+        Self {
+            client: PodmanClient::new(user),
+        }
+    }
+}
+
+#[async_trait]
+impl ContainerRuntime for PodmanRuntime {
+    async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> {
+        self.client.pull_image(image, signature).await
+    }
+
+    async fn create_container(
+        &self,
+        manifest: &AppManifest,
+        name: &str,
+        port_offset: u16,
+    ) -> Result<String> {
+        // Apply port offset to manifest ports
+        let mut dev_manifest = manifest.clone();
+        for port in &mut dev_manifest.app.ports {
+            port.host = port.host + port_offset;
+        }
+
+        // PodmanClient doesn't take port_offset, so we use the modified manifest
+        self.client.create_container(&dev_manifest, name).await
+    }
+
+    async fn start_container(&self, name: &str) -> Result<()> {
+        self.client.start_container(name).await
+    }
+
+    async fn stop_container(&self, name: &str) -> Result<()> {
+        self.client.stop_container(name).await
+    }
+
+    async fn remove_container(&self, name: &str) -> Result<()> {
+        self.client.remove_container(name).await
+    }
+
+    async fn get_container_status(&self, name: &str) -> Result<ContainerStatus> {
+        self.client.get_container_status(name).await
+    }
+
+    async fn get_container_logs(&self, name: &str, lines: u32) -> Result<Vec<String>> {
+        self.client.get_container_logs(name, lines).await
+    }
+
+    async fn list_containers(&self) -> Result<Vec<ContainerStatus>> {
+        self.client.list_containers().await
+    }
+}
+
+pub struct DockerRuntime {
+    user: String,
+}
+
+impl DockerRuntime {
+    pub fn new(user: String) -> Self {
+        Self { user }
+    }
+
+    fn docker_async(&self) -> TokioCommand {
+        let mut cmd = TokioCommand::new("docker");
+        cmd.env("HOME", format!("/home/{}", self.user));
+        cmd
+    }
+
+    #[allow(dead_code)]
+    fn docker_command(&self) -> Command {
+        let mut cmd = Command::new("docker");
+        cmd.env("HOME", format!("/home/{}", self.user));
+        cmd
+    }
+}
+
+#[async_trait]
+impl ContainerRuntime for DockerRuntime {
+    async fn pull_image(&self, image: &str, _signature: Option<&str>) -> Result<()> {
+        let mut cmd = self.docker_async();
+        cmd.arg("pull").arg(image);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to execute docker pull")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to pull image: {}", stderr));
+        }
+
+        Ok(())
+    }
+
+    async fn create_container(
+        &self,
+        manifest: &AppManifest,
+        name: &str,
+        port_offset: u16,
+    ) -> Result<String> {
+        let mut cmd = self.docker_async();
+        cmd.arg("create");
+
+        cmd.arg("--name").arg(name);
+
+        if manifest.app.security.readonly_root {
+            cmd.arg("--read-only");
+        }
+
+        match manifest.app.security.network_policy.as_str() {
+            "host" => {
+                cmd.arg("--network").arg("host");
+            }
+            "isolated" => {
+                // Docker uses bridge network by default
+            }
+            _ => {
+                cmd.arg("--network").arg(&manifest.app.security.network_policy);
+            }
+        }
+
+        // Port mappings with offset
+        for port in &manifest.app.ports {
+            let host_port = port.host + port_offset;
+            cmd.arg("-p").arg(format!("{}:{}", host_port, port.container));
+        }
+
+        // Volumes
+        for volume in &manifest.app.volumes {
+            let mut mount = format!("{}:{}", volume.source, volume.target);
+            if !volume.options.is_empty() {
+                mount.push_str(&format!(":{}", volume.options.join(",")));
+            }
+            cmd.arg("-v").arg(mount);
+        }
+
+        // Devices
+        for device in &manifest.app.devices {
+            cmd.arg("--device").arg(device);
+        }
+
+        // Environment variables
+        for env in &manifest.app.environment {
+            cmd.arg("-e").arg(env);
+        }
+
+        // Resource limits
+        if let Some(cpu) = manifest.app.resources.cpu_limit {
+            cmd.arg("--cpus").arg(cpu.to_string());
+        }
+
+        if let Some(memory) = &manifest.app.resources.memory_limit {
+            cmd.arg("--memory").arg(memory);
+        }
+
+        // Capabilities
+        cmd.arg("--cap-drop").arg("ALL");
+        for cap in &manifest.app.security.capabilities {
+            cmd.arg("--cap-add").arg(cap);
+        }
+
+        cmd.arg(&manifest.app.container.image);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to create container")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to create container: {}", stderr));
+        }
+
+        let container_id = String::from_utf8_lossy(&output.stdout)
+            .trim()
+            .to_string();
+
+        Ok(container_id)
+    }
+
+    async fn start_container(&self, name: &str) -> Result<()> {
+        let mut cmd = self.docker_async();
+        cmd.arg("start").arg(name);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to start container")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to start container: {}", stderr));
+        }
+
+        Ok(())
+    }
+
+    async fn stop_container(&self, name: &str) -> Result<()> {
+        let mut cmd = self.docker_async();
+        cmd.arg("stop").arg(name);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to stop container")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to stop container: {}", stderr));
+        }
+
+        Ok(())
+    }
+
+    async fn remove_container(&self, name: &str) -> Result<()> {
+        let mut cmd = self.docker_async();
+        cmd.arg("rm").arg("-f").arg(name);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to remove container")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to remove container: {}", stderr));
+        }
+
+        Ok(())
+    }
+
+    async fn get_container_status(&self, name: &str) -> Result<ContainerStatus> {
+        let mut cmd = self.docker_async();
+        cmd.arg("inspect")
+            .arg("--format")
+            .arg("{{.Id}}|{{.Name}}|{{.State.Status}}|{{.Config.Image}}|{{.Created}}|{{.NetworkSettings.Ports}}")
+            .arg(name);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to inspect container")?;
+
+        if !output.status.success() {
+            return Err(anyhow::anyhow!("Container not found: {}", name));
+        }
+
+        let info = String::from_utf8_lossy(&output.stdout);
+        let parts: Vec<&str> = info.trim().split('|').collect();
+
+        if parts.len() < 5 {
+            return Err(anyhow::anyhow!("Invalid container inspect output"));
+        }
+
+        Ok(ContainerStatus {
+            id: parts[0].to_string(),
+            name: parts[1].to_string(),
+            state: crate::podman_client::ContainerState::from(parts[2]),
+            image: parts[3].to_string(),
+            created: parts[4].to_string(),
+            ports: vec![],
+        })
+    }
+
+    async fn get_container_logs(&self, name: &str, lines: u32) -> Result<Vec<String>> {
+        let mut cmd = self.docker_async();
+        cmd.arg("logs")
+            .arg("--tail")
+            .arg(lines.to_string())
+            .arg(name);
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to get container logs")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to get logs: {}", stderr));
+        }
+
+        let logs = String::from_utf8_lossy(&output.stdout);
+        Ok(logs.lines().map(|s| s.to_string()).collect())
+    }
+
+    async fn list_containers(&self) -> Result<Vec<ContainerStatus>> {
+        let mut cmd = self.docker_async();
+        cmd.arg("ps")
+            .arg("-a")
+            .arg("--format")
+            .arg("json");
+
+        let output = cmd
+            .output()
+            .await
+            .context("Failed to list containers")?;
+
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            return Err(anyhow::anyhow!("Failed to list containers: {}", stderr));
+        }
+
+        let json = String::from_utf8_lossy(&output.stdout);
+        let containers: Vec<serde_json::Value> = serde_json::from_str(&json)
+            .context("Failed to parse container list")?;
+
+        let mut result = Vec::new();
+        for container in containers {
+            result.push(ContainerStatus {
+                id: container["ID"].as_str().unwrap_or("").to_string(),
+                name: container["Names"].as_str().unwrap_or("").to_string(),
+                state: ContainerState::from(
+                    container["State"].as_str().unwrap_or("unknown")
+                ),
+                image: container["Image"].as_str().unwrap_or("").to_string(),
+                created: container["CreatedAt"].as_str().unwrap_or("").to_string(),
+                ports: vec![],
+            });
+        }
+
+        Ok(result)
+    }
+}
+
+pub struct AutoRuntime {
+    runtime: Box<dyn ContainerRuntime>,
+}
+
+impl AutoRuntime {
+    pub async fn new(user: String) -> Result<Self> {
+        // Try Podman first
+        if Self::check_podman_available() {
+            Ok(Self {
+                runtime: Box::new(PodmanRuntime::new(user)),
+            })
+        } else if Self::check_docker_available() {
+            Ok(Self {
+                runtime: Box::new(DockerRuntime::new(user)),
+            })
+        } else {
+            Err(anyhow::anyhow!(
+                "Neither Podman nor Docker is available"
+            ))
+        }
+    }
+
+    fn check_podman_available() -> bool {
+        Command::new("podman")
+            .arg("--version")
+            .output()
+            .is_ok()
+    }
+
+    fn check_docker_available() -> bool {
+        Command::new("docker")
+            .arg("--version")
+            .output()
+            .is_ok()
+    }
+}
+
+#[async_trait]
+impl ContainerRuntime for AutoRuntime {
+    async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> {
+        self.runtime.pull_image(image, signature).await
+    }
+
+    async fn create_container(
+        &self,
+        manifest: &AppManifest,
+        name: &str,
+        port_offset: u16,
+    ) -> Result<String> {
+        self.runtime.create_container(manifest, name, port_offset).await
+    }
+
+    async fn start_container(&self, name: &str) -> Result<()> {
+        self.runtime.start_container(name).await
+    }
+
+    async fn stop_container(&self, name: &str) -> Result<()> {
+        self.runtime.stop_container(name).await
+    }
+
+    async fn remove_container(&self, name: &str) -> Result<()> {
+        self.runtime.remove_container(name).await
+    }
+
+    async fn get_container_status(&self, name: &str) -> Result<ContainerStatus> {
+        self.runtime.get_container_status(name).await
+    }
+
+    async fn get_container_logs(&self, name: &str, lines: u32) -> Result<Vec<String>> {
+        self.runtime.get_container_logs(name, lines).await
+    }
+
+    async fn list_containers(&self) -> Result<Vec<ContainerStatus>> {
+        self.runtime.list_containers().await
+    }
+}
+
+// Runtime factory functions will be provided by the archipelago crate
+// that imports this library and has access to Config

core/helpers/Cargo.toml

@@ -0,0 +1,19 @@
+[package]
+name = "helpers"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+color-eyre = "0.6.2"
+futures = "0.3.28"
+lazy_async_pool = "0.3.3"
+models = { path = "../models" }
+pin-project = "1.1.3"
+serde = { version = "1.0", features = ["derive", "rc"] }
+serde_json = "1.0"
+tokio = { version = "1", features = ["full"] }
+tokio-stream = { version = "0.1.14", features = ["io-util", "sync"] }
+tracing = "0.1.39"
+yajrc = { version = "*", git = "https://github.com/dr-bonez/yajrc.git", branch = "develop" }

core/helpers/src/byte_replacement_reader.rs

@@ -0,0 +1,31 @@
+use std::task::Poll;
+
+use tokio::io::{AsyncRead, ReadBuf};
+
+#[pin_project::pin_project]
+pub struct ByteReplacementReader<R> {
+    pub replace: u8,
+    pub with: u8,
+    #[pin]
+    pub inner: R,
+}
+impl<R: AsyncRead> AsyncRead for ByteReplacementReader<R> {
+    fn poll_read(
+        self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+        buf: &mut ReadBuf<'_>,
+    ) -> std::task::Poll<std::io::Result<()>> {
+        let this = self.project();
+        match this.inner.poll_read(cx, buf) {
+            Poll::Ready(Ok(())) => {
+                for idx in 0..buf.filled().len() {
+                    if buf.filled()[idx] == *this.replace {
+                        buf.filled_mut()[idx] = *this.with;
+                    }
+                }
+                Poll::Ready(Ok(()))
+            }
+            a => a,
+        }
+    }
+}

core/helpers/src/lib.rs

@@ -0,0 +1,262 @@
+use std::future::Future;
+use std::ops::{Deref, DerefMut};
+use std::path::{Path, PathBuf};
+use std::time::Duration;
+
+use color_eyre::eyre::{eyre, Context, Error};
+use futures::future::BoxFuture;
+use futures::FutureExt;
+use tokio::fs::File;
+use tokio::sync::oneshot;
+use tokio::task::{JoinError, JoinHandle, LocalSet};
+
+mod byte_replacement_reader;
+mod rpc_client;
+mod rsync;
+mod script_dir;
+pub use byte_replacement_reader::*;
+pub use rpc_client::{RpcClient, UnixRpcClient};
+pub use rsync::*;
+pub use script_dir::*;
+
+pub fn const_true() -> bool {
+    true
+}
+
+pub fn to_tmp_path(path: impl AsRef<Path>) -> Result<PathBuf, Error> {
+    let path = path.as_ref();
+    if let (Some(parent), Some(file_name)) =
+        (path.parent(), path.file_name().and_then(|f| f.to_str()))
+    {
+        Ok(parent.join(format!(".{}.tmp", file_name)))
+    } else {
+        Err(eyre!("invalid path: {}", path.display()))
+    }
+}
+
+pub async fn canonicalize(
+    path: impl AsRef<Path> + Send + Sync,
+    create_parent: bool,
+) -> Result<PathBuf, Error> {
+    fn create_canonical_folder<'a>(
+        path: impl AsRef<Path> + Send + Sync + 'a,
+    ) -> BoxFuture<'a, Result<PathBuf, Error>> {
+        async move {
+            let path = canonicalize(path, true).await?;
+            tokio::fs::create_dir(&path)
+                .await
+                .with_context(|| path.display().to_string())?;
+            Ok(path)
+        }
+        .boxed()
+    }
+    let path = path.as_ref();
+    if tokio::fs::metadata(path).await.is_err() {
+        if let (Some(parent), Some(file_name)) = (path.parent(), path.file_name()) {
+            if create_parent && tokio::fs::metadata(parent).await.is_err() {
+                return Ok(create_canonical_folder(parent).await?.join(file_name));
+            } else {
+                return Ok(tokio::fs::canonicalize(parent)
+                    .await
+                    .with_context(|| parent.display().to_string())?
+                    .join(file_name));
+            }
+        }
+    }
+    tokio::fs::canonicalize(&path)
+        .await
+        .with_context(|| path.display().to_string())
+}
+
+#[pin_project::pin_project(PinnedDrop)]
+pub struct NonDetachingJoinHandle<T>(#[pin] JoinHandle<T>);
+impl<T> NonDetachingJoinHandle<T> {
+    pub async fn wait_for_abort(self) -> Result<T, JoinError> {
+        self.abort();
+        self.await
+    }
+}
+impl<T> From<JoinHandle<T>> for NonDetachingJoinHandle<T> {
+    fn from(t: JoinHandle<T>) -> Self {
+        NonDetachingJoinHandle(t)
+    }
+}
+
+impl<T> Deref for NonDetachingJoinHandle<T> {
+    type Target = JoinHandle<T>;
+    fn deref(&self) -> &Self::Target {
+        &self.0
+    }
+}
+impl<T> DerefMut for NonDetachingJoinHandle<T> {
+    fn deref_mut(&mut self) -> &mut Self::Target {
+        &mut self.0
+    }
+}
+#[pin_project::pinned_drop]
+impl<T> PinnedDrop for NonDetachingJoinHandle<T> {
+    fn drop(self: std::pin::Pin<&mut Self>) {
+        let this = self.project();
+        this.0.into_ref().get_ref().abort()
+    }
+}
+impl<T> Future for NonDetachingJoinHandle<T> {
+    type Output = Result<T, JoinError>;
+    fn poll(
+        self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+    ) -> std::task::Poll<Self::Output> {
+        let this = self.project();
+        this.0.poll(cx)
+    }
+}
+
+pub struct AtomicFile {
+    tmp_path: PathBuf,
+    path: PathBuf,
+    file: Option<File>,
+}
+impl AtomicFile {
+    pub async fn new(
+        path: impl AsRef<Path> + Send + Sync,
+        tmp_path: Option<impl AsRef<Path> + Send + Sync>,
+    ) -> Result<Self, Error> {
+        let path = canonicalize(&path, true).await?;
+        let tmp_path = if let Some(tmp_path) = tmp_path {
+            canonicalize(&tmp_path, true).await?
+        } else {
+            to_tmp_path(&path)?
+        };
+        let file = File::create(&tmp_path)
+            .await
+            .with_context(|| tmp_path.display().to_string())?;
+        Ok(Self {
+            tmp_path,
+            path,
+            file: Some(file),
+        })
+    }
+
+    pub async fn rollback(mut self) -> Result<(), Error> {
+        drop(self.file.take());
+        tokio::fs::remove_file(&self.tmp_path)
+            .await
+            .with_context(|| format!("rm {}", self.tmp_path.display()))?;
+        Ok(())
+    }
+
+    pub async fn save(mut self) -> Result<(), Error> {
+        use tokio::io::AsyncWriteExt;
+        if let Some(file) = self.file.as_mut() {
+            file.flush().await?;
+            file.shutdown().await?;
+            file.sync_all().await?;
+        }
+        drop(self.file.take());
+        tokio::fs::rename(&self.tmp_path, &self.path)
+            .await
+            .with_context(|| {
+                format!("mv {} -> {}", self.tmp_path.display(), self.path.display())
+            })?;
+        Ok(())
+    }
+}
+impl std::ops::Deref for AtomicFile {
+    type Target = File;
+    fn deref(&self) -> &Self::Target {
+        self.file.as_ref().unwrap()
+    }
+}
+impl std::ops::DerefMut for AtomicFile {
+    fn deref_mut(&mut self) -> &mut Self::Target {
+        self.file.as_mut().unwrap()
+    }
+}
+impl Drop for AtomicFile {
+    fn drop(&mut self) {
+        if let Some(file) = self.file.take() {
+            drop(file);
+            let path = std::mem::take(&mut self.tmp_path);
+            tokio::spawn(async move { tokio::fs::remove_file(path).await.unwrap() });
+        }
+    }
+}
+
+pub struct TimedResource<T: 'static + Send> {
+    handle: NonDetachingJoinHandle<Option<T>>,
+    ready: oneshot::Sender<()>,
+}
+impl<T: 'static + Send> TimedResource<T> {
+    pub fn new(resource: T, timer: Duration) -> Self {
+        let (send, recv) = oneshot::channel();
+        let handle = tokio::spawn(async move {
+            tokio::select! {
+                _ = tokio::time::sleep(timer) => {
+                    drop(resource);
+                    None
+                },
+                _ = recv => Some(resource),
+            }
+        });
+        Self {
+            handle: handle.into(),
+            ready: send,
+        }
+    }
+
+    pub fn new_with_destructor<
+        Fn: FnOnce(T) -> Fut + Send + 'static,
+        Fut: Future<Output = ()> + Send,
+    >(
+        resource: T,
+        timer: Duration,
+        destructor: Fn,
+    ) -> Self {
+        let (send, recv) = oneshot::channel();
+        let handle = tokio::spawn(async move {
+            tokio::select! {
+                _ = tokio::time::sleep(timer) => {
+                    destructor(resource).await;
+                    None
+                },
+                _ = recv => Some(resource),
+            }
+        });
+        Self {
+            handle: handle.into(),
+            ready: send,
+        }
+    }
+
+    pub async fn get(self) -> Option<T> {
+        let _ = self.ready.send(());
+        self.handle.await.unwrap()
+    }
+
+    pub fn is_timed_out(&self) -> bool {
+        self.ready.is_closed()
+    }
+}
+
+pub async fn spawn_local<
+    T: 'static + Send,
+    F: FnOnce() -> Fut + Send + 'static,
+    Fut: Future<Output = T> + 'static,
+>(
+    fut: F,
+) -> NonDetachingJoinHandle<T> {
+    let (send, recv) = tokio::sync::oneshot::channel();
+    std::thread::spawn(move || {
+        tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .unwrap()
+            .block_on(async move {
+                let set = LocalSet::new();
+                send.send(set.spawn_local(fut()).into())
+                    .unwrap_or_else(|_| unreachable!());
+                set.await
+            })
+    });
+    recv.await.unwrap()
+}

core/helpers/src/rpc_client.rs

@@ -0,0 +1,192 @@
+use std::collections::BTreeMap;
+use std::path::PathBuf;
+use std::sync::atomic::AtomicUsize;
+use std::sync::{Arc, Weak};
+
+use futures::future::BoxFuture;
+use futures::{FutureExt, TryFutureExt};
+use lazy_async_pool::Pool;
+use models::{Error, ErrorKind, ResultExt};
+use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWrite, AsyncWriteExt, BufReader};
+use tokio::net::UnixStream;
+use tokio::runtime::Handle;
+use tokio::sync::{oneshot, Mutex};
+use yajrc::{Id, RpcError, RpcMethod, RpcRequest, RpcResponse};
+
+use crate::NonDetachingJoinHandle;
+
+type DynWrite = Box<dyn AsyncWrite + Unpin + Send + Sync + 'static>;
+type ResponseMap = BTreeMap<Id, oneshot::Sender<Result<Value, RpcError>>>;
+
+const MAX_TRIES: u64 = 3;
+
+pub struct RpcClient {
+    id: Arc<AtomicUsize>,
+    _handler: NonDetachingJoinHandle<()>,
+    writer: DynWrite,
+    responses: Weak<Mutex<ResponseMap>>,
+}
+impl RpcClient {
+    pub fn new<
+        W: AsyncWrite + Unpin + Send + Sync + 'static,
+        R: AsyncRead + Unpin + Send + Sync + 'static,
+    >(
+        writer: W,
+        reader: R,
+        id: Arc<AtomicUsize>,
+    ) -> Self {
+        let writer: DynWrite = Box::new(writer);
+        let responses = Arc::new(Mutex::new(ResponseMap::new()));
+        let weak_responses = Arc::downgrade(&responses);
+        RpcClient {
+            id,
+            _handler: tokio::spawn(async move {
+                let mut lines = BufReader::new(reader).lines();
+                while let Some(line) = lines.next_line().await.transpose() {
+                    match line.map_err(Error::from).and_then(|l| {
+                        serde_json::from_str::<RpcResponse>(&l)
+                            .with_kind(ErrorKind::Deserialization)
+                    }) {
+                        Ok(l) => {
+                            if let Some(id) = l.id {
+                                if let Some(res) = responses.lock().await.remove(&id) {
+                                    if let Err(e) = res.send(l.result) {
+                                        tracing::warn!(
+                                            "RpcClient Response for Unknown ID: {:?}",
+                                            e
+                                        );
+                                    }
+                                } else {
+                                    tracing::warn!(
+                                        "RpcClient Response for Unknown ID: {:?}",
+                                        l.result
+                                    );
+                                }
+                            } else {
+                                tracing::info!("RpcClient Notification: {:?}", l);
+                            }
+                        }
+                        Err(e) => {
+                            tracing::error!("RpcClient Error: {}", e);
+                            tracing::debug!("{:?}", e);
+                        }
+                    }
+                }
+            })
+            .into(),
+            writer,
+            responses: weak_responses,
+        }
+    }
+
+    pub async fn request<T: RpcMethod>(
+        &mut self,
+        method: T,
+        params: T::Params,
+    ) -> Result<T::Response, RpcError>
+    where
+        T: Serialize,
+        T::Params: Serialize,
+        T::Response: for<'de> Deserialize<'de>,
+    {
+        let id = Id::Number(
+            self.id
+                .fetch_add(1, std::sync::atomic::Ordering::SeqCst)
+                .into(),
+        );
+        let request = RpcRequest {
+            id: Some(id.clone()),
+            method,
+            params,
+        };
+        if let Some(w) = self.responses.upgrade() {
+            let (send, recv) = oneshot::channel();
+            w.lock().await.insert(id.clone(), send);
+            self.writer
+                .write_all((serde_json::to_string(&request)? + "\n").as_bytes())
+                .await
+                .map_err(|e| {
+                    let mut err = yajrc::INTERNAL_ERROR.clone();
+                    err.data = Some(json!(e.to_string()));
+                    err
+                })?;
+            match recv.await {
+                Ok(val) => {
+                    return Ok(serde_json::from_value(val?)?);
+                }
+                Err(_err) => {
+                    tokio::task::yield_now().await;
+                }
+            }
+        }
+        tracing::debug!(
+            "Client has finished {:?}",
+            futures::poll!(&mut self._handler)
+        );
+        let mut err = yajrc::INTERNAL_ERROR.clone();
+        err.data = Some(json!("RpcClient thread has terminated"));
+        Err(err)
+    }
+}
+
+pub struct UnixRpcClient {
+    pool: Pool<
+        RpcClient,
+        Box<dyn Fn() -> BoxFuture<'static, Result<RpcClient, std::io::Error>> + Send + Sync>,
+        BoxFuture<'static, Result<RpcClient, std::io::Error>>,
+        std::io::Error,
+    >,
+}
+impl UnixRpcClient {
+    pub fn new(path: PathBuf) -> Self {
+        let rt = Handle::current();
+        let id = Arc::new(AtomicUsize::new(0));
+        Self {
+            pool: Pool::new(
+                0,
+                Box::new(move || {
+                    let path = path.clone();
+                    let id = id.clone();
+                    rt.spawn(async move {
+                        let (r, w) = UnixStream::connect(&path).await?.into_split();
+                        Ok(RpcClient::new(w, r, id))
+                    })
+                    .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))
+                    .and_then(|x| async move { x })
+                    .boxed()
+                }),
+            ),
+        }
+    }
+
+    pub async fn request<T: RpcMethod>(
+        &self,
+        method: T,
+        params: T::Params,
+    ) -> Result<T::Response, RpcError>
+    where
+        T: Serialize + Clone,
+        T::Params: Serialize + Clone,
+        T::Response: for<'de> Deserialize<'de>,
+    {
+        let mut tries = 0;
+        let res = loop {
+            tries += 1;
+            let mut client = self.pool.clone().get().await?;
+            let res = client.request(method.clone(), params.clone()).await;
+            match &res {
+                Err(e) if e.code == yajrc::INTERNAL_ERROR.code => {
+                    client.destroy();
+                }
+                _ => break res,
+            }
+            if tries > MAX_TRIES {
+                tracing::warn!("Max Tries exceeded");
+                break res;
+            }
+        };
+        res
+    }
+}

core/helpers/src/rsync.rs

@@ -0,0 +1,224 @@
+use std::path::Path;
+
+use color_eyre::eyre::eyre;
+use futures::StreamExt;
+use models::{Error, ErrorKind};
+use tokio::io::{AsyncBufReadExt, AsyncReadExt, BufReader};
+use tokio::process::{Child, Command};
+use tokio::sync::watch;
+use tokio_stream::wrappers::WatchStream;
+
+use crate::{const_true, ByteReplacementReader, NonDetachingJoinHandle};
+
+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct RsyncOptions {
+    #[serde(default = "const_true")]
+    pub delete: bool,
+    #[serde(default = "const_true")]
+    pub force: bool,
+    #[serde(default)]
+    pub ignore_existing: bool,
+    #[serde(default)]
+    pub exclude: Vec<String>,
+    #[serde(default = "const_true")]
+    pub no_permissions: bool,
+    #[serde(default = "const_true")]
+    pub no_owner: bool,
+}
+impl Default for RsyncOptions {
+    fn default() -> Self {
+        Self {
+            delete: true,
+            force: true,
+            ignore_existing: false,
+            exclude: Vec::new(),
+            no_permissions: false,
+            no_owner: false,
+        }
+    }
+}
+
+pub struct Rsync {
+    pub command: Child,
+    _progress_task: NonDetachingJoinHandle<Result<(), Error>>,
+    stderr: NonDetachingJoinHandle<Result<String, Error>>,
+    pub progress: WatchStream<f64>,
+}
+impl Rsync {
+    pub async fn new(
+        src: impl AsRef<Path>,
+        dst: impl AsRef<Path>,
+        options: RsyncOptions,
+    ) -> Result<Self, Error> {
+        let mut cmd = Command::new("rsync");
+        if options.delete {
+            cmd.arg("--delete");
+        }
+        if options.force {
+            cmd.arg("--force");
+        }
+        if options.ignore_existing {
+            cmd.arg("--ignore-existing");
+        }
+        if options.no_permissions {
+            cmd.arg("--no-perms");
+        }
+        if options.no_owner {
+            cmd.arg("--no-owner");
+        }
+        for exclude in options.exclude {
+            cmd.arg(format!("--exclude={}", exclude));
+        }
+        let mut command = cmd
+            .arg("-actAXH")
+            .arg("--info=progress2")
+            .arg("--no-inc-recursive")
+            .arg(src.as_ref())
+            .arg(dst.as_ref())
+            .kill_on_drop(true)
+            .stdout(std::process::Stdio::piped())
+            .stderr(std::process::Stdio::piped())
+            .spawn()?;
+        let cmd_stdout = match command.stdout.take() {
+            None => {
+                return Err(Error::new(
+                    eyre!("rsync command stdout is none"),
+                    ErrorKind::Filesystem,
+                ))
+            }
+            Some(a) => a,
+        };
+        let mut cmd_stderr = match command.stderr.take() {
+            None => {
+                return Err(Error::new(
+                    eyre!("rsync command stderr is none"),
+                    ErrorKind::Filesystem,
+                ))
+            }
+            Some(a) => a,
+        };
+        let (send, recv) = watch::channel(0.0);
+        let stderr = tokio::spawn(async move {
+            let mut res = String::new();
+            cmd_stderr.read_to_string(&mut res).await?;
+            Ok(res)
+        })
+        .into();
+        let progress_task = tokio::spawn(async move {
+            let mut lines = BufReader::new(ByteReplacementReader {
+                replace: b'\r',
+                with: b'\n',
+                inner: cmd_stdout,
+            })
+            .lines();
+            while let Some(line) = lines.next_line().await? {
+                if let Some(percentage) = parse_percentage(&line) {
+                    if percentage > 0.0 {
+                        if let Err(err) = send.send(percentage / 100.0) {
+                            return Err(Error::new(
+                                eyre!("rsync progress send error: {}", err),
+                                ErrorKind::Filesystem,
+                            ));
+                        }
+                    }
+                }
+            }
+            Ok(())
+        })
+        .into();
+        let mut progress = WatchStream::new(recv);
+        progress.next().await;
+        Ok(Rsync {
+            command,
+            _progress_task: progress_task,
+            stderr,
+            progress,
+        })
+    }
+    pub async fn wait(mut self) -> Result<(), Error> {
+        let status = self.command.wait().await?;
+        let stderr = match self.stderr.await {
+            Err(err) => {
+                return Err(Error::new(
+                    eyre!("rsync stderr error: {}", err),
+                    ErrorKind::Filesystem,
+                ))
+            }
+            Ok(a) => a?,
+        };
+        if status.success() {
+            tracing::info!("rsync: {}", stderr);
+        } else {
+            return Err(Error::new(
+                eyre!(
+                    "rsync error: {} {} ",
+                    status.code().map(|x| x.to_string()).unwrap_or_default(),
+                    stderr
+                ),
+                ErrorKind::Filesystem,
+            ));
+        }
+        Ok(())
+    }
+}
+
+fn parse_percentage(line: &str) -> Option<f64> {
+    if let Some(percentage) = line
+        .split_ascii_whitespace()
+        .find_map(|col| col.strip_suffix("%"))
+    {
+        return percentage.parse().ok();
+    }
+    None
+}
+
+#[test]
+fn test_parse() {
+    let input = "          1.07G 57%   95.20MB/s    0:00:10 (xfr#1, to-chk=0/2)";
+    assert_eq!(Some(57.0), parse_percentage(input));
+}
+
+#[tokio::test]
+async fn test_rsync() {
+    use futures::StreamExt;
+    use tokio::fs;
+    let mut seen_zero = false;
+    let mut seen_in_between = false;
+    let mut seen_hundred = false;
+    fs::remove_dir_all("/tmp/test_rsync")
+        .await
+        .unwrap_or_default();
+    fs::create_dir_all("/tmp/test_rsync/a").await.unwrap();
+    fs::create_dir_all("/tmp/test_rsync/b").await.unwrap();
+    for i in 0..100 {
+        tokio::io::copy(
+            &mut fs::File::open("/dev/urandom").await.unwrap().take(100_000),
+            &mut fs::File::create(format!("/tmp/test_rsync/a/sample.{i}.bin"))
+                .await
+                .unwrap(),
+        )
+        .await
+        .unwrap();
+    }
+    let mut rsync = Rsync::new(
+        "/tmp/test_rsync/a/",
+        "/tmp/test_rsync/b/",
+        Default::default(),
+    )
+    .await
+    .unwrap();
+    while let Some(progress) = rsync.progress.next().await {
+        if progress <= 0.05 {
+            seen_zero = true;
+        } else if progress > 0.05 && progress < 1.0 {
+            seen_in_between = true
+        } else {
+            seen_hundred = true;
+        }
+    }
+    rsync.wait().await.unwrap();
+    assert!(seen_zero, "seen zero");
+    assert!(seen_in_between, "seen in between 0 and 100");
+    assert!(seen_hundred, "seen 100");
+}

core/helpers/src/script_dir.rs

@@ -0,0 +1,13 @@
+use std::path::{Path, PathBuf};
+
+use models::{PackageId, Version};
+
+pub const PKG_SCRIPT_DIR: &str = "package-data/scripts";
+
+pub fn script_dir<P: AsRef<Path>>(datadir: P, pkg_id: &PackageId, version: &Version) -> PathBuf {
+    datadir
+        .as_ref()
+        .join(&*PKG_SCRIPT_DIR)
+        .join(pkg_id)
+        .join(version.as_str())
+}

core/js-engine/Cargo.toml

@@ -0,0 +1,23 @@
+[package]
+name = "js-engine"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+async-trait = "0.1.74"
+dashmap = "5.5.3"
+deno_core = "=0.222.0"
+deno_ast = { version = "=0.29.5", features = ["transpiling"] }
+container-init = { path = "../container-init" }
+reqwest = { version = "0.11.22" }
+sha2 = "0.10.8"
+itertools = "0.11.0"
+lazy_static = "1.4.0"
+models = { path = "../models" }
+helpers = { path = "../helpers" }
+serde = { version = "1.0", features = ["derive", "rc"] }
+serde_json = "1.0"
+tokio = { version = "1", features = ["full"] }
+tracing = "0.1"

core/js-engine/src/artifacts/loadModule.js

@@ -0,0 +1,242 @@
+import Deno from "/deno_global.js";
+import * as mainModule from "/embassy.js";
+
+function requireParam(param) {
+  throw new Error(`Missing required parameter ${param}`);
+}
+
+/**
+ * This is using the simplified json pointer spec, using no escapes and arrays
+ * @param {object} obj
+ * @param {string} pointer
+ * @returns
+ */
+function jsonPointerValue(obj, pointer) {
+  const paths = pointer.substring(1).split("/");
+  for (const path of paths) {
+    if (obj == null) {
+      return null;
+    }
+    obj = (obj || {})[path];
+  }
+  return obj;
+}
+
+function maybeDate(value) {
+  if (!value) return value;
+  return new Date(value);
+}
+const writeFile = (
+  {
+    path = requireParam("path"),
+    volumeId = requireParam("volumeId"),
+    toWrite = requireParam("toWrite"),
+  } = requireParam("options"),
+) => Deno.core.opAsync("write_file", volumeId, path, toWrite);
+
+const readFile = (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => Deno.core.opAsync("read_file", volumeId, path);
+
+
+
+const runDaemon = (
+  { command = requireParam("command"), args = [] } = requireParam("options"),
+) => {
+  let id = Deno.core.opAsync("start_command", command, args, "inherit", null);
+  let processId = id.then(x => x.processId)
+  let waitPromise = null;
+  return {
+    processId,
+    async wait() {
+      waitPromise = waitPromise || Deno.core.opAsync("wait_command", await processId)
+      return waitPromise
+    },
+    async term(signal = 15) {
+      return Deno.core.opAsync("send_signal", await processId, 15)
+    }
+  }
+};
+const runCommand = async (
+  { command = requireParam("command"), args = [], timeoutMillis = 30000 } = requireParam("options"),
+) => {
+  let id = Deno.core.opAsync("start_command", command, args, "collect", timeoutMillis);
+  let pid = id.then(x => x.processId)
+  return Deno.core.opAsync("wait_command", await pid)
+};
+const signalGroup = async (
+  { gid = requireParam("gid"), signal = requireParam("signal") } = requireParam("gid and signal")
+) => {
+  return Deno.core.opAsync("signal_group", gid, signal);
+};
+const sleep = (timeMs = requireParam("timeMs"),
+) => Deno.core.opAsync("sleep", timeMs);
+
+const rename = (
+  {
+    srcVolume = requireParam("srcVolume"),
+    dstVolume = requirePapram("dstVolume"),
+    srcPath = requireParam("srcPath"),
+    dstPath = requireParam("dstPath"),
+  } = requireParam("options"),
+) => Deno.core.opAsync("rename", srcVolume, srcPath, dstVolume, dstPath);
+const metadata = async (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => {
+  const data = await Deno.core.opAsync("metadata", volumeId, path);
+  return {
+    ...data,
+    modified: maybeDate(data.modified),
+    created: maybeDate(data.created),
+    accessed: maybeDate(data.accessed),
+  };
+};
+const removeFile = (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => Deno.core.opAsync("remove_file", volumeId, path);
+const isSandboxed = () => Deno.core.ops["is_sandboxed"]();
+
+const writeJsonFile = (
+  {
+    volumeId = requireParam("volumeId"),
+    path = requireParam("path"),
+    toWrite = requireParam("toWrite"),
+  } = requireParam("options"),
+) =>
+  writeFile({
+    volumeId,
+    path,
+    toWrite: JSON.stringify(toWrite),
+  });
+
+const chown = async (
+  {
+    volumeId = requireParam("volumeId"),
+    path = requireParam("path"),
+    uid = requireParam("uid"),
+  } = requireParam("options"),
+) => {
+  return await Deno.core.opAsync("chown", volumeId, path, uid);
+};
+
+const chmod = async (
+  {
+    volumeId = requireParam("volumeId"),
+    path = requireParam("path"),
+    mode = requireParam("mode"),
+  } = requireParam("options"),
+) => {
+  return await Deno.core.opAsync("chmod", volumeId, path, mode);
+};
+const readJsonFile = async (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => JSON.parse(await readFile({ volumeId, path }));
+const createDir = (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => Deno.core.opAsync("create_dir", volumeId, path);
+
+const readDir = (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => Deno.core.opAsync("read_dir", volumeId, path);
+const removeDir = (
+  { volumeId = requireParam("volumeId"), path = requireParam("path") } = requireParam("options"),
+) => Deno.core.opAsync("remove_dir", volumeId, path);
+const trace = (whatToTrace = requireParam('whatToTrace')) => Deno.core.opAsync("log_trace", whatToTrace);
+const warn = (whatToTrace = requireParam('whatToTrace')) => Deno.core.opAsync("log_warn", whatToTrace);
+const error = (whatToTrace = requireParam('whatToTrace')) => Deno.core.opAsync("log_error", whatToTrace);
+const debug = (whatToTrace = requireParam('whatToTrace')) => Deno.core.opAsync("log_debug", whatToTrace);
+const info = (whatToTrace = requireParam('whatToTrace')) => Deno.core.opAsync("log_info", whatToTrace);
+const fetch = async (url = requireParam ('url'), options = null) => {
+  const { body, ...response } = await Deno.core.opAsync("fetch", url, options);
+  const textValue = Promise.resolve(body);
+  return {
+    ...response,
+    text() {
+      return textValue;
+    },
+    json() {
+      return textValue.then((x) => JSON.parse(x));
+    },
+  };
+};
+
+const runRsync = (
+  { 
+    srcVolume = requireParam("srcVolume"),
+    dstVolume = requireParam("dstVolume"),
+    srcPath = requireParam("srcPath"),
+    dstPath = requireParam("dstPath"),
+    options = requireParam("options"),
+ } = requireParam("options"),
+) => {
+  let id = Deno.core.opAsync("rsync", srcVolume, srcPath, dstVolume, dstPath, options);
+  let waitPromise = null;
+  return {
+    async id() {
+      return id
+    },
+    async wait() {
+      waitPromise = waitPromise || Deno.core.opAsync("rsync_wait", await id)
+      return waitPromise
+    },
+    async progress() {
+      return Deno.core.opAsync("rsync_progress", await id)
+    }
+  }
+};
+
+const diskUsage = async ({
+  volumeId = requireParam("volumeId"),
+  path = requireParam("path"),
+} = { volumeId: null, path: null }) => {
+  const [used, total] = await Deno.core.opAsync("disk_usage", volumeId, path);
+  return { used, total }
+}
+
+const currentFunction = Deno.core.ops.current_function();
+const input = Deno.core.ops.get_input();
+const variable_args = Deno.core.ops.get_variable_args();
+const setState = (x) => Deno.core.ops.set_value(x);
+const effects = {
+  chmod,
+  chown,
+  writeFile,
+  readFile,
+  writeJsonFile,
+  readJsonFile,
+  error,
+  warn,
+  debug,
+  trace,
+  info,
+  isSandboxed,
+  fetch,
+  removeFile,
+  createDir,
+  removeDir,
+  metadata,
+  rename,
+  runCommand,
+  sleep,
+  runDaemon,
+  signalGroup,
+  runRsync,
+  readDir,
+  diskUsage,
+};
+
+const defaults = {
+  "handleSignal": (effects, { gid, signal }) => {
+    return effects.signalGroup({ gid, signal })
+  }
+}
+
+const runFunction = jsonPointerValue(mainModule, currentFunction) || jsonPointerValue(defaults, currentFunction);
+(async () => {
+  if (typeof runFunction !== "function") {
+    error(`Expecting ${currentFunction} to be a function`);
+    throw new Error(`Expecting ${currentFunction} to be a function`);
+  }
+  const answer = await runFunction(effects, input, ...variable_args);
+  setState(answer);
+})();

core/models/Cargo.toml

@@ -0,0 +1,39 @@
+[package]
+name = "models"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+base64 = "0.21.4"
+color-eyre = "0.6.2"
+ed25519-dalek = { version = "2.0.0", features = ["serde"] }
+lazy_static = "1.4"
+mbrman = "0.5.2"
+emver = { git = "https://github.com/Start9Labs/emver-rs.git", rev = "61cf0bc96711b4d6f3f30df8efef025e0cc02bad", features = [
+  "serde",
+] }
+ipnet = "2.8.0"
+openssl = { version = "0.10.57", features = ["vendored"] }
+patch-db = { version = "*", path = "../../patch-db/patch-db", features = [
+  "trace",
+] }
+rand = "0.8.5"
+regex = "1.10.2"
+reqwest = "0.11.22"
+rpc-toolkit = "0.2.2"
+serde = { version = "1.0", features = ["derive", "rc"] }
+serde_json = "1.0"
+sqlx = { version = "0.7.2", features = [
+  "chrono",
+  "runtime-tokio-rustls",
+  "postgres",
+] }
+ssh-key = "0.6.2"
+thiserror = "1.0"
+tokio = { version = "1", features = ["full"] }
+torut = "0.2.1"
+tracing = "0.1.39"
+# Pinned via [patch] to vendored 0.1.6 (edition 2021). 0.1.6+ on crates.io use edition2024, which Cargo 1.75 doesn't support.
+yasi = "0.1.6"

core/models/src/data_url.rs

@@ -0,0 +1,171 @@
+use std::borrow::Cow;
+use std::path::Path;
+
+use base64::Engine;
+use color_eyre::eyre::eyre;
+use reqwest::header::CONTENT_TYPE;
+use serde::{Deserialize, Serialize};
+use tokio::io::{AsyncRead, AsyncReadExt};
+use yasi::InternedString;
+
+use crate::{mime, Error, ErrorKind, ResultExt};
+
+#[derive(Clone)]
+pub struct DataUrl<'a> {
+    mime: InternedString,
+    data: Cow<'a, [u8]>,
+}
+impl<'a> DataUrl<'a> {
+    pub const DEFAULT_MIME: &'static str = "application/octet-stream";
+    pub const MAX_SIZE: u64 = 100 * 1024;
+
+    // data:{mime};base64,{data}
+    pub fn to_string(&self) -> String {
+        use std::fmt::Write;
+        let mut res = String::with_capacity(self.data_url_len_without_mime() + self.mime.len());
+        let _ = write!(res, "data:{};base64,", self.mime);
+        base64::engine::general_purpose::STANDARD.encode_string(&self.data, &mut res);
+        res
+    }
+
+    fn data_url_len_without_mime(&self) -> usize {
+        5 + 8 + (4 * self.data.len() / 3) + 3
+    }
+
+    pub fn data_url_len(&self) -> usize {
+        self.data_url_len_without_mime() + self.mime.len()
+    }
+
+    pub fn from_slice(mime: &str, data: &'a [u8]) -> Self {
+        Self {
+            mime: InternedString::intern(mime),
+            data: Cow::Borrowed(data),
+        }
+    }
+}
+impl DataUrl<'static> {
+    pub async fn from_reader(
+        mime: &str,
+        rdr: impl AsyncRead + Unpin,
+        size_est: Option<u64>,
+    ) -> Result<Self, Error> {
+        let check_size = |s| {
+            if s > Self::MAX_SIZE {
+                Err(Error::new(
+                    eyre!("Data URLs must be smaller than 100KiB"),
+                    ErrorKind::Filesystem,
+                ))
+            } else {
+                Ok(s)
+            }
+        };
+        let mut buf = size_est
+            .map(check_size)
+            .transpose()?
+            .map(|s| Vec::with_capacity(s as usize))
+            .unwrap_or_default();
+        rdr.take(Self::MAX_SIZE + 1).read_to_end(&mut buf).await?;
+        check_size(buf.len() as u64)?;
+
+        Ok(Self {
+            mime: InternedString::intern(mime),
+            data: Cow::Owned(buf),
+        })
+    }
+
+    pub async fn from_path(path: impl AsRef<Path>) -> Result<Self, Error> {
+        let path = path.as_ref();
+        let f = tokio::fs::File::open(path).await?;
+        let m = f.metadata().await?;
+        let mime = path
+            .extension()
+            .and_then(|s| s.to_str())
+            .and_then(mime)
+            .unwrap_or(Self::DEFAULT_MIME);
+        Self::from_reader(mime, f, Some(m.len())).await
+    }
+
+    pub async fn from_response(res: reqwest::Response) -> Result<Self, Error> {
+        let mime = InternedString::intern(
+            res.headers()
+                .get(CONTENT_TYPE)
+                .and_then(|h| h.to_str().ok())
+                .unwrap_or(Self::DEFAULT_MIME),
+        );
+        let data = res.bytes().await.with_kind(ErrorKind::Network)?.to_vec();
+        Ok(Self {
+            mime,
+            data: Cow::Owned(data),
+        })
+    }
+
+    pub fn from_vec(mime: &str, data: Vec<u8>) -> Self {
+        Self {
+            mime: InternedString::intern(mime),
+            data: Cow::Owned(data),
+        }
+    }
+}
+
+impl<'a> std::fmt::Debug for DataUrl<'a> {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str(&self.to_string())
+    }
+}
+
+impl<'de> Deserialize<'de> for DataUrl<'static> {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        struct Visitor;
+        impl<'de> serde::de::Visitor<'de> for Visitor {
+            type Value = DataUrl<'static>;
+            fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
+                write!(formatter, "a valid base64 data url")
+            }
+            fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>
+            where
+                E: serde::de::Error,
+            {
+                v.strip_prefix("data:")
+                    .and_then(|v| v.split_once(";base64,"))
+                    .and_then(|(mime, data)| {
+                        Some(DataUrl {
+                            mime: InternedString::intern(mime),
+                            data: Cow::Owned(
+                                base64::engine::general_purpose::STANDARD
+                                    .decode(data)
+                                    .ok()?,
+                            ),
+                        })
+                    })
+                    .ok_or_else(|| {
+                        E::invalid_value(serde::de::Unexpected::Str(v), &"a valid base64 data url")
+                    })
+            }
+        }
+        deserializer.deserialize_any(Visitor)
+    }
+}
+
+impl<'a> Serialize for DataUrl<'a> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        serializer.serialize_str(&self.to_string())
+    }
+}
+
+#[test]
+fn doesnt_reallocate() {
+    let random: [u8; 10] = rand::random();
+    for i in 0..10 {
+        let icon = DataUrl {
+            mime: InternedString::intern("png"),
+            data: Cow::Borrowed(&random[..i]),
+        };
+        assert_eq!(dbg!(icon.to_string()).capacity(), icon.data_url_len());
+    }
+}

core/models/src/errors.rs

@@ -0,0 +1,434 @@
+use std::fmt::Display;
+
+use color_eyre::eyre::eyre;
+use patch_db::Revision;
+use rpc_toolkit::hyper::http::uri::InvalidUri;
+use rpc_toolkit::reqwest;
+use rpc_toolkit::yajrc::RpcError;
+
+use crate::InvalidId;
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ErrorKind {
+    Unknown = 1,
+    Filesystem = 2,
+    Docker = 3,
+    ConfigSpecViolation = 4,
+    ConfigRulesViolation = 5,
+    NotFound = 6,
+    IncorrectPassword = 7,
+    VersionIncompatible = 8,
+    Network = 9,
+    Registry = 10,
+    Serialization = 11,
+    Deserialization = 12,
+    Utf8 = 13,
+    ParseVersion = 14,
+    IncorrectDisk = 15,
+    // Nginx = 16,
+    Dependency = 17,
+    ParseS9pk = 18,
+    ParseUrl = 19,
+    DiskNotAvailable = 20,
+    BlockDevice = 21,
+    InvalidOnionAddress = 22,
+    Pack = 23,
+    ValidateS9pk = 24,
+    DiskCorrupted = 25, // Remove
+    Tor = 26,
+    ConfigGen = 27,
+    ParseNumber = 28,
+    Database = 29,
+    InvalidPackageId = 30,
+    InvalidSignature = 31,
+    Backup = 32,
+    Restore = 33,
+    Authorization = 34,
+    AutoConfigure = 35,
+    Action = 36,
+    RateLimited = 37,
+    InvalidRequest = 38,
+    MigrationFailed = 39,
+    Uninitialized = 40,
+    ParseNetAddress = 41,
+    ParseSshKey = 42,
+    SoundError = 43,
+    ParseTimestamp = 44,
+    ParseSysInfo = 45,
+    Wifi = 46,
+    Journald = 47,
+    DiskManagement = 48,
+    OpenSsl = 49,
+    PasswordHashGeneration = 50,
+    DiagnosticMode = 51,
+    ParseDbField = 52,
+    Duplicate = 53,
+    MultipleErrors = 54,
+    Incoherent = 55,
+    InvalidBackupTargetId = 56,
+    ProductKeyMismatch = 57,
+    LanPortConflict = 58,
+    Javascript = 59,
+    Pem = 60,
+    TLSInit = 61,
+    Ascii = 62,
+    MissingHeader = 63,
+    Grub = 64,
+    Systemd = 65,
+    OpenSsh = 66,
+    Zram = 67,
+    Lshw = 68,
+    CpuSettings = 69,
+    Firmware = 70,
+    Timeout = 71,
+}
+impl ErrorKind {
+    pub fn as_str(&self) -> &'static str {
+        use ErrorKind::*;
+        match self {
+            Unknown => "Unknown Error",
+            Filesystem => "Filesystem I/O Error",
+            Docker => "Docker Error",
+            ConfigSpecViolation => "Config Spec Violation",
+            ConfigRulesViolation => "Config Rules Violation",
+            NotFound => "Not Found",
+            IncorrectPassword => "Incorrect Password",
+            VersionIncompatible => "Version Incompatible",
+            Network => "Network Error",
+            Registry => "Registry Error",
+            Serialization => "Serialization Error",
+            Deserialization => "Deserialization Error",
+            Utf8 => "UTF-8 Parse Error",
+            ParseVersion => "Version Parsing Error",
+            IncorrectDisk => "Incorrect Disk",
+            // Nginx => "Nginx Error",
+            Dependency => "Dependency Error",
+            ParseS9pk => "S9PK Parsing Error",
+            ParseUrl => "URL Parsing Error",
+            DiskNotAvailable => "Disk Not Available",
+            BlockDevice => "Block Device Error",
+            InvalidOnionAddress => "Invalid Onion Address",
+            Pack => "Pack Error",
+            ValidateS9pk => "S9PK Validation Error",
+            DiskCorrupted => "Disk Corrupted", // Remove
+            Tor => "Tor Daemon Error",
+            ConfigGen => "Config Generation Error",
+            ParseNumber => "Number Parsing Error",
+            Database => "Database Error",
+            InvalidPackageId => "Invalid Package ID",
+            InvalidSignature => "Invalid Signature",
+            Backup => "Backup Error",
+            Restore => "Restore Error",
+            Authorization => "Unauthorized",
+            AutoConfigure => "Auto-Configure Error",
+            Action => "Action Failed",
+            RateLimited => "Rate Limited",
+            InvalidRequest => "Invalid Request",
+            MigrationFailed => "Migration Failed",
+            Uninitialized => "Uninitialized",
+            ParseNetAddress => "Net Address Parsing Error",
+            ParseSshKey => "SSH Key Parsing Error",
+            SoundError => "Sound Interface Error",
+            ParseTimestamp => "Timestamp Parsing Error",
+            ParseSysInfo => "System Info Parsing Error",
+            Wifi => "WiFi Internal Error",
+            Journald => "Journald Error",
+            DiskManagement => "Disk Management Error",
+            OpenSsl => "OpenSSL Internal Error",
+            PasswordHashGeneration => "Password Hash Generation Error",
+            DiagnosticMode => "Server is in Diagnostic Mode",
+            ParseDbField => "Database Field Parse Error",
+            Duplicate => "Duplication Error",
+            MultipleErrors => "Multiple Errors",
+            Incoherent => "Incoherent",
+            InvalidBackupTargetId => "Invalid Backup Target ID",
+            ProductKeyMismatch => "Incompatible Product Keys",
+            LanPortConflict => "Incompatible LAN Port Configuration",
+            Javascript => "Javascript Engine Error",
+            Pem => "PEM Encoding Error",
+            TLSInit => "TLS Backend Initialization Error",
+            Ascii => "ASCII Parse Error",
+            MissingHeader => "Missing Header",
+            Grub => "Grub Error",
+            Systemd => "Systemd Error",
+            OpenSsh => "OpenSSH Error",
+            Zram => "Zram Error",
+            Lshw => "LSHW Error",
+            CpuSettings => "CPU Settings Error",
+            Firmware => "Firmware Error",
+            Timeout => "Timeout Error",
+        }
+    }
+}
+impl Display for ErrorKind {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.as_str())
+    }
+}
+
+#[derive(Debug)]
+pub struct Error {
+    pub source: color_eyre::eyre::Error,
+    pub kind: ErrorKind,
+    pub revision: Option<Revision>,
+}
+
+impl Display for Error {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}: {}", self.kind.as_str(), self.source)
+    }
+}
+impl Error {
+    pub fn new<E: Into<color_eyre::eyre::Error>>(source: E, kind: ErrorKind) -> Self {
+        Error {
+            source: source.into(),
+            kind,
+            revision: None,
+        }
+    }
+}
+impl From<InvalidId> for Error {
+    fn from(err: InvalidId) -> Self {
+        Error::new(err, ErrorKind::InvalidPackageId)
+    }
+}
+impl From<std::io::Error> for Error {
+    fn from(e: std::io::Error) -> Self {
+        Error::new(e, ErrorKind::Filesystem)
+    }
+}
+impl From<std::str::Utf8Error> for Error {
+    fn from(e: std::str::Utf8Error) -> Self {
+        Error::new(e, ErrorKind::Utf8)
+    }
+}
+impl From<std::string::FromUtf8Error> for Error {
+    fn from(e: std::string::FromUtf8Error) -> Self {
+        Error::new(e, ErrorKind::Utf8)
+    }
+}
+impl From<emver::ParseError> for Error {
+    fn from(e: emver::ParseError) -> Self {
+        Error::new(e, ErrorKind::ParseVersion)
+    }
+}
+impl From<rpc_toolkit::url::ParseError> for Error {
+    fn from(e: rpc_toolkit::url::ParseError) -> Self {
+        Error::new(e, ErrorKind::ParseUrl)
+    }
+}
+impl From<std::num::ParseIntError> for Error {
+    fn from(e: std::num::ParseIntError) -> Self {
+        Error::new(e, ErrorKind::ParseNumber)
+    }
+}
+impl From<std::num::ParseFloatError> for Error {
+    fn from(e: std::num::ParseFloatError) -> Self {
+        Error::new(e, ErrorKind::ParseNumber)
+    }
+}
+impl From<patch_db::Error> for Error {
+    fn from(e: patch_db::Error) -> Self {
+        Error::new(e, ErrorKind::Database)
+    }
+}
+impl From<sqlx::Error> for Error {
+    fn from(e: sqlx::Error) -> Self {
+        Error::new(e, ErrorKind::Database)
+    }
+}
+impl From<ed25519_dalek::SignatureError> for Error {
+    fn from(e: ed25519_dalek::SignatureError) -> Self {
+        Error::new(e, ErrorKind::InvalidSignature)
+    }
+}
+impl From<std::net::AddrParseError> for Error {
+    fn from(e: std::net::AddrParseError) -> Self {
+        Error::new(e, ErrorKind::ParseNetAddress)
+    }
+}
+impl From<torut::control::ConnError> for Error {
+    fn from(e: torut::control::ConnError) -> Self {
+        Error::new(e, ErrorKind::Tor)
+    }
+}
+impl From<ipnet::AddrParseError> for Error {
+    fn from(e: ipnet::AddrParseError) -> Self {
+        Error::new(e, ErrorKind::ParseNetAddress)
+    }
+}
+impl From<openssl::error::ErrorStack> for Error {
+    fn from(e: openssl::error::ErrorStack) -> Self {
+        Error::new(eyre!("{}", e), ErrorKind::OpenSsl)
+    }
+}
+impl From<mbrman::Error> for Error {
+    fn from(e: mbrman::Error) -> Self {
+        Error::new(e, ErrorKind::DiskManagement)
+    }
+}
+impl From<InvalidUri> for Error {
+    fn from(e: InvalidUri) -> Self {
+        Error::new(eyre!("{}", e), ErrorKind::ParseUrl)
+    }
+}
+impl From<ssh_key::Error> for Error {
+    fn from(e: ssh_key::Error) -> Self {
+        Error::new(e, ErrorKind::OpenSsh)
+    }
+}
+impl From<reqwest::Error> for Error {
+    fn from(e: reqwest::Error) -> Self {
+        let kind = match e {
+            _ if e.is_builder() => ErrorKind::ParseUrl,
+            _ if e.is_decode() => ErrorKind::Deserialization,
+            _ => ErrorKind::Network,
+        };
+        Error::new(e, kind)
+    }
+}
+impl From<patch_db::value::Error> for Error {
+    fn from(value: patch_db::value::Error) -> Self {
+        match value.kind {
+            patch_db::value::ErrorKind::Serialization => {
+                Error::new(value.source, ErrorKind::Serialization)
+            }
+            patch_db::value::ErrorKind::Deserialization => {
+                Error::new(value.source, ErrorKind::Deserialization)
+            }
+        }
+    }
+}
+
+impl From<Error> for RpcError {
+    fn from(e: Error) -> Self {
+        let mut data_object = serde_json::Map::with_capacity(3);
+        data_object.insert("details".to_owned(), format!("{}", e.source).into());
+        data_object.insert("debug".to_owned(), format!("{:?}", e.source).into());
+        data_object.insert(
+            "revision".to_owned(),
+            match serde_json::to_value(&e.revision) {
+                Ok(a) => a,
+                Err(e) => {
+                    tracing::warn!("Error serializing revision for Error object: {}", e);
+                    serde_json::Value::Null
+                }
+            },
+        );
+        RpcError {
+            code: e.kind as i32,
+            message: e.kind.as_str().into(),
+            data: Some(data_object.into()),
+        }
+    }
+}
+
+#[derive(Debug, Default)]
+pub struct ErrorCollection(Vec<Error>);
+impl ErrorCollection {
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    pub fn handle<T, E: Into<Error>>(&mut self, result: Result<T, E>) -> Option<T> {
+        match result {
+            Ok(a) => Some(a),
+            Err(e) => {
+                self.0.push(e.into());
+                None
+            }
+        }
+    }
+
+    pub fn into_result(self) -> Result<(), Error> {
+        if self.0.is_empty() {
+            Ok(())
+        } else {
+            Err(Error::new(eyre!("{}", self), ErrorKind::MultipleErrors))
+        }
+    }
+}
+impl From<ErrorCollection> for Result<(), Error> {
+    fn from(e: ErrorCollection) -> Self {
+        e.into_result()
+    }
+}
+impl<T, E: Into<Error>> Extend<Result<T, E>> for ErrorCollection {
+    fn extend<I: IntoIterator<Item = Result<T, E>>>(&mut self, iter: I) {
+        for item in iter {
+            self.handle(item);
+        }
+    }
+}
+impl std::fmt::Display for ErrorCollection {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        for (idx, e) in self.0.iter().enumerate() {
+            if idx > 0 {
+                write!(f, "; ")?;
+            }
+            write!(f, "{}", e)?;
+        }
+        Ok(())
+    }
+}
+
+pub trait ResultExt<T, E>
+where
+    Self: Sized,
+{
+    fn with_kind(self, kind: ErrorKind) -> Result<T, Error>;
+    fn with_ctx<F: FnOnce(&E) -> (ErrorKind, D), D: Display + Send + Sync + 'static>(
+        self,
+        f: F,
+    ) -> Result<T, Error>;
+}
+impl<T, E> ResultExt<T, E> for Result<T, E>
+where
+    color_eyre::eyre::Error: From<E>,
+{
+    fn with_kind(self, kind: ErrorKind) -> Result<T, Error> {
+        self.map_err(|e| Error {
+            source: e.into(),
+            kind,
+            revision: None,
+        })
+    }
+
+    fn with_ctx<F: FnOnce(&E) -> (ErrorKind, D), D: Display + Send + Sync + 'static>(
+        self,
+        f: F,
+    ) -> Result<T, Error> {
+        self.map_err(|e| {
+            let (kind, ctx) = f(&e);
+            let source = color_eyre::eyre::Error::from(e);
+            let ctx = format!("{}: {}", ctx, source);
+            let source = source.wrap_err(ctx);
+            Error {
+                kind,
+                source,
+                revision: None,
+            }
+        })
+    }
+}
+
+pub trait OptionExt<T>
+where
+    Self: Sized,
+{
+    fn or_not_found(self, message: impl std::fmt::Display) -> Result<T, Error>;
+}
+impl<T> OptionExt<T> for Option<T> {
+    fn or_not_found(self, message: impl std::fmt::Display) -> Result<T, Error> {
+        self.ok_or_else(|| Error::new(eyre!("{}", message), ErrorKind::NotFound))
+    }
+}
+
+#[macro_export]
+macro_rules! ensure_code {
+    ($x:expr, $c:expr, $fmt:expr $(, $arg:expr)*) => {
+        if !($x) {
+            return Err(Error::new(color_eyre::eyre::eyre!($fmt, $($arg, )*), $c));
+        }
+    };
+}

core/models/src/id/action.rs

@@ -0,0 +1,43 @@
+use std::path::Path;
+use std::str::FromStr;
+
+use serde::{Deserialize, Serialize};
+
+use crate::{Id, InvalidId};
+
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize)]
+pub struct ActionId(Id);
+impl FromStr for ActionId {
+    type Err = InvalidId;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(ActionId(Id::try_from(s.to_owned())?))
+    }
+}
+impl AsRef<ActionId> for ActionId {
+    fn as_ref(&self) -> &ActionId {
+        self
+    }
+}
+impl std::fmt::Display for ActionId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &self.0)
+    }
+}
+impl AsRef<str> for ActionId {
+    fn as_ref(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl AsRef<Path> for ActionId {
+    fn as_ref(&self) -> &Path {
+        self.0.as_ref().as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for ActionId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::de::Deserializer<'de>,
+    {
+        Ok(ActionId(serde::Deserialize::deserialize(deserializer)?))
+    }
+}

core/models/src/id/address.rs

@@ -0,0 +1,59 @@
+use std::path::Path;
+
+use serde::{Deserialize, Deserializer, Serialize};
+
+use crate::Id;
+
+#[derive(Clone, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize)]
+pub struct AddressId(Id);
+impl From<Id> for AddressId {
+    fn from(id: Id) -> Self {
+        Self(id)
+    }
+}
+impl std::fmt::Display for AddressId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &self.0)
+    }
+}
+impl std::ops::Deref for AddressId {
+    type Target = str;
+    fn deref(&self) -> &Self::Target {
+        &*self.0
+    }
+}
+impl AsRef<str> for AddressId {
+    fn as_ref(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for AddressId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        Ok(AddressId(Deserialize::deserialize(deserializer)?))
+    }
+}
+impl AsRef<Path> for AddressId {
+    fn as_ref(&self) -> &Path {
+        self.0.as_ref().as_ref()
+    }
+}
+impl<'q> sqlx::Encode<'q, sqlx::Postgres> for AddressId {
+    fn encode_by_ref(
+        &self,
+        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
+    ) -> sqlx::encode::IsNull {
+        <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
+    }
+}
+impl sqlx::Type<sqlx::Postgres> for AddressId {
+    fn type_info() -> sqlx::postgres::PgTypeInfo {
+        <&str as sqlx::Type<sqlx::Postgres>>::type_info()
+    }
+
+    fn compatible(ty: &sqlx::postgres::PgTypeInfo) -> bool {
+        <&str as sqlx::Type<sqlx::Postgres>>::compatible(ty)
+    }
+}

core/models/src/id/health_check.rs

@@ -0,0 +1,31 @@
+use std::path::Path;
+
+use serde::{Deserialize, Deserializer, Serialize};
+
+use crate::Id;
+
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize)]
+pub struct HealthCheckId(Id);
+impl std::fmt::Display for HealthCheckId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &self.0)
+    }
+}
+impl AsRef<str> for HealthCheckId {
+    fn as_ref(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for HealthCheckId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        Ok(HealthCheckId(Deserialize::deserialize(deserializer)?))
+    }
+}
+impl AsRef<Path> for HealthCheckId {
+    fn as_ref(&self) -> &Path {
+        self.0.as_ref().as_ref()
+    }
+}

core/models/src/id/image.rs

@@ -0,0 +1,38 @@
+use std::fmt::Debug;
+use std::str::FromStr;
+
+use serde::{Deserialize, Deserializer, Serialize};
+
+use crate::{Id, InvalidId, PackageId, Version};
+
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize)]
+pub struct ImageId(Id);
+impl std::fmt::Display for ImageId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &self.0)
+    }
+}
+impl ImageId {
+    pub fn for_package(&self, pkg_id: &PackageId, pkg_version: Option<&Version>) -> String {
+        format!(
+            "start9/{}/{}:{}",
+            pkg_id,
+            self.0,
+            pkg_version.map(|v| { v.as_str() }).unwrap_or("latest")
+        )
+    }
+}
+impl FromStr for ImageId {
+    type Err = InvalidId;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(ImageId(Id::try_from(s.to_owned())?))
+    }
+}
+impl<'de> Deserialize<'de> for ImageId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        Ok(ImageId(Deserialize::deserialize(deserializer)?))
+    }
+}

core/models/src/id/interface.rs

@@ -0,0 +1,66 @@
+use std::path::Path;
+use std::str::FromStr;
+
+use serde::{Deserialize, Deserializer, Serialize};
+
+use crate::{Id, InvalidId};
+
+#[derive(Clone, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize)]
+pub struct InterfaceId(Id);
+impl FromStr for InterfaceId {
+    type Err = InvalidId;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(Self(Id::try_from(s.to_owned())?))
+    }
+}
+impl From<Id> for InterfaceId {
+    fn from(id: Id) -> Self {
+        Self(id)
+    }
+}
+impl std::fmt::Display for InterfaceId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &self.0)
+    }
+}
+impl std::ops::Deref for InterfaceId {
+    type Target = str;
+    fn deref(&self) -> &Self::Target {
+        &*self.0
+    }
+}
+impl AsRef<str> for InterfaceId {
+    fn as_ref(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for InterfaceId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        Ok(InterfaceId(Deserialize::deserialize(deserializer)?))
+    }
+}
+impl AsRef<Path> for InterfaceId {
+    fn as_ref(&self) -> &Path {
+        self.0.as_ref().as_ref()
+    }
+}
+impl<'q> sqlx::Encode<'q, sqlx::Postgres> for InterfaceId {
+    fn encode_by_ref(
+        &self,
+        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
+    ) -> sqlx::encode::IsNull {
+        <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
+    }
+}
+impl sqlx::Type<sqlx::Postgres> for InterfaceId {
+    fn type_info() -> sqlx::postgres::PgTypeInfo {
+        <&str as sqlx::Type<sqlx::Postgres>>::type_info()
+    }
+
+    fn compatible(ty: &sqlx::postgres::PgTypeInfo) -> bool {
+        <&str as sqlx::Type<sqlx::Postgres>>::compatible(ty)
+    }
+}

core/models/src/id/invalid_id.rs

@@ -0,0 +1,3 @@
+#[derive(Debug, thiserror::Error)]
+#[error("Invalid ID")]
+pub struct InvalidId;

core/models/src/id/mod.rs

@@ -0,0 +1,116 @@
+use std::borrow::Borrow;
+
+use regex::Regex;
+use serde::{Deserialize, Deserializer, Serialize, Serializer};
+use yasi::InternedString;
+
+mod action;
+mod address;
+mod health_check;
+mod image;
+mod interface;
+mod invalid_id;
+mod package;
+mod volume;
+
+pub use action::ActionId;
+pub use address::AddressId;
+pub use health_check::HealthCheckId;
+pub use image::ImageId;
+pub use interface::InterfaceId;
+pub use invalid_id::InvalidId;
+pub use package::{PackageId, SYSTEM_PACKAGE_ID};
+pub use volume::VolumeId;
+
+lazy_static::lazy_static! {
+    static ref ID_REGEX: Regex = Regex::new("^[a-z]+(-[a-z]+)*$").unwrap();
+    pub static ref SYSTEM_ID: Id = Id(InternedString::intern("x_system"));
+}
+
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Default)]
+pub struct Id(InternedString);
+impl TryFrom<InternedString> for Id {
+    type Error = InvalidId;
+    fn try_from(value: InternedString) -> Result<Self, Self::Error> {
+        if ID_REGEX.is_match(&*value) {
+            Ok(Id(value))
+        } else {
+            Err(InvalidId)
+        }
+    }
+}
+impl TryFrom<String> for Id {
+    type Error = InvalidId;
+    fn try_from(value: String) -> Result<Self, Self::Error> {
+        if ID_REGEX.is_match(&value) {
+            Ok(Id(InternedString::intern(value)))
+        } else {
+            Err(InvalidId)
+        }
+    }
+}
+impl TryFrom<&str> for Id {
+    type Error = InvalidId;
+    fn try_from(value: &str) -> Result<Self, Self::Error> {
+        if ID_REGEX.is_match(&value) {
+            Ok(Id(InternedString::intern(value)))
+        } else {
+            Err(InvalidId)
+        }
+    }
+}
+impl std::ops::Deref for Id {
+    type Target = str;
+    fn deref(&self) -> &Self::Target {
+        &*self.0
+    }
+}
+impl std::fmt::Display for Id {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &*self.0)
+    }
+}
+impl AsRef<str> for Id {
+    fn as_ref(&self) -> &str {
+        &*self.0
+    }
+}
+impl Borrow<str> for Id {
+    fn borrow(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for Id {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let unchecked: InternedString = Deserialize::deserialize(deserializer)?;
+        Id::try_from(unchecked).map_err(serde::de::Error::custom)
+    }
+}
+impl Serialize for Id {
+    fn serialize<Ser>(&self, serializer: Ser) -> Result<Ser::Ok, Ser::Error>
+    where
+        Ser: Serializer,
+    {
+        serializer.serialize_str(&*self)
+    }
+}
+impl<'q> sqlx::Encode<'q, sqlx::Postgres> for Id {
+    fn encode_by_ref(
+        &self,
+        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
+    ) -> sqlx::encode::IsNull {
+        <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
+    }
+}
+impl sqlx::Type<sqlx::Postgres> for Id {
+    fn type_info() -> sqlx::postgres::PgTypeInfo {
+        <&str as sqlx::Type<sqlx::Postgres>>::type_info()
+    }
+
+    fn compatible(ty: &sqlx::postgres::PgTypeInfo) -> bool {
+        <&str as sqlx::Type<sqlx::Postgres>>::compatible(ty)
+    }
+}

core/models/src/id/package.rs

@@ -0,0 +1,88 @@
+use std::borrow::Borrow;
+use std::path::Path;
+use std::str::FromStr;
+
+use serde::{Deserialize, Serialize, Serializer};
+
+use crate::{Id, InvalidId, SYSTEM_ID};
+
+lazy_static::lazy_static! {
+    pub static ref SYSTEM_PACKAGE_ID: PackageId = PackageId(SYSTEM_ID.clone());
+}
+#[derive(Clone, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct PackageId(Id);
+impl FromStr for PackageId {
+    type Err = InvalidId;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(PackageId(Id::try_from(s.to_owned())?))
+    }
+}
+impl From<Id> for PackageId {
+    fn from(id: Id) -> Self {
+        PackageId(id)
+    }
+}
+impl std::ops::Deref for PackageId {
+    type Target = str;
+    fn deref(&self) -> &Self::Target {
+        &*self.0
+    }
+}
+impl AsRef<PackageId> for PackageId {
+    fn as_ref(&self) -> &PackageId {
+        self
+    }
+}
+impl std::fmt::Display for PackageId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", &self.0)
+    }
+}
+impl AsRef<str> for PackageId {
+    fn as_ref(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl Borrow<str> for PackageId {
+    fn borrow(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+impl AsRef<Path> for PackageId {
+    fn as_ref(&self) -> &Path {
+        self.0.as_ref().as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for PackageId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::de::Deserializer<'de>,
+    {
+        Ok(PackageId(Deserialize::deserialize(deserializer)?))
+    }
+}
+impl Serialize for PackageId {
+    fn serialize<Ser>(&self, serializer: Ser) -> Result<Ser::Ok, Ser::Error>
+    where
+        Ser: Serializer,
+    {
+        Serialize::serialize(&self.0, serializer)
+    }
+}
+impl<'q> sqlx::Encode<'q, sqlx::Postgres> for PackageId {
+    fn encode_by_ref(
+        &self,
+        buf: &mut <sqlx::Postgres as sqlx::database::HasArguments<'q>>::ArgumentBuffer,
+    ) -> sqlx::encode::IsNull {
+        <&str as sqlx::Encode<'q, sqlx::Postgres>>::encode_by_ref(&&**self, buf)
+    }
+}
+impl sqlx::Type<sqlx::Postgres> for PackageId {
+    fn type_info() -> sqlx::postgres::PgTypeInfo {
+        <&str as sqlx::Type<sqlx::Postgres>>::type_info()
+    }
+
+    fn compatible(ty: &sqlx::postgres::PgTypeInfo) -> bool {
+        <&str as sqlx::Type<sqlx::Postgres>>::compatible(ty)
+    }
+}

core/models/src/id/volume.rs

@@ -0,0 +1,58 @@
+use std::borrow::Borrow;
+use std::path::Path;
+
+use serde::{Deserialize, Deserializer, Serialize};
+
+use crate::Id;
+
+#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub enum VolumeId {
+    Backup,
+    Custom(Id),
+}
+impl std::fmt::Display for VolumeId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            VolumeId::Backup => write!(f, "BACKUP"),
+            VolumeId::Custom(id) => write!(f, "{}", id),
+        }
+    }
+}
+impl AsRef<str> for VolumeId {
+    fn as_ref(&self) -> &str {
+        match self {
+            VolumeId::Backup => "BACKUP",
+            VolumeId::Custom(id) => id.as_ref(),
+        }
+    }
+}
+impl Borrow<str> for VolumeId {
+    fn borrow(&self) -> &str {
+        self.as_ref()
+    }
+}
+impl AsRef<Path> for VolumeId {
+    fn as_ref(&self) -> &Path {
+        AsRef::<str>::as_ref(self).as_ref()
+    }
+}
+impl<'de> Deserialize<'de> for VolumeId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let unchecked: String = Deserialize::deserialize(deserializer)?;
+        Ok(match unchecked.as_ref() {
+            "BACKUP" => VolumeId::Backup,
+            _ => VolumeId::Custom(Id::try_from(unchecked).map_err(serde::de::Error::custom)?),
+        })
+    }
+}
+impl Serialize for VolumeId {
+    fn serialize<Ser>(&self, serializer: Ser) -> Result<Ser::Ok, Ser::Error>
+    where
+        Ser: serde::Serializer,
+    {
+        serializer.serialize_str(self.as_ref())
+    }
+}

core/models/src/js_engine_types.rs

@@ -0,0 +1,164 @@
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::Arc;
+use std::time::Duration;
+
+use color_eyre::eyre::bail;
+use container_init::{Input, Output, ProcessId, RpcId};
+use tokio::sync::mpsc::{UnboundedReceiver, UnboundedSender};
+use tokio::sync::Mutex;
+
+/// Used by the js-executor, it is the ability to just create a command in an already running exec
+pub type ExecCommand = Arc<
+    dyn Fn(
+            String,
+            Vec<String>,
+            UnboundedSender<container_init::Output>,
+            Option<Duration>,
+        ) -> Pin<Box<dyn Future<Output = Result<RpcId, String>> + 'static>>
+        + Send
+        + Sync
+        + 'static,
+>;
+
+/// Used by the js-executor, it is the ability to just create a command in an already running exec
+pub type SendKillSignal = Arc<
+    dyn Fn(RpcId, u32) -> Pin<Box<dyn Future<Output = Result<(), String>> + 'static>>
+        + Send
+        + Sync
+        + 'static,
+>;
+
+pub trait CommandInserter {
+    fn insert_command(
+        &self,
+        command: String,
+        args: Vec<String>,
+        sender: UnboundedSender<container_init::Output>,
+        timeout: Option<Duration>,
+    ) -> Pin<Box<dyn Future<Output = Option<RpcId>>>>;
+
+    fn send_signal(&self, id: RpcId, command: u32) -> Pin<Box<dyn Future<Output = ()>>>;
+}
+
+pub type ArcCommandInserter = Arc<Mutex<Option<Box<dyn CommandInserter>>>>;
+
+pub struct ExecutingCommand {
+    rpc_id: RpcId,
+    /// Will exist until killed
+    command_inserter: Arc<Mutex<Option<ArcCommandInserter>>>,
+    owned_futures: Arc<Mutex<Vec<Pin<Box<dyn Future<Output = ()>>>>>>,
+}
+
+impl ExecutingCommand {
+    pub async fn new(
+        command_inserter: ArcCommandInserter,
+        command: String,
+        args: Vec<String>,
+        timeout: Option<Duration>,
+    ) -> Result<ExecutingCommand, color_eyre::Report> {
+        let (sender, receiver) = tokio::sync::mpsc::unbounded_channel::<Output>();
+        let rpc_id = {
+            let locked_command_inserter = command_inserter.lock().await;
+            let locked_command_inserter = match &*locked_command_inserter {
+                Some(a) => a,
+                None => bail!("Expecting containers.main in the package manifest".to_string()),
+            };
+            match locked_command_inserter
+                .insert_command(command, args, sender, timeout)
+                .await
+            {
+                Some(a) => a,
+                None => bail!("Couldn't get command started ".to_string()),
+            }
+        };
+        let executing_commands = ExecutingCommand {
+            rpc_id,
+            command_inserter: Arc::new(Mutex::new(Some(command_inserter.clone()))),
+            owned_futures: Default::default(),
+        };
+        // let waiting = self.wait()
+        Ok(executing_commands)
+    }
+
+    async fn wait(
+        rpc_id: RpcId,
+        mut outputs: UnboundedReceiver<Output>,
+    ) -> Result<String, (Option<i32>, String)> {
+        let (process_id_send, process_id_recv) = tokio::sync::oneshot::channel::<ProcessId>();
+        let mut answer = String::new();
+        let mut command_error = String::new();
+        let mut status: Option<i32> = None;
+        let mut process_id_send = Some(process_id_send);
+        while let Some(output) = outputs.recv().await {
+            match output {
+                Output::ProcessId(process_id) => {
+                    if let Some(process_id_send) = process_id_send.take() {
+                        if let Err(err) = process_id_send.send(process_id) {
+                            tracing::error!(
+                                "Could not get a process id {process_id:?} sent for {rpc_id:?}"
+                            );
+                            tracing::debug!("{err:?}");
+                        }
+                    }
+                }
+                Output::Line(value) => {
+                    answer.push_str(&value);
+                    answer.push('\n');
+                }
+                Output::Error(error) => {
+                    command_error.push_str(&error);
+                    command_error.push('\n');
+                }
+                Output::Done(error_code) => {
+                    status = error_code;
+                    break;
+                }
+            }
+        }
+        if !command_error.is_empty() {
+            return Err((status, command_error));
+        }
+
+        Ok(answer)
+    }
+
+    async fn send_signal(&self, signal: u32) {
+        let locked = self.command_inserter.lock().await;
+        let inner = match &*locked {
+            Some(a) => a,
+            None => return,
+        };
+        let locked = inner.lock().await;
+        let command_inserter = match &*locked {
+            Some(a) => a,
+            None => return,
+        };
+        command_inserter.send_signal(self.rpc_id, signal);
+    }
+    /// Should only be called when output::done
+    async fn killed(&self) {
+        *self.owned_futures.lock().await = Default::default();
+        *self.command_inserter.lock().await = Default::default();
+    }
+    pub fn rpc_id(&self) -> RpcId {
+        self.rpc_id
+    }
+}
+
+impl Drop for ExecutingCommand {
+    fn drop(&mut self) {
+        let command_inserter = self.command_inserter.clone();
+        let rpc_id = self.rpc_id.clone();
+        tokio::spawn(async move {
+            let command_inserter_lock = command_inserter.lock().await;
+            let command_inserter = match &*command_inserter_lock {
+                Some(a) => a,
+                None => {
+                    return;
+                }
+            };
+            command_inserter.send_kill_command(rpc_id, 9).await;
+        });
+    }
+}

core/models/src/lib.rs

@@ -0,0 +1,13 @@
+mod data_url;
+mod errors;
+mod id;
+mod mime;
+mod procedure_name;
+mod version;
+
+pub use data_url::*;
+pub use errors::*;
+pub use id::*;
+pub use mime::*;
+pub use procedure_name::*;
+pub use version::*;

core/models/src/mime.rs

@@ -0,0 +1,47 @@
+pub fn mime(extension: &str) -> Option<&'static str> {
+    match extension {
+        "apng" => Some("image/apng"),
+        "avif" => Some("image/avif"),
+        "flif" => Some("image/flif"),
+        "gif" => Some("image/gif"),
+        "jpg" | "jpeg" | "jfif" | "pjpeg" | "pjp" => Some("image/jpeg"),
+        "jxl" => Some("image/jxl"),
+        "png" => Some("image/png"),
+        "svg" => Some("image/svg+xml"),
+        "webp" => Some("image/webp"),
+        "mng" | "x-mng" => Some("image/x-mng"),
+        "css" => Some("text/css"),
+        "csv" => Some("text/csv"),
+        "html" => Some("text/html"),
+        "php" => Some("text/php"),
+        "plain" | "md" | "txt" => Some("text/plain"),
+        "xml" => Some("text/xml"),
+        "js" => Some("text/javascript"),
+        "wasm" => Some("application/wasm"),
+        _ => None,
+    }
+}
+
+pub fn unmime(mime: &str) -> Option<&'static str> {
+    match mime {
+        "image/apng" => Some("apng"),
+        "image/avif" => Some("avif"),
+        "image/flif" => Some("flif"),
+        "image/gif" => Some("gif"),
+        "jpg" | "jpeg" | "jfif" | "pjpeg" | "image/jpeg" => Some("pjp"),
+        "image/jxl" => Some("jxl"),
+        "image/png" => Some("png"),
+        "image/svg+xml" => Some("svg"),
+        "image/webp" => Some("webp"),
+        "mng" | "image/x-mng" => Some("x-mng"),
+        "text/css" => Some("css"),
+        "text/csv" => Some("csv"),
+        "text/html" => Some("html"),
+        "text/php" => Some("php"),
+        "plain" | "md" | "text/plain" => Some("txt"),
+        "text/xml" => Some("xml"),
+        "text/javascript" => Some("js"),
+        "application/wasm" => Some("wasm"),
+        _ => None,
+    }
+}

core/models/src/procedure_name.rs

@@ -0,0 +1,57 @@
+use serde::{Deserialize, Serialize};
+
+use crate::{ActionId, HealthCheckId, PackageId};
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum ProcedureName {
+    Main, // Usually just run container
+    CreateBackup,
+    RestoreBackup,
+    GetConfig,
+    SetConfig,
+    Migration,
+    Properties,
+    LongRunning,
+    Check(PackageId),
+    AutoConfig(PackageId),
+    Health(HealthCheckId),
+    Action(ActionId),
+    Signal,
+}
+
+impl ProcedureName {
+    pub fn docker_name(&self) -> Option<String> {
+        match self {
+            ProcedureName::Main => None,
+            ProcedureName::LongRunning => None,
+            ProcedureName::CreateBackup => Some("CreateBackup".to_string()),
+            ProcedureName::RestoreBackup => Some("RestoreBackup".to_string()),
+            ProcedureName::GetConfig => Some("GetConfig".to_string()),
+            ProcedureName::SetConfig => Some("SetConfig".to_string()),
+            ProcedureName::Migration => Some("Migration".to_string()),
+            ProcedureName::Properties => Some(format!("Properties-{}", rand::random::<u64>())),
+            ProcedureName::Health(id) => Some(format!("{}Health", id)),
+            ProcedureName::Action(id) => Some(format!("{}Action", id)),
+            ProcedureName::Check(_) => None,
+            ProcedureName::AutoConfig(_) => None,
+            ProcedureName::Signal => None,
+        }
+    }
+    pub fn js_function_name(&self) -> Option<String> {
+        match self {
+            ProcedureName::Main => Some("/main".to_string()),
+            ProcedureName::LongRunning => None,
+            ProcedureName::CreateBackup => Some("/createBackup".to_string()),
+            ProcedureName::RestoreBackup => Some("/restoreBackup".to_string()),
+            ProcedureName::GetConfig => Some("/getConfig".to_string()),
+            ProcedureName::SetConfig => Some("/setConfig".to_string()),
+            ProcedureName::Migration => Some("/migration".to_string()),
+            ProcedureName::Properties => Some("/properties".to_string()),
+            ProcedureName::Health(id) => Some(format!("/health/{}", id)),
+            ProcedureName::Action(id) => Some(format!("/action/{}", id)),
+            ProcedureName::Check(id) => Some(format!("/dependencies/{}/check", id)),
+            ProcedureName::AutoConfig(id) => Some(format!("/dependencies/{}/autoConfigure", id)),
+            ProcedureName::Signal => Some("/handleSignal".to_string()),
+        }
+    }
+}

core/models/src/version.rs

@@ -0,0 +1,106 @@
+use std::hash::{Hash, Hasher};
+use std::ops::Deref;
+use std::str::FromStr;
+
+use serde::{Deserialize, Deserializer, Serialize, Serializer};
+
+#[derive(Debug, Clone)]
+pub struct Version {
+    version: emver::Version,
+    string: String,
+}
+impl Version {
+    pub fn as_str(&self) -> &str {
+        self.string.as_str()
+    }
+    pub fn into_version(self) -> emver::Version {
+        self.version
+    }
+}
+impl std::fmt::Display for Version {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.string)
+    }
+}
+impl std::str::FromStr for Version {
+    type Err = <emver::Version as FromStr>::Err;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(Version {
+            string: s.to_owned(),
+            version: s.parse()?,
+        })
+    }
+}
+impl From<emver::Version> for Version {
+    fn from(v: emver::Version) -> Self {
+        Version {
+            string: v.to_string(),
+            version: v,
+        }
+    }
+}
+impl From<Version> for emver::Version {
+    fn from(v: Version) -> Self {
+        v.version
+    }
+}
+impl Default for Version {
+    fn default() -> Self {
+        Self::from(emver::Version::default())
+    }
+}
+impl Deref for Version {
+    type Target = emver::Version;
+    fn deref(&self) -> &Self::Target {
+        &self.version
+    }
+}
+impl AsRef<emver::Version> for Version {
+    fn as_ref(&self) -> &emver::Version {
+        &self.version
+    }
+}
+impl AsRef<str> for Version {
+    fn as_ref(&self) -> &str {
+        self.as_str()
+    }
+}
+impl PartialEq for Version {
+    fn eq(&self, other: &Version) -> bool {
+        self.version.eq(&other.version)
+    }
+}
+impl Eq for Version {}
+impl PartialOrd for Version {
+    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
+        self.version.partial_cmp(&other.version)
+    }
+}
+impl Ord for Version {
+    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
+        self.version.cmp(&other.version)
+    }
+}
+impl Hash for Version {
+    fn hash<H: Hasher>(&self, state: &mut H) {
+        self.version.hash(state)
+    }
+}
+impl<'de> Deserialize<'de> for Version {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        let string = String::deserialize(deserializer)?;
+        let version = emver::Version::from_str(&string).map_err(::serde::de::Error::custom)?;
+        Ok(Self { string, version })
+    }
+}
+impl Serialize for Version {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        self.string.serialize(serializer)
+    }
+}

core/startos/.sqlx/query-1ce5254f27de971fd87f5ab66d300f2b22433c86617a0dbf796bf2170186dd2e.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Bytea"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "1ce5254f27de971fd87f5ab66d300f2b22433c86617a0dbf796bf2170186dd2e"
+}

core/startos/.sqlx/query-21471490cdc3adb206274cc68e1ea745ffa5da4479478c1fd2158a45324b1930.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM ssh_keys WHERE fingerprint = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "21471490cdc3adb206274cc68e1ea745ffa5da4479478c1fd2158a45324b1930"
+}

core/startos/.sqlx/query-28ea34bbde836e0618c5fc9bb7c36e463c20c841a7d6a0eb15be0f24f4a928ec.json

@@ -0,0 +1,40 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT hostname, path, username, password FROM cifs_shares WHERE id = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "hostname",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "path",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "username",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 3,
+        "name": "password",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int4"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "28ea34bbde836e0618c5fc9bb7c36e463c20c841a7d6a0eb15be0f24f4a928ec"
+}

core/startos/.sqlx/query-350ab82048fb4a049042e4fdbe1b8c606ca400e43e31b9a05d2937217e0f6962.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM tor WHERE package = $1 AND interface = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "350ab82048fb4a049042e4fdbe1b8c606ca400e43e31b9a05d2937217e0f6962"
+}

core/startos/.sqlx/query-4099028a5c0de578255bf54a67cef6cb0f1e9a4e158260700f1639dd4b438997.json

@@ -0,0 +1,34 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT * FROM ssh_keys WHERE fingerprint = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "fingerprint",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "openssh_pubkey",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "created_at",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "4099028a5c0de578255bf54a67cef6cb0f1e9a4e158260700f1639dd4b438997"
+}

core/startos/.sqlx/query-4691e3a2ce80b59009ac17124f54f925f61dc5ea371903e62cdffa5d7b67ca96.json

@@ -0,0 +1,50 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT * FROM session WHERE logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "logged_in",
+        "type_info": "Timestamp"
+      },
+      {
+        "ordinal": 2,
+        "name": "logged_out",
+        "type_info": "Timestamp"
+      },
+      {
+        "ordinal": 3,
+        "name": "last_active",
+        "type_info": "Timestamp"
+      },
+      {
+        "ordinal": 4,
+        "name": "user_agent",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 5,
+        "name": "metadata",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false,
+      true,
+      false,
+      true,
+      false
+    ]
+  },
+  "hash": "4691e3a2ce80b59009ac17124f54f925f61dc5ea371903e62cdffa5d7b67ca96"
+}

core/startos/.sqlx/query-4bcfbefb1eb3181343871a1cd7fc3afb81c2be5c681cfa8b4be0ce70610e9c3a.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE session SET logged_out = CURRENT_TIMESTAMP WHERE id = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "4bcfbefb1eb3181343871a1cd7fc3afb81c2be5c681cfa8b4be0ce70610e9c3a"
+}

core/startos/.sqlx/query-629be61c3c341c131ddbbff0293a83dbc6afd07cae69d246987f62cf0cc35c2a.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT password FROM account",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "password",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "629be61c3c341c131ddbbff0293a83dbc6afd07cae69d246987f62cf0cc35c2a"
+}

core/startos/.sqlx/query-687688055e63d27123cdc89a5bbbd8361776290a9411d527eaf1fdb40bef399d.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT key FROM tor WHERE package = $1 AND interface = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "key",
+        "type_info": "Bytea"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "687688055e63d27123cdc89a5bbbd8361776290a9411d527eaf1fdb40bef399d"
+}

core/startos/.sqlx/query-6d35ccf780fb2bb62586dd1d3df9c1550a41ee580dad3f49d35cb843ebef10ca.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE session SET last_active = CURRENT_TIMESTAMP WHERE id = $1 AND logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "6d35ccf780fb2bb62586dd1d3df9c1550a41ee580dad3f49d35cb843ebef10ca"
+}

core/startos/.sqlx/query-770c1017734720453dc87b58c385b987c5af5807151ff71a59000014586752e0.json

@@ -0,0 +1,24 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO UPDATE SET package = EXCLUDED.package RETURNING key",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "key",
+        "type_info": "Bytea"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Bytea"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "770c1017734720453dc87b58c385b987c5af5807151ff71a59000014586752e0"
+}

core/startos/.sqlx/query-7b64f032d507e8ffe37c41f4c7ad514a66c421a11ab04c26d89a7aa8f6b67210.json

@@ -0,0 +1,65 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id, package_id, created_at, code, level, title, message, data FROM notifications WHERE id < $1 ORDER BY id DESC LIMIT $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "package_id",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "created_at",
+        "type_info": "Timestamp"
+      },
+      {
+        "ordinal": 3,
+        "name": "code",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 4,
+        "name": "level",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 5,
+        "name": "title",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 6,
+        "name": "message",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 7,
+        "name": "data",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int4",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      true,
+      false,
+      false,
+      false,
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "7b64f032d507e8ffe37c41f4c7ad514a66c421a11ab04c26d89a7aa8f6b67210"
+}

core/startos/.sqlx/query-7c7a3549c997eb75bf964ea65fbb98a73045adf618696cd838d79203ef5383fb.json

@@ -0,0 +1,19 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n            INSERT INTO account (\n                id,\n                server_id,\n                hostname,\n                password,\n                network_key,\n                root_ca_key_pem,\n                root_ca_cert_pem\n            ) VALUES (\n                0, $1, $2, $3, $4, $5, $6\n            ) ON CONFLICT (id) DO UPDATE SET\n                server_id = EXCLUDED.server_id,\n                hostname = EXCLUDED.hostname,\n                password = EXCLUDED.password,\n                network_key = EXCLUDED.network_key,\n                root_ca_key_pem = EXCLUDED.root_ca_key_pem,\n                root_ca_cert_pem = EXCLUDED.root_ca_cert_pem\n            ",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text",
+        "Bytea",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "7c7a3549c997eb75bf964ea65fbb98a73045adf618696cd838d79203ef5383fb"
+}

core/startos/.sqlx/query-7e0649d839927e57fa03ee51a2c9f96a8bdb0fc97ee8a3c6df1069e1e2b98576.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM tor WHERE package = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "7e0649d839927e57fa03ee51a2c9f96a8bdb0fc97ee8a3c6df1069e1e2b98576"
+}

core/startos/.sqlx/query-8951b9126fbf60dbb5997241e11e3526b70bccf3e407327917294a993bc17ed5.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO tor (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Bytea"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "8951b9126fbf60dbb5997241e11e3526b70bccf3e407327917294a993bc17ed5"
+}

core/startos/.sqlx/query-94d471bb374b4965c6cbedf8c17bbf6bea226d38efaf6559923c79a36d5ca08c.json

@@ -0,0 +1,64 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id, package_id, created_at, code, level, title, message, data FROM notifications ORDER BY id DESC LIMIT $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "package_id",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "created_at",
+        "type_info": "Timestamp"
+      },
+      {
+        "ordinal": 3,
+        "name": "code",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 4,
+        "name": "level",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 5,
+        "name": "title",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 6,
+        "name": "message",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 7,
+        "name": "data",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      true,
+      false,
+      false,
+      false,
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "94d471bb374b4965c6cbedf8c17bbf6bea226d38efaf6559923c79a36d5ca08c"
+}

core/startos/.sqlx/query-95c4ab4c645f3302568c6ff13d85ab58252362694cf0f56999bf60194d20583a.json

@@ -0,0 +1,44 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id, hostname, path, username, password FROM cifs_shares",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "hostname",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "path",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 3,
+        "name": "username",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 4,
+        "name": "password",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false,
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "95c4ab4c645f3302568c6ff13d85ab58252362694cf0f56999bf60194d20583a"
+}

core/startos/.sqlx/query-a60d6e66719325b08dc4ecfacaf337527233c84eee758ac9be967906e5841d27.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM cifs_shares WHERE id = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int4"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "a60d6e66719325b08dc4ecfacaf337527233c84eee758ac9be967906e5841d27"
+}

core/startos/.sqlx/query-a6b0c8909a3a5d6d9156aebfb359424e6b5a1d1402e028219e21726f1ebd282e.json

@@ -0,0 +1,32 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT fingerprint, openssh_pubkey, created_at FROM ssh_keys",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "fingerprint",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "openssh_pubkey",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "created_at",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "a6b0c8909a3a5d6d9156aebfb359424e6b5a1d1402e028219e21726f1ebd282e"
+}

core/startos/.sqlx/query-b1147beaaabbed89f2ab8c1e13ec4393a9a8fde2833cf096af766a979d94dee6.json

@@ -0,0 +1,18 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE cifs_shares SET hostname = $1, path = $2, username = $3, password = $4 WHERE id = $5",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text",
+        "Text",
+        "Int4"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "b1147beaaabbed89f2ab8c1e13ec4393a9a8fde2833cf096af766a979d94dee6"
+}

core/startos/.sqlx/query-b203820ee1c553a4b246eac74b79bd10d5717b2a0ddecf22330b7d531aac7c5d.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM network_keys WHERE package = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "b203820ee1c553a4b246eac74b79bd10d5717b2a0ddecf22330b7d531aac7c5d"
+}

core/startos/.sqlx/query-b81592b3a74940ab56d41537484090d45cfa4c85168a587b1a41dc5393cccea1.json

@@ -0,0 +1,12 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE account SET tor_key = NULL, network_key = gen_random_bytes(32)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": []
+  },
+  "hash": "b81592b3a74940ab56d41537484090d45cfa4c85168a587b1a41dc5393cccea1"
+}

core/startos/.sqlx/query-d5117054072476377f3c4f040ea429d4c9b2cf534e76f35c80a2bf60e8599cca.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT openssh_pubkey FROM ssh_keys",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "openssh_pubkey",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "d5117054072476377f3c4f040ea429d4c9b2cf534e76f35c80a2bf60e8599cca"
+}

core/startos/.sqlx/query-da71f94b29798d1738d2b10b9a721ea72db8cfb362e7181c8226d9297507c62b.json

@@ -0,0 +1,19 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO notifications (package_id, code, level, title, message, data) VALUES ($1, $2, $3, $4, $5, $6)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Int4",
+        "Text",
+        "Text",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "da71f94b29798d1738d2b10b9a721ea72db8cfb362e7181c8226d9297507c62b"
+}

core/startos/.sqlx/query-dfc23b7e966c3853284753a7e934351ba0cae3825988b3e0ecd3b6781bcff524.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM network_keys WHERE package = $1 AND interface = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "dfc23b7e966c3853284753a7e934351ba0cae3825988b3e0ecd3b6781bcff524"
+}

core/startos/.sqlx/query-e185203cf84e43b801dfb23b4159e34aeaef1154dcd3d6811ab504915497ccf7.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM notifications WHERE id = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int4"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "e185203cf84e43b801dfb23b4159e34aeaef1154dcd3d6811ab504915497ccf7"
+}

core/startos/.sqlx/query-e545696735f202f9d13cf22a561f3ff3f9aed7f90027a9ba97634bcb47d772f0.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT tor_key FROM account WHERE id = 0",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "tor_key",
+        "type_info": "Bytea"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      true
+    ]
+  },
+  "hash": "e545696735f202f9d13cf22a561f3ff3f9aed7f90027a9ba97634bcb47d772f0"
+}

core/startos/.sqlx/query-e5843c5b0e7819b29aa1abf2266799bd4f82e761837b526a0972c3d4439a264d.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO session (id, user_agent, metadata) VALUES ($1, $2, $3)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "e5843c5b0e7819b29aa1abf2266799bd4f82e761837b526a0972c3d4439a264d"
+}

core/startos/.sqlx/query-e95322a8e2ae3b93f1e974b24c0b81803f1e9ec9e8ebbf15cafddfc1c5a028ed.json

@@ -0,0 +1,40 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n                SELECT\n                    network_keys.package,\n                    network_keys.interface,\n                    network_keys.key,\n                    tor.key AS \"tor_key?\"\n                FROM\n                    network_keys\n                LEFT JOIN\n                    tor\n                ON\n                    network_keys.package = tor.package\n                AND\n                    network_keys.interface = tor.interface\n                WHERE\n                    network_keys.package = $1\n            ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "package",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "interface",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "key",
+        "type_info": "Bytea"
+      },
+      {
+        "ordinal": 3,
+        "name": "tor_key?",
+        "type_info": "Bytea"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "e95322a8e2ae3b93f1e974b24c0b81803f1e9ec9e8ebbf15cafddfc1c5a028ed"
+}

core/startos/.sqlx/query-eb750adaa305bdbf3c5b70aaf59139c7b7569602adb58f2d6b3a94da4f167b0a.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM notifications WHERE id < $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int4"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "eb750adaa305bdbf3c5b70aaf59139c7b7569602adb58f2d6b3a94da4f167b0a"
+}

core/startos/.sqlx/query-ecc765d8205c0876956f95f76944ac6a5f34dd820c4073b7728c7067aab9fded.json

@@ -0,0 +1,25 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO cifs_shares (hostname, path, username, password) VALUES ($1, $2, $3, $4) RETURNING id",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "ecc765d8205c0876956f95f76944ac6a5f34dd820c4073b7728c7067aab9fded"
+}

core/startos/.sqlx/query-f6d1c5ef0f9d9577bea8382318967b9deb46da75788c7fe6082b43821c22d556.json

@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO ssh_keys (fingerprint, openssh_pubkey, created_at) VALUES ($1, $2, $3)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "f6d1c5ef0f9d9577bea8382318967b9deb46da75788c7fe6082b43821c22d556"
+}

core/startos/.sqlx/query-f7d2dae84613bcef330f7403352cc96547f3f6dbec11bf2eadfaf53ad8ab51b5.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT network_key FROM account WHERE id = 0",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "network_key",
+        "type_info": "Bytea"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "f7d2dae84613bcef330f7403352cc96547f3f6dbec11bf2eadfaf53ad8ab51b5"
+}

core/startos/.sqlx/query-fe6e4f09f3028e5b6b6259e86cbad285680ce157aae9d7837ac020c8b2945e7f.json

@@ -0,0 +1,62 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT * FROM account WHERE id = 0",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "password",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 2,
+        "name": "tor_key",
+        "type_info": "Bytea"
+      },
+      {
+        "ordinal": 3,
+        "name": "server_id",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 4,
+        "name": "hostname",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 5,
+        "name": "network_key",
+        "type_info": "Bytea"
+      },
+      {
+        "ordinal": 6,
+        "name": "root_ca_key_pem",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 7,
+        "name": "root_ca_cert_pem",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false,
+      true,
+      true,
+      true,
+      false,
+      false,
+      false
+    ]
+  },
+  "hash": "fe6e4f09f3028e5b6b6259e86cbad285680ce157aae9d7837ac020c8b2945e7f"
+}

core/startos/Cargo.toml

@@ -0,0 +1,169 @@
+[package]
+authors = ["Aiden McClelland <me@drbonez.dev>"]
+description = "The core of StartOS"
+documentation = "https://docs.rs/start-os"
+edition = "2021"
+keywords = [
+  "self-hosted",
+  "raspberry-pi",
+  "privacy",
+  "bitcoin",
+  "full-node",
+  "lightning",
+]
+name = "start-os"
+readme = "README.md"
+repository = "https://github.com/Start9Labs/start-os"
+version = "0.3.5-rev.1"
+license = "MIT"
+
+[lib]
+name = "startos"
+path = "src/lib.rs"
+
+[[bin]]
+name = "startbox"
+path = "src/main.rs"
+
+[features]
+avahi = ["avahi-sys"]
+avahi-alias = ["avahi"]
+cli = []
+daemon = []
+default = ["cli", "sdk", "daemon", "js-engine"]
+dev = []
+docker = []
+sdk = []
+unstable = ["console-subscriber", "tokio/tracing"]
+
+[dependencies]
+aes = { version = "0.7.5", features = ["ctr"] }
+async-compression = { version = "0.4.4", features = [
+  "gzip",
+  "brotli",
+  "tokio",
+] }
+async-stream = "0.3.5"
+async-trait = "0.1.74"
+avahi-sys = { git = "https://github.com/Start9Labs/avahi-sys", version = "0.10.0", branch = "feature/dynamic-linking", features = [
+  "dynamic",
+], optional = true }
+base32 = "0.4.0"
+base64 = "0.21.4"
+base64ct = "1.6.0"
+basic-cookies = "0.1.4"
+bytes = "1"
+chrono = { version = "0.4.31", features = ["serde"] }
+clap = "3.2.25"
+color-eyre = "0.6.2"
+console = "0.15.7"
+console-subscriber = { version = "0.2", optional = true }
+cookie = "0.18.0"
+cookie_store = "0.20.0"
+current_platform = "0.2.0"
+digest = "0.10.7"
+divrem = "1.0.0"
+ed25519 = { version = "2.2.3", features = ["pkcs8", "pem", "alloc"] }
+ed25519-dalek = { version = "2.0.0", features = [
+  "serde",
+  "zeroize",
+  "rand_core",
+  "digest",
+] }
+ed25519-dalek-v1 = { package = "ed25519-dalek", version = "1" }
+container-init = { path = "../container-init" }
+emver = { git = "https://github.com/Start9Labs/emver-rs.git", rev = "61cf0bc96711b4d6f3f30df8efef025e0cc02bad", features = [
+  "serde",
+] }
+fd-lock-rs = "0.1.4"
+futures = "0.3.28"
+gpt = "3.1.0"
+helpers = { path = "../helpers" }
+hex = "0.4.3"
+hmac = "0.12.1"
+http = "0.2.9"
+hyper = { version = "0.14.27", features = ["full"] }
+hyper-ws-listener = "0.3.0"
+imbl = "2.0.2"
+imbl-value = { git = "https://github.com/Start9Labs/imbl-value.git" }
+include_dir = "0.7.3"
+indexmap = { version = "2.0.2", features = ["serde"] }
+indicatif = { version = "0.17.7", features = ["tokio"] }
+ipnet = { version = "2.8.0", features = ["serde"] }
+iprange = { version = "0.6.7", features = ["serde"] }
+isocountry = "0.3.2"
+itertools = "0.11.0"
+jaq-core = "0.10.1"
+jaq-std = "0.10.0"
+josekit = "0.8.4"
+js-engine = { path = '../js-engine', optional = true }
+jsonpath_lib = { git = "https://github.com/Start9Labs/jsonpath.git" }
+lazy_static = "1.4.0"
+libc = "0.2.149"
+log = "0.4.20"
+mbrman = "0.5.2"
+models = { version = "*", path = "../models" }
+new_mime_guess = "4"
+nix = { version = "0.27.1", features = ["user", "process", "signal", "fs"] }
+nom = "7.1.3"
+num = "0.4.1"
+num_enum = "0.7.0"
+openssh-keys = "0.6.2"
+openssl = { version = "0.10.57", features = ["vendored"] }
+p256 = { version = "0.13.2", features = ["pem"] }
+patch-db = { version = "*", path = "../../patch-db/patch-db", features = [
+  "trace",
+] }
+pbkdf2 = "0.12.2"
+pin-project = "1.1.3"
+pkcs8 = { version = "0.10.2", features = ["std"] }
+prettytable-rs = "0.10.0"
+proptest = "1.3.1"
+proptest-derive = "0.4.0"
+rand = { version = "0.8.5", features = ["std"] }
+regex = "1.10.2"
+reqwest = { version = "0.11.22", features = ["stream", "json", "socks"] }
+reqwest_cookie_store = "0.6.0"
+rpassword = "7.2.0"
+rpc-toolkit = "0.2.2"
+rust-argon2 = "2.0.0"
+scopeguard = "1.1" # because avahi-sys fucks your shit up
+semver = { version = "1.0.20", features = ["serde"] }
+serde = { version = "1.0", features = ["derive", "rc"] }
+serde_cbor = { package = "ciborium", version = "0.2.1" }
+serde_json = "1.0"
+serde_toml = { package = "toml", version = "0.8.2" }
+serde_with = { version = "3.4.0", features = ["macros", "json"] }
+serde_yaml = "0.9.25"
+sha2 = "0.10.2"
+simple-logging = "2.0.2"
+sqlx = { version = "0.7.2", features = [
+  "chrono",
+  "runtime-tokio-rustls",
+  "postgres",
+] }
+sscanf = "0.4.1"
+ssh-key = { version = "0.6.2", features = ["ed25519"] }
+stderrlog = "0.5.4"
+tar = "0.4.40"
+thiserror = "1.0.49"
+tokio = { version = "1", features = ["full"] }
+tokio-rustls = "0.24.1"
+tokio-socks = "0.5.1"
+tokio-stream = { version = "0.1.14", features = ["io-util", "sync", "net"] }
+tokio-tar = { git = "https://github.com/dr-bonez/tokio-tar.git" }
+tokio-tungstenite = { version = "0.20.1", features = ["native-tls"] }
+tokio-util = { version = "0.7.9", features = ["io"] }
+torut = "0.2.1"
+tracing = "0.1.39"
+tracing-error = "0.2.0"
+tracing-futures = "0.2.5"
+tracing-journald = "0.3.0"
+tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }
+trust-dns-server = "0.23.1"
+typed-builder = "0.17.0"
+url = { version = "2.4.1", features = ["serde"] }
+urlencoding = "2.1.3"
+uuid = { version = "1.4.1", features = ["v4"] }
+zeroize = "1.6.0"
+

core/startos/deny.toml

@@ -0,0 +1,22 @@
+[licenses]
+unlicensed = "warn"
+allow-osi-fsf-free = "neither"
+copyleft = "deny"
+confidence-threshold = 0.93
+allow = [
+    "Apache-2.0",
+    "Apache-2.0 WITH LLVM-exception",
+    "MIT",
+    "ISC",
+    "MPL-2.0",
+    "CC0-1.0",
+    "BSD-2-Clause",
+    "BSD-3-Clause",
+    "LGPL-3.0",
+    "OpenSSL",
+]
+
+clarify = [
+    { name = "webpki", expression = "ISC", license-files = [ { path = "LICENSE", hash = 0x001c7e6c } ] },
+    { name = "ring", expression = "OpenSSL", license-files = [ { path = "LICENSE", hash = 0xbd0eed23 } ] },
+]

core/startos/migrations/20210629193146_Init.sql

@@ -0,0 +1,60 @@
+-- Add migration script here
+CREATE TABLE IF NOT EXISTS tor (
+    package TEXT NOT NULL,
+    interface TEXT NOT NULL,
+    key BYTEA NOT NULL CHECK (length(key) = 64),
+    PRIMARY KEY (package, interface)
+);
+
+CREATE TABLE IF NOT EXISTS session (
+    id TEXT NOT NULL PRIMARY KEY,
+    logged_in TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    logged_out TIMESTAMP,
+    last_active TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    user_agent TEXT,
+    metadata TEXT NOT NULL DEFAULT 'null'
+);
+
+CREATE TABLE IF NOT EXISTS account (
+    id SERIAL PRIMARY KEY CHECK (id = 0),
+    password TEXT NOT NULL,
+    tor_key BYTEA NOT NULL CHECK (length(tor_key) = 64)
+);
+
+CREATE TABLE IF NOT EXISTS ssh_keys (
+    fingerprint TEXT NOT NULL,
+    openssh_pubkey TEXT NOT NULL,
+    created_at TEXT NOT NULL,
+    PRIMARY KEY (fingerprint)
+);
+
+CREATE TABLE IF NOT EXISTS certificates (
+    id SERIAL PRIMARY KEY,
+    -- Root = 0, Int = 1, Other = 2..
+    priv_key_pem TEXT NOT NULL,
+    certificate_pem TEXT NOT NULL,
+    lookup_string TEXT UNIQUE,
+    created_at TEXT,
+    updated_at TEXT
+);
+
+ALTER SEQUENCE certificates_id_seq START 2 RESTART 2;
+
+CREATE TABLE IF NOT EXISTS notifications (
+    id SERIAL PRIMARY KEY,
+    package_id TEXT,
+    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    code INTEGER NOT NULL,
+    level TEXT NOT NULL,
+    title TEXT NOT NULL,
+    message TEXT NOT NULL,
+    data TEXT
+);
+
+CREATE TABLE IF NOT EXISTS cifs_shares (
+    id SERIAL PRIMARY KEY,
+    hostname TEXT NOT NULL,
+    path TEXT NOT NULL,
+    username TEXT NOT NULL,
+    password TEXT
+);

core/startos/migrations/20230118185232_NetworkKeys.sql

@@ -0,0 +1,62 @@
+-- Add migration script here
+CREATE EXTENSION pgcrypto;
+
+ALTER TABLE
+    account
+ADD
+    COLUMN server_id TEXT,
+ADD
+    COLUMN hostname TEXT,
+ADD
+    COLUMN network_key BYTEA CHECK (length(network_key) = 32),
+ADD
+    COLUMN root_ca_key_pem TEXT,
+ADD
+    COLUMN root_ca_cert_pem TEXT;
+
+UPDATE
+    account
+SET
+    network_key = gen_random_bytes(32),
+    root_ca_key_pem = (
+        SELECT
+            priv_key_pem
+        FROM
+            certificates
+        WHERE
+            id = 0
+    ),
+    root_ca_cert_pem = (
+        SELECT
+            certificate_pem
+        FROM
+            certificates
+        WHERE
+            id = 0
+    )
+WHERE
+    id = 0;
+
+ALTER TABLE
+    account
+ALTER COLUMN
+    tor_key DROP NOT NULL,
+ALTER COLUMN
+    network_key
+SET
+    NOT NULL,
+ALTER COLUMN
+    root_ca_key_pem
+SET
+    NOT NULL,
+ALTER COLUMN
+    root_ca_cert_pem
+SET
+    NOT NULL;
+
+CREATE TABLE IF NOT EXISTS network_keys (
+    package TEXT NOT NULL,
+    interface TEXT NOT NULL,
+    key BYTEA NOT NULL CHECK (length(key) = 32),
+    PRIMARY KEY (package, interface)
+);

core/startos/src/account.rs

@@ -0,0 +1,132 @@
+use std::time::SystemTime;
+
+use ed25519_dalek::SecretKey;
+use openssl::pkey::{PKey, Private};
+use openssl::x509::X509;
+use sqlx::PgExecutor;
+
+use crate::hostname::{generate_hostname, generate_id, Hostname};
+use crate::net::keys::Key;
+use crate::net::ssl::{generate_key, make_root_cert};
+use crate::prelude::*;
+use crate::util::crypto::ed25519_expand_key;
+
+fn hash_password(password: &str) -> Result<String, Error> {
+    argon2::hash_encoded(
+        password.as_bytes(),
+        &rand::random::<[u8; 16]>()[..],
+        &argon2::Config::rfc9106_low_mem(),
+    )
+    .with_kind(crate::ErrorKind::PasswordHashGeneration)
+}
+
+#[derive(Debug, Clone)]
+pub struct AccountInfo {
+    pub server_id: String,
+    pub hostname: Hostname,
+    pub password: String,
+    pub key: Key,
+    pub root_ca_key: PKey<Private>,
+    pub root_ca_cert: X509,
+}
+impl AccountInfo {
+    pub fn new(password: &str, start_time: SystemTime) -> Result<Self, Error> {
+        let server_id = generate_id();
+        let hostname = generate_hostname();
+        let root_ca_key = generate_key()?;
+        let root_ca_cert = make_root_cert(&root_ca_key, &hostname, start_time)?;
+        Ok(Self {
+            server_id,
+            hostname,
+            password: hash_password(password)?,
+            key: Key::new(None),
+            root_ca_key,
+            root_ca_cert,
+        })
+    }
+
+    pub async fn load(secrets: impl PgExecutor<'_>) -> Result<Self, Error> {
+        let r = sqlx::query!("SELECT * FROM account WHERE id = 0")
+            .fetch_one(secrets)
+            .await?;
+
+        let server_id = r.server_id.unwrap_or_else(generate_id);
+        let hostname = r.hostname.map(Hostname).unwrap_or_else(generate_hostname);
+        let password = r.password;
+        let network_key = SecretKey::try_from(r.network_key).map_err(|e| {
+            Error::new(
+                eyre!("expected vec of len 32, got len {}", e.len()),
+                ErrorKind::ParseDbField,
+            )
+        })?;
+        let tor_key = if let Some(k) = &r.tor_key {
+            <[u8; 64]>::try_from(&k[..]).map_err(|_| {
+                Error::new(
+                    eyre!("expected vec of len 64, got len {}", k.len()),
+                    ErrorKind::ParseDbField,
+                )
+            })?
+        } else {
+            ed25519_expand_key(&network_key)
+        };
+        let key = Key::from_pair(None, network_key, tor_key);
+        let root_ca_key = PKey::private_key_from_pem(r.root_ca_key_pem.as_bytes())?;
+        let root_ca_cert = X509::from_pem(r.root_ca_cert_pem.as_bytes())?;
+
+        Ok(Self {
+            server_id,
+            hostname,
+            password,
+            key,
+            root_ca_key,
+            root_ca_cert,
+        })
+    }
+
+    pub async fn save(&self, secrets: impl PgExecutor<'_>) -> Result<(), Error> {
+        let server_id = self.server_id.as_str();
+        let hostname = self.hostname.0.as_str();
+        let password = self.password.as_str();
+        let network_key = self.key.as_bytes();
+        let network_key = network_key.as_slice();
+        let root_ca_key = String::from_utf8(self.root_ca_key.private_key_to_pem_pkcs8()?)?;
+        let root_ca_cert = String::from_utf8(self.root_ca_cert.to_pem()?)?;
+
+        sqlx::query!(
+            r#"
+            INSERT INTO account (
+                id,
+                server_id,
+                hostname,
+                password,
+                network_key,
+                root_ca_key_pem,
+                root_ca_cert_pem
+            ) VALUES (
+                0, $1, $2, $3, $4, $5, $6
+            ) ON CONFLICT (id) DO UPDATE SET
+                server_id = EXCLUDED.server_id,
+                hostname = EXCLUDED.hostname,
+                password = EXCLUDED.password,
+                network_key = EXCLUDED.network_key,
+                root_ca_key_pem = EXCLUDED.root_ca_key_pem,
+                root_ca_cert_pem = EXCLUDED.root_ca_cert_pem
+            "#,
+            server_id,
+            hostname,
+            password,
+            network_key,
+            root_ca_key,
+            root_ca_cert,
+        )
+        .execute(secrets)
+        .await?;
+
+        Ok(())
+    }
+
+    pub fn set_password(&mut self, password: &str) -> Result<(), Error> {
+        self.password = hash_password(password)?;
+        Ok(())
+    }
+}

core/startos/src/action.rs

@@ -0,0 +1,163 @@
+use std::collections::{BTreeMap, BTreeSet};
+
+use clap::ArgMatches;
+use color_eyre::eyre::eyre;
+use indexmap::IndexSet;
+pub use models::ActionId;
+use models::ImageId;
+use rpc_toolkit::command;
+use serde::{Deserialize, Serialize};
+use tracing::instrument;
+
+use crate::config::{Config, ConfigSpec};
+use crate::context::RpcContext;
+use crate::prelude::*;
+use crate::procedure::docker::DockerContainers;
+use crate::procedure::{PackageProcedure, ProcedureName};
+use crate::s9pk::manifest::PackageId;
+use crate::util::serde::{display_serializable, parse_stdin_deserializable, IoFormat};
+use crate::util::Version;
+use crate::volume::Volumes;
+use crate::{Error, ResultExt};
+#[derive(Clone, Debug, Default, Deserialize, Serialize)]
+pub struct Actions(pub BTreeMap<ActionId, Action>);
+
+#[derive(Debug, Serialize, Deserialize)]
+#[serde(tag = "version")]
+pub enum ActionResult {
+    #[serde(rename = "0")]
+    V0(ActionResultV0),
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct ActionResultV0 {
+    pub message: String,
+    pub value: Option<String>,
+    pub copyable: bool,
+    pub qr: bool,
+}
+
+#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub enum DockerStatus {
+    Running,
+    Stopped,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct Action {
+    pub name: String,
+    pub description: String,
+    #[serde(default)]
+    pub warning: Option<String>,
+    pub implementation: PackageProcedure,
+    pub allowed_statuses: IndexSet<DockerStatus>,
+    #[serde(default)]
+    pub input_spec: ConfigSpec,
+}
+impl Action {
+    #[instrument(skip_all)]
+    pub fn validate(
+        &self,
+        _container: &Option<DockerContainers>,
+        eos_version: &Version,
+        volumes: &Volumes,
+        image_ids: &BTreeSet<ImageId>,
+    ) -> Result<(), Error> {
+        self.implementation
+            .validate(eos_version, volumes, image_ids, true)
+            .with_ctx(|_| {
+                (
+                    crate::ErrorKind::ValidateS9pk,
+                    format!("Action {}", self.name),
+                )
+            })
+    }
+
+    #[instrument(skip_all)]
+    pub async fn execute(
+        &self,
+        ctx: &RpcContext,
+        pkg_id: &PackageId,
+        pkg_version: &Version,
+        action_id: &ActionId,
+        volumes: &Volumes,
+        input: Option<Config>,
+    ) -> Result<ActionResult, Error> {
+        if let Some(ref input) = input {
+            self.input_spec
+                .matches(&input)
+                .with_kind(crate::ErrorKind::ConfigSpecViolation)?;
+        }
+        self.implementation
+            .execute(
+                ctx,
+                pkg_id,
+                pkg_version,
+                ProcedureName::Action(action_id.clone()),
+                volumes,
+                input,
+                None,
+            )
+            .await?
+            .map_err(|e| Error::new(eyre!("{}", e.1), crate::ErrorKind::Action))
+    }
+}
+
+fn display_action_result(action_result: ActionResult, matches: &ArgMatches) {
+    if matches.is_present("format") {
+        return display_serializable(action_result, matches);
+    }
+    match action_result {
+        ActionResult::V0(ar) => {
+            println!(
+                "{}: {}",
+                ar.message,
+                serde_json::to_string(&ar.value).unwrap()
+            );
+        }
+    }
+}
+
+#[command(about = "Executes an action", display(display_action_result))]
+#[instrument(skip_all)]
+pub async fn action(
+    #[context] ctx: RpcContext,
+    #[arg(rename = "id")] pkg_id: PackageId,
+    #[arg(rename = "action-id")] action_id: ActionId,
+    #[arg(stdin, parse(parse_stdin_deserializable))] input: Option<Config>,
+    #[allow(unused_variables)]
+    #[arg(long = "format")]
+    format: Option<IoFormat>,
+) -> Result<ActionResult, Error> {
+    let manifest = ctx
+        .db
+        .peek()
+        .await
+        .as_package_data()
+        .as_idx(&pkg_id)
+        .or_not_found(&pkg_id)?
+        .as_installed()
+        .or_not_found(&pkg_id)?
+        .as_manifest()
+        .de()?;
+
+    if let Some(action) = manifest.actions.0.get(&action_id) {
+        action
+            .execute(
+                &ctx,
+                &manifest.id,
+                &manifest.version,
+                &action_id,
+                &manifest.volumes,
+                input,
+            )
+            .await
+    } else {
+        Err(Error::new(
+            eyre!("Action not found in manifest"),
+            crate::ErrorKind::NotFound,
+        ))
+    }
+}

core/startos/src/auth.rs

@@ -0,0 +1,391 @@
+use std::collections::BTreeMap;
+use std::marker::PhantomData;
+
+use chrono::{DateTime, Utc};
+use clap::ArgMatches;
+use color_eyre::eyre::eyre;
+use josekit::jwk::Jwk;
+use rpc_toolkit::command;
+use rpc_toolkit::command_helpers::prelude::{RequestParts, ResponseParts};
+use rpc_toolkit::yajrc::RpcError;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+use sqlx::{Executor, Postgres};
+use tracing::instrument;
+
+use crate::context::{CliContext, RpcContext};
+use crate::middleware::auth::{AsLogoutSessionId, HasLoggedOutSessions, HashSessionToken};
+use crate::middleware::encrypt::EncryptedWire;
+use crate::prelude::*;
+use crate::util::display_none;
+use crate::util::serde::{display_serializable, IoFormat};
+use crate::{ensure_code, Error, ResultExt};
+#[derive(Clone, Serialize, Deserialize)]
+#[serde(untagged)]
+pub enum PasswordType {
+    EncryptedWire(EncryptedWire),
+    String(String),
+}
+impl PasswordType {
+    pub fn decrypt(self, current_secret: impl AsRef<Jwk>) -> Result<String, Error> {
+        match self {
+            PasswordType::String(x) => Ok(x),
+            PasswordType::EncryptedWire(x) => x.decrypt(current_secret).ok_or_else(|| {
+                Error::new(
+                    color_eyre::eyre::eyre!("Couldn't decode password"),
+                    crate::ErrorKind::Unknown,
+                )
+            }),
+        }
+    }
+}
+impl Default for PasswordType {
+    fn default() -> Self {
+        PasswordType::String(String::default())
+    }
+}
+impl std::fmt::Debug for PasswordType {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "<REDACTED_PASSWORD>")?;
+        Ok(())
+    }
+}
+
+impl std::str::FromStr for PasswordType {
+    type Err = String;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        Ok(match serde_json::from_str(s) {
+            Ok(a) => a,
+            Err(_) => PasswordType::String(s.to_string()),
+        })
+    }
+}
+
+#[command(subcommands(login, logout, session, reset_password, get_pubkey))]
+pub fn auth() -> Result<(), Error> {
+    Ok(())
+}
+
+pub fn cli_metadata() -> Value {
+    serde_json::json!({
+        "platforms": ["cli"],
+    })
+}
+
+pub fn parse_metadata(_: &str, _: &ArgMatches) -> Result<Value, Error> {
+    Ok(cli_metadata())
+}
+
+#[test]
+fn gen_pwd() {
+    println!(
+        "{:?}",
+        argon2::hash_encoded(
+            b"testing1234",
+            &rand::random::<[u8; 16]>()[..],
+            &argon2::Config::rfc9106_low_mem()
+        )
+        .unwrap()
+    )
+}
+
+#[instrument(skip_all)]
+async fn cli_login(
+    ctx: CliContext,
+    password: Option<PasswordType>,
+    metadata: Value,
+) -> Result<(), RpcError> {
+    let password = if let Some(password) = password {
+        password.decrypt(&ctx)?
+    } else {
+        rpassword::prompt_password("Password: ")?
+    };
+
+    rpc_toolkit::command_helpers::call_remote(
+        ctx,
+        "auth.login",
+        serde_json::json!({ "password": password, "metadata": metadata }),
+        PhantomData::<()>,
+    )
+    .await?
+    .result?;
+
+    Ok(())
+}
+
+pub fn check_password(hash: &str, password: &str) -> Result<(), Error> {
+    ensure_code!(
+        argon2::verify_encoded(&hash, password.as_bytes()).map_err(|_| {
+            Error::new(
+                eyre!("Password Incorrect"),
+                crate::ErrorKind::IncorrectPassword,
+            )
+        })?,
+        crate::ErrorKind::IncorrectPassword,
+        "Password Incorrect"
+    );
+    Ok(())
+}
+
+pub async fn check_password_against_db<Ex>(secrets: &mut Ex, password: &str) -> Result<(), Error>
+where
+    for<'a> &'a mut Ex: Executor<'a, Database = Postgres>,
+{
+    let pw_hash = sqlx::query!("SELECT password FROM account")
+        .fetch_one(secrets)
+        .await?
+        .password;
+    check_password(&pw_hash, password)?;
+    Ok(())
+}
+
+#[command(
+    custom_cli(cli_login(async, context(CliContext))),
+    display(display_none),
+    metadata(authenticated = false)
+)]
+#[instrument(skip_all)]
+pub async fn login(
+    #[context] ctx: RpcContext,
+    #[request] req: &RequestParts,
+    #[response] res: &mut ResponseParts,
+    #[arg] password: Option<PasswordType>,
+    #[arg(
+        parse(parse_metadata),
+        default = "cli_metadata",
+        help = "RPC Only: This value cannot be overidden from the cli"
+    )]
+    metadata: Value,
+) -> Result<(), Error> {
+    let password = password.unwrap_or_default().decrypt(&ctx)?;
+    let mut handle = ctx.secret_store.acquire().await?;
+    check_password_against_db(handle.as_mut(), &password).await?;
+
+    let hash_token = HashSessionToken::new();
+    let user_agent = req.headers.get("user-agent").and_then(|h| h.to_str().ok());
+    let metadata = serde_json::to_string(&metadata).with_kind(crate::ErrorKind::Database)?;
+    let hash_token_hashed = hash_token.hashed();
+    sqlx::query!(
+        "INSERT INTO session (id, user_agent, metadata) VALUES ($1, $2, $3)",
+        hash_token_hashed,
+        user_agent,
+        metadata,
+    )
+    .execute(handle.as_mut())
+    .await?;
+    res.headers.insert(
+        "set-cookie",
+        hash_token.header_value()?, // Should be impossible, but don't want to panic
+    );
+
+    Ok(())
+}
+
+#[command(display(display_none), metadata(authenticated = false))]
+#[instrument(skip_all)]
+pub async fn logout(
+    #[context] ctx: RpcContext,
+    #[request] req: &RequestParts,
+) -> Result<Option<HasLoggedOutSessions>, Error> {
+    let auth = match HashSessionToken::from_request_parts(req) {
+        Err(_) => return Ok(None),
+        Ok(a) => a,
+    };
+    Ok(Some(HasLoggedOutSessions::new(vec![auth], &ctx).await?))
+}
+
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct Session {
+    logged_in: DateTime<Utc>,
+    last_active: DateTime<Utc>,
+    user_agent: Option<String>,
+    metadata: Value,
+}
+
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct SessionList {
+    current: String,
+    sessions: BTreeMap<String, Session>,
+}
+
+#[command(subcommands(list, kill))]
+pub async fn session() -> Result<(), Error> {
+    Ok(())
+}
+
+fn display_sessions(arg: SessionList, matches: &ArgMatches) {
+    use prettytable::*;
+
+    if matches.is_present("format") {
+        return display_serializable(arg, matches);
+    }
+
+    let mut table = Table::new();
+    table.add_row(row![bc =>
+        "ID",
+        "LOGGED IN",
+        "LAST ACTIVE",
+        "USER AGENT",
+        "METADATA",
+    ]);
+    for (id, session) in arg.sessions {
+        let mut row = row![
+            &id,
+            &format!("{}", session.logged_in),
+            &format!("{}", session.last_active),
+            session.user_agent.as_deref().unwrap_or("N/A"),
+            &format!("{}", session.metadata),
+        ];
+        if id == arg.current {
+            row.iter_mut()
+                .map(|c| c.style(Attr::ForegroundColor(color::GREEN)))
+                .collect::<()>()
+        }
+        table.add_row(row);
+    }
+    table.print_tty(false).unwrap();
+}
+
+#[command(display(display_sessions))]
+#[instrument(skip_all)]
+pub async fn list(
+    #[context] ctx: RpcContext,
+    #[request] req: &RequestParts,
+    #[allow(unused_variables)]
+    #[arg(long = "format")]
+    format: Option<IoFormat>,
+) -> Result<SessionList, Error> {
+    Ok(SessionList {
+        current: HashSessionToken::from_request_parts(req)?.as_hash(),
+        sessions: sqlx::query!(
+            "SELECT * FROM session WHERE logged_out IS NULL OR logged_out > CURRENT_TIMESTAMP"
+        )
+        .fetch_all(ctx.secret_store.acquire().await?.as_mut())
+        .await?
+        .into_iter()
+        .map(|row| {
+            Ok((
+                row.id,
+                Session {
+                    logged_in: DateTime::from_utc(row.logged_in, Utc),
+                    last_active: DateTime::from_utc(row.last_active, Utc),
+                    user_agent: row.user_agent,
+                    metadata: serde_json::from_str(&row.metadata)
+                        .with_kind(crate::ErrorKind::Database)?,
+                },
+            ))
+        })
+        .collect::<Result<_, Error>>()?,
+    })
+}
+
+fn parse_comma_separated(arg: &str, _: &ArgMatches) -> Result<Vec<String>, RpcError> {
+    Ok(arg.split(",").map(|s| s.trim().to_owned()).collect())
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct KillSessionId(String);
+
+impl AsLogoutSessionId for KillSessionId {
+    fn as_logout_session_id(self) -> String {
+        self.0
+    }
+}
+
+#[command(display(display_none))]
+#[instrument(skip_all)]
+pub async fn kill(
+    #[context] ctx: RpcContext,
+    #[arg(parse(parse_comma_separated))] ids: Vec<String>,
+) -> Result<(), Error> {
+    HasLoggedOutSessions::new(ids.into_iter().map(KillSessionId), &ctx).await?;
+    Ok(())
+}
+
+#[instrument(skip_all)]
+async fn cli_reset_password(
+    ctx: CliContext,
+    old_password: Option<PasswordType>,
+    new_password: Option<PasswordType>,
+) -> Result<(), RpcError> {
+    let old_password = if let Some(old_password) = old_password {
+        old_password.decrypt(&ctx)?
+    } else {
+        rpassword::prompt_password("Current Password: ")?
+    };
+
+    let new_password = if let Some(new_password) = new_password {
+        new_password.decrypt(&ctx)?
+    } else {
+        let new_password = rpassword::prompt_password("New Password: ")?;
+        if new_password != rpassword::prompt_password("Confirm: ")? {
+            return Err(Error::new(
+                eyre!("Passwords do not match"),
+                crate::ErrorKind::IncorrectPassword,
+            )
+            .into());
+        }
+        new_password
+    };
+
+    rpc_toolkit::command_helpers::call_remote(
+        ctx,
+        "auth.reset-password",
+        serde_json::json!({ "old-password": old_password, "new-password": new_password }),
+        PhantomData::<()>,
+    )
+    .await?
+    .result?;
+
+    Ok(())
+}
+
+#[command(
+    rename = "reset-password",
+    custom_cli(cli_reset_password(async, context(CliContext))),
+    display(display_none)
+)]
+#[instrument(skip_all)]
+pub async fn reset_password(
+    #[context] ctx: RpcContext,
+    #[arg(rename = "old-password")] old_password: Option<PasswordType>,
+    #[arg(rename = "new-password")] new_password: Option<PasswordType>,
+) -> Result<(), Error> {
+    let old_password = old_password.unwrap_or_default().decrypt(&ctx)?;
+    let new_password = new_password.unwrap_or_default().decrypt(&ctx)?;
+
+    let mut account = ctx.account.write().await;
+    if !argon2::verify_encoded(&account.password, old_password.as_bytes())
+        .with_kind(crate::ErrorKind::IncorrectPassword)?
+    {
+        return Err(Error::new(
+            eyre!("Incorrect Password"),
+            crate::ErrorKind::IncorrectPassword,
+        ));
+    }
+    account.set_password(&new_password)?;
+    account.save(&ctx.secret_store).await?;
+    let account_password = &account.password;
+    ctx.db
+        .mutate(|d| {
+            d.as_server_info_mut()
+                .as_password_hash_mut()
+                .ser(account_password)
+        })
+        .await
+}
+
+#[command(
+    rename = "get-pubkey",
+    display(display_none),
+    metadata(authenticated = false)
+)]
+#[instrument(skip_all)]
+pub async fn get_pubkey(#[context] ctx: RpcContext) -> Result<Jwk, RpcError> {
+    let secret = ctx.as_ref().clone();
+    let pub_key = secret.to_public_key()?;
+    Ok(pub_key)
+}

core/startos/src/backup/backup_bulk.rs

@@ -0,0 +1,322 @@
+use std::collections::BTreeMap;
+use std::panic::UnwindSafe;
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+
+use chrono::Utc;
+use clap::ArgMatches;
+use color_eyre::eyre::eyre;
+use helpers::AtomicFile;
+use imbl::OrdSet;
+use models::Version;
+use rpc_toolkit::command;
+use tokio::io::AsyncWriteExt;
+use tokio::sync::Mutex;
+use tracing::instrument;
+
+use super::target::BackupTargetId;
+use super::PackageBackupReport;
+use crate::auth::check_password_against_db;
+use crate::backup::os::OsBackup;
+use crate::backup::{BackupReport, ServerBackupReport};
+use crate::context::RpcContext;
+use crate::db::model::BackupProgress;
+use crate::db::package::get_packages;
+use crate::disk::mount::backup::BackupMountGuard;
+use crate::disk::mount::filesystem::ReadWrite;
+use crate::disk::mount::guard::TmpMountGuard;
+use crate::manager::BackupReturn;
+use crate::notifications::NotificationLevel;
+use crate::prelude::*;
+use crate::s9pk::manifest::PackageId;
+use crate::util::display_none;
+use crate::util::io::dir_copy;
+use crate::util::serde::IoFormat;
+use crate::version::VersionT;
+
+fn parse_comma_separated(arg: &str, _: &ArgMatches) -> Result<OrdSet<PackageId>, Error> {
+    arg.split(',')
+        .map(|s| s.trim().parse::<PackageId>().map_err(Error::from))
+        .collect()
+}
+
+#[command(rename = "create", display(display_none))]
+#[instrument(skip(ctx, old_password, password))]
+pub async fn backup_all(
+    #[context] ctx: RpcContext,
+    #[arg(rename = "target-id")] target_id: BackupTargetId,
+    #[arg(rename = "old-password", long = "old-password")] old_password: Option<
+        crate::auth::PasswordType,
+    >,
+    #[arg(
+        rename = "package-ids",
+        long = "package-ids",
+        parse(parse_comma_separated)
+    )]
+    package_ids: Option<OrdSet<PackageId>>,
+    #[arg] password: crate::auth::PasswordType,
+) -> Result<(), Error> {
+    let db = ctx.db.peek().await;
+    let old_password_decrypted = old_password
+        .as_ref()
+        .unwrap_or(&password)
+        .clone()
+        .decrypt(&ctx)?;
+    let password = password.decrypt(&ctx)?;
+    check_password_against_db(ctx.secret_store.acquire().await?.as_mut(), &password).await?;
+    let fs = target_id
+        .load(ctx.secret_store.acquire().await?.as_mut())
+        .await?;
+    let mut backup_guard = BackupMountGuard::mount(
+        TmpMountGuard::mount(&fs, ReadWrite).await?,
+        &old_password_decrypted,
+    )
+    .await?;
+    let package_ids = if let Some(ids) = package_ids {
+        ids.into_iter()
+            .flat_map(|package_id| {
+                let version = db
+                    .as_package_data()
+                    .as_idx(&package_id)?
+                    .as_manifest()
+                    .as_version()
+                    .de()
+                    .ok()?;
+                Some((package_id, version))
+            })
+            .collect()
+    } else {
+        get_packages(db.clone())?.into_iter().collect()
+    };
+    if old_password.is_some() {
+        backup_guard.change_password(&password)?;
+    }
+    assure_backing_up(&ctx.db, &package_ids).await?;
+    tokio::task::spawn(async move {
+        let backup_res = perform_backup(&ctx, backup_guard, &package_ids).await;
+        match backup_res {
+            Ok(report) if report.iter().all(|(_, rep)| rep.error.is_none()) => ctx
+                .notification_manager
+                .notify(
+                    ctx.db.clone(),
+                    None,
+                    NotificationLevel::Success,
+                    "Backup Complete".to_owned(),
+                    "Your backup has completed".to_owned(),
+                    BackupReport {
+                        server: ServerBackupReport {
+                            attempted: true,
+                            error: None,
+                        },
+                        packages: report
+                            .into_iter()
+                            .map(|((package_id, _), value)| (package_id, value))
+                            .collect(),
+                    },
+                    None,
+                )
+                .await
+                .expect("failed to send notification"),
+            Ok(report) => ctx
+                .notification_manager
+                .notify(
+                    ctx.db.clone(),
+                    None,
+                    NotificationLevel::Warning,
+                    "Backup Complete".to_owned(),
+                    "Your backup has completed, but some package(s) failed to backup".to_owned(),
+                    BackupReport {
+                        server: ServerBackupReport {
+                            attempted: true,
+                            error: None,
+                        },
+                        packages: report
+                            .into_iter()
+                            .map(|((package_id, _), value)| (package_id, value))
+                            .collect(),
+                    },
+                    None,
+                )
+                .await
+                .expect("failed to send notification"),
+            Err(e) => {
+                tracing::error!("Backup Failed: {}", e);
+                tracing::debug!("{:?}", e);
+                ctx.notification_manager
+                    .notify(
+                        ctx.db.clone(),
+                        None,
+                        NotificationLevel::Error,
+                        "Backup Failed".to_owned(),
+                        "Your backup failed to complete.".to_owned(),
+                        BackupReport {
+                            server: ServerBackupReport {
+                                attempted: true,
+                                error: Some(e.to_string()),
+                            },
+                            packages: BTreeMap::new(),
+                        },
+                        None,
+                    )
+                    .await
+                    .expect("failed to send notification");
+            }
+        }
+        ctx.db
+            .mutate(|v| {
+                v.as_server_info_mut()
+                    .as_status_info_mut()
+                    .as_backup_progress_mut()
+                    .ser(&None)
+            })
+            .await?;
+        Ok::<(), Error>(())
+    });
+    Ok(())
+}
+
+#[instrument(skip(db, packages))]
+async fn assure_backing_up(
+    db: &PatchDb,
+    packages: impl IntoIterator<Item = &(PackageId, Version)> + UnwindSafe + Send,
+) -> Result<(), Error> {
+    db.mutate(|v| {
+        let backing_up = v
+            .as_server_info_mut()
+            .as_status_info_mut()
+            .as_backup_progress_mut();
+        if backing_up
+            .clone()
+            .de()?
+            .iter()
+            .flat_map(|x| x.values())
+            .fold(false, |acc, x| {
+                if !x.complete {
+                    return true;
+                }
+                acc
+            })
+        {
+            return Err(Error::new(
+                eyre!("Server is already backing up!"),
+                ErrorKind::InvalidRequest,
+            ));
+        }
+        backing_up.ser(&Some(
+            packages
+                .into_iter()
+                .map(|(x, _)| (x.clone(), BackupProgress { complete: false }))
+                .collect(),
+        ))?;
+        Ok(())
+    })
+    .await
+}
+
+#[instrument(skip(ctx, backup_guard))]
+async fn perform_backup(
+    ctx: &RpcContext,
+    backup_guard: BackupMountGuard<TmpMountGuard>,
+    package_ids: &OrdSet<(PackageId, Version)>,
+) -> Result<BTreeMap<(PackageId, Version), PackageBackupReport>, Error> {
+    let mut backup_report = BTreeMap::new();
+    let backup_guard = Arc::new(Mutex::new(backup_guard));
+
+    for package_id in package_ids {
+        let (response, _report) = match ctx
+            .managers
+            .get(package_id)
+            .await
+            .ok_or_else(|| Error::new(eyre!("Manager not found"), ErrorKind::InvalidRequest))?
+            .backup(backup_guard.clone())
+            .await
+        {
+            BackupReturn::Ran { report, res } => (res, report),
+            BackupReturn::AlreadyRunning(report) => {
+                backup_report.insert(package_id.clone(), report);
+                continue;
+            }
+            BackupReturn::Error(error) => {
+                tracing::warn!("Backup thread error");
+                tracing::debug!("{error:?}");
+                backup_report.insert(
+                    package_id.clone(),
+                    PackageBackupReport {
+                        error: Some("Backup thread error".to_owned()),
+                    },
+                );
+                continue;
+            }
+        };
+        backup_report.insert(
+            package_id.clone(),
+            PackageBackupReport {
+                error: response.as_ref().err().map(|e| e.to_string()),
+            },
+        );
+
+        if let Ok(pkg_meta) = response {
+            backup_guard
+                .lock()
+                .await
+                .metadata
+                .package_backups
+                .insert(package_id.0.clone(), pkg_meta);
+        }
+    }
+
+    let ui = ctx.db.peek().await.into_ui().de()?;
+
+    let mut os_backup_file = AtomicFile::new(
+        backup_guard.lock().await.as_ref().join("os-backup.cbor"),
+        None::<PathBuf>,
+    )
+    .await
+    .with_kind(ErrorKind::Filesystem)?;
+    os_backup_file
+        .write_all(&IoFormat::Cbor.to_vec(&OsBackup {
+            account: ctx.account.read().await.clone(),
+            ui,
+        })?)
+        .await?;
+    os_backup_file
+        .save()
+        .await
+        .with_kind(ErrorKind::Filesystem)?;
+
+    let luks_folder_old = backup_guard.lock().await.as_ref().join("luks.old");
+    if tokio::fs::metadata(&luks_folder_old).await.is_ok() {
+        tokio::fs::remove_dir_all(&luks_folder_old).await?;
+    }
+    let luks_folder_bak = backup_guard.lock().await.as_ref().join("luks");
+    if tokio::fs::metadata(&luks_folder_bak).await.is_ok() {
+        tokio::fs::rename(&luks_folder_bak, &luks_folder_old).await?;
+    }
+    let luks_folder = Path::new("/media/embassy/config/luks");
+    if tokio::fs::metadata(&luks_folder).await.is_ok() {
+        dir_copy(&luks_folder, &luks_folder_bak, None).await?;
+    }
+
+    let timestamp = Some(Utc::now());
+    let mut backup_guard = Arc::try_unwrap(backup_guard)
+        .map_err(|_err| {
+            Error::new(
+                eyre!("Backup guard could not ensure that the others where dropped"),
+                ErrorKind::Unknown,
+            )
+        })?
+        .into_inner();
+
+    backup_guard.unencrypted_metadata.version = crate::version::Current::new().semver().into();
+    backup_guard.unencrypted_metadata.full = true;
+    backup_guard.metadata.version = crate::version::Current::new().semver().into();
+    backup_guard.metadata.timestamp = timestamp;
+
+    backup_guard.save_and_unmount().await?;
+
+    ctx.db
+        .mutate(|v| v.as_server_info_mut().as_last_backup_mut().ser(&timestamp))
+        .await?;
+
+    Ok(backup_report)
+}

core/startos/src/backup/mod.rs

@@ -0,0 +1,226 @@
+use std::collections::{BTreeMap, BTreeSet};
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+
+use chrono::{DateTime, Utc};
+use color_eyre::eyre::eyre;
+use helpers::AtomicFile;
+use models::{ImageId, OptionExt};
+use reqwest::Url;
+use rpc_toolkit::command;
+use serde::{Deserialize, Serialize};
+use tokio::fs::File;
+use tokio::io::AsyncWriteExt;
+use tracing::instrument;
+
+use self::target::PackageBackupInfo;
+use crate::context::RpcContext;
+use crate::install::PKG_ARCHIVE_DIR;
+use crate::manager::manager_seed::ManagerSeed;
+use crate::net::interface::InterfaceId;
+use crate::net::keys::Key;
+use crate::prelude::*;
+use crate::procedure::docker::DockerContainers;
+use crate::procedure::{NoOutput, PackageProcedure, ProcedureName};
+use crate::s9pk::manifest::PackageId;
+use crate::util::serde::{Base32, Base64, IoFormat};
+use crate::util::Version;
+use crate::version::{Current, VersionT};
+use crate::volume::{backup_dir, Volume, VolumeId, Volumes, BACKUP_DIR};
+use crate::{Error, ErrorKind, ResultExt};
+
+pub mod backup_bulk;
+pub mod os;
+pub mod restore;
+pub mod target;
+
+#[derive(Debug, Deserialize, Serialize)]
+pub struct BackupReport {
+    server: ServerBackupReport,
+    packages: BTreeMap<PackageId, PackageBackupReport>,
+}
+
+#[derive(Debug, Deserialize, Serialize)]
+pub struct ServerBackupReport {
+    attempted: bool,
+    error: Option<String>,
+}
+
+#[derive(Debug, Deserialize, Serialize)]
+pub struct PackageBackupReport {
+    pub error: Option<String>,
+}
+
+#[command(subcommands(backup_bulk::backup_all, target::target))]
+pub fn backup() -> Result<(), Error> {
+    Ok(())
+}
+
+#[command(rename = "backup", subcommands(restore::restore_packages_rpc))]
+pub fn package_backup() -> Result<(), Error> {
+    Ok(())
+}
+
+#[derive(Deserialize, Serialize)]
+struct BackupMetadata {
+    pub timestamp: DateTime<Utc>,
+    #[serde(default)]
+    pub network_keys: BTreeMap<InterfaceId, Base64<[u8; 32]>>,
+    #[serde(default)]
+    pub tor_keys: BTreeMap<InterfaceId, Base32<[u8; 64]>>, // DEPRECATED
+    pub marketplace_url: Option<Url>,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize, HasModel)]
+#[model = "Model<Self>"]
+pub struct BackupActions {
+    pub create: PackageProcedure,
+    pub restore: PackageProcedure,
+}
+impl BackupActions {
+    pub fn validate(
+        &self,
+        _container: &Option<DockerContainers>,
+        eos_version: &Version,
+        volumes: &Volumes,
+        image_ids: &BTreeSet<ImageId>,
+    ) -> Result<(), Error> {
+        self.create
+            .validate(eos_version, volumes, image_ids, false)
+            .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Backup Create"))?;
+        self.restore
+            .validate(eos_version, volumes, image_ids, false)
+            .with_ctx(|_| (crate::ErrorKind::ValidateS9pk, "Backup Restore"))?;
+        Ok(())
+    }
+
+    #[instrument(skip_all)]
+    pub async fn create(&self, seed: Arc<ManagerSeed>) -> Result<PackageBackupInfo, Error> {
+        let manifest = &seed.manifest;
+        let mut volumes = seed.manifest.volumes.to_readonly();
+        let ctx = &seed.ctx;
+        let pkg_id = &manifest.id;
+        let pkg_version = &manifest.version;
+        volumes.insert(VolumeId::Backup, Volume::Backup { readonly: false });
+        let backup_dir = backup_dir(&manifest.id);
+        if tokio::fs::metadata(&backup_dir).await.is_err() {
+            tokio::fs::create_dir_all(&backup_dir).await?
+        }
+        self.create
+            .execute::<(), NoOutput>(
+                ctx,
+                pkg_id,
+                pkg_version,
+                ProcedureName::CreateBackup,
+                &volumes,
+                None,
+                None,
+            )
+            .await?
+            .map_err(|e| eyre!("{}", e.1))
+            .with_kind(crate::ErrorKind::Backup)?;
+        let (network_keys, tor_keys): (Vec<_>, Vec<_>) =
+            Key::for_package(&ctx.secret_store, pkg_id)
+                .await?
+                .into_iter()
+                .filter_map(|k| {
+                    let interface = k.interface().map(|(_, i)| i)?;
+                    Some((
+                        (interface.clone(), Base64(k.as_bytes())),
+                        (interface, Base32(k.tor_key().as_bytes())),
+                    ))
+                })
+                .unzip();
+        let marketplace_url = ctx
+            .db
+            .peek()
+            .await
+            .as_package_data()
+            .as_idx(&pkg_id)
+            .or_not_found(pkg_id)?
+            .expect_as_installed()?
+            .as_installed()
+            .as_marketplace_url()
+            .de()?;
+        let tmp_path = Path::new(BACKUP_DIR)
+            .join(pkg_id)
+            .join(format!("{}.s9pk", pkg_id));
+        let s9pk_path = ctx
+            .datadir
+            .join(PKG_ARCHIVE_DIR)
+            .join(pkg_id)
+            .join(pkg_version.as_str())
+            .join(format!("{}.s9pk", pkg_id));
+        let mut infile = File::open(&s9pk_path).await?;
+        let mut outfile = AtomicFile::new(&tmp_path, None::<PathBuf>)
+            .await
+            .with_kind(ErrorKind::Filesystem)?;
+        tokio::io::copy(&mut infile, &mut *outfile)
+            .await
+            .with_ctx(|_| {
+                (
+                    crate::ErrorKind::Filesystem,
+                    format!("cp {} -> {}", s9pk_path.display(), tmp_path.display()),
+                )
+            })?;
+        outfile.save().await.with_kind(ErrorKind::Filesystem)?;
+        let timestamp = Utc::now();
+        let metadata_path = Path::new(BACKUP_DIR).join(pkg_id).join("metadata.cbor");
+        let mut outfile = AtomicFile::new(&metadata_path, None::<PathBuf>)
+            .await
+            .with_kind(ErrorKind::Filesystem)?;
+        let network_keys = network_keys.into_iter().collect();
+        let tor_keys = tor_keys.into_iter().collect();
+        outfile
+            .write_all(&IoFormat::Cbor.to_vec(&BackupMetadata {
+                timestamp,
+                network_keys,
+                tor_keys,
+                marketplace_url,
+            })?)
+            .await?;
+        outfile.save().await.with_kind(ErrorKind::Filesystem)?;
+        Ok(PackageBackupInfo {
+            os_version: Current::new().semver().into(),
+            title: manifest.title.clone(),
+            version: pkg_version.clone(),
+            timestamp,
+        })
+    }
+
+    #[instrument(skip_all)]
+    pub async fn restore(
+        &self,
+        ctx: &RpcContext,
+        pkg_id: &PackageId,
+        pkg_version: &Version,
+        volumes: &Volumes,
+    ) -> Result<Option<Url>, Error> {
+        let mut volumes = volumes.clone();
+        volumes.insert(VolumeId::Backup, Volume::Backup { readonly: true });
+        self.restore
+            .execute::<(), NoOutput>(
+                ctx,
+                pkg_id,
+                pkg_version,
+                ProcedureName::RestoreBackup,
+                &volumes,
+                None,
+                None,
+            )
+            .await?
+            .map_err(|e| eyre!("{}", e.1))
+            .with_kind(crate::ErrorKind::Restore)?;
+        let metadata_path = Path::new(BACKUP_DIR).join(pkg_id).join("metadata.cbor");
+        let metadata: BackupMetadata = IoFormat::Cbor.from_slice(
+            &tokio::fs::read(&metadata_path).await.with_ctx(|_| {
+                (
+                    crate::ErrorKind::Filesystem,
+                    metadata_path.display().to_string(),
+                )
+            })?,
+        )?;
+
+        Ok(metadata.marketplace_url)
+    }
+}

core/startos/src/backup/os.rs

@@ -0,0 +1,122 @@
+use openssl::pkey::PKey;
+use openssl::x509::X509;
+use patch_db::Value;
+use serde::{Deserialize, Serialize};
+
+use crate::account::AccountInfo;
+use crate::hostname::{generate_hostname, generate_id, Hostname};
+use crate::net::keys::Key;
+use crate::prelude::*;
+use crate::util::serde::Base64;
+
+pub struct OsBackup {
+    pub account: AccountInfo,
+    pub ui: Value,
+}
+impl<'de> Deserialize<'de> for OsBackup {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        let tagged = OsBackupSerDe::deserialize(deserializer)?;
+        match tagged.version {
+            0 => patch_db::value::from_value::<OsBackupV0>(tagged.rest)
+                .map_err(serde::de::Error::custom)?
+                .project()
+                .map_err(serde::de::Error::custom),
+            1 => patch_db::value::from_value::<OsBackupV1>(tagged.rest)
+                .map_err(serde::de::Error::custom)?
+                .project()
+                .map_err(serde::de::Error::custom),
+            v => Err(serde::de::Error::custom(&format!(
+                "Unknown backup version {v}"
+            ))),
+        }
+    }
+}
+impl Serialize for OsBackup {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        OsBackupSerDe {
+            version: 1,
+            rest: patch_db::value::to_value(
+                &OsBackupV1::unproject(self).map_err(serde::ser::Error::custom)?,
+            )
+            .map_err(serde::ser::Error::custom)?,
+        }
+        .serialize(serializer)
+    }
+}
+
+#[derive(Deserialize, Serialize)]
+struct OsBackupSerDe {
+    #[serde(default)]
+    version: usize,
+    #[serde(flatten)]
+    rest: Value,
+}
+
+/// V0
+#[derive(Deserialize)]
+#[serde(rename = "kebab-case")]
+struct OsBackupV0 {
+    // tor_key: Base32<[u8; 64]>,
+    root_ca_key: String,  // PEM Encoded OpenSSL Key
+    root_ca_cert: String, // PEM Encoded OpenSSL X509 Certificate
+    ui: Value,            // JSON Value
+}
+impl OsBackupV0 {
+    fn project(self) -> Result<OsBackup, Error> {
+        Ok(OsBackup {
+            account: AccountInfo {
+                server_id: generate_id(),
+                hostname: generate_hostname(),
+                password: Default::default(),
+                key: Key::new(None),
+                root_ca_key: PKey::private_key_from_pem(self.root_ca_key.as_bytes())?,
+                root_ca_cert: X509::from_pem(self.root_ca_cert.as_bytes())?,
+            },
+            ui: self.ui,
+        })
+    }
+}
+
+/// V1
+#[derive(Deserialize, Serialize)]
+#[serde(rename = "kebab-case")]
+struct OsBackupV1 {
+    server_id: String,         // uuidv4
+    hostname: String,          // embassy-<adjective>-<noun>
+    net_key: Base64<[u8; 32]>, // Ed25519 Secret Key
+    root_ca_key: String,       // PEM Encoded OpenSSL Key
+    root_ca_cert: String,      // PEM Encoded OpenSSL X509 Certificate
+    ui: Value,                 // JSON Value
+                               // TODO add more
+}
+impl OsBackupV1 {
+    fn project(self) -> Result<OsBackup, Error> {
+        Ok(OsBackup {
+            account: AccountInfo {
+                server_id: self.server_id,
+                hostname: Hostname(self.hostname),
+                password: Default::default(),
+                key: Key::from_bytes(None, self.net_key.0),
+                root_ca_key: PKey::private_key_from_pem(self.root_ca_key.as_bytes())?,
+                root_ca_cert: X509::from_pem(self.root_ca_cert.as_bytes())?,
+            },
+            ui: self.ui,
+        })
+    }
+    fn unproject(backup: &OsBackup) -> Result<Self, Error> {
+        Ok(Self {
+            server_id: backup.account.server_id.clone(),
+            hostname: backup.account.hostname.0.clone(),
+            net_key: Base64(backup.account.key.as_bytes()),
+            root_ca_key: String::from_utf8(backup.account.root_ca_key.private_key_to_pem_pkcs8()?)?,
+            root_ca_cert: String::from_utf8(backup.account.root_ca_cert.to_pem()?)?,
+            ui: backup.ui.clone(),
+        })
+    }
+}

core/startos/src/backup/restore.rs

@@ -0,0 +1,461 @@
+use std::collections::BTreeMap;
+use std::path::Path;
+use std::sync::atomic::Ordering;
+use std::sync::Arc;
+use std::time::Duration;
+
+use clap::ArgMatches;
+use futures::future::BoxFuture;
+use futures::{stream, FutureExt, StreamExt};
+use openssl::x509::X509;
+use rpc_toolkit::command;
+use sqlx::Connection;
+use tokio::fs::File;
+use torut::onion::OnionAddressV3;
+use tracing::instrument;
+
+use super::target::BackupTargetId;
+use crate::backup::os::OsBackup;
+use crate::backup::BackupMetadata;
+use crate::context::rpc::RpcContextConfig;
+use crate::context::{RpcContext, SetupContext};
+use crate::db::model::{PackageDataEntry, PackageDataEntryRestoring, StaticFiles};
+use crate::disk::mount::backup::{BackupMountGuard, PackageBackupMountGuard};
+use crate::disk::mount::filesystem::ReadWrite;
+use crate::disk::mount::guard::TmpMountGuard;
+use crate::hostname::Hostname;
+use crate::init::init;
+use crate::install::progress::InstallProgress;
+use crate::install::{download_install_s9pk, PKG_PUBLIC_DIR};
+use crate::notifications::NotificationLevel;
+use crate::prelude::*;
+use crate::s9pk::manifest::{Manifest, PackageId};
+use crate::s9pk::reader::S9pkReader;
+use crate::setup::SetupStatus;
+use crate::util::display_none;
+use crate::util::io::dir_size;
+use crate::util::serde::IoFormat;
+use crate::volume::{backup_dir, BACKUP_DIR, PKG_VOLUME_DIR};
+
+fn parse_comma_separated(arg: &str, _: &ArgMatches) -> Result<Vec<PackageId>, Error> {
+    arg.split(',')
+        .map(|s| s.trim().parse().map_err(Error::from))
+        .collect()
+}
+
+#[command(rename = "restore", display(display_none))]
+#[instrument(skip(ctx, password))]
+pub async fn restore_packages_rpc(
+    #[context] ctx: RpcContext,
+    #[arg(parse(parse_comma_separated))] ids: Vec<PackageId>,
+    #[arg(rename = "target-id")] target_id: BackupTargetId,
+    #[arg] password: String,
+) -> Result<(), Error> {
+    let fs = target_id
+        .load(ctx.secret_store.acquire().await?.as_mut())
+        .await?;
+    let backup_guard =
+        BackupMountGuard::mount(TmpMountGuard::mount(&fs, ReadWrite).await?, &password).await?;
+
+    let (backup_guard, tasks, _) = restore_packages(&ctx, backup_guard, ids).await?;
+
+    tokio::spawn(async move {
+        stream::iter(tasks.into_iter().map(|x| (x, ctx.clone())))
+            .for_each_concurrent(5, |(res, ctx)| async move {
+                match res.await {
+                    (Ok(_), _) => (),
+                    (Err(err), package_id) => {
+                        if let Err(err) = ctx
+                            .notification_manager
+                            .notify(
+                                ctx.db.clone(),
+                                Some(package_id.clone()),
+                                NotificationLevel::Error,
+                                "Restoration Failure".to_string(),
+                                format!("Error restoring package {}: {}", package_id, err),
+                                (),
+                                None,
+                            )
+                            .await
+                        {
+                            tracing::error!("Failed to notify: {}", err);
+                            tracing::debug!("{:?}", err);
+                        };
+                        tracing::error!("Error restoring package {}: {}", package_id, err);
+                        tracing::debug!("{:?}", err);
+                    }
+                }
+            })
+            .await;
+        if let Err(e) = backup_guard.unmount().await {
+            tracing::error!("Error unmounting backup drive: {}", e);
+            tracing::debug!("{:?}", e);
+        }
+    });
+
+    Ok(())
+}
+
+async fn approximate_progress(
+    rpc_ctx: &RpcContext,
+    progress: &mut ProgressInfo,
+) -> Result<(), Error> {
+    for (id, size) in &mut progress.target_volume_size {
+        let dir = rpc_ctx.datadir.join(PKG_VOLUME_DIR).join(id).join("data");
+        if tokio::fs::metadata(&dir).await.is_err() {
+            *size = 0;
+        } else {
+            *size = dir_size(&dir, None).await?;
+        }
+    }
+    Ok(())
+}
+
+async fn approximate_progress_loop(
+    ctx: &SetupContext,
+    rpc_ctx: &RpcContext,
+    mut starting_info: ProgressInfo,
+) {
+    loop {
+        if let Err(e) = approximate_progress(rpc_ctx, &mut starting_info).await {
+            tracing::error!("Failed to approximate restore progress: {}", e);
+            tracing::debug!("{:?}", e);
+        } else {
+            *ctx.setup_status.write().await = Some(Ok(starting_info.flatten()));
+        }
+        tokio::time::sleep(Duration::from_secs(1)).await;
+    }
+}
+
+#[derive(Debug, Default)]
+struct ProgressInfo {
+    package_installs: BTreeMap<PackageId, Arc<InstallProgress>>,
+    src_volume_size: BTreeMap<PackageId, u64>,
+    target_volume_size: BTreeMap<PackageId, u64>,
+}
+impl ProgressInfo {
+    fn flatten(&self) -> SetupStatus {
+        let mut total_bytes = 0;
+        let mut bytes_transferred = 0;
+
+        for progress in self.package_installs.values() {
+            total_bytes += ((progress.size.unwrap_or(0) as f64) * 2.2) as u64;
+            bytes_transferred += progress.downloaded.load(Ordering::SeqCst);
+            bytes_transferred += ((progress.validated.load(Ordering::SeqCst) as f64) * 0.2) as u64;
+            bytes_transferred += progress.unpacked.load(Ordering::SeqCst);
+        }
+
+        for size in self.src_volume_size.values() {
+            total_bytes += *size;
+        }
+
+        for size in self.target_volume_size.values() {
+            bytes_transferred += *size;
+        }
+
+        if bytes_transferred > total_bytes {
+            bytes_transferred = total_bytes;
+        }
+
+        SetupStatus {
+            total_bytes: Some(total_bytes),
+            bytes_transferred,
+            complete: false,
+        }
+    }
+}
+
+#[instrument(skip(ctx))]
+pub async fn recover_full_embassy(
+    ctx: SetupContext,
+    disk_guid: Arc<String>,
+    embassy_password: String,
+    recovery_source: TmpMountGuard,
+    recovery_password: Option<String>,
+) -> Result<(Arc<String>, Hostname, OnionAddressV3, X509), Error> {
+    let backup_guard = BackupMountGuard::mount(
+        recovery_source,
+        recovery_password.as_deref().unwrap_or_default(),
+    )
+    .await?;
+
+    let os_backup_path = backup_guard.as_ref().join("os-backup.cbor");
+    let mut os_backup: OsBackup = IoFormat::Cbor.from_slice(
+        &tokio::fs::read(&os_backup_path)
+            .await
+            .with_ctx(|_| (ErrorKind::Filesystem, os_backup_path.display().to_string()))?,
+    )?;
+
+    os_backup.account.password = argon2::hash_encoded(
+        embassy_password.as_bytes(),
+        &rand::random::<[u8; 16]>()[..],
+        &argon2::Config::rfc9106_low_mem(),
+    )
+    .with_kind(ErrorKind::PasswordHashGeneration)?;
+
+    let secret_store = ctx.secret_store().await?;
+
+    os_backup.account.save(&secret_store).await?;
+
+    secret_store.close().await;
+
+    let cfg = RpcContextConfig::load(ctx.config_path.clone()).await?;
+
+    init(&cfg).await?;
+
+    let rpc_ctx = RpcContext::init(ctx.config_path.clone(), disk_guid.clone()).await?;
+
+    let ids: Vec<_> = backup_guard
+        .metadata
+        .package_backups
+        .keys()
+        .cloned()
+        .collect();
+    let (backup_guard, tasks, progress_info) =
+        restore_packages(&rpc_ctx, backup_guard, ids).await?;
+    let task_consumer_rpc_ctx = rpc_ctx.clone();
+    tokio::select! {
+        _ = async move {
+            stream::iter(tasks.into_iter().map(|x| (x,  task_consumer_rpc_ctx.clone())))
+                .for_each_concurrent(5, |(res, ctx)| async move {
+                    match res.await {
+                        (Ok(_), _) => (),
+                        (Err(err), package_id) => {
+                            if let Err(err) = ctx.notification_manager.notify(
+                                ctx.db.clone(),
+                                Some(package_id.clone()),
+                                NotificationLevel::Error,
+                                "Restoration Failure".to_string(), format!("Error restoring package {}: {}", package_id,err), (), None).await{
+                                tracing::error!("Failed to notify: {}", err);
+                                tracing::debug!("{:?}", err);
+                                };
+                            tracing::error!("Error restoring package {}: {}", package_id, err);
+                            tracing::debug!("{:?}", err);
+                        },
+                    }
+                }).await;
+
+        } => {
+
+        },
+        _ = approximate_progress_loop(&ctx, &rpc_ctx, progress_info) => unreachable!(concat!(module_path!(), "::approximate_progress_loop should not terminate")),
+    }
+
+    backup_guard.unmount().await?;
+    rpc_ctx.shutdown().await?;
+
+    Ok((
+        disk_guid,
+        os_backup.account.hostname,
+        os_backup.account.key.tor_address(),
+        os_backup.account.root_ca_cert,
+    ))
+}
+
+#[instrument(skip(ctx, backup_guard))]
+async fn restore_packages(
+    ctx: &RpcContext,
+    backup_guard: BackupMountGuard<TmpMountGuard>,
+    ids: Vec<PackageId>,
+) -> Result<
+    (
+        BackupMountGuard<TmpMountGuard>,
+        Vec<BoxFuture<'static, (Result<(), Error>, PackageId)>>,
+        ProgressInfo,
+    ),
+    Error,
+> {
+    let guards = assure_restoring(ctx, ids, &backup_guard).await?;
+
+    let mut progress_info = ProgressInfo::default();
+
+    let mut tasks = Vec::with_capacity(guards.len());
+    for (manifest, guard) in guards {
+        let id = manifest.id.clone();
+        let (progress, task) = restore_package(ctx.clone(), manifest, guard).await?;
+        progress_info
+            .package_installs
+            .insert(id.clone(), progress.clone());
+        progress_info
+            .src_volume_size
+            .insert(id.clone(), dir_size(backup_dir(&id), None).await?);
+        progress_info.target_volume_size.insert(id.clone(), 0);
+        let package_id = id.clone();
+        tasks.push(
+            async move {
+                if let Err(e) = task.await {
+                    tracing::error!("Error restoring package {}: {}", id, e);
+                    tracing::debug!("{:?}", e);
+                    Err(e)
+                } else {
+                    Ok(())
+                }
+            }
+            .map(|x| (x, package_id))
+            .boxed(),
+        );
+    }
+
+    Ok((backup_guard, tasks, progress_info))
+}
+
+#[instrument(skip(ctx, backup_guard))]
+async fn assure_restoring(
+    ctx: &RpcContext,
+    ids: Vec<PackageId>,
+    backup_guard: &BackupMountGuard<TmpMountGuard>,
+) -> Result<Vec<(Manifest, PackageBackupMountGuard)>, Error> {
+    let mut guards = Vec::with_capacity(ids.len());
+
+    let mut insert_packages = BTreeMap::new();
+
+    for id in ids {
+        let peek = ctx.db.peek().await;
+
+        let model = peek.as_package_data().as_idx(&id);
+
+        if !model.is_none() {
+            return Err(Error::new(
+                eyre!("Can't restore over existing package: {}", id),
+                crate::ErrorKind::InvalidRequest,
+            ));
+        }
+        let guard = backup_guard.mount_package_backup(&id).await?;
+        let s9pk_path = Path::new(BACKUP_DIR).join(&id).join(format!("{}.s9pk", id));
+        let mut rdr = S9pkReader::open(&s9pk_path, false).await?;
+
+        let manifest = rdr.manifest().await?;
+        let version = manifest.version.clone();
+        let progress = Arc::new(InstallProgress::new(Some(
+            tokio::fs::metadata(&s9pk_path).await?.len(),
+        )));
+
+        let public_dir_path = ctx
+            .datadir
+            .join(PKG_PUBLIC_DIR)
+            .join(&id)
+            .join(version.as_str());
+        tokio::fs::create_dir_all(&public_dir_path).await?;
+
+        let license_path = public_dir_path.join("LICENSE.md");
+        let mut dst = File::create(&license_path).await?;
+        tokio::io::copy(&mut rdr.license().await?, &mut dst).await?;
+        dst.sync_all().await?;
+
+        let instructions_path = public_dir_path.join("INSTRUCTIONS.md");
+        let mut dst = File::create(&instructions_path).await?;
+        tokio::io::copy(&mut rdr.instructions().await?, &mut dst).await?;
+        dst.sync_all().await?;
+
+        let icon_path = Path::new("icon").with_extension(&manifest.assets.icon_type());
+        let icon_path = public_dir_path.join(&icon_path);
+        let mut dst = File::create(&icon_path).await?;
+        tokio::io::copy(&mut rdr.icon().await?, &mut dst).await?;
+        dst.sync_all().await?;
+        insert_packages.insert(
+            id.clone(),
+            PackageDataEntry::Restoring(PackageDataEntryRestoring {
+                install_progress: progress.clone(),
+                static_files: StaticFiles::local(&id, &version, manifest.assets.icon_type()),
+                manifest: manifest.clone(),
+            }),
+        );
+
+        guards.push((manifest, guard));
+    }
+    ctx.db
+        .mutate(|db| {
+            for (id, package) in insert_packages {
+                db.as_package_data_mut().insert(&id, &package)?;
+            }
+            Ok(())
+        })
+        .await?;
+    Ok(guards)
+}
+
+#[instrument(skip(ctx, guard))]
+async fn restore_package<'a>(
+    ctx: RpcContext,
+    manifest: Manifest,
+    guard: PackageBackupMountGuard,
+) -> Result<(Arc<InstallProgress>, BoxFuture<'static, Result<(), Error>>), Error> {
+    let id = manifest.id.clone();
+    let s9pk_path = Path::new(BACKUP_DIR)
+        .join(&manifest.id)
+        .join(format!("{}.s9pk", id));
+
+    let metadata_path = Path::new(BACKUP_DIR).join(&id).join("metadata.cbor");
+    let metadata: BackupMetadata = IoFormat::Cbor.from_slice(
+        &tokio::fs::read(&metadata_path)
+            .await
+            .with_ctx(|_| (ErrorKind::Filesystem, metadata_path.display().to_string()))?,
+    )?;
+
+    let mut secrets = ctx.secret_store.acquire().await?;
+    let mut secrets_tx = secrets.begin().await?;
+    for (iface, key) in metadata.network_keys {
+        let k = key.0.as_slice();
+        sqlx::query!(
+            "INSERT INTO network_keys (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING",
+            id.to_string(),
+            iface.to_string(),
+            k,
+        )
+        .execute(secrets_tx.as_mut()).await?;
+    }
+    // DEPRECATED
+    for (iface, key) in metadata.tor_keys {
+        let k = key.0.as_slice();
+        sqlx::query!(
+            "INSERT INTO tor (package, interface, key) VALUES ($1, $2, $3) ON CONFLICT (package, interface) DO NOTHING",
+            id.to_string(),
+            iface.to_string(),
+            k,
+        )
+        .execute(secrets_tx.as_mut()).await?;
+    }
+    secrets_tx.commit().await?;
+    drop(secrets);
+
+    let len = tokio::fs::metadata(&s9pk_path)
+        .await
+        .with_ctx(|_| (ErrorKind::Filesystem, s9pk_path.display().to_string()))?
+        .len();
+    let file = File::open(&s9pk_path)
+        .await
+        .with_ctx(|_| (ErrorKind::Filesystem, s9pk_path.display().to_string()))?;
+
+    let progress = InstallProgress::new(Some(len));
+    let marketplace_url = metadata.marketplace_url;
+
+    let progress = Arc::new(progress);
+
+    ctx.db
+        .mutate(|db| {
+            db.as_package_data_mut().insert(
+                &id,
+                &PackageDataEntry::Restoring(PackageDataEntryRestoring {
+                    install_progress: progress.clone(),
+                    static_files: StaticFiles::local(
+                        &id,
+                        &manifest.version,
+                        manifest.assets.icon_type(),
+                    ),
+                    manifest: manifest.clone(),
+                }),
+            )
+        })
+        .await?;
+    Ok((
+        progress.clone(),
+        async move {
+            download_install_s9pk(ctx, manifest, marketplace_url, progress, file, None).await?;
+
+            guard.unmount().await?;
+
+            Ok(())
+        }
+        .boxed(),
+    ))
+}

core/startos/src/backup/target/cifs.rs

@@ -0,0 +1,211 @@
+use std::path::{Path, PathBuf};
+
+use color_eyre::eyre::eyre;
+use futures::TryStreamExt;
+use rpc_toolkit::command;
+use serde::{Deserialize, Serialize};
+use sqlx::{Executor, Postgres};
+
+use super::{BackupTarget, BackupTargetId};
+use crate::context::RpcContext;
+use crate::disk::mount::filesystem::cifs::Cifs;
+use crate::disk::mount::filesystem::ReadOnly;
+use crate::disk::mount::guard::TmpMountGuard;
+use crate::disk::util::{recovery_info, EmbassyOsRecoveryInfo};
+use crate::prelude::*;
+use crate::util::display_none;
+use crate::util::serde::KeyVal;
+
+#[derive(Debug, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct CifsBackupTarget {
+    hostname: String,
+    path: PathBuf,
+    username: String,
+    mountable: bool,
+    embassy_os: Option<EmbassyOsRecoveryInfo>,
+}
+
+#[command(subcommands(add, update, remove))]
+pub fn cifs() -> Result<(), Error> {
+    Ok(())
+}
+
+#[command(display(display_none))]
+pub async fn add(
+    #[context] ctx: RpcContext,
+    #[arg] hostname: String,
+    #[arg] path: PathBuf,
+    #[arg] username: String,
+    #[arg] password: Option<String>,
+) -> Result<KeyVal<BackupTargetId, BackupTarget>, Error> {
+    let cifs = Cifs {
+        hostname,
+        path,
+        username,
+        password,
+    };
+    let guard = TmpMountGuard::mount(&cifs, ReadOnly).await?;
+    let embassy_os = recovery_info(&guard).await?;
+    guard.unmount().await?;
+    let path_string = Path::new("/").join(&cifs.path).display().to_string();
+    let id: i32 = sqlx::query!(
+        "INSERT INTO cifs_shares (hostname, path, username, password) VALUES ($1, $2, $3, $4) RETURNING id",
+        cifs.hostname,
+        path_string,
+        cifs.username,
+        cifs.password,
+    )
+    .fetch_one(&ctx.secret_store)
+    .await?.id;
+    Ok(KeyVal {
+        key: BackupTargetId::Cifs { id },
+        value: BackupTarget::Cifs(CifsBackupTarget {
+            hostname: cifs.hostname,
+            path: cifs.path,
+            username: cifs.username,
+            mountable: true,
+            embassy_os,
+        }),
+    })
+}
+
+#[command(display(display_none))]
+pub async fn update(
+    #[context] ctx: RpcContext,
+    #[arg] id: BackupTargetId,
+    #[arg] hostname: String,
+    #[arg] path: PathBuf,
+    #[arg] username: String,
+    #[arg] password: Option<String>,
+) -> Result<KeyVal<BackupTargetId, BackupTarget>, Error> {
+    let id = if let BackupTargetId::Cifs { id } = id {
+        id
+    } else {
+        return Err(Error::new(
+            eyre!("Backup Target ID {} Not Found", id),
+            ErrorKind::NotFound,
+        ));
+    };
+    let cifs = Cifs {
+        hostname,
+        path,
+        username,
+        password,
+    };
+    let guard = TmpMountGuard::mount(&cifs, ReadOnly).await?;
+    let embassy_os = recovery_info(&guard).await?;
+    guard.unmount().await?;
+    let path_string = Path::new("/").join(&cifs.path).display().to_string();
+    if sqlx::query!(
+        "UPDATE cifs_shares SET hostname = $1, path = $2, username = $3, password = $4 WHERE id = $5",
+        cifs.hostname,
+        path_string,
+        cifs.username,
+        cifs.password,
+        id,
+    )
+    .execute(&ctx.secret_store)
+    .await?
+    .rows_affected()
+        == 0
+    {
+        return Err(Error::new(
+            eyre!("Backup Target ID {} Not Found", BackupTargetId::Cifs { id }),
+            ErrorKind::NotFound,
+        ));
+    };
+    Ok(KeyVal {
+        key: BackupTargetId::Cifs { id },
+        value: BackupTarget::Cifs(CifsBackupTarget {
+            hostname: cifs.hostname,
+            path: cifs.path,
+            username: cifs.username,
+            mountable: true,
+            embassy_os,
+        }),
+    })
+}
+
+#[command(display(display_none))]
+pub async fn remove(#[context] ctx: RpcContext, #[arg] id: BackupTargetId) -> Result<(), Error> {
+    let id = if let BackupTargetId::Cifs { id } = id {
+        id
+    } else {
+        return Err(Error::new(
+            eyre!("Backup Target ID {} Not Found", id),
+            ErrorKind::NotFound,
+        ));
+    };
+    if sqlx::query!("DELETE FROM cifs_shares WHERE id = $1", id)
+        .execute(&ctx.secret_store)
+        .await?
+        .rows_affected()
+        == 0
+    {
+        return Err(Error::new(
+            eyre!("Backup Target ID {} Not Found", BackupTargetId::Cifs { id }),
+            ErrorKind::NotFound,
+        ));
+    };
+    Ok(())
+}
+
+pub async fn load<Ex>(secrets: &mut Ex, id: i32) -> Result<Cifs, Error>
+where
+    for<'a> &'a mut Ex: Executor<'a, Database = Postgres>,
+{
+    let record = sqlx::query!(
+        "SELECT hostname, path, username, password FROM cifs_shares WHERE id = $1",
+        id
+    )
+    .fetch_one(secrets)
+    .await?;
+
+    Ok(Cifs {
+        hostname: record.hostname,
+        path: PathBuf::from(record.path),
+        username: record.username,
+        password: record.password,
+    })
+}
+
+pub async fn list<Ex>(secrets: &mut Ex) -> Result<Vec<(i32, CifsBackupTarget)>, Error>
+where
+    for<'a> &'a mut Ex: Executor<'a, Database = Postgres>,
+{
+    let mut records =
+        sqlx::query!("SELECT id, hostname, path, username, password FROM cifs_shares")
+            .fetch_many(secrets);
+
+    let mut cifs = Vec::new();
+    while let Some(query_result) = records.try_next().await? {
+        if let Some(record) = query_result.right() {
+            let mount_info = Cifs {
+                hostname: record.hostname,
+                path: PathBuf::from(record.path),
+                username: record.username,
+                password: record.password,
+            };
+            let embassy_os = async {
+                let guard = TmpMountGuard::mount(&mount_info, ReadOnly).await?;
+                let embassy_os = recovery_info(&guard).await?;
+                guard.unmount().await?;
+                Ok::<_, Error>(embassy_os)
+            }
+            .await;
+            cifs.push((
+                record.id,
+                CifsBackupTarget {
+                    hostname: mount_info.hostname,
+                    path: mount_info.path,
+                    username: mount_info.username,
+                    mountable: embassy_os.is_ok(),
+                    embassy_os: embassy_os.ok().and_then(|a| a),
+                },
+            ));
+        }
+    }
+
+    Ok(cifs)
+}

core/startos/src/backup/target/mod.rs

@@ -0,0 +1,307 @@
+use std::collections::BTreeMap;
+use std::path::{Path, PathBuf};
+
+use async_trait::async_trait;
+use chrono::{DateTime, Utc};
+use clap::ArgMatches;
+use color_eyre::eyre::eyre;
+use digest::generic_array::GenericArray;
+use digest::OutputSizeUser;
+use rpc_toolkit::command;
+use serde::{Deserialize, Serialize};
+use sha2::Sha256;
+use sqlx::{Executor, Postgres};
+use tokio::sync::Mutex;
+use tracing::instrument;
+
+use self::cifs::CifsBackupTarget;
+use crate::context::RpcContext;
+use crate::disk::mount::backup::BackupMountGuard;
+use crate::disk::mount::filesystem::block_dev::BlockDev;
+use crate::disk::mount::filesystem::cifs::Cifs;
+use crate::disk::mount::filesystem::{FileSystem, MountType, ReadWrite};
+use crate::disk::mount::guard::TmpMountGuard;
+use crate::disk::util::PartitionInfo;
+use crate::prelude::*;
+use crate::s9pk::manifest::PackageId;
+use crate::util::serde::{deserialize_from_str, display_serializable, serialize_display};
+use crate::util::{display_none, Version};
+
+pub mod cifs;
+
+#[derive(Debug, Deserialize, Serialize)]
+#[serde(tag = "type")]
+#[serde(rename_all = "kebab-case")]
+pub enum BackupTarget {
+    #[serde(rename_all = "kebab-case")]
+    Disk {
+        vendor: Option<String>,
+        model: Option<String>,
+        #[serde(flatten)]
+        partition_info: PartitionInfo,
+    },
+    Cifs(CifsBackupTarget),
+}
+
+#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone)]
+pub enum BackupTargetId {
+    Disk { logicalname: PathBuf },
+    Cifs { id: i32 },
+}
+impl BackupTargetId {
+    pub async fn load<Ex>(self, secrets: &mut Ex) -> Result<BackupTargetFS, Error>
+    where
+        for<'a> &'a mut Ex: Executor<'a, Database = Postgres>,
+    {
+        Ok(match self {
+            BackupTargetId::Disk { logicalname } => {
+                BackupTargetFS::Disk(BlockDev::new(logicalname))
+            }
+            BackupTargetId::Cifs { id } => BackupTargetFS::Cifs(cifs::load(secrets, id).await?),
+        })
+    }
+}
+impl std::fmt::Display for BackupTargetId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            BackupTargetId::Disk { logicalname } => write!(f, "disk-{}", logicalname.display()),
+            BackupTargetId::Cifs { id } => write!(f, "cifs-{}", id),
+        }
+    }
+}
+impl std::str::FromStr for BackupTargetId {
+    type Err = Error;
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s.split_once('-') {
+            Some(("disk", logicalname)) => Ok(BackupTargetId::Disk {
+                logicalname: Path::new(logicalname).to_owned(),
+            }),
+            Some(("cifs", id)) => Ok(BackupTargetId::Cifs { id: id.parse()? }),
+            _ => Err(Error::new(
+                eyre!("Invalid Backup Target ID"),
+                ErrorKind::InvalidBackupTargetId,
+            )),
+        }
+    }
+}
+impl<'de> Deserialize<'de> for BackupTargetId {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        deserialize_from_str(deserializer)
+    }
+}
+impl Serialize for BackupTargetId {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        serialize_display(self, serializer)
+    }
+}
+
+#[derive(Debug, Deserialize, Serialize)]
+#[serde(tag = "type")]
+#[serde(rename_all = "kebab-case")]
+pub enum BackupTargetFS {
+    Disk(BlockDev<PathBuf>),
+    Cifs(Cifs),
+}
+#[async_trait]
+impl FileSystem for BackupTargetFS {
+    async fn mount<P: AsRef<Path> + Send + Sync>(
+        &self,
+        mountpoint: P,
+        mount_type: MountType,
+    ) -> Result<(), Error> {
+        match self {
+            BackupTargetFS::Disk(a) => a.mount(mountpoint, mount_type).await,
+            BackupTargetFS::Cifs(a) => a.mount(mountpoint, mount_type).await,
+        }
+    }
+    async fn source_hash(
+        &self,
+    ) -> Result<GenericArray<u8, <Sha256 as OutputSizeUser>::OutputSize>, Error> {
+        match self {
+            BackupTargetFS::Disk(a) => a.source_hash().await,
+            BackupTargetFS::Cifs(a) => a.source_hash().await,
+        }
+    }
+}
+
+#[command(subcommands(cifs::cifs, list, info, mount, umount))]
+pub fn target() -> Result<(), Error> {
+    Ok(())
+}
+
+#[command(display(display_serializable))]
+pub async fn list(
+    #[context] ctx: RpcContext,
+) -> Result<BTreeMap<BackupTargetId, BackupTarget>, Error> {
+    let mut sql_handle = ctx.secret_store.acquire().await?;
+    let (disks_res, cifs) = tokio::try_join!(
+        crate::disk::util::list(&ctx.os_partitions),
+        cifs::list(sql_handle.as_mut()),
+    )?;
+    Ok(disks_res
+        .into_iter()
+        .flat_map(|mut disk| {
+            std::mem::take(&mut disk.partitions)
+                .into_iter()
+                .map(|part| {
+                    (
+                        BackupTargetId::Disk {
+                            logicalname: part.logicalname.clone(),
+                        },
+                        BackupTarget::Disk {
+                            vendor: disk.vendor.clone(),
+                            model: disk.model.clone(),
+                            partition_info: part,
+                        },
+                    )
+                })
+                .collect::<Vec<_>>()
+        })
+        .chain(
+            cifs.into_iter()
+                .map(|(id, cifs)| (BackupTargetId::Cifs { id }, BackupTarget::Cifs(cifs))),
+        )
+        .collect())
+}
+
+#[derive(Clone, Debug, Default, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct BackupInfo {
+    pub version: Version,
+    pub timestamp: Option<DateTime<Utc>>,
+    pub package_backups: BTreeMap<PackageId, PackageBackupInfo>,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct PackageBackupInfo {
+    pub title: String,
+    pub version: Version,
+    pub os_version: Version,
+    pub timestamp: DateTime<Utc>,
+}
+
+fn display_backup_info(info: BackupInfo, matches: &ArgMatches) {
+    use prettytable::*;
+
+    if matches.is_present("format") {
+        return display_serializable(info, matches);
+    }
+
+    let mut table = Table::new();
+    table.add_row(row![bc =>
+        "ID",
+        "VERSION",
+        "OS VERSION",
+        "TIMESTAMP",
+    ]);
+    table.add_row(row![
+        "EMBASSY OS",
+        info.version.as_str(),
+        info.version.as_str(),
+        &if let Some(ts) = &info.timestamp {
+            ts.to_string()
+        } else {
+            "N/A".to_owned()
+        },
+    ]);
+    for (id, info) in info.package_backups {
+        let row = row![
+            &*id,
+            info.version.as_str(),
+            info.os_version.as_str(),
+            &info.timestamp.to_string(),
+        ];
+        table.add_row(row);
+    }
+    table.print_tty(false).unwrap();
+}
+
+#[command(display(display_backup_info))]
+#[instrument(skip(ctx, password))]
+pub async fn info(
+    #[context] ctx: RpcContext,
+    #[arg(rename = "target-id")] target_id: BackupTargetId,
+    #[arg] password: String,
+) -> Result<BackupInfo, Error> {
+    let guard = BackupMountGuard::mount(
+        TmpMountGuard::mount(
+            &target_id
+                .load(ctx.secret_store.acquire().await?.as_mut())
+                .await?,
+            ReadWrite,
+        )
+        .await?,
+        &password,
+    )
+    .await?;
+
+    let res = guard.metadata.clone();
+
+    guard.unmount().await?;
+
+    Ok(res)
+}
+
+lazy_static::lazy_static! {
+    static ref USER_MOUNTS: Mutex<BTreeMap<BackupTargetId, BackupMountGuard<TmpMountGuard>>> =
+        Mutex::new(BTreeMap::new());
+}
+
+#[command]
+#[instrument(skip_all)]
+pub async fn mount(
+    #[context] ctx: RpcContext,
+    #[arg(rename = "target-id")] target_id: BackupTargetId,
+    #[arg] password: String,
+) -> Result<String, Error> {
+    let mut mounts = USER_MOUNTS.lock().await;
+
+    if let Some(existing) = mounts.get(&target_id) {
+        return Ok(existing.as_ref().display().to_string());
+    }
+
+    let guard = BackupMountGuard::mount(
+        TmpMountGuard::mount(
+            &target_id
+                .clone()
+                .load(ctx.secret_store.acquire().await?.as_mut())
+                .await?,
+            ReadWrite,
+        )
+        .await?,
+        &password,
+    )
+    .await?;
+
+    let res = guard.as_ref().display().to_string();
+
+    mounts.insert(target_id, guard);
+
+    Ok(res)
+}
+#[command(display(display_none))]
+#[instrument(skip_all)]
+pub async fn umount(
+    #[context] _ctx: RpcContext,
+    #[arg(rename = "target-id")] target_id: Option<BackupTargetId>,
+) -> Result<(), Error> {
+    let mut mounts = USER_MOUNTS.lock().await;
+    if let Some(target_id) = target_id {
+        if let Some(existing) = mounts.remove(&target_id) {
+            existing.unmount().await?;
+        }
+    } else {
+        for (_, existing) in std::mem::take(&mut *mounts) {
+            existing.unmount().await?;
+        }
+    }
+
+    Ok(())
+}

core/startos/src/bins/avahi_alias.rs

@@ -0,0 +1,163 @@
+use avahi_sys::{
+    self, avahi_client_errno, avahi_entry_group_add_service, avahi_entry_group_commit,
+    avahi_strerror, AvahiClient,
+};
+
+fn log_str_error(action: &str, e: i32) {
+    unsafe {
+        let e_str = avahi_strerror(e);
+        eprintln!(
+            "Could not {}: {:?}",
+            action,
+            std::ffi::CStr::from_ptr(e_str)
+        );
+    }
+}
+
+pub fn main() {
+    let aliases: Vec<_> = std::env::args().skip(1).collect();
+    unsafe {
+        let simple_poll = avahi_sys::avahi_simple_poll_new();
+        let poll = avahi_sys::avahi_simple_poll_get(simple_poll);
+        let mut box_err = Box::pin(0 as i32);
+        let err_c: *mut i32 = box_err.as_mut().get_mut();
+        let avahi_client = avahi_sys::avahi_client_new(
+            poll,
+            avahi_sys::AvahiClientFlags::AVAHI_CLIENT_NO_FAIL,
+            Some(client_callback),
+            std::ptr::null_mut(),
+            err_c,
+        );
+        if avahi_client == std::ptr::null_mut::<AvahiClient>() {
+            log_str_error("create Avahi client", *box_err);
+            panic!("Failed to create Avahi Client");
+        }
+        let group = avahi_sys::avahi_entry_group_new(
+            avahi_client,
+            Some(entry_group_callback),
+            std::ptr::null_mut(),
+        );
+        if group == std::ptr::null_mut() {
+            log_str_error("create Avahi entry group", avahi_client_errno(avahi_client));
+            panic!("Failed to create Avahi Entry Group");
+        }
+        let mut hostname_buf = vec![0];
+        let hostname_raw = avahi_sys::avahi_client_get_host_name_fqdn(avahi_client);
+        hostname_buf.extend_from_slice(std::ffi::CStr::from_ptr(hostname_raw).to_bytes_with_nul());
+        let buflen = hostname_buf.len();
+        debug_assert!(hostname_buf.ends_with(b".local\0"));
+        debug_assert!(!hostname_buf[..(buflen - 7)].contains(&b'.'));
+        // assume fixed length prefix on hostname due to local address
+        hostname_buf[0] = (buflen - 8) as u8; // set the prefix length to len - 8 (leading byte, .local, nul) for the main address
+        hostname_buf[buflen - 7] = 5; // set the prefix length to 5 for "local"
+        let mut res;
+        let http_tcp_cstr =
+            std::ffi::CString::new("_http._tcp").expect("Could not cast _http._tcp to c string");
+        res = avahi_entry_group_add_service(
+            group,
+            avahi_sys::AVAHI_IF_UNSPEC,
+            avahi_sys::AVAHI_PROTO_UNSPEC,
+            avahi_sys::AvahiPublishFlags_AVAHI_PUBLISH_USE_MULTICAST,
+            hostname_raw,
+            http_tcp_cstr.as_ptr(),
+            std::ptr::null(),
+            std::ptr::null(),
+            443,
+            // below is a secret final argument that the type signature of this function does not tell you that it
+            // needs. This is because the C lib function takes a variable number of final arguments indicating the
+            // desired TXT records to add to this service entry. The way it decides when to stop taking arguments
+            // from the stack and dereferencing them is when it finds a null pointer...because fuck you, that's why.
+            // The consequence of this is that forgetting this last argument will cause segfaults or other undefined
+            // behavior. Welcome back to the stone age motherfucker.
+            std::ptr::null::<libc::c_char>(),
+        );
+        if res < avahi_sys::AVAHI_OK {
+            log_str_error("add service to Avahi entry group", res);
+            panic!("Failed to load Avahi services");
+        }
+        eprintln!("Published {:?}", std::ffi::CStr::from_ptr(hostname_raw));
+        for alias in aliases {
+            let lan_address = alias + ".local";
+            let lan_address_ptr = std::ffi::CString::new(lan_address)
+                .expect("Could not cast lan address to c string");
+            res = avahi_sys::avahi_entry_group_add_record(
+                group,
+                avahi_sys::AVAHI_IF_UNSPEC,
+                avahi_sys::AVAHI_PROTO_UNSPEC,
+                avahi_sys::AvahiPublishFlags_AVAHI_PUBLISH_USE_MULTICAST
+                    | avahi_sys::AvahiPublishFlags_AVAHI_PUBLISH_ALLOW_MULTIPLE,
+                lan_address_ptr.as_ptr(),
+                avahi_sys::AVAHI_DNS_CLASS_IN as u16,
+                avahi_sys::AVAHI_DNS_TYPE_CNAME as u16,
+                avahi_sys::AVAHI_DEFAULT_TTL,
+                hostname_buf.as_ptr().cast(),
+                hostname_buf.len(),
+            );
+            if res < avahi_sys::AVAHI_OK {
+                log_str_error("add CNAME record to Avahi entry group", res);
+                panic!("Failed to load Avahi services");
+            }
+            eprintln!("Published {:?}", lan_address_ptr);
+        }
+        let commit_err = avahi_entry_group_commit(group);
+        if commit_err < avahi_sys::AVAHI_OK {
+            log_str_error("reset Avahi entry group", commit_err);
+            panic!("Failed to load Avahi services: reset");
+        }
+    }
+    std::thread::park()
+}
+
+unsafe extern "C" fn entry_group_callback(
+    _group: *mut avahi_sys::AvahiEntryGroup,
+    state: avahi_sys::AvahiEntryGroupState,
+    _userdata: *mut core::ffi::c_void,
+) {
+    match state {
+        avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_FAILURE => {
+            eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_FAILURE");
+        }
+        avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_COLLISION => {
+            eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_COLLISION");
+        }
+        avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_UNCOMMITED => {
+            eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_UNCOMMITED");
+        }
+        avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_ESTABLISHED => {
+            eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_ESTABLISHED");
+        }
+        avahi_sys::AvahiEntryGroupState_AVAHI_ENTRY_GROUP_REGISTERING => {
+            eprintln!("AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_REGISTERING");
+        }
+        other => {
+            eprintln!("AvahiCallback: EntryGroupState = {}", other);
+        }
+    }
+}
+
+unsafe extern "C" fn client_callback(
+    _group: *mut avahi_sys::AvahiClient,
+    state: avahi_sys::AvahiClientState,
+    _userdata: *mut core::ffi::c_void,
+) {
+    match state {
+        avahi_sys::AvahiClientState_AVAHI_CLIENT_FAILURE => {
+            eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_FAILURE");
+        }
+        avahi_sys::AvahiClientState_AVAHI_CLIENT_S_RUNNING => {
+            eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_S_RUNNING");
+        }
+        avahi_sys::AvahiClientState_AVAHI_CLIENT_CONNECTING => {
+            eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_CONNECTING");
+        }
+        avahi_sys::AvahiClientState_AVAHI_CLIENT_S_COLLISION => {
+            eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_S_COLLISION");
+        }
+        avahi_sys::AvahiClientState_AVAHI_CLIENT_S_REGISTERING => {
+            eprintln!("AvahiCallback: ClientState = AVAHI_CLIENT_S_REGISTERING");
+        }
+        other => {
+            eprintln!("AvahiCallback: ClientState = {}", other);
+        }
+    }
+}