# Routstr over FIPS with a local Llama provider Status: candidate provider/runtime packaging and mesh request policy implemented; no deployment or live acceptance yet. Branch: `feat/routstr-fips-llama`, based on `fcc18b3`. ## Goal Node A selects Node B by identity, sees its model and pricing, authorizes a bounded spend, obtains inference over FIPS through B's Routstr server backed by llama-server, and records the correct payment outcome. Failures and restarts must not trigger duplicate payments, expose the raw runtime, or silently change transport/provider. Initial runtime assumption: llama.cpp's llama-server serving a Llama model. Verify upstream compatibility before selecting runtime/server versions. ## Source audit at kickoff - `core/archipelago/src/api/handler/routstr_proxy.rs`: AIUI's separate, session-authenticated client uses a fixed public HTTPS aggregator, model price estimates and Cashu payment/refund handling. - `core/archipelago/src/assistant/backends/routstr.rs`: assistant adapter uses Nostr discovery and documents an originally unobserved provider wire contract. Do not treat those historical fixtures as current upstream compatibility proof. - `core/archipelago/src/fips/dial.rs`: mesh identity resolution, restricted peer port 5679, explicit FIPS selection and single-delivery retry distinctions. Reuse these guarantees; do not introduce generic retries for paid POSTs. - `core/archipelago/src/api/rpc/package/config.rs`: legacy Routstr configuration exposes port 8200 and copies node identity material into environment variables. This is not the proposed server packaging/security contract. - No `apps/routstr/manifest.yml` existed at kickoff. A candidate manifest and `docker/routstr/Dockerfile` now implement the private server package foundation. Runtime lifecycle remains unqualified. Never hand-edit generated app ports. - `docs/external-access-and-websites.md`: setup/funding UI and private Framework UAT results are recorded; real paid inference remains a separate acceptance gate. ## Architecture and invariants AIUI / assistant / website generation -> unified node Routstr client -> FIPS peer gateway -> Routstr server -> private llama-server. - One provider selection, capability model and spending ledger across callers. - Public HTTPS providers remain an explicit alternative. FIPS-selected requests fail closed when the mesh is unavailable; never silently use Tor or HTTPS. - Restrict peer inference routes independently of owner dashboard sessions. Specify peer authentication, identity binding, authorization and replay protection before exposing a gateway. Do not expose wallet administration or the raw model server. - Treat announcements, URLs, redirects, pricing and model capabilities as untrusted. - Keep service signing scoped; do not pass the node master secret to the container. - Separate wallet funding from permission to spend. Discovery and setup spend nothing. - Persist allowance reservations, submissions, settlement and uncertain outcomes. Never automatically pay again to recover a potentially delivered request. - Preserve app data, payment records, wallets and explicit uninstall decisions. - FIPS inference does not imply mint settlement/discovery uses FIPS. Document each network dependency without overstating transport privacy. ## Delivery checklist ### 1. Upstream contract verification (next) - [x] Pin candidate upstream source revisions: Routstr v0.5.0 and llama.cpp b9860. Image builds, base-image digest pins and runtime compatibility remain pending. - [ ] Verify catalog, chat, tools, streaming, pricing and mint contracts against those revisions; record citations and reproducible disposable fixtures. - [ ] Verify payment authentication, refunds, idempotency and status/recovery support. Explicitly document missing upstream guarantees and required extensions. - [ ] Specify authenticated mesh gateway routes and identity-bound provider records. - [ ] Define a disposable test environment with no production wallet or service access. ### 2. Local provider - [ ] Add manifest-driven Routstr app with private upstream configuration, scoped secrets, persistent accounting, resource limits and truthful readiness. - [ ] Connect an existing private llama-server; then add managed local runtime support. - [ ] Require explicit model acquisition consent; record provenance, storage/memory needs, context limits and tested chat templates/capabilities. - [ ] Demonstrate one non-streaming completion through the local provider. ### 3. FIPS vertical slice - [ ] Support explicit provider identity entry without depending on public discovery. - [ ] Add restricted, authenticated inference/catalog gateway and mesh-only selection. - [ ] Demonstrate two-node inference and no fallback on mesh loss. - [ ] Verify raw model server and administrative routes remain inaccessible to peers. ### 4. Shared client and payment correctness - [ ] Delegate both current Routstr paths to a common provider/transport/payment service. - [ ] Enforce capabilities, privacy screening and allowance limits across all callers. - [ ] Implement durable request states: prepared, reserved, submitted, completed, settled, uncertain and reconciled; specify transitions before coding settlement. - [ ] Test concurrent spends, duplicate submissions, response loss, cancellation, process restarts, mint failures and refund failures with isolated fixtures. - [ ] Add validated discovery and configurable public providers without silent failover. - [ ] Qualify streaming and tool calls after non-streaming payment correctness. ### 5. Operator experience and lifecycle - [ ] Consumer: identity, model, transport, pricing, readiness, separate balance and allowance, uncertain transactions and recovery actions. - [ ] Provider: runtime/model readiness, mesh publication opt-in, pricing, accepted mints, accounting and concurrency/resource limits. - [ ] Test install/start/stop/restart/manager restart/upgrade/preserved uninstall/reinstall. - [ ] Qualify embedded/standalone responsive and accessible setup/error states. ### 6. Acceptance and publication - [ ] Record source tests separately from disposable integration and live acceptance. - [ ] Run backend tests only through `scripts/test-backend-isolated.sh`. - [ ] Obtain explicit authorization and a spending cap before real paid inference tests. - [ ] Record two-node FIPS paid completion and fault/recovery evidence with no second payment. - [ ] Review/contribute through ngit; mirror identical accepted commits to Gitea per `AGENTS.md`. No publication is authorized by this implementation kickoff. - [ ] Read and retain all active release regression tasks before release preparation; this feature does not close unrelated paid-file or lifecycle incidents. ## Evidence Upstream source audit: Routstr Core v0.5.0 resolves to `4aa58bf380dd41f8e3ab7dc768b8e1d112d92019`. The source was fetched read-only into `/tmp/archy-routstr-research/routstr-core`; no upstream scripts were executed. Pinned source references: - [Generic OpenAI upstream](https://github.com/Routstr/routstr-core/blob/4aa58bf380dd41f8e3ab7dc768b8e1d112d92019/routstr/upstream/generic.py) - [Payment dispatch](https://github.com/Routstr/routstr-core/blob/4aa58bf380dd41f8e3ab7dc768b8e1d112d92019/routstr/proxy.py) - [Refund retrieval](https://github.com/Routstr/routstr-core/blob/4aa58bf380dd41f8e3ab7dc768b8e1d112d92019/routstr/balance.py) - [Settings and secret bootstrap](https://github.com/Routstr/routstr-core/blob/4aa58bf380dd41f8e3ab7dc768b8e1d112d92019/routstr/core/settings.py) Important finding: X-Cashu per-request and bearer-funded-key flows are distinct. Current Archipelago bearer-token clients must not be presumed compatible with per-request change accounting. Refund/status and actual paid-inference behavior remain unverified. The source supports a generic OpenAI upstream; actual Llama model compatibility is not yet demonstrated. Candidate package includes immutable source checkout, frozen dependency install, loopback-only API, dedicated admin secret, read-only root and app-owned persistent state. It defaults to no upstream/mint/publication identity. Upstream saved settings can override defaults, and exchange-rate/background tasks still exist; this is not a claim of zero external traffic. See `apps/routstr/README.md` for remaining gates. No image build, container start, model download, live-node change, wallet spending, deployment or publication performed. Source-only validation results are recorded below; they are not lifecycle or protocol acceptance. Source-only results: - Manifest preflight: 15 passed, zero failed, one secret-like environment warning (the manifest contains an encryption-key file path, not a literal key). - `python3 -m unittest discover -s tests/routstr-provider -p 'test_*.py' -v`: twelve provider/runtime packaging-invariant tests passed. - `git diff --check`: passed (later committed as `bc08073`). - Canonical Rust parsing: one isolated container test passed for both manifests through `ARCHY_TEST_PACKAGE=archipelago-container scripts/test-backend-isolated.sh routstr_provider_candidates`. - Image builds, catalog generation and runtime tests pending. Catalog surfaces were intentionally not regenerated for unqualified candidates. ## Second implementation slice: runtime and FIPS policy - Added `apps/llama-server` and `docker/llama-server`, pinned to llama.cpp b9860 (`fdb1db877c526ec90f668eca1b858da5dba85560`). Reviewed its server documentation, CPU Dockerfile and CMake options at that exact revision. CPU-only source build, no published host ports, dedicated API key, read-only operator-provided GGUF, stable `archy-llama` alias and explicit initial resource limits. - Llama manifest preflight: 16 passed, zero failures/warnings. Missing model import UX and prerequisite checks intentionally block catalog readiness. - Added `assistant/routstr_transport.rs`: canonical npub validation, exact typed catalog/completion routes, explicit FIPS-only requests, no fallback host, and single-delivery policy for completion requests. Four source tests cover address, route and delivery-policy construction. This is a request-policy foundation, **not an enabled client or server gateway**. It attaches no credentials; peer identity binding, authorization and durable settlement remain required. - Full backend compilation through the isolated runner is blocked by pre-existing `monitoring/store.rs:105` use of `AtomicU64::try_update`, unstable on the installed Rust 1.94.1 toolchain (`E0658`). The new FIPS unit tests did not execute. No unrestricted host test run or unrelated source workaround was used. - This environment has no `podman` executable; no image build or container lifecycle claim is made. No model downloads, node changes, paid requests or publication. Next: compile/run the FIPS tests with the supported backend toolchain; build the candidate images in a disposable rootless runtime; then implement the authenticated mesh gateway and durable request journal before connecting paid callers. Configure and qualify the llama-server upstream/key handoff without exposing the raw runtime. ## Third slice: toolchain, images and local paid inference (2026-10-25) Supersedes the second slice's blockers. Disposable rootless Podman 5.4.2 on the development host (not a node); fake-Lightning test mint; no real funds. Toolchain: - `088eee55` deliberately moved `monitoring/store.rs` to `AtomicU64::try_update`; the distro Rust 1.94.1 is simply too old. The installed rustup stable 1.96.0 compiles it unchanged. The four `routstr_transport` tests **passed** through `scripts/test-backend-isolated.sh` with that toolchain first on `PATH` and a dedicated `CARGO_TARGET_DIR`. No monitoring code was patched. Image defects found by running them, both fixed with regression tests: - llama-server exited 132 (SIGILL) on a CPU without AVX: `GGML_NATIVE=OFF` still bakes in AVX/AVX2. Now built like upstream's `cpu.Dockerfile` (`GGML_BACKEND_DL=ON`, `GGML_CPU_ALL_VARIANTS=ON`): sse42 through zen4 backends, best one loaded at start. - Routstr failed every startup: `alembic.ini` uses a cwd-relative `script_location`, and the image's cwd is `/data`. Now `%(here)s/migrations`. Local stack (manifest security settings: read-only root, `--cap-drop=all`, no-new-privileges, tmpfs `/tmp`, model mounted read-only): - Model: TinyLlama-1.1B-Chat-v1.0 Q4_K_M, Apache-2.0, Llama architecture, `TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF` revision `52e7645ba7c309695bec7ac98f4f005b139cf465`, 668788096 bytes, sha256 `9fecc3b3cd76bba89d504f29b616eedf7da85b96540e490ca5824d3f7d2776a0` (verified). - llama-server: model loaded <1s; chat completions return 401 without or with a wrong key and 200 with usage accounting with the dedicated key. `/health` and `/v1/models` are public by upstream design (alias only). - Routstr with `UPSTREAM_BASE_URL=http://llama-server:8080/v1` and the llama key as `UPSTREAM_API_KEY`: admin API redacts the key. - **Pricing is mandatory.** Routstr fails closed: an unpriced upstream model is imported disabled and `/v1/models` is empty. `MODELS_PATH` is unread at this revision; pricing goes through `POST /admin/api/upstream-providers/{id}/models` (USD per token, converted to sats via Kraken/Coinbase/Binance price feeds). Install therefore needs a post-start pricing step before the provider is usable. - Test mint: `docker.io/cashubtc/nutshell:0.20.0` (`sha256:fae7b126ecc2ca797f3f572e5e86b3623bda26fdee0f97489ea9e3d09089950b`), FakeWallet backend. 0.17.0 is incompatible with Routstr's bundled cashu 0.20.0 wallet (`KeysResponse.keysets.active` missing). Tokens must name the exact URL in `CASHU_MINTS`; others are rejected as `untrusted_mint`. Payment contract, observed against Routstr v0.5.0 (20-sat token each): - `X-Cashu: `: 200, charged 70 msat, change returned immediately as a new token in the `x-cashu` response header (redeemed: +17 sat). On a 503 the unspent token is also returned in `x-cashu`. - `Authorization: Bearer `: 200, charged 53 msat, **no change header**; the token becomes a server-side `sk-…` account holding the remainder (18947 msat). `POST /v1/balance/refund` returns it as a token; a repeated refund call replays the same (already-spent) token rather than paying twice. - Consequence: `api/handler/routstr_proxy.rs` sends Bearer but harvests change from `x-cashu`/`x-cashu-refund`. Against a v0.5.0 server that change stays on the provider's account and is never refunded. The public aggregator AIUI uses may run a different version; this is proven only for routstr-core v0.5.0. The FIPS buyer must use `X-Cashu` (per-request, change in-band) or an explicit Bearer account with journaled refund. Still pending: manifest upstream/key wiring and post-install pricing, private stack network isolation, lifecycle qualification, the FIPS gateway, the buyer journal, and two-node acceptance. ## Fourth slice: node wallet integration and default pricing (2026-10-25) Operator decisions: the seller trusts the node ecash wallet's mints (Minibits by default) and pays earnings to the wallet's Minibits Lightning address; the model ships priced, and mesh selling stays off until the operator opts in. - An empty `CASHU_MINTS` was shown live to make Routstr trust `https://mint.minibits.cash/Bitcoin` silently (`_compute_primary_mint`). - Routstr persists settings on first start and its database then wins over env (shown live: a changed `CASHU_MINTS` was ignored on restart). - New `derived_env` placeholders `{{ECASH_MINTS}}` (accepted mints; empty list refuses the start) and `{{ECASH_LN_ADDRESS}}` (empty with a warning when unavailable). Orchestrator and container tests pass in the isolated runner. - `docker/routstr/archy-start.py` is PID 1: starts uvicorn, re-applies mints and payout address via `PATCH /admin/api/settings` on every start, prices `archy-llama` once at $1/$2 per million tokens unless already priced, and maps an orchestrator stop to exit 0. Verified live: fresh install priced and served; changed wallet mints/address applied on restart; operator price kept; paid X-Cashu completion at the operator price; stop exit 0, SIGKILL 137. - The buyer-side Bearer/X-Cashu defect is fixed separately on `fix/routstr-xcashu-change`, ngit proposal `437748339cad1c5d70d8f399dac084244be06b01844f121823552659c2dd450f`. Next: authenticated FIPS gateway with serving opt-in (default off). ## Fifth slice: seller FIPS gateway (2026-10-25) - `api/handler/routstr_gateway.rs` serves exactly `GET /routstr/v1/models` and `POST /routstr/v1/chat/completions` (`RoutstrOperation::from_request`), forwarded to the Routstr app on `127.0.0.1:8200`. Both are on `is_peer_allowed_path`; every other `/routstr/...` path stays 404. - Refused with 404 unless `settings/routstr-selling.json` has `enabled: true` (`system.settings` key `routstr_selling`; missing or corrupt reads as off; not in the assistant's settable keys) **and** the request arrived on the FIPS peer listener (new `FipsPeerListener` request extension) from a ULA source. The path allowlist also admits these paths on the main listener, where the handler refuses them. - Payment is the authentication: completions need an `X-Cashu` token (402 otherwise). Only `X-Cashu` and `Content-Type` are forwarded (no `Authorization`, cookies, forwarded-for or query strings). Routstr's `x-cashu` change header is passed back, also on refusals. `stream: true` is refused (change arrives in a header). 256 KiB request / 4 MiB response caps, two concurrent requests (503 beyond), no retries. The caller's FIPS address is logged; tokens never are. - Tests: six gateway tests against a fake Routstr, the settings test and the peer path filter, in the isolated runner (39 `routstr`/filter tests pass). - Not yet shown: a request from a second node over a real FIPS link. Next: buyer side (`RoutstrFipsTarget`, X-Cashu, durable request journal), an operator toggle in the UI, then two-node acceptance with a spending cap. ## Sixth slice: buyer, settings UI and first two-node run (2026-10-25) - Buyer: `assistant/routstr_peer.rs`. With `routstr_peer` set in the AI provider settings, AIUI's Routstr catalog and completions go to that peer over FIPS only. The token is the seller's `sats_pricing.max_cost` rounded up plus one, journaled (`assistant/routstr-peer-journal/`) before sending; change, or the whole token after a refusal or no answer, is received back; a token the mint reports spent is final; unreachable-mint cases stay journaled and are retried before the next paid request. The X-Cashu fix (ngit `43774833…`) is cherry-picked onto this branch. - UI: Settings → System → "AI over the mesh" (sell on/off with this node's npub; buy-from-peer npub). The AI connection modal keeps `routstr_peer`. - Two-node run, operator-authorized with a 100-sat cap, on branch build `e77fbe02`: seller archy-x250-pa3 (llama-server + Routstr installed from preloaded images; mint `https://mint.minibits.cash/Bitcoin`; `archy-llama` `max_cost` 6.44 sats), buyer archy-x250-beta (hostname bellevue). - Selling off: the buyer's catalog request reached pa3 over FIPS and got 404. - Selling on: catalog 200 over FIPS; one completion (28 prompt + 23 completion tokens) paid with an 8-sat token, 94 msat charged, 7 sats change returned and received. Buyer wallet 120 → 119 sats, budget 1/100 spent, journal empty. Seller logged `change_returned=true` and the caller's FIPS address. - Found on the way, not fixed here: - pa3's `archipelago-source` image build has failed every reconcile pass since at least 2026-10-24; each pass holds the lifecycle lock, so installs fail with "An app lifecycle operation is active" unless sent between passes. - `package.install` requires `dockerImage` even for manifest-built apps. - Existing-only reconcile resolves env (and logs an error) for a manifest that is on disk but not installed, e.g. routstr before llama-server's key exists. - Still open: the assistant's own Routstr fallback still uses the public aggregator when a peer is set; streaming; choosing a peer from federation instead of pasting an npub; lifecycle gate; base-image digest pins.