archy/docs/routstr-fips-llama-integration.md

Routstr over FIPS with a local Llama provider

Status: candidate provider/runtime packaging and mesh request policy implemented; no deployment or live acceptance yet. Branch: feat/routstr-fips-llama, based on fcc18b3.

Goal

Node A selects Node B by identity, sees its model and pricing, authorizes a bounded spend, obtains inference over FIPS through B's Routstr server backed by llama-server, and records the correct payment outcome. Failures and restarts must not trigger duplicate payments, expose the raw runtime, or silently change transport/provider.

Initial runtime assumption: llama.cpp's llama-server serving a Llama model. Verify upstream compatibility before selecting runtime/server versions.

Source audit at kickoff

• core/archipelago/src/api/handler/routstr_proxy.rs: AIUI's separate, session-authenticated client uses a fixed public HTTPS aggregator, model price estimates and Cashu payment/refund handling.
• core/archipelago/src/assistant/backends/routstr.rs: assistant adapter uses Nostr discovery and documents an originally unobserved provider wire contract. Do not treat those historical fixtures as current upstream compatibility proof.
• core/archipelago/src/fips/dial.rs: mesh identity resolution, restricted peer port 5679, explicit FIPS selection and single-delivery retry distinctions. Reuse these guarantees; do not introduce generic retries for paid POSTs.
• core/archipelago/src/api/rpc/package/config.rs: legacy Routstr configuration exposes port 8200 and copies node identity material into environment variables. This is not the proposed server packaging/security contract.
• No apps/routstr/manifest.yml existed at kickoff. A candidate manifest and docker/routstr/Dockerfile now implement the private server package foundation. Runtime lifecycle remains unqualified. Never hand-edit generated app ports.
• docs/external-access-and-websites.md: setup/funding UI and private Framework UAT results are recorded; real paid inference remains a separate acceptance gate.

Architecture and invariants

AIUI / assistant / website generation -> unified node Routstr client -> FIPS peer gateway -> Routstr server -> private llama-server.

• One provider selection, capability model and spending ledger across callers.
• Public HTTPS providers remain an explicit alternative. FIPS-selected requests fail closed when the mesh is unavailable; never silently use Tor or HTTPS.
• Restrict peer inference routes independently of owner dashboard sessions. Specify peer authentication, identity binding, authorization and replay protection before exposing a gateway. Do not expose wallet administration or the raw model server.
• Treat announcements, URLs, redirects, pricing and model capabilities as untrusted.
• Keep service signing scoped; do not pass the node master secret to the container.
• Separate wallet funding from permission to spend. Discovery and setup spend nothing.
• Persist allowance reservations, submissions, settlement and uncertain outcomes. Never automatically pay again to recover a potentially delivered request.
• Preserve app data, payment records, wallets and explicit uninstall decisions.
• FIPS inference does not imply mint settlement/discovery uses FIPS. Document each network dependency without overstating transport privacy.

Delivery checklist

1. Upstream contract verification (next)

• Pin candidate upstream source revisions: Routstr v0.5.0 and llama.cpp b9860. Image builds, base-image digest pins and runtime compatibility remain pending.
• Verify catalog, chat, tools, streaming, pricing and mint contracts against those revisions; record citations and reproducible disposable fixtures.
• Verify payment authentication, refunds, idempotency and status/recovery support. Explicitly document missing upstream guarantees and required extensions.
• Specify authenticated mesh gateway routes and identity-bound provider records.
• Define a disposable test environment with no production wallet or service access.

2. Local provider

• Add manifest-driven Routstr app with private upstream configuration, scoped secrets, persistent accounting, resource limits and truthful readiness.
• Connect an existing private llama-server; then add managed local runtime support.
• Require explicit model acquisition consent; record provenance, storage/memory needs, context limits and tested chat templates/capabilities.
• Demonstrate one non-streaming completion through the local provider.

3. FIPS vertical slice

• Support explicit provider identity entry without depending on public discovery.
• Add restricted, authenticated inference/catalog gateway and mesh-only selection.
• Demonstrate two-node inference and no fallback on mesh loss.
• Verify raw model server and administrative routes remain inaccessible to peers.

4. Shared client and payment correctness

• Delegate both current Routstr paths to a common provider/transport/payment service.
• Enforce capabilities, privacy screening and allowance limits across all callers.
• Implement durable request states: prepared, reserved, submitted, completed, settled, uncertain and reconciled; specify transitions before coding settlement.
• Test concurrent spends, duplicate submissions, response loss, cancellation, process restarts, mint failures and refund failures with isolated fixtures.
• Add validated discovery and configurable public providers without silent failover.
• Qualify streaming and tool calls after non-streaming payment correctness.

5. Operator experience and lifecycle

• Consumer: identity, model, transport, pricing, readiness, separate balance and allowance, uncertain transactions and recovery actions.
• Provider: runtime/model readiness, mesh publication opt-in, pricing, accepted mints, accounting and concurrency/resource limits.
• Test install/start/stop/restart/manager restart/upgrade/preserved uninstall/reinstall.
• Qualify embedded/standalone responsive and accessible setup/error states.

6. Acceptance and publication

• Record source tests separately from disposable integration and live acceptance.
• Run backend tests only through scripts/test-backend-isolated.sh.
• Obtain explicit authorization and a spending cap before real paid inference tests.
• Record two-node FIPS paid completion and fault/recovery evidence with no second payment.
• Review/contribute through ngit; mirror identical accepted commits to Gitea per AGENTS.md. No publication is authorized by this implementation kickoff.
• Read and retain all active release regression tasks before release preparation; this feature does not close unrelated paid-file or lifecycle incidents.

Evidence

Upstream source audit: Routstr Core v0.5.0 resolves to 4aa58bf380dd41f8e3ab7dc768b8e1d112d92019. The source was fetched read-only into /tmp/archy-routstr-research/routstr-core; no upstream scripts were executed.

Pinned source references:

• Generic OpenAI upstream
• Payment dispatch
• Refund retrieval
• Settings and secret bootstrap

Important finding: X-Cashu per-request and bearer-funded-key flows are distinct. Current Archipelago bearer-token clients must not be presumed compatible with per-request change accounting. Refund/status and actual paid-inference behavior remain unverified. The source supports a generic OpenAI upstream; actual Llama model compatibility is not yet demonstrated.

Candidate package includes immutable source checkout, frozen dependency install, loopback-only API, dedicated admin secret, read-only root and app-owned persistent state. It defaults to no upstream/mint/publication identity. Upstream saved settings can override defaults, and exchange-rate/background tasks still exist; this is not a claim of zero external traffic. See apps/routstr/README.md for remaining gates.

No image build, container start, model download, live-node change, wallet spending, deployment or publication performed. Source-only validation results are recorded below; they are not lifecycle or protocol acceptance.

Source-only results:

• Manifest preflight: 15 passed, zero failed, one secret-like environment warning (the manifest contains an encryption-key file path, not a literal key).
• python3 -m unittest discover -s tests/routstr-provider -p 'test_*.py' -v: twelve provider/runtime packaging-invariant tests passed.
• git diff --check: passed (later committed as bc08073).
• Canonical Rust parsing: one isolated container test passed for both manifests through ARCHY_TEST_PACKAGE=archipelago-container scripts/test-backend-isolated.sh routstr_provider_candidates.
• Image builds, catalog generation and runtime tests pending. Catalog surfaces were intentionally not regenerated for unqualified candidates.

Second implementation slice: runtime and FIPS policy

• Added apps/llama-server and docker/llama-server, pinned to llama.cpp b9860 (fdb1db877c526ec90f668eca1b858da5dba85560). Reviewed its server documentation, CPU Dockerfile and CMake options at that exact revision. CPU-only source build, no published host ports, dedicated API key, read-only operator-provided GGUF, stable archy-llama alias and explicit initial resource limits.
• Llama manifest preflight: 16 passed, zero failures/warnings. Missing model import UX and prerequisite checks intentionally block catalog readiness.
• Added assistant/routstr_transport.rs: canonical npub validation, exact typed catalog/completion routes, explicit FIPS-only requests, no fallback host, and single-delivery policy for completion requests. Four source tests cover address, route and delivery-policy construction. This is a request-policy foundation, not an enabled client or server gateway. It attaches no credentials; peer identity binding, authorization and durable settlement remain required.
• Full backend compilation through the isolated runner is blocked by pre-existing monitoring/store.rs:105 use of AtomicU64::try_update, unstable on the installed Rust 1.94.1 toolchain (E0658). The new FIPS unit tests did not execute. No unrestricted host test run or unrelated source workaround was used.
• This environment has no podman executable; no image build or container lifecycle claim is made. No model downloads, node changes, paid requests or publication.

Next: compile/run the FIPS tests with the supported backend toolchain; build the candidate images in a disposable rootless runtime; then implement the authenticated mesh gateway and durable request journal before connecting paid callers. Configure and qualify the llama-server upstream/key handoff without exposing the raw runtime.

Third slice: toolchain, images and local paid inference (2026-10-25)

Supersedes the second slice's blockers. Disposable rootless Podman 5.4.2 on the development host (not a node); fake-Lightning test mint; no real funds.

Toolchain:

• 088eee55 deliberately moved monitoring/store.rs to AtomicU64::try_update; the distro Rust 1.94.1 is simply too old. The installed rustup stable 1.96.0 compiles it unchanged. The four routstr_transport tests passed through scripts/test-backend-isolated.sh with that toolchain first on PATH and a dedicated CARGO_TARGET_DIR. No monitoring code was patched.

Image defects found by running them, both fixed with regression tests:

• llama-server exited 132 (SIGILL) on a CPU without AVX: GGML_NATIVE=OFF still bakes in AVX/AVX2. Now built like upstream's cpu.Dockerfile (GGML_BACKEND_DL=ON, GGML_CPU_ALL_VARIANTS=ON): sse42 through zen4 backends, best one loaded at start.
• Routstr failed every startup: alembic.ini uses a cwd-relative script_location, and the image's cwd is /data. Now %(here)s/migrations.

Local stack (manifest security settings: read-only root, --cap-drop=all, no-new-privileges, tmpfs /tmp, model mounted read-only):

• Model: TinyLlama-1.1B-Chat-v1.0 Q4_K_M, Apache-2.0, Llama architecture, TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF revision 52e7645ba7c309695bec7ac98f4f005b139cf465, 668788096 bytes, sha256 9fecc3b3cd76bba89d504f29b616eedf7da85b96540e490ca5824d3f7d2776a0 (verified).
• llama-server: model loaded <1s; chat completions return 401 without or with a wrong key and 200 with usage accounting with the dedicated key. /health and /v1/models are public by upstream design (alias only).
• Routstr with UPSTREAM_BASE_URL=http://llama-server:8080/v1 and the llama key as UPSTREAM_API_KEY: admin API redacts the key.
• Pricing is mandatory. Routstr fails closed: an unpriced upstream model is imported disabled and /v1/models is empty. MODELS_PATH is unread at this revision; pricing goes through POST /admin/api/upstream-providers/{id}/models (USD per token, converted to sats via Kraken/Coinbase/Binance price feeds). Install therefore needs a post-start pricing step before the provider is usable.
• Test mint: docker.io/cashubtc/nutshell:0.20.0 (sha256:fae7b126ecc2ca797f3f572e5e86b3623bda26fdee0f97489ea9e3d09089950b), FakeWallet backend. 0.17.0 is incompatible with Routstr's bundled cashu 0.20.0 wallet (KeysResponse.keysets.active missing). Tokens must name the exact URL in CASHU_MINTS; others are rejected as untrusted_mint.

Payment contract, observed against Routstr v0.5.0 (20-sat token each):

• X-Cashu: <token>: 200, charged 70 msat, change returned immediately as a new token in the x-cashu response header (redeemed: +17 sat). On a 503 the unspent token is also returned in x-cashu.
• Authorization: Bearer <token>: 200, charged 53 msat, no change header; the token becomes a server-side sk-… account holding the remainder (18947 msat). POST /v1/balance/refund returns it as a token; a repeated refund call replays the same (already-spent) token rather than paying twice.
• Consequence: api/handler/routstr_proxy.rs sends Bearer but harvests change from x-cashu/x-cashu-refund. Against a v0.5.0 server that change stays on the provider's account and is never refunded. The public aggregator AIUI uses may run a different version; this is proven only for routstr-core v0.5.0. The FIPS buyer must use X-Cashu (per-request, change in-band) or an explicit Bearer account with journaled refund.

Still pending: manifest upstream/key wiring and post-install pricing, private stack network isolation, lifecycle qualification, the FIPS gateway, the buyer journal, and two-node acceptance.

Fourth slice: node wallet integration and default pricing (2026-10-25)

Operator decisions: the seller trusts the node ecash wallet's mints (Minibits by default) and pays earnings to the wallet's Minibits Lightning address; the model ships priced, and mesh selling stays off until the operator opts in.

• An empty CASHU_MINTS was shown live to make Routstr trust https://mint.minibits.cash/Bitcoin silently (_compute_primary_mint).
• Routstr persists settings on first start and its database then wins over env (shown live: a changed CASHU_MINTS was ignored on restart).
• New derived_env placeholders {{ECASH_MINTS}} (accepted mints; empty list refuses the start) and {{ECASH_LN_ADDRESS}} (empty with a warning when unavailable). Orchestrator and container tests pass in the isolated runner.
• docker/routstr/archy-start.py is PID 1: starts uvicorn, re-applies mints and payout address via PATCH /admin/api/settings on every start, prices archy-llama once at $1/$2 per million tokens unless already priced, and maps an orchestrator stop to exit 0. Verified live: fresh install priced and served; changed wallet mints/address applied on restart; operator price kept; paid X-Cashu completion at the operator price; stop exit 0, SIGKILL 137.
• The buyer-side Bearer/X-Cashu defect is fixed separately on fix/routstr-xcashu-change, ngit proposal 437748339cad1c5d70d8f399dac084244be06b01844f121823552659c2dd450f.

Next: authenticated FIPS gateway with serving opt-in (default off).

Fifth slice: seller FIPS gateway (2026-10-25)

• api/handler/routstr_gateway.rs serves exactly GET /routstr/v1/models and POST /routstr/v1/chat/completions (RoutstrOperation::from_request), forwarded to the Routstr app on 127.0.0.1:8200. Both are on is_peer_allowed_path; every other /routstr/... path stays 404.
• Refused with 404 unless settings/routstr-selling.json has enabled: true (system.settings key routstr_selling; missing or corrupt reads as off; not in the assistant's settable keys) and the request arrived on the FIPS peer listener (new FipsPeerListener request extension) from a ULA source. The path allowlist also admits these paths on the main listener, where the handler refuses them.
• Payment is the authentication: completions need an X-Cashu token (402 otherwise). Only X-Cashu and Content-Type are forwarded (no Authorization, cookies, forwarded-for or query strings). Routstr's x-cashu change header is passed back, also on refusals. stream: true is refused (change arrives in a header). 256 KiB request / 4 MiB response caps, two concurrent requests (503 beyond), no retries. The caller's FIPS address is logged; tokens never are.
• Tests: six gateway tests against a fake Routstr, the settings test and the peer path filter, in the isolated runner (39 routstr/filter tests pass).
• Not yet shown: a request from a second node over a real FIPS link.

Next: buyer side (RoutstrFipsTarget, X-Cashu, durable request journal), an operator toggle in the UI, then two-node acceptance with a spending cap.

Sixth slice: buyer, settings UI and first two-node run (2026-10-25)

• Buyer: assistant/routstr_peer.rs. With routstr_peer set in the AI provider settings, AIUI's Routstr catalog and completions go to that peer over FIPS only. The token is the seller's sats_pricing.max_cost rounded up plus one, journaled (assistant/routstr-peer-journal/) before sending; change, or the whole token after a refusal or no answer, is received back; a token the mint reports spent is final; unreachable-mint cases stay journaled and are retried before the next paid request. The X-Cashu fix (ngit 43774833…) is cherry-picked onto this branch.
• UI: Settings → System → "AI over the mesh" (sell on/off with this node's npub; buy-from-peer npub). The AI connection modal keeps routstr_peer.
• Two-node run, operator-authorized with a 100-sat cap, on branch build e77fbe02: seller archy-x250-pa3 (llama-server + Routstr installed from preloaded images; mint https://mint.minibits.cash/Bitcoin; archy-llama max_cost 6.44 sats), buyer archy-x250-beta (hostname bellevue).
  • Selling off: the buyer's catalog request reached pa3 over FIPS and got 404.
  • Selling on: catalog 200 over FIPS; one completion (28 prompt + 23 completion tokens) paid with an 8-sat token, 94 msat charged, 7 sats change returned and received. Buyer wallet 120 → 119 sats, budget 1/100 spent, journal empty. Seller logged change_returned=true and the caller's FIPS address.
• Found on the way, not fixed here:
  • pa3's archipelago-source image build has failed every reconcile pass since at least 2026-10-24; each pass holds the lifecycle lock, so installs fail with "An app lifecycle operation is active" unless sent between passes.
  • package.install requires dockerImage even for manifest-built apps.
  • Existing-only reconcile resolves env (and logs an error) for a manifest that is on disk but not installed, e.g. routstr before llama-server's key exists.
• Still open: streaming; catalog publication.

Seventh slice: assistant, seller discovery, earnings, pins (2026-10-25)

• Earnings card: the gateway records each answered sale (token minus change); routstr_selling_status adds what Routstr holds for payout. Live on pa3: 17 sats held, matching the buyer's logs at the time.
• The assistant's Routstr leg buys from the chosen peer too (no public fallback). Live on beta: one question, two paid turns via pa3, 13 sent and 8 returned each, journal empty; 217 s end to end on pa3's CPU. Qwen2.5 tool calls through llama-server b9860 work without extra flags.
• "Find sellers": routstr.find-sellers probes every federated peer's catalog over FIPS. Live on beta: 21 peers probed, pa3 found, 17 s.
• Base images pinned by multi-arch index digest (ubuntu 24.04 68ee2200…, uv python3.14-bookworm-slim 7cf77f59…). Rebuilding with the pins gives image IDs identical to the deployed candidates.

Eighth slice: lifecycle gate on a real node (2026-10-25)

• New suite tests/lifecycle/bats/routstr.bats.
• First run on archy-x250-pa3: stop and start failed with "An app lifecycle operation is active". The reconcile pass holds the lifecycle lock for the whole pass (~50 s with ~70 apps). This is the bug fixed by ngit PR 2c9782a7 (operator actions wait; the pass yields).
• With that PR applied locally on top of this branch (not pushed), on pa3:
  • single run: 9/9 read-only and destructive tests pass;
  • cascade tier (update; uninstall with preserve_data → reinstall): pass, Routstr's held ecash unchanged (data dir backed up first to /var/lib/archipelago/routstr.bak-gate-20261025);
  • run-gate.sh routstr, 5 iterations, destructive tier: 5/5 pass, 0 failures, 1980 s.
• This branch therefore depends on PR 2c9782a7 for the gate. Merge that first.
• pa3's archipelago-source build loop, which made reconcile passes longer and blocked installs, is fixed separately in ngit PR 84ecf252 (GitWorkshop dependency refresh).

Ninth slice: claiming change a reply never carried (2026-10-25)

Routstr saves the change it owes for an X-Cashu request as an out row tied to the paying token, and POST /v1/balance/refund with that token in X-Cashu returns the same saved change on every call (425 while the request is still running, 404 when nothing was saved, 410 once swept). Upstream routstr/balance.py has not changed since v0.5.0.

• Seller gateway: a third route, POST /routstr/v1/balance/refund, on the peer allowlist. It forwards only the X-Cashu token (no body, so no buyer-chosen payout address reaches Routstr), never waits for an inference slot, and stays open when selling is off, so change owed for earlier sales can still be claimed. FIPS peer listener only, as before.
• Buyer (assistant/routstr_peer.rs): when a paid request ends without change and taking the token back shows the seller redeemed it, the buyer claims the saved change over FIPS (10 s timeout, not single-delivery: a claim is idempotent). "Pending" (425, 429, 5xx, unreachable) keeps the journal entry and is retried before later paid requests, at most once a minute and for at most 24 hours. Recovered sats are credited to the budget as before.
• Old journal entries that stored an unreceivable token as its own "change" are read as plain tokens, so they are claimed too.
• Recovery skips journal entries younger than 300 s (longer than a paid request can take). Before, a second paid request (AIUI and the assistant at once) could take an in-flight request's token back before the seller redeemed it.
• This covers the paths where the seller saved the change. Which paths those are depends on the seller's Routstr version; see the tenth slice.

Tenth slice: Routstr main with X-Cashu failure refunds (2026-10-25)

The image now builds Routstr Core main at 8ce9ef69fc1b5ecd6f92f1baf54d8c2620ea0e89 (2026-10-24, v0.5.0 plus 87 commits; upstream CI green on Python 3.11/3.12/3.14). No release is tagged after v0.5.0. It includes #829 (refund an X-Cashu payment when forwarding or settlement fails) and #872 (refund at most once). App version 0.5.0-archy2, image localhost/archy-routstr:0.5.0-archy2; Routstr reports 0.5.0+g8ce9ef6. Interfaces we use are unchanged (admin login, settings, balances, custom models, /v1/info, /v1/models, /v1/balance/refund); one additive migration (models.image_pricing, nullable, with a downgrade).

Observed on the development host (fake-Lightning nutshell mint, 60-sat tokens, the new image started on a copy of the v0.5.0 database):

Case	v0.5.0	8ce9ef69
Upgrade from a v0.5.0 database	n/a	migrations ran; mints re-applied; operator price kept
Answered request	change in x-cashu	same; a claim returns the same change, every time
Upstream timeout (read timeout 2 s)	token kept, no change	424 with the refund in x-cashu and refund_token; claim returns it; all 59 sats Routstr received
Buyer hangs up 3 s in	request aborted before redemption; buyer receives its own token back	request finishes; change saved; claim answers 425, then 200 with the change
Seller Routstr restarts mid-request	not tested	token spent, no change saved; claim answers 425 indefinitely

(The mint charged 1 sat per swap, so a 60-sat token redeemed as 59.)

The last row is the remaining upstream gap: a request cancelled by a restart leaves a redeemed in row with no out row, so POST /v1/balance/refund says "pending" forever. The buyer drops such a claim after 24 hours. An upstream report is drafted (repro, cause, suggested fix: refund orphaned in rows at startup and in the sweeper, refund under asyncio.shield on cancellation, and a terminal status instead of an endless 425). Filed as Routstr/routstr-core#876.

Eleventh slice: both changes on the real nodes (2026-10-25)

Deployed to archy-x250-pa3 (seller) and archy-x250-beta (buyer): this branch plus ngit PR 2c9782a7 (the lifecycle lock fix the gate needs), local build only. pa3's Routstr data was backed up first to /var/lib/archipelago/routstr.bak-pre-main-20261025 (with a consistent SQLite copy, integrity ok).

• pa3: the orchestrator rewrote the Quadlet for the new image and applied it on package.restart. Routstr reports 0.5.0+g8ce9ef6; migrations ran; the node wallet's mints and payout address were re-applied; the operator price and the held earnings (75 sats) were kept.
• From beta over FIPS: POST /routstr/v1/balance/refund without a token is refused by the gateway (402); with an unknown token it reaches Routstr ("Refund not found", 404); /routstr/v1/wallet/refund stays 404.
• One paid AIUI request beta → pa3: paid 13 sats, 12 returned as change, the journal left empty; pa3 then held 76 sats.
• Lifecycle on pa3: cascade tier (package.update; uninstall with preserve_data → reinstall) pass, held earnings kept; run-gate.sh routstr, 5 iterations, destructive tier: 5/5 pass, 0 failures, 2009 s.