Files
archy/docs/routstr-fips-llama-integration.md

20 KiB

Routstr over FIPS with a local Llama provider

Status: candidate provider/runtime packaging and mesh request policy implemented; no deployment or live acceptance yet. Branch: feat/routstr-fips-llama, based on fcc18b3.

Goal

Node A selects Node B by identity, sees its model and pricing, authorizes a bounded spend, obtains inference over FIPS through B's Routstr server backed by llama-server, and records the correct payment outcome. Failures and restarts must not trigger duplicate payments, expose the raw runtime, or silently change transport/provider.

Initial runtime assumption: llama.cpp's llama-server serving a Llama model. Verify upstream compatibility before selecting runtime/server versions.

Source audit at kickoff

  • core/archipelago/src/api/handler/routstr_proxy.rs: AIUI's separate, session-authenticated client uses a fixed public HTTPS aggregator, model price estimates and Cashu payment/refund handling.
  • core/archipelago/src/assistant/backends/routstr.rs: assistant adapter uses Nostr discovery and documents an originally unobserved provider wire contract. Do not treat those historical fixtures as current upstream compatibility proof.
  • core/archipelago/src/fips/dial.rs: mesh identity resolution, restricted peer port 5679, explicit FIPS selection and single-delivery retry distinctions. Reuse these guarantees; do not introduce generic retries for paid POSTs.
  • core/archipelago/src/api/rpc/package/config.rs: legacy Routstr configuration exposes port 8200 and copies node identity material into environment variables. This is not the proposed server packaging/security contract.
  • No apps/routstr/manifest.yml existed at kickoff. A candidate manifest and docker/routstr/Dockerfile now implement the private server package foundation. Runtime lifecycle remains unqualified. Never hand-edit generated app ports.
  • docs/external-access-and-websites.md: setup/funding UI and private Framework UAT results are recorded; real paid inference remains a separate acceptance gate.

Architecture and invariants

AIUI / assistant / website generation -> unified node Routstr client -> FIPS peer gateway -> Routstr server -> private llama-server.

  • One provider selection, capability model and spending ledger across callers.
  • Public HTTPS providers remain an explicit alternative. FIPS-selected requests fail closed when the mesh is unavailable; never silently use Tor or HTTPS.
  • Restrict peer inference routes independently of owner dashboard sessions. Specify peer authentication, identity binding, authorization and replay protection before exposing a gateway. Do not expose wallet administration or the raw model server.
  • Treat announcements, URLs, redirects, pricing and model capabilities as untrusted.
  • Keep service signing scoped; do not pass the node master secret to the container.
  • Separate wallet funding from permission to spend. Discovery and setup spend nothing.
  • Persist allowance reservations, submissions, settlement and uncertain outcomes. Never automatically pay again to recover a potentially delivered request.
  • Preserve app data, payment records, wallets and explicit uninstall decisions.
  • FIPS inference does not imply mint settlement/discovery uses FIPS. Document each network dependency without overstating transport privacy.

Delivery checklist

1. Upstream contract verification (next)

  • Pin candidate upstream source revisions: Routstr v0.5.0 and llama.cpp b9860. Image builds, base-image digest pins and runtime compatibility remain pending.
  • Verify catalog, chat, tools, streaming, pricing and mint contracts against those revisions; record citations and reproducible disposable fixtures.
  • Verify payment authentication, refunds, idempotency and status/recovery support. Explicitly document missing upstream guarantees and required extensions.
  • Specify authenticated mesh gateway routes and identity-bound provider records.
  • Define a disposable test environment with no production wallet or service access.

2. Local provider

  • Add manifest-driven Routstr app with private upstream configuration, scoped secrets, persistent accounting, resource limits and truthful readiness.
  • Connect an existing private llama-server; then add managed local runtime support.
  • Require explicit model acquisition consent; record provenance, storage/memory needs, context limits and tested chat templates/capabilities.
  • Demonstrate one non-streaming completion through the local provider.

3. FIPS vertical slice

  • Support explicit provider identity entry without depending on public discovery.
  • Add restricted, authenticated inference/catalog gateway and mesh-only selection.
  • Demonstrate two-node inference and no fallback on mesh loss.
  • Verify raw model server and administrative routes remain inaccessible to peers.

4. Shared client and payment correctness

  • Delegate both current Routstr paths to a common provider/transport/payment service.
  • Enforce capabilities, privacy screening and allowance limits across all callers.
  • Implement durable request states: prepared, reserved, submitted, completed, settled, uncertain and reconciled; specify transitions before coding settlement.
  • Test concurrent spends, duplicate submissions, response loss, cancellation, process restarts, mint failures and refund failures with isolated fixtures.
  • Add validated discovery and configurable public providers without silent failover.
  • Qualify streaming and tool calls after non-streaming payment correctness.

5. Operator experience and lifecycle

  • Consumer: identity, model, transport, pricing, readiness, separate balance and allowance, uncertain transactions and recovery actions.
  • Provider: runtime/model readiness, mesh publication opt-in, pricing, accepted mints, accounting and concurrency/resource limits.
  • Test install/start/stop/restart/manager restart/upgrade/preserved uninstall/reinstall.
  • Qualify embedded/standalone responsive and accessible setup/error states.

6. Acceptance and publication

  • Record source tests separately from disposable integration and live acceptance.
  • Run backend tests only through scripts/test-backend-isolated.sh.
  • Obtain explicit authorization and a spending cap before real paid inference tests.
  • Record two-node FIPS paid completion and fault/recovery evidence with no second payment.
  • Review/contribute through ngit; mirror identical accepted commits to Gitea per AGENTS.md. No publication is authorized by this implementation kickoff.
  • Read and retain all active release regression tasks before release preparation; this feature does not close unrelated paid-file or lifecycle incidents.

Evidence

Upstream source audit: Routstr Core v0.5.0 resolves to 4aa58bf380dd41f8e3ab7dc768b8e1d112d92019. The source was fetched read-only into /tmp/archy-routstr-research/routstr-core; no upstream scripts were executed.

Pinned source references:

Important finding: X-Cashu per-request and bearer-funded-key flows are distinct. Current Archipelago bearer-token clients must not be presumed compatible with per-request change accounting. Refund/status and actual paid-inference behavior remain unverified. The source supports a generic OpenAI upstream; actual Llama model compatibility is not yet demonstrated.

Candidate package includes immutable source checkout, frozen dependency install, loopback-only API, dedicated admin secret, read-only root and app-owned persistent state. It defaults to no upstream/mint/publication identity. Upstream saved settings can override defaults, and exchange-rate/background tasks still exist; this is not a claim of zero external traffic. See apps/routstr/README.md for remaining gates.

No image build, container start, model download, live-node change, wallet spending, deployment or publication performed. Source-only validation results are recorded below; they are not lifecycle or protocol acceptance.

Source-only results:

  • Manifest preflight: 15 passed, zero failed, one secret-like environment warning (the manifest contains an encryption-key file path, not a literal key).
  • python3 -m unittest discover -s tests/routstr-provider -p 'test_*.py' -v: twelve provider/runtime packaging-invariant tests passed.
  • git diff --check: passed (later committed as bc08073).
  • Canonical Rust parsing: one isolated container test passed for both manifests through ARCHY_TEST_PACKAGE=archipelago-container scripts/test-backend-isolated.sh routstr_provider_candidates.
  • Image builds, catalog generation and runtime tests pending. Catalog surfaces were intentionally not regenerated for unqualified candidates.

Second implementation slice: runtime and FIPS policy

  • Added apps/llama-server and docker/llama-server, pinned to llama.cpp b9860 (fdb1db877c526ec90f668eca1b858da5dba85560). Reviewed its server documentation, CPU Dockerfile and CMake options at that exact revision. CPU-only source build, no published host ports, dedicated API key, read-only operator-provided GGUF, stable archy-llama alias and explicit initial resource limits.
  • Llama manifest preflight: 16 passed, zero failures/warnings. Missing model import UX and prerequisite checks intentionally block catalog readiness.
  • Added assistant/routstr_transport.rs: canonical npub validation, exact typed catalog/completion routes, explicit FIPS-only requests, no fallback host, and single-delivery policy for completion requests. Four source tests cover address, route and delivery-policy construction. This is a request-policy foundation, not an enabled client or server gateway. It attaches no credentials; peer identity binding, authorization and durable settlement remain required.
  • Full backend compilation through the isolated runner is blocked by pre-existing monitoring/store.rs:105 use of AtomicU64::try_update, unstable on the installed Rust 1.94.1 toolchain (E0658). The new FIPS unit tests did not execute. No unrestricted host test run or unrelated source workaround was used.
  • This environment has no podman executable; no image build or container lifecycle claim is made. No model downloads, node changes, paid requests or publication.

Next: compile/run the FIPS tests with the supported backend toolchain; build the candidate images in a disposable rootless runtime; then implement the authenticated mesh gateway and durable request journal before connecting paid callers. Configure and qualify the llama-server upstream/key handoff without exposing the raw runtime.

Third slice: toolchain, images and local paid inference (2026-10-25)

Supersedes the second slice's blockers. Disposable rootless Podman 5.4.2 on the development host (not a node); fake-Lightning test mint; no real funds.

Toolchain:

  • 088eee55 deliberately moved monitoring/store.rs to AtomicU64::try_update; the distro Rust 1.94.1 is simply too old. The installed rustup stable 1.96.0 compiles it unchanged. The four routstr_transport tests passed through scripts/test-backend-isolated.sh with that toolchain first on PATH and a dedicated CARGO_TARGET_DIR. No monitoring code was patched.

Image defects found by running them, both fixed with regression tests:

  • llama-server exited 132 (SIGILL) on a CPU without AVX: GGML_NATIVE=OFF still bakes in AVX/AVX2. Now built like upstream's cpu.Dockerfile (GGML_BACKEND_DL=ON, GGML_CPU_ALL_VARIANTS=ON): sse42 through zen4 backends, best one loaded at start.
  • Routstr failed every startup: alembic.ini uses a cwd-relative script_location, and the image's cwd is /data. Now %(here)s/migrations.

Local stack (manifest security settings: read-only root, --cap-drop=all, no-new-privileges, tmpfs /tmp, model mounted read-only):

  • Model: TinyLlama-1.1B-Chat-v1.0 Q4_K_M, Apache-2.0, Llama architecture, TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF revision 52e7645ba7c309695bec7ac98f4f005b139cf465, 668788096 bytes, sha256 9fecc3b3cd76bba89d504f29b616eedf7da85b96540e490ca5824d3f7d2776a0 (verified).
  • llama-server: model loaded <1s; chat completions return 401 without or with a wrong key and 200 with usage accounting with the dedicated key. /health and /v1/models are public by upstream design (alias only).
  • Routstr with UPSTREAM_BASE_URL=http://llama-server:8080/v1 and the llama key as UPSTREAM_API_KEY: admin API redacts the key.
  • Pricing is mandatory. Routstr fails closed: an unpriced upstream model is imported disabled and /v1/models is empty. MODELS_PATH is unread at this revision; pricing goes through POST /admin/api/upstream-providers/{id}/models (USD per token, converted to sats via Kraken/Coinbase/Binance price feeds). Install therefore needs a post-start pricing step before the provider is usable.
  • Test mint: docker.io/cashubtc/nutshell:0.20.0 (sha256:fae7b126ecc2ca797f3f572e5e86b3623bda26fdee0f97489ea9e3d09089950b), FakeWallet backend. 0.17.0 is incompatible with Routstr's bundled cashu 0.20.0 wallet (KeysResponse.keysets.active missing). Tokens must name the exact URL in CASHU_MINTS; others are rejected as untrusted_mint.

Payment contract, observed against Routstr v0.5.0 (20-sat token each):

  • X-Cashu: <token>: 200, charged 70 msat, change returned immediately as a new token in the x-cashu response header (redeemed: +17 sat). On a 503 the unspent token is also returned in x-cashu.
  • Authorization: Bearer <token>: 200, charged 53 msat, no change header; the token becomes a server-side sk-… account holding the remainder (18947 msat). POST /v1/balance/refund returns it as a token; a repeated refund call replays the same (already-spent) token rather than paying twice.
  • Consequence: api/handler/routstr_proxy.rs sends Bearer but harvests change from x-cashu/x-cashu-refund. Against a v0.5.0 server that change stays on the provider's account and is never refunded. The public aggregator AIUI uses may run a different version; this is proven only for routstr-core v0.5.0. The FIPS buyer must use X-Cashu (per-request, change in-band) or an explicit Bearer account with journaled refund.

Still pending: manifest upstream/key wiring and post-install pricing, private stack network isolation, lifecycle qualification, the FIPS gateway, the buyer journal, and two-node acceptance.

Fourth slice: node wallet integration and default pricing (2026-10-25)

Operator decisions: the seller trusts the node ecash wallet's mints (Minibits by default) and pays earnings to the wallet's Minibits Lightning address; the model ships priced, and mesh selling stays off until the operator opts in.

  • An empty CASHU_MINTS was shown live to make Routstr trust https://mint.minibits.cash/Bitcoin silently (_compute_primary_mint).
  • Routstr persists settings on first start and its database then wins over env (shown live: a changed CASHU_MINTS was ignored on restart).
  • New derived_env placeholders {{ECASH_MINTS}} (accepted mints; empty list refuses the start) and {{ECASH_LN_ADDRESS}} (empty with a warning when unavailable). Orchestrator and container tests pass in the isolated runner.
  • docker/routstr/archy-start.py is PID 1: starts uvicorn, re-applies mints and payout address via PATCH /admin/api/settings on every start, prices archy-llama once at $1/$2 per million tokens unless already priced, and maps an orchestrator stop to exit 0. Verified live: fresh install priced and served; changed wallet mints/address applied on restart; operator price kept; paid X-Cashu completion at the operator price; stop exit 0, SIGKILL 137.
  • The buyer-side Bearer/X-Cashu defect is fixed separately on fix/routstr-xcashu-change, ngit proposal 437748339cad1c5d70d8f399dac084244be06b01844f121823552659c2dd450f.

Next: authenticated FIPS gateway with serving opt-in (default off).

Fifth slice: seller FIPS gateway (2026-10-25)

  • api/handler/routstr_gateway.rs serves exactly GET /routstr/v1/models and POST /routstr/v1/chat/completions (RoutstrOperation::from_request), forwarded to the Routstr app on 127.0.0.1:8200. Both are on is_peer_allowed_path; every other /routstr/... path stays 404.
  • Refused with 404 unless settings/routstr-selling.json has enabled: true (system.settings key routstr_selling; missing or corrupt reads as off; not in the assistant's settable keys) and the request arrived on the FIPS peer listener (new FipsPeerListener request extension) from a ULA source. The path allowlist also admits these paths on the main listener, where the handler refuses them.
  • Payment is the authentication: completions need an X-Cashu token (402 otherwise). Only X-Cashu and Content-Type are forwarded (no Authorization, cookies, forwarded-for or query strings). Routstr's x-cashu change header is passed back, also on refusals. stream: true is refused (change arrives in a header). 256 KiB request / 4 MiB response caps, two concurrent requests (503 beyond), no retries. The caller's FIPS address is logged; tokens never are.
  • Tests: six gateway tests against a fake Routstr, the settings test and the peer path filter, in the isolated runner (39 routstr/filter tests pass).
  • Not yet shown: a request from a second node over a real FIPS link.

Next: buyer side (RoutstrFipsTarget, X-Cashu, durable request journal), an operator toggle in the UI, then two-node acceptance with a spending cap.

Sixth slice: buyer, settings UI and first two-node run (2026-10-25)

  • Buyer: assistant/routstr_peer.rs. With routstr_peer set in the AI provider settings, AIUI's Routstr catalog and completions go to that peer over FIPS only. The token is the seller's sats_pricing.max_cost rounded up plus one, journaled (assistant/routstr-peer-journal/) before sending; change, or the whole token after a refusal or no answer, is received back; a token the mint reports spent is final; unreachable-mint cases stay journaled and are retried before the next paid request. The X-Cashu fix (ngit 43774833…) is cherry-picked onto this branch.
  • UI: Settings → System → "AI over the mesh" (sell on/off with this node's npub; buy-from-peer npub). The AI connection modal keeps routstr_peer.
  • Two-node run, operator-authorized with a 100-sat cap, on branch build e77fbe02: seller archy-x250-pa3 (llama-server + Routstr installed from preloaded images; mint https://mint.minibits.cash/Bitcoin; archy-llama max_cost 6.44 sats), buyer archy-x250-beta (hostname bellevue).
    • Selling off: the buyer's catalog request reached pa3 over FIPS and got 404.
    • Selling on: catalog 200 over FIPS; one completion (28 prompt + 23 completion tokens) paid with an 8-sat token, 94 msat charged, 7 sats change returned and received. Buyer wallet 120 → 119 sats, budget 1/100 spent, journal empty. Seller logged change_returned=true and the caller's FIPS address.
  • Found on the way, not fixed here:
    • pa3's archipelago-source image build has failed every reconcile pass since at least 2026-10-24; each pass holds the lifecycle lock, so installs fail with "An app lifecycle operation is active" unless sent between passes.
    • package.install requires dockerImage even for manifest-built apps.
    • Existing-only reconcile resolves env (and logs an error) for a manifest that is on disk but not installed, e.g. routstr before llama-server's key exists.
  • Still open: the assistant's own Routstr fallback still uses the public aggregator when a peer is set; streaming; choosing a peer from federation instead of pasting an npub; lifecycle gate; base-image digest pins.