20 KiB
Routstr over FIPS with a local Llama provider
Status: candidate provider/runtime packaging and mesh request policy implemented;
no deployment or live acceptance yet.
Branch: feat/routstr-fips-llama, based on fcc18b3.
Goal
Node A selects Node B by identity, sees its model and pricing, authorizes a bounded spend, obtains inference over FIPS through B's Routstr server backed by llama-server, and records the correct payment outcome. Failures and restarts must not trigger duplicate payments, expose the raw runtime, or silently change transport/provider.
Initial runtime assumption: llama.cpp's llama-server serving a Llama model. Verify upstream compatibility before selecting runtime/server versions.
Source audit at kickoff
core/archipelago/src/api/handler/routstr_proxy.rs: AIUI's separate, session-authenticated client uses a fixed public HTTPS aggregator, model price estimates and Cashu payment/refund handling.core/archipelago/src/assistant/backends/routstr.rs: assistant adapter uses Nostr discovery and documents an originally unobserved provider wire contract. Do not treat those historical fixtures as current upstream compatibility proof.core/archipelago/src/fips/dial.rs: mesh identity resolution, restricted peer port 5679, explicit FIPS selection and single-delivery retry distinctions. Reuse these guarantees; do not introduce generic retries for paid POSTs.core/archipelago/src/api/rpc/package/config.rs: legacy Routstr configuration exposes port 8200 and copies node identity material into environment variables. This is not the proposed server packaging/security contract.- No
apps/routstr/manifest.ymlexisted at kickoff. A candidate manifest anddocker/routstr/Dockerfilenow implement the private server package foundation. Runtime lifecycle remains unqualified. Never hand-edit generated app ports. docs/external-access-and-websites.md: setup/funding UI and private Framework UAT results are recorded; real paid inference remains a separate acceptance gate.
Architecture and invariants
AIUI / assistant / website generation -> unified node Routstr client -> FIPS peer gateway -> Routstr server -> private llama-server.
- One provider selection, capability model and spending ledger across callers.
- Public HTTPS providers remain an explicit alternative. FIPS-selected requests fail closed when the mesh is unavailable; never silently use Tor or HTTPS.
- Restrict peer inference routes independently of owner dashboard sessions. Specify peer authentication, identity binding, authorization and replay protection before exposing a gateway. Do not expose wallet administration or the raw model server.
- Treat announcements, URLs, redirects, pricing and model capabilities as untrusted.
- Keep service signing scoped; do not pass the node master secret to the container.
- Separate wallet funding from permission to spend. Discovery and setup spend nothing.
- Persist allowance reservations, submissions, settlement and uncertain outcomes. Never automatically pay again to recover a potentially delivered request.
- Preserve app data, payment records, wallets and explicit uninstall decisions.
- FIPS inference does not imply mint settlement/discovery uses FIPS. Document each network dependency without overstating transport privacy.
Delivery checklist
1. Upstream contract verification (next)
- Pin candidate upstream source revisions: Routstr v0.5.0 and llama.cpp b9860. Image builds, base-image digest pins and runtime compatibility remain pending.
- Verify catalog, chat, tools, streaming, pricing and mint contracts against those revisions; record citations and reproducible disposable fixtures.
- Verify payment authentication, refunds, idempotency and status/recovery support. Explicitly document missing upstream guarantees and required extensions.
- Specify authenticated mesh gateway routes and identity-bound provider records.
- Define a disposable test environment with no production wallet or service access.
2. Local provider
- Add manifest-driven Routstr app with private upstream configuration, scoped secrets, persistent accounting, resource limits and truthful readiness.
- Connect an existing private llama-server; then add managed local runtime support.
- Require explicit model acquisition consent; record provenance, storage/memory needs, context limits and tested chat templates/capabilities.
- Demonstrate one non-streaming completion through the local provider.
3. FIPS vertical slice
- Support explicit provider identity entry without depending on public discovery.
- Add restricted, authenticated inference/catalog gateway and mesh-only selection.
- Demonstrate two-node inference and no fallback on mesh loss.
- Verify raw model server and administrative routes remain inaccessible to peers.
4. Shared client and payment correctness
- Delegate both current Routstr paths to a common provider/transport/payment service.
- Enforce capabilities, privacy screening and allowance limits across all callers.
- Implement durable request states: prepared, reserved, submitted, completed, settled, uncertain and reconciled; specify transitions before coding settlement.
- Test concurrent spends, duplicate submissions, response loss, cancellation, process restarts, mint failures and refund failures with isolated fixtures.
- Add validated discovery and configurable public providers without silent failover.
- Qualify streaming and tool calls after non-streaming payment correctness.
5. Operator experience and lifecycle
- Consumer: identity, model, transport, pricing, readiness, separate balance and allowance, uncertain transactions and recovery actions.
- Provider: runtime/model readiness, mesh publication opt-in, pricing, accepted mints, accounting and concurrency/resource limits.
- Test install/start/stop/restart/manager restart/upgrade/preserved uninstall/reinstall.
- Qualify embedded/standalone responsive and accessible setup/error states.
6. Acceptance and publication
- Record source tests separately from disposable integration and live acceptance.
- Run backend tests only through
scripts/test-backend-isolated.sh. - Obtain explicit authorization and a spending cap before real paid inference tests.
- Record two-node FIPS paid completion and fault/recovery evidence with no second payment.
- Review/contribute through ngit; mirror identical accepted commits to Gitea per
AGENTS.md. No publication is authorized by this implementation kickoff. - Read and retain all active release regression tasks before release preparation; this feature does not close unrelated paid-file or lifecycle incidents.
Evidence
Upstream source audit: Routstr Core v0.5.0 resolves to
4aa58bf380dd41f8e3ab7dc768b8e1d112d92019. The source was fetched read-only into
/tmp/archy-routstr-research/routstr-core; no upstream scripts were executed.
Pinned source references:
Important finding: X-Cashu per-request and bearer-funded-key flows are distinct. Current Archipelago bearer-token clients must not be presumed compatible with per-request change accounting. Refund/status and actual paid-inference behavior remain unverified. The source supports a generic OpenAI upstream; actual Llama model compatibility is not yet demonstrated.
Candidate package includes immutable source checkout, frozen dependency install,
loopback-only API, dedicated admin secret, read-only root and app-owned persistent
state. It defaults to no upstream/mint/publication identity. Upstream saved settings
can override defaults, and exchange-rate/background tasks still exist; this is not
a claim of zero external traffic. See apps/routstr/README.md for remaining gates.
No image build, container start, model download, live-node change, wallet spending, deployment or publication performed. Source-only validation results are recorded below; they are not lifecycle or protocol acceptance.
Source-only results:
- Manifest preflight: 15 passed, zero failed, one secret-like environment warning (the manifest contains an encryption-key file path, not a literal key).
python3 -m unittest discover -s tests/routstr-provider -p 'test_*.py' -v: twelve provider/runtime packaging-invariant tests passed.git diff --check: passed (later committed asbc08073).- Canonical Rust parsing: one isolated container test passed for both manifests
through
ARCHY_TEST_PACKAGE=archipelago-container scripts/test-backend-isolated.sh routstr_provider_candidates. - Image builds, catalog generation and runtime tests pending. Catalog surfaces were intentionally not regenerated for unqualified candidates.
Second implementation slice: runtime and FIPS policy
- Added
apps/llama-serveranddocker/llama-server, pinned to llama.cpp b9860 (fdb1db877c526ec90f668eca1b858da5dba85560). Reviewed its server documentation, CPU Dockerfile and CMake options at that exact revision. CPU-only source build, no published host ports, dedicated API key, read-only operator-provided GGUF, stablearchy-llamaalias and explicit initial resource limits. - Llama manifest preflight: 16 passed, zero failures/warnings. Missing model import UX and prerequisite checks intentionally block catalog readiness.
- Added
assistant/routstr_transport.rs: canonical npub validation, exact typed catalog/completion routes, explicit FIPS-only requests, no fallback host, and single-delivery policy for completion requests. Four source tests cover address, route and delivery-policy construction. This is a request-policy foundation, not an enabled client or server gateway. It attaches no credentials; peer identity binding, authorization and durable settlement remain required. - Full backend compilation through the isolated runner is blocked by pre-existing
monitoring/store.rs:105use ofAtomicU64::try_update, unstable on the installed Rust 1.94.1 toolchain (E0658). The new FIPS unit tests did not execute. No unrestricted host test run or unrelated source workaround was used. - This environment has no
podmanexecutable; no image build or container lifecycle claim is made. No model downloads, node changes, paid requests or publication.
Next: compile/run the FIPS tests with the supported backend toolchain; build the candidate images in a disposable rootless runtime; then implement the authenticated mesh gateway and durable request journal before connecting paid callers. Configure and qualify the llama-server upstream/key handoff without exposing the raw runtime.
Third slice: toolchain, images and local paid inference (2026-10-25)
Supersedes the second slice's blockers. Disposable rootless Podman 5.4.2 on the development host (not a node); fake-Lightning test mint; no real funds.
Toolchain:
088eee55deliberately movedmonitoring/store.rstoAtomicU64::try_update; the distro Rust 1.94.1 is simply too old. The installed rustup stable 1.96.0 compiles it unchanged. The fourroutstr_transporttests passed throughscripts/test-backend-isolated.shwith that toolchain first onPATHand a dedicatedCARGO_TARGET_DIR. No monitoring code was patched.
Image defects found by running them, both fixed with regression tests:
- llama-server exited 132 (SIGILL) on a CPU without AVX:
GGML_NATIVE=OFFstill bakes in AVX/AVX2. Now built like upstream'scpu.Dockerfile(GGML_BACKEND_DL=ON,GGML_CPU_ALL_VARIANTS=ON): sse42 through zen4 backends, best one loaded at start. - Routstr failed every startup:
alembic.iniuses a cwd-relativescript_location, and the image's cwd is/data. Now%(here)s/migrations.
Local stack (manifest security settings: read-only root, --cap-drop=all,
no-new-privileges, tmpfs /tmp, model mounted read-only):
- Model: TinyLlama-1.1B-Chat-v1.0 Q4_K_M, Apache-2.0, Llama architecture,
TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUFrevision52e7645ba7c309695bec7ac98f4f005b139cf465, 668788096 bytes, sha2569fecc3b3cd76bba89d504f29b616eedf7da85b96540e490ca5824d3f7d2776a0(verified). - llama-server: model loaded <1s; chat completions return 401 without or with a
wrong key and 200 with usage accounting with the dedicated key.
/healthand/v1/modelsare public by upstream design (alias only). - Routstr with
UPSTREAM_BASE_URL=http://llama-server:8080/v1and the llama key asUPSTREAM_API_KEY: admin API redacts the key. - Pricing is mandatory. Routstr fails closed: an unpriced upstream model is
imported disabled and
/v1/modelsis empty.MODELS_PATHis unread at this revision; pricing goes throughPOST /admin/api/upstream-providers/{id}/models(USD per token, converted to sats via Kraken/Coinbase/Binance price feeds). Install therefore needs a post-start pricing step before the provider is usable. - Test mint:
docker.io/cashubtc/nutshell:0.20.0(sha256:fae7b126ecc2ca797f3f572e5e86b3623bda26fdee0f97489ea9e3d09089950b), FakeWallet backend. 0.17.0 is incompatible with Routstr's bundled cashu 0.20.0 wallet (KeysResponse.keysets.activemissing). Tokens must name the exact URL inCASHU_MINTS; others are rejected asuntrusted_mint.
Payment contract, observed against Routstr v0.5.0 (20-sat token each):
X-Cashu: <token>: 200, charged 70 msat, change returned immediately as a new token in thex-cashuresponse header (redeemed: +17 sat). On a 503 the unspent token is also returned inx-cashu.Authorization: Bearer <token>: 200, charged 53 msat, no change header; the token becomes a server-sidesk-…account holding the remainder (18947 msat).POST /v1/balance/refundreturns it as a token; a repeated refund call replays the same (already-spent) token rather than paying twice.- Consequence:
api/handler/routstr_proxy.rssends Bearer but harvests change fromx-cashu/x-cashu-refund. Against a v0.5.0 server that change stays on the provider's account and is never refunded. The public aggregator AIUI uses may run a different version; this is proven only for routstr-core v0.5.0. The FIPS buyer must useX-Cashu(per-request, change in-band) or an explicit Bearer account with journaled refund.
Still pending: manifest upstream/key wiring and post-install pricing, private stack network isolation, lifecycle qualification, the FIPS gateway, the buyer journal, and two-node acceptance.
Fourth slice: node wallet integration and default pricing (2026-10-25)
Operator decisions: the seller trusts the node ecash wallet's mints (Minibits by default) and pays earnings to the wallet's Minibits Lightning address; the model ships priced, and mesh selling stays off until the operator opts in.
- An empty
CASHU_MINTSwas shown live to make Routstr trusthttps://mint.minibits.cash/Bitcoinsilently (_compute_primary_mint). - Routstr persists settings on first start and its database then wins over env
(shown live: a changed
CASHU_MINTSwas ignored on restart). - New
derived_envplaceholders{{ECASH_MINTS}}(accepted mints; empty list refuses the start) and{{ECASH_LN_ADDRESS}}(empty with a warning when unavailable). Orchestrator and container tests pass in the isolated runner. docker/routstr/archy-start.pyis PID 1: starts uvicorn, re-applies mints and payout address viaPATCH /admin/api/settingson every start, pricesarchy-llamaonce at $1/$2 per million tokens unless already priced, and maps an orchestrator stop to exit 0. Verified live: fresh install priced and served; changed wallet mints/address applied on restart; operator price kept; paid X-Cashu completion at the operator price; stop exit 0, SIGKILL 137.- The buyer-side Bearer/X-Cashu defect is fixed separately on
fix/routstr-xcashu-change, ngit proposal437748339cad1c5d70d8f399dac084244be06b01844f121823552659c2dd450f.
Next: authenticated FIPS gateway with serving opt-in (default off).
Fifth slice: seller FIPS gateway (2026-10-25)
api/handler/routstr_gateway.rsserves exactlyGET /routstr/v1/modelsandPOST /routstr/v1/chat/completions(RoutstrOperation::from_request), forwarded to the Routstr app on127.0.0.1:8200. Both are onis_peer_allowed_path; every other/routstr/...path stays 404.- Refused with 404 unless
settings/routstr-selling.jsonhasenabled: true(system.settingskeyroutstr_selling; missing or corrupt reads as off; not in the assistant's settable keys) and the request arrived on the FIPS peer listener (newFipsPeerListenerrequest extension) from a ULA source. The path allowlist also admits these paths on the main listener, where the handler refuses them. - Payment is the authentication: completions need an
X-Cashutoken (402 otherwise). OnlyX-CashuandContent-Typeare forwarded (noAuthorization, cookies, forwarded-for or query strings). Routstr'sx-cashuchange header is passed back, also on refusals.stream: trueis refused (change arrives in a header). 256 KiB request / 4 MiB response caps, two concurrent requests (503 beyond), no retries. The caller's FIPS address is logged; tokens never are. - Tests: six gateway tests against a fake Routstr, the settings test and the
peer path filter, in the isolated runner (39
routstr/filter tests pass). - Not yet shown: a request from a second node over a real FIPS link.
Next: buyer side (RoutstrFipsTarget, X-Cashu, durable request journal), an
operator toggle in the UI, then two-node acceptance with a spending cap.
Sixth slice: buyer, settings UI and first two-node run (2026-10-25)
- Buyer:
assistant/routstr_peer.rs. Withroutstr_peerset in the AI provider settings, AIUI's Routstr catalog and completions go to that peer over FIPS only. The token is the seller'ssats_pricing.max_costrounded up plus one, journaled (assistant/routstr-peer-journal/) before sending; change, or the whole token after a refusal or no answer, is received back; a token the mint reports spent is final; unreachable-mint cases stay journaled and are retried before the next paid request. The X-Cashu fix (ngit43774833…) is cherry-picked onto this branch. - UI: Settings → System → "AI over the mesh" (sell on/off with this node's
npub; buy-from-peer npub). The AI connection modal keeps
routstr_peer. - Two-node run, operator-authorized with a 100-sat cap, on branch build
e77fbe02: seller archy-x250-pa3 (llama-server + Routstr installed from preloaded images; minthttps://mint.minibits.cash/Bitcoin;archy-llamamax_cost6.44 sats), buyer archy-x250-beta (hostname bellevue).- Selling off: the buyer's catalog request reached pa3 over FIPS and got 404.
- Selling on: catalog 200 over FIPS; one completion (28 prompt + 23
completion tokens) paid with an 8-sat token, 94 msat charged, 7 sats change
returned and received. Buyer wallet 120 → 119 sats, budget 1/100 spent,
journal empty. Seller logged
change_returned=trueand the caller's FIPS address.
- Found on the way, not fixed here:
- pa3's
archipelago-sourceimage build has failed every reconcile pass since at least 2026-10-24; each pass holds the lifecycle lock, so installs fail with "An app lifecycle operation is active" unless sent between passes. package.installrequiresdockerImageeven for manifest-built apps.- Existing-only reconcile resolves env (and logs an error) for a manifest that is on disk but not installed, e.g. routstr before llama-server's key exists.
- pa3's
- Still open: the assistant's own Routstr fallback still uses the public aggregator when a peer is set; streaming; choosing a peer from federation instead of pasting an npub; lifecycle gate; base-image digest pins.