archy/apps/llama-server/README.md

Private llama-server — development candidate

CPU-only llama.cpp b9860, pinned to source revision fdb1db877c526ec90f668eca1b858da5dba85560. The build uses the upstream-documented GCC 14 toolchain with host-specific optimization disabled. No model is bundled; no Hugging Face model or Web UI asset download is requested. The image and lifecycle have been run on real nodes (archy-x250-pa3); the base image is pinned by multi-arch index digest.

Connection boundary

• No published host ports, no browser interface, no automatic FIPS exposure.
• Container DNS alias llama-server on archy-net; the candidate Routstr server can be configured with generic upstream base URL http://llama-server:8080/v1.
• A dedicated generated llama-server-api-key is injected through Podman secrets as LLAMA_API_KEY, an upstream-supported authentication option.
• The provider configuration must receive that key through a private trusted configuration path; it must never pass through AIUI, a model prompt or logs. Automatic key handoff/configuration RPC is not implemented yet.
• archy-net is shared. API authentication is required, but this is not dedicated network isolation. A private stack network remains an acceptance gate.
• The raw model server is not a payment boundary. Only the Routstr gateway should grant remote paid inference after peer authorization and accounting are complete.

Model prerequisites

The operator must explicitly import a compatible GGUF as /var/lib/archipelago/llama-server/models/model.gguf and verify its provenance, license and checksum. The model mount is read-only. Nothing downloads a model merely because the app is installed. Without a model it will fail to start; the model-import/readiness UI and install prerequisite checks are still pending, so this candidate is not ready for catalog inclusion.

Tested model: Qwen2.5-1.5B-Instruct q4_k_m (Apache-2.0) from Qwen/Qwen2.5-1.5B-Instruct-GGUF revision 91cad51170dc346986eccefdc2dd33a9da36ead9, sha256 6a1a2eb6d15622bf3c96857206351ba97e1af16c30d7a74ee38970e434e9407e. TinyLlama 1.1B was dropped: it is trained on 2048 tokens, llama-server silently caps the context to that, and AIUI's instructions alone are about 2.4k tokens.

Initial limits: 8 GiB container memory, threads defaulting to the physical core count, 8192-token context, one slot and a 1024-token generation setting. These are an initial small-model test profile, not a promise that an arbitrary Llama model fits. Caller token bounds, admission control and actual limit enforcement must be tested separately; do not treat command-line defaults as the paid request budget ceiling.

The stable model alias is archy-llama. Test a non-streaming completion and exact usage reporting before enabling it in Routstr. Tool support, streaming and GPU variants remain unqualified. /health is the runtime's model-loading health endpoint; it does not prove billing correctness.

Source references

All references are to the pinned upstream revision:

• server API and options
• upstream CPU build
• build options

Offline validation from the repository root:

bash scripts/validate-app-manifest.sh apps/llama-server/manifest.yml
python3 -m unittest discover -s tests/routstr-provider -p 'test_*.py' -v