docs: did:dht integration architecture and DWN protocol schemas

- DHT-01: docs/did-dht-integration.md — did:dht spec analysis, DNS packet encoding, mainline crate, publication/resolution flows, security notes - SCHEMA-01: docs/dwn-protocols.md — 4 DWN protocol definitions with JSON schemas: node-identity, file-catalog, federation, app-deploy Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 02:59:16 +00:00 · 2026-03-14 02:59:16 +00:00 · 9afe93a5b4
commit 9afe93a5b4
parent 952b7d1c92
3 changed files with 562 additions and 2 deletions
--- a/docs/did-dht-integration.md
+++ b/docs/did-dht-integration.md
@ -0,0 +1,160 @@
+# did:dht Integration Architecture
+
+## Overview
+
+Archipelago currently uses `did:key` for node identities. This document describes integrating `did:dht` as a **complementary** DID method that makes node identities discoverable via the BitTorrent Mainline DHT, without relying on centralized registries, Nostr relays, or Tor hidden services.
+
+**Goal**: Each Archipelago node has two DID types:
+- `did:key` — Offline, self-certifying, works without network (primary identity)
+- `did:dht` — Published to Mainline DHT for decentralized discovery (optional, discoverable)
+
+## What is did:dht?
+
+The `did:dht` method stores DID Documents in the [BitTorrent Mainline DHT](https://www.bittorrent.org/beps/bep_0044.html) using BEP-44 (mutable items). Key properties:
+
+- **No server needed**: Uses the public DHT network (~15M+ nodes)
+- **Ed25519 keypair**: Same key type Archipelago already uses
+- **DNS-encoded**: DID Document stored as a DNS packet (compact, standardized)
+- **Mutable**: Documents can be updated by the key holder
+- **TTL-based**: Published records have a TTL and must be refreshed periodically (every 2 hours recommended)
+- **Identifier format**: `did:dht:{z-base-32-encoded-ed25519-pubkey}`
+
+## Architecture
+
+### DID Relationship
+
+```
+Node Identity (Ed25519 keypair)
+  ├── did:key:z6Mk...     (derived from pubkey, offline, stable)
+  └── did:dht:z6Mk...     (published to DHT, discoverable, same key)
+```
+
+Both DIDs use the same underlying Ed25519 keypair. The `did:dht` identifier is the z-base-32 encoding of the 32-byte public key. This means the same keypair produces both DID types — no additional key management.
+
+### DHT Publication Flow
+
+```
+1. Node generates Ed25519 keypair (already exists)
+2. Node creates DID Document with:
+   - Ed25519 verification key (signing)
+   - X25519 key agreement key (derived)
+   - Service endpoints (optional: Tor onion, federation endpoint)
+3. DID Document encoded as DNS packet (RFC 1035)
+4. DNS packet signed with Ed25519 key (BEP-44 mutable item)
+5. Published to Mainline DHT under the public key
+6. Refreshed every 2 hours to maintain availability
+```
+
+### DNS Packet Encoding
+
+The DID Document maps to DNS resource records:
+
+| Record Type | Name | Purpose |
+|------------|------|---------|
+| TXT `_did.` | `vm=k0` | Verification method: key 0 (Ed25519) |
+| TXT `_did.` | `auth=0` | Authentication uses key 0 |
+| TXT `_did.` | `asm=0` | AssertionMethod uses key 0 |
+| TXT `_k0._did.` | `id=0;t=0;k={base64url_pubkey}` | Key 0: Ed25519 public key |
+| TXT `_s0._did.` | `id=tor;t=TorHiddenService;se={onion}` | Service endpoint (optional) |
+
+### Resolution Flow
+
+```
+1. Receive did:dht:{identifier}
+2. Decode z-base-32 → 32-byte Ed25519 public key
+3. Query Mainline DHT for BEP-44 mutable item under that key
+4. Verify signature on the DHT payload
+5. Parse DNS packet → reconstruct DID Document
+6. Cache for 1 hour (reduce DHT load)
+```
+
+## Implementation Plan
+
+### Rust Crate: `mainline`
+
+Use the `mainline` crate for Mainline DHT access. It provides:
+- `Dht::client()` for resolution-only nodes
+- `Dht::server()` for full DHT participation
+- `MutableItem` for BEP-44 put/get operations
+- Ed25519 signing compatible with `ed25519-dalek`
+
+Additional crates needed:
+- `simple-dns` or `trust-dns-proto` for DNS packet encoding/decoding
+- `zbase32` for z-base-32 encoding (did:dht identifier format)
+
+### New Files
+
+```
+core/archipelago/src/identity/
+├── did_dht.rs          — did:dht creation, publication, resolution
+└── dns_packet.rs       — DID Document ↔ DNS packet encoding
+```
+
+### New RPC Endpoints
+
+| Endpoint | Description |
+|----------|-------------|
+| `identity.create-dht-did` | Publish current node's DID to DHT |
+| `identity.resolve-dht-did` | Resolve a did:dht from the DHT |
+| `identity.refresh-dht-did` | Force refresh the DHT publication |
+| `identity.dht-status` | Check if node's did:dht is published and resolvable |
+
+### Integration Points
+
+1. **Server startup**: Optionally publish did:dht in background (non-blocking)
+2. **Identity manager**: Store did:dht alongside did:key in identity records
+3. **Federation**: Accept did:dht in peer join/discovery
+4. **Web5 UI**: Display both DID types, add publish/resolve buttons
+5. **Credentials**: Accept did:dht as issuer/subject in VCs
+
+### Background Refresh
+
+A background tokio task refreshes the DHT publication every 2 hours:
+
+```
+spawn background task:
+  loop {
+    publish_to_dht(keypair, did_document_dns_packet)
+    sleep(2 hours)
+  }
+```
+
+If the node is offline when the TTL expires, the record drops from the DHT. It gets re-published when the node comes back online.
+
+## Security Considerations
+
+1. **Same key for both DIDs**: No new key material to protect. The Ed25519 key already in `/var/lib/archipelago/identity/node_key` is used for both.
+
+2. **DHT is public**: Publishing to the DHT makes the node's DID Document visible to anyone querying the DHT. This is intentional for discoverability. Sensitive information (Tor addresses) should only be included in the service endpoints if the user explicitly opts in.
+
+3. **No Tor address by default**: The DID Document published to DHT should NOT include Tor hidden service addresses by default (per the security rule about not publishing onion addresses publicly). Tor addresses are exchanged privately via federation.
+
+4. **BEP-44 signature verification**: All DHT records are signed with Ed25519. Resolvers verify the signature, preventing tampering.
+
+5. **Sybil resistance**: did:dht identifiers are derived from public keys, so creating a fake identity requires generating a new keypair. The federation trust system already handles this via trust levels.
+
+## Comparison: did:key vs did:dht
+
+| Property | did:key | did:dht |
+|----------|---------|---------|
+| Offline creation | Yes | No (needs DHT access) |
+| Discoverable | No (must share manually) | Yes (query by identifier) |
+| Persistence | Permanent (derived from key) | TTL-based (needs refresh) |
+| Network requirement | None | UDP to DHT peers |
+| Resolution | Local computation only | DHT query (~1-5s) |
+| Privacy | Key not published anywhere | Key is on the DHT |
+| W3C standard | Yes (DID Core) | Yes (DID Core) |
+
+## Timeline
+
+1. **DHT-02**: Implement did:dht creation + publication (~2 days)
+2. **DHT-03**: Implement did:dht resolution + caching (~1 day)
+3. **DHT-04**: Web5 UI integration (~1 day)
+4. **Testing**: Cross-node resolution via DHT (separate from Tor) (~1 day)
+
+## References
+
+- [did:dht Method Specification](https://did-dht.com/)
+- [BEP-44: Storing arbitrary data in the DHT](https://www.bittorrent.org/beps/bep_0044.html)
+- [Mainline DHT crate](https://crates.io/crates/mainline)
+- [W3C DID Core 1.0](https://www.w3.org/TR/did-core/)
--- a/docs/dwn-protocols.md
+++ b/docs/dwn-protocols.md
@ -0,0 +1,400 @@
+# Archipelago DWN Protocol Definitions
+
+## Overview
+
+These protocol definitions specify the data schemas used when Archipelago nodes exchange data via Decentralized Web Nodes (DWN). By defining protocols in the standard DWN format, any DWN-compatible application can read and write Archipelago data.
+
+Each protocol defines:
+- A unique URI identifier
+- Data types with JSON schemas
+- Structure rules (who can read/write what)
+
+## Protocol 1: Node Identity Announcements
+
+**URI**: `https://archipelago.dev/protocols/node-identity/v1`
+
+Nodes publish their identity information so federated peers can discover capabilities and status.
+
+```json
+{
+  "protocol": "https://archipelago.dev/protocols/node-identity/v1",
+  "published": true,
+  "types": {
+    "announcement": {
+      "schema": "https://archipelago.dev/schemas/node-announcement/v1",
+      "dataFormats": ["application/json"]
+    }
+  },
+  "structure": {
+    "announcement": {
+      "$actions": [
+        { "who": "anyone", "can": ["read"] },
+        { "who": "author", "of": "announcement", "can": ["write", "update", "delete"] }
+      ]
+    }
+  }
+}
+```
+
+**Schema**: `node-announcement/v1`
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "required": ["did", "version", "apps", "timestamp"],
+  "properties": {
+    "did": {
+      "type": "string",
+      "description": "Node's DID (did:key or did:dht)"
+    },
+    "version": {
+      "type": "string",
+      "description": "Archipelago version (semver)"
+    },
+    "apps": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "List of installed app IDs"
+    },
+    "capabilities": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Node capabilities (e.g., 'file-sharing', 'dwn-sync', 'tor')"
+    },
+    "timestamp": {
+      "type": "string",
+      "format": "date-time",
+      "description": "ISO 8601 timestamp of announcement"
+    }
+  }
+}
+```
+
+## Protocol 2: File Sharing Catalog
+
+**URI**: `https://archipelago.dev/protocols/file-catalog/v1`
+
+Nodes publish their shared file catalogs so peers can browse available content.
+
+```json
+{
+  "protocol": "https://archipelago.dev/protocols/file-catalog/v1",
+  "published": true,
+  "types": {
+    "entry": {
+      "schema": "https://archipelago.dev/schemas/file-entry/v1",
+      "dataFormats": ["application/json"]
+    }
+  },
+  "structure": {
+    "entry": {
+      "$actions": [
+        { "who": "anyone", "can": ["read"] },
+        { "who": "author", "of": "entry", "can": ["write", "update", "delete"] }
+      ]
+    }
+  }
+}
+```
+
+**Schema**: `file-entry/v1`
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "required": ["id", "title", "access", "size_bytes", "created_at"],
+  "properties": {
+    "id": {
+      "type": "string",
+      "description": "Unique file entry ID"
+    },
+    "title": {
+      "type": "string",
+      "description": "Display title"
+    },
+    "description": {
+      "type": "string",
+      "description": "Optional description"
+    },
+    "content_type": {
+      "type": "string",
+      "description": "MIME type (e.g., 'application/pdf')"
+    },
+    "size_bytes": {
+      "type": "integer",
+      "description": "File size in bytes"
+    },
+    "access": {
+      "type": "string",
+      "enum": ["free", "peers-only", "paid"],
+      "description": "Access level"
+    },
+    "price_sats": {
+      "type": "integer",
+      "description": "Price in satoshis (for paid access)"
+    },
+    "hash": {
+      "type": "string",
+      "description": "SHA-256 hash of file content"
+    },
+    "created_at": {
+      "type": "string",
+      "format": "date-time"
+    },
+    "tags": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Content tags for filtering"
+    }
+  }
+}
+```
+
+## Protocol 3: Federation State
+
+**URI**: `https://archipelago.dev/protocols/federation/v1`
+
+Nodes publish their federation membership and peer trust status.
+
+```json
+{
+  "protocol": "https://archipelago.dev/protocols/federation/v1",
+  "published": false,
+  "types": {
+    "membership": {
+      "schema": "https://archipelago.dev/schemas/federation-membership/v1",
+      "dataFormats": ["application/json"]
+    },
+    "peerStatus": {
+      "schema": "https://archipelago.dev/schemas/peer-status/v1",
+      "dataFormats": ["application/json"]
+    }
+  },
+  "structure": {
+    "membership": {
+      "$actions": [
+        { "who": "recipient", "of": "membership", "can": ["read"] },
+        { "who": "author", "of": "membership", "can": ["write", "update", "delete"] }
+      ]
+    },
+    "peerStatus": {
+      "$actions": [
+        { "who": "recipient", "of": "peerStatus", "can": ["read"] },
+        { "who": "author", "of": "peerStatus", "can": ["write", "update"] }
+      ]
+    }
+  }
+}
+```
+
+**Schema**: `federation-membership/v1`
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "required": ["node_did", "trust_level", "joined_at"],
+  "properties": {
+    "node_did": {
+      "type": "string",
+      "description": "DID of the federated node"
+    },
+    "trust_level": {
+      "type": "string",
+      "enum": ["trusted", "verified", "untrusted"],
+      "description": "Trust level assigned to this peer"
+    },
+    "joined_at": {
+      "type": "string",
+      "format": "date-time"
+    },
+    "last_seen": {
+      "type": "string",
+      "format": "date-time"
+    },
+    "apps": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "Apps reported by this peer"
+    },
+    "credential_id": {
+      "type": "string",
+      "description": "VC ID proving federation relationship"
+    }
+  }
+}
+```
+
+**Schema**: `peer-status/v1`
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "required": ["node_did", "online", "timestamp"],
+  "properties": {
+    "node_did": {
+      "type": "string"
+    },
+    "online": {
+      "type": "boolean"
+    },
+    "cpu_percent": {
+      "type": "number"
+    },
+    "memory_used_mb": {
+      "type": "integer"
+    },
+    "disk_used_percent": {
+      "type": "number"
+    },
+    "container_count": {
+      "type": "integer"
+    },
+    "uptime_seconds": {
+      "type": "integer"
+    },
+    "timestamp": {
+      "type": "string",
+      "format": "date-time"
+    }
+  }
+}
+```
+
+## Protocol 4: App Deployment Requests
+
+**URI**: `https://archipelago.dev/protocols/app-deploy/v1`
+
+Enables trusted peers to request app installations on remote nodes.
+
+```json
+{
+  "protocol": "https://archipelago.dev/protocols/app-deploy/v1",
+  "published": false,
+  "types": {
+    "request": {
+      "schema": "https://archipelago.dev/schemas/deploy-request/v1",
+      "dataFormats": ["application/json"]
+    },
+    "response": {
+      "schema": "https://archipelago.dev/schemas/deploy-response/v1",
+      "dataFormats": ["application/json"]
+    }
+  },
+  "structure": {
+    "request": {
+      "$actions": [
+        { "who": "recipient", "of": "request", "can": ["read"] },
+        { "who": "author", "of": "request", "can": ["write"] }
+      ],
+      "response": {
+        "$actions": [
+          { "who": "author", "of": "request", "can": ["read"] },
+          { "who": "recipient", "of": "request", "can": ["write"] }
+        ]
+      }
+    }
+  }
+}
+```
+
+**Schema**: `deploy-request/v1`
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "required": ["app_id", "requester_did", "target_did", "action"],
+  "properties": {
+    "app_id": {
+      "type": "string",
+      "description": "Application identifier (e.g., 'bitcoin-knots')"
+    },
+    "requester_did": {
+      "type": "string",
+      "description": "DID of the requesting node"
+    },
+    "target_did": {
+      "type": "string",
+      "description": "DID of the target node"
+    },
+    "action": {
+      "type": "string",
+      "enum": ["install", "update", "uninstall"],
+      "description": "Deployment action"
+    },
+    "image": {
+      "type": "string",
+      "description": "Docker/OCI image reference"
+    },
+    "version": {
+      "type": "string",
+      "description": "Requested version"
+    },
+    "reason": {
+      "type": "string",
+      "description": "Human-readable reason for request"
+    },
+    "requested_at": {
+      "type": "string",
+      "format": "date-time"
+    }
+  }
+}
+```
+
+**Schema**: `deploy-response/v1`
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "required": ["request_id", "status"],
+  "properties": {
+    "request_id": {
+      "type": "string",
+      "description": "ID of the original request"
+    },
+    "status": {
+      "type": "string",
+      "enum": ["accepted", "rejected", "completed", "failed"],
+      "description": "Response status"
+    },
+    "reason": {
+      "type": "string",
+      "description": "Reason for rejection or failure"
+    },
+    "completed_at": {
+      "type": "string",
+      "format": "date-time"
+    }
+  }
+}
+```
+
+## Registration
+
+On backend startup, all 4 protocols should be auto-registered via `dwn.register-protocol`:
+
+```rust
+const ARCHIPELAGO_PROTOCOLS: &[&str] = &[
+    "https://archipelago.dev/protocols/node-identity/v1",
+    "https://archipelago.dev/protocols/file-catalog/v1",
+    "https://archipelago.dev/protocols/federation/v1",
+    "https://archipelago.dev/protocols/app-deploy/v1",
+];
+```
+
+## Interoperability
+
+These protocols follow the DWN protocol definition format. Any application that implements DWN can:
+
+1. **Read** node announcements to discover Archipelago nodes
+2. **Browse** file catalogs to find shared content
+3. **Query** federation state to understand network topology
+4. **Request** app deployments through the standard DWN messaging interface
+
+The `published: true` flag on protocols 1 and 2 means any DWN node can query for these records. Protocols 3 and 4 are `published: false` (private — only shared with authorized peers).
--- a/loop/plan.md
+++ b/loop/plan.md
@ -257,7 +257,7 @@ Every test must pass **10 consecutive times** from BOTH .228→.198 AND .198→.

 ### Sprint 9: did:dht Implementation

- [ ] **DHT-01** — Research and document did:dht integration approach. Study the did:dht spec (uses BitTorrent DHT — Mainline DHT). Document: how to publish DIDs to the DHT, how to resolve them, what library/crate to use (or implement), how it fits alongside existing did:key. Write to `docs/did-dht-integration.md`. **Acceptance**: Architecture document with specific implementation plan.
+- [x] **DHT-01** — Created `docs/did-dht-integration.md`. Covers: did:dht spec (BEP-44 mutable DHT items), DNS packet encoding, z-base-32 identifiers, publication/resolution flows, `mainline` crate for Rust DHT access, security considerations (no Tor addresses in public DHT), comparison with did:key, new RPC endpoints, background refresh every 2h, integration points with federation/VCs/Web5 UI.

 - [ ] **DHT-02** — Implement did:dht creation in identity_manager.rs. Add `create_dht_did()` method that: (1) generates Ed25519 keypair, (2) creates a DNS packet encoding per did:dht spec, (3) publishes to Mainline DHT using a Rust BitTorrent DHT library (e.g., `mainline` crate). The node should have BOTH did:key (local, offline) and did:dht (discoverable, no server needed). Add `identity.create-dht-did` RPC endpoint. **Acceptance**: Can create a did:dht and resolve it from another machine using the DHT.

@ -267,7 +267,7 @@ Every test must pass **10 consecutive times** from BOTH .228→.198 AND .198→.

 ### Sprint 10: DWN Protocol Definitions for Interoperable Schemas

- [ ] **SCHEMA-01** — Define Archipelago DWN protocol schemas. Create protocol definitions for the data types Archipelago shares between nodes: (1) Node identity announcements, (2) File sharing catalogs, (3) Federation state, (4) App deployment requests. Use the DWN protocol definition format so other apps implementing DWN could read Archipelago data. Document in `docs/dwn-protocols.md`. **Acceptance**: 4 protocol definitions documented with JSON schemas.
+- [x] **SCHEMA-01** — Created `docs/dwn-protocols.md` with 4 protocol definitions: (1) Node Identity Announcements (node-identity/v1) — public, node DID/version/apps/capabilities. (2) File Sharing Catalog (file-catalog/v1) — public, file entries with access levels/pricing. (3) Federation State (federation/v1) — private, membership + peer status with trust levels. (4) App Deployment Requests (app-deploy/v1) — private, request/response for remote app install. All with JSON schemas, DWN protocol definition format, and interoperability notes.

 - [ ] **SCHEMA-02** — Register Archipelago protocols in DWN on both nodes. On startup, the backend should auto-register all 4 Archipelago protocols via `dwn.register-protocol`. Verify protocols are registered on both .228 and .198. **Acceptance**: `dwn.list-protocols` on both nodes shows all 4 Archipelago protocols.