Expose Fleet alert failures and distinguish report provenance

docs/fleet-recovery-qualification-20261007.md

@@ -98,3 +98,55 @@ qualification conditions are stable. Yaya was not changed.
 
 Logs: `/tmp/archy-combined-ui-dev-deploy-retry.log` and the private preflight
 receipt `/tmp/archy-combined-ui-dev-preflight-retry.json`.
+
+## Additional partial-failure and provenance fixtures
+
+The later artwork UI artifact did successfully deliver the earlier Fleet recovery
+changes to dev and Yaya; see `docs/companion-audio-investigation-20261007.md` for
+the guarded deployment receipt. The failed combined deployment above remains
+historical evidence, not the latest UI deployment state.
+
+A new isolated fixture found alerts failures were swallowed: a denied, unsupported
+or timed-out alerts method left an empty panel saying “No alerts across the fleet.”
+A malformed alert list could also replace valid retained alerts with null entries.
+The UI now reports alerts unavailable or explicitly marks retained alerts while
+keeping successful node status results usable. A successful alert response clears
+the warning; an older failed request cannot overwrite that recovery.
+
+Node cards now distinguish collector/unverified reports from trusted federation.
+The trusted label requires all three server-owned fields: `source=federation`,
+`trust_level=trusted`, and strict `identity_authenticated=true`. Collector reports
+say “Collector · identity unverified”; legacy/missing or incomplete provenance says
+“Source unverified.” This is presentation of server evidence, not frontend identity
+verification. It MUST ship with the corrected backend provenance gate: historical
+collector ingestion accepted caller-supplied fields, so UI labels alone cannot
+repair that boundary. No isolated Fleet change in this section is deployed.
+
+Focused suite: **28 tests in five files passed**. Seven added cases cover three
+partial-alert failure classes with mixed-version node reports and real zero versus
+missing metrics, malformed alerts retaining prior evidence, late history after
+unmount, old alert denial after newer success, and provenance labels. These are
+synthetic RPC/component fixtures, not proof of real authorization or partition
+behavior. Before the change, four new partial-failure cases failed and the existing
+late-history safeguard passed. Initial wrong-project test invocation failed before
+test collection; the corrected invocation established the failures. Logs are
+`/tmp/archy-fleet-partial-before.log`,
+`/tmp/archy-fleet-partial-before-corrected.log`, and
+`/tmp/archy-fleet-acceptance-final.log`.
+
+Current Fleet UI exposes status, alerts, history and export only. It has no remote
+restart/update/action RPC, so those action-success/partial-failure acceptance cells
+remain unsupported/open; no service operation was invented for the fixture.
+
+Exact app-project typecheck passed (`vue-tsc --noEmit -p tsconfig.app.json`,
+2 GB heap cap, low priority); log `/tmp/archy-fleet-acceptance-typecheck.log`.
+Real Chromium rendered the actual source component at 390px and 1440px:
+collector and trusted labels, denied-alert warning, successful status cards,
+no false empty-alert claim, and Refresh recovery passed with no page errors or
+horizontal overflow. All non-loopback requests were blocked. The first cold-load
+attempt exceeded its 30-second load timeout before assertions; the corrected
+DOMContentLoaded/60-second fixture passed both widths. Logs:
+`/tmp/archy-fleet-acceptance-browser.log` and
+`/tmp/archy-fleet-acceptance-browser-retry.log`. Owned browser/server jobs stopped.
+This is source-browser evidence, not a production build or real identity/transport
+acceptance. Backend provenance qualification and coupled release remain required.

neode-ui/src/views/Fleet.vue

@@ -117,6 +117,7 @@
       <FleetAlerts
         :alerts="fleet.fleetAlerts.value"
         :alerts-loading="fleet.alertsLoading.value"
+        :error-message="fleet.alertsErrorMessage.value"
       />
 
       <FleetNodeDetail

neode-ui/src/views/fleet/FleetAlerts.vue

@@ -2,10 +2,11 @@
   <div class="glass-card p-5 mb-6">
     <h3 class="text-sm font-medium text-white/80 mb-4">Fleet Alerts</h3>
 
+    <p v-if="errorMessage" role="status" class="text-sm text-amber-200 mb-3">{{ errorMessage }}</p>
     <div v-if="alertsLoading" class="text-white/40 text-sm py-4 text-center">
       Loading alerts...
     </div>
-    <div v-else-if="!alerts.length" class="text-white/40 text-sm py-4 text-center">
+    <div v-else-if="!alerts.length && !errorMessage" class="text-white/40 text-sm py-4 text-center">
       No alerts across the fleet.
     </div>
     <div v-else class="space-y-2 max-h-80 overflow-y-auto">
@@ -37,5 +38,6 @@ import { type FleetAlert, alertSeverityDot, alertTypeLabel, formatTimestamp } fr
 defineProps<{
   alerts: FleetAlert[]
   alertsLoading: boolean
+  errorMessage?: string
 }>()
 </script>

neode-ui/src/views/fleet/FleetNodeGrid.vue

@@ -42,6 +42,8 @@
           {{ fleetNodeSubtitle(node) }}
         </div>
 
+        <div class="mb-3 text-xs text-white/50">{{ fleetSourceLabel(node) }}</div>
+
         <div class="space-y-2 mb-3">
           <div class="fleet-metric-row">
             <span class="text-xs text-white/50">CPU</span>
@@ -94,7 +96,7 @@
 <script setup lang="ts">
 import {
   type FleetNode, type SortOption, SORT_OPTIONS,
-  isOnline, fleetStatus, formatMetric, healthBarClass, formatUptime, timeAgo, fleetNodeDisplayName, fleetNodeSubtitle,
+  isOnline, fleetStatus, formatMetric, healthBarClass, formatUptime, timeAgo, fleetNodeDisplayName, fleetNodeSubtitle, fleetSourceLabel,
 } from './useFleetData'
 
 defineProps<{

neode-ui/src/views/fleet/__tests__/FleetNodeGrid.test.ts

@@ -51,3 +51,26 @@ describe('fleet unavailable readings', () => {
     wrapper.unmount()
   })
 })
+
+it('labels collector, legacy, and incomplete provenance as unverified', () => {
+  const nodes = [
+    normalizeFleetNode({ node_id: 'trusted', source: 'federation', trust_level: 'trusted', identity_authenticated: true }),
+    normalizeFleetNode({ node_id: 'collector', source: 'collector', trust_level: 'unverified', identity_authenticated: false }),
+    normalizeFleetNode({ node_id: 'legacy', source: 'federation', trust_level: 'trusted' }),
+    normalizeFleetNode({ node_id: 'unknown' }),
+    normalizeFleetNode({ node_id: 'denied', source: 'federation', trust_level: 'untrusted', identity_authenticated: true }),
+  ]
+  const wrapper = mount(FleetNodeGrid, {
+    props: { nodes, sortedNodes: nodes, sortBy: 'status', selectedNodeId: null },
+    global: { mocks: { $ver: (v: string) => v } },
+  })
+  try {
+    const cards = wrapper.findAll('.fleet-node-card')
+    expect(cards[0]!.text()).toContain('Trusted federation')
+    expect(cards[1]!.text()).toContain('Collector · identity unverified')
+    for (const card of cards.slice(2)) {
+      expect(card.text()).toContain('Source unverified')
+      expect(card.text()).not.toContain('Trusted federation')
+    }
+  } finally { wrapper.unmount() }
+})

neode-ui/src/views/fleet/__tests__/FleetPartialFailure.test.ts

@@ -0,0 +1,84 @@
+import { afterEach, expect, it, vi } from 'vitest'
+import { defineComponent } from 'vue'
+import { flushPromises, mount } from '@vue/test-utils'
+import { rpcClient } from '@/api/rpc-client'
+import { useFleetData } from '../useFleetData'
+import FleetAlerts from '../FleetAlerts.vue'
+
+vi.mock('@/api/rpc-client', () => ({ rpcClient: { call: vi.fn() } }))
+afterEach(() => { sessionStorage.clear(); vi.clearAllMocks() })
+
+it.each(['Unauthorized', 'Method not found', 'Timeout'])('keeps mixed-version nodes usable when alerts fail: %s', async reason => {
+  vi.mocked(rpcClient.call).mockImplementation(({ method }) => method === 'telemetry.fleet-status'
+    ? Promise.resolve({ nodes: [{ node_id: 'legacy', version: 'old' }, { node_id: 'current', cpu_pct: 0, mem_pct: 0, disk_pct: 0 }] }) as never
+    : Promise.reject(new Error(reason)))
+  let fleet!: ReturnType<typeof useFleetData>
+  const wrapper = mount(defineComponent({ setup() { fleet = useFleetData(); return () => null } }))
+  try {
+    await flushPromises()
+    expect(fleet.nodes.value.map(n => n.node_id)).toEqual(['legacy', 'current'])
+    expect(fleet.nodes.value[0]?.cpu_pct).toBeNull()
+    expect(fleet.nodes.value[1]?.cpu_pct).toBe(0)
+    expect(fleet.errorMessage.value).toBe('')
+    expect(fleet.alertsErrorMessage.value).toContain('unavailable')
+    expect(fleet.alertsLoading.value).toBe(false)
+    const panel = mount(FleetAlerts, { props: { alerts: [], alertsLoading: false, errorMessage: fleet.alertsErrorMessage.value } })
+    expect(panel.text()).not.toContain('No alerts across the fleet')
+    expect(panel.get('[role="status"]').text()).toContain('unavailable')
+    panel.unmount()
+    vi.mocked(rpcClient.call).mockResolvedValue({ nodes: [], alerts: [] })
+    await fleet.refreshAll()
+    expect(fleet.alertsErrorMessage.value).toBe('')
+  } finally { wrapper.unmount() }
+})
+
+it('retains previous alerts on malformed refresh without calling them current', async () => {
+  const alert = { node_id: 'node', rule: 'cpu_high', message: 'Earlier alert', timestamp: '2026-10-07T12:00:00Z' }
+  sessionStorage.setItem('archipelago.fleet.cache.v2', JSON.stringify({ fleetAlerts: [alert] }))
+  vi.mocked(rpcClient.call).mockResolvedValue({ nodes: [], alerts: [null] })
+  let fleet!: ReturnType<typeof useFleetData>
+  const wrapper = mount(defineComponent({ setup() { fleet = useFleetData(); return () => null } }))
+  try {
+    await flushPromises()
+    expect(fleet.fleetAlerts.value).toEqual([alert])
+    expect(fleet.alertsErrorMessage.value).toContain('last received alerts')
+    const panel = mount(FleetAlerts, { props: { alerts: fleet.fleetAlerts.value, alertsLoading: false, errorMessage: fleet.alertsErrorMessage.value } })
+    expect(panel.text()).toContain('Earlier alert')
+    expect(panel.get('[role="status"]').text()).toContain('last received alerts')
+    panel.unmount()
+  } finally { wrapper.unmount() }
+})
+
+it('does not apply selected-node history after unmount', async () => {
+  let finish!: (value: unknown) => void
+  vi.mocked(rpcClient.call).mockImplementation(({ method }) => method === 'telemetry.fleet-node-history'
+    ? new Promise(done => { finish = done }) as never
+    : Promise.resolve({ nodes: [], alerts: [] }) as never)
+  let fleet!: ReturnType<typeof useFleetData>
+  const wrapper = mount(defineComponent({ setup() { fleet = useFleetData(); return () => null } }))
+  await flushPromises()
+  fleet.selectNode('node'); await flushPromises()
+  wrapper.unmount()
+  finish({ entries: [{ timestamp: 'late', cpu_pct: 80 }] })
+  await flushPromises()
+  expect(fleet.nodeHistory.value).toEqual([])
+})
+
+it('ignores an older alerts failure after a newer successful refresh', async () => {
+  let failOld!: (reason: Error) => void
+  let alertCalls = 0
+  vi.mocked(rpcClient.call).mockImplementation(({ method }) => {
+    if (method === 'telemetry.fleet-status') return Promise.resolve({ nodes: [] }) as never
+    if (++alertCalls === 1) return new Promise((_, reject) => { failOld = reject }) as never
+    return Promise.resolve({ alerts: [] }) as never
+  })
+  let fleet!: ReturnType<typeof useFleetData>
+  const wrapper = mount(defineComponent({ setup() { fleet = useFleetData(); return () => null } }))
+  try {
+    await fleet.refreshAll()
+    failOld(new Error('Old permission denial'))
+    await flushPromises()
+    expect(fleet.alertsErrorMessage.value).toBe('')
+    expect(fleet.alertsLoading.value).toBe(false)
+  } finally { wrapper.unmount() }
+})

neode-ui/src/views/fleet/useFleetData.ts

@@ -8,6 +8,9 @@ import type { ChartDataset } from '@/components/LineChart.vue'
 
 export interface FleetNode {
   node_id: string
+  identity_authenticated?: boolean
+  source?: string
+  trust_level?: string
   node_name?: string | null
   hostname?: string | null
   server_url?: string | null
@@ -137,6 +140,12 @@ export function fleetNodeDisplayName(node: FleetNode): string {
   return name || node.node_id.slice(0, 8)
 }
 
+export function fleetSourceLabel(node: FleetNode): string {
+  if (node.source === 'collector') return 'Collector · identity unverified'
+  if (node.source === 'federation' && node.trust_level === 'trusted' && node.identity_authenticated === true) return 'Trusted federation'
+  return 'Source unverified'
+}
+
 export function fleetNodeSubtitle(node: FleetNode): string {
   const host = node.hostname?.trim()
   if (host && host !== fleetNodeDisplayName(node)) return host
@@ -188,6 +197,9 @@ function numberOrZero(value: unknown): number {
 export function normalizeFleetNode(node: Partial<FleetNode>): FleetNode {
   return {
     node_id: typeof node.node_id === 'string' ? node.node_id : 'unknown',
+    identity_authenticated: node.identity_authenticated === true,
+    source: typeof node.source === 'string' ? node.source : 'unknown',
+    trust_level: typeof node.trust_level === 'string' ? node.trust_level : 'unverified',
     node_name: typeof node.node_name === 'string' ? node.node_name : null,
     hostname: typeof node.hostname === 'string' ? node.hostname : null,
     server_url: typeof node.server_url === 'string' ? node.server_url : null,
@@ -265,6 +277,7 @@ export function useFleetData() {
   const fleetAlerts = ref<FleetAlert[]>(cached.fleetAlerts ?? [])
   const refreshing = ref(false)
   const alertsLoading = ref(false)
+  const alertsErrorMessage = ref('')
   const selectedNodeId = ref<string | null>(cached.selectedNodeId ?? null)
   const nodeHistory = ref<NodeHistoryEntry[]>([])
   const nodeHistoryLoading = ref(false)
@@ -379,8 +392,14 @@ export function useFleetData() {
         method: 'telemetry.fleet-alerts',
       })
       if (disposed || request !== alertsRequest) return
-      if (Array.isArray(data?.alerts)) {
+      if (!Array.isArray(data?.alerts) || data.alerts.some(alert => !alert ||
+        typeof alert.node_id !== 'string' || typeof alert.rule !== 'string' ||
+        typeof alert.message !== 'string' || typeof alert.timestamp !== 'string')) {
+        throw new Error('Invalid fleet alerts response')
+      }
+      {
         fleetAlerts.value = data.alerts
+        alertsErrorMessage.value = ''
         writeFleetCache({
           nodes: nodes.value,
           fleetAlerts: fleetAlerts.value,
@@ -390,7 +409,11 @@ export function useFleetData() {
         })
       }
     } catch {
-      // Non-critical, retry on next poll
+      if (!disposed && request === alertsRequest) {
+        alertsErrorMessage.value = fleetAlerts.value.length
+          ? 'Alert refresh failed. Showing the last received alerts.'
+          : 'Fleet alerts are unavailable. Retry refresh to check for alerts.'
+      }
     } finally {
       if (!disposed && request === alertsRequest) alertsLoading.value = false
     }
@@ -575,7 +598,7 @@ export function useFleetData() {
   })
 
   return {
-    loading, refreshing, errorMessage, nodes, fleetAlerts, alertsLoading,
+    loading, refreshing, errorMessage, nodes, fleetAlerts, alertsLoading, alertsErrorMessage,
     selectedNodeId, selectedNode, nodeHistory, nodeHistoryLoading,
     autoRefresh, lastRefreshed, sortBy, chartWidth, now,
     onlineCount, offlineCount, unknownCount, healthyCount, fleetHealthPct,