Observability Spec (formerly SPEC §9) — metrics and trace backend live¶

Metrics (§9.1): kube-prometheus-stack (Prometheus + Alertmanager + Grafana). The agentgateway chart supplies control-plane/proxy monitors and GenAI token, latency, TTFT, and TPOT series; kagent exposes controller /metrics; CNPG enables a PodMonitor. License note: Grafana and Loki are AGPL-3.0 (fine to run unmodified, documented as swappable — VictoriaMetrics/VictoriaLogs are the Apache-2.0 alternates for AGPL-banning shops; Prometheus/OTel/Jaeger are Apache-2.0).
Traces (§9.2): a dedicated OTel Collector accepts OTLP/gRPC (4317) and OTLP/HTTP (4318) from every workload namespace, batches traces, and exports only traces to Jaeger. Jaeger is the Apache-2.0 default over Tempo (AGPL); its query API has no HTTPRoute and is available to operators through the provisioned Grafana datasource. The bridge emits one content-free span per delegation (queue → A2A send/poll → Matrix reply) and injects W3C traceparent; kagent tracing and agentgateway's tracing policy target the same Collector. Do not claim cross-component continuity until the integration test proves one trace covers mention → gateway → agent → LLM.
Bridge metrics and audit (§9.3): Prometheus /metrics on a side port — delegations total/by agent, A2A latency histogram, queue depth, dedup hits, rate-limit rejections, and normalized 👍/👎 counts on known terminal agent replies. The bridge separately marks content-free fgentic.delegation.v1 JSON records with log_stream=audit; each records terminal reason/duration and explicit dedup/rate-limit verdicts, including suppressed redeliveries. The reference deploys no log database: VictoriaLogs remains a future explicit, pinned, access-controlled per-overlay opt-in and never a bridge runtime dependency. See the attribution runbook.
Declarative dashboards: Grafana's kube-prometheus-stack sidecar provisions Fgentic — Bridge (delegations, latency quantiles, queue/inflight pressure, rate-limit rejections, dedup) and Fgentic — LLM Token & Cost Guard (provider/model/route token signals, cost-catalog coverage, and 15-minute burn against the per-cluster guard) from versioned JSON in infra/observability/dashboards/. mise run check:dashboards parses every panel/query and validates the Flux-rendered ConfigMap. Live rendering and non-empty data remain cluster acceptance checks. The cost-dashboard runbook maps each panel to its attribution limit.
Agent evaluation (§9.4): MLflow (Apache-2.0, LF) is the optional eval/experiment store and is deployed off by default in its own namespace and database. The reference trace stream is content-suppressed and is not itself a prompt/response evaluation corpus; any content-bearing evaluation path requires the separately approved privacy, access, and retention design tracked by #38. Evaluation remains analysis tooling, not a runtime dependency.
LLM budget (§9.5): the provider-neutral system_model_token_type:agentgateway_gen_ai_client_token_usage_sum:increase15m recording rule preserves provider/model/token-type dimensions, while LLMTokenBurnHigh aggregates every profile against the per-cluster llm_usage_budget_15m threshold in tokens. This is a token guard, not a currency-cost dashboard. The repository has no versioned model cost catalog, provider billing export, or deterministic model-request-to-task correlation, so exact per-task currency cost is explicitly unavailable; see the attribution runbook.
Self-hosted model health: the central vllm PodMonitor scrapes /metrics on the serving engine's internal OpenAI API port. Runtime acceptance requires both an up target and non-empty vLLM request metrics after a chat; the agentgateway token histogram remains the provider-neutral budget signal.
NetworkPolicies (§9.6) already admit the monitoring namespace everywhere (D14).
Database audit (§9.7): CloudNativePG emits content-suppressed pgAudit DDL/ROLE records as structured JSON stdout. They remain node-runtime logs today and are an explicit selected stream for the future opt-in log pipeline in #157, not a durable store by themselves.
Kubernetes API audit (§9.8): local k3d writes selected request/response-body-suppressed API Metadata events to a size/backup/age-bounded node file. GKE uses its managed Cloud Audit Logs path instead. Neither is ingested by the current reference, so #157 owns transport, access, and durable retention.
Runtime image vulnerabilities (§9.9): Trivy Operator scans the immutable images used by running workloads and exposes report-level metrics. A warning fires only when the High or Critical count rises for the same namespace and image digest relative to its lowest value in the previous 48 hours. This complements D13's build-time gate; it does not replace digest pinning, signatures, SBOMs, or human review.
Matrix identity audit (§9.10): D19 and ADR 0018 define an opt-in regulated profile over exact pinned MAS/Synapse source records. The default platform emits no named MAS authentication or homeserver-wide event stream, and generic TerseJson/access logs are never promoted into audit evidence.

9.1 Metrics and showback boundary¶

Prometheus remains an aggregate operations signal, not the source of truth for a person, room, tenant, or invoice. The committed metrics retain only bounded dimensions already owned by their producers:

Signal	Retained dimensions	Safe interpretation
`fgentic_delegations_total`	configured ghost and bounded outcome	Delegation activity and terminal outcomes for one configured agent target; not a unique request, person, room, or team
`agentgateway_gen_ai_client_token_usage_sum`	provider, request/response model, route, token type	Aggregate provider-reported model tokens on the selected route; not a Matrix identity, tenant, partner, or invoice
`system_model_token_type:...:increase15m`	provider, requested model, token type	The same aggregate consumption over the alert window; not a hard budget or currency value
MCP tool-call quota state	authenticated Agent and resolved tool inside Redis	Fixed-window request admissions used to deny excess tool calls; not execution success, model tokens, or consumption
Federation `maxTokens` rate-limit state	verified client key inside the reservation component	Admission capacity reserved for a partner client; not observed downstream model consumption
`fgentic_room_token_budget_exhaustions_total`	none (deliberately label-free)	Count of delegations refused because a room reached its per-period token budget (#99); the room is in the audit log (`fgentic.room_token_budget.v1`), never a label

The room/team decision is to keep raw MXIDs and room IDs out of Prometheus. Hashing does not bound cardinality and does not remove linkability; an allowlist would still make a personal room identifier an operational label whose churn grows series over the retention window. Exact room, sender, event, and task correlation therefore stays in the restricted audit/evidence path described in Delegation Attribution Audit, not in metrics. A future tenant budget may add one operator-defined, bounded tenant key only after its authenticated mapping, lifecycle, and privacy review land with the tenancy work; it must not derive that key from a raw or hashed MXID/room ID.

These agentgateway_gen_ai_client_token_usage_sum labels track OpenTelemetry's gen_ai.* semantic conventions, which remain pre-stable: as verified on 2026-07-26 no GenAI attribute, metric, or span is marked Stable, and semconv v1.42.0 (June 2026) moved the gen_ai.* set into a dedicated semantic-conventions-genai repository. Treat an upstream label rename as an expected convention update to reconcile, not a metric regression; pin the naming to the deployed agentgateway version rather than to a frozen spec (#1432).

The pinned agentgateway v1.3.1 source confirms that its GenAI metric labels contain operation, provider, request/response model, route, token type, and optional custom fields, but no stable Fgentic user or team identity. The platform does not inject an identity custom field on the LLM path. Both Redis-backed rate-limit services run with StatsD disabled and expose no committed per-Agent/tool or per-client Prometheus series. Their state is an admission control input, while the content-free fgentic.mcp_tool_call.v1 log records an individual tool-call outcome without arguments or quota values. Dashboards must show these signals as aggregate consumption or reservation posture only when the corresponding metric exists; they must never join them by timing and call the result attributed usage.

9.2 Trace data plane¶

workload SDK --OTLP--> otel-collector --OTLP--> Jaeger memory store
                                                    ^
                                                    |
                                      Grafana datasource proxy

The reference is intentionally small: one Collector and one Jaeger v2 all-in-one replica, each limited to 256 MiB. Both images and Helm charts are immutable-version pinned, service-account tokens are disabled, containers are non-root with read-only filesystems, and NetworkPolicies allow only the two OTLP ingress ports. Prometheus scrapes their internal telemetry on 8888. Jaeger retains at most 10,000 traces in memory, so a restart loses trace history by design.

Storage choices are deployment policy, not an application dependency:

Memory (reference default): no PVC or external service, lowest cost, and disposable history. Use it for local development and the low-cost reference deployment.
Badger + PVC: survives restarts without another service and suits a single Jaeger all-in-one instance. It cannot scale horizontally; use it only for modest single-node installations and include the PVC in backup policy.
OpenSearch/Elasticsearch or Cassandra: use an external, persistent backend when retention, replication, or horizontal scale matters. Size and secure it independently; these options materially increase operational cost and are not enabled by the reference.

The Collector-to-Jaeger hop is unencrypted inside the cluster and constrained by NetworkPolicy. A deployment whose threat model does not trust its pod network must enable OTLP TLS/mTLS or a service-mesh identity layer before carrying sensitive span attributes.

The trace-content boundary is enforced at each producer, not by the Collector. Every managed Python Agent carries exactly ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false, and TRACELOOP_TRACE_CONTENT=false. These independently disable Google ADK, standard OpenTelemetry GenAI, and OpenLLMetry content capture while leaving OTLP tracing enabled. Admission rejects a missing, changed, duplicated, valueFrom-backed, or additional entry.

The bridge represents A2A send, task-timeout, and task-poll failures with fixed-name, attribute-free events rather than exception events, so server-controlled error text cannot become exception.message. The agentgateway tracing policy deliberately adds no prompt or completion expressions. The Collector has only an OTLP traces pipeline with memory_limiter and batch; it has no logs pipeline, content enrichment, or redaction processor. It forwards producer fields unchanged, so every new or changed instrumentation path must prove content suppression at its source. This gate covers exported traces only: ordinary kagent and bridge diagnostic logs can contain content and must not be admitted to #157's future log store as content-free streams.

Content-free means payload-free, not anonymous. Spans retain valid trace IDs and linkable operational metadata, including Matrix room, event, and sender identifiers; agent and sender-origin identity; provider and model identity; token counts; bounded outcomes and terminal reasons; and success or failure status. Restrict Collector ingestion and Grafana/Jaeger query access to authorized operators, and choose retention and deletion controls for the deployment threat model. The reference Jaeger memory cap and restart loss bound storage but are not a retention policy. A persistent trace or evaluation backend must define authentication, query access, retention, deletion, and its content boundary before it is enabled.

mise run test:tracing proves only the Collector → Jaeger → Grafana backend path with a synthetic span. mise run test:kagent-trace-privacy runs the pinned kagent runtime against local mock model and tool endpoints, inserts prompt, completion, tool-argument, and tool-result sentinels, and proves their absence from the final exported representation while retaining expected trace identity and bounded GenAI metadata. Bridge tests independently prove that A2A error sentinels do not enter spans. These gates do not prove cross-component continuity; full acceptance for #35 still requires one installed-cluster mention whose bridge, gateway, agent, and model spans share a trace ID. No Jaeger Ingress or HTTPRoute is enabled.

For installed-cluster acceptance, port-forward the private Jaeger query service and run the content-free topology assertion:

kubectl --context k3d-fgentic --namespace monitoring port-forward service/jaeger 16686:16686
mise run trace:continuity

The assertion searches recent matrix-a2a-bridge traces and requires one exact parent/descendant chain: bridge fgentic.delegation → agentgateway POST /api/a2a/* → kagent controller route and invoke_agent → managed-agent POST / and invoke_agent → generate_content. Its output is limited to the trace ID, fixed service/span names, and fixed service edges. It never emits span tags, logs, Matrix sender/room/event identifiers, prompts, or responses. The default target is the provider-free docs-qa/docs_qa demo agent; set FGENTIC_TRACE_AGENT and FGENTIC_TRACE_AGENT_SERVICE for another locally deployed agent.

9.3 Bridge reply-quality reaction signal¶

A human 👍 or 👎 reaction on a successfully projected terminal agent m.notice increments fgentic_agent_reply_quality_signals_total{ghost,signal} with signal="positive" or signal="negative". The low-cardinality metric deliberately omits room, event, and sender labels. A separate fgentic.agent_reply.quality_signal span retains the Matrix room, reaction event, reacting sender, target reply event, agent ghost, and normalized signal for authorized operational correlation. Neither surface contains reply text, reaction payload beyond the normalized signal, secrets, model output, or token figures.

The handler accepts only reply IDs in a 4,096-entry process-local, content-free registry populated after a terminal m.notice is projected successfully. Unknown events, events from another room, and reactions authored by the bridge bot or agent ghosts are ignored. Registry eviction or a bridge restart can therefore lose a late quality signal; Matrix remains the event source of record, and this metric is not a durable feedback ledger. Repeated valid reaction events are observations, not unique users or task-success proof. No reaction calls Matrix, A2A, an LLM, or emits another room event, so it cannot create invocation spend or an automation loop.

9.4 Sovereign LLM-as-judge eval scores¶

The optional eval:models judge lane (#355) scores qualitative scenarios for groundedness and task success using a judge model composed from the self-hosted vllm profile over the existing A2A/agentgateway route — eval prompts and agent outputs never leave the cluster and no evaluator holds a model credential (docs/models.md). The lane is opt-in and runs only against a residency: self-hosted catalog model; a metered external provider blocks it fail-closed.

The judge answer is parsed into a trusted type at the boundary (strict JSON, scores bounded to [0,1]); a malformed, missing, or out-of-range judge answer fails the scenario rather than silently passing it. Only the two bounded numbers are recorded (judge_scores.groundedness/judge_scores.task_success per scenario in the eval report, alongside existing token/latency deltas); the judge rationale, prompt, and agent output are never persisted, logged, or metered. These scores feed the optional MLflow eval store as content-free quality evidence, not a live production signal.

9.7 Database audit stream¶

The shared CNPG cluster enables pgAudit through four operator-managed parameters: pgaudit.log=ddl, role, with catalog-only noise, SQL statement text, and parameters disabled. CloudNativePG manages shared_preload_libraries and the extension lifecycle for every connectable database, then emits each parsed record with logger=pgaudit, msg=record, and typed fields under record.audit. Normal Synapse and application READ/WRITE traffic is not audited, so ordinary platform activity does not become a high-volume content stream.

scripts/lib/postgres-audit.jq is the reviewed minimal SQL/payload-suppressed projection for operators and #157: it retains time, pod, database/session role, database, session ID, class, command, statement IDs, and object type/name while dropping statement and parameter fields. Those retained identifiers remain sensitive operational metadata. Pod stdout has no repository-defined retention or access layer. The future log pipeline must select that projection, enforce its own retention/authentication/TLS controls, and keep debug and ordinary PostgreSQL records out; until then, no durable pgAudit claim is made.

pg_stat_statements is deliberately deferred. Although CNPG can manage it, query statistics are a distinct performance-observability feature with sizing, access, reset, query-text, and retention decisions; enabling it is neither free nor required to establish the DDL/ROLE audit boundary.

9.8 Kubernetes API audit stream¶

The local profile's input contract for #157 is /var/log/kubernetes/audit/audit.log plus its rotated siblings inside each k3d server node. infra/k3d-audit-policy.yaml admits only high-signal Metadata records and captures no Kubernetes object body. Request URIs remain sensitive and can expose pods/exec command arguments. A future collector must select that path explicitly, preserve body suppression, restrict query access, and test retention; it must not broaden the API policy or scrape arbitrary node logs as part of this handoff. Deleting the local cluster deletes the current evidence.

GKE writes Kubernetes API evidence through managed Cloud Audit Logs, not the local node path. Production routing must use Cloud Logging sinks and IAM with separately approved retention and Data Access settings. The repository does not deploy or spend on that cloud path automatically.

9.9 Runtime image vulnerability drift¶

The local and gcp overlays reconcile a structurally separate trivy-operator Flux layer into the restricted trivy-system namespace; the evaluation demo and disposable federation overlays remove that layer and its namespace budget, so a fresh reconciliation installs no operator, scan Jobs, report CRDs, monitor, or rules. The source is Trivy Operator v0.32.0 at release commit 1006872c1463e81a40d48298145625aefef2a02f, whose in-tree chart is 0.34.0. The two runtime images are independently immutable:

mirror.gcr.io/aquasec/trivy-operator:0.32.0@sha256:d4a61c4607e2931bd2615bf3bcf8912669d11d194c44c77edd413e6301b50c5b
mirror.gcr.io/aquasec/trivy:0.72.0@sha256:cffe3f5161a47a6823fbd23d985795b3ed72a4c806da4c4df16266c02accdd6f

The scanner watches exactly cert-manager, gateway, cnpg-system, postgres, knowledge, keycloak, matrix, agentgateway-system, kagent, bridge, bridges, monitoring, and models. It creates namespaced VulnerabilityReport objects for Pods, ReplicaSets, StatefulSets, DaemonSets, CronJobs, and Jobs. The operator automatically caches trivy-system so it can observe centralized Jobs, but its install-mode predicate deliberately excludes that namespace from workload scanning; the pinned operator/scanner images remain in the human-reviewed dependency-update lane rather than this self-scan signal. Reports expire after 24 hours so a still-running image is rescanned against refreshed vulnerability data; completed scan Jobs expire after 30 seconds. The operator's configured concurrency limit is not trusted as an atomic guard: v0.32.0 can race its informer cache during simultaneous events, so a dedicated count/jobs.batch: 1 ResourceQuota enforces one Job at Kubernetes admission. Configuration, RBAC, secret, infrastructure, compliance, and SBOM scanners are disabled. The security boundary documents the resulting permissions and residual risks.

The trivy-operator ServiceMonitor lives in monitoring, selects the exact operator Service in trivy-system, and preserves each report's workload namespace label. namespace_image_vulnerability_severity:trivy_image_vulnerabilities:max accepts only canonical sha256:<64 lowercase hex> identities and deduplicates replica/workload series per namespace, digest, and severity; tag-only reports and reports whose registry did not resolve a digest are deliberately outside this immutable-identity signal. The rule renames the source severity to vulnerability_severity so Alertmanager can use severity=warning for routing. TrivyImageVulnerabilityDrift waits ten minutes before alerting on a positive difference from the lowest count in the preceding 48 hours.

This is a count-drift signal, not a vulnerability identity diff. A newly deployed digest starts a fresh series and does not alert on its initial baseline. Replacing one CVE with another at the same count, a report that disappears, or a change below High is also silent. Always inspect the referenced VulnerabilityReport and the image digest; do not infer remediation from the alert merely clearing. The production runbook defines triage and acceptance.

9.10 Matrix identity audit boundary¶

The reference stack's ordinary Synapse and MAS logs are diagnostic data, not versioned audit schemas. Synapse's TerseJson formatter serializes arbitrary LogRecord messages and extra fields; MAS request telemetry retains method/route/path/query/status/User-Agent but its login handlers deliberately omit request fields. Neither source proves a stable authenticated identity or event-acceptance claim.

D19 and ADR 0018 select two narrower records for regulated deployments: committed MAS authentication rows and non-rejected, non-outlier Synapse event metadata. The component is disabled by default, depends on #157's durable authenticated query/retention boundary, pins the exact private database schemas, and has a closed negative-content contract. Keycloak LOGIN/LOGIN_ERROR evidence remains a separate upstream-IdP claim owned by #300.

9.11 Synthetic delegation canary¶

The local and gcp overlays can opt into a scheduled end-to-end canary (synthetic_canary: enabled in platform-settings); it is disabled by default and structurally absent in demo and federation (their canary Kustomization resolves to profiles/disabled, which prunes to zero objects, and the whole observability-monitors layer that holds its alert is removed there). On a quiet cluster a broken delegation plane — appservice registration drift, a bridge/agentgateway/kagent outage, a model-credential expiry — produces no failing metric, only silence; HTTP blackbox probes cannot prove the loop. The canary does: a CronJob (canary namespace) posts one nonce-bearing @mention into a dedicated, unencrypted canary room and asserts the target ghost's m.notice reply within a deadline, exiting non-zero on timeout.

The probe is content-free by construction — the message body is only a fixed marker plus a random nonce and a timestamp, never real prompts or data — and it targets the configured @agent-docs-qa ghost, so it exercises bridge intake, the ledger, agentgateway routing, kagent A2A, and the cluster's selected model. The committed local default selects Vertex google/gemini-2.5-flash, so its canary crosses that provider's prompt-egress and billing boundary. The gcp default selects in-cluster vLLM and consumes GPU/model-runtime resources without serving-time prompt egress; its funded live-GPU proof remains open under issue #10. The canary sender is an ordinary allowlisted @canary user on the normal admission and rate-limit path — no privileged bypass — and its invocation volume is bounded entirely by the CronJob schedule, not made free by it, so it cannot amplify (D7/D8). Its credential, room id, and allowlist entry are provisioning outputs kept in the SOPS canary-credential Secret and the per-cluster agents.yaml. Restricted Matrix/audit evidence can identify the canary sender, but agentgateway model telemetry retains only provider/model/route/token-type dimensions: canary tokens remain in aggregate consumption and cannot be excluded from showback/chargeback or quality dashboards without a separately reviewed bounded telemetry key.

The alerts read kube-state-metrics (no Pushgateway): SyntheticCanaryDelegationFailing fires when a canary Job fails (kube_job_status_failed), and SyntheticCanaryStale fires when the CronJob has not scheduled a run in thirty minutes — keyed on kube_cronjob_status_last_schedule_time, which persists with the CronJob object rather than being garbage-collected with each Job, so a suspended or stuck canary keeps alerting instead of self-silencing while a scheduling-but-failing run is caught by the failing alert. Login runs against MAS (the MSC3861 compat endpoint), sync and send against Synapse. Honest limits: this proves the delegation loop, not homeserver liveness — an alert that Synapse/Matrix itself is down cannot be delivered through Matrix (that dead-man concern belongs to an external heartbeat, a deployment-policy decision this canary does not ship); and deleting the CronJob object entirely removes its schedule metric, so full-removal is a git/Flux-visible change rather than an alert.

9.12 Sovereign alert delivery¶

The platform emits real, adoption-critical alerts — LLMTokenBurnHigh (the D7/D8 spend guard), TrivyImageVulnerabilityDrift, and the synthetic-canary alerts (§9.11) — but Alertmanager ships with only resource limits: no receivers, no routes, so every alert fires into the default null configuration. A sovereign adopter cannot default to PagerDuty/Slack SaaS; the platform already runs the natural on-call surface — a Matrix room.

The local and gcp overlays can opt into delivery (alert_delivery: enabled in platform-settings); it is disabled by default and structurally absent in demo/federation (their alert-delivery Kustomization resolves to profiles/disabled, which prunes to zero, and they delete the whole observability layer, so with the profile off Alertmanager keeps its null posture). Fgentic OWNS the routing contract and the deny-by-default posture toward agents; it COMPOSES a self-hosted receiver — a small standard-library Python webhook handler in the already-pinned python:3.14-slim image (no external or copyleft image, no SaaS), hardened like the tracing stack (non-root, read-only root filesystem, no service-account token, digest-pinned) with a default-deny NetworkPolicy that admits only the Alertmanager webhook (ingress from monitoring) and Synapse (egress to matrix). An AlertmanagerConfig (adopted globally via alertmanagerConfigMatcherStrategy: None) routes by the conventional severity label with sane group_by/group_wait/repeat_interval and an explicit catch-all, so an alert storm becomes a bounded stream and no alert is silently dropped.

Delivery goes to a dedicated private ops room as a plain @alertbot user — not an agent ghost: it is outside the bridge's @a2a-bridge/@agent-.* exclusive namespace and in no agent's allowedSenders, so an alert can never invoke an agent (D7/D8 — no alert-storm → LLM amplification), asserted offline by check:alert-delivery. Payloads are content-free by construction — only the alert name, severity, firing/resolved status, a bounded set of low-cardinality labels (namespace, job, …), a count, and the generator link; never alert annotation prose, Matrix event content, prompts, or secrets. The @alertbot access token lives in the per-cluster SOPS alertbot-credential Secret; the receiver holds no other credential. Honest limit: an alert that Synapse/Matrix itself is down cannot be delivered through Matrix — that dead-man concern belongs to an external heartbeat, a deployment-policy decision this path does not ship.