Day-2 Operations Handbook

This handbook is the operator entrypoint after Production Installation. It connects Fgentic's GitOps composition to the upstream components it assembles, while keeping three kinds of evidence separate:

1. Declared posture is what the reviewed Git revision and effective overlay render.
2. Control evidence is what the repository's static and isolated test gates prove.
3. Operational evidence is what the target cluster actually observed under its traffic, capacity, failure, restore, and upgrade conditions.

Do not promote one kind into another. Rendered replicas do not prove availability, a successful answer does not prove authorization, a backup object does not prove restoration, and a quota or token reservation is not measured consumption.

1. Establish the operating baseline

Record the following together for every production change and incident:

• Git revision, cluster overlay, effective platform settings, component/chart versions, and bridge image digest;
• first non-Ready Flux Kustomization or HelmRelease, affected namespace/workload, and event timestamps;
• relevant content-free metric windows, alert state, and the exact validation or recovery procedure run;
• owner, decision, rollback boundary, and any evidence that was not collected.

Flux owns production reconciliation. Inspect the dependency DAG before the workloads it creates:

flux get kustomizations
flux get helmreleases -A

Debug the first non-Ready layer rather than applying around it with kubectl apply or helm upgrade. The Flux troubleshooting guide is the component-level reference; Production Installation documents Fgentic's layer order and target-cluster acceptance gates.

2. Read the service signals

Grafana exposes the bridge and LLM dashboards at grafana.<server_name>. Prometheus scrapes the bridge, agentgateway, CloudNativePG, and the other configured monitors. Start from the symptom, then correlate the smallest relevant signals:

Question	Primary signal	Interpretation boundary
Are delegations completing?	fgentic_delegations_total{ghost,outcome}	Outcomes include success and explicit failure states; the counter is not a unique-human-task count.
Is bridge work accumulating?	fgentic_queue_depth, fgentic_inflight_delegations	A queue increase can mean downstream latency or insufficient admitted capacity; it is not permission to increase limits blindly.
Is durable recovery healthy?	fgentic_delegation_ledger_transitions_total{from_state,to_state}, fgentic_delegation_recovery_outcomes_total{outcome}	ambiguous is a deliberate terminal safety result. Never resend it automatically.
Are retries being suppressed?	fgentic_dedup_skips_total	Correlate with Matrix transaction handling before calling repeated user messages duplicates.
How long does initial delegation take?	fgentic_a2a_request_seconds	Measures SendMessage; long-task GetTask polling is outside this histogram.
How many model tokens were reported?	agentgateway_gen_ai_client_token_usage_sum by provider, model, route, and token type	Provider-reported token telemetry is aggregate usage, not an invoice, per-user attribution, or a federation reservation.
Is recent aggregate token burn above the configured warning budget?	LLMTokenBurnHigh and system_model_token_type:agentgateway_gen_ai_client_token_usage_sum:increase15m	The alert sums the 15-minute increase and waits 5 minutes. It is a warning detector, not a hard token or currency limit.

The LLM dashboard's catalog panel reports whether a provider/model rate lookup exists; it does not establish a bill. D7 sender/room rate limits and queue bounds constrain invocation pressure. Cross-organization maxTokens reserves admission capacity per verified client; actual model telemetry remains aggregate. See Observability §9 for metric labels, dashboard panels, privacy limits, and the token-cost boundary, and the agentgateway observability reference for the upstream telemetry surface.

2.1 Triage the nightly smoke

The default-branch nightly smoke workflow is a hosted, provider-free composition signal. Its three jobs deliberately keep running long enough to preserve independent outcomes and diagnostics:

Job	Boundaries exercised	Evidence limit
Small-profile mention to reply	demo:up, one Matrix → bridge → agentgateway → kagent reply, live namespace quota accounting, and ownership-guarded demo:down	Proves the disposable demo on that hosted runner and Git revision; it is not production, GKE, provider, or availability evidence.
Authorization, API audit, traces, and NetworkPolicy conformance	A2A authorization, governed MCP quota behavior, OTLP traces, kind/Calico NetworkPolicies, ResourceQuota enforcement, and Kubernetes API audit rotation	Each sub-check has its own outcome. One passing deny path does not prove its paired allow path, and an isolated cluster does not report target-cluster state.
Continuous image vulnerability scan	The reconciled Trivy Operator report and alert boundary in a disposable k3d cluster	Proves that the pinned scanner composition detects the fixture under test; it does not prove that every deployed image or target cluster is vulnerability-free.

Diagnostic steps use continue-on-error so later checks, teardown, and artifact upload still run. The Actions UI can therefore show a check mark beside a step whose command failed: treat the final job conclusion and the aggregate Enforce ... result step as authoritative. Those enforcers read steps.<id>.outcome, so failure, cancelled, or skipped cannot silently become success.

On the default branch, the final report job opens or refreshes the single nightly-smoke tracker with the demo, policy, and scanner job conclusions. A fully green run comments with the recovery run and closes it. A feature-branch dispatch exercises the same jobs but cannot mutate that repository-wide tracker.

Inspect one run without any kubeconfig, cluster, provider credential, or local runtime:

smoke_run="$(gh run list --repo fmind-ai/fgentic --workflow smoke.yml \
  --branch main --limit 1 --json databaseId --jq '.[0].databaseId')"
gh run view "${smoke_run}" --repo fmind-ai/fgentic \
  --json conclusion,headSha,jobs,url \
  --jq '{url, conclusion, headSha, jobs: [.jobs[] | {
    name, conclusion,
    failed_steps: [.steps[] | select(.conclusion == "failure") | .name]
  }]}'
gh run view "${smoke_run}" --repo fmind-ai/fgentic --log-failed
gh run view "${smoke_run}" --repo fmind-ai/fgentic --log

The lookup selects the latest main run. If the tracker references an earlier failure, assign the numeric ID from that run URL instead. Confirm headSha before attributing a failure to current main; an older scheduled run or branch dispatch may exercise different source. --log-failed exposes the aggregate enforcer that failed, while --log also includes the underlying continue-on-error output that explains which contract failed. Then download the retained failure bundles locally:

smoke_run="$(gh run list --repo fmind-ai/fgentic --workflow smoke.yml \
  --branch main --limit 1 --json databaseId --jq '.[0].databaseId')"
artifact_dir="$(mktemp -d)"
gh run download "${smoke_run}" --repo fmind-ai/fgentic --dir "${artifact_dir}"
rg --files "${artifact_dir}"

When the final upload step runs and finds diagnostic files, it retains demo-diagnostics-<run>-<attempt>, policy-diagnostics-<run>-<attempt>, or trivy-diagnostics-<run>-<attempt> for 14 days. Cancellation, timeout, runner loss, or an early failure with an empty diagnostics directory can leave no bundle; use the hosted full log in that case. Read the smallest relevant result or overview first, then the matching event, description, or log. Keep the bundle local; before quoting evidence in a public issue, remove room content, prompts, credentials, tokens, personal data, and unrelated workload logs.

Route the root cause, not merely the red job. Action setup, permissions, outcome projection, artifact retention, and tracker behavior belong to area/ci. Demo lifecycle sources and composed manifests belong to their runtime area; policy failures belong to the enforcing security or infrastructure surface; scanner composition belongs to its infrastructure/security owner. Record the exact head SHA, failed contract, and artifact filename in the handoff. Do not weaken an enforcer, suppress a negative control, or rerun until the first failure evidence disappears.

3. Scale without changing semantics

Only clusters/gcp composes infra/production-ha/cluster. The local, demo, and federation profiles intentionally do not. Use the component table and evidence limits in Production Installation: Availability posture before changing capacity.

1. Measure normal traffic, a representative rollout, and the intended failure condition. Retain headroom for scheduling and recovery.
2. Change workload values or the three namespace quota profiles in clusters/<env>/platform-settings.yaml; do not patch generated Deployments or individual ResourceQuota objects.
3. Render and validate the effective overlay. A ResourceQuota is a ceiling, not a reservation or measured target.
4. Reconcile through Flux and verify placement, readiness, disruption behavior, and queue/latency signals on the target cluster.
5. Record the observed scope. The default two-node, one-zone GKE reference demonstrates host separation, not zone or region availability.

The Matrix-to-A2A bridge is intentionally one ready intake replica. Its chart rejects replicaCount values other than 1 to preserve per-room ordering. On graceful termination it stops intake and claims, closes transaction connections, and drains active leases for up to 25 seconds within a 45-second Pod grace period; nonterminal canceled work stays recoverable. mise run test:availability proves that graceful SIGTERM path and deduplication with a replacement process. It does not prove SIGKILL, node loss, automatic Synapse retry, or a production RTO.

mise run test:crash-recovery separately exercises twelve persisted job/control SIGKILL boundaries. If A2A send, cancel, or continuation transport may have started but returned no acknowledgement, recovery records ambiguous and does not resend: deterministic message IDs do not establish target idempotency. Matrix question/progress/pin projections instead converge through deterministic transaction IDs and current room state. This is at-most-once recovery at the uncertain A2A boundary, not distributed exactly-once delivery. Run target-specific drills before setting an SLO or RTO.

4. Back up and restore the state and its keys

4.1 Current declared posture

Scope	Checked-in posture	Loss blast radius and boundary
GCP platform-pg	CloudNativePG native barmanObjectStore to GCS through Workload Identity; continuous WAL, 30-day retention, and platform-pg-nightly at 03:00 daily with an immediate first base backup	This is the repository's current implementation. Upstream has deprecated native Barman Cloud configuration in favor of its plugin; #170 owns that migration.
Local platform-pg	The overlay removes the GCS backup block, service-account template, and ScheduledBackup	Local development does not inherit a production backup merely because the base contains one.
GCP Synapse media	ESS retains ess-synapse-media on standard-rwo; fgentic-synapse-media retains PD CSI snapshot contents	No snapshot is created merely because the class exists. #64 owns the coordinated database+media restore evidence.
Local Synapse media	ESS retains ess-synapse-media on local-path	k3d local-path has no snapshot boundary. This is development persistence, not a backup or production recovery claim.
SOPS and age recovery key	Ciphertext is committed under clusters/<env>/secrets/*.sops.yaml; the matching age private key remains offline and is installed as cluster-local sops-age material	A database backup contains neither Kubernetes Secrets nor the age private key. Losing the private key with the cluster makes every committed ciphertext unusable.
Synapse synapse-signing-key	identity-keys.sops.yaml supplies ESS synapse.signingKey; fresh generation happens once, and existing clusters use scripts/escrow-identity-keys.sh	Loss changes the homeserver's federation identity, invalidates partner-pinned trust, and prevents verification of events signed only by the lost key.
MAS mas-encryption-secret	The same SOPS Secret supplies ESS matrixAuthenticationService.encryptionSecret	Loss strands encrypted values in a restored MAS database; restoring database bytes alone is insufficient.
MAS RSA and P-256 private keys	The same SOPS Secret supplies ESS matrixAuthenticationService.privateKeys.{rsa,ecdsaPrime256v1}	Loss changes MAS signing identity and invalidates outstanding signed tokens or sessions. The keys are restored with, not regenerated after, the MAS database.
Other ESS-generated shared/random secrets	Synapse macaroon/registration secrets and the MAS↔Synapse shared secret remain ESS-managed	They are availability/session or paired-authentication credentials, not federation identity or database decryption roots. Recover them by a deliberate coordinated rotation; never regenerate only one consumer.
Keycloak, kagent, agentgateway, and bridge	Keycloak realm keys and runtime sessions live in their backed-up scoped databases; deployment credentials and every bridge/appservice pair are SOPS-managed	Database recovery and the matching SOPS revision are both required. No additional in-cluster-only federation identity or encrypted-data root was found in this inventory.

CloudNativePG point-in-time recovery consumes a valid base backup and WAL archive into a new Cluster; it is not an in-place rewind. Follow the CloudNativePG recovery reference for the pinned operator version and review the recovery manifest before reconciliation.

4.2 Backup checks

1. Require the CloudNativePG Cluster and the latest Backup objects to report success, and confirm WAL archival remains current.
2. Verify retention and object-store access from the intended Workload Identity; never copy a provider key into a Pod as a shortcut.
3. On GKE, require ess-synapse-media to use standard-rwo, the PD CSI addon and snapshot CRDs to be available, and the selected media VolumeSnapshot to report readyToUse: true before treating it as a recovery point.
4. Preserve the exact SOPS ciphertext revision and test access to the original age recovery key through the organization's approved offline procedure.
5. Store the age private key and the ability to retrieve identity-keys.sops.yaml in the same offline recovery posture, but as independently controlled material. Test decryption without printing key values.
6. Alert on failed or stale backup evidence. Do not infer restorability from object presence alone.

4.3 Adopt identity-key escrow without rotation

Fresh clusters get identity-keys.sops.yaml from scripts/gen-secrets.sh; the file is bootstrap-only and survives --force and every rotate-secrets.sh set. For an existing cluster, do not run the generator first. Export the live ESS values through an explicit kubeconfig:

k3d kubeconfig get fgentic >"$(pwd)/.agents/tmp/fgentic-kubeconfig"
scripts/escrow-identity-keys.sh fgentic.localhost local "$(pwd)/.agents/tmp/fgentic-kubeconfig"

The migration reads matrix/ess-generated, encrypts all four values directly into the environment's SOPS set, verifies byte equality, and prints only the Synapse signing-key SHA-256 fingerprint. Review and commit the ciphertext plus generated kustomization.yaml, then deploy the HelmRelease references. Before and after reconciliation, hash SYNAPSE_SIGNING_KEY from the live Secret and query /_matrix/key/v2/server; the key ID and public key must be unchanged. Remove the temporary kubeconfig when the comparison is recorded.

Recovery and rotation are separate operations. Recovery restores the original ciphertext before databases or identity consumers start. scripts/rotate-secrets.sh deliberately has no identity-key set; changing a Synapse federation key or MAS signing/encryption root requires its own protocol-aware, live-first ceremony and must never be a side effect of database-password or bulk credential rotation.

4.4 Isolated restore drill

Run the database drill quarterly and after any backup backend, PostgreSQL image, CNPG operator, or recovery-policy change. Use a reviewed UTC recovery point and retain the JSON evidence with the operations record:

scripts/restore-drill.sh \
  --context <explicit-approved-context> \
  --target-time 2026-07-31T12:00:00Z \
  --mas-url https://matrix.restore.example \
  --matrix-access-token "$RESTORE_TEST_ACCESS_TOKEN" \
  --room-id '!restore-fixture:fgentic.example' \
  --evidence restore-drill-2026-Q3.json

The command applies a separately named CNPG recovery Cluster in restore-drill, waits for PITR, requires user tables in the synapse, mas, bridge, and kagent databases, verifies MAS discovery and authenticated room history, and records consistently shaped fgentic.restore-drill.v1 timing evidence. Re-running the same command converges the namespace and recovery manifest. Changing an immutable recovery source or target requires explicit scratch-Cluster deletion; never point the script at the source Cluster name. The access token is used only in memory and is excluded from the evidence.

1. Choose a reviewed recovery point and create a separately named CloudNativePG recovery Cluster in an isolated namespace or cluster. Do not overwrite the production Cluster.
2. Upload a deterministic media fixture and record its content hash. Quiesce Synapse media writes, create an explicitly named VolumeSnapshot of ess-synapse-media with class fgentic-synapse-media, wait for readyToUse: true, and record the database and media recovery timestamps. A retained class without a ready snapshot is not a backup.
3. Restore the exact committed SOPS ciphertext with the original age key before starting Synapse, MAS, or other consumers. scripts/gen-secrets.sh creates new credentials only when their bootstrap-only files are absent; it is not recovery of the old credential set.
4. Provision a new media PVC from that exact snapshot and attach only the isolated Synapse recovery instance. Verify the fixture downloads with the original hash; do not infer media recovery from a bound PVC alone.
5. Verify every expected database, scoped role, collation requirement, extension such as pgvector, ownership grant, and HBA boundary before connecting consumers.
6. Verify Matrix, MAS, bridge durable state, Keycloak, kagent, knowledge, and every enabled optional bridge against the recovered credential pairs. Appservice, database, OIDC, A2A, and MCP copies must remain coherent.
7. Compare the pre-loss Synapse signing-key endpoint with the rebuilt endpoint, verify MAS can decrypt its restored database by completing an existing-user login/session flow, and verify the bridge registration copies still match. A same-cluster scratch namespace is not full-loss evidence.
8. Run positive and negative service checks, then record the observed recovery point, data loss, duration, revision, and deviations. Only those observations may become RPO/RTO evidence.
9. Destroy the isolated recovery environment only after its evidence and retention decision are recorded.

This drill validates the database recovery layer, not complete identity-key recovery. Synapse signing and MAS key-material escrow remain tracked by #458; until that work is complete, supply the original isolated identity keys through the approved recovery procedure before starting consumers. Do not substitute scripts/gen-secrets.sh, which creates new credentials rather than recovering coherent old ones.

Use scripts/rotate-secrets.sh and the SOPS secrets runbook for an intentional coherent rotation after recovery. Never regenerate one half of a paired credential or place a model credential in an Agent.

5. Respond from the signal, not the guess

Signal or symptom	Enforcing control and file	Diagnose and contain	Do not claim or do
LLMTokenBurnHigh	The threshold and recording rule live in cost-alert.yaml; bridge sender/room enforcement is in handler.go and its bounded configuration in values.yaml (D7/D8).	Break down token growth by provider/model/route/token type and correlate bridge outcomes and queue. Stop a loop at the sender/room allowlist or Agent mapping, retain rate/queue bounds, and correct the reviewed route or budget through Git.	Do not call the threshold currency spend, per-user usage, or a hard cap. Do not raise it merely to silence the alert.
Rising queue/inflight or delegation latency	Bridge gauges and histograms are declared in metrics.go; production scaling composition is infra/production-ha/.	Check downstream A2A, agentgateway/model, Postgres, and Matrix before capacity. Separate SendMessage latency from polling; remove the bottleneck or reduce admitted pressure, then scale only a component whose protocol and measured headroom support it.	Do not increase every quota or bridge replicas.
Durable ambiguous, dead, or recovery failures	The durable state machine and fail-closed uncertain boundary are in durable_job.go; transitions and recovery results are exported by metrics.go.	Inspect the affected content-free job identity and persisted transition. Preserve evidence, resolve downstream state explicitly, and use a new user-authorized request if another action is required.	Never resend ambiguous work automatically or relabel it success to clear a dashboard.
Model/A2A backend down mid-delegation	Bounded retry plus the §6.1 catalog notice in durable_job.go (retryOrDead, DELEGATION_MAX_ATTEMPTS, capped backoff) and failure_message.go; exercised by mise run test:model-outage.	The user receives the §6.1 "could not be recovered after repeated failures" notice; retries are bounded to DELEGATION_MAX_ATTEMPTS with capped backoff and no model spend reaches the unreachable backend; a new mention succeeds once the backend returns. Recovery needs no bridge action — investigate and restore the backend.	Do not treat the notice as data loss or resend it automatically. The drill is a local kind fixture injecting a fast connection failure; it is not a model-provider SLA or availability guarantee.
Synapse homeserver restarts with a task in flight	Postgres-backed durable delegation with deterministic Matrix transaction IDs in durable_job.go and appservice-transaction replay dedup; exercised by mise run test:synapse-restart.	The in-flight task's reply is delivered exactly once after the homeserver returns, appservice-transaction replay dedups cleanly, and there is no duplicate A2A execution and no silent forever-⏳. Wait for the homeserver to become ready; the bridge reconnects and completes.	Do not resend or restart the in-flight task while the homeserver is down. The drill exercised a single-replica Recreate restart on a local kind fixture (~22s observed down window); it is not a node/zone-loss, RTO, or HA proof.
Suspected prompt injection or tool abuse	Per-Agent full-MXID allowedSenders, allowedServers, and MCP policy are parsed in config.go; Security §7 maps the room, NetworkPolicy, gateway, and tool enforcement points.	Treat Matrix content and external artifacts as untrusted. Review room membership, sender/server rules, exact Agent mapping, MCP subset, downstream identity, and egress. Disable only the compromised route, rotate exposed credentials coherently, and retain content-free evidence.	Do not paste room content, prompts, credentials, or private artifacts into public issues or support bundles.
SSO/identity plane locked out (Keycloak down)	The SSO-only flag mas_local_login_enabled in platform-settings.yaml and the disabled-by-default recovery credential break-glass-recovery, emitted only by FGENTIC_SECRET_SET=break-glass in gen-secrets.sh and gated fail-closed by mise run check:break-glass; the break-glass ladder is the runbook.	Administer through the lowest sufficient rung: the MAS Admin API (client credentials) for user/session actions, a scoped MAS compatibility token for Synapse admin actions, and only as a last resort the deliberate GitOps flip of mas_local_login_enabled plus the pre-provisioned recovery admin whose password lives only in clusters/<env>/secrets/break-glass-recovery.sops.yaml. Break-glass admin actions and the recovery login are audited content-free per audit.md. Close the window by reverting the flip and re-disabling the account.	Do not leave mas_local_login_enabled: "true" or the recovery account enabled after the window; do not treat the least-audited MAS Admin API rung as a fully attributed trail; do not store a standing superuser credential in git.
Federated peer unexpectedly allowed or denied	Closed homeserver allowlists are in infra/federation/matrix-a/; the room-v12 m.room.server_acl constructor is federation-matrix.sh, and callback policy reload is owned by federation.sh (D6).	Check the ACL, closed-federation allowlist, callback policy version, Signed AgentCard pin, JWT client, and exact route independently. Revoke or restore the narrow plane through Git/Flux and use Federation Offboarding when trust is withdrawn.	fed:policy-reload is an owned disposable-lab proof that restores deny; it is not a production mutation command. Reachability alone is not governance acceptance.
One local agent must be silenced immediately	The fail-closed mapping/sender re-read before spend is in handler.go/config.go (docs/bridge.md §6); the full lifecycle is Runbook: retire an agent in the matrix-agents skill.	Emergency fast path: in Git, remove the agent's agents.yaml mapping or narrow its allowedSenders to empty, commit, and flux reconcile kustomization bridge. The next atomic reload refuses the agent before any A2A spend with no pod restart. A plain mention is then silently dropped (removal) or logged as sender not allowed to invoke agent without a room reply (emptied allowlist); verify by the absence of a reply and any A2A call in the bridge logs / delegation audit and by !agents no longer offering it — not a visible refusal notice. Capture the Git revision, reconcile time, and content-free per-step timestamps. Follow with full room-leave and session purge as deliberate retirement.	Do not kubectl edit the Flux-owned bridge Deployment or mapping to silence an agent; do not treat mapping removal as data deletion (Matrix history and federated copies remain).
Flux object not Ready	The ordered production DAG is clusters/base/; Flux Kustomization dependencies and health checks are the reconciliation enforcement point.	Find the earliest failed source/Kustomization/HelmRelease and inspect its conditions/events. Correct the source revision, dependency, substitution, secret, or health check in Git, then reconcile and observe the DAG.	Do not bypass Flux with a manual production apply or upgrade.
Backup failed or restore point is stale	GCS/WAL retention is in cluster.yaml, and the base-backup schedule is scheduledbackup.yaml (D12).	Inspect Backup and WAL archival state plus object-store identity and retention. Restore archival, take a valid backup, and run the isolated recovery drill before revising recovery claims.	Do not delete the last valid chain or claim a backup success proves restore.
TrivyImageVulnerabilityDrift	The digest-scoped recording and alert rules are in trivy-alert.yaml; the reconciled scanner boundary is infra/trivy-operator/.	Inspect report age, affected digest/workload, severity, fix availability, and signed source image. Upgrade the reviewed pin through the supply-chain gates or document the owned exception.	The alert measures report-count drift after its window; it is not proof of exploitability or complete risk.

Security vulnerabilities use the private path in SECURITY.md, never a public issue. For component failures, preserve the minimum content-free reproduction and route it according to the ownership matrix below.

6. Upgrade and roll back through Git

1. Read the owning topic spec, stability contract, release notes, and current upstream compatibility guidance. Stable public-surface breaks require the documented deprecation/upgrade path; settled design changes require an ADR.
2. Confirm a valid recovery point, SOPS-key recovery, scheduling headroom, and an explicit rollback revision before changing pins.
3. Change the canonical chart/image/manifests in one reviewable Git change. Run focused render/schema tests, then the repository check and test gates sequentially.
4. Reconcile through Flux and inspect the first non-Ready layer. Run component smoke, negative-control, data, metric, and representative rollout checks on the target cluster.
5. Roll back by reverting the reviewed Git change and reconciling. Do not hand-edit a HelmRelease, Deployment, or digest to create an unrecorded state.

The following pins are coupled operational contracts, not independent search-and-replace values:

• Gateway API v1.5.1 experimental, Traefik chart 41.0.2 (proxy v3.7.6), and agentgateway v1.3.1 are the current supported overlap: Traefik and agentgateway both support Gateway API 1.5, while agentgateway 1.3 still watches TCPRoute v1alpha2. Gateway API v1.6 deprecates rather than removes that API version, but remains outside agentgateway 1.3.x's supported range; move the coupled pins together.
• kagent charts come from a HelmRepository with type: oci; do not replace it with an OCIRepository, because Flux digest build metadata then enters labels generated by kagent.
• Bridge CD builds, scans, signs, and commits the immutable multi-architecture image digest into its HelmRelease. Do not edit that digest by hand or deploy latest.
• After recreating the full local cluster, run mise run cluster:overrides to restore the gitignored platform-settings-overrides ConfigMap when present.

Use the kagent debug and operational considerations references for kagent-owned behavior; Fgentic's rendered compatibility and negative-control gates remain required around it.

7. Own the composition; escalate component internals

Surface	Fgentic/operator ownership	Component escalation boundary
GitOps and environment contract	Overlay composition, substitutions, DAG dependencies, reviewed revision, SOPS integration, and acceptance evidence	Flux source/reconcile/controller internals after the composition and object evidence are isolated
Matrix collaboration	Room and appservice policy, homeserver-neutral bridge contract, identity mapping, federation controls, and deployment evidence	ESS/Synapse/MAS/Element internals with the exact pinned version and sanitized reproduction
Delegation	Bridge lifecycle, durable ledger, allowlists, A2A route policy, remote-card pin, and fail-closed results	kagent or A2A SDK/runtime internals after the bridge boundary is isolated
Model path	agentgateway route, credential boundary, NetworkPolicy, provider inventory, token telemetry, and budget warning	agentgateway or selected model-provider behavior after route-level evidence is isolated
Data and recovery	CNPG composition, scoped roles/databases, recovery manifests, SOPS pairing, restore drills, and recorded RPO/RTO	CloudNativePG/PostgreSQL internals with backup/WAL/operator evidence and no secrets
Security and supply chain	Admission policy, least privilege, signed/digest-pinned artifacts, vulnerability response, and private disclosure	Upstream vulnerability or component defect after Fgentic containment remains in force

The operator remains accountable for the assembled service even when an upstream component owns the defect. Escalation never means disabling Fgentic's negative controls, publishing secrets/content, or claiming upstream acceptance before it occurs.

8. Operate the public demo

The public reference deployment is a Matrix homeserver on the open internet running an agent execution plane. The 2026 cautionary case is instructive rather than abstract: an exposed self-hosted agent surface reached six figures of internet-reachable instances and a critical CVE in the same year. This section is the day-2 contract for fgentic.fmind.ai, and every control below is a Git change reconciled by Flux, never a console action. The full moderation stack (#136) is v1.1; the demo does not wait for it, so the controls here are the ones that work without it.

8.1 Registration policy

The public profile is closed by construction. clusters/gcp/platform-settings.yaml sets mas_local_login_enabled: "false", so MAS serves no local password login and the only way in is the reviewed upstream IdP. Nothing about the demo requires self-service registration, and the demo user is provisioned by the seeding path rather than by a public sign-up form.

Before public DNS goes live, verify all four, and treat a failure as a release blocker rather than a follow-up:

1. mas_local_login_enabled is "false" in the deployed overlay, and the reconciled MAS configuration reflects it.
2. The upstream IdP's own registration is closed or invite-only. MAS delegates identity; an open IdP makes an open homeserver regardless of what MAS does.
3. Guest access is off and the homeserver is not published in any public room directory.
4. Federation remains a closed allowlist. Deny-by-default is what keeps an open homeserver from becoming an open relay; see federation §8.

Changing any of these is a pull request against the overlay, so the policy has a reviewable history. There is no supported runtime toggle.

8.2 Abuse reports and takedown without the moderation stack

Publish a contact route before the deployment is reachable: the security contact in SECURITY.md is the reporting address for demo abuse as well, and reports also arrive through the Matrix client's built-in report action, which Synapse records for the server administrator.

The takedown ladder, narrowest first:

1. Redact the events. Redaction is a Matrix operation available to a room moderator; it removes the content while the redacted event itself remains part of history. Retention of the original content behind it follows retention.
2. Lock or deactivate the account. ESS delegates authentication to MAS under MSC3861, so use mas-cli manage lock-user "<localpart>" --deactivate and never Synapse's legacy deactivate endpoint. Read retention §"Deactivation" first: deactivation blocks login and revokes tokens, but has documented residuals and does not delete uploaded media.
3. Remove the participating server. For abuse arriving over federation, drop the server from the allowlist and add a server ACL. This is a Git change, so it is reviewable and revertible, and it takes effect through reconciliation rather than a live edit.
4. Withdraw the deployment. The demo is rebuildable from Git in one pass (§8.5), so taking it down is a legitimate response to an incident rather than a last resort.

Agent-specific abuse has one extra lever the moderation stack does not own: a mapped ghost's sender policy and room binding live in the bridge's agents.yaml. Removing a ghost from a room binding stops delegation into that room immediately on the next reload, without touching the room's human membership.

8.3 Credential and token rotation

Every demo credential is SOPS-age encrypted and cluster-specific. scripts/rotate-secrets.sh performs selective rotation and mise run check:secret-rotation proves the cross-consumer invariants offline, so a rotation can be rehearsed before it is performed.

Material	Cadence	Note
Demo user password	Every rebuild	Generated per install and never reused; a rebuild is the rotation.
Appservice as_token/hs_token	Quarterly, and on exposure	Rotate both halves together — Synapse and the bridge must agree or delegation stops.
Database roles	Quarterly, and on exposure	Per-service scoped roles; rotate one service at a time and confirm its Kustomization is Ready before the next.
Model provider credential	Per provider policy	Held only at agentgateway. No agent and no bridge holds it, so the blast radius is one component.
SOPS age recipient	On operator change	Re-encrypting every *.sops.yaml for the new recipient is the whole operation; it is not a per-secret change.
AgentCard signing key	On exposure, and per pin	A partner pins the public key, so rotation is bilateral and must follow the overlap window in federation.

Rotate on suspicion, not only on schedule. An unexplained delegation in the audit log is grounds to rotate the appservice tokens the same day.

8.4 Uptime posture and outage communication

State the posture plainly rather than implying an SLA. The demo is a single-region, best-effort evaluation deployment with no availability commitment, run to let people try the product and to prove the manifests. It is not a service anyone should depend on, and it carries no support obligation.

What that means concretely: the bridge runs one ready intake replica by design, so a bridge restart is a short delegation gap rather than a transparent failover; durable work survives it through the ledger, but intake does not continue during it. A cost-reduced profile drops the HA composition entirely, which trades availability for spend deliberately.

Communicate outages where people already are: a notice in the demo room, and the repository's issue tracker for anything lasting beyond a restart. If the deployment is withdrawn rather than repaired, say so on the front page rather than leaving a dead link — a stale "reference deployment" claim is the exact failure #1611 removed.

8.5 Teardown and rebuild from Git in one pass

The rebuild drill is the control that makes every other control above affordable, because it makes withdrawal cheap. It is also the fastest way to rotate everything at once.

1. Confirm the current revision is reconciled and record it, so the rebuild has a known target.
2. Destroy the deployment.
3. Re-apply from the same revision and let Flux reconcile, then re-seed.
4. Verify the acceptance path end to end: a mention in the demo room produces an agent reply, and the audit record for it is content-free.
5. Confirm no credential survived the cycle — every generated secret must differ from the recorded pre-teardown values.

Run this drill before public DNS and after any change to the overlay's composition. The cost model and the cost-reduced overlay it depends on are tracked in #1649 and #1648.

Publication status

This handbook is repository source, not proof of publication or a completed production drill. Delivery through the planned documentation site remains tracked by #72. The M31 tenancy epic #384 checklist update remains a separate lane-owned repository action; this source change does not claim it.