feat(infra): retire fc-redis bootstrap after Redis Manager adoption

Per Q-SO-1 operator resolution 2026-05-11 PM, Redis SignalR backplane lands
in Phase A (was Phase C deferral). Treats Redis as a managed FC infrastructure
component, not a deferred scaling escalation.

Lands the minimal Phase A surface:
- Namespace fc-redis
- Single Redis 7-alpine pod with 1Gi Longhorn RWO PVC
- ConfigMap with AOF persistence (everysec), 256Mi maxmemory, allkeys-lru
- ClusterIP Service `redis.fc-redis.svc.cluster.local:6379` (in-cluster only)
- No AUTH Phase A (Phase B add via 1Password Connect rotation)
- No IngressRoute (backplane is server-to-server)

Consumers (Phase A IMPL across FC services) add:
  services.AddSignalR().AddStackExchangeRedis(
      "redis.fc-redis.svc.cluster.local:6379",
      opts => opts.Configuration.ChannelPrefix =
          StackExchange.Redis.RedisChannel.Literal("fc-opsconsole"));

Phase B/C follow-ons (not in this commit): Sentinel for HA, AUTH password
from 1Password, redis_exporter sidecar for Prometheus, network policies.

See FlowerCore.Notes/docs/signage/operations-console-phase-2-design.md
section 3.5 (rewritten) and decisions-waiting.html Q-SO-1 (RESOLVED).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

apps/fc-updater/fc-updater.yaml

@@ -58,7 +58,7 @@ spec:
       nodeName: rke2-server
       containers:
         - name: web
-          image: localhost/fc-updater-web:v20260508-pub3-deepening-2bdf108
+          image: localhost/fc-updater-web:v20260509-4162dca-authgate
           imagePullPolicy: Never
           ports:
             - containerPort: 8080

apps/kubevirt-vms/ci1.yaml

@@ -377,7 +377,22 @@ spec:
         firmware:
           bootloader:
             efi:
-              secureBoot: true
+              # 2026-05-08: SecureBoot=false during initial install. With SecureBoot
+              # enabled, OVMF's BdsDxe times out reading Boot0001 from the SCSI
+              # CDROM ("BdsDxe: failed to start Boot0001 ... Time out") before the
+              # EFI bootloader signature can verify against the OVMF VARS trust DB.
+              # KubeVirt's `/usr/share/OVMF/OVMF_VARS.secboot.fd` template doesn't
+              # appear to include the Microsoft KEK/DB by default, so signed
+              # Windows EFI bootloaders fail validation. Disabling SecureBoot lets
+              # OVMF skip the chain check and boot directly. This is acceptable for
+              # a CI runner — TPM 2.0 is still emulated (`tpm: {}` below) so
+              # BitLocker / Hyper-V / WSL still work.
+              # When the operator wants SecureBoot back, the path is:
+              #   1. Custom-build OVMF_VARS.fd with Microsoft KEK/DB enrolled
+              #   2. Mount it into the VM via firmware.bootloader.efi.persistent
+              #   3. Set secureBoot: true again
+              # Tracked separately from the install unblock.
+              secureBoot: false
         devices:
           tpm: {}             # Non-persistent vTPM — sufficient for runner; no BitLocker
           disks:
@@ -396,11 +411,18 @@ spec:
             # Confirmed via debug pod: PVC content IS a real bootable ISO9660
             # (file: "ISO 9660 CD-ROM filesystem data ... (bootable)"), so the
             # only bug was boot priority.
-            # 2026-05-08 PM: cdrom bus flipped sata→scsi for windows-iso to address
+            # 2026-05-08 PM: cdrom bus SCSI + containerDisk delivery. This
-            # the OVMF SATA-CDROM read timeout (`BdsDxe: failed to start Boot0001 ...
+            # combination boots qemu cleanly and reaches OVMF, but OVMF
-            # Time out`). The SCSI CDROM uses virtio-scsi controller which has a
+            # BdsDxe still hits "starting Boot0001 ... Time out" on the
-            # longer read window and works cleanly on Filesystem-backed PVCs.
+            # cdrom — see HANDOFF.md / CODEX-STATUS.md "OPEN — ci1" for the
-            # See diagnostic chain in HANDOFF.md / CODEX-STATUS.md "OPEN — ci1".
+            # full diagnostic chain. virtio-blk disk swap was attempted as a
+            # workaround but introduced a separate QEMU rootdisk flock issue
+            # without fixing the underlying OVMF cdrom problem; reverted.
+            # Operator decision needed for next architectural step (OVMF
+            # custom build with extended timeout, KubeVirt version bump,
+            # Hyper-V/VirtualBox-and-export, or BIOS legacy boot). The
+            # containerDisk distribution pipeline (build/save/scp/ctr import)
+            # is proven and ready to reuse for any of those.
             - name: windows-iso
               bootOrder: 1
               cdrom:
@@ -435,25 +457,40 @@ spec:
           persistentVolumeClaim:
             claimName: ci1-rootdisk
         - name: windows-iso
-          # 2026-05-08 PM: REVERTED from NFS Path B back to the original CDI
+          # 2026-05-08 PM (Path C, CONTAINERDISK): the ISO is now packaged as
-          # Longhorn Filesystem PVC. NFS Path B (commit fc2aca0) failed at the
+          # a KubeVirt containerDisk OCI image baked from
-          # storage layer because the Synology export `/volume1/ISOs` denies
+          # `FROM scratch ; ADD --chown=107:107 disk.img /disk/disk.img`.
-          # non-root client UIDs at the directory level (qemu uid 107 cannot
+          # The qemu user (uid 107) reads the ISO directly from a tmpfs view
-          # `ls /iso/` even with file mode 0777). Confirmed via uid-107 +
+          # of the OCI layer, bypassing both:
-          # uid-0 busybox probe pods on rke2-agent2 — same export-only-root
+          #   - Synology NFS export ACL (Path B failed: uid 107 denied at
-          # pattern as `/volume1/kubernetes` documented in
+          #     directory level even with mode 0777, see memory
-          # `feedback_synology_nfs_kubernetes_export_root_only`. Memory:
+          #     feedback_synology_iso_export_root_only_uid_107_denied)
-          # `feedback_synology_iso_export_root_only_uid_107_denied.md`.
+          #   - OVMF cdrom read-window timeout (Path A and Path B's SCSI
+          #     retry both hit `BdsDxe: failed to start Boot0001 ... Time out`
+          #     when the cdrom was backed by a PVC the storage controller
+          #     couldn't satisfy reads from fast enough).
           #
-          # The Longhorn PVC `windows-server-2025-iso` (CDI Filesystem mode,
+          # Image build (one-time, per ISO version):
-          # 10Gi) was confirmed to contain valid ISO bytes that uid 107 CAN
+          #   1. Copy ISO to disk.img, write Dockerfile
-          # read (mode 0660 root:107). The OVMF SATA-CDROM read timeout from
+          #   2. podman build --tag localhost/win-server-2025:1.0 .  (on noc1)
-          # the original Path A is now addressed by the `bus: scsi` swap on
+          #   3. podman save -o win-server-2025-1.0.tar localhost/win-server-2025:1.0
-          # the disks block above. The NFS PVC + PV are RETAINED on disk so
+          #   4. SCP tar to all 3 RKE2 nodes (rke2-server, rke2-agent1, rke2-agent2)
-          # the Path B state is recoverable; they can be pruned in a
+          #   5. sudo /var/lib/rancher/rke2/bin/ctr -a /run/k3s/containerd/containerd.sock \
-          # follow-up commit once SCSI boot is proven.
+          #        -n k8s.io images import /tmp/win-server-2025-1.0.tar
-          persistentVolumeClaim:
+          # Standard FC pattern per `feedback_rke2_localhost_imagepullpolicy`.
-            claimName: windows-server-2025-iso
+          #
+          # When a new Windows ISO version ships, bump the tag (1.1, 1.2, ...),
+          # rebuild + redistribute, and update the image: line below in a new
+          # commit. KubeVirt picks up the new image via a VM restart.
+          #
+          # The legacy NFS PVC + PV (apps/kubevirt-vms/win2025-iso-nfs-pv.yaml)
+          # and CDI Longhorn PVC (`windows-server-2025-iso`) are RETAINED for
+          # this commit so the prior states are recoverable. Once the
+          # containerDisk path proves on a successful Windows install, both
+          # legacy artifacts can be pruned in a follow-up commit.
+          containerDisk:
+            image: localhost/win-server-2025:1.0
+            imagePullPolicy: Never
         - name: virtio-drivers
           containerDisk:
             # Pinned to v1.8.2 (latest stable as of 2026-05-08).

apps/monitoring/noc-monitoring.yaml

@@ -974,6 +974,39 @@ data:
               summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
               description: "Spec wants {{ $labels.spec_replicas }} but only {{ $value }} available. Likely a rollout stuck on probe failure, scheduling, or PVC."
 
+          # Q-MR-3 (2026-05-11): multus memory pressure — catches the next OOM
+          # cascade BEFORE multus is OOM-killed cluster-wide. The 2026-05-10
+          # outage (21h) hit because no alert fired on the rising multus working
+          # set — only downstream blackbox / Traefik / service alerts. With
+          # 1Gi limit (bluejay-infra@eb8693e), 80% = ~800MiB; steady-state
+          # runs ~150-250MiB so this only fires when an avalanche starts.
+          - alert: MultusMemoryPressure
+            expr: |
+              container_memory_working_set_bytes{container="kube-multus"}
+                / container_spec_memory_limit_bytes{container="kube-multus"} > 0.8
+            for: 5m
+            labels:
+              severity: critical
+              alert_channel: thermal_print
+            annotations:
+              summary: "kube-multus memory >80% of limit on {{ $labels.node }} for 5m"
+              description: "kube-multus working set is {{ $value | humanizePercentage }} of its memory limit on node {{ $labels.node }}. If this keeps climbing, multus will OOM and all new pod networking will halt cluster-wide (precedent: 2026-05-10 outage)."
+
+          # Q-MR-3 (2026-05-11): namespace pending-pod backlog — catches the
+          # operator-leak avalanche pattern BEFORE it cascades into a multus
+          # CNI OOM. Any FC operator (RemoteDesktop / Distribution / WorldBuilder)
+          # emitting pods without ownerReferences will accumulate them when
+          # the operator crashes. >25 pending pods in any namespace for 30m
+          # is the signal to investigate the reconciler.
+          - alert: NamespacePendingPodBacklog
+            expr: sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 25
+            for: 30m
+            labels:
+              severity: warning
+            annotations:
+              summary: "Namespace {{ $labels.namespace }} has {{ $value }} Pending pods for 30m"
+              description: "Pending pod count in {{ $labels.namespace }} exceeds 25 sustained for 30m. Likely operator-leak avalanche pattern — children emitted without ownerReferences. Risk of multus CNI OOM cascade."
+
       # Longhorn storage health alerts. Required: longhorn scrape job
       # (added 2026-04-26 — see scrape_configs above). The K8s events
       # for "snapshot becomes not ready to use" are transient lifecycle

apps/multus/multus.yaml

@@ -188,13 +188,24 @@ spec:
         - name: kube-multus
           image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
           command: [ "/usr/src/multus-cni/bin/multus-daemon" ]
+          # 2026-05-11: upstream default of 50Mi memory limit OOM-cascades when
+          # an operator-owned namespace accumulates >100 pending pods retrying
+          # CNI ADD. RemoteDesktop emitted 219 orphan rd-browser-only pods
+          # (missing OwnerReferences), kubelet's CNI ADD avalanche pushed multus
+          # over 50Mi, OOMKilled, restarted with even bigger backlog → loop.
+          # 21h cluster outage. See FlowerCore.Notes:
+          #   feedback_multus_50mi_limit_oom_orphan_pod_avalanche.md
+          # 1Gi limit / 512Mi request comfortably handles a 200+ pod CNI
+          # catchup burst on 64GB nodes (nodes are <25% used in steady-state).
+          # Drop back toward 256Mi only after MultusMemoryPressure alert
+          # proves steady-state working set sits well below 200Mi.
           resources:
             requests:
               cpu: "100m"
-              memory: "50Mi"
+              memory: "512Mi"
             limits:
               cpu: "100m"
-              memory: "50Mi"
+              memory: "1Gi"
           securityContext:
             privileged: true
           terminationMessagePolicy: FallbackToLogsOnError

apps/telephony/telephony.yaml

@@ -127,10 +127,13 @@ spec:
       initContainers:
         - name: fix-data-perms
           image: busybox:latest
-          # Also chown /shared-tts (hostPath /tmp/tts-audio) so the non-root
+          # Must run as root to chown the hostPath /tmp/tts-audio that may be
-          # app user (uid 1654) can write Piper .sln16 files that Asterisk
+          # root-owned after node reboot. Pod-level runAsNonRoot:true would
-          # reads at /var/lib/asterisk/sounds/tts. World-readable (755) is
+          # otherwise inherit and chown would fail with EPERM (see Notes memory
-          # fine — Asterisk runs as a different uid in the other pod.
+          # feedback_hostpath_initcontainer_chown_perms).
+          securityContext:
+            runAsUser: 0
+            runAsNonRoot: false
           command: ["sh", "-c", "chown -R 1654:1654 /data && chown 1654:1654 /shared-tts && chmod 0755 /shared-tts"]
           volumeMounts:
             - name: telephony-data