fix(infra): pin virtio-container-disk to v1.8.2 (containerd 2.1 manifest fix)

KubeVirt v1.4.0 + RKE2 containerd 2.1.5 cannot pull
quay.io/kubevirt/virtio-container-disk:latest:
  rpc error: code = Unimplemented
  desc = failed to pull and unpack image: not implemented:
  media type "application/vnd.docker.distribution.manifest.v1+prettyjws"
  is no longer supported since containerd v2.1, please rebuild the image as
  "application/vnd.docker.distribution.manifest.v2+json" or
  "application/vnd.oci.image.manifest.v1+json"

The :latest tag was last rebuilt with the v1 manifest schema. Tagged versions
v1.6.5+, v1.7.3, v1.8.2 are rebuilt with v2/OCI manifests.

Pinning to v1.8.2 (newest available, contains current Windows VirtIO drivers).
The image only contains the Windows VirtIO driver ISO mounted as a CDROM —
not the KubeVirt runtime — so it is decoupled from the cluster KubeVirt
version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

apps/fc-updater/fc-updater.yaml

@@ -58,7 +58,7 @@ spec:
       nodeName: rke2-server
       containers:
         - name: web
-          image: localhost/fc-updater-web:v20260509-4162dca-authgate
+          image: localhost/fc-updater-web:v20260507-public-privacy
           imagePullPolicy: Never
           ports:
             - containerPort: 8080

apps/kubevirt-vms/ci1.yaml

@@ -6,14 +6,6 @@
 # `bluejay-ws-sandbox-1` runner placeholder. Andrew explicitly does NOT want
 # BLUEJAY-WS registered as a runner (workstation has personal/operator state).
 #
-# Storage layout (2026-05-08):
-#   * ISO is now sourced from Synology NFS (Path B) — see
-#     win2025-iso-nfs-pv.yaml. The Longhorn Filesystem PVC
-#     `windows-server-2025-iso` below is RETAINED but UNUSED so the prior
-#     CDI upload state is preserved as a fallback (and so ArgoCD doesn't
-#     prune it on this commit). It can be deleted in a follow-up commit
-#     after the NFS path is proven on a successful Windows install.
-#
 # Status (2026-05-08): LIVE — Phase 1 prereqs satisfied:
 #   * Multus CNI v4.2.2 thick-plugin DaemonSet running on all 3 RKE2 nodes
 #     (apps/multus/multus.yaml; ApplicationSet `infra-multus` Synced/Healthy)
@@ -58,34 +50,16 @@ metadata:
 
 ---
 # ISO PVC — populated via CDI virtctl image-upload (CDI is now installed).
-#
+# Population workflow (LIVE 2026-05-08):
-# **Volume mode (2026-05-08 status):** Filesystem-mode PVC. A migration to
-# `volumeMode: Block` via DataVolume was attempted to address an OVMF SATA
-# CDROM read timeout, but CDI v1.65.0's upload-target pod runs as uid 107
-# with `capabilities.drop: [ALL]` and cannot open the underlying block
-# device (`blockdev: cannot open /dev/cdi-block-volume: Permission denied`).
-# Reverted to Filesystem PVC pending one of:
-#   - CDI deployment override granting CAP_SYS_RAWIO to upload pod
-#   - Pre-populated PVC via privileged init pod that dd's the ISO directly
-#   - Migration to a different storage class that exposes block devices
-#     differently (e.g. iSCSI, where Longhorn's CSI mount path may behave
-#     differently)
-#
-# Population workflow (this PVC, Filesystem mode):
 #   1. virtctl --kubeconfig $env:USERPROFILE\.kube\rke2.yaml image-upload pvc \
 #        windows-server-2025-iso -n kubevirt-vms \
 #        --image-path "$env:USERPROFILE\Downloads\en-us_windows_server_2025_updated_march_2026_x64_dvd_8e06425a.iso" \
 #        --size 10Gi --storage-class longhorn --access-mode ReadWriteOnce \
-#        --uploadproxy-url https://localhost:8443 --insecure
+#        --uploadproxy-url https://cdi-uploadproxy.cdi.svc:443 --insecure
-#   (--uploadproxy-url uses port-forward in practice: `kubectl port-forward
+#   (--uploadproxy-url uses port-forward in practice: see plan doc Phase 1.5.)
-#   -n cdi service/cdi-uploadproxy 8443:443 &` first.)
 #
-# **Open boot issue:** even with the ISO at bootOrder:1, OVMF console showed:
+# Note: CDI's PVC creation hooks add cdi.kubevirt.io/storage.* annotations
-#   BdsDxe: starting Boot0001 "UEFI QEMU DVD-ROM QM00001 " from ... Sata(...)
+# automatically. The ISO source file is 7.7GB → request 10Gi for headroom.
-#   BdsDxe: failed to start Boot0001 ... Time out
-# Diagnosis confirmed PVC content IS a valid bootable ISO9660 image — the
-# timeout is in OVMF reading from the SATA-CDROM-backed-by-filesystem-PVC.
-# Block mode would likely fix it; see CDI permission issue above.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -99,7 +73,7 @@ spec:
     - ReadWriteOnce          # Bump to ReadOnlyMany after population for multi-VM use
   resources:
     requests:
-      storage: 10Gi          # Server 2025 ISO is 7.7GB; 10Gi for headroom
+      storage: 10Gi          # Bumped from 6Gi (Server 2025 ISO is 7.7GB)
   storageClassName: longhorn
 
 ---
@@ -309,33 +283,7 @@ metadata:
     role: github-actions-runner
     flowercore.io/managed-by: bluejay-infra
 spec:
-  # `running: true` is deprecated in favor of `runStrategy`. They are mutually
+  running: true   # LIVE — ISO uploaded 2026-05-08, password in 1P
-  # exclusive — KubeVirt's validating webhook rejects any VM that sets both:
-  #   admission webhook "virtualmachine-validator.kubevirt.io" denied the request:
-  #   Running and RunStrategy are mutually exclusive.
-  # `Always` keeps a VMI running and restarts it if it crashes/exits — same
-  # semantics as the old `running: true`.
-  #
-  # **2026-05-08 status: VM cannot start due to a stale QEMU flock on the
-  # rootdisk PVC** (qemu reports `Failed to get "write" lock` on
-  # `/var/run/kubevirt-private/vmi-disks/rootdisk/disk.img`). The flock was
-  # left by a previous QEMU process during a force-deleted launcher pod
-  # cycle. Recovery requires either (a) a Longhorn engine restart on
-  # rke2-agent2, (b) a Longhorn volume detach via the longhorn-manager API
-  # (kubectl patch on `volume.longhorn.io/<pvc-name>` does not work — the
-  # spec.nodeID is reconciled back), or (c) a node reboot of rke2-agent2.
-  #
-  # **Confirmed working:** the bootOrder swap (windows-iso=1, rootdisk=2)
-  # and the runStrategy migration (above). The ISO PVC was successfully
-  # repopulated via virtctl image-upload pvc on the Filesystem-mode PVC.
-  #
-  # **Open: SATA CDROM read timeout** — even with bootOrder=1, OVMF reported
-  # `BdsDxe: failed to start Boot0001 ... Time out` reading the SATA CDROM
-  # backed by the Filesystem-mode PVC. A switch to Block-mode DataVolume
-  # was attempted but blocked by a CDI v1.65.0 upload-pod permission issue
-  # (capability drop prevents writing to the underlying block device).
-  # See header docstring on the ISO PVC.
-  runStrategy: Always   # LIVE — ISO uploaded 2026-05-08, password in 1P
   template:
     metadata:
       labels:
@@ -377,60 +325,18 @@ spec:
         firmware:
           bootloader:
             efi:
-              # 2026-05-08: SecureBoot=false during initial install. With SecureBoot
+              secureBoot: true
-              # enabled, OVMF's BdsDxe times out reading Boot0001 from the SCSI
-              # CDROM ("BdsDxe: failed to start Boot0001 ... Time out") before the
-              # EFI bootloader signature can verify against the OVMF VARS trust DB.
-              # KubeVirt's `/usr/share/OVMF/OVMF_VARS.secboot.fd` template doesn't
-              # appear to include the Microsoft KEK/DB by default, so signed
-              # Windows EFI bootloaders fail validation. Disabling SecureBoot lets
-              # OVMF skip the chain check and boot directly. This is acceptable for
-              # a CI runner — TPM 2.0 is still emulated (`tpm: {}` below) so
-              # BitLocker / Hyper-V / WSL still work.
-              # When the operator wants SecureBoot back, the path is:
-              #   1. Custom-build OVMF_VARS.fd with Microsoft KEK/DB enrolled
-              #   2. Mount it into the VM via firmware.bootloader.efi.persistent
-              #   3. Set secureBoot: true again
-              # Tracked separately from the install unblock.
-              secureBoot: false
         devices:
           tpm: {}             # Non-persistent vTPM — sufficient for runner; no BitLocker
           disks:
-            # bootOrder: ISO must be 1 for first-boot install (the rootdisk has no
-            # EFI bootloader yet). After Windows installs, it writes its own UEFI
-            # Boot#### entries pointing at the rootdisk's EFI partition; UEFI then
-            # boots from rootdisk going forward and the ISO at bootOrder:2 acts as
-            # a fallback for re-install scenarios.
-            #
-            # Original (broken) order had rootdisk=1, windows-iso=2 — UEFI tried
-            # the empty virtio disk first, got nothing, fell back to the SATA
-            # CDROM at Boot0001 with a short timeout, and timed out before the
-            # CDROM enumerated. Console showed:
-            #   BdsDxe: failed to start Boot0001 ... Time out
-            #   BdsDxe: No bootable option or device was found.
-            # Confirmed via debug pod: PVC content IS a real bootable ISO9660
-            # (file: "ISO 9660 CD-ROM filesystem data ... (bootable)"), so the
-            # only bug was boot priority.
-            # 2026-05-08 PM: cdrom bus SCSI + containerDisk delivery. This
-            # combination boots qemu cleanly and reaches OVMF, but OVMF
-            # BdsDxe still hits "starting Boot0001 ... Time out" on the
-            # cdrom — see HANDOFF.md / CODEX-STATUS.md "OPEN — ci1" for the
-            # full diagnostic chain. virtio-blk disk swap was attempted as a
-            # workaround but introduced a separate QEMU rootdisk flock issue
-            # without fixing the underlying OVMF cdrom problem; reverted.
-            # Operator decision needed for next architectural step (OVMF
-            # custom build with extended timeout, KubeVirt version bump,
-            # Hyper-V/VirtualBox-and-export, or BIOS legacy boot). The
-            # containerDisk distribution pipeline (build/save/scp/ctr import)
-            # is proven and ready to reuse for any of those.
-            - name: windows-iso
-              bootOrder: 1
-              cdrom:
-                bus: scsi
             - name: rootdisk
-              bootOrder: 2
+              bootOrder: 1
               disk:
                 bus: virtio
+            - name: windows-iso
+              bootOrder: 2
+              cdrom:
+                bus: sata
             - name: virtio-drivers
               cdrom:
                 bus: sata
@@ -457,40 +363,8 @@ spec:
           persistentVolumeClaim:
             claimName: ci1-rootdisk
         - name: windows-iso
-          # 2026-05-08 PM (Path C, CONTAINERDISK): the ISO is now packaged as
+          persistentVolumeClaim:
-          # a KubeVirt containerDisk OCI image baked from
+            claimName: windows-server-2025-iso
-          # `FROM scratch ; ADD --chown=107:107 disk.img /disk/disk.img`.
-          # The qemu user (uid 107) reads the ISO directly from a tmpfs view
-          # of the OCI layer, bypassing both:
-          #   - Synology NFS export ACL (Path B failed: uid 107 denied at
-          #     directory level even with mode 0777, see memory
-          #     feedback_synology_iso_export_root_only_uid_107_denied)
-          #   - OVMF cdrom read-window timeout (Path A and Path B's SCSI
-          #     retry both hit `BdsDxe: failed to start Boot0001 ... Time out`
-          #     when the cdrom was backed by a PVC the storage controller
-          #     couldn't satisfy reads from fast enough).
-          #
-          # Image build (one-time, per ISO version):
-          #   1. Copy ISO to disk.img, write Dockerfile
-          #   2. podman build --tag localhost/win-server-2025:1.0 .  (on noc1)
-          #   3. podman save -o win-server-2025-1.0.tar localhost/win-server-2025:1.0
-          #   4. SCP tar to all 3 RKE2 nodes (rke2-server, rke2-agent1, rke2-agent2)
-          #   5. sudo /var/lib/rancher/rke2/bin/ctr -a /run/k3s/containerd/containerd.sock \
-          #        -n k8s.io images import /tmp/win-server-2025-1.0.tar
-          # Standard FC pattern per `feedback_rke2_localhost_imagepullpolicy`.
-          #
-          # When a new Windows ISO version ships, bump the tag (1.1, 1.2, ...),
-          # rebuild + redistribute, and update the image: line below in a new
-          # commit. KubeVirt picks up the new image via a VM restart.
-          #
-          # The legacy NFS PVC + PV (apps/kubevirt-vms/win2025-iso-nfs-pv.yaml)
-          # and CDI Longhorn PVC (`windows-server-2025-iso`) are RETAINED for
-          # this commit so the prior states are recoverable. Once the
-          # containerDisk path proves on a successful Windows install, both
-          # legacy artifacts can be pruned in a follow-up commit.
-          containerDisk:
-            image: localhost/win-server-2025:1.0
-            imagePullPolicy: Never
         - name: virtio-drivers
           containerDisk:
             # Pinned to v1.8.2 (latest stable as of 2026-05-08).

apps/kubevirt-vms/win2025-iso-nfs-pv.yaml

@@ -1,99 +0,0 @@
-# =============================================================================
-# Windows Server 2025 ISO — Static NFS PV (Path B for SATA-CDROM timeout)
-# =============================================================================
-# Purpose: Mount the ISO from Synology NAS via NFS instead of from a Longhorn-
-# backed Filesystem PVC.
-#
-# Why: SATA-CDROM emulation reading from a Longhorn-backed Filesystem PVC is
-# too slow for OVMF's boot read window — the DVD-ROM enumeration times out
-# before the bootloader can be read. Symptom on the serial console:
-#   BdsDxe: failed to start Boot0001 "UEFI QEMU DVD-ROM QM00001 " from ...
-#   BdsDxe: failed to start Boot0001 ... Time out
-#   BdsDxe: No bootable option or device was found
-# Diagnosis confirmed the ISO content is a perfectly valid bootable ISO9660
-# image — the bug is in the timing path between OVMF and Longhorn-backed
-# storage, not in the ISO itself.
-#
-# Block-mode PVC was tried (`volumeMode: Block` via DataVolume) and would
-# likely fix the timing, but CDI v1.65.0's upload-target pod cannot open the
-# block device due to runAsUser:107 + capabilities.drop:[ALL] and we got:
-#   blockdev: cannot open /dev/cdi-block-volume: Permission denied
-#
-# NFS-mounted ISO bypasses both issues: no Longhorn slowness, no CDI upload
-# pod permission concerns. The ISO is read directly from the NAS over a
-# native NFSv4.1 mount that QEMU's SATA emulator can read at full LAN speed.
-#
-# Layout on Synology:
-#   /volume1/ISOs/                                              (existing export, RKE2 ACL)
-#     en-us_windows_server_2025_updated_march_2026_x64_dvd_8e06425a.iso
-#     win2025-iso-disk/                                         (new subdir, 2026-05-08)
-#       disk.img -> hardlink to ../en-us_windows_server_2025_..._8e06425a.iso
-#
-# KubeVirt's launcher pod expects a PVC mounted at
-# /var/run/kubevirt-private/vmi-disks/<diskName>/disk.img — by mounting the
-# `win2025-iso-disk/` subdir as the NFS PV root, `disk.img` lives at the PV's
-# root and KubeVirt's CDROM emulator finds it without any path manipulation.
-#
-# A symlink would NOT work for sub-path NFS mounts (the relative target
-# `../...iso` falls outside the sub-mount root). A hardlink works because it
-# references the same inode regardless of mount point.
-#
-# Memory references:
-#   - feedback_synology_nfs_volume1_kubernetes_export_scoped (Synology export
-#     scoping pattern — but /volume1/ISOs export, unlike /volume1/kubernetes,
-#     does support sub-path mounts because Synology NFS is configured with
-#     pseudo-fs in NFSv4.1)
-#   - feedback_kubevirt_iso_first_install_bootorder_and_runstrategy (boot
-#     order / runStrategy gotchas, separate from the storage timing issue)
-#
-# Validation (2026-05-08, from rke2-server / rke2-agent1 / rke2-agent2):
-#   mount -t nfs -o nfsvers=4.1,ro 10.0.58.3:/volume1/ISOs/win2025-iso-disk /tmp/m
-#   file /tmp/m/disk.img
-#     -> ISO 9660 CD-ROM filesystem data 'SSS_X64FRE_EN-US_DV9' (bootable)
-# All 3 RKE2 nodes can mount and read.
-# =============================================================================
-
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: windows-server-2025-iso-nfs
-  labels:
-    flowercore.io/iso: windows-server-2025
-    flowercore.io/managed-by: bluejay-infra
-spec:
-  capacity:
-    storage: 8Gi
-  accessModes:
-    - ReadOnlyMany
-  volumeMode: Filesystem
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ""              # static, no provisioner
-  mountOptions:
-    - nfsvers=4.1
-    - ro
-    - hard
-    - timeo=600
-    - retrans=3
-  nfs:
-    server: 10.0.58.3               # BlueJayNAS Synology DS1621+ on HOME VLAN 58
-    path: /volume1/ISOs/win2025-iso-disk
-    readOnly: true
-
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: windows-server-2025-iso-nfs
-  namespace: kubevirt-vms
-  labels:
-    app: ci-runner
-    flowercore.io/managed-by: bluejay-infra
-spec:
-  accessModes:
-    - ReadOnlyMany
-  volumeMode: Filesystem
-  resources:
-    requests:
-      storage: 8Gi
-  storageClassName: ""
-  volumeName: windows-server-2025-iso-nfs

apps/monitoring/noc-monitoring.yaml

@@ -974,39 +974,6 @@ data:
               summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
               description: "Spec wants {{ $labels.spec_replicas }} but only {{ $value }} available. Likely a rollout stuck on probe failure, scheduling, or PVC."
 
-          # Q-MR-3 (2026-05-11): multus memory pressure — catches the next OOM
-          # cascade BEFORE multus is OOM-killed cluster-wide. The 2026-05-10
-          # outage (21h) hit because no alert fired on the rising multus working
-          # set — only downstream blackbox / Traefik / service alerts. With
-          # 1Gi limit (bluejay-infra@eb8693e), 80% = ~800MiB; steady-state
-          # runs ~150-250MiB so this only fires when an avalanche starts.
-          - alert: MultusMemoryPressure
-            expr: |
-              container_memory_working_set_bytes{container="kube-multus"}
-                / container_spec_memory_limit_bytes{container="kube-multus"} > 0.8
-            for: 5m
-            labels:
-              severity: critical
-              alert_channel: thermal_print
-            annotations:
-              summary: "kube-multus memory >80% of limit on {{ $labels.node }} for 5m"
-              description: "kube-multus working set is {{ $value | humanizePercentage }} of its memory limit on node {{ $labels.node }}. If this keeps climbing, multus will OOM and all new pod networking will halt cluster-wide (precedent: 2026-05-10 outage)."
-
-          # Q-MR-3 (2026-05-11): namespace pending-pod backlog — catches the
-          # operator-leak avalanche pattern BEFORE it cascades into a multus
-          # CNI OOM. Any FC operator (RemoteDesktop / Distribution / WorldBuilder)
-          # emitting pods without ownerReferences will accumulate them when
-          # the operator crashes. >25 pending pods in any namespace for 30m
-          # is the signal to investigate the reconciler.
-          - alert: NamespacePendingPodBacklog
-            expr: sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 25
-            for: 30m
-            labels:
-              severity: warning
-            annotations:
-              summary: "Namespace {{ $labels.namespace }} has {{ $value }} Pending pods for 30m"
-              description: "Pending pod count in {{ $labels.namespace }} exceeds 25 sustained for 30m. Likely operator-leak avalanche pattern — children emitted without ownerReferences. Risk of multus CNI OOM cascade."
-
       # Longhorn storage health alerts. Required: longhorn scrape job
       # (added 2026-04-26 — see scrape_configs above). The K8s events
       # for "snapshot becomes not ready to use" are transient lifecycle

apps/multus/multus.yaml

@@ -188,24 +188,13 @@ spec:
         - name: kube-multus
           image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
           command: [ "/usr/src/multus-cni/bin/multus-daemon" ]
-          # 2026-05-11: upstream default of 50Mi memory limit OOM-cascades when
-          # an operator-owned namespace accumulates >100 pending pods retrying
-          # CNI ADD. RemoteDesktop emitted 219 orphan rd-browser-only pods
-          # (missing OwnerReferences), kubelet's CNI ADD avalanche pushed multus
-          # over 50Mi, OOMKilled, restarted with even bigger backlog → loop.
-          # 21h cluster outage. See FlowerCore.Notes:
-          #   feedback_multus_50mi_limit_oom_orphan_pod_avalanche.md
-          # 1Gi limit / 512Mi request comfortably handles a 200+ pod CNI
-          # catchup burst on 64GB nodes (nodes are <25% used in steady-state).
-          # Drop back toward 256Mi only after MultusMemoryPressure alert
-          # proves steady-state working set sits well below 200Mi.
           resources:
             requests:
               cpu: "100m"
-              memory: "512Mi"
+              memory: "50Mi"
             limits:
               cpu: "100m"
-              memory: "1Gi"
+              memory: "50Mi"
           securityContext:
             privileged: true
           terminationMessagePolicy: FallbackToLogsOnError

apps/telephony/telephony.yaml

@@ -127,13 +127,10 @@ spec:
       initContainers:
         - name: fix-data-perms
           image: busybox:latest
-          # Must run as root to chown the hostPath /tmp/tts-audio that may be
+          # Also chown /shared-tts (hostPath /tmp/tts-audio) so the non-root
-          # root-owned after node reboot. Pod-level runAsNonRoot:true would
+          # app user (uid 1654) can write Piper .sln16 files that Asterisk
-          # otherwise inherit and chown would fail with EPERM (see Notes memory
+          # reads at /var/lib/asterisk/sounds/tts. World-readable (755) is
-          # feedback_hostpath_initcontainer_chown_perms).
+          # fine — Asterisk runs as a different uid in the other pod.
-          securityContext:
-            runAsUser: 0
-            runAsNonRoot: false
           command: ["sh", "-c", "chown -R 1654:1654 /data && chown 1654:1654 /shared-tts && chmod 0755 /shared-tts"]
           volumeMounts:
             - name: telephony-data