Documenting the remaining 2 unresolved issues for the next operator
session, with the recovery paths from this session captured inline so
the next agent doesn't repeat the same blind alleys:
1. **rootdisk QEMU flock** — every new launcher pod fails QEMU start with
`Failed to get "write" lock` on the rootdisk Filesystem-mode disk.img.
Stale flock from a previous force-deleted virt-launcher pod. Longhorn
engine on rke2-agent2 needs to release the lock; `kubectl patch
volume.longhorn.io/<pvc-name> spec.nodeID=""` is reverted by the
Longhorn controller. Operator-level recovery only.
2. **SATA CDROM read timeout** — even with bootOrder=1 (windows-iso first),
OVMF UEFI fails Boot0001 with "Time out" reading the SATA CDROM backed
by the Filesystem-mode PVC. Block-mode DataVolume migration was
attempted but blocked by CDI v1.65.0's upload pod running with
`capabilities.drop: [ALL]` and `runAsUser: 107`, preventing direct
block-device writes (`blockdev: cannot open /dev/cdi-block-volume:
Permission denied`). See ISO PVC header docstring for 3 forward paths.
Net commits during this session:
- 1c4145a: bootOrder swap (windows-iso=1, rootdisk=2)
- 87a7d7c: deprecated `running:` -> `runStrategy: Always`
- 0bf47df: ISO migration to Block-mode DataVolume (REVERTED)
- 9f6dc1a: revert to Filesystem PVC (CDI block-upload blocked)
- 1c4145a + 87a7d7c + 9f6dc1a are the live, correct configuration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Block-mode DataVolume migration (commit 0bf47df) hit a CDI v1.65.0 limitation:
the upload-target pod runs as uid 107 with `capabilities.drop: [ALL]`, so it
cannot open the underlying block device:
blockdev: cannot open /dev/cdi-block-volume: Permission denied
Saving stream failed: Unable to transfer source data to target file:
error determining if block device exists: exit status 1
Reverting to a Filesystem-mode PVC + virtctl image-upload pvc, which DID work
(uploaded the 7.7 GiB ISO with valid ISO9660 magic intact). Boot timeout is
unresolved (header docstring captures the open issue + 3 paths to revisit).
The bootOrder swap (1c4145a) and runStrategy migration (87a7d7c) stay landed —
those are correct improvements regardless of the volume-mode question.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The bootOrder swap alone didn't fix the install — even with `windows-iso` at
bootOrder:1, OVMF UEFI still timed out reading the SATA CDROM:
BdsDxe: starting Boot0001 "UEFI QEMU DVD-ROM QM00001 " from ... Sata(...)
BdsDxe: failed to start Boot0001 ... : Time out
BdsDxe: No bootable option or device was found.
Diagnosis (debug pod mounting the live PVC):
- /pvc/disk.img IS a valid bootable ISO9660 image — `file` reports
"ISO 9660 CD-ROM filesystem data 'SSS_X64FRE_EN-US_DV9' (bootable)".
- bytes 0..15: zeros (NOT QCOW2 magic 51 46 49 fb).
- bytes 32769..32773: "CD001" — ISO9660 primary volume descriptor at the
correct offset.
So content was fine. The bug is in how KubeVirt + QEMU + Longhorn expose a
Filesystem-mode PVC's `/disk.img` as a SATA CDROM. With Block-mode the
underlying volume IS the raw ISO9660 sectors, OVMF reads them directly,
no QEMU file-emulation layer. This is the recommended pattern for ISO
install media on KubeVirt + Longhorn.
Migration:
- Replace `kind: PersistentVolumeClaim` with `kind: DataVolume` (CDI manages
the underlying PVC + upload-target pod).
- Set `pvc.volumeMode: Block`.
- Annotate `cdi.kubevirt.io/storage.contentType: kubevirt` so CDI keeps raw
bytes (no QCOW2 wrap).
- VM volume reference changes from `persistentVolumeClaim.claimName` to
`dataVolume.name`. KubeVirt's VMI controller blocks VM start until DV
phase is Succeeded (upload completed).
Operator step after this lands:
1. Wait for DV `phase: UploadReady`
kubectl get dv -n kubevirt-vms windows-server-2025-iso -w
2. virtctl image-upload dv windows-server-2025-iso -n kubevirt-vms \
--image-path "...\en-us_windows_server_2025...iso" \
--uploadproxy-url https://localhost:8443 --insecure --no-create
3. Re-flip runStrategy to Always (was set to Halted live-side during
migration; this commit keeps the manifest at Always).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Required to clear OutOfSync state after the bootOrder fix. Live VM had
runStrategy: Halted (set during diagnosis to release the PVC for inspection).
Manifest had running: true. KubeVirt's validating webhook rejects sync:
admission webhook "virtualmachine-validator.kubevirt.io" denied the request:
Running and RunStrategy are mutually exclusive.
Switching to runStrategy: Always preserves the original "auto-start +
auto-restart" semantics with the non-deprecated field, and gives ArgoCD a
clean diff target to flip Halted -> Always.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Original order: rootdisk=1 (empty 200Gi virtio), windows-iso=2 (SATA CDROM).
UEFI tried the empty virtio disk first, got nothing, fell back to Boot0001
(the SATA CDROM) with a short timeout, and aborted with:
BdsDxe: failed to start Boot0001 ... Time out
BdsDxe: No bootable option or device was found.
VM had been running 38+ min with rootdisk actualSize stuck at 4.13 GiB and
no AgentConnected condition — install never started.
Diagnosis via debug pod mounting the windows-server-2025-iso PVC:
/pvc/disk.img: ISO 9660 CD-ROM filesystem data 'SSS_X64FRE_EN-US_DV9' (bootable)
bytes 0..15: zeros (NOT QCOW2 magic 51 46 49 fb)
bytes 32769..32773: "CD001" (ISO9660 primary volume descriptor)
So the PVC content is a real bootable ISO — the only fix needed is to make
the ISO bootOrder=1 for first install. After Windows installs, it writes its
own UEFI Boot#### entries pointing at the rootdisk EFI partition; UEFI then
boots from rootdisk going forward and the ISO at bootOrder:2 is a fallback
for re-install scenarios.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
KubeVirt v1.4.0 + RKE2 containerd 2.1.5 cannot pull
quay.io/kubevirt/virtio-container-disk:latest:
rpc error: code = Unimplemented
desc = failed to pull and unpack image: not implemented:
media type "application/vnd.docker.distribution.manifest.v1+prettyjws"
is no longer supported since containerd v2.1, please rebuild the image as
"application/vnd.docker.distribution.manifest.v2+json" or
"application/vnd.oci.image.manifest.v1+json"
The :latest tag was last rebuilt with the v1 manifest schema. Tagged versions
v1.6.5+, v1.7.3, v1.8.2 are rebuilt with v2/OCI manifests.
Pinning to v1.8.2 (newest available, contains current Windows VirtIO drivers).
The image only contains the Windows VirtIO driver ISO mounted as a CDROM —
not the KubeVirt runtime — so it is decoupled from the cluster KubeVirt
version.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 prereqs all satisfied:
- Multus CNI v4.2.2 thick-plugin DS Running on rke2-server/agent1/agent2
- CDI v1.65.0 operator + CR Deployed (cdi-apiserver/deployment/uploadproxy
all Running 1/1)
- Windows Server 2025 ISO (7.7GiB, March 2026 update) uploaded via CDI
virtctl image-upload to PVC windows-server-2025-iso. Verified via PVC
annotations: cdi.kubevirt.io/storage.condition.running.message="Upload
Complete", storage.pod.phase="Succeeded"
- Local Administrator password generated (26 char, FANTASTIC strength).
Stored in 1Password vault IAmWorkin (qaphopopkryhbg353ukzhhuqoq) item
h3ix4mgfk65gmkcmvh6ly3d3hu. UTF-16-LE base64 in autounattend.xml Value
field matches the 1P "autounattend AdministratorPassword Value" field.
Changes:
- ISO PVC bumped 6Gi → 10Gi (ISO is 7.7GiB, need headroom)
- Added labels app=ci-runner, flowercore.io/managed-by=bluejay-infra
- autounattend.xml AdministratorPassword Value: real base64-encoded password
- spec.running: false → true (VM starts on next ArgoCD sync)
- Header comment refreshed to LIVE state with prereq references
Network: still pod-network masquerade. Multus NAD prod-vlan57 is registered
but the VM doesn't use it yet (Phase 1.5 host bridge needed first).
Verify after sync:
kubectl --kubeconfig $env:USERPROFILE\.kube\rke2.yaml -n kubevirt-vms get vm,vmi
virtctl --kubeconfig $env:USERPROFILE\.kube\rke2.yaml vnc ci1 -n kubevirt-vms
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds three new bluejay-infra apps that auto-pickup via ApplicationSet (apps/*
directory generator on main):
* apps/multus/multus.yaml — Multus CNI v4.2.2 thick-plugin daemonset (verbatim
upstream, project-annotated). Enables KubeVirt VMs to attach additional
network interfaces. Required by ci1 to bridge onto PROD VLAN 57.
* apps/cdi/{cdi-operator.yaml,cdi-cr.yaml,README.md} — Containerized Data
Importer v1.65.0 (verbatim upstream). Operator + CR pattern. Enables
populating PVCs from HTTP/registry/upload sources, used to load the Windows
Server 2025 ISO into the windows-server-2025-iso PVC.
* apps/kubevirt-vms/prod-vlan57-nad.yaml — NetworkAttachmentDefinition for
PROD VLAN 57 bridge. **Deploy gated on Phase 1.5 host work**: requires
br-prod bridge enslaving enp86s0.57 on each RKE2 node (Puppet config-as-code).
ci1.yaml continues to use pod-network masquerade until that lands; switching
to multus.networkName: kubevirt-vms/prod-vlan57 is a one-line YAML edit
followed by a GitOps push.
Cluster verification (2026-05-08):
- KubeVirt LIVE (3 nodes, virt-api/controller/handler/operator all Running)
- Calico CNI on /etc/cni/net.d + /opt/cni/bin (Multus default paths)
- ApplicationSet `bluejay-infra` already watches `apps/*` on main
Reproducibility: upstream YAMLs vendored verbatim with project header diffs
only. Bumping versions = re-curl + git push. No deploy-time internet fetch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stages a draft VirtualMachine + Namespace + ISO PVC + rootdisk PVC + sysprep
ConfigMap for the dedicated GitHub Actions self-hosted runner that replaces
the never-registered bluejay-ws-sandbox-1 placeholder.
Status: STAGED ONLY. spec.running = false. ISO PVC empty. Two operator
decisions still pending before this can boot:
1. Network choice — pod-network fallback (in this draft) vs Multus +
PROD VLAN NAD (preferred, requires Multus install).
2. ISO path — manual upload via helper pod (Path A) vs CDI HTTP import
(Path B, requires CDI install).
Cluster baseline 2026-05-08:
- KubeVirt operator: installed, healthy, 14d
- CDI: NOT installed
- Multus: NOT installed
- Calico-only CNI
See docs/infrastructure/windows-server-build-runner-plan.md "Phase 1 readiness
gate" for the full operator pickup checklist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
