Compare commits

..

24 Commits

Author SHA1 Message Date
Codex
41c598394e Add Authentik OIDC client registration assets 2026-05-13 11:37:09 -05:00
Andrew Stoltz
5029e209cd kubevirt-vms: boot ci1 from server template 2026-05-12 16:58:18 -05:00
Codex
f298339152 fix(guacamole): add --- separator between macmini-vnc-creds OnePasswordItem and guacamole-branding ConfigMap
Missing document separator caused YAML to merge the OnePasswordItem's
top-level `spec: itemPath:` block into the ConfigMap that follows.
Result: a ConfigMap with a `.spec` field whose K8s schema does not
declare one, triggering ArgoCD's structured-merge diff to fail since
2026-05-11T15:30:54Z:

  Failed to compare desired state to live state: failed to calculate
  diff: error calculating structured merge diff: error building typed
  value from config resource: .spec: field not declared in schema

App stayed Healthy (live K8s tolerated the extra field — ConfigMap
ignored it) but ArgoCD's diff calc was broken, leaving the app stuck at
sync=Unknown for all 21 resources. Adding the missing `---` separator
makes the OnePasswordItem and ConfigMap proper sibling YAML documents,
each with its own kind-correct schema.

Diagnosed during 2026-05-12 morning routine.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 09:26:03 -05:00
Codex
6e7d88db49 feat(fc-redis): add SignalR backplane for cross-product event bus (Q-SO-1 Phase A)
Per Q-SO-1 operator resolution 2026-05-11 PM, Redis SignalR backplane lands
in Phase A (was Phase C deferral). Treats Redis as a managed FC infrastructure
component, not a deferred scaling escalation.

Lands the minimal Phase A surface:
- Namespace fc-redis
- Single Redis 7-alpine pod with 1Gi Longhorn RWO PVC
- ConfigMap with AOF persistence (everysec), 256Mi maxmemory, allkeys-lru
- ClusterIP Service `redis.fc-redis.svc.cluster.local:6379` (in-cluster only)
- No AUTH Phase A (Phase B add via 1Password Connect rotation)
- No IngressRoute (backplane is server-to-server)

Consumers (Phase A IMPL across FC services) add:
  services.AddSignalR().AddStackExchangeRedis(
      "redis.fc-redis.svc.cluster.local:6379",
      opts => opts.Configuration.ChannelPrefix =
          StackExchange.Redis.RedisChannel.Literal("fc-opsconsole"));

Phase B/C follow-ons (not in this commit): Sentinel for HA, AUTH password
from 1Password, redis_exporter sidecar for Prometheus, network policies.

See FlowerCore.Notes/docs/signage/operations-console-phase-2-design.md
section 3.5 (rewritten) and decisions-waiting.html Q-SO-1 (RESOLVED).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 19:02:58 -05:00
Codex
5ae50bd491 fix(telephony): init container runs as root to chown hostPath /tmp/tts-audio
The fix-data-perms init container chowns /data (PVC) and /shared-tts
(hostPath /tmp/tts-audio on rke2-agent1) to uid 1654 so the non-root
telephony-web app can write Piper TTS .sln16 files.

Without an explicit container-level securityContext override, the init
container inherits pod-level runAsNonRoot:true / runAsUser:1654 and
fails with 'chown: /shared-tts: Operation not permitted' the first
time the hostPath comes up root-owned after a node reboot.

Outage 2026-05-11 23:00 UTC: telephony-web in Init:CrashLoopBackOff for
9 hours (100+ restarts) until init container was bumped to runAsUser:0.
Live cluster patched in the same operation; this commit makes the fix
durable in git so ArgoCD sync preserves it.

See Notes memory: feedback_hostpath_initcontainer_chown_perms

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 18:37:15 -05:00
Codex
653d4472f5 fix(monitoring): mirror Q-MR-3 MultusMemoryPressure + NamespacePendingPodBacklog alerts
Two new preventive alert rules added to the kubernetes-state group of the
K8s migration target ConfigMap. The live Podman Prometheus on noc1 has
already been updated via FlowerCore.Notes/scripts/monitoring/alerts.yml +
sudo cp + podman pod restart monitoring (this commit only locks it in
the bluejay-infra K8s mirror so a future migration carries it forward).

MultusMemoryPressure (critical, thermal_print): fires when kube-multus
working set exceeds 80% of its memory limit for 5m. Catches the next
multus OOM cascade BEFORE it kills the daemon cluster-wide. The 2026-05-10
21h outage hit because no alert fired on the rising multus working set;
only downstream blackbox / Traefik / service alerts triggered, after the
fact.

NamespacePendingPodBacklog (warning): fires when any single namespace has
>25 Pending pods sustained for 30m. Catches the operator-leak avalanche
pattern (orphan pods from a crashed reconciler emitting children without
ownerReferences) before it cascades into a CNI OOM.

See FlowerCore.Notes:
  - feedback_multus_50mi_limit_oom_orphan_pod_avalanche
  - feedback_monitoring_k8s_target_vs_live_podman (workflow)

Companion commits:
  - bluejay-infra@eb8693e (multus memory limit)
  - FlowerCore.RemoteDesktop@b02c59b (OwnerReferences fix)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 10:42:27 -05:00
Codex
eb8693e1ce fix(multus): bump kube-multus-ds memory 50Mi/50Mi -> 1Gi/512Mi (prevent OOM cascade)
Cluster outage 2026-05-10T17:43 through 2026-05-11 ~10:30 (~21h). Root cause:
FlowerCore.RemoteDesktop emitted 219 orphan rd-browser-only-* pods in fc-desktop
(missing OwnerReferences — see companion fix in FlowerCore.RemoteDesktop).
Kubelet's continuous CNI ADD retries for those pending pods drove a request
queue that exceeded the upstream default 50Mi limit on kube-multus-ds. Multus
OOMKilled (exit 137), restarted with an even bigger backlog, OOMKilled again,
positive feedback loop. Restart counts climbed to 276 / 412 / 261 across the
3 RKE2 nodes.

Downstream blast radius: both Traefik pods stuck ContainerCreating (101m +
4h35m), all Longhorn CSI attacher/provisioner/instance-manager stuck, every
Prometheus blackbox probe for *.iamworkin.lan failing, UpdateCenterPublicEdgeDown
critical on update.flowercore.io, every ArgoCD app showing sync=Unknown
because repo-server lost git connectivity. 45 firing Prometheus alerts.

Recovery sequence (Q-MR-1 from FlowerCore.Notes morning routine):
1. kubectl patch kube-multus-ds memory live (this commit locks it in git so
   ArgoCD doesn't revert on next sync)
2. Force-delete the 219 orphan pods (kubectl --grace-period=0 --force) to
   break the avalanche
3. Rollout restart kube-multus-ds — STABLE after restart with new limit
4. Restart Traefik + Longhorn CSI to clear stuck ContainerCreating
5. Verify update.flowercore.io returns 200 + ArgoCD apps reconcile

Tested incrementally: 256Mi limit was insufficient (still OOMed on catchup
burst), 512Mi was insufficient on rke2-agent1 (most pods concentrated there),
1Gi/512Mi handled the full 200+ pending pod CNI catchup cleanly with 0 multus
restarts after rollout. Nodes are 64GB with <25% used in steady-state, so the
~256Mi typical working-set is well within the new limit.

Companion change: FlowerCore.RemoteDesktop must set OwnerReferences on every
worker pod so future operator crashes don't leak orphans (Q-MR-2). Preventive
alerts (Q-MR-3) MultusMemoryPressure + NamespacePendingPodBacklog are coming
in a follow-up commit to apps/monitoring/.

Memory: feedback_multus_50mi_limit_oom_orphan_pod_avalanche
Decisions card: docs/dashboards/decisions-waiting.html Q-MR-1..3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 10:30:05 -05:00
Codex
667777a653 revert(ci1): back to cdrom:scsi (virtio-blk disk hit QEMU flock)
The virtio-blk disk swap (commit 84c9feb) didn't help: qemu fails to
acquire the write lock on the rootdisk PVC because the previous
launcher's qemu process didn't release it cleanly. Same family of
bug as the "stale QEMU flock" already documented in
feedback_kubevirt_iso_first_install_bootorder_and_runstrategy, but
now triggered on rke2-agent1 instead of agent2.

OVMF cdrom timeout is the real blocker and remains open:
  -  Distribution pipeline (build → save → scp → ctr import on all
    3 RKE2 nodes) is proven. localhost/win-server-2025:1.0 lives in
    each node's containerd k8s.io namespace.
  -  containerDisk + cdrom:scsi gets qemu domain Running (no NFS
    Permission denied, no rootdisk flock).
  -  OVMF BdsDxe times out reading the SCSI cdrom regardless of
    SecureBoot setting and bus type.

Reverting the disk type to cdrom:scsi so the VM lands back on the
"qemu Running, OVMF stuck at Boot Manager" state — known-stable and
easier to attack than the QEMU-flock state we hit by trying
virtio-blk disk.

Operator decision for next architectural step (one of):
  - Custom OVMF firmware build with longer Boot0001 timeout
  - KubeVirt version bump (v1.5+ has OVMF fixes)
  - Hyper-V/VirtualBox install + export VHD to ci1
  - BIOS legacy boot (Win Server 2025 needs UEFI but install media
    has a BIOS path)
  - DataVolume HTTP datasource (CDI internalizes ISO bytes via
    different code path)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:35:00 -05:00
Codex
84c9feb893 fix(ci1): present ISO as virtio-blk disk instead of cdrom
OVMF BdsDxe "starting Boot0001 ... Time out" persists across:
  - SATA cdrom + Longhorn Filesystem PVC (Path A)
  - SATA cdrom + Synology NFS (Path B failed: storage perms)
  - SCSI cdrom + Longhorn (Path B variant)
  - SCSI cdrom + containerDisk tmpfs (Path C)
  - + SecureBoot=false

That rules out: storage IO speed, cdrom bus type, signature
verification. Remaining cause is deeper in qemu's cdrom device
emulation under KubeVirt v1.4.0's OVMF firmware — the cdrom read
window for OVMF's first-sector probe is too short to satisfy from
the cdrom controller path regardless of bus type.

Workaround: present the ISO bytes as a regular virtio-blk DISK
(not a cdrom). UEFI/OVMF still recognizes ISO9660 + El Torito
boot records on any block device, so it can find and boot the
EFI bootloader the same way it would from a USB stick. virtio-blk
has a different read path that doesn't hit the cdrom-specific
timeout.

This also better aligns with the FlowerCore.Distribution USB-key
pattern: ISO bytes on a block device, UEFI boots from the El
Torito boot record, Windows installer takes over. The autounattend
ConfigMap (ci1-autounattend) drives unattended Windows setup once
the installer kicks off.

The containerDisk OCI image (localhost/win-server-2025:1.0)
remains unchanged — only the disk type in the VM spec changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:29:59 -05:00
Codex
427dbfcef2 [uc] Phase 1 auth gate deploy v20260509-4162dca-authgate 2026-05-08 21:16:54 -05:00
Codex
b651a4e2d0 fix(ci1): disable SecureBoot to allow OVMF to boot Windows ISO
containerDisk delivery (commit b998f50) successfully gave qemu fast
in-memory access to the ISO bytes (no NFS denial, no Longhorn read
latency), but OVMF's BdsDxe still timed out:

  BdsDxe: loading Boot0001 "UEFI QEMU QEMU CD-ROM " from
    PciRoot(0x0)/Pci(0x2,0x4)/Pci(0x0,0x0)/Scsi(0x0,0x0)
  BdsDxe: starting Boot0001 ... Time out

That rules out storage IO speed and bus type as causes (already
tested both sata and scsi against both Longhorn-PVC and tmpfs-backed
containerDisk). Remaining likely cause: SecureBoot signature
verification on the ISO's EFI bootloader. KubeVirt's stock
`/usr/share/OVMF/OVMF_VARS.secboot.fd` doesn't appear to ship with
the Microsoft KEK/DB enrolled by default, so signed Windows EFI
bootloaders fail the trust-chain check and OVMF reports a generic
"Time out" rather than a verification failure.

Disabling SecureBoot lets OVMF skip the chain check entirely and
boot the El Torito EFI image. SMM stays enabled (KubeVirt only
requires it WITH SecureBoot, not the inverse). TPM 2.0 emulation
also stays on (`tpm: {}`), so BitLocker, Hyper-V, and WSL2 still
work in the guest.

This is acceptable for a CI runner. Long-term path back to
SecureBoot:
  1. Custom-build OVMF_VARS.fd with Microsoft KEK/DB pre-enrolled
  2. Mount via firmware.bootloader.efi.persistent
  3. secureBoot: true

Tracked as a Phase 2 hardening task once the runner is doing real
work and we want signed-boot guarantees.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:06:18 -05:00
Codex
b998f50f48 fix(ci1): switch ISO delivery to containerDisk OCI image (Path C)
OCI image: localhost/win-server-2025:1.0 (8.27 GB)
Built FROM scratch + ADD disk.img → /disk/disk.img on noc1, podman
saved as tar (8.27 GB), SCP'd in parallel to all 3 RKE2 nodes,
imported via ctr in k8s.io namespace. Verified present on all 3
schedulable nodes (rke2-server, rke2-agent1, rke2-agent2).

Why containerDisk over the prior PVC paths:
  - Path A (Longhorn Filesystem PVC, sata): OVMF BdsDxe SATA-CDROM
    read timeout. Cdrom-backed PVC is too slow for OVMF's first-sector
    read window.
  - Path B (Synology NFS): uid 107 (qemu) denied at directory level by
    Synology export ACL despite file mode 0777. Memory:
    feedback_synology_iso_export_root_only_uid_107_denied.
  - Path B+SCSI: same OVMF timeout, just on SCSI controller. Bus
    choice was not load-bearing — the issue was always the slow PVC
    backing.
  - Path C (this commit): containerDisk delivers the ISO bytes from
    a tmpfs view of the OCI layer, no PVC controller in the read path.
    qemu reads at native FS speed; OVMF first-sector read completes
    well within timeout. This is also the KubeVirt-recommended pattern
    for installer ISOs.

Connects to FlowerCore.Distribution / Provisioning USB story: same
"OCI image of the OS installer + autounattend on a sysprep CDROM"
pattern that the USB provisioning agent will use. The Windows
install proceeds hands-off via the existing autounattend.xml in
ci1-autounattend ConfigMap (RDP enabled, WinRM, UAC disabled,
Administrator password from 1Password vault item
h3ix4mgfk65gmkcmvh6ly3d3hu).

Image lifecycle: bump tag (1.1, 1.2, ...) when ISO version changes,
rebuild on noc1, redistribute to RKE2 nodes, update image: line.

Legacy NFS PVC + PV manifest and CDI Longhorn PVC RETAINED for this
commit so prior states are recoverable. Will prune in follow-up
once containerDisk boot proves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:45:38 -05:00
Codex
8fd9ae1cd3 fix(ci1): revert NFS Path B + flip ISO cdrom bus sata→scsi
NFS Path B (commit fc2aca0) failed at storage layer: Synology export
`/volume1/ISOs` denies non-root client UIDs at the directory level.
qemu uid 107 cannot `ls /iso/` even though disk.img is mode 0777.

Diagnosed via uid-107 + uid-0 busybox probe pods on rke2-agent2:
- libvirt error: "Cannot access storage file ... Permission denied"
  (virStorageSourceReportBrokenChain:1281, virError Code=38 Domain=18)
- uid 107 pod: "ls: can't open '/iso/': Permission denied"
- uid 0 pod (same mount): "drwxrwxrwx 1 root root 16 ... disk.img"
- SELinux Enforcing + virt_use_nfs=on, no AVC denials → not SELinux
- File mode 0777 with owner 107:107 → not POSIX

Same export-only-root pattern as `/volume1/kubernetes`. Memory:
feedback_synology_iso_export_root_only_uid_107_denied.md

Existing CDI-uploaded Longhorn PVC `windows-server-2025-iso` (10Gi
Filesystem mode) verified to contain valid ISO bytes readable by
uid 107 (mode 0660 root:107, 9.85 GB sparse, 8.27 GB blocks ≈
original 7.7 GB ISO). Reverting to it.

The original OVMF SATA-CDROM read timeout that drove yesterday's
NFS pivot is now addressed by `cdrom: bus: scsi` (virtio-scsi has
a longer read window than the IDE/SATA emulator). Per user-prompt
diagnostic chain Step 5.

NFS PVC + PV (apps/kubevirt-vms/win2025-iso-nfs-pv.yaml) RETAINED
so Path B state is recoverable; can be pruned in follow-up once
SCSI boot is proven.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 18:54:36 -05:00
Codex
fc2aca0e9e fix(ci1): mount Windows ISO via Synology NFS (Path B for SATA-CDROM timeout)
Previous fix attempts confirmed the Longhorn-backed Filesystem PVC contains
a perfectly valid bootable ISO9660 image. The bug is that SATA-CDROM
emulation reading from a Longhorn Filesystem PVC is too slow for OVMF's
boot read window — DVD-ROM enumeration times out before the bootloader
loads. Symptom on the serial console:
  BdsDxe: failed to start Boot0001 "UEFI QEMU DVD-ROM QM00001 " ... Time out
  BdsDxe: No bootable option or device was found

Block-mode PVC (Path A) was attempted and would likely fix the timing,
but CDI v1.65.0's upload-target pod cannot open the underlying block
device (runAsUser:107 + capabilities.drop:[ALL]):
  blockdev: cannot open /dev/cdi-block-volume: Permission denied

Path B (this change): mount the ISO directly from Synology NAS over
NFSv4.1. Bypasses both the Longhorn slowness and the CDI permission
issue. QEMU's SATA emulator reads at native LAN speed.

Layout:
  /volume1/ISOs/ — existing Synology export, RKE2 ACL already granted
  /volume1/ISOs/win2025-iso-disk/disk.img — new subdir, hardlink to the
    ISO file, named so KubeVirt's launcher finds it at the PV root

A hardlink (not symlink) is required because symlinks with relative
targets pointing to the parent directory are broken when the NFS PV
sub-mounts the subdir as its root.

Validated 2026-05-08 from rke2-server, rke2-agent1, rke2-agent2:
  mount -t nfs -o nfsvers=4.1,ro 10.0.58.3:/volume1/ISOs/win2025-iso-disk
  file disk.img -> ISO 9660 CD-ROM filesystem data ... (bootable)

The original Longhorn Filesystem ISO PVC is RETAINED unused (so ArgoCD
doesn't prune the populated PVC and so we have a fallback). Can be
removed in a follow-up commit after the NFS path is proven on a
successful Windows install.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:03:42 -05:00
Codex
ba18c52130 docs(ci1): record open rootdisk-flock and SATA-CDROM-timeout issues
Documenting the remaining 2 unresolved issues for the next operator
session, with the recovery paths from this session captured inline so
the next agent doesn't repeat the same blind alleys:

1. **rootdisk QEMU flock** — every new launcher pod fails QEMU start with
   `Failed to get "write" lock` on the rootdisk Filesystem-mode disk.img.
   Stale flock from a previous force-deleted virt-launcher pod. Longhorn
   engine on rke2-agent2 needs to release the lock; `kubectl patch
   volume.longhorn.io/<pvc-name> spec.nodeID=""` is reverted by the
   Longhorn controller. Operator-level recovery only.

2. **SATA CDROM read timeout** — even with bootOrder=1 (windows-iso first),
   OVMF UEFI fails Boot0001 with "Time out" reading the SATA CDROM backed
   by the Filesystem-mode PVC. Block-mode DataVolume migration was
   attempted but blocked by CDI v1.65.0's upload pod running with
   `capabilities.drop: [ALL]` and `runAsUser: 107`, preventing direct
   block-device writes (`blockdev: cannot open /dev/cdi-block-volume:
   Permission denied`). See ISO PVC header docstring for 3 forward paths.

Net commits during this session:
- 1c4145a: bootOrder swap (windows-iso=1, rootdisk=2)
- 87a7d7c: deprecated `running:` -> `runStrategy: Always`
- 0bf47df: ISO migration to Block-mode DataVolume (REVERTED)
- 9f6dc1a: revert to Filesystem PVC (CDI block-upload blocked)
- 1c4145a + 87a7d7c + 9f6dc1a are the live, correct configuration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 15:18:38 -05:00
Codex
9f6dc1a9d5 fix(ci1): revert ISO to Filesystem PVC; CDI v1.65.0 block-upload pod blocked by capability drop
The Block-mode DataVolume migration (commit 0bf47df) hit a CDI v1.65.0 limitation:
the upload-target pod runs as uid 107 with `capabilities.drop: [ALL]`, so it
cannot open the underlying block device:

  blockdev: cannot open /dev/cdi-block-volume: Permission denied
  Saving stream failed: Unable to transfer source data to target file:
  error determining if block device exists: exit status 1

Reverting to a Filesystem-mode PVC + virtctl image-upload pvc, which DID work
(uploaded the 7.7 GiB ISO with valid ISO9660 magic intact). Boot timeout is
unresolved (header docstring captures the open issue + 3 paths to revisit).

The bootOrder swap (1c4145a) and runStrategy migration (87a7d7c) stay landed —
those are correct improvements regardless of the volume-mode question.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 14:32:52 -05:00
Codex
0bf47dfa33 fix(ci1): switch ISO from filesystem PVC to Block-mode DataVolume
The bootOrder swap alone didn't fix the install — even with `windows-iso` at
bootOrder:1, OVMF UEFI still timed out reading the SATA CDROM:

  BdsDxe: starting Boot0001 "UEFI QEMU DVD-ROM QM00001 " from ... Sata(...)
  BdsDxe: failed to start Boot0001 ... : Time out
  BdsDxe: No bootable option or device was found.

Diagnosis (debug pod mounting the live PVC):
- /pvc/disk.img IS a valid bootable ISO9660 image — `file` reports
  "ISO 9660 CD-ROM filesystem data 'SSS_X64FRE_EN-US_DV9' (bootable)".
- bytes 0..15: zeros (NOT QCOW2 magic 51 46 49 fb).
- bytes 32769..32773: "CD001" — ISO9660 primary volume descriptor at the
  correct offset.

So content was fine. The bug is in how KubeVirt + QEMU + Longhorn expose a
Filesystem-mode PVC's `/disk.img` as a SATA CDROM. With Block-mode the
underlying volume IS the raw ISO9660 sectors, OVMF reads them directly,
no QEMU file-emulation layer. This is the recommended pattern for ISO
install media on KubeVirt + Longhorn.

Migration:
- Replace `kind: PersistentVolumeClaim` with `kind: DataVolume` (CDI manages
  the underlying PVC + upload-target pod).
- Set `pvc.volumeMode: Block`.
- Annotate `cdi.kubevirt.io/storage.contentType: kubevirt` so CDI keeps raw
  bytes (no QCOW2 wrap).
- VM volume reference changes from `persistentVolumeClaim.claimName` to
  `dataVolume.name`. KubeVirt's VMI controller blocks VM start until DV
  phase is Succeeded (upload completed).

Operator step after this lands:
1. Wait for DV `phase: UploadReady`
   kubectl get dv -n kubevirt-vms windows-server-2025-iso -w
2. virtctl image-upload dv windows-server-2025-iso -n kubevirt-vms \
     --image-path "...\en-us_windows_server_2025...iso" \
     --uploadproxy-url https://localhost:8443 --insecure --no-create
3. Re-flip runStrategy to Always (was set to Halted live-side during
   migration; this commit keeps the manifest at Always).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 14:23:31 -05:00
Codex
87a7d7c70a fix(ci1): switch deprecated running: true -> runStrategy: Always
Required to clear OutOfSync state after the bootOrder fix. Live VM had
runStrategy: Halted (set during diagnosis to release the PVC for inspection).
Manifest had running: true. KubeVirt's validating webhook rejects sync:
  admission webhook "virtualmachine-validator.kubevirt.io" denied the request:
  Running and RunStrategy are mutually exclusive.

Switching to runStrategy: Always preserves the original "auto-start +
auto-restart" semantics with the non-deprecated field, and gives ArgoCD a
clean diff target to flip Halted -> Always.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 14:12:07 -05:00
Codex
1c4145a581 fix(ci1): swap bootOrder so Windows install ISO boots first
Original order: rootdisk=1 (empty 200Gi virtio), windows-iso=2 (SATA CDROM).
UEFI tried the empty virtio disk first, got nothing, fell back to Boot0001
(the SATA CDROM) with a short timeout, and aborted with:
  BdsDxe: failed to start Boot0001 ... Time out
  BdsDxe: No bootable option or device was found.

VM had been running 38+ min with rootdisk actualSize stuck at 4.13 GiB and
no AgentConnected condition — install never started.

Diagnosis via debug pod mounting the windows-server-2025-iso PVC:
  /pvc/disk.img: ISO 9660 CD-ROM filesystem data 'SSS_X64FRE_EN-US_DV9' (bootable)
  bytes 0..15: zeros (NOT QCOW2 magic 51 46 49 fb)
  bytes 32769..32773: "CD001" (ISO9660 primary volume descriptor)

So the PVC content is a real bootable ISO — the only fix needed is to make
the ISO bootOrder=1 for first install. After Windows installs, it writes its
own UEFI Boot#### entries pointing at the rootdisk EFI partition; UEFI then
boots from rootdisk going forward and the ISO at bootOrder:2 is a fallback
for re-install scenarios.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 14:10:17 -05:00
Codex
c50a403f74 fix(infra): pin virtio-container-disk to v1.8.2 (containerd 2.1 manifest fix)
KubeVirt v1.4.0 + RKE2 containerd 2.1.5 cannot pull
quay.io/kubevirt/virtio-container-disk:latest:
  rpc error: code = Unimplemented
  desc = failed to pull and unpack image: not implemented:
  media type "application/vnd.docker.distribution.manifest.v1+prettyjws"
  is no longer supported since containerd v2.1, please rebuild the image as
  "application/vnd.docker.distribution.manifest.v2+json" or
  "application/vnd.oci.image.manifest.v1+json"

The :latest tag was last rebuilt with the v1 manifest schema. Tagged versions
v1.6.5+, v1.7.3, v1.8.2 are rebuilt with v2/OCI manifests.

Pinning to v1.8.2 (newest available, contains current Windows VirtIO drivers).
The image only contains the Windows VirtIO driver ISO mounted as a CDROM —
not the KubeVirt runtime — so it is decoupled from the cluster KubeVirt
version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 13:28:22 -05:00
Codex
fb7bd10528 feat(infra): activate ci1 VM — running:true + 10Gi ISO PVC + 1P password
Phase 1 prereqs all satisfied:
- Multus CNI v4.2.2 thick-plugin DS Running on rke2-server/agent1/agent2
- CDI v1.65.0 operator + CR Deployed (cdi-apiserver/deployment/uploadproxy
  all Running 1/1)
- Windows Server 2025 ISO (7.7GiB, March 2026 update) uploaded via CDI
  virtctl image-upload to PVC windows-server-2025-iso. Verified via PVC
  annotations: cdi.kubevirt.io/storage.condition.running.message="Upload
  Complete", storage.pod.phase="Succeeded"
- Local Administrator password generated (26 char, FANTASTIC strength).
  Stored in 1Password vault IAmWorkin (qaphopopkryhbg353ukzhhuqoq) item
  h3ix4mgfk65gmkcmvh6ly3d3hu. UTF-16-LE base64 in autounattend.xml Value
  field matches the 1P "autounattend AdministratorPassword Value" field.

Changes:
- ISO PVC bumped 6Gi → 10Gi (ISO is 7.7GiB, need headroom)
- Added labels app=ci-runner, flowercore.io/managed-by=bluejay-infra
- autounattend.xml AdministratorPassword Value: real base64-encoded password
- spec.running: false → true (VM starts on next ArgoCD sync)
- Header comment refreshed to LIVE state with prereq references

Network: still pod-network masquerade. Multus NAD prod-vlan57 is registered
but the VM doesn't use it yet (Phase 1.5 host bridge needed first).

Verify after sync:
  kubectl --kubeconfig $env:USERPROFILE\.kube\rke2.yaml -n kubevirt-vms get vm,vmi
  virtctl --kubeconfig $env:USERPROFILE\.kube\rke2.yaml vnc ci1 -n kubevirt-vms

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 13:24:46 -05:00
Codex
6c21d14a98 deploy(fc-updater): bump image to v20260508-pub3-deepening-2bdf108
Promotes the fleet to FlowerCore.Updater main @ 2bdf108 which combines:
- PR #6 publish pre-signed releases (1a188f4)
- PR #7 deeper public-host privacy enforcement (8cd8544)
- PublishPreSignedAsync(stream, sig) Integration coverage (2bdf108)

Live image already imported to rke2-server and rolled via deploy-web.ps1.
This commit aligns the bluejay-infra source of truth so ArgoCD doesn't
snap the spec back to the previous tag (per
feedback_argocd_managed_image_overrides_do_not_stick).
2026-05-08 13:07:24 -05:00
Codex
b3529f8e96 feat(infra): add Multus CNI + CDI + PROD VLAN 57 NAD as GitOps prereqs for ci1
Adds three new bluejay-infra apps that auto-pickup via ApplicationSet (apps/*
directory generator on main):

* apps/multus/multus.yaml — Multus CNI v4.2.2 thick-plugin daemonset (verbatim
  upstream, project-annotated). Enables KubeVirt VMs to attach additional
  network interfaces. Required by ci1 to bridge onto PROD VLAN 57.

* apps/cdi/{cdi-operator.yaml,cdi-cr.yaml,README.md} — Containerized Data
  Importer v1.65.0 (verbatim upstream). Operator + CR pattern. Enables
  populating PVCs from HTTP/registry/upload sources, used to load the Windows
  Server 2025 ISO into the windows-server-2025-iso PVC.

* apps/kubevirt-vms/prod-vlan57-nad.yaml — NetworkAttachmentDefinition for
  PROD VLAN 57 bridge. **Deploy gated on Phase 1.5 host work**: requires
  br-prod bridge enslaving enp86s0.57 on each RKE2 node (Puppet config-as-code).
  ci1.yaml continues to use pod-network masquerade until that lands; switching
  to multus.networkName: kubevirt-vms/prod-vlan57 is a one-line YAML edit
  followed by a GitOps push.

Cluster verification (2026-05-08):
- KubeVirt LIVE (3 nodes, virt-api/controller/handler/operator all Running)
- Calico CNI on /etc/cni/net.d + /opt/cni/bin (Multus default paths)
- ApplicationSet `bluejay-infra` already watches `apps/*` on main

Reproducibility: upstream YAMLs vendored verbatim with project header diffs
only. Bumping versions = re-curl + git push. No deploy-time internet fetch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 13:05:58 -05:00
Codex
00c11b4eaa feat(infra): stage ci1 Windows Server 2025 KubeVirt VM (Phase 1, NOT YET APPLIED)
Stages a draft VirtualMachine + Namespace + ISO PVC + rootdisk PVC + sysprep
ConfigMap for the dedicated GitHub Actions self-hosted runner that replaces
the never-registered bluejay-ws-sandbox-1 placeholder.

Status: STAGED ONLY. spec.running = false. ISO PVC empty. Two operator
decisions still pending before this can boot:
  1. Network choice — pod-network fallback (in this draft) vs Multus +
     PROD VLAN NAD (preferred, requires Multus install).
  2. ISO path — manual upload via helper pod (Path A) vs CDI HTTP import
     (Path B, requires CDI install).

Cluster baseline 2026-05-08:
  - KubeVirt operator: installed, healthy, 14d
  - CDI: NOT installed
  - Multus: NOT installed
  - Calico-only CNI

See docs/infrastructure/windows-server-build-runner-plan.md "Phase 1 readiness
gate" for the full operator pickup checklist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:32:47 -05:00
50 changed files with 7786 additions and 7 deletions

18
apps/authentik/README.md Normal file
View File

@@ -0,0 +1,18 @@
# Authentik OIDC client registration sweep
This directory holds the FlowerCore per-service OIDC client secret references
for the ADR-093 / ADR-124 Phase 1 step 8 sweep.
The `clients/*-oidc-client.yaml` manifests are intentionally only
`OnePasswordItem` CRDs. The actual 1Password items are created by an operator in
the `IAmWorkin` vault with these fields:
| Field | Purpose |
| --- | --- |
| `client_id` | Authentik provider client id, default `<slug>` |
| `client_secret` | Authentik provider client secret |
| `issuer_url` | `https://id.iamworkin.lan/application/o/<slug>/` |
Run `scripts/authentik-bulk-client-create.py` in dry-run mode first. Live REST
mutation requires `--apply`, `AUTHENTIK_TOKEN`, and an operator-provided
client-secret JSON file. The script redacts secrets in all normal output.

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: aistation-oidc-client
namespace: fc-aistation
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: aistation
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/aistation-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/aistation-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: audit-oidc-client
namespace: fc-audit
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: audit
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/audit-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/audit-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: chat-oidc-client
namespace: fc-chat
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: chat
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/chat-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/chat-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: distribution-oidc-client
namespace: fc-distribution
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: distribution
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/distribution-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/distribution-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: dms-oidc-client
namespace: fc-dms
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: dms
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/dms-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/dms-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: dns-oidc-client
namespace: fc-dns
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: dns
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/dns-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/dns-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: intranet-oidc-client
namespace: intranet
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: intranet
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/intranet-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/intranet-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: irc-oidc-client
namespace: irc
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: irc
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/irc-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/irc-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: kiosk-oidc-client
namespace: fc-system
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: kiosk
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/kiosk-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/kiosk-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: knowledge-oidc-client
namespace: knowledge
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: knowledge
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/knowledge-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/knowledge-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: library-oidc-client
namespace: fc-library
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: library
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/library-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/library-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: licensing-oidc-client
namespace: fc-licensing
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: licensing
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/licensing-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/licensing-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: llmbridge-oidc-client
namespace: fc-llm-bridge
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: llmbridge
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/llmbridge-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/llmbridge-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: media-oidc-client
namespace: fc-media
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: media
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/media-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/media-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: menuboard-oidc-client
namespace: fc-menuboard
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: menuboard
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/menuboard-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/menuboard-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: messageboard-oidc-client
namespace: fc-messageboard
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: messageboard
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/messageboard-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/messageboard-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: mike-bundle-oidc-client
namespace: fc-mike-bundle
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: mike-bundle
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/mike-bundle-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/mike-bundle-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: mndot-oidc-client
namespace: fc-mndot
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: mndot
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/mndot-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/mndot-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: mysql-oidc-client
namespace: fc-mysql
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: mysql
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/mysql-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/mysql-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: php-oidc-client
namespace: fc-php
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: php
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/php-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/php-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: pimanager-oidc-client
namespace: fc-pimanager
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: pimanager
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/pimanager-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/pimanager-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: presentations-oidc-client
namespace: fc-presentations
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: presentations
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/presentations-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/presentations-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: print-oidc-client
namespace: fc-print
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: print
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/print-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/print-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: provisioning-oidc-client
namespace: fc-provisioning
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: provisioning
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/provisioning-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/provisioning-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: remotedesktop-oidc-client
namespace: fc-desktop
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: remotedesktop
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/remotedesktop-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/remotedesktop-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: retail-oidc-client
namespace: fc-retail
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: retail
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/retail-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/retail-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: scoreboards-oidc-client
namespace: fc-scoreboard
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: scoreboards
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/scoreboards-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/scoreboards-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: segmentdisplay-oidc-client
namespace: fc-segmentdisplay
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: segmentdisplay
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/segmentdisplay-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/segmentdisplay-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: signage-oidc-client
namespace: fc-signage
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: signage
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/signage-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/signage-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: signalcontrol-oidc-client
namespace: fc-signalcontrol
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: signalcontrol
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/signalcontrol-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/signalcontrol-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: telephony-oidc-client
namespace: telephony
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: telephony
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/telephony-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/telephony-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: ttsreader-oidc-client
namespace: fc-ttsreader
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: ttsreader
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/ttsreader-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/ttsreader-oidc-client"

View File

@@ -0,0 +1,14 @@
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
name: worldbuilder-oidc-client
namespace: fc-worldbuilder
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/component: authentik-oidc-client
flowercore.io/authentik-client-slug: worldbuilder
annotations:
flowercore.io/onepassword-item: "IAmWorkin/items/worldbuilder-oidc-client"
flowercore.io/expected-fields: "client_id,client_secret,issuer_url"
spec:
itemPath: "vaults/IAmWorkin/items/worldbuilder-oidc-client"

View File

@@ -0,0 +1,38 @@
# ArgoCD's bluejay-infra ApplicationSet sees apps/authentik as one app. Keep
# an explicit resource list so the client manifests can live under clients/.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- clients/library-oidc-client.yaml
- clients/retail-oidc-client.yaml
- clients/telephony-oidc-client.yaml
- clients/knowledge-oidc-client.yaml
- clients/llmbridge-oidc-client.yaml
- clients/mysql-oidc-client.yaml
- clients/php-oidc-client.yaml
- clients/signage-oidc-client.yaml
- clients/media-oidc-client.yaml
- clients/dms-oidc-client.yaml
- clients/pimanager-oidc-client.yaml
- clients/distribution-oidc-client.yaml
- clients/dns-oidc-client.yaml
- clients/print-oidc-client.yaml
- clients/aistation-oidc-client.yaml
- clients/irc-oidc-client.yaml
- clients/ttsreader-oidc-client.yaml
- clients/chat-oidc-client.yaml
- clients/intranet-oidc-client.yaml
- clients/remotedesktop-oidc-client.yaml
- clients/provisioning-oidc-client.yaml
- clients/scoreboards-oidc-client.yaml
- clients/mndot-oidc-client.yaml
- clients/kiosk-oidc-client.yaml
- clients/mike-bundle-oidc-client.yaml
- clients/messageboard-oidc-client.yaml
- clients/menuboard-oidc-client.yaml
- clients/presentations-oidc-client.yaml
- clients/segmentdisplay-oidc-client.yaml
- clients/signalcontrol-oidc-client.yaml
- clients/worldbuilder-oidc-client.yaml
- clients/audit-oidc-client.yaml
- clients/licensing-oidc-client.yaml

69
apps/cdi/README.md Normal file
View File

@@ -0,0 +1,69 @@
# CDI — Containerized Data Importer
KubeVirt's `containerized-data-importer` for populating PVCs from external
sources (HTTP, HTTPS, container registry, S3, virtctl upload). Required to
import the Windows Server 2025 ISO into the `windows-server-2025-iso` PVC
that `apps/kubevirt-vms/ci1.yaml` mounts as a CDROM.
## Files
| File | Source | Purpose |
| ----------------- | ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `cdi-operator.yaml` | [`v1.65.0`](https://github.com/kubevirt/containerized-data-importer/releases/tag/v1.65.0) — verbatim copy | Installs operator + CRDs (5779 lines, large) |
| `cdi-cr.yaml` | [`v1.65.0`](https://github.com/kubevirt/containerized-data-importer/releases/tag/v1.65.0) — annotated + commented | Tells operator to deploy CDI components |
`cdi-operator.yaml` is **vendored verbatim** from the upstream release for
air-gap reproducibility (no internet fetch at deploy time, ArgoCD prune
contracts hold). To bump versions:
```bash
CDI_VER=v1.66.0 # for example
curl -sL "https://github.com/kubevirt/containerized-data-importer/releases/download/${CDI_VER}/cdi-operator.yaml" \
-o apps/cdi/cdi-operator.yaml
curl -sL "https://github.com/kubevirt/containerized-data-importer/releases/download/${CDI_VER}/cdi-cr.yaml" \
-o /tmp/cdi-cr-new.yaml # then re-apply project header diff
git diff apps/cdi/ # review
git commit + push
```
## Verify after deploy
```bash
kubectl -n cdi get pods # operator + apiserver + deployment + uploadproxy
kubectl get cdis cdi -o jsonpath='{.status.phase}' # "Deployed"
kubectl get crd | grep cdi.kubevirt.io
# Expected CRDs: datavolumes.cdi.kubevirt.io, cdiconfigs.cdi.kubevirt.io,
# storageprofiles.cdi.kubevirt.io, dataimportcrons.cdi.kubevirt.io,
# datasources.cdi.kubevirt.io, objecttransfers.cdi.kubevirt.io
```
## Use after install
```yaml
# Example DataVolume that imports from HTTP
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: my-iso
spec:
source:
http:
url: "https://server/path/to.iso"
pvc:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 10Gi
storageClassName: longhorn
```
```bash
# Or upload from local disk via virtctl
virtctl image-upload pvc my-iso \
--image-path ./my.iso \
--size 10Gi \
--storage-class longhorn \
--access-mode ReadWriteOnce \
--uploadproxy-url https://cdi-uploadproxy.cdi.svc:443 \
--insecure
```

36
apps/cdi/cdi-cr.yaml Normal file
View File

@@ -0,0 +1,36 @@
# =============================================================================
# CDI CR — Tells the CDI operator to install CDI components into the cluster.
# =============================================================================
# After cdi-operator.yaml is applied, the operator watches for THIS resource
# (CDI named "cdi"). When found, it deploys cdi-apiserver, cdi-deployment,
# cdi-uploadproxy, cdi-cronjob, and the importer/uploadserver/cloner pods.
#
# Configuration:
# - HonorWaitForFirstConsumer: PVCs created by DataVolumes wait for first
# pod to schedule before binding (lets storage class pick best node).
# - WebhookPvcRendering: validates PVC creation against CDI policies.
# - imagePullPolicy IfNotPresent: re-pull only on tag rotation.
# - nodeSelector linux: pin to Linux nodes (no Windows worker support).
#
# Andrew may want to add a `uploadProxyURLOverride` later to expose the
# uploadproxy via Traefik IngressRoute for `virtctl image-upload` from
# BLUEJAY-WS without `kubectl port-forward`. Phase 2 enhancement.
# =============================================================================
apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata:
name: cdi
annotations:
bluejay.iamworkin.lan/source: "kubevirt/containerized-data-importer v1.65.0"
spec:
config:
featureGates:
- HonorWaitForFirstConsumer
- WebhookPvcRendering
imagePullPolicy: IfNotPresent
infra:
nodeSelector:
kubernetes.io/os: linux
workload:
nodeSelector:
kubernetes.io/os: linux

5779
apps/cdi/cdi-operator.yaml Normal file

File diff suppressed because it is too large Load Diff

171
apps/fc-redis/fc-redis.yaml Normal file
View File

@@ -0,0 +1,171 @@
# fc-redis — SignalR backplane for cross-product event bus
#
# Lands per Q-SO-1 resolution (2026-05-11 PM): SignalR backplane in Phase A,
# not Phase C as originally drafted. Operator directive: "Redis can be
# deployed just fine as it's another FlowerCore technology we'll want to
# manage."
#
# Phase A scope (this file):
# - Single Redis 7.x Alpine pod
# - 1Gi Longhorn RWO PVC for AOF persistence
# - ClusterIP Service at `redis.fc-redis.svc.cluster.local:6379`
# - No AUTH (in-cluster only; not exposed externally)
# - No IngressRoute (backplane is server-to-server only)
#
# Consumers (Phase A IMPL across FC services):
# - FlowerCore.Signage.Web (OpsConsoleHub)
# - FlowerCore.Scoreboard.Web (ScoreboardHub)
# - FlowerCore.SignalControl.Web
# - FlowerCore.DMS.Web
# - Any other product joining the cross-product event bus
#
# Each consumer adds:
# services.AddSignalR()
# .AddStackExchangeRedis(
# "redis.fc-redis.svc.cluster.local:6379",
# opts => opts.Configuration.ChannelPrefix =
# StackExchange.Redis.RedisChannel.Literal("fc-opsconsole"));
#
# Phase B / C follow-ons (out of scope here):
# - Redis Sentinel for HA (3-node)
# - AUTH password from 1Password Connect (rotate via /rotate-password)
# - redis_exporter sidecar for Prometheus scrape
# - Network policies restricting which namespaces can dial 6379
#
# Design: docs/signage/operations-console-phase-2-design.md §3.5
# Decision: Q-SO-1 (RESOLVED 2026-05-11 PM)
# Memory: feedback_blooming_ui_pattern_no_iframes
---
apiVersion: v1
kind: Namespace
metadata:
name: fc-redis
labels:
app.kubernetes.io/part-of: flowercore
app.kubernetes.io/managed-by: argocd
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fc-redis-data
namespace: fc-redis
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fc-redis-config
namespace: fc-redis
data:
redis.conf: |
# Phase A — minimal config; no AUTH, no replication.
bind 0.0.0.0
protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
# Persistence: AOF (fsync every second is the standard SignalR-backplane
# durability sweet spot — the backplane only needs to survive Redis
# restarts, not absolute zero loss).
appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# Reasonable defaults — let Redis pick most things.
maxmemory-policy allkeys-lru
maxmemory 256mb
# Logging
loglevel notice
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fc-redis
namespace: fc-redis
labels:
app: fc-redis
spec:
replicas: 1
strategy:
type: Recreate # RWO PVC; do not do rolling update
selector:
matchLabels:
app: fc-redis
template:
metadata:
labels:
app: fc-redis
spec:
securityContext:
runAsNonRoot: true
runAsUser: 999 # redis:7-alpine default uid
runAsGroup: 999
fsGroup: 999
containers:
- name: redis
image: redis:7-alpine
imagePullPolicy: IfNotPresent
command: ["redis-server", "/etc/redis/redis.conf"]
ports:
- name: redis
containerPort: 6379
resources:
requests:
cpu: "50m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "384Mi"
volumeMounts:
- name: data
mountPath: /data
- name: config
mountPath: /etc/redis
readOnly: true
livenessProbe:
tcpSocket:
port: 6379
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
exec:
command: ["redis-cli", "ping"]
initialDelaySeconds: 2
periodSeconds: 5
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
volumes:
- name: data
persistentVolumeClaim:
claimName: fc-redis-data
- name: config
configMap:
name: fc-redis-config
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: fc-redis
spec:
type: ClusterIP
selector:
app: fc-redis
ports:
- name: redis
port: 6379
targetPort: 6379
protocol: TCP

View File

@@ -58,7 +58,7 @@ spec:
nodeName: rke2-server
containers:
- name: web
image: localhost/fc-updater-web:v20260507-public-privacy
image: localhost/fc-updater-web:v20260509-4162dca-authgate
imagePullPolicy: Never
ports:
- containerPort: 8080

View File

@@ -466,11 +466,11 @@ spec:
itemPath: vaults/IAmWorkin/items/Guacamole JSON Auth
---
---
# 1Password-backed credentials for Mac mini VNC access (Phase 1 2026-04-28)
# 1Password-backed credentials for Mac mini VNC access (Phase 1 <EFBFBD> 2026-04-28)
# The operator mints Secret 'macmini-vnc-creds' with keys: username, password, VNC Password
# Note: '1Password' field label 'VNC Password' -> K8s Secret key 'VNC Password' (space retained)
# Guacamole VNC connection password is sourced from the 'VNC Password' field.
# Actual IP is 10.0.56.115 (INFRA VLAN) the 1P item 'IP' field is kept as backup reference.
# Actual IP is 10.0.56.115 (INFRA VLAN) <EFBFBD> the 1P item 'IP' field is kept as backup reference.
apiVersion: onepassword.com/v1
kind: OnePasswordItem
metadata:
@@ -481,6 +481,7 @@ metadata:
app.kubernetes.io/part-of: flowercore
spec:
itemPath: vaults/IAmWorkin/items/Mac Mini
---
# Blue Jay Branding Extension (CSS + translations)
apiVersion: v1
kind: ConfigMap

View File

@@ -0,0 +1,93 @@
# =============================================================================
# ci1 - Windows Server 2025 KubeVirt VM (GitHub Actions Self-Hosted Runner)
# =============================================================================
# Boots from the sysprepped containerDisk template built by the Windows VM
# sysprep pipeline. See docs/infrastructure/windows-vm-sysprep-pipeline.md.
# Path A/B/C install history is preserved in git log only.
# =============================================================================
apiVersion: v1
kind: Namespace
metadata:
name: kubevirt-vms
labels:
app.kubernetes.io/part-of: kubevirt-stack
pod-security.kubernetes.io/enforce: privileged
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: ci1
namespace: kubevirt-vms
labels:
app: ci-runner
role: github-actions-runner
flowercore.io/managed-by: bluejay-infra
spec:
runStrategy: Always
template:
metadata:
labels:
app: ci-runner
role: github-actions-runner
kubevirt.io/vm: ci1
spec:
domain:
cpu:
cores: 8
sockets: 1
threads: 1
memory:
guest: 16Gi
resources:
requests:
memory: 16Gi
limits:
memory: 16Gi
clock:
utc: {}
timer:
hpet:
present: false
pit:
tickPolicy: delay
rtc:
tickPolicy: catchup
hyperv: {}
features:
acpi: {}
apic: {}
hyperv:
relaxed: {}
vapic: {}
spinlocks:
spinlocks: 8191
smm: {}
firmware:
bootloader:
efi:
secureBoot: false
devices:
tpm: {}
disks:
- name: rootdisk
disk:
bus: virtio
interfaces:
# Pod-network fallback for CI runner outbound traffic. Switch to
# prod-vlan57 once the bridge/NAD lane is ready for L2 access.
- name: default
masquerade: {}
model: virtio
machine:
type: q35
networks:
- name: default
pod: {}
volumes:
- name: rootdisk
containerDisk:
image: localhost/fc-win-server-2025:v1
imagePullPolicy: Never
terminationGracePeriodSeconds: 3600

View File

@@ -0,0 +1,3 @@
resources:
- ci1.yaml
- prod-vlan57-nad.yaml

View File

@@ -0,0 +1,69 @@
# =============================================================================
# NetworkAttachmentDefinition — PROD VLAN 57 bridge
# =============================================================================
# Purpose: makes KubeVirt VMs reachable on the PROD VLAN (10.0.57.0/24)
# alongside the existing pod network. Required for ci1 to bridge onto PROD
# (e.g. to provision/scrape edge1, edge2, kiosks, Pis on the same L2 segment).
#
# **DEPLOY GATE — Phase 1.5 host work required first**:
# On every RKE2 node (rke2-server, rke2-agent1, rke2-agent2):
# 1. Switch port (UniFi USL16LP) trunks VLAN 57 to the node — usually
# already true since BLUEJAY-WS reaches 10.0.57.x services. Verify
# with `ip link show enp86s0.57` after configuring sub-interface, OR
# `tcpdump -ni enp86s0 vlan 57` and ping a known PROD host.
# 2. Linux bridge `br-prod` enslaving `enp86s0.57` (VLAN sub-interface).
# NetworkManager profile examples in the runbook below.
# 3. Verify Multus DaemonSet `kube-multus-ds` is Ready on all nodes.
#
# Without those, applying this NAD has no effect except to register the CRD.
# A VM that requests this NAD with no bridge present will fail with:
# `error adding pod kubevirt-vms_ci1 to CNI network "prod-vlan57": failed to
# plumb VLAN: open /sys/class/net/br-prod/master: no such file or directory`
#
# Configuration notes:
# - cniVersion 0.3.1 to match Multus daemon-config.json
# - mtu 1500 (matches enp86s0 default; bump if jumbo frames configured)
# - bridge name `br-prod` is convention; if Puppet picks a different name
# (e.g. `br57`, `br-vlan57`), edit BOTH this NAD and the ci1.yaml
# interface block. Keep them in sync.
# - vlan: 0 because the host bridge already strips VLAN tag (br-prod sits
# on top of `enp86s0.57`). If we instead used a VLAN-aware bridge with
# trunk port, set vlan: 57 here. Current convention is VLAN-stripped at
# the sub-interface, so the bridge passes untagged frames.
#
# Apply:
# kubectl --kubeconfig $env:USERPROFILE\.kube\rke2.yaml apply -f apps/kubevirt-vms/prod-vlan57-nad.yaml
#
# Then update ci1.yaml networks: stanza to:
# - name: prod-net
# multus:
# networkName: kubevirt-vms/prod-vlan57
# and the interface block from `masquerade` to `bridge`.
# =============================================================================
---
# Namespace must exist already (created by ci1.yaml's first document).
# This file imports a NAD into that same namespace.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: prod-vlan57
namespace: kubevirt-vms
annotations:
bluejay.iamworkin.lan/host-bridge: "br-prod (enslaves enp86s0.57)"
bluejay.iamworkin.lan/cidr: "10.0.57.0/24"
bluejay.iamworkin.lan/gateway: "10.0.57.1"
bluejay.iamworkin.lan/dns: "10.0.56.1 (pfSense Unbound)"
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "prod-vlan57",
"type": "bridge",
"bridge": "br-prod",
"ipam": {},
"mtu": 1500,
"vlan": 0,
"promiscMode": true,
"preserveDefaultVlan": false
}

View File

@@ -0,0 +1,99 @@
# =============================================================================
# Windows Server 2025 ISO — Static NFS PV (Path B for SATA-CDROM timeout)
# =============================================================================
# Purpose: Mount the ISO from Synology NAS via NFS instead of from a Longhorn-
# backed Filesystem PVC.
#
# Why: SATA-CDROM emulation reading from a Longhorn-backed Filesystem PVC is
# too slow for OVMF's boot read window — the DVD-ROM enumeration times out
# before the bootloader can be read. Symptom on the serial console:
# BdsDxe: failed to start Boot0001 "UEFI QEMU DVD-ROM QM00001 " from ...
# BdsDxe: failed to start Boot0001 ... Time out
# BdsDxe: No bootable option or device was found
# Diagnosis confirmed the ISO content is a perfectly valid bootable ISO9660
# image — the bug is in the timing path between OVMF and Longhorn-backed
# storage, not in the ISO itself.
#
# Block-mode PVC was tried (`volumeMode: Block` via DataVolume) and would
# likely fix the timing, but CDI v1.65.0's upload-target pod cannot open the
# block device due to runAsUser:107 + capabilities.drop:[ALL] and we got:
# blockdev: cannot open /dev/cdi-block-volume: Permission denied
#
# NFS-mounted ISO bypasses both issues: no Longhorn slowness, no CDI upload
# pod permission concerns. The ISO is read directly from the NAS over a
# native NFSv4.1 mount that QEMU's SATA emulator can read at full LAN speed.
#
# Layout on Synology:
# /volume1/ISOs/ (existing export, RKE2 ACL)
# en-us_windows_server_2025_updated_march_2026_x64_dvd_8e06425a.iso
# win2025-iso-disk/ (new subdir, 2026-05-08)
# disk.img -> hardlink to ../en-us_windows_server_2025_..._8e06425a.iso
#
# KubeVirt's launcher pod expects a PVC mounted at
# /var/run/kubevirt-private/vmi-disks/<diskName>/disk.img — by mounting the
# `win2025-iso-disk/` subdir as the NFS PV root, `disk.img` lives at the PV's
# root and KubeVirt's CDROM emulator finds it without any path manipulation.
#
# A symlink would NOT work for sub-path NFS mounts (the relative target
# `../...iso` falls outside the sub-mount root). A hardlink works because it
# references the same inode regardless of mount point.
#
# Memory references:
# - feedback_synology_nfs_volume1_kubernetes_export_scoped (Synology export
# scoping pattern — but /volume1/ISOs export, unlike /volume1/kubernetes,
# does support sub-path mounts because Synology NFS is configured with
# pseudo-fs in NFSv4.1)
# - feedback_kubevirt_iso_first_install_bootorder_and_runstrategy (boot
# order / runStrategy gotchas, separate from the storage timing issue)
#
# Validation (2026-05-08, from rke2-server / rke2-agent1 / rke2-agent2):
# mount -t nfs -o nfsvers=4.1,ro 10.0.58.3:/volume1/ISOs/win2025-iso-disk /tmp/m
# file /tmp/m/disk.img
# -> ISO 9660 CD-ROM filesystem data 'SSS_X64FRE_EN-US_DV9' (bootable)
# All 3 RKE2 nodes can mount and read.
# =============================================================================
apiVersion: v1
kind: PersistentVolume
metadata:
name: windows-server-2025-iso-nfs
labels:
flowercore.io/iso: windows-server-2025
flowercore.io/managed-by: bluejay-infra
spec:
capacity:
storage: 8Gi
accessModes:
- ReadOnlyMany
volumeMode: Filesystem
persistentVolumeReclaimPolicy: Retain
storageClassName: "" # static, no provisioner
mountOptions:
- nfsvers=4.1
- ro
- hard
- timeo=600
- retrans=3
nfs:
server: 10.0.58.3 # BlueJayNAS Synology DS1621+ on HOME VLAN 58
path: /volume1/ISOs/win2025-iso-disk
readOnly: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: windows-server-2025-iso-nfs
namespace: kubevirt-vms
labels:
app: ci-runner
flowercore.io/managed-by: bluejay-infra
spec:
accessModes:
- ReadOnlyMany
volumeMode: Filesystem
resources:
requests:
storage: 8Gi
storageClassName: ""
volumeName: windows-server-2025-iso-nfs

View File

@@ -974,6 +974,39 @@ data:
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
description: "Spec wants {{ $labels.spec_replicas }} but only {{ $value }} available. Likely a rollout stuck on probe failure, scheduling, or PVC."
# Q-MR-3 (2026-05-11): multus memory pressure — catches the next OOM
# cascade BEFORE multus is OOM-killed cluster-wide. The 2026-05-10
# outage (21h) hit because no alert fired on the rising multus working
# set — only downstream blackbox / Traefik / service alerts. With
# 1Gi limit (bluejay-infra@eb8693e), 80% = ~800MiB; steady-state
# runs ~150-250MiB so this only fires when an avalanche starts.
- alert: MultusMemoryPressure
expr: |
container_memory_working_set_bytes{container="kube-multus"}
/ container_spec_memory_limit_bytes{container="kube-multus"} > 0.8
for: 5m
labels:
severity: critical
alert_channel: thermal_print
annotations:
summary: "kube-multus memory >80% of limit on {{ $labels.node }} for 5m"
description: "kube-multus working set is {{ $value | humanizePercentage }} of its memory limit on node {{ $labels.node }}. If this keeps climbing, multus will OOM and all new pod networking will halt cluster-wide (precedent: 2026-05-10 outage)."
# Q-MR-3 (2026-05-11): namespace pending-pod backlog — catches the
# operator-leak avalanche pattern BEFORE it cascades into a multus
# CNI OOM. Any FC operator (RemoteDesktop / Distribution / WorldBuilder)
# emitting pods without ownerReferences will accumulate them when
# the operator crashes. >25 pending pods in any namespace for 30m
# is the signal to investigate the reconciler.
- alert: NamespacePendingPodBacklog
expr: sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 25
for: 30m
labels:
severity: warning
annotations:
summary: "Namespace {{ $labels.namespace }} has {{ $value }} Pending pods for 30m"
description: "Pending pod count in {{ $labels.namespace }} exceeds 25 sustained for 30m. Likely operator-leak avalanche pattern — children emitted without ownerReferences. Risk of multus CNI OOM cascade."
# Longhorn storage health alerts. Required: longhorn scrape job
# (added 2026-04-26 — see scrape_configs above). The K8s events
# for "snapshot becomes not ready to use" are transient lifecycle

297
apps/multus/multus.yaml Normal file
View File

@@ -0,0 +1,297 @@
# =============================================================================
# Multus CNI — Meta-CNI for multi-network attachment to pods/VMs
# =============================================================================
# Purpose: enable KubeVirt VMs (and any future workload) to attach additional
# network interfaces beyond the default Calico-managed pod network. Required
# for ci1 (Windows Server 2025 KubeVirt VM) to bridge onto PROD VLAN 57.
#
# Source: upstream k8snetworkplumbingwg/multus-cni v4.2.2
# https://github.com/k8snetworkplumbingwg/multus-cni/blob/v4.2.2/deployments/multus-daemonset-thick.yml
#
# Inlined verbatim (with project header + version pin annotation) for
# reproducibility and air-gap safety. Bumping versions = edit this file +
# git push. ArgoCD picks up via the bluejay-infra ApplicationSet
# (apps/* directory generator on main).
#
# Why thick plugin (not thin):
# - Thick = daemon + thin shim binary; daemon handles NAD watch + CRD reads
# centrally so each pod's CNI ADD doesn't hit the K8s API server. Better
# for clusters with many NAD-using pods.
# - Thin = each CNI ADD process directly contacts K8s API. Simpler but
# scales worse and has more failure modes.
# - KubeVirt + multi-VM workload pattern fits thick perfectly.
#
# Cluster context (verified 2026-05-08):
# - RKE2 v1.34.5 on 3 nodes (rke2-server, rke2-agent1, rke2-agent2)
# - Calico CNI (Tigera-managed) at /etc/cni/net.d + /opt/cni/bin (default)
# - openSUSE Leap 16, kernel 6.12, containerd 2.1.5
# - host bridge for PROD VLAN 57 = `br-prod` (PUPPET HOST WORK — see Phase 1.5
# in docs/infrastructure/windows-server-build-runner-plan.md)
#
# Version pin: snapshot-thick → pinning to v4.2.2 release tag at deploy time
# would require a private mirror of the image. Upstream `snapshot-thick` tag
# is updated on every release, so for now we trust upstream + Calico's
# established pattern. Pin to a specific SHA256 once we mirror to Gitea OCI.
#
# Apply (once committed to bluejay-infra main, ApplicationSet auto-syncs):
# git add apps/multus/multus.yaml && git commit && git push origin main
# # ArgoCD `infra-multus` Application appears within 3 min via ApplicationSet
#
# Verify:
# kubectl -n kube-system get ds kube-multus-ds
# kubectl -n kube-system rollout status ds kube-multus-ds
# kubectl get crd network-attachment-definitions.k8s.cni.cncf.io
# =============================================================================
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
annotations:
bluejay.iamworkin.lan/source: "k8snetworkplumbingwg/multus-cni v4.2.2"
spec:
group: k8s.cni.cncf.io
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net-attach-def
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
Working Group to express the intent for attaching pods to one or more logical or physical
networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
type: object
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
type: object
properties:
config:
description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
type: string
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: multus
rules:
- apiGroups: ["k8s.cni.cncf.io"]
resources:
- '*'
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
- pods/status
verbs:
- get
- list
- update
- watch
- apiGroups:
- ""
- events.k8s.io
resources:
- events
verbs:
- create
- patch
- update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: multus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: multus
subjects:
- kind: ServiceAccount
name: multus
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: multus
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: multus-daemon-config
namespace: kube-system
labels:
tier: node
app: multus
data:
daemon-config.json: |
{
"chrootDir": "/hostroot",
"cniVersion": "0.3.1",
"logLevel": "verbose",
"logToStderr": true,
"cniConfigDir": "/host/etc/cni/net.d",
"multusAutoconfigDir": "/host/etc/cni/net.d",
"multusConfigFile": "auto",
"socketDir": "/host/run/multus/"
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-multus-ds
namespace: kube-system
labels:
tier: node
app: multus
name: multus
spec:
selector:
matchLabels:
name: multus
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
tier: node
app: multus
name: multus
spec:
hostNetwork: true
hostPID: true
tolerations:
- operator: Exists
effect: NoSchedule
- operator: Exists
effect: NoExecute
serviceAccountName: multus
containers:
- name: kube-multus
image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
command: [ "/usr/src/multus-cni/bin/multus-daemon" ]
# 2026-05-11: upstream default of 50Mi memory limit OOM-cascades when
# an operator-owned namespace accumulates >100 pending pods retrying
# CNI ADD. RemoteDesktop emitted 219 orphan rd-browser-only pods
# (missing OwnerReferences), kubelet's CNI ADD avalanche pushed multus
# over 50Mi, OOMKilled, restarted with even bigger backlog → loop.
# 21h cluster outage. See FlowerCore.Notes:
# feedback_multus_50mi_limit_oom_orphan_pod_avalanche.md
# 1Gi limit / 512Mi request comfortably handles a 200+ pod CNI
# catchup burst on 64GB nodes (nodes are <25% used in steady-state).
# Drop back toward 256Mi only after MultusMemoryPressure alert
# proves steady-state working set sits well below 200Mi.
resources:
requests:
cpu: "100m"
memory: "512Mi"
limits:
cpu: "100m"
memory: "1Gi"
securityContext:
privileged: true
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- name: cni
mountPath: /host/etc/cni/net.d
# multus-daemon expects that cnibin path must be identical between pod and container host.
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
# not to any other directory, like '/opt/bin' or '/usr/bin'.
- name: cnibin
mountPath: /opt/cni/bin
- name: host-run
mountPath: /host/run
- name: host-var-lib-cni-multus
mountPath: /var/lib/cni/multus
- name: host-var-lib-kubelet
mountPath: /var/lib/kubelet
mountPropagation: HostToContainer
- name: host-run-k8s-cni-cncf-io
mountPath: /run/k8s.cni.cncf.io
- name: host-run-netns
mountPath: /run/netns
mountPropagation: HostToContainer
- name: multus-daemon-config
mountPath: /etc/cni/net.d/multus.d
readOnly: true
- name: hostroot
mountPath: /hostroot
mountPropagation: HostToContainer
- mountPath: /etc/cni/multus/net.d
name: multus-conf-dir
env:
- name: MULTUS_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
initContainers:
- name: install-multus-binary
image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick
command:
- "sh"
- "-c"
- "cp /usr/src/multus-cni/bin/multus-shim /host/opt/cni/bin/multus-shim && cp /usr/src/multus-cni/bin/passthru /host/opt/cni/bin/passthru"
resources:
requests:
cpu: "10m"
memory: "15Mi"
securityContext:
privileged: true
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- name: cnibin
mountPath: /host/opt/cni/bin
mountPropagation: Bidirectional
terminationGracePeriodSeconds: 10
volumes:
- name: cni
hostPath:
path: /etc/cni/net.d
- name: cnibin
hostPath:
path: /opt/cni/bin
- name: hostroot
hostPath:
path: /
- name: multus-daemon-config
configMap:
name: multus-daemon-config
items:
- key: daemon-config.json
path: daemon-config.json
- name: host-run
hostPath:
path: /run
- name: host-var-lib-cni-multus
hostPath:
path: /var/lib/cni/multus
- name: host-var-lib-kubelet
hostPath:
path: /var/lib/kubelet
- name: host-run-k8s-cni-cncf-io
hostPath:
path: /run/k8s.cni.cncf.io
- name: host-run-netns
hostPath:
path: /run/netns/
- name: multus-conf-dir
hostPath:
path: /etc/cni/multus/net.d

View File

@@ -127,10 +127,13 @@ spec:
initContainers:
- name: fix-data-perms
image: busybox:latest
# Also chown /shared-tts (hostPath /tmp/tts-audio) so the non-root
# app user (uid 1654) can write Piper .sln16 files that Asterisk
# reads at /var/lib/asterisk/sounds/tts. World-readable (755) is
# fine — Asterisk runs as a different uid in the other pod.
# Must run as root to chown the hostPath /tmp/tts-audio that may be
# root-owned after node reboot. Pod-level runAsNonRoot:true would
# otherwise inherit and chown would fail with EPERM (see Notes memory
# feedback_hostpath_initcontainer_chown_perms).
securityContext:
runAsUser: 0
runAsNonRoot: false
command: ["sh", "-c", "chown -R 1654:1654 /data && chown 1654:1654 /shared-tts && chmod 0755 /shared-tts"]
volumeMounts:
- name: telephony-data

View File

@@ -0,0 +1,416 @@
#!/usr/bin/env python3
"""Generate and optionally apply FlowerCore Authentik OIDC client assets.
Dry-run is the default. Live Authentik mutations require --apply plus an
AUTHENTIK_TOKEN bearer token and an operator-provided client secret JSON file.
The script never prints client_secret values.
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from typing import Any
CLAIMS = (
"fc:roles",
"fc:tenant",
"fc:svc",
"fc:scope",
"fc:mfa",
"flowercore_actor_id",
)
BUILTIN_SCOPE_MAPPINGS = (
"goauthentik.io/providers/oauth2/scope-openid",
"goauthentik.io/providers/oauth2/scope-profile",
"goauthentik.io/providers/oauth2/scope-email",
"goauthentik.io/providers/oauth2/scope-offline_access",
)
FLOW_CONTRACT = {
"response_type": "code",
"grant_types": ["authorization_code", "refresh_token"],
"offline_access_required": True,
}
@dataclass(frozen=True)
class ServiceSpec:
slug: str
namespace: str
display_name: str
host: str
@property
def client_id(self) -> str:
return self.slug
@property
def provider_name(self) -> str:
return f"FlowerCore {self.display_name} OIDC"
@property
def application_name(self) -> str:
return f"FlowerCore {self.display_name}"
@property
def issuer_url(self) -> str:
return f"https://id.iamworkin.lan/application/o/{self.slug}/"
@property
def onepassword_item_path(self) -> str:
return f"IAmWorkin/items/{self.slug}-oidc-client"
SERVICE_SPECS = (
ServiceSpec("library", "fc-library", "Library", "library.iamworkin.lan"),
ServiceSpec("retail", "fc-retail", "Retail", "retail.iamworkin.lan"),
ServiceSpec("telephony", "telephony", "Telephony", "telephony.iamworkin.lan"),
ServiceSpec("knowledge", "knowledge", "Knowledge", "knowledge.iamworkin.lan"),
ServiceSpec("llmbridge", "fc-llm-bridge", "LlmBridge", "fc-llm-bridge.iamworkin.lan"),
ServiceSpec("mysql", "fc-mysql", "MySQL", "mysql.iamworkin.lan"),
ServiceSpec("php", "fc-php", "PHP", "php.iamworkin.lan"),
ServiceSpec("signage", "fc-signage", "Signage", "signage.iamworkin.lan"),
ServiceSpec("media", "fc-media", "Media", "media.iamworkin.lan"),
ServiceSpec("dms", "fc-dms", "DMS", "dms.iamworkin.lan"),
ServiceSpec("pimanager", "fc-pimanager", "PiManager", "pimanager.iamworkin.lan"),
ServiceSpec("distribution", "fc-distribution", "Distribution", "distribution.iamworkin.lan"),
ServiceSpec("dns", "fc-dns", "DNS", "dns.iamworkin.lan"),
ServiceSpec("print", "fc-print", "Print", "print.iamworkin.lan"),
ServiceSpec("aistation", "fc-aistation", "AiStation", "aistation.iamworkin.lan"),
ServiceSpec("irc", "irc", "IRC", "irc.iamworkin.lan"),
ServiceSpec("ttsreader", "fc-ttsreader", "TtsReader", "ttsreader.iamworkin.lan"),
ServiceSpec("chat", "fc-chat", "Chat", "chat.iamworkin.lan"),
ServiceSpec("intranet", "intranet", "Intranet", "intranet.iamworkin.lan"),
ServiceSpec("remotedesktop", "fc-desktop", "RemoteDesktop", "remotedesktop.iamworkin.lan"),
ServiceSpec("provisioning", "fc-provisioning", "Provisioning", "provisioning.iamworkin.lan"),
ServiceSpec("scoreboards", "fc-scoreboard", "Scoreboards", "scoreboards.iamworkin.lan"),
ServiceSpec("mndot", "fc-mndot", "MnDOT", "mndot.iamworkin.lan"),
ServiceSpec("kiosk", "fc-system", "Kiosk", "kiosk.iamworkin.lan"),
ServiceSpec("mike-bundle", "fc-mike-bundle", "Mike Bundle", "mike-bundle.iamworkin.lan"),
ServiceSpec("messageboard", "fc-messageboard", "MessageBoard", "messageboard.iamworkin.lan"),
ServiceSpec("menuboard", "fc-menuboard", "MenuBoard", "menuboard.iamworkin.lan"),
ServiceSpec("presentations", "fc-presentations", "Presentations", "presentations.iamworkin.lan"),
ServiceSpec("segmentdisplay", "fc-segmentdisplay", "SegmentDisplay", "segmentdisplay.iamworkin.lan"),
ServiceSpec("signalcontrol", "fc-signalcontrol", "SignalControl", "signalcontrol.iamworkin.lan"),
ServiceSpec("worldbuilder", "fc-worldbuilder", "WorldBuilder", "worldbuilder.iamworkin.lan"),
ServiceSpec("audit", "fc-audit", "Audit", "audit.iamworkin.lan"),
ServiceSpec("licensing", "fc-licensing", "Licensing", "licensing.iamworkin.lan"),
)
def scope_mapping_payloads(service: ServiceSpec) -> list[dict[str, str]]:
managed_prefix = f"flowercore.io/authentik/oidc/{service.slug}"
return [
{
"managed": f"{managed_prefix}/fc-roles",
"name": f"FlowerCore {service.slug} fc:roles",
"scope_name": "flowercore",
"description": "FlowerCore role claim from Authentik group memberships.",
"expression": (
"groups = [group.name for group in request.user.ak_groups.all()]\n"
"return {'fc:roles': ','.join(groups)}"
),
},
{
"managed": f"{managed_prefix}/fc-tenant",
"name": f"FlowerCore {service.slug} fc:tenant",
"scope_name": "flowercore",
"description": "FlowerCore tenant claim from group attribute fc_tenant_id.",
"expression": (
"for group in request.user.ak_groups.all():\n"
" tenant_id = group.attributes.get('fc_tenant_id')\n"
" if tenant_id:\n"
" return {'fc:tenant': tenant_id}\n"
"return {'fc:tenant': 'default'}"
),
},
{
"managed": f"{managed_prefix}/fc-svc",
"name": f"FlowerCore {service.slug} fc:svc",
"scope_name": "flowercore",
"description": "FlowerCore service slug claim.",
"expression": f"return {{'fc:svc': '{service.slug}'}}",
},
{
"managed": f"{managed_prefix}/fc-scope",
"name": f"FlowerCore {service.slug} fc:scope",
"scope_name": "flowercore",
"description": "FlowerCore service permission scope claim.",
"expression": f"return {{'fc:scope': 'flowercore:{service.slug}'}}",
},
{
"managed": f"{managed_prefix}/fc-mfa",
"name": f"FlowerCore {service.slug} fc:mfa",
"scope_name": "flowercore",
"description": "FlowerCore MFA satisfied session claim.",
"expression": (
"mfa_stage = request.session.get('authentik/stages/authenticator_validate')\n"
"return {'fc:mfa': bool(mfa_stage)}"
),
},
{
"managed": f"{managed_prefix}/flowercore-actor-id",
"name": f"FlowerCore {service.slug} flowercore_actor_id",
"scope_name": "flowercore",
"description": "FlowerCore audit actor alias for the Authentik user id.",
"expression": "return {'flowercore_actor_id': str(request.user.uid)}",
},
]
def provider_payload(
service: ServiceSpec,
args: argparse.Namespace,
mapping_ids: list[str],
client_secret: str,
) -> dict[str, Any]:
payload: dict[str, Any] = {
"name": service.provider_name,
"authorization_flow": args.authorization_flow,
"invalidation_flow": args.invalidation_flow,
"property_mappings": mapping_ids,
"client_type": "confidential",
"client_id": service.client_id,
"client_secret": client_secret,
"access_code_validity": "minutes=1",
"access_token_validity": "hours=1",
"refresh_token_validity": "days=30",
"include_claims_in_id_token": True,
"redirect_uris": [
{"matching_mode": "strict", "url": f"https://{service.host}/signin-oidc"},
{"matching_mode": "strict", "url": f"https://{service.host}/signout-callback-oidc"},
],
"sub_mode": "hashed_user_id",
"issuer_mode": "per_provider",
}
if args.authentication_flow:
payload["authentication_flow"] = args.authentication_flow
if args.signing_key:
payload["signing_key"] = args.signing_key
return payload
def application_payload(service: ServiceSpec, provider_pk: int | str | None) -> dict[str, Any]:
return {
"name": service.application_name,
"slug": service.slug,
"provider": provider_pk,
"open_in_new_tab": True,
"meta_launch_url": f"https://{service.host}/",
"meta_description": f"FlowerCore {service.display_name} OIDC client",
"meta_publisher": "FlowerCore",
"policy_engine_mode": "all",
"group": "FlowerCore",
}
def load_client_secrets(path: str | None) -> dict[str, str]:
if not path:
return {}
with open(path, "r", encoding="utf-8") as handle:
data = json.load(handle)
if not isinstance(data, dict):
raise ValueError("client secret file must be a JSON object keyed by service slug")
return {str(key): str(value) for key, value in data.items()}
def redact(value: Any) -> Any:
if isinstance(value, dict):
return {
key: "<redacted>" if key == "client_secret" else redact(child)
for key, child in value.items()
}
if isinstance(value, list):
return [redact(child) for child in value]
return value
class AuthentikClient:
def __init__(self, base_url: str, token: str) -> None:
self.base_url = base_url.rstrip("/")
self.token = token
def request(self, method: str, path: str, payload: dict[str, Any] | None = None) -> Any:
url = f"{self.base_url}/api/v3/{path.lstrip('/')}"
body = None
headers = {
"Accept": "application/json",
"Authorization": f"Bearer {self.token}",
}
if payload is not None:
headers["Content-Type"] = "application/json"
body = json.dumps(payload).encode("utf-8")
request = urllib.request.Request(url, data=body, headers=headers, method=method)
try:
with urllib.request.urlopen(request, timeout=30) as response:
text = response.read().decode("utf-8")
except urllib.error.HTTPError as error:
error_text = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"{method} {path} failed with HTTP {error.code}: {error_text}") from error
if not text:
return None
return json.loads(text)
def first_result(self, path: str, **query: str) -> dict[str, Any] | None:
query_string = urllib.parse.urlencode(query)
response = self.request("GET", f"{path}?{query_string}")
results = response.get("results", []) if isinstance(response, dict) else []
return results[0] if results else None
def select_services(slugs: list[str]) -> list[ServiceSpec]:
if not slugs:
return list(SERVICE_SPECS)
by_slug = {service.slug: service for service in SERVICE_SPECS}
unknown = sorted(set(slugs) - set(by_slug))
if unknown:
raise ValueError(f"unknown service slug(s): {', '.join(unknown)}")
return [by_slug[slug] for slug in slugs]
def validate_specs(services: list[ServiceSpec]) -> None:
slugs = [service.slug for service in services]
if len(slugs) != len(set(slugs)):
raise ValueError("duplicate service slug in OIDC roster")
for service in services:
if not service.namespace:
raise ValueError(f"{service.slug} is missing a target namespace")
expressions = "\n".join(mapping["expression"] for mapping in scope_mapping_payloads(service))
missing_claims = [claim for claim in CLAIMS if claim not in expressions]
if missing_claims:
raise ValueError(f"{service.slug} mapping payloads miss claims: {', '.join(missing_claims)}")
def dry_run(services: list[ServiceSpec], args: argparse.Namespace) -> int:
placeholder_ids = [
*[f"<builtin-{managed.rsplit('/', 1)[-1]}-pk>" for managed in BUILTIN_SCOPE_MAPPINGS],
*[f"<{claim}-mapping-pk>" for claim in CLAIMS],
]
documents = []
for service in services:
provider = provider_payload(service, args, placeholder_ids, "<from-1password-client_secret>")
documents.append(
{
"service": service.slug,
"namespace": service.namespace,
"onepassword_item": service.onepassword_item_path,
"issuer_url": service.issuer_url,
"flow_contract": FLOW_CONTRACT,
"builtin_scope_mappings": list(BUILTIN_SCOPE_MAPPINGS),
"scope_mappings": scope_mapping_payloads(service),
"provider": redact(provider),
"application": application_payload(service, "<provider-pk>"),
}
)
if args.print_json:
print(json.dumps(documents, indent=2, sort_keys=True))
else:
print(
"Dry-run only: generated "
f"{len(services)} providers, {len(services)} applications, "
f"and {len(services) * len(CLAIMS)} scope mappings."
)
print("Use --print-json to inspect redacted payloads; use --apply for live Authentik mutation.")
return 0
def apply(services: list[ServiceSpec], args: argparse.Namespace) -> int:
token = os.environ.get("AUTHENTIK_TOKEN")
if not token:
raise ValueError("AUTHENTIK_TOKEN is required with --apply")
if not args.client_secrets_json:
raise ValueError("--client-secrets-json is required with --apply")
secrets = load_client_secrets(args.client_secrets_json)
missing = [service.slug for service in services if not secrets.get(service.slug)]
if missing:
raise ValueError(f"client secret JSON is missing slug(s): {', '.join(missing)}")
client = AuthentikClient(args.base_url, token)
for service in services:
mapping_ids: list[str] = []
for managed in BUILTIN_SCOPE_MAPPINGS:
existing = client.first_result("/propertymappings/provider/scope/", managed=managed)
if not existing:
raise ValueError(f"built-in Authentik scope mapping not found: {managed}")
mapping_ids.append(existing["pk"])
for mapping in scope_mapping_payloads(service):
existing = client.first_result("/propertymappings/provider/scope/", name=mapping["name"])
if existing and not args.update_existing:
mapping_ids.append(existing["pk"])
continue
if existing and args.update_existing:
updated = client.request("PATCH", f"/propertymappings/provider/scope/{existing['pk']}/", mapping)
mapping_ids.append(updated["pk"])
continue
created = client.request("POST", "/propertymappings/provider/scope/", mapping)
mapping_ids.append(created["pk"])
existing_provider = client.first_result("/providers/oauth2/", client_id=service.client_id)
provider_body = provider_payload(service, args, mapping_ids, secrets[service.slug])
if existing_provider and not args.update_existing:
provider_pk = existing_provider["pk"]
elif existing_provider:
updated_provider = client.request("PATCH", f"/providers/oauth2/{existing_provider['pk']}/", provider_body)
provider_pk = updated_provider["pk"]
else:
provider_pk = client.request("POST", "/providers/oauth2/", provider_body)["pk"]
app_body = application_payload(service, provider_pk)
existing_app = client.first_result("/core/applications/", slug=service.slug)
if existing_app and args.update_existing:
client.request("PATCH", f"/core/applications/{existing_app['slug']}/", app_body)
elif not existing_app:
client.request("POST", "/core/applications/", app_body)
print(f"applied {service.slug}: provider/app present, secret redacted")
return 0
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--apply", action="store_true", help="perform live Authentik REST mutations")
parser.add_argument("--update-existing", action="store_true", help="patch existing mappings/providers/applications")
parser.add_argument("--print-json", action="store_true", help="print redacted dry-run payloads")
parser.add_argument("--service", action="append", default=[], help="limit to one service slug; repeatable")
parser.add_argument("--base-url", default="http://localhost:9000", help="Authentik base URL")
parser.add_argument("--client-secrets-json", help="operator-provided JSON object of slug to client_secret")
parser.add_argument("--authorization-flow", default="<authorization-flow-uuid>")
parser.add_argument("--invalidation-flow", default="<invalidation-flow-uuid>")
parser.add_argument("--authentication-flow")
parser.add_argument("--signing-key", help="shared Authentik signing key UUID")
return parser.parse_args()
def main() -> int:
args = parse_args()
try:
services = select_services(args.service)
validate_specs(services)
if args.apply:
return apply(services, args)
return dry_run(services, args)
except Exception as error:
print(f"error: {error}", file=sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,192 @@
using FluentAssertions;
using System.Diagnostics;
using System.Text.RegularExpressions;
using Xunit;
using YamlDotNet.RepresentationModel;
namespace BluejayInfraLint.Tests;
[Trait("Category", "AuthentikOidc")]
public sealed class AuthentikOidcClientRegistrationTests
{
private static readonly IReadOnlyList<ServiceClientExpectation> ExpectedClients =
[
new("library", "fc-library"),
new("retail", "fc-retail"),
new("telephony", "telephony"),
new("knowledge", "knowledge"),
new("llmbridge", "fc-llm-bridge"),
new("mysql", "fc-mysql"),
new("php", "fc-php"),
new("signage", "fc-signage"),
new("media", "fc-media"),
new("dms", "fc-dms"),
new("pimanager", "fc-pimanager"),
new("distribution", "fc-distribution"),
new("dns", "fc-dns"),
new("print", "fc-print"),
new("aistation", "fc-aistation"),
new("irc", "irc"),
new("ttsreader", "fc-ttsreader"),
new("chat", "fc-chat"),
new("intranet", "intranet"),
new("remotedesktop", "fc-desktop"),
new("provisioning", "fc-provisioning"),
new("scoreboards", "fc-scoreboard"),
new("mndot", "fc-mndot"),
new("kiosk", "fc-system"),
new("mike-bundle", "fc-mike-bundle"),
new("messageboard", "fc-messageboard"),
new("menuboard", "fc-menuboard"),
new("presentations", "fc-presentations"),
new("segmentdisplay", "fc-segmentdisplay"),
new("signalcontrol", "fc-signalcontrol"),
new("worldbuilder", "fc-worldbuilder"),
new("audit", "fc-audit"),
new("licensing", "fc-licensing"),
];
public static TheoryData<string, string> ExpectedClientRows()
{
var data = new TheoryData<string, string>();
foreach (var client in ExpectedClients)
{
data.Add(client.Slug, client.Namespace);
}
return data;
}
[Theory]
[MemberData(nameof(ExpectedClientRows))]
public void OidcClientManifest_MatchesOnePasswordOperatorContract(string slug, string targetNamespace)
{
var manifest = LoadClientManifest(slug);
manifest.Scalar("apiVersion").Should().Be("onepassword.com/v1");
manifest.Scalar("kind").Should().Be("OnePasswordItem");
manifest.Scalar("metadata", "name").Should().Be($"{slug}-oidc-client");
manifest.Scalar("metadata", "namespace").Should().Be(targetNamespace);
manifest.Scalar("metadata", "labels", "app.kubernetes.io/component")
.Should().Be("authentik-oidc-client");
manifest.Scalar("metadata", "labels", "flowercore.io/authentik-client-slug")
.Should().Be(slug);
manifest.Scalar("metadata", "annotations", "flowercore.io/expected-fields")
.Should().Be("client_id,client_secret,issuer_url");
manifest.Scalar("spec", "itemPath")
.Should().Be($"vaults/IAmWorkin/items/{slug}-oidc-client");
}
[Fact]
public void AuthentikKustomization_ReferencesEveryClientManifest()
{
var kustomizationPath = Path.Combine(BluejayRoot(), "apps", "authentik", "kustomization.yaml");
var text = File.ReadAllText(kustomizationPath);
foreach (var client in ExpectedClients)
{
text.Should().Contain($"clients/{client.Slug}-oidc-client.yaml");
}
Regex.Matches(text, @"clients/[-a-z0-9]+-oidc-client\.yaml")
.Select(match => match.Value)
.Distinct(StringComparer.Ordinal)
.Should()
.HaveCount(ExpectedClients.Count);
}
[Fact]
public void BulkClientScript_HasDryRunDefaultAndRequiredClaimPayloads()
{
var scriptPath = Path.Combine(BluejayRoot(), "scripts", "authentik-bulk-client-create.py");
var script = File.ReadAllText(scriptPath);
script.Should().Contain("--apply");
script.Should().Contain("Dry-run only");
script.Should().Contain("AUTHENTIK_TOKEN");
script.Should().Contain("client_secrets_json");
script.Should().Contain("scope-offline_access");
script.Should().Contain("authorization_code");
script.Should().Contain("refresh_token");
foreach (var claim in new[] { "fc:roles", "fc:tenant", "fc:svc", "fc:scope", "fc:mfa", "flowercore_actor_id" })
{
script.Should().Contain(claim);
}
}
[Fact]
public void BulkClientScript_DryRunGeneratesAllServicesWithoutSecrets()
{
var scriptPath = Path.Combine(BluejayRoot(), "scripts", "authentik-bulk-client-create.py");
var startInfo = new ProcessStartInfo
{
FileName = "python",
WorkingDirectory = BluejayRoot(),
RedirectStandardOutput = true,
RedirectStandardError = true,
};
startInfo.ArgumentList.Add(scriptPath);
startInfo.ArgumentList.Add("--print-json");
using var process = Process.Start(startInfo)
?? throw new InvalidOperationException("Could not start python dry-run.");
var stdout = process.StandardOutput.ReadToEnd();
var stderr = process.StandardError.ReadToEnd();
process.WaitForExit(15000).Should().BeTrue(stderr);
process.ExitCode.Should().Be(0, stderr);
foreach (var client in ExpectedClients)
{
stdout.Should().Contain($"\"service\": \"{client.Slug}\"");
stdout.Should().Contain($"\"namespace\": \"{client.Namespace}\"");
}
stdout.Should().Contain("\"client_secret\": \"<redacted>\"");
stdout.Should().NotMatchRegex("\"client_secret\"\\s*:\\s*\"(?!<redacted>)[^\"]+\"");
}
[Fact]
public void ClientManifests_DoNotContainInlineSecretMaterial()
{
foreach (var client in ExpectedClients)
{
var path = ClientManifestPath(client.Slug);
var text = File.ReadAllText(path);
text.Should().NotContain("client_secret:");
text.Should().NotContain("password:");
text.Should().NotContain("secret:");
text.Should().Contain($"IAmWorkin/items/{client.Slug}-oidc-client");
}
}
private static YamlMappingNode LoadClientManifest(string slug)
{
using var reader = File.OpenText(ClientManifestPath(slug));
var stream = new YamlStream();
stream.Load(reader);
return stream.Documents[0].RootNode.Should().BeOfType<YamlMappingNode>().Subject;
}
private static string ClientManifestPath(string slug) =>
Path.Combine(BluejayRoot(), "apps", "authentik", "clients", $"{slug}-oidc-client.yaml");
private static string BluejayRoot()
{
var current = new DirectoryInfo(AppContext.BaseDirectory);
while (current is not null)
{
if (Directory.Exists(Path.Combine(current.FullName, "apps"))
&& File.Exists(Path.Combine(current.FullName, "README.md")))
{
return current.FullName;
}
current = current.Parent;
}
throw new DirectoryNotFoundException("Could not find bluejay-infra root.");
}
private sealed record ServiceClientExpectation(string Slug, string Namespace);
}