Wire SignalControl platform observability

monitoring(probe): signage/mysql/php blackbox probe / -> /healthz (K8s-target mirror)
Mirrors the live noc1 podman fix + Notes scripts/monitoring/prometheus.yml. These services enforce OIDC bearer auth (FlowerCore__Auth__Enabled=true), so an anonymous probe of / returns 401 -> false TraefikServiceDown. All three expose anonymous /healthz=200. This noc-monitoring.yaml is the forward K8s-migration target (not live); brings it in sync with the live config.
2026-06-02 03:02:13 -05:00 · 2026-06-02 01:09:57 -05:00 · 2026-06-01 15:55:38 -05:00 · 2026-05-31 12:35:45 -05:00 · 2026-05-31 11:34:12 -05:00 · 2026-05-31 00:32:48 -05:00
11 changed files with 521 additions and 77 deletions
--- a/apps/fc-signalcontrol/README.md
+++ b/apps/fc-signalcontrol/README.md
@@ -0,0 +1,33 @@
+# FlowerCore SignalControl platform notes
+
+This app owns the cluster web manager at `signalcontrol.iamworkin.lan` and documents the physical Pi pilot at `signal-a.iamworkin.lan` / `pirelay`.
+
+## mTLS enrollment pattern
+
+Do not install or restart anything from this repo. The intended pirelay pattern is the Pi-signage step-ca-agent shape:
+
+- stable node identity: `pirelay`
+- local private key and CSR generated on the node
+- CSR submitted through the approved DeviceManagement/step-ca enrollment path
+- client certificate and chain stored node-local under `/etc/flowercore/signalcontrol/mtls/`
+- daily renewal timer, renewing only when fewer than 30 days remain
+- certificate used for DM-agent to DM-web traffic and future SignalControl inter-service calls
+
+Secrets, enrollment codes, private keys, p12 passphrases, and OIDC client secrets stay out of Git.
+
+## Telemetry
+
+Monitoring manifests add a dedicated Prometheus job:
+
+- `signalcontrol-pi-app`
+- target `10.0.58.113:5200`
+- path `/metrics/prometheus`
+- labels `instance="pirelay"`, `host="signal-a.iamworkin.lan"`, `service="signalcontrol-pi"`
+
+Host metrics continue through the `edge-nodes` node_exporter target at `10.0.58.113:9100`.
+
+## Physical-control audit
+
+The app ships with `FlowerCore:SignalControl:PhysicalAudit:Enabled=false` and `ForwardingEnabled=false`. Enabling local audit creates a SHA-256 hash chain for physical-control mutations. Forwarding to `https://audit.iamworkin.lan/api/v1/audit/signalcontrol` requires flipping the forwarding gate separately.
+
+Telemetry reads and `/metrics` scrapes are not audited.
--- a/apps/fc-signalcontrol/fc-signalcontrol.yaml
+++ b/apps/fc-signalcontrol/fc-signalcontrol.yaml
@@ -46,7 +46,7 @@ spec:
    spec:
      containers:
        - name: signalcontrol-web
-          image: localhost/fc-signalcontrol-web:latest
+          image: localhost/fc-signalcontrol-web:s50cx12-20260602-1d26c58
          imagePullPolicy: Never
          ports:
            - containerPort: 5000
@@ -65,6 +65,48 @@ spec:
                secretKeyRef:
                  name: signalcontrol-auth
                  key: Auth__ApiKey
+            - name: Auth__AdminApiKey
+              valueFrom:
+                secretKeyRef:
+                  name: signalcontrol-auth
+                  key: Auth__AdminApiKey
+                  optional: true
+            - name: Auth__Enabled
+              value: "false"
+            - name: FlowerCore__Auth__Enabled
+              value: "false"
+            - name: FlowerCore__Auth__Oidc__Enabled
+              value: "true"
+            - name: FlowerCore__Auth__Oidc__Authority
+              valueFrom:
+                secretKeyRef:
+                  name: signalcontrol-oidc-client
+                  key: issuer_url
+                  optional: true
+            - name: FlowerCore__Auth__Oidc__ClientId
+              valueFrom:
+                secretKeyRef:
+                  name: signalcontrol-oidc-client
+                  key: client_id
+                  optional: true
+            - name: FlowerCore__Auth__Oidc__ClientSecret
+              valueFrom:
+                secretKeyRef:
+                  name: signalcontrol-oidc-client
+                  key: client_secret
+                  optional: true
+            - name: TrafficSignal__RelayBridge__Enabled
+              value: "true"
+            - name: TrafficSignal__RelayBridge__BaseUrl
+              value: https://pirelay.iamworkin.lan
+            - name: TrafficSignal__RelayBridge__ApiKey
+              valueFrom:
+                secretKeyRef:
+                  name: signalcontrol-pirelay
+                  key: ApiKey
+                  optional: true
+            - name: LiveStatus__TrafficSignal__BaseAddress
+              value: https://signalcontrol.iamworkin.lan
          volumeMounts:
            - name: data
              mountPath: /data
--- a/apps/fc-ttsreader/fc-ttsreader.yaml
+++ b/apps/fc-ttsreader/fc-ttsreader.yaml
@@ -532,7 +532,7 @@ spec:
        fsGroupChangePolicy: OnRootMismatch
      containers:
        - name: web
-          image: localhost/fc-ttsreader-web:v20260518-sprint36-demo-finish-b132cbf
+          image: localhost/fc-ttsreader-web:v20260531-tts-corrections-r2
          imagePullPolicy: Never
          ports:
            - containerPort: 5217
@@ -554,6 +554,8 @@ spec:
              value: "/data/chapter-context.db"
            - name: TtsReader__Jobs__Root
              value: "/data/jobs"
+            - name: TtsReader__Export__LocalCasRoot
+              value: "/data/bundles/cas"
            - name: TtsReader__Piper__Host
              value: "10.0.57.17"
            - name: TtsReader__Piper__Port
--- a/apps/fc-updater/fc-updater.yaml
+++ b/apps/fc-updater/fc-updater.yaml
@@ -58,7 +58,7 @@ spec:
      nodeName: rke2-server
      containers:
        - name: web
-          image: localhost/fc-updater-web:v20260509-4162dca-authgate
+          image: localhost/fc-updater-web:v202605310029-7974fc4
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
@@ -88,6 +88,8 @@ spec:
              value: Faith AI Mike Edition
            - name: FlowerCore__Updater__PublicShares__Links__0__Description
              value: Private release link for Mike's Faith AI bundle.
+            - name: FlowerCore__Audit__Sinks__Loki__Enabled
+              value: "false"
            - name: FlowerCore__Updater__Auth__Bootstrap__Enabled
              value: "true"
            - name: FlowerCore__Updater__Auth__Bootstrap__Username
--- a/apps/github-runner/github-runner.yaml
+++ b/apps/github-runner/github-runner.yaml
@@ -241,7 +241,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -306,7 +306,7 @@ spec:
  # UN-PARKED 2026-05-21: Shared.Pos #5 fixed the non-root setup-dotnet path
  # (DOTNET_INSTALL_DIR step-scoped). Sprint 30 Cl-8 capacity Q-CI-52: raised
  # to replicas: 2 to absorb top-8 burst load per substrate-recommended default.
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-sharedpos
@@ -397,7 +397,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -448,7 +448,7 @@ metadata:
    flowercore.io/runner-repo: puppet
    flowercore.io/github-repo: FlowerCore.Puppet
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-puppet
@@ -533,7 +533,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -580,7 +580,7 @@ metadata:
    flowercore.io/runner-repo: signage
    flowercore.io/github-repo: FlowerCore.Signage
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-signage
@@ -665,7 +665,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -712,7 +712,7 @@ metadata:
    flowercore.io/runner-repo: dms
    flowercore.io/github-repo: FlowerCore.DMS
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-dms
@@ -797,7 +797,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -844,7 +844,7 @@ metadata:
    flowercore.io/runner-repo: telephony
    flowercore.io/github-repo: FlowerCore.Telephony
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-telephony
@@ -929,7 +929,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -979,7 +979,7 @@ spec:
  # Sprint 33 morning-routine (2026-05-25): bumped 2 → 3 because help-screenshots
  # AAT job holds a runner 30+ min, causing head-of-line blocking on parallel PRs.
  # 12 runs in trailing 5d.
-  replicas: 3
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-print-web
@@ -1064,7 +1064,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -1111,7 +1111,7 @@ metadata:
    flowercore.io/runner-repo: chat
    flowercore.io/github-repo: FlowerCore.Chat
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-chat
@@ -1196,7 +1196,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -1243,7 +1243,7 @@ metadata:
    flowercore.io/runner-repo: mysql
    flowercore.io/github-repo: FlowerCore.MySQL
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-mysql
@@ -1328,7 +1328,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -1375,7 +1375,7 @@ metadata:
    flowercore.io/runner-repo: kiosk-linux
    flowercore.io/github-repo: FlowerCore.Kiosk.Linux
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-kiosk-linux
@@ -1460,7 +1460,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -1509,7 +1509,7 @@ metadata:
    flowercore.io/runner-repo: marquee
    flowercore.io/github-repo: FlowerCore.Marquee
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-marquee
@@ -1594,7 +1594,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -1643,7 +1643,7 @@ metadata:
    flowercore.io/runner-repo: tts-reader
    flowercore.io/github-repo: FlowerCore.TtsReader
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-tts-reader
@@ -1732,7 +1732,7 @@ spec:
          # symptoms surfaced. 8Gi gives ~30% headroom over peak observed.
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "2Gi"
            limits:
              cpu: "2000m"
@@ -1867,7 +1867,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2001,7 +2001,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2135,7 +2135,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2269,7 +2269,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2317,7 +2317,7 @@ metadata:
    flowercore.io/runner-repo: remote-desktop
    flowercore.io/github-repo: FlowerCore.RemoteDesktop
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-remote-desktop
@@ -2402,7 +2402,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2536,7 +2536,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2584,7 +2584,7 @@ metadata:
    flowercore.io/runner-repo: distribution
    flowercore.io/github-repo: FlowerCore.Distribution
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-distribution
@@ -2669,7 +2669,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2717,7 +2717,7 @@ metadata:
    flowercore.io/runner-repo: scoreboard
    flowercore.io/github-repo: FlowerCore.Scoreboard
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-scoreboard
@@ -2802,7 +2802,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2850,7 +2850,7 @@ metadata:
    flowercore.io/runner-repo: segment-display
    flowercore.io/github-repo: FlowerCore.SegmentDisplay
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-segment-display
@@ -2935,7 +2935,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -2983,7 +2983,7 @@ metadata:
    flowercore.io/runner-repo: signage-contracts
    flowercore.io/github-repo: FlowerCore.Signage.Contracts
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-signage-contracts
@@ -3068,7 +3068,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3116,7 +3116,7 @@ metadata:
    flowercore.io/runner-repo: signal-control
    flowercore.io/github-repo: FlowerCore.SignalControl
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-signal-control
@@ -3201,7 +3201,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3335,7 +3335,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3469,7 +3469,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3603,7 +3603,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3737,7 +3737,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3871,7 +3871,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -3919,7 +3919,7 @@ metadata:
    flowercore.io/runner-repo: pimanager
    flowercore.io/github-repo: FlowerCore.PiManager
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-pimanager
@@ -4004,7 +4004,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -4053,7 +4053,7 @@ metadata:
    flowercore.io/runner-repo: updater
    flowercore.io/github-repo: FlowerCore.Updater
 spec:
-  replicas: 2
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-updater
@@ -4138,7 +4138,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -4189,7 +4189,9 @@ metadata:
    flowercore.io/runner-repo: device-management
    flowercore.io/github-repo: FlowerCore.DeviceManagement
 spec:
-  replicas: 2
+  # Single replica until cluster CPU pressure resolves; the fleet-wide
+  # request right-sizing pass is queued for a future sweep.
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-device-management
@@ -4273,8 +4275,11 @@ spec:
            - name: RUN_AS_ROOT
              value: "false"
          resources:
+            # Reduced from 500m → 100m 2026-05-26 because cluster CPU
+            # requests were at 99% across all 3 nodes; idle runners use ~1m.
+            # Burst headroom preserved by limits.cpu: 2000m.
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -4323,7 +4328,9 @@ metadata:
    flowercore.io/runner-repo: aistation-linux
    flowercore.io/github-repo: FlowerCore.AiStation.Linux
 spec:
-  replicas: 2
+  # Single replica until cluster CPU pressure resolves; the fleet-wide
+  # request right-sizing pass is queued for a future sweep.
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-aistation-linux
@@ -4408,7 +4415,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
@@ -4456,7 +4463,9 @@ metadata:
    flowercore.io/runner-repo: worldbuilder
    flowercore.io/github-repo: FlowerCore.WorldBuilder
 spec:
-  replicas: 2
+  # Single replica until cluster CPU pressure resolves; the fleet-wide
+  # request right-sizing pass is queued for a future sweep.
+  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-worldbuilder
@@ -4541,7 +4550,7 @@ spec:
              value: "false"
          resources:
            requests:
-              cpu: "500m"
+              cpu: "100m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
--- a/apps/intranet/intranet.yaml
+++ b/apps/intranet/intranet.yaml
@@ -46,7 +46,7 @@ spec:
    spec:
      containers:
        - name: intranet-web
-          image: localhost/fc-intranet-web:v20260508-brochure-w1
+          image: localhost/fc-intranet-web:v20260531-ttsreader-bridge
          imagePullPolicy: Never
          ports:
            - containerPort: 5300
--- a/apps/kubevirt-vms/ci1.yaml
+++ b/apps/kubevirt-vms/ci1.yaml
@@ -25,7 +25,7 @@ metadata:
    role: github-actions-runner
    flowercore.io/managed-by: bluejay-infra
 spec:
-  runStrategy: Always
+  runStrategy: Halted
  template:
    metadata:
      labels:
--- a/apps/mail/mail.yaml
+++ b/apps/mail/mail.yaml
@@ -207,20 +207,13 @@ spec:
    - port: 993
      targetPort: 993
      name: imaps
---
-# TLS Certificate via cert-manager
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: mail-tls
-  namespace: mail
-spec:
-  secretName: mail-tls
-  issuerRef:
-    name: step-ca-acme
-    kind: ClusterIssuer
-  dnsNames:
-    - mail.iamworkin.lan
+# --- mail-tls Certificate REMOVED 2026-06-01 ---
+# mail-tls is now managed OUTSIDE cert-manager: issued from step-ca's JWK 'admin'
+# provisioner and auto-renewed by a systemd timer on noc1 (step ca renew), which
+# writes the mail-tls secret directly. step-ca-acme only has an HTTP-01 (Traefik)
+# solver, but mail.iamworkin.lan must resolve to the dedicated MetalLB IP 10.0.56.202
+# (SMTP/IMAP), so HTTP-01 cannot validate. Do NOT re-add a cert-manager Certificate
+# here unless a DNS-01 solver is deployed for step-ca-acme.
 ---
 # Traefik IngressRoute - Webmail placeholder
 apiVersion: traefik.io/v1alpha1
--- a/apps/monitoring/grafana-dashboard-signalcontrol.yaml
+++ b/apps/monitoring/grafana-dashboard-signalcontrol.yaml
@@ -0,0 +1,260 @@
+# Grafana dashboard ConfigMap for FlowerCore.SignalControl on pirelay.
+#
+# The Grafana Deployment in noc-monitoring.yaml mounts this ConfigMap at
+# /var/lib/grafana/dashboards/signalcontrol. The paired Prometheus jobs are:
+# - signalcontrol-pi-app: 10.0.58.113:5200 /metrics/prometheus
+# - edge-nodes: 10.0.58.113:9100 with instance="pirelay"
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-signalcontrol
+  namespace: monitoring
+data:
+  signalcontrol.json: |
+    {
+      "annotations": { "list": [] },
+      "editable": true,
+      "fiscalYearStartMonth": 0,
+      "graphTooltip": 0,
+      "id": null,
+      "links": [],
+      "panels": [
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "red", "value": null },
+                  { "color": "green", "value": 1 }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 5, "w": 6, "x": 0, "y": 0 },
+          "id": 1,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false },
+            "textMode": "auto"
+          },
+          "targets": [
+            { "editorMode": "code", "expr": "up{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}", "range": true, "refId": "A" }
+          ],
+          "title": "SignalControl App Up",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "red", "value": null },
+                  { "color": "green", "value": 1 }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 5, "w": 6, "x": 6, "y": 0 },
+          "id": 2,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "none",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false },
+            "textMode": "auto"
+          },
+          "targets": [
+            { "editorMode": "code", "expr": "up{job=\"edge-nodes\",instance=\"pirelay\"}", "range": true, "refId": "A" }
+          ],
+          "title": "pirelay node_exporter Up",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+          "gridPos": { "h": 5, "w": 6, "x": 12, "y": 0 },
+          "id": 3,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false },
+            "textMode": "name"
+          },
+          "targets": [
+            { "editorMode": "code", "expr": "signalcontrol_active_pattern{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}", "legendFormat": "{{pattern}}", "range": true, "refId": "A" }
+          ],
+          "title": "Active Pattern",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+          "gridPos": { "h": 5, "w": 6, "x": 18, "y": 0 },
+          "id": 4,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false },
+            "textMode": "name"
+          },
+          "targets": [
+            { "editorMode": "code", "expr": "signalcontrol_phase{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}", "legendFormat": "{{phase}}", "range": true, "refId": "A" }
+          ],
+          "title": "Current Phase",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 5 },
+          "id": 5,
+          "options": { "legend": { "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "single" } },
+          "targets": [
+            {
+              "editorMode": "code",
+              "expr": "sum by (channel, state) (rate(signal_relay_writes_total{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}[$__rate_interval]))",
+              "legendFormat": "channel {{channel}} {{state}}",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Relay Activations",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 5 },
+          "id": 6,
+          "options": { "legend": { "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "single" } },
+          "targets": [
+            {
+              "editorMode": "code",
+              "expr": "sum by (source, to) (rate(signal_transitions_total{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}[$__rate_interval]))",
+              "legendFormat": "{{source}} -> {{to}}",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Phase Dwell / Transitions",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
+          "id": 7,
+          "options": { "legend": { "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "single" } },
+          "targets": [
+            {
+              "editorMode": "code",
+              "expr": "sum by (action, outcome) (increase(signal_schedule_fires_total{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}[24h]))",
+              "legendFormat": "{{action}} {{outcome}}",
+              "range": true,
+              "refId": "A"
+            },
+            {
+              "editorMode": "code",
+              "expr": "sum by (from_pattern, to_pattern) (increase(flowercore_signalcontrol_pattern_switches_total{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}[24h]))",
+              "legendFormat": "{{from_pattern}} -> {{to_pattern}}",
+              "range": true,
+              "refId": "B"
+            }
+          ],
+          "title": "Schedule Fires and Pattern Switches",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
+          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
+          "id": 8,
+          "options": { "legend": { "displayMode": "table", "placement": "bottom" }, "tooltip": { "mode": "single" } },
+          "targets": [
+            {
+              "editorMode": "code",
+              "expr": "1 - avg by (instance) (rate(node_cpu_seconds_total{job=\"edge-nodes\",instance=\"pirelay\",mode=\"idle\"}[$__rate_interval]))",
+              "legendFormat": "CPU",
+              "range": true,
+              "refId": "A"
+            },
+            {
+              "editorMode": "code",
+              "expr": "1 - (node_memory_MemAvailable_bytes{job=\"edge-nodes\",instance=\"pirelay\"} / node_memory_MemTotal_bytes{job=\"edge-nodes\",instance=\"pirelay\"})",
+              "legendFormat": "Memory",
+              "range": true,
+              "refId": "B"
+            }
+          ],
+          "title": "pirelay Host Utilization",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+          "gridPos": { "h": 6, "w": 12, "x": 0, "y": 21 },
+          "id": 9,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false },
+            "textMode": "auto"
+          },
+          "targets": [
+            { "editorMode": "code", "expr": "signalcontrol_screen_saver_enabled{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}", "range": true, "refId": "A" }
+          ],
+          "title": "Screen-saver Enabled",
+          "type": "stat"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+          "gridPos": { "h": 6, "w": 12, "x": 12, "y": 21 },
+          "id": 10,
+          "options": {
+            "colorMode": "value",
+            "graphMode": "area",
+            "justifyMode": "auto",
+            "orientation": "auto",
+            "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false },
+            "textMode": "name"
+          },
+          "targets": [
+            { "editorMode": "code", "expr": "signalcontrol_animation_active{job=\"signalcontrol-pi-app\",instance=\"pirelay\"}", "legendFormat": "{{planner}}", "range": true, "refId": "A" }
+          ],
+          "title": "Screen-saver / Animation Engaged",
+          "type": "stat"
+        }
+      ],
+      "refresh": "30s",
+      "schemaVersion": 39,
+      "style": "dark",
+      "tags": [ "flowercore", "signalcontrol", "pirelay" ],
+      "templating": { "list": [] },
+      "time": { "from": "now-24h", "to": "now" },
+      "timezone": "browser",
+      "title": "FlowerCore SignalControl",
+      "uid": "flowercore-signalcontrol",
+      "version": 1
+    }
--- a/apps/monitoring/noc-monitoring.yaml
+++ b/apps/monitoring/noc-monitoring.yaml
@@ -230,6 +230,19 @@ data:
              vlan: "home"
              device: "pi3-ks0212"

+      # SignalControl Pi-edition app metrics (pirelay / signal-a)
+      - job_name: "signalcontrol-pi-app"
+        scrape_interval: 15s
+        metrics_path: /metrics/prometheus
+        static_configs:
+          - targets: ["10.0.58.113:5200"]
+            labels:
+              instance: "pirelay"
+              host: "signal-a.iamworkin.lan"
+              service: "signalcontrol-pi"
+              vlan: "home"
+              device: "pi3-ks0212"
+
      # Epson ET-3750 EcoTank Printer SNMP
      - job_name: "snmp-printer"
        scrape_interval: 5m
@@ -479,11 +492,11 @@ data:
              - "https://gitea.iamworkin.lan/"
              - "https://argocd.iamworkin.lan/"
              - "https://intranet.iamworkin.lan/"
-              - "https://signage.iamworkin.lan/"
+              - "https://signage.iamworkin.lan/healthz"   # root 401 auth-gated 2026-06-01; /healthz anon 200
              - "https://kiosk.iamworkin.lan/"
              - "https://media.iamworkin.lan/"
-              - "https://mysql.iamworkin.lan/"
-              - "https://php.iamworkin.lan/"
+              - "https://mysql.iamworkin.lan/healthz"   # root 401 auth-gated 2026-06-01; /healthz anon 200
+              - "https://php.iamworkin.lan/healthz"     # root 401 auth-gated 2026-06-01; /healthz anon 200
              - "https://zabbix.iamworkin.lan/"
              - "https://desktop.iamworkin.lan/"
              - "https://print.iamworkin.lan/"
@@ -4051,6 +4064,9 @@ spec:
            - name: dashboards-remotedesktop
              mountPath: /var/lib/grafana/dashboards/remotedesktop
              readOnly: true
+            - name: dashboards-signalcontrol
+              mountPath: /var/lib/grafana/dashboards/signalcontrol
+              readOnly: true
            - name: datasource-provisioning
              mountPath: /etc/grafana/provisioning/datasources
              readOnly: true
@@ -4104,6 +4120,9 @@ spec:
        - name: dashboards-remotedesktop
          configMap:
            name: grafana-dashboard-remotedesktop
+        - name: dashboards-signalcontrol
+          configMap:
+            name: grafana-dashboard-signalcontrol
        - name: datasource-provisioning
          configMap:
            name: grafana-datasource-provisioning
--- a/tests/bluejay-infra-lint/FleetManifestLintTests.cs
+++ b/tests/bluejay-infra-lint/FleetManifestLintTests.cs
@@ -227,6 +227,50 @@ public sealed class FleetManifestLintTests
        violations.Should().BeEmpty();
    }

+    [Fact]
+    public void SignalControlDeployment_MustKeepAuthOffAndStageOidcSecret()
+    {
+        var deployment = Inventory.Documents.Single(document =>
+            document.Kind == "Deployment"
+            && document.Namespace == "fc-signalcontrol"
+            && document.Name == "signalcontrol-web"
+            && document.RelativePath == "fc-signalcontrol/fc-signalcontrol.yaml");
+        var container = deployment.MainContainerMappings().Single(container =>
+            ManifestNodeExtensions.Scalar(container, "name") == "signalcontrol-web");
+
+        EnvValue(container, "Auth__Enabled").Should().Be("false");
+        EnvValue(container, "FlowerCore__Auth__Enabled").Should().Be("false");
+        EnvValue(container, "FlowerCore__Auth__Oidc__Enabled").Should().Be("true");
+        EnvSecretName(container, "FlowerCore__Auth__Oidc__Authority").Should().Be("signalcontrol-oidc-client");
+        EnvSecretKey(container, "FlowerCore__Auth__Oidc__Authority").Should().Be("issuer_url");
+        EnvSecretName(container, "FlowerCore__Auth__Oidc__ClientId").Should().Be("signalcontrol-oidc-client");
+        EnvSecretKey(container, "FlowerCore__Auth__Oidc__ClientId").Should().Be("client_id");
+        EnvSecretName(container, "FlowerCore__Auth__Oidc__ClientSecret").Should().Be("signalcontrol-oidc-client");
+        EnvSecretKey(container, "FlowerCore__Auth__Oidc__ClientSecret").Should().Be("client_secret");
+        EnvSecretOptional(container, "FlowerCore__Auth__Oidc__Authority").Should().BeTrue();
+        EnvSecretOptional(container, "FlowerCore__Auth__Oidc__ClientId").Should().BeTrue();
+        EnvSecretOptional(container, "FlowerCore__Auth__Oidc__ClientSecret").Should().BeTrue();
+    }
+
+    [Fact]
+    public void SignalControlDeployment_MustWirePirelayRelayBridgeSecret()
+    {
+        var deployment = Inventory.Documents.Single(document =>
+            document.Kind == "Deployment"
+            && document.Namespace == "fc-signalcontrol"
+            && document.Name == "signalcontrol-web"
+            && document.RelativePath == "fc-signalcontrol/fc-signalcontrol.yaml");
+        var container = deployment.MainContainerMappings().Single(container =>
+            ManifestNodeExtensions.Scalar(container, "name") == "signalcontrol-web");
+
+        EnvValue(container, "TrafficSignal__RelayBridge__Enabled").Should().Be("true");
+        EnvValue(container, "TrafficSignal__RelayBridge__BaseUrl").Should().Be("https://pirelay.iamworkin.lan");
+        EnvSecretName(container, "TrafficSignal__RelayBridge__ApiKey").Should().Be("signalcontrol-pirelay");
+        EnvSecretKey(container, "TrafficSignal__RelayBridge__ApiKey").Should().Be("ApiKey");
+        EnvSecretOptional(container, "TrafficSignal__RelayBridge__ApiKey").Should().BeTrue();
+        EnvValue(container, "LiveStatus__TrafficSignal__BaseAddress").Should().Be("https://signalcontrol.iamworkin.lan");
+    }
+
    [Fact]
    public void GitHubRunnerFleet_MustRegisterRequiredReposAsRepoScopedDeployments()
    {
@@ -424,6 +468,36 @@ public sealed class FleetManifestLintTests
        monitoring.Should().Contain("alert_channel: irc");
    }

+    [Fact]
+    public void Monitoring_MustScrapeSignalControlPiAppAndMountDashboard()
+    {
+        var monitoring = File.ReadAllText(Path.Combine(Inventory.BluejayRoot, "apps", "monitoring", "noc-monitoring.yaml"));
+
+        monitoring.Should().Contain("job_name: \"signalcontrol-pi-app\"");
+        monitoring.Should().Contain("metrics_path: /metrics/prometheus");
+        monitoring.Should().Contain("10.0.58.113:5200");
+        monitoring.Should().Contain("host: \"signal-a.iamworkin.lan\"");
+        monitoring.Should().Contain("mountPath: /var/lib/grafana/dashboards/signalcontrol");
+        monitoring.Should().Contain("name: grafana-dashboard-signalcontrol");
+    }
+
+    [Fact]
+    public void SignalControlGrafanaDashboard_MustCoverAppNodeAndPhysicalControlMetrics()
+    {
+        var dashboard = File.ReadAllText(Path.Combine(
+            Inventory.BluejayRoot,
+            "apps",
+            "monitoring",
+            "grafana-dashboard-signalcontrol.yaml"));
+
+        dashboard.Should().Contain("uid\": \"flowercore-signalcontrol\"");
+        dashboard.Should().Contain("up{job=\\\"signalcontrol-pi-app\\\",instance=\\\"pirelay\\\"}");
+        dashboard.Should().Contain("up{job=\\\"edge-nodes\\\",instance=\\\"pirelay\\\"}");
+        dashboard.Should().Contain("signal_relay_writes_total");
+        dashboard.Should().Contain("signal_schedule_fires_total");
+        dashboard.Should().Contain("signalcontrol_screen_saver_enabled");
+    }
+
    [Fact]
    public void StatefulSets_WithVolumeClaimTemplates_MustDeclareFilesystemDefaults()
    {
@@ -762,6 +836,16 @@ public sealed class FleetManifestLintTests
            : null;
    }

+    private static bool EnvSecretOptional(YamlMappingNode container, string name)
+    {
+        return string.Equals(
+            EnvMapping(container, name) is { } env
+                ? ManifestNodeExtensions.Scalar(env, "valueFrom", "secretKeyRef", "optional")
+                : null,
+            "true",
+            StringComparison.Ordinal);
+    }
+
    private static YamlMappingNode? EnvMapping(YamlMappingNode container, string name)
    {
        return ManifestNodeExtensions.MappingSequence(container, "env")
Author	SHA1	Message	Date
Andrew Stoltz	2c8968f5d0	Wire SignalControl platform observability	2026-06-02 03:02:13 -05:00
Andrew Stoltz	0307ae16ae	monitoring(probe): signage/mysql/php blackbox probe / -> /healthz (K8s-target mirror) Mirrors the live noc1 podman fix + Notes scripts/monitoring/prometheus.yml. These services enforce OIDC bearer auth (FlowerCore__Auth__Enabled=true), so an anonymous probe of / returns 401 -> false TraefikServiceDown. All three expose anonymous /healthz=200. This noc-monitoring.yaml is the forward K8s-migration target (not live); brings it in sync with the live config.	2026-06-02 01:09:57 -05:00
Andrew Stoltz	6c18f69cf2	mail: remove cert-manager Certificate (manage mail-tls via step-ca JWK + noc1 renew timer) step-ca-acme only has an HTTP-01 (Traefik) solver, but mail.iamworkin.lan must resolve to the dedicated MetalLB IP 10.0.56.202 (SMTP/IMAP), so HTTP-01 cannot validate (order stuck pending since 2026-05-06; cert expired 2026-05-24). mail-tls is now issued from step-ca's JWK 'admin' provisioner and auto-renewed by a systemd timer on noc1 that writes the mail-tls secret directly. The secret + Deployment mount + webmail IngressRoute are unchanged. Re-add a Certificate only if a DNS-01 solver is deployed for step-ca-acme. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-01 15:55:38 -05:00
Andrew Stoltz	47e2256556	Deploy TtsReader correction bridge images	2026-05-31 12:35:45 -05:00
Andrew Stoltz	9d77f8ba0e	fc-updater: disable loki audit sink	2026-05-31 11:34:12 -05:00
Andrew Stoltz	2f4be19c85	fc-updater: bump signing diagnostics image	2026-05-31 00:32:48 -05:00
Andrew Stoltz	2a62c40990	fc-updater: bump image to MSI installer surface	2026-05-30 23:31:48 -05:00
Andrew Stoltz	7be98e5efc	Bump UpdateCenter image to hosted-service fix	2026-05-30 20:22:13 -05:00
Andrew Stoltz	a65b356c9d	deploy(fc-updater): roll UC to v202605301823-a6c3354 (Phase 3 SQLite fixes) Durable image bump for FlowerCore.Updater main a6c3354 (PRs #63-#66): hosted-service + request-path SQLite DateTimeOffset fixes, StopHost restored + per-tick resilience, Shared.Settings 1.0.1. Image built + imported to rke2-server. Un-degrades the Phase-9 provenance verifier + settings poll (were stopped under the removed global Ignore mask). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-05-30 18:27:45 -05:00
Andrew Stoltz	08c17ef1b4	fc-updater: bump to v202605301703-296f350-fix2 (BackgroundServiceExceptionBehavior=Ignore so a hosted-service SQLite query crash can't stop the host) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-05-30 17:04:54 -05:00
Andrew Stoltz	06f2f002b7	fc-updater: bump image to v202605301657-296f350-fix1 (Shared.Settings SQLite poll fix) The v202605301642-296f350-rework image crash-looped: FlowerCore.Shared.Settings SettingsDbPollHostedService ran a DateTimeOffset Where/OrderBy on SettingsRecordChanges that SQLite can't translate, and as a BackgroundService it stopped the host. Shared.Settings 1.0.1 materializes the change-log then filters/orders in memory; Updater Web bumped to 1.0.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-05-30 16:59:37 -05:00
Andrew Stoltz	7ac4a8b4b7	fc-updater: bump image to v202605301642-296f350-rework (ADR-179 rework live) Deploy the current FlowerCore.Updater main (PRs #52-#61) to prod: MSI-first packaging, beta gating + per-install tokens, interactive+bearer Authentik OIDC, native installer apply, and the .fcsetup.exe retirement (DropReleaseInstallers migration runs on the now-empty DB). Image pre-imported to rke2-server + agent1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-05-30 16:47:28 -05:00
Andrew Stoltz	90f2a86819	ops: trim load for degraded 2-node cluster (agent2 PSU dead) Scale all github-runner deployments to 1 replica and halt the ci1 KubeVirt VM. With agent2 down (failed PSU) the cluster runs on two passively-cooled NUCs; the ci1 8-vCPU VM drove agent1 to ~100C. Keep total load trimmed until replacement hardware is in place. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 13:47:13 -05:00
Andrew Stoltz	cbdefb2b23	Revert "ci1: expose WinRM/RDP/SSH ports on masquerade interface for Phase 2 bootstrap" The port additions caused the new VMI to stick at phase=Scheduled with reason=GuestNotRunning. The guest-console-log sidecar exited 1 and qemu never started. Reverting to the working 9-day-stable shape until the port-add path is verified in a non-production VM. Phase 2 (Windows runner install + registration) needs an operator- interactive virtctl-vnc session against the rebuilt VM, OR a separate investigation of why this port-add tipped over the VM. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 11:35:10 -05:00
Andrew Stoltz	1c36fe3a0a	ci1: expose WinRM/RDP/SSH ports on masquerade interface for Phase 2 bootstrap The Phase 1 VM has been Running for 9 days but Phase 2 (Puppet bootstrap + runner registration) was deferred because the operator-interactive virtctl-vnc path was the only way in. The masquerade interface listed no exposed ports, so virtctl ssh and kubectl port-forward both hit 'no route to host' — qemu user-mode NAT does not forward inbound by default. Adding 5985 (WinRM HTTP) lets a kubectl port-forward + PowerShell remoting path drive runner registration entirely from outside the VM. 3389 + 22 are reserved for desktop access via Guacamole or virtctl ssh once OpenSSH Server is installed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 11:24:34 -05:00
Andrew Stoltz	2b420ce8a4	runners: fleet-wide right-size CPU requests from 500m to 100m All 33 runner Deployments now request 100m CPU instead of 500m, freeing roughly 50 idle pods × 400m = ~20 cores back to the cluster. Observed CPU usage on idle runners is ~1m via kubectl top; the 500m request was a 500× over-provision that was eating allocatable CPU and blocking new workload scheduling — WorldBuilder runner could not be scheduled even at the new 100m request because the pre-existing fleet held the cluster at 99% requested. Burst headroom preserved by limits.cpu: 2000m unchanged. TtsReader keeps its 8Gi memory limit from the 2026-05-25 OOMKill fix; only the CPU request line moves. Recreate strategy on each deployment means a brief offline window per runner during rollout; in-flight CI jobs complete on the existing container before the new spec takes effect. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 10:09:24 -05:00
Andrew Stoltz	5cbc1a06b1	runners: scale DM/AiStation.Linux/WorldBuilder down to 1 replica until cluster relieved After cutting requests to 100m, 4 of 6 new pods scheduled and 2 stayed Pending — cluster CPU REQUEST utilization is 49.6 of 48 allocatable cores because the existing fleet of ~50 idle runners reserves 25.6 cores (500m × ~50) for ~50m actual use. Single-replica per new repo gets the service online without competing with in-flight CI from the rest of the fleet. When the broader fleet-wide request right-sizing pass lands (500m → 100m on all idle runners would free ~20 cores), these can be bumped back to 2 replicas if PR-CI backlog warrants it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 10:03:30 -05:00
Andrew Stoltz	9e7ee39b3a	runners: drop CPU request 500m→100m on DM/AiStation.Linux/WorldBuilder All 3 fleet nodes were at 99% CPU REQUEST allocation; the 6 new pods from the previous commit (3 deployments × 2 replicas × 500m) couldn't schedule. Idle runners actually use ~1m CPU per `kubectl top pods`; the 500m request was significantly over-provisioned. Burst headroom preserved by limits.cpu: 2000m unchanged. Follow-up: similar request right-sizing pass across the rest of the runner fleet is queued for a future morning-routine sweep — 25 cores reserved for ~50m actual use is a large slack we can reclaim cluster- wide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 10:00:23 -05:00