feat(github-runner): add top Linux repo runners

fix(github-runner): switch RUNNER_TOKEN -> ACCESS_TOKEN; set RUN_AS_ROOT=false
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 13:55:55 -05:00 · 2026-05-16 22:08:56 +00:00 · 2026-05-16 22:04:03 +00:00 · 2026-05-16 22:01:41 +00:00 · 2026-05-16 10:27:58 -05:00 · 2026-05-15 16:18:19 -05:00
5 changed files with 1188 additions and 2 deletions
--- a/apps/github-runner/README.md
+++ b/apps/github-runner/README.md
@@ -0,0 +1,38 @@
+# github-runner
+
+ArgoCD-managed repo-scoped Linux GitHub Actions runners for FlowerCore.
+
+`astoltz` is a GitHub user account, not an organization, so each repository
+needs its own runner registration. The existing Common runner remains
+`Deployment/github-runner`; Sprint 29 adds one single-replica Deployment for
+each top Linux-cost repo:
+
+- `FlowerCore.Puppet`
+- `FlowerCore.Signage`
+- `FlowerCore.DMS`
+- `FlowerCore.Telephony`
+- `FlowerCore.Print.Web`
+- `FlowerCore.Chat`
+- `FlowerCore.MySQL`
+- `FlowerCore.Kiosk.Linux`
+
+Each runner uses `myoung34/github-runner:latest`, `EPHEMERAL=true`, and labels
+`self-hosted,linux,fc-build-linux`. The shared `github-runner-token` Secret is
+synced from the existing 1Password item `GitHub PAT (Runner Registration)` and
+is consumed as `ACCESS_TOKEN`.
+
+Do not `kubectl apply` this app over ArgoCD. Merge to `main`, let
+`infra-github-runner` sync, then verify from `noc1`:
+
+```bash
+kubectl -n github-runner get deploy,pods,pvc
+
+for repo in FlowerCore.Puppet FlowerCore.Signage FlowerCore.DMS FlowerCore.Telephony FlowerCore.Print.Web FlowerCore.Chat FlowerCore.MySQL FlowerCore.Kiosk.Linux; do
+  gh api "/repos/astoltz/$repo/actions/runners" \
+    --jq '.runners[] | select((.labels[].name == "fc-build-linux") and (.status == "online")) | {name,status,busy,labels:[.labels[].name]}'
+done
+```
+
+`LinuxRunnerOffline` is declared in `apps/monitoring/noc-monitoring.yaml` and
+fires when any Common or top-8 Linux runner deployment has no available replica
+for 10 minutes.
--- a/apps/github-runner/github-runner.yaml
+++ b/apps/github-runner/github-runner.yaml
@@ -0,0 +1,999 @@
+# GitHub Actions self-hosted Linux runner fleet
+#
+# ArgoCD owns this namespace. Do not kubectl-apply ad hoc runner changes over
+# it; update this manifest and let the bluejay-infra ApplicationSet reconcile.
+#
+# astoltz is a GitHub user account, not an org, so runners must be repo-scoped.
+# Each Deployment below registers exactly one ephemeral myoung34/github-runner
+# instance against one private FlowerCore repo using the shared PAT from the
+# github-runner-token Secret.
+#
+# Current shape:
+#   - Common runner preserved from the phase-2 pilot.
+#   - Sprint 29 top-8 Linux-cost repos added first:
+#     Puppet, Signage, DMS, Telephony, Print.Web, Chat, MySQL, Kiosk.Linux.
+#
+# Security:
+#   - No ClusterRole / ClusterRoleBinding.
+#   - ServiceAccount has no K8s API privileges.
+#   - Self-hosted runners are for private repos and trusted branches only.
+#   - Fork pull-request approval must remain required in GitHub repo settings.
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: github-runner
+  labels:
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+---
+# 1Password secret sync — creates github-runner-token K8s Secret.
+# Fields expected in the 1Password item:
+#   credential — GitHub fine-grained PAT with Administration:read/write on
+#                each target repo. myoung34/github-runner uses ACCESS_TOKEN to
+#                mint fresh short-lived registration tokens at pod startup.
+# Item path: IAmWorkin vault > "GitHub PAT (Runner Registration)"
+apiVersion: onepassword.com/v1
+kind: OnePasswordItem
+metadata:
+  name: github-runner-token
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: credentials
+    app.kubernetes.io/part-of: flowercore
+spec:
+  itemPath: vaults/IAmWorkin/items/GitHub PAT (Runner Registration)
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: github-runner
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Common
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-puppet-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Puppet
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-signage-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Signage
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-dms-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.DMS
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-telephony-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Telephony
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-print-web-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Print.Web
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-chat-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Chat
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-mysql-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.MySQL
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: github-runner-kiosk-linux-nuget-cache
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: flowercore
+    flowercore.io/github-repo: FlowerCore.Kiosk.Linux
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 5Gi
+  volumeMode: Filesystem
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Common
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Common
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Common"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-puppet
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-puppet
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Puppet
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-puppet
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-puppet
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Puppet
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Puppet"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-puppet"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-puppet-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-signage
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-signage
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Signage
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-signage
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-signage
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Signage
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Signage"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-signage"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-signage-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-dms
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-dms
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.DMS
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-dms
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-dms
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.DMS
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.DMS"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-dms"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-dms-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-telephony
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-telephony
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Telephony
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-telephony
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-telephony
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Telephony
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Telephony"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-telephony"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-telephony-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-print-web
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-print-web
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Print.Web
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-print-web
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-print-web
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Print.Web
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Print.Web"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-print-web"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-print-web-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-chat
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-chat
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Chat
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-chat
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-chat
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Chat
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Chat"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-chat"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-chat-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-mysql
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-mysql
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.MySQL
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-mysql
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-mysql
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.MySQL
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.MySQL"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-mysql"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-mysql-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: github-runner-kiosk-linux
+  namespace: github-runner
+  labels:
+    app.kubernetes.io/name: github-runner-kiosk-linux
+    app.kubernetes.io/component: runner
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/created-by: argocd
+    flowercore.io/github-repo: FlowerCore.Kiosk.Linux
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: github-runner-kiosk-linux
+  strategy:
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: github-runner-kiosk-linux
+        app.kubernetes.io/component: runner
+        app.kubernetes.io/part-of: flowercore
+        flowercore.io/created-by: argocd
+        flowercore.io/github-repo: FlowerCore.Kiosk.Linux
+    spec:
+      serviceAccountName: github-runner
+      nodeSelector:
+        kubernetes.io/hostname: rke2-server
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1001
+        runAsGroup: 1001
+        fsGroup: 1001
+      containers:
+        - name: runner
+          image: myoung34/github-runner:latest
+          imagePullPolicy: Always
+          env:
+            - name: REPO_URL
+              value: "https://github.com/astoltz/FlowerCore.Kiosk.Linux"
+            - name: RUNNER_NAME_PREFIX
+              value: "rke2-linux-kiosk-linux"
+            - name: RUNNER_WORKDIR
+              value: "/tmp/runner/work"
+            - name: EPHEMERAL
+              value: "true"
+            - name: LABELS
+              value: "self-hosted,linux,fc-build-linux"
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: github-runner-token
+                  key: credential
+            - name: RUN_AS_ROOT
+              value: "false"
+          resources:
+            requests:
+              cpu: "500m"
+              memory: "1Gi"
+            limits:
+              cpu: "2000m"
+              memory: "4Gi"
+          volumeMounts:
+            - name: nuget-cache
+              mountPath: /home/runner/.nuget/packages
+            - name: tmp
+              mountPath: /tmp
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - "pgrep -f Runner.Listener > /dev/null"
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+      volumes:
+        - name: nuget-cache
+          persistentVolumeClaim:
+            claimName: github-runner-kiosk-linux-nuget-cache
+        - name: tmp
+          emptyDir: {}
+      restartPolicy: Always
--- a/apps/monitoring/noc-monitoring.yaml
+++ b/apps/monitoring/noc-monitoring.yaml
@@ -974,6 +974,19 @@ data:
              summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
              description: "Spec wants {{ $labels.spec_replicas }} but only {{ $value }} available. Likely a rollout stuck on probe failure, scheduling, or PVC."

+          - alert: LinuxRunnerOffline
+            expr: |
+              kube_deployment_status_replicas_available{namespace="github-runner",deployment=~"github-runner(|-(puppet|signage|dms|telephony|print-web|chat|mysql|kiosk-linux))"} < 1
+            for: 10m
+            labels:
+              severity: warning
+              service: github-runner
+              alert_channel: thermal_print
+            annotations:
+              summary: "Linux GitHub Actions runner offline: {{ $labels.deployment }}"
+              description: "{{ $labels.deployment }} has no available runner pod for 10 minutes. GitHub jobs using [self-hosted, linux, fc-build-linux] for its repo will queue at $0 until the runner returns."
+              runbook_url: "https://gitea.iamworkin.lan/bluejay/FlowerCore.Notes/src/branch/master/docs/infrastructure/self-hosted-runner-fleet.md"
+
          # Q-MR-3 (2026-05-11): multus memory pressure — catches the next OOM
          # cascade BEFORE multus is OOM-killed cluster-wide. The 2026-05-10
          # outage (21h) hit because no alert fired on the rising multus working
@@ -3427,6 +3440,33 @@ data:
                relativeTimeRange: {from: 120, to: 0}
                datasourceUid: __expr__
                model: {type: threshold, expression: B, conditions: [{evaluator: {params: [1], type: lt}}], refId: C}
+          - uid: linux-runner-offline
+            title: LinuxRunnerOffline
+            condition: C
+            for: 10m
+            noDataState: Alerting
+            execErrState: OK
+            annotations:
+              summary: Linux GitHub Actions runner offline
+              description: "A repo-scoped fc-build-linux runner deployment has no available pod. Jobs will queue at $0 until ArgoCD/K8s returns the runner."
+              runbook_url: "https://gitea.iamworkin.lan/bluejay/FlowerCore.Notes/src/branch/master/docs/infrastructure/self-hosted-runner-fleet.md"
+            labels:
+              severity: warning
+              service: github-runner
+              alert_channel: thermal_print
+            data:
+              - refId: A
+                relativeTimeRange: {from: 600, to: 0}
+                datasourceUid: prometheus
+                model: {expr: 'min by(deployment) (kube_deployment_status_replicas_available{namespace="github-runner",deployment=~"github-runner(|-(puppet|signage|dms|telephony|print-web|chat|mysql|kiosk-linux))"})', instant: true, refId: A}
+              - refId: B
+                relativeTimeRange: {from: 600, to: 0}
+                datasourceUid: __expr__
+                model: {type: reduce, expression: A, reducer: last, refId: B}
+              - refId: C
+                relativeTimeRange: {from: 600, to: 0}
+                datasourceUid: __expr__
+                model: {type: threshold, expression: B, conditions: [{evaluator: {params: [1], type: lt}}], refId: C}
          - uid: high-cpu
            title: High CPU (>85%)
            condition: C
--- a/apps/zabbix/zabbix.yaml
+++ b/apps/zabbix/zabbix.yaml
@@ -305,15 +305,17 @@ spec:
              path: /
              port: 8080
            initialDelaySeconds: 60
-            timeoutSeconds: 5
+            timeoutSeconds: 15
            periodSeconds: 10
+            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
-            timeoutSeconds: 5
+            timeoutSeconds: 15
+            failureThreshold: 3
 ---
 apiVersion: v1
 kind: Service
--- a/tests/bluejay-infra-lint/FleetManifestLintTests.cs
+++ b/tests/bluejay-infra-lint/FleetManifestLintTests.cs
@@ -54,6 +54,18 @@ public sealed class FleetManifestLintTests
        "ttsreader-piper",
    };

+    private static readonly IReadOnlyDictionary<string, string> TopLinuxRunnerRepos = new Dictionary<string, string>(StringComparer.Ordinal)
+    {
+        ["github-runner-puppet"] = "https://github.com/astoltz/FlowerCore.Puppet",
+        ["github-runner-signage"] = "https://github.com/astoltz/FlowerCore.Signage",
+        ["github-runner-dms"] = "https://github.com/astoltz/FlowerCore.DMS",
+        ["github-runner-telephony"] = "https://github.com/astoltz/FlowerCore.Telephony",
+        ["github-runner-print-web"] = "https://github.com/astoltz/FlowerCore.Print.Web",
+        ["github-runner-chat"] = "https://github.com/astoltz/FlowerCore.Chat",
+        ["github-runner-mysql"] = "https://github.com/astoltz/FlowerCore.MySQL",
+        ["github-runner-kiosk-linux"] = "https://github.com/astoltz/FlowerCore.Kiosk.Linux",
+    };
+
    [Fact]
    public void IngressRoutes_MustKeepServiceReferencesInTheSameNamespace()
    {
@@ -187,6 +199,76 @@ public sealed class FleetManifestLintTests
        violations.Should().BeEmpty();
    }

+    [Fact]
+    public void GitHubRunnerFleet_MustRegisterTopLinuxReposAsRepoScopedDeployments()
+    {
+        var deployments = Inventory.Documents
+            .Where(document => document.Kind == "Deployment")
+            .Where(document => document.Namespace == "github-runner")
+            .ToDictionary(document => document.Name, StringComparer.Ordinal);
+
+        foreach (var expectedRunner in TopLinuxRunnerRepos)
+        {
+            deployments.Should().ContainKey(expectedRunner.Key);
+
+            var container = deployments[expectedRunner.Key].ContainerMappings().Should().ContainSingle().Subject;
+            EnvValue(container, "REPO_URL").Should().Be(expectedRunner.Value);
+            EnvValue(container, "EPHEMERAL").Should().Be("true");
+            EnvValue(container, "LABELS").Should().Be("self-hosted,linux,fc-build-linux");
+            EnvValue(container, "ACCESS_TOKEN").Should().BeNull("ACCESS_TOKEN must come from github-runner-token Secret, not a literal");
+            EnvSecretName(container, "ACCESS_TOKEN").Should().Be("github-runner-token");
+            EnvSecretKey(container, "ACCESS_TOKEN").Should().Be("credential");
+        }
+    }
+
+    [Fact]
+    public void GitHubRunnerFleet_MustPreserveExistingCommonRunnerShape()
+    {
+        var common = Inventory.Documents
+            .Single(document => document.Kind == "Deployment"
+                && document.Namespace == "github-runner"
+                && document.Name == "github-runner");
+
+        var container = common.ContainerMappings().Should().ContainSingle().Subject;
+        EnvValue(container, "REPO_URL").Should().Be("https://github.com/astoltz/FlowerCore.Common");
+        EnvValue(container, "RUNNER_NAME_PREFIX").Should().Be("rke2-linux");
+        EnvValue(container, "LABELS").Should().Be("self-hosted,linux,fc-build-linux");
+
+        var claimNames = common.MappingSequence("spec", "template", "spec", "volumes")
+            .Select(volume => ManifestNodeExtensions.Scalar(volume, "persistentVolumeClaim", "claimName"))
+            .Where(value => !string.IsNullOrWhiteSpace(value))
+            .ToList();
+
+        claimNames.Should().Contain("github-runner-nuget-cache");
+    }
+
+    [Fact]
+    public void GitHubRunnerFleet_MustUseOneRwoCachePerRepoScopedDeployment()
+    {
+        var pvcNames = Inventory.Documents
+            .Where(document => document.Kind == "PersistentVolumeClaim")
+            .Where(document => document.Namespace == "github-runner")
+            .Select(document => document.Name)
+            .ToHashSet(StringComparer.Ordinal);
+
+        foreach (var deploymentName in TopLinuxRunnerRepos.Keys)
+        {
+            var suffix = deploymentName["github-runner-".Length..];
+            pvcNames.Should().Contain($"github-runner-{suffix}-nuget-cache");
+        }
+    }
+
+    [Fact]
+    public void Monitoring_MustAlertWhenTopLinuxRunnerDeploymentIsUnavailable()
+    {
+        var monitoring = File.ReadAllText(Path.Combine(Inventory.BluejayRoot, "apps", "monitoring", "noc-monitoring.yaml"));
+
+        monitoring.Should().Contain("LinuxRunnerOffline");
+        monitoring.Should().Contain("kube_deployment_status_replicas_available{namespace=\"github-runner\"");
+        monitoring.Should().Contain("github-runner(|-(puppet|signage|dms|telephony|print-web|chat|mysql|kiosk-linux))");
+        monitoring.Should().Contain("runbook_url: \"https://gitea.iamworkin.lan/bluejay/FlowerCore.Notes/src/branch/master/docs/infrastructure/self-hosted-runner-fleet.md\"");
+    }
+
    [Fact]
    public void StatefulSets_WithVolumeClaimTemplates_MustDeclareFilesystemDefaults()
    {
@@ -314,6 +396,31 @@ public sealed class FleetManifestLintTests
            $"{document.Descriptor} container '{containerName}' still uses {probeKey}.httpGet on /health.",
        };
    }
+
+    private static string? EnvValue(YamlMappingNode container, string name)
+    {
+        return EnvMapping(container, name) is { } env ? ManifestNodeExtensions.Scalar(env, "value") : null;
+    }
+
+    private static string? EnvSecretName(YamlMappingNode container, string name)
+    {
+        return EnvMapping(container, name) is { } env
+            ? ManifestNodeExtensions.Scalar(env, "valueFrom", "secretKeyRef", "name")
+            : null;
+    }
+
+    private static string? EnvSecretKey(YamlMappingNode container, string name)
+    {
+        return EnvMapping(container, name) is { } env
+            ? ManifestNodeExtensions.Scalar(env, "valueFrom", "secretKeyRef", "key")
+            : null;
+    }
+
+    private static YamlMappingNode? EnvMapping(YamlMappingNode container, string name)
+    {
+        return ManifestNodeExtensions.MappingSequence(container, "env")
+            .SingleOrDefault(env => string.Equals(ManifestNodeExtensions.Scalar(env, "name"), name, StringComparison.Ordinal));
+    }
 }

 internal sealed class ManifestInventory
Author	SHA1	Message	Date
Codex	266b9cb8be	feat(github-runner): add top Linux repo runners	2026-05-17 13:55:55 -05:00
Codex	6f6ca50987	fix(github-runner): switch RUNNER_TOKEN -> ACCESS_TOKEN; set RUN_AS_ROOT=false Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 22:08:56 +00:00
Codex	c7be58c1f7	chore(github-runner): bump replicas 0 -> 1 (PAT provisioned) Operator provisioned GitHub PAT (Runner Registration) 1P item. OnePasswordItem CRD already materialized the secret. Unblocks CI fleet-wide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 22:04:03 +00:00
Codex	a1f5a393cd	chore(github-runner): rename 1P item to GitHub PAT (Runner Registration) Renames the OnePasswordItem.itemPath from "GitHub Runner Registration Token" to "GitHub PAT (Runner Registration)" so the runner 1P entry sits next to its siblings — GitHub PAT (Gitea Mirrors) and GitHub PAT (NuGet Packages) — under a consistent "GitHub PAT (...)" naming pattern and API_CREDENTIAL category. Existing field "credential" remains the consumer (RUNNER_TOKEN env). Comment block clarified to require Administration:read/write fine-grained PAT scope on target repos. Old 1P item renamed to "[DEPRECATED 2026-05-16] GitHub Runner Registration" — kept as recovery backup; can be hard-deleted after the first successful runner pod start against the new item path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 22:01:41 +00:00
Codex	710340d8be	chore(github-runner): rename 1P item to GitHub PAT (Runner Registration) Renames the OnePasswordItem.itemPath from "GitHub Runner Registration Token" to "GitHub PAT (Runner Registration)" so the runner 1P entry sits next to its siblings — GitHub PAT (Gitea Mirrors) and GitHub PAT (NuGet Packages) — under a consistent "GitHub PAT (...)" naming pattern and API_CREDENTIAL category. Existing field "credential" remains the consumer (RUNNER_TOKEN env). Comment block clarified to require Administration:read/write fine-grained PAT scope on target repos. Old 1P item renamed to "[DEPRECATED 2026-05-16] GitHub Runner Registration" — kept as recovery backup; can be hard-deleted after the first successful runner pod start against the new item path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 10:27:58 -05:00
Andrew Stoltz	7d2daaa4f8	chore(github-runner): replicas 1 → 0 until 1Password token provisioned github-runner-token OnePasswordItem exists but the underlying 1Password vault item hasn't been created yet, so the operator can't mint the K8s Secret. Pod stuck in CreateContainerConfigError → DeploymentReplicasMismatch alert fires. Scaling to 0 keeps the manifest infrastructure intact but stops trying to schedule until operator: 1. Creates "GitHub Runner Registration Token" item in IAmWorkin vault 2. Generates a token at github.com/astoltz/<repo>/settings/actions/runners/new 3. Updates the OnePasswordItem itemPath to point at it 4. Bumps replicas back to 1 via PR	2026-05-15 16:18:19 -05:00
Andrew Stoltz	e50e103ba0	fix(zabbix): bump web probe timeouts 5s→15s + add failureThreshold zabbix-web nginx+PHP-FPM container serves / at ~3-5s baseline with occasional 6-7s spikes (probe path renders full dashboard via PHP). kube-probe was killing the container after 3 consecutive 5s-timeout 499s, producing CrashLoopBackOff alert noise even though the app was serving real traffic fine. 15s timeout absorbs the natural variance; explicit failureThreshold=3 documents the policy (was implicit default). Closes the firing PodCrashLoopBackOff (zabbix-web) + pending HTTPServiceSlow/HTTPServiceDegraded alerts. zabbix.iamworkin.lan remains slow at the application layer (separate work — PHP-FPM warm-up + Zabbix server "host not found" agent lookup spam need their own fixes) but the pod restart loop stops.	2026-05-15 15:59:04 -05:00
Codex	e8094eb0bd	ci(github-runner): add Phase 2 ephemeral Linux runner K8s manifest Namespace github-runner with myoung34/github-runner:latest Deployment, 5Gi Longhorn RWO NuGet cache PVC, zero-privilege ServiceAccount, and OnePasswordItem CRD for the registration token. EPHEMERAL=true mode re-registers after each job; Recreate strategy avoids RWO multi-attach. Targets fc-build-linux label; single replica pinned to rke2-server node. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-14 12:46:25 -05:00