feat(github-runner): add top Linux repo runners

fix(github-runner): switch RUNNER_TOKEN -> ACCESS_TOKEN; set RUN_AS_ROOT=false
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 13:55:55 -05:00 · 2026-05-16 22:08:56 +00:00 · 2026-05-16 22:04:03 +00:00 · 2026-05-16 22:01:41 +00:00 · 2026-05-16 10:27:58 -05:00 · 2026-05-15 16:18:19 -05:00
5 changed files with 1188 additions and 2 deletions
--- a/apps/github-runner/README.md
+++ b/apps/github-runner/README.md
@@ -0,0 +1,38 @@
 # github-runner
 ArgoCD-managed repo-scoped Linux GitHub Actions runners for FlowerCore.
 `astoltz` is a GitHub user account, not an organization, so each repository
 needs its own runner registration. The existing Common runner remains
 `Deployment/github-runner`; Sprint 29 adds one single-replica Deployment for
 each top Linux-cost repo:
 - `FlowerCore.Puppet`
 - `FlowerCore.Signage`
 - `FlowerCore.DMS`
 - `FlowerCore.Telephony`
 - `FlowerCore.Print.Web`
 - `FlowerCore.Chat`
 - `FlowerCore.MySQL`
 - `FlowerCore.Kiosk.Linux`
 Each runner uses `myoung34/github-runner:latest`, `EPHEMERAL=true`, and labels
 `self-hosted,linux,fc-build-linux`. The shared `github-runner-token` Secret is
 synced from the existing 1Password item `GitHub PAT (Runner Registration)` and
 is consumed as `ACCESS_TOKEN`.
 Do not `kubectl apply` this app over ArgoCD. Merge to `main`, let
 `infra-github-runner` sync, then verify from `noc1`:
 ```bash
 kubectl -n github-runner get deploy,pods,pvc
 for repo in FlowerCore.Puppet FlowerCore.Signage FlowerCore.DMS FlowerCore.Telephony FlowerCore.Print.Web FlowerCore.Chat FlowerCore.MySQL FlowerCore.Kiosk.Linux; do
  gh api "/repos/astoltz/$repo/actions/runners" \
    --jq '.runners[] | select((.labels[].name == "fc-build-linux") and (.status == "online")) | {name,status,busy,labels:[.labels[].name]}'
 done
 ```
 `LinuxRunnerOffline` is declared in `apps/monitoring/noc-monitoring.yaml` and
 fires when any Common or top-8 Linux runner deployment has no available replica
 for 10 minutes.
--- a/apps/github-runner/github-runner.yaml
+++ b/apps/github-runner/github-runner.yaml
@@ -0,0 +1,999 @@
 # GitHub Actions self-hosted Linux runner fleet
 #
 # ArgoCD owns this namespace. Do not kubectl-apply ad hoc runner changes over
 # it; update this manifest and let the bluejay-infra ApplicationSet reconcile.
 #
 # astoltz is a GitHub user account, not an org, so runners must be repo-scoped.
 # Each Deployment below registers exactly one ephemeral myoung34/github-runner
 # instance against one private FlowerCore repo using the shared PAT from the
 # github-runner-token Secret.
 #
 # Current shape:
 #   - Common runner preserved from the phase-2 pilot.
 #   - Sprint 29 top-8 Linux-cost repos added first:
 #     Puppet, Signage, DMS, Telephony, Print.Web, Chat, MySQL, Kiosk.Linux.
 #
 # Security:
 #   - No ClusterRole / ClusterRoleBinding.
 #   - ServiceAccount has no K8s API privileges.
 #   - Self-hosted runners are for private repos and trusted branches only.
 #   - Fork pull-request approval must remain required in GitHub repo settings.
 ---
 apiVersion: v1
 kind: Namespace
 metadata:
  name: github-runner
  labels:
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
 ---
 # 1Password secret sync — creates github-runner-token K8s Secret.
 # Fields expected in the 1Password item:
 #   credential — GitHub fine-grained PAT with Administration:read/write on
 #                each target repo. myoung34/github-runner uses ACCESS_TOKEN to
 #                mint fresh short-lived registration tokens at pod startup.
 # Item path: IAmWorkin vault > "GitHub PAT (Runner Registration)"
 apiVersion: onepassword.com/v1
 kind: OnePasswordItem
 metadata:
  name: github-runner-token
  namespace: github-runner
  labels:
    app.kubernetes.io/component: credentials
    app.kubernetes.io/part-of: flowercore
 spec:
  itemPath: vaults/IAmWorkin/items/GitHub PAT (Runner Registration)
 ---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: github-runner
  namespace: github-runner
  labels:
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Common
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-puppet-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Puppet
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-signage-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Signage
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-dms-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.DMS
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-telephony-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Telephony
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-print-web-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Print.Web
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-chat-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Chat
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-mysql-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.MySQL
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: github-runner-kiosk-linux-nuget-cache
  namespace: github-runner
  labels:
    app.kubernetes.io/component: cache
    app.kubernetes.io/part-of: flowercore
    flowercore.io/github-repo: FlowerCore.Kiosk.Linux
 spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
  volumeMode: Filesystem
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Common
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Common
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Common"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-puppet
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-puppet
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Puppet
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-puppet
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-puppet
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Puppet
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Puppet"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-puppet"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-puppet-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-signage
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-signage
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Signage
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-signage
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-signage
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Signage
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Signage"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-signage"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-signage-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-dms
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-dms
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.DMS
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-dms
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-dms
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.DMS
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.DMS"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-dms"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-dms-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-telephony
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-telephony
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Telephony
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-telephony
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-telephony
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Telephony
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Telephony"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-telephony"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-telephony-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-print-web
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-print-web
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Print.Web
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-print-web
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-print-web
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Print.Web
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Print.Web"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-print-web"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-print-web-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-chat
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-chat
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Chat
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-chat
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-chat
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Chat
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Chat"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-chat"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-chat-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-mysql
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-mysql
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.MySQL
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-mysql
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.MySQL
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.MySQL"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-mysql"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-mysql-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: github-runner-kiosk-linux
  namespace: github-runner
  labels:
    app.kubernetes.io/name: github-runner-kiosk-linux
    app.kubernetes.io/component: runner
    app.kubernetes.io/part-of: flowercore
    app.kubernetes.io/managed-by: argocd
    flowercore.io/created-by: argocd
    flowercore.io/github-repo: FlowerCore.Kiosk.Linux
 spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: github-runner-kiosk-linux
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: github-runner-kiosk-linux
        app.kubernetes.io/component: runner
        app.kubernetes.io/part-of: flowercore
        flowercore.io/created-by: argocd
        flowercore.io/github-repo: FlowerCore.Kiosk.Linux
    spec:
      serviceAccountName: github-runner
      nodeSelector:
        kubernetes.io/hostname: rke2-server
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      containers:
        - name: runner
          image: myoung34/github-runner:latest
          imagePullPolicy: Always
          env:
            - name: REPO_URL
              value: "https://github.com/astoltz/FlowerCore.Kiosk.Linux"
            - name: RUNNER_NAME_PREFIX
              value: "rke2-linux-kiosk-linux"
            - name: RUNNER_WORKDIR
              value: "/tmp/runner/work"
            - name: EPHEMERAL
              value: "true"
            - name: LABELS
              value: "self-hosted,linux,fc-build-linux"
            - name: ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-runner-token
                  key: credential
            - name: RUN_AS_ROOT
              value: "false"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "4Gi"
          volumeMounts:
            - name: nuget-cache
              mountPath: /home/runner/.nuget/packages
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "pgrep -f Runner.Listener > /dev/null"
            initialDelaySeconds: 30
            periodSeconds: 30
            failureThreshold: 3
      volumes:
        - name: nuget-cache
          persistentVolumeClaim:
            claimName: github-runner-kiosk-linux-nuget-cache
        - name: tmp
          emptyDir: {}
      restartPolicy: Always
--- a/apps/monitoring/noc-monitoring.yaml
+++ b/apps/monitoring/noc-monitoring.yaml
@@ -974,6 +974,19 @@ data:
              summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
              description: "Spec wants {{ $labels.spec_replicas }} but only {{ $value }} available. Likely a rollout stuck on probe failure, scheduling, or PVC."
          - alert: LinuxRunnerOffline
            expr: |
              kube_deployment_status_replicas_available{namespace="github-runner",deployment=~"github-runner(|-(puppet|signage|dms|telephony|print-web|chat|mysql|kiosk-linux))"} < 1
            for: 10m
            labels:
              severity: warning
              service: github-runner
              alert_channel: thermal_print
            annotations:
              summary: "Linux GitHub Actions runner offline: {{ $labels.deployment }}"
              description: "{{ $labels.deployment }} has no available runner pod for 10 minutes. GitHub jobs using [self-hosted, linux, fc-build-linux] for its repo will queue at $0 until the runner returns."
              runbook_url: "https://gitea.iamworkin.lan/bluejay/FlowerCore.Notes/src/branch/master/docs/infrastructure/self-hosted-runner-fleet.md"
          # Q-MR-3 (2026-05-11): multus memory pressure — catches the next OOM
          # cascade BEFORE multus is OOM-killed cluster-wide. The 2026-05-10
          # outage (21h) hit because no alert fired on the rising multus working
@@ -3427,6 +3440,33 @@ data:
                relativeTimeRange: {from: 120, to: 0}
                datasourceUid: __expr__
                model: {type: threshold, expression: B, conditions: [{evaluator: {params: [1], type: lt}}], refId: C}
          - uid: linux-runner-offline
            title: LinuxRunnerOffline
            condition: C
            for: 10m
            noDataState: Alerting
            execErrState: OK
            annotations:
              summary: Linux GitHub Actions runner offline
              description: "A repo-scoped fc-build-linux runner deployment has no available pod. Jobs will queue at $0 until ArgoCD/K8s returns the runner."
              runbook_url: "https://gitea.iamworkin.lan/bluejay/FlowerCore.Notes/src/branch/master/docs/infrastructure/self-hosted-runner-fleet.md"
            labels:
              severity: warning
              service: github-runner
              alert_channel: thermal_print
            data:
              - refId: A
                relativeTimeRange: {from: 600, to: 0}
                datasourceUid: prometheus
                model: {expr: 'min by(deployment) (kube_deployment_status_replicas_available{namespace="github-runner",deployment=~"github-runner(|-(puppet|signage|dms|telephony|print-web|chat|mysql|kiosk-linux))"})', instant: true, refId: A}
              - refId: B
                relativeTimeRange: {from: 600, to: 0}
                datasourceUid: __expr__
                model: {type: reduce, expression: A, reducer: last, refId: B}
              - refId: C
                relativeTimeRange: {from: 600, to: 0}
                datasourceUid: __expr__
                model: {type: threshold, expression: B, conditions: [{evaluator: {params: [1], type: lt}}], refId: C}
          - uid: high-cpu
            title: High CPU (>85%)
            condition: C
--- a/apps/zabbix/zabbix.yaml
+++ b/apps/zabbix/zabbix.yaml
@@ -305,15 +305,17 @@ spec:
              path: /
              port: 8080
            initialDelaySeconds: 60
-            timeoutSeconds: 5
+            timeoutSeconds: 15
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 5
-            timeoutSeconds: 5
+            timeoutSeconds: 15
            failureThreshold: 3
 ---
 apiVersion: v1
 kind: Service
--- a/tests/bluejay-infra-lint/FleetManifestLintTests.cs
+++ b/tests/bluejay-infra-lint/FleetManifestLintTests.cs
@@ -54,6 +54,18 @@ public sealed class FleetManifestLintTests
        "ttsreader-piper",
    };
    private static readonly IReadOnlyDictionary<string, string> TopLinuxRunnerRepos = new Dictionary<string, string>(StringComparer.Ordinal)
    {
        ["github-runner-puppet"] = "https://github.com/astoltz/FlowerCore.Puppet",
        ["github-runner-signage"] = "https://github.com/astoltz/FlowerCore.Signage",
        ["github-runner-dms"] = "https://github.com/astoltz/FlowerCore.DMS",
        ["github-runner-telephony"] = "https://github.com/astoltz/FlowerCore.Telephony",
        ["github-runner-print-web"] = "https://github.com/astoltz/FlowerCore.Print.Web",
        ["github-runner-chat"] = "https://github.com/astoltz/FlowerCore.Chat",
        ["github-runner-mysql"] = "https://github.com/astoltz/FlowerCore.MySQL",
        ["github-runner-kiosk-linux"] = "https://github.com/astoltz/FlowerCore.Kiosk.Linux",
    };
    [Fact]
    public void IngressRoutes_MustKeepServiceReferencesInTheSameNamespace()
    {
@@ -187,6 +199,76 @@ public sealed class FleetManifestLintTests
        violations.Should().BeEmpty();
    }
    [Fact]
    public void GitHubRunnerFleet_MustRegisterTopLinuxReposAsRepoScopedDeployments()
    {
        var deployments = Inventory.Documents
            .Where(document => document.Kind == "Deployment")
            .Where(document => document.Namespace == "github-runner")
            .ToDictionary(document => document.Name, StringComparer.Ordinal);
        foreach (var expectedRunner in TopLinuxRunnerRepos)
        {
            deployments.Should().ContainKey(expectedRunner.Key);
            var container = deployments[expectedRunner.Key].ContainerMappings().Should().ContainSingle().Subject;
            EnvValue(container, "REPO_URL").Should().Be(expectedRunner.Value);
            EnvValue(container, "EPHEMERAL").Should().Be("true");
            EnvValue(container, "LABELS").Should().Be("self-hosted,linux,fc-build-linux");
            EnvValue(container, "ACCESS_TOKEN").Should().BeNull("ACCESS_TOKEN must come from github-runner-token Secret, not a literal");
            EnvSecretName(container, "ACCESS_TOKEN").Should().Be("github-runner-token");
            EnvSecretKey(container, "ACCESS_TOKEN").Should().Be("credential");
        }
    }
    [Fact]
    public void GitHubRunnerFleet_MustPreserveExistingCommonRunnerShape()
    {
        var common = Inventory.Documents
            .Single(document => document.Kind == "Deployment"
                && document.Namespace == "github-runner"
                && document.Name == "github-runner");
        var container = common.ContainerMappings().Should().ContainSingle().Subject;
        EnvValue(container, "REPO_URL").Should().Be("https://github.com/astoltz/FlowerCore.Common");
        EnvValue(container, "RUNNER_NAME_PREFIX").Should().Be("rke2-linux");
        EnvValue(container, "LABELS").Should().Be("self-hosted,linux,fc-build-linux");
        var claimNames = common.MappingSequence("spec", "template", "spec", "volumes")
            .Select(volume => ManifestNodeExtensions.Scalar(volume, "persistentVolumeClaim", "claimName"))
            .Where(value => !string.IsNullOrWhiteSpace(value))
            .ToList();
        claimNames.Should().Contain("github-runner-nuget-cache");
    }
    [Fact]
    public void GitHubRunnerFleet_MustUseOneRwoCachePerRepoScopedDeployment()
    {
        var pvcNames = Inventory.Documents
            .Where(document => document.Kind == "PersistentVolumeClaim")
            .Where(document => document.Namespace == "github-runner")
            .Select(document => document.Name)
            .ToHashSet(StringComparer.Ordinal);
        foreach (var deploymentName in TopLinuxRunnerRepos.Keys)
        {
            var suffix = deploymentName["github-runner-".Length..];
            pvcNames.Should().Contain($"github-runner-{suffix}-nuget-cache");
        }
    }
    [Fact]
    public void Monitoring_MustAlertWhenTopLinuxRunnerDeploymentIsUnavailable()
    {
        var monitoring = File.ReadAllText(Path.Combine(Inventory.BluejayRoot, "apps", "monitoring", "noc-monitoring.yaml"));
        monitoring.Should().Contain("LinuxRunnerOffline");
        monitoring.Should().Contain("kube_deployment_status_replicas_available{namespace=\"github-runner\"");
        monitoring.Should().Contain("github-runner(|-(puppet|signage|dms|telephony|print-web|chat|mysql|kiosk-linux))");
        monitoring.Should().Contain("runbook_url: \"https://gitea.iamworkin.lan/bluejay/FlowerCore.Notes/src/branch/master/docs/infrastructure/self-hosted-runner-fleet.md\"");
    }
    [Fact]
    public void StatefulSets_WithVolumeClaimTemplates_MustDeclareFilesystemDefaults()
    {
@@ -314,6 +396,31 @@ public sealed class FleetManifestLintTests
            $"{document.Descriptor} container '{containerName}' still uses {probeKey}.httpGet on /health.",
        };
    }
    private static string? EnvValue(YamlMappingNode container, string name)
    {
        return EnvMapping(container, name) is { } env ? ManifestNodeExtensions.Scalar(env, "value") : null;
    }
    private static string? EnvSecretName(YamlMappingNode container, string name)
    {
        return EnvMapping(container, name) is { } env
            ? ManifestNodeExtensions.Scalar(env, "valueFrom", "secretKeyRef", "name")
            : null;
    }
    private static string? EnvSecretKey(YamlMappingNode container, string name)
    {
        return EnvMapping(container, name) is { } env
            ? ManifestNodeExtensions.Scalar(env, "valueFrom", "secretKeyRef", "key")
            : null;
    }
    private static YamlMappingNode? EnvMapping(YamlMappingNode container, string name)
    {
        return ManifestNodeExtensions.MappingSequence(container, "env")
            .SingleOrDefault(env => string.Equals(ManifestNodeExtensions.Scalar(env, "name"), name, StringComparison.Ordinal));
    }
 }
 internal sealed class ManifestInventory
Author	SHA1	Message	Date
Codex	266b9cb8be	feat(github-runner): add top Linux repo runners	2026-05-17 13:55:55 -05:00
Codex	6f6ca50987	fix(github-runner): switch RUNNER_TOKEN -> ACCESS_TOKEN; set RUN_AS_ROOT=false Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 22:08:56 +00:00
Codex	c7be58c1f7	chore(github-runner): bump replicas 0 -> 1 (PAT provisioned) Operator provisioned GitHub PAT (Runner Registration) 1P item. OnePasswordItem CRD already materialized the secret. Unblocks CI fleet-wide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 22:04:03 +00:00
Codex	a1f5a393cd	chore(github-runner): rename 1P item to GitHub PAT (Runner Registration) Renames the OnePasswordItem.itemPath from "GitHub Runner Registration Token" to "GitHub PAT (Runner Registration)" so the runner 1P entry sits next to its siblings — GitHub PAT (Gitea Mirrors) and GitHub PAT (NuGet Packages) — under a consistent "GitHub PAT (...)" naming pattern and API_CREDENTIAL category. Existing field "credential" remains the consumer (RUNNER_TOKEN env). Comment block clarified to require Administration:read/write fine-grained PAT scope on target repos. Old 1P item renamed to "[DEPRECATED 2026-05-16] GitHub Runner Registration" — kept as recovery backup; can be hard-deleted after the first successful runner pod start against the new item path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 22:01:41 +00:00
Codex	710340d8be	chore(github-runner): rename 1P item to GitHub PAT (Runner Registration) Renames the OnePasswordItem.itemPath from "GitHub Runner Registration Token" to "GitHub PAT (Runner Registration)" so the runner 1P entry sits next to its siblings — GitHub PAT (Gitea Mirrors) and GitHub PAT (NuGet Packages) — under a consistent "GitHub PAT (...)" naming pattern and API_CREDENTIAL category. Existing field "credential" remains the consumer (RUNNER_TOKEN env). Comment block clarified to require Administration:read/write fine-grained PAT scope on target repos. Old 1P item renamed to "[DEPRECATED 2026-05-16] GitHub Runner Registration" — kept as recovery backup; can be hard-deleted after the first successful runner pod start against the new item path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-16 10:27:58 -05:00
Andrew Stoltz	7d2daaa4f8	chore(github-runner): replicas 1 → 0 until 1Password token provisioned github-runner-token OnePasswordItem exists but the underlying 1Password vault item hasn't been created yet, so the operator can't mint the K8s Secret. Pod stuck in CreateContainerConfigError → DeploymentReplicasMismatch alert fires. Scaling to 0 keeps the manifest infrastructure intact but stops trying to schedule until operator: 1. Creates "GitHub Runner Registration Token" item in IAmWorkin vault 2. Generates a token at github.com/astoltz/<repo>/settings/actions/runners/new 3. Updates the OnePasswordItem itemPath to point at it 4. Bumps replicas back to 1 via PR	2026-05-15 16:18:19 -05:00
Andrew Stoltz	e50e103ba0	fix(zabbix): bump web probe timeouts 5s→15s + add failureThreshold zabbix-web nginx+PHP-FPM container serves / at ~3-5s baseline with occasional 6-7s spikes (probe path renders full dashboard via PHP). kube-probe was killing the container after 3 consecutive 5s-timeout 499s, producing CrashLoopBackOff alert noise even though the app was serving real traffic fine. 15s timeout absorbs the natural variance; explicit failureThreshold=3 documents the policy (was implicit default). Closes the firing PodCrashLoopBackOff (zabbix-web) + pending HTTPServiceSlow/HTTPServiceDegraded alerts. zabbix.iamworkin.lan remains slow at the application layer (separate work — PHP-FPM warm-up + Zabbix server "host not found" agent lookup spam need their own fixes) but the pod restart loop stops.	2026-05-15 15:59:04 -05:00
Codex	e8094eb0bd	ci(github-runner): add Phase 2 ephemeral Linux runner K8s manifest Namespace github-runner with myoung34/github-runner:latest Deployment, 5Gi Longhorn RWO NuGet cache PVC, zero-privilege ServiceAccount, and OnePasswordItem CRD for the registration token. EPHEMERAL=true mode re-registers after each job; Recreate strategy avoids RWO multi-attach. Targets fc-build-linux label; single replica pinned to rke2-server node. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-14 12:46:25 -05:00