fc-apple-mdm: add NanoHUB GitOps workload

Ships the Blazor circuit-resilience + mood + telemetry fixes to live chat:
- FcAiChat reconnect-resync (stuck _generating after a circuit drop)
- fc-blazor-start.js client serverTimeout 60s (fewer spurious 1006 reconnects)
- ChatToolVisibility (tool plumbing hidden in personality chat)
- mood empathy fix (avatar no longer "excited" on bad news)
- fc-circuit-telemetry.js + /api/clientlog sink (kubectl-readable circuit data)

Image built on noc1 + imported to rke2-server + rke2-agent1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

apps/fc-apple-mdm/1password-item.yaml

@@ -0,0 +1,32 @@
+# Runtime secret placeholder for the self-hosted Apple MDM substrate.
+#
+# OnePasswordItem operator syncs this item into a Kubernetes Secret with the
+# same name. Expected fields for MDM-N1:
+#   NANOHUB_API_KEY
+#
+# Optional fields for later lanes:
+#   NANOHUB_WEBHOOK_URL
+#   APNS_MDM_CERT_PEM
+#   APNS_MDM_KEY_PEM
+#   APNS_MDM_TOPIC
+#   SCEP_CA_CERT_PEM
+#   SCEP_CA_KEY_PEM
+#   PROFILE_SIGNING_CERT_PEM
+#   PROFILE_SIGNING_KEY_PEM
+#
+# Do not commit APNs, SCEP, profile-signing, webhook, or API key material to
+# Git. MDM-N1 only consumes NANOHUB_API_KEY and optional NANOHUB_WEBHOOK_URL.
+apiVersion: onepassword.com/v1
+kind: OnePasswordItem
+metadata:
+  name: fc-apple-mdm-runtime
+  namespace: fc-apple-mdm
+  labels:
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: secrets
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+spec:
+  itemPath: "vaults/IAmWorkin/items/FlowerCore Apple MDM Runtime"

apps/fc-apple-mdm/README.md

@@ -0,0 +1,65 @@
+# FlowerCore Apple MDM Infra
+
+This app hosts the private NanoHUB bootstrap service for FlowerCore iPad
+management at `https://mdm.iamworkin.lan`.
+
+## Runtime Shape
+
+- Namespace: `fc-apple-mdm`
+- Host: `mdm.iamworkin.lan`
+- Image: `localhost/fc-apple-mdm-nanohub:v0.2.0-20260617`
+- Upstream baseline: NanoHUB `v0.2.0`, published 2025-12-25
+- Persistent data: `fc-apple-mdm-data` mounted at `/var/lib/nanohub`
+- NanoHUB file backend root: `/var/lib/nanohub/db`
+- Runtime secret: `OnePasswordItem/fc-apple-mdm-runtime`
+- Required secret field: `NANOHUB_API_KEY`
+- Optional secret field: `NANOHUB_WEBHOOK_URL`
+
+NanoHUB listens on HTTP `:9004` inside the pod; Traefik owns TLS using
+`Certificate/fc-apple-mdm-tls`. The public route intentionally exposes only
+`/mdm`, `/checkin`, and `/version`. The NanoHUB APIs under `/api/v1/*` stay
+cluster-internal for MDM-N1 and are intended for the FlowerCore
+DeviceManagement bridge.
+
+## NanoHUB Endpoints
+
+- Device command/report and default check-in endpoint: `/mdm`
+- Separate check-in endpoint enabled by `NANOHUB_CHECKIN=true`: `/checkin`
+- Health/version endpoint: `/version`
+- Internal NanoMDM API: `/api/v1/nanomdm/`
+- Internal NanoCMD API: `/api/v1/nanocmd/`
+- Internal KMFDDM API: `/api/v1/ddm/`
+
+NanoHUB API authentication is HTTP Basic with username `nanohub` and password
+from `NANOHUB_API_KEY`.
+
+## Operator Gates
+
+1. Create `FlowerCore Apple MDM Runtime` in the `IAmWorkin` 1Password vault with
+   field `NANOHUB_API_KEY`. Add `NANOHUB_WEBHOOK_URL` only after the
+   DeviceManagement Nano bridge endpoint is live.
+2. Add or confirm `mdm.iamworkin.lan -> 10.0.56.200` in FlowerCore.DNS/pfSense
+   before cert-manager syncs the certificate.
+3. Mirror or build the pinned NanoHUB image, then import it on every schedulable
+   RKE2 node:
+
+   ```bash
+   podman pull --arch arm64 ghcr.io/micromdm/nanohub:latest@sha256:e36a50db2dc3d2bf736645e58712f622c04b05b28487390981905ef4d0be5fbd
+   podman tag ghcr.io/micromdm/nanohub@sha256:e36a50db2dc3d2bf736645e58712f622c04b05b28487390981905ef4d0be5fbd localhost/fc-apple-mdm-nanohub:v0.2.0-20260617
+   podman save localhost/fc-apple-mdm-nanohub:v0.2.0-20260617 -o fc-apple-mdm-nanohub-v0.2.0-20260617.tar
+   # copy to each RKE2 node, then:
+   sudo ctr -n k8s.io images import fc-apple-mdm-nanohub-v0.2.0-20260617.tar
+   ```
+
+   If GHCR changes or becomes unavailable, rebuild/import from
+   `nanohub-linux-arm64-v0.2.0.zip` with SHA-256
+   `b05968322a9bc34e79169ebee28d16554046f981eaee48a12cf80899f51a9dbd`.
+
+4. Sync the ArgoCD app and prove `https://mdm.iamworkin.lan/version`.
+
+## Support Boundary
+
+This MDM-N1 lane deploys the protocol substrate only. It does not create an APNs
+MDM push certificate, enrollment profile, SCEP/device identity service, managed
+Wi-Fi payload, managed app install, or supervised iPad enrollment. Those stay in
+MDM-N2 through MDM-N8.

apps/fc-apple-mdm/certificate.yaml

@@ -0,0 +1,29 @@
+# Certificate for mdm.iamworkin.lan.
+#
+# Preflight gate: FlowerCore.DNS / pfSense must contain an explicit A record:
+#   mdm.iamworkin.lan -> 10.0.56.200
+# before this Certificate is synced. step-ca ACME cannot see the CoreDNS
+# wildcard, so missing pfSense DNS produces cert-manager HTTP-01 backoff.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: fc-apple-mdm-tls
+  namespace: fc-apple-mdm
+  labels:
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+  annotations:
+    flowercore.io/dns-preflight: "mdm.iamworkin.lan must resolve to 10.0.56.200 before ACME sync"
+spec:
+  secretName: fc-apple-mdm-tls
+  issuerRef:
+    name: step-ca-acme
+    kind: ClusterIssuer
+  dnsNames:
+    - mdm.iamworkin.lan
+  duration: 720h
+  renewBefore: 240h

apps/fc-apple-mdm/deployment.yaml

@@ -0,0 +1,127 @@
+# Self-hosted NanoHUB lane for FlowerCore Apple device management.
+#
+# Image contract:
+#   Mirror/import localhost/fc-apple-mdm-nanohub:v0.2.0-20260617 from
+#   ghcr.io/micromdm/nanohub:latest@sha256:e36a50db2dc3d2bf736645e58712f622c04b05b28487390981905ef4d0be5fbd
+#   or rebuild from nanohub-linux-arm64-v0.2.0.zip with SHA-256
+#   b05968322a9bc34e79169ebee28d16554046f981eaee48a12cf80899f51a9dbd.
+#   Keep imagePullPolicy: Never so the RKE2 nodes do not depend on GHCR at
+#   runtime.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: fc-apple-mdm
+  namespace: fc-apple-mdm
+  labels:
+    app: fc-apple-mdm
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+  annotations:
+    flowercore.io/traceability-standard: k8s-pod-ownership-and-traceability-standard
+spec:
+  replicas: 1
+  revisionHistoryLimit: 3
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: fc-apple-mdm
+  template:
+    metadata:
+      labels:
+        app: fc-apple-mdm
+        app.kubernetes.io/name: fc-apple-mdm
+        app.kubernetes.io/component: mdm
+        app.kubernetes.io/part-of: flowercore
+        app.kubernetes.io/managed-by: argocd
+        flowercore.io/tenant-id: system
+        flowercore.io/created-by: bluejay-infra
+      annotations:
+        fc.flowercore.io/healthz-anon: "true"
+        fc.flowercore.io/probe-path: "/version"
+        prometheus.io/scrape: "false"
+        flowercore.io/audit-trace-id: "apple-mdm-nanohub-runtime-trace"
+    spec:
+      securityContext:
+        fsGroup: 1654
+        fsGroupChangePolicy: OnRootMismatch
+      containers:
+        - name: nanohub
+          image: localhost/fc-apple-mdm-nanohub:v0.2.0-20260617
+          imagePullPolicy: Never
+          ports:
+            - name: http
+              containerPort: 9004
+          env:
+            - name: HOME
+              value: "/var/lib/nanohub"
+            - name: NANOHUB_LISTEN
+              value: ":9004"
+            - name: NANOHUB_STORAGE
+              value: "file"
+            - name: NANOHUB_STORAGE_DSN
+              value: "/var/lib/nanohub/db"
+            - name: NANOHUB_CHECKIN
+              value: "true"
+            - name: NANOHUB_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: fc-apple-mdm-runtime
+                  key: NANOHUB_API_KEY
+            - name: NANOHUB_WEBHOOK_URL
+              valueFrom:
+                secretKeyRef:
+                  name: fc-apple-mdm-runtime
+                  key: NANOHUB_WEBHOOK_URL
+                  optional: true
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
+          startupProbe:
+            httpGet:
+              path: /version
+              port: 9004
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            failureThreshold: 30
+          readinessProbe:
+            httpGet:
+              path: /version
+              port: 9004
+            periodSeconds: 10
+            failureThreshold: 3
+          livenessProbe:
+            httpGet:
+              path: /version
+              port: 9004
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            failureThreshold: 3
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1654
+            runAsGroup: 1654
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          volumeMounts:
+            - name: data
+              mountPath: /var/lib/nanohub
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: fc-apple-mdm-data
+        - name: tmp
+          emptyDir: {}

apps/fc-apple-mdm/ingressroute.yaml

@@ -0,0 +1,29 @@
+# LAN ingress for NanoHUB.
+#
+# Traefik terminates step-ca TLS; NanoHUB listens on HTTP :9004 and serves the
+# Apple MDM protocol endpoints. The NanoHUB API stays cluster-internal for
+# MDM-N1; do not route /api/v1 through Traefik until the operator approves an
+# API exposure model.
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: fc-apple-mdm
+  namespace: fc-apple-mdm
+  labels:
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`mdm.iamworkin.lan`) && (PathPrefix(`/mdm`) || PathPrefix(`/checkin`) || PathPrefix(`/version`))
+      kind: Rule
+      services:
+        - name: fc-apple-mdm
+          port: 80
+  tls:
+    secretName: fc-apple-mdm-tls

apps/fc-apple-mdm/kustomization.yaml

@@ -0,0 +1,13 @@
+# ArgoCD's bluejay-infra ApplicationSet discovers apps/* directories on main.
+# The kustomization is included for local previews and single-app validation.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - 1password-item.yaml
+  - pvc.yaml
+  - deployment.yaml
+  - service.yaml
+  - certificate.yaml
+  - ingressroute.yaml
+  - network-policy.yaml

apps/fc-apple-mdm/namespace.yaml

@@ -0,0 +1,13 @@
+# FlowerCore Apple MDM namespace.
+#
+# ArgoCD discovers this directory as Application `infra-fc-apple-mdm`.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: fc-apple-mdm
+  labels:
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra

apps/fc-apple-mdm/network-policy.yaml

@@ -0,0 +1,94 @@
+# FlowerCore Apple MDM network isolation.
+#
+# Public/LAN device traffic enters through Traefik. NanoHUB API access is kept
+# cluster-internal for MDM-N1 and is reachable by the DeviceManagement bridge.
+# Egress 443 is required for Apple APNs/ADE/VPP endpoints once APNs and Apple
+# enrollment material are configured in later lanes.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: fc-apple-mdm-isolation
+  namespace: fc-apple-mdm
+  labels:
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+spec:
+  podSelector:
+    matchLabels:
+      app: fc-apple-mdm
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: traefik-system
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: traefik
+      ports:
+        - port: 9004
+          protocol: TCP
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: fc-devicemgmt
+      ports:
+        - port: 9004
+          protocol: TCP
+  egress:
+    # CoreDNS.
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+          podSelector:
+            matchLabels:
+              k8s-app: kube-dns
+      ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Apple APNs/ADE/VPP endpoints and upstream certificate checks.
+    - to:
+        - ipBlock:
+            cidr: 0.0.0.0/0
+      ports:
+        - port: 443
+          protocol: TCP
+    # Traefik VIP / in-cluster Traefik for public URL self-checks. Include
+    # post-DNAT backend ports 8443 + 8080.
+    - to:
+        - ipBlock:
+            cidr: 10.0.56.200/32
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: traefik-system
+          podSelector:
+            matchLabels:
+              app.kubernetes.io/name: traefik
+      ports:
+        - port: 80
+          protocol: TCP
+        - port: 443
+          protocol: TCP
+        - port: 8080
+          protocol: TCP
+        - port: 8443
+          protocol: TCP
+    # DeviceManagement bridge webhook/API target.
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: fc-devicemgmt
+      ports:
+        - port: 80
+          protocol: TCP
+        - port: 8080
+          protocol: TCP

apps/fc-apple-mdm/pvc.yaml

@@ -0,0 +1,25 @@
+# Persistent NanoHUB file backend state.
+#
+# NanoHUB stores NanoMDM, NanoCMD, and KMFDDM data under the file backend root.
+# RWO: keep a single replica and use Recreate for disruptive image/runtime
+# changes.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: fc-apple-mdm-data
+  namespace: fc-apple-mdm
+  labels:
+    app: fc-apple-mdm
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 2Gi

apps/fc-apple-mdm/service.yaml

@@ -0,0 +1,22 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: fc-apple-mdm
+  namespace: fc-apple-mdm
+  labels:
+    app: fc-apple-mdm
+    app.kubernetes.io/name: fc-apple-mdm
+    app.kubernetes.io/component: mdm
+    app.kubernetes.io/part-of: flowercore
+    app.kubernetes.io/managed-by: argocd
+    flowercore.io/tenant-id: system
+    flowercore.io/created-by: bluejay-infra
+spec:
+  type: ClusterIP
+  selector:
+    app: fc-apple-mdm
+  ports:
+    - name: http
+      port: 80
+      targetPort: 9004
+      protocol: TCP

tests/bluejay-infra-lint/FleetManifestLintTests.cs

@@ -814,6 +814,78 @@ public sealed class FleetManifestLintTests
         ns.FileText.Should().Contain("ArgoCD discovers this directory as Application `infra-fc-devicemgmt`.");
     }
 
+    [Fact]
+    public void FcAppleMdm_NanoHubWorkloadMustStayPinnedAndInternalApiOnly()
+    {
+        var documents = AppDocuments("fc-apple-mdm");
+
+        documents.Should().Contain(document => document.Kind == "Namespace" && document.Name == "fc-apple-mdm");
+        documents.Should().Contain(document => document.Kind == "OnePasswordItem" && document.Name == "fc-apple-mdm-runtime");
+        documents.Should().NotContain(document => document.Kind == "Secret");
+
+        var item = documents.Single(document => document.Kind == "OnePasswordItem" && document.Name == "fc-apple-mdm-runtime");
+        item.Scalar("spec", "itemPath").Should().Be("vaults/IAmWorkin/items/FlowerCore Apple MDM Runtime");
+
+        var deployment = documents.Single(document => document.Kind == "Deployment" && document.Name == "fc-apple-mdm");
+        deployment.Scalar("spec", "strategy", "type").Should().Be("Recreate");
+        PodAnnotation(deployment, "fc.flowercore.io/healthz-anon").Should().Be("true");
+        PodAnnotation(deployment, "fc.flowercore.io/probe-path").Should().Be("/version");
+        PodAnnotation(deployment, "flowercore.io/audit-trace-id").Should().Be("apple-mdm-nanohub-runtime-trace");
+
+        var container = deployment.MainContainerMappings().Should().ContainSingle().Subject;
+        ManifestNodeExtensions.Scalar(container, "name").Should().Be("nanohub");
+        ManifestNodeExtensions.Scalar(container, "image").Should().Be("localhost/fc-apple-mdm-nanohub:v0.2.0-20260617");
+        ManifestNodeExtensions.Scalar(container, "imagePullPolicy").Should().Be("Never");
+        EnvValue(container, "NANOHUB_LISTEN").Should().Be(":9004");
+        EnvValue(container, "NANOHUB_STORAGE").Should().Be("file");
+        EnvValue(container, "NANOHUB_STORAGE_DSN").Should().Be("/var/lib/nanohub/db");
+        EnvValue(container, "NANOHUB_CHECKIN").Should().Be("true");
+        EnvSecretName(container, "NANOHUB_API_KEY").Should().Be("fc-apple-mdm-runtime");
+        EnvSecretKey(container, "NANOHUB_API_KEY").Should().Be("NANOHUB_API_KEY");
+        EnvSecretName(container, "NANOHUB_WEBHOOK_URL").Should().Be("fc-apple-mdm-runtime");
+        EnvSecretKey(container, "NANOHUB_WEBHOOK_URL").Should().Be("NANOHUB_WEBHOOK_URL");
+        EnvSecretOptional(container, "NANOHUB_WEBHOOK_URL").Should().Be("true");
+        ProbePath(container, "readinessProbe").Should().Be("/version");
+        ProbePath(container, "startupProbe").Should().Be("/version");
+        ProbePath(container, "livenessProbe").Should().Be("/version");
+
+        var certificate = documents.Single(document => document.Kind == "Certificate" && document.Name == "fc-apple-mdm-tls");
+        certificate.Scalar("spec", "issuerRef", "name").Should().Be("step-ca-acme");
+        certificate.Scalar("spec", "issuerRef", "kind").Should().Be("ClusterIssuer");
+        ManifestNodeExtensions.ScalarSequence(certificate.Root, "spec", "dnsNames")
+            .Should()
+            .ContainSingle("mdm.iamworkin.lan");
+
+        var ingress = documents.Single(document => document.Kind == "IngressRoute" && document.Name == "fc-apple-mdm");
+        var match = ingress.MappingSequence("spec", "routes")
+            .Select(route => ManifestNodeExtensions.Scalar(route, "match") ?? string.Empty)
+            .Should()
+            .ContainSingle()
+            .Subject;
+
+        match.Should().Contain("Host(`mdm.iamworkin.lan`)");
+        match.Should().Contain("PathPrefix(`/mdm`)");
+        match.Should().Contain("PathPrefix(`/checkin`)");
+        match.Should().Contain("PathPrefix(`/version`)");
+        match.Should().NotContain("/api/v1", "NanoHUB API access is cluster-internal for MDM-N1");
+
+        var service = documents.Single(document => document.Kind == "Service" && document.Name == "fc-apple-mdm");
+        service.AllScalars().Should().Contain("9004");
+
+        var policy = documents.Single(document => document.Kind == "NetworkPolicy" && document.Name == "fc-apple-mdm-isolation");
+        policy.AllScalars().Should().Contain(new[]
+        {
+            "traefik-system",
+            "fc-devicemgmt",
+            "10.0.56.200/32",
+        });
+        policy.EgressPorts().Should().Contain(new[] { "53", "80", "443", "8080", "8443" });
+
+        documents.Should().NotContain(document => document.AllScalars().Any(value =>
+            value.Contains("micromdm", StringComparison.OrdinalIgnoreCase)
+            || value.Contains("MICROMDM", StringComparison.Ordinal)));
+    }
+
     [Fact]
     public void BroaderHardeningDeployments_MustAnnotateAnonymousHealthProbeIntent()
     {