Compare commits
116 Commits
cae03296f5
...
claude/blu
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
851f8e673b | ||
|
|
2489464d4f | ||
|
|
4b777b16ac | ||
|
|
8c60e3a4d3 | ||
|
|
df02b4c3c3 | ||
|
|
c0dceafffd | ||
|
|
490db8f9e6 | ||
|
|
1926bdaf3b | ||
|
|
ca8d062826 | ||
|
|
1889462fc4 | ||
|
|
523ba61232 | ||
|
|
53f67c8713 | ||
|
|
6b9cf3d12c | ||
|
|
0b52093b36 | ||
|
|
7a9098d3bd | ||
|
|
57d7ba46a7 | ||
|
|
9ec2e2d52e | ||
|
|
b4d62a8a50 | ||
|
|
fbbc07023b | ||
|
|
4b0eef0fb0 | ||
|
|
bb09a3786f | ||
|
|
006dbcf671 | ||
|
|
1be71d6ba7 | ||
|
|
0c8026c912 | ||
|
|
621ae47e00 | ||
|
|
ae6b8c0142 | ||
|
|
da55220218 | ||
|
|
b1ad253dd6 | ||
|
|
ee935f6e07 | ||
|
|
2853ee2024 | ||
|
|
b4a34e16ca | ||
|
|
0d5a1fd530 | ||
|
|
1b633f57b2 | ||
|
|
ee8afd0a08 | ||
|
|
cf35884eae | ||
|
|
9881767b11 | ||
|
|
c9bf23834b | ||
|
|
174002023d | ||
|
|
b71f9e4ec9 | ||
|
|
f1431f7324 | ||
|
|
35bd055cb4 | ||
|
|
f604ab419e | ||
|
|
b2786252b0 | ||
|
|
45ee40920d | ||
|
|
8ad7eb714b | ||
|
|
3cb44c3104 | ||
|
|
2400329acd | ||
|
|
c17af882cc | ||
|
|
76b1938afa | ||
|
|
ced04a6148 | ||
|
|
f2258b92a2 | ||
|
|
979a7c7b25 | ||
|
|
0df8f7b936 | ||
|
|
38558641c1 | ||
|
|
63d905b4df | ||
|
|
d95f4e0caf | ||
|
|
7bc565d17e | ||
|
|
dfe9c3b67e | ||
|
|
37f8db89e4 | ||
|
|
00c7d8df24 | ||
|
|
c6811eadd8 | ||
|
|
4d9d537d83 | ||
|
|
0f9d56ee16 | ||
|
|
3bf6511d5d | ||
|
|
3e0b9055b0 | ||
|
|
c828832808 | ||
|
|
e2c71c2b8a | ||
|
|
b3028f5119 | ||
|
|
05a273d3a6 | ||
|
|
ab6ade4e46 | ||
|
|
4848f72eec | ||
|
|
f5eafc5def | ||
|
|
2d3fd74bab | ||
|
|
df4e1f78b0 | ||
|
|
2a10b775a8 | ||
|
|
447ddd339d | ||
|
|
7833143c1c | ||
|
|
8ed77c4627 | ||
|
|
437f346aee | ||
|
|
bc32b5ef04 | ||
|
|
263d06acb9 | ||
|
|
25dbb2967f | ||
|
|
a89a774eaf | ||
|
|
dc39747f3f | ||
|
|
87050e72a9 | ||
|
|
e8c5d2afd2 | ||
|
|
eef492125f | ||
|
|
b51ee35bfa | ||
|
|
4abc2fa95d | ||
|
|
d7628a6945 | ||
|
|
df115e4d1e | ||
|
|
9df26620b8 | ||
|
|
08aa7a5bff | ||
|
|
38e20a8b64 | ||
|
|
c945d44b9e | ||
|
|
1f1354f634 | ||
|
|
76ece92cfd | ||
|
|
a760a58846 | ||
|
|
9fb526c7c5 | ||
|
|
dd7980642e | ||
|
|
1d4ad64226 | ||
|
|
774f82c431 | ||
|
|
d2cc36ea0e | ||
|
|
299070e4bf | ||
|
|
a9debd8668 | ||
|
|
675b9da4f9 | ||
|
|
2b471a55b0 | ||
|
|
37ce0aed85 | ||
|
|
a37fc83584 | ||
|
|
3a8aae9e2d | ||
|
|
020a806d08 | ||
|
|
e65de2938b | ||
|
|
5c0c21790e | ||
|
|
292528ec15 | ||
|
|
bb39a0c1fd | ||
|
|
c23e903ba7 |
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
# .NET build outputs (lint test project)
|
||||||
|
**/bin/
|
||||||
|
**/obj/
|
||||||
|
|
||||||
|
# Editor / temp
|
||||||
|
.DS_Store
|
||||||
|
*.swp
|
||||||
15
README.md
15
README.md
@@ -99,8 +99,23 @@ curl -sk -X DELETE https://dns.iamworkin.lan/api/v1/servers/<serverId>/zones/iam
|
|||||||
- **CoreDNS template + ndots:5 collision**: inside pods, `<svc>.<ns>.svc.cluster.local` with <5 dots gets search-expanded through `iamworkin.lan` FIRST and hits the wildcard template → resolves to Traefik VIP, not the real ClusterIP. Use short service names (`<svc>`) in K8s manifests. See memory `feedback_coredns_ndots_template_collision.md`.
|
- **CoreDNS template + ndots:5 collision**: inside pods, `<svc>.<ns>.svc.cluster.local` with <5 dots gets search-expanded through `iamworkin.lan` FIRST and hits the wildcard template → resolves to Traefik VIP, not the real ClusterIP. Use short service names (`<svc>`) in K8s manifests. See memory `feedback_coredns_ndots_template_collision.md`.
|
||||||
- **Image not on node**: pods stuck `ErrImageNeverPull` means the image wasn't imported to the node Kubernetes scheduled the pod onto. `ctr images import` on all of rke2-server, rke2-agent1, rke2-agent2.
|
- **Image not on node**: pods stuck `ErrImageNeverPull` means the image wasn't imported to the node Kubernetes scheduled the pod onto. `ctr images import` on all of rke2-server, rke2-agent1, rke2-agent2.
|
||||||
- **StatefulSet PVC drift**: `volumeClaimTemplates` needs explicit `volumeMode: Filesystem` or ArgoCD SSA self-heals forever. See memory `feedback_argocd_statefulset_pvc_drift.md`.
|
- **StatefulSet PVC drift**: `volumeClaimTemplates` needs explicit `volumeMode: Filesystem` or ArgoCD SSA self-heals forever. See memory `feedback_argocd_statefulset_pvc_drift.md`.
|
||||||
|
- **IngressRoute namespace split**: this RKE2 Traefik install does not allow cross-namespace service refs. Keep the `IngressRoute`, backend `Service`, and TLS secret in the same namespace; if one host is shared across namespaces, duplicate the `Certificate` and move the route next to the destination service.
|
||||||
|
- **Public read-only hosts**: if a public host fronts a service that also exposes admin writes internally, add a Traefik route match like `Host(...) && (Method(GET) || Method(HEAD))` on the public edge instead of trusting the app to reject unsafe methods.
|
||||||
|
- **Public read-write allowlist hosts**: if a public host accepts a tightly bounded write surface (e.g. bootstrap-JWT POST), pin the allowlist as `(Method(GET) || Method(HEAD) || Method(POST) || Method(OPTIONS))`. PUT/PATCH/DELETE must still 404 at the route. Track A's `updatecenter.iamworkin.lan` / `updates.iamworkin.lan` are the canonical example. The lint test enforces this invariant.
|
||||||
|
- **Traefik VIP netpols**: when a `NetworkPolicy` allows `10.0.56.200`, also allow the post-DNAT backend ports (`8443` for TLS plus `8080` or `8000` for HTTP) or Calico will drop the rewritten flow.
|
||||||
|
- **Auth-safe probes**: services behind API-key or global auth middleware should prefer `tcpSocket` probes unless `/health` is explicitly exempted before the middleware runs.
|
||||||
- **ArgoCD must use internal Gitea URL**: `http://gitea-clusterip.gitea.svc.cluster.local:3000/bluejay/bluejay-infra.git`, not the external HTTPS URL (step-ca cert isn't trusted by ArgoCD). The `ApplicationSet` and any hand-created `Application` must both use the internal URL.
|
- **ArgoCD must use internal Gitea URL**: `http://gitea-clusterip.gitea.svc.cluster.local:3000/bluejay/bluejay-infra.git`, not the external HTTPS URL (step-ca cert isn't trusted by ArgoCD). The `ApplicationSet` and any hand-created `Application` must both use the internal URL.
|
||||||
|
|
||||||
|
## Local manifest lint
|
||||||
|
|
||||||
|
The repo now carries a local-first lint pass for the recurring K8s gotchas that have burned the fleet:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet test tests/bluejay-infra-lint/BluejayInfraLint.Tests.csproj -c Release
|
||||||
|
```
|
||||||
|
|
||||||
|
That test project sweeps `bluejay-infra/apps/**` plus the canonical sibling `FlowerCore.*\\k8s` manifests that share the same workspace. Matching `conftest.dev` policy files live under `tests/bluejay-infra-lint/conftest.dev/` for environments that also have `conftest` or `opa`.
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
- Cert-manager recovery playbook: `FlowerCore.Notes/memory/project_cert_manager_recovery_2026_04_22.md`
|
- Cert-manager recovery playbook: `FlowerCore.Notes/memory/project_cert_manager_recovery_2026_04_22.md`
|
||||||
|
|||||||
@@ -2,14 +2,15 @@
|
|||||||
# Agent Zero AI Stack — NUC Deployment (RKE2 Bare-Metal)
|
# Agent Zero AI Stack — NUC Deployment (RKE2 Bare-Metal)
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Deploys: AgentZero (agent UI) on RKE2 cluster with Blue Jay profile
|
# Deploys: AgentZero (agent UI) on RKE2 cluster with Blue Jay profile
|
||||||
# Ollama: workstation-first via BLUEJAY-WS (10.0.56.20:11434) with edge1 Pi 5
|
# Ollama: edge1 Pi 5 + AI HAT+ ONLY (10.0.57.17:11434).
|
||||||
# fallback (10.0.57.17:11434)
|
# Workstation Ollama (BLUEJAY-WS) is intentionally NOT in the upstream —
|
||||||
|
# the workstation is private dev hardware, not a cluster dependency.
|
||||||
# Target: RKE2 bare-metal cluster, namespace: agent-zero
|
# Target: RKE2 bare-metal cluster, namespace: agent-zero
|
||||||
# Profile: Blue Jay (21 tools, 3 prompts, 4 extensions, theme)
|
# Profile: Blue Jay (21 tools, 3 prompts, 4 extensions, theme)
|
||||||
#
|
#
|
||||||
# Differences from LOCAL (WSL K3s):
|
# Differences from LOCAL (WSL K3s):
|
||||||
# - Uses Longhorn StorageClass (not local-path)
|
# - Uses Longhorn StorageClass (not local-path)
|
||||||
# - Prefers workstation Ollama on the R9700, falls back to edge1 Pi 5
|
# - Cluster-only Ollama path (edge1) — keeps workstation private
|
||||||
# - NO Anthropic API key (free/local models only)
|
# - NO Anthropic API key (free/local models only)
|
||||||
# - NO Piper TTS or Kiwix (edge1 handles TTS, no Wikipedia needed)
|
# - NO Piper TTS or Kiwix (edge1 handles TTS, no Wikipedia needed)
|
||||||
# - NO hostPath volumes — profile/tools/extensions loaded via ConfigMaps
|
# - NO hostPath volumes — profile/tools/extensions loaded via ConfigMaps
|
||||||
@@ -91,14 +92,17 @@ subjects:
|
|||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Agent Zero — AI Agent Web UI (NUC Edition, Blue Jay Profile)
|
# Agent Zero — AI Agent Web UI (NUC Edition, Blue Jay Profile)
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Connects to a local proxy that routes to workstation Ollama first and edge1 second
|
# Connects directly to fc-llm-bridge for chat + internal util/embed + browser.
|
||||||
# Blue Jay profile with 21 tools, 3 prompts, 4 extensions
|
# Agent Zero's internal util/embed slots stay on the bridge's OpenAI-compatible
|
||||||
|
# /v1 surface, while browser + corpus-search use the Ollama-compatible /api/*
|
||||||
|
# surface through OLLAMA_HOST.
|
||||||
|
# Blue Jay profile with 21 tools, 3 prompts, 4 extensions.
|
||||||
|
|
||||||
---
|
---
|
||||||
# FC LLM Bridge API key for Agent Zero (ADR-088 chat_model routing).
|
# FC LLM Bridge API key for Agent Zero (ADR-088 chat/util/embed/browser routing).
|
||||||
# Syncs from 1Password item "FC LLM Bridge API Keys" (field: agent-zero-k8s).
|
# Syncs from 1Password item "FC LLM Bridge API Keys" (field: agent-zero-k8s).
|
||||||
# Consumed by the chat_model only; util / embedding / browser stay on local
|
# Consumed by chat, internal util/embed, browser, and corpus-search requests
|
||||||
# Ollama via the 127.0.0.1 sidecar proxy.
|
# that traverse fc-llm-bridge.
|
||||||
apiVersion: onepassword.com/v1
|
apiVersion: onepassword.com/v1
|
||||||
kind: OnePasswordItem
|
kind: OnePasswordItem
|
||||||
metadata:
|
metadata:
|
||||||
@@ -107,6 +111,34 @@ metadata:
|
|||||||
spec:
|
spec:
|
||||||
itemPath: "vaults/IAmWorkin/items/FC LLM Bridge API Keys"
|
itemPath: "vaults/IAmWorkin/items/FC LLM Bridge API Keys"
|
||||||
|
|
||||||
|
---
|
||||||
|
# Print.Web API key for Agent Zero's print_web.py Python tool.
|
||||||
|
# Syncs from 1Password item "Print.Web API Keys" (password field = API key).
|
||||||
|
# The print_web.py tool reads PRINT_WEB_API_KEY env var for all HTTP requests
|
||||||
|
# to the thermal print service (GET /api/mcp/tools, POST /api/print/*, etc.).
|
||||||
|
# Note: Print.Web uses the legacy REST MCP shape (/api/mcp/tools/*), not the
|
||||||
|
# streamable-http MCP protocol. The print_web Python tool bridges this gap
|
||||||
|
# and is already present in bluejay-tools ConfigMaps.
|
||||||
|
apiVersion: onepassword.com/v1
|
||||||
|
kind: OnePasswordItem
|
||||||
|
metadata:
|
||||||
|
name: print-web-api-keys
|
||||||
|
namespace: agent-zero
|
||||||
|
spec:
|
||||||
|
itemPath: "vaults/IAmWorkin/items/Print.Web API Keys"
|
||||||
|
|
||||||
|
---
|
||||||
|
# Knowledge MCP bearer token for the direct Agent Zero -> Knowledge.Web path.
|
||||||
|
# The 1Password item currently stores the raw token in its concealed PASSWORD
|
||||||
|
# field, which the operator syncs to Secret key `password`.
|
||||||
|
apiVersion: onepassword.com/v1
|
||||||
|
kind: OnePasswordItem
|
||||||
|
metadata:
|
||||||
|
name: knowledge-mcp-tokens
|
||||||
|
namespace: agent-zero
|
||||||
|
spec:
|
||||||
|
itemPath: "vaults/IAmWorkin/items/FlowerCore Knowledge MCP Tokens"
|
||||||
|
|
||||||
---
|
---
|
||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
@@ -118,7 +150,7 @@ metadata:
|
|||||||
annotations:
|
annotations:
|
||||||
agent-zero/deployment: "nuc"
|
agent-zero/deployment: "nuc"
|
||||||
agent-zero/profile: "bluejay"
|
agent-zero/profile: "bluejay"
|
||||||
agent-zero/ollama: "BLUEJAY-WS primary (10.0.56.20:11434), edge1 fallback (10.0.57.17:11434)"
|
agent-zero/ollama: "fc-llm-bridge fronts edge1 Pi 5 + AI HAT+ Ollama for cluster browser/corpus-search traffic; internal chat/util/embed route through the bridge's authenticated OpenAI surface"
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
selector:
|
selector:
|
||||||
@@ -133,19 +165,18 @@ spec:
|
|||||||
spec:
|
spec:
|
||||||
serviceAccountName: agent-zero
|
serviceAccountName: agent-zero
|
||||||
initContainers:
|
initContainers:
|
||||||
# Wait for either workstation or edge1 Ollama to be reachable before starting Agent Zero.
|
# Wait for fc-llm-bridge to be reachable before starting Agent Zero.
|
||||||
- name: wait-for-ollama
|
- name: wait-for-llm-bridge
|
||||||
image: busybox:1.37
|
image: busybox:1.37
|
||||||
command: ["sh", "-c"]
|
command: ["sh", "-c"]
|
||||||
args:
|
args:
|
||||||
- |
|
- |
|
||||||
echo "Waiting for Ollama at BLUEJAY-WS or edge1..."
|
echo "Waiting for fc-llm-bridge..."
|
||||||
until wget -qO- --timeout=2 http://10.0.56.20:11434/api/tags >/dev/null 2>&1 || \
|
until wget -qO- --timeout=2 http://fc-llm-bridge.fc-llm-bridge.svc:8080/healthz >/dev/null 2>&1; do
|
||||||
wget -qO- --timeout=2 http://10.0.57.17:11434/api/tags >/dev/null 2>&1; do
|
echo "fc-llm-bridge not ready yet, retrying in 5s..."
|
||||||
echo "No Ollama endpoint ready yet, retrying in 5s..."
|
|
||||||
sleep 5
|
sleep 5
|
||||||
done
|
done
|
||||||
echo "At least one Ollama endpoint is reachable."
|
echo "fc-llm-bridge is reachable."
|
||||||
# Assemble the Blue Jay profile directory structure from ConfigMaps.
|
# Assemble the Blue Jay profile directory structure from ConfigMaps.
|
||||||
# ConfigMaps can't create nested dirs, so we copy into the workspace PVC.
|
# ConfigMaps can't create nested dirs, so we copy into the workspace PVC.
|
||||||
- name: setup-bluejay
|
- name: setup-bluejay
|
||||||
@@ -192,50 +223,6 @@ spec:
|
|||||||
- name: bluejay-theme
|
- name: bluejay-theme
|
||||||
mountPath: /tmp/bluejay-theme
|
mountPath: /tmp/bluejay-theme
|
||||||
containers:
|
containers:
|
||||||
- name: ollama-proxy
|
|
||||||
image: nginx:1.27-alpine
|
|
||||||
command: ["/bin/sh", "-c"]
|
|
||||||
args:
|
|
||||||
- |
|
|
||||||
cat > /etc/nginx/nginx.conf <<'NGINX'
|
|
||||||
worker_processes 1;
|
|
||||||
events { worker_connections 1024; }
|
|
||||||
http {
|
|
||||||
upstream ollama_upstream {
|
|
||||||
server 10.0.56.20:11434 max_fails=2 fail_timeout=10s;
|
|
||||||
server 10.0.57.17:11434 backup;
|
|
||||||
keepalive 16;
|
|
||||||
}
|
|
||||||
server {
|
|
||||||
listen 11434;
|
|
||||||
location / {
|
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Connection "";
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_connect_timeout 5s;
|
|
||||||
proxy_read_timeout 600s;
|
|
||||||
proxy_send_timeout 600s;
|
|
||||||
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
|
|
||||||
proxy_pass http://ollama_upstream;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
NGINX
|
|
||||||
exec nginx -g 'daemon off;'
|
|
||||||
ports:
|
|
||||||
- containerPort: 11434
|
|
||||||
readinessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /api/tags
|
|
||||||
port: 11434
|
|
||||||
initialDelaySeconds: 5
|
|
||||||
periodSeconds: 15
|
|
||||||
livenessProbe:
|
|
||||||
httpGet:
|
|
||||||
path: /api/tags
|
|
||||||
port: 11434
|
|
||||||
initialDelaySeconds: 10
|
|
||||||
periodSeconds: 30
|
|
||||||
- name: agent-zero
|
- name: agent-zero
|
||||||
image: agent0ai/agent-zero:latest
|
image: agent0ai/agent-zero:latest
|
||||||
command: ["/bin/bash", "-c"]
|
command: ["/bin/bash", "-c"]
|
||||||
@@ -256,23 +243,41 @@ spec:
|
|||||||
# chat_model: FlowerCore LLM Bridge (ADR-088) — OpenAI-compat,
|
# chat_model: FlowerCore LLM Bridge (ADR-088) — OpenAI-compat,
|
||||||
# spend-tracked, tier-aliased (fc:balanced → Claude Sonnet).
|
# spend-tracked, tier-aliased (fc:balanced → Claude Sonnet).
|
||||||
# api_key comes from A0_SET_chat_model_api_key env var (overrides
|
# api_key comes from A0_SET_chat_model_api_key env var (overrides
|
||||||
# config.json). util + embedding stay on local 127.0.0.1 Ollama
|
# config.json). Utility + embedding stay on the authenticated
|
||||||
# proxy (workstation primary, edge1 fallback).
|
# OpenAI-compatible /v1 surface; browser and direct tool traffic
|
||||||
|
# use the bridge's Ollama-compatible root via OLLAMA_HOST.
|
||||||
mkdir -p /a0/usr/plugins/_model_config
|
mkdir -p /a0/usr/plugins/_model_config
|
||||||
cat > /a0/usr/plugins/_model_config/config.json << 'MODELCFG'
|
cat > /a0/usr/plugins/_model_config/config.json << 'MODELCFG'
|
||||||
{"allow_chat_override":true,"chat_model":{"provider":"openai","name":"fc:balanced","api_base":"http://fc-llm-bridge.fc-llm-bridge.svc:8080/v1","ctx_length":8192,"ctx_history":0.7,"vision":false,"kwargs":{"temperature":0,"num_ctx":8192}},"utility_model":{"provider":"ollama","name":"qwen2.5:1.5b","api_base":"http://127.0.0.1:11434","ctx_length":8192,"ctx_input":0.7,"kwargs":{"num_ctx":8192}},"embedding_model":{"provider":"ollama","name":"nomic-embed-text","api_base":"http://127.0.0.1:11434","kwargs":{}}}
|
{"allow_chat_override":true,"chat_model":{"provider":"openai","name":"fc:balanced","api_base":"http://fc-llm-bridge.fc-llm-bridge.svc:8080/v1","ctx_length":8192,"ctx_history":0.7,"vision":false,"kwargs":{"temperature":0,"num_ctx":8192}},"utility_model":{"provider":"openai","name":"fc:cheap","api_base":"http://fc-llm-bridge.fc-llm-bridge.svc:8080/v1","ctx_length":8192,"ctx_input":0.7,"kwargs":{"num_ctx":8192}},"embedding_model":{"provider":"openai","name":"openai/fc:embedding","api_base":"http://fc-llm-bridge.fc-llm-bridge.svc:8080/v1","kwargs":{}}}
|
||||||
MODELCFG
|
MODELCFG
|
||||||
# Strip heredoc indentation
|
# Strip heredoc indentation
|
||||||
sed -i 's/^ //' /a0/usr/plugins/_model_config/config.json
|
sed -i 's/^ //' /a0/usr/plugins/_model_config/config.json
|
||||||
# Phase 0 Chat MCP pilot: Agent Zero does not interpolate env vars
|
# Phase 0 Chat MCP pilot: Agent Zero does not interpolate env vars
|
||||||
# inside A0_SET_mcp_servers JSON, so build the final JSON here from
|
# inside A0_SET_mcp_servers JSON, so build the final JSON here from
|
||||||
# the secret-backed CHAT_MCP_API_KEY env var before initialize.sh.
|
# the secret-backed env vars before initialize.sh. Keep the local
|
||||||
# Use the in-cluster Chat service URL rather than the public
|
# corpus_search.py tool mounted either way so outage fallback
|
||||||
# Traefik hostname so the pod stays off the private VIP lane that
|
# remains available even when fc_knowledge is not advertised.
|
||||||
# the default egress rule blocks.
|
export KNOWLEDGE_MCP_ENABLED=false
|
||||||
if [ -n "${CHAT_MCP_API_KEY:-}" ]; then
|
if [ -n "${KNOWLEDGE_MCP_BEARER_TOKEN:-}" ]; then
|
||||||
export A0_SET_mcp_servers="{\"mcpServers\":{\"fc-chat\":{\"type\":\"streamable-http\",\"url\":\"http://chat-web.fc-chat.svc/mcp\",\"headers\":{\"X-Api-Key\":\"${CHAT_MCP_API_KEY}\"}}}}"
|
if curl -sf --connect-timeout 3 "${KNOWLEDGE_MCP_HEALTH_URL}" > /dev/null && \
|
||||||
|
curl -sf --connect-timeout 5 \
|
||||||
|
-H "Authorization: Bearer ${KNOWLEDGE_MCP_BEARER_TOKEN}" \
|
||||||
|
-H "Accept: application/json, text/event-stream" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"jsonrpc":"2.0","id":"fc-knowledge-bootstrap","method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"agent-zero-bootstrap","version":"1.0"}}}' \
|
||||||
|
"${KNOWLEDGE_MCP_URL}" > /dev/null; then
|
||||||
|
export KNOWLEDGE_MCP_ENABLED=true
|
||||||
|
echo "fc_knowledge enabled from ${KNOWLEDGE_MCP_URL}."
|
||||||
|
else
|
||||||
|
echo "fc_knowledge unavailable or unauthorized; keeping local corpus_search.py as the fallback path."
|
||||||
fi
|
fi
|
||||||
|
else
|
||||||
|
echo "fc_knowledge token missing; keeping local corpus_search.py as the fallback path."
|
||||||
|
fi
|
||||||
|
|
||||||
|
export A0_SET_mcp_servers="$(
|
||||||
|
python3 -c 'import json, os; servers = {}; chat_key = os.getenv("CHAT_MCP_API_KEY"); knowledge_enabled = os.getenv("KNOWLEDGE_MCP_ENABLED", "false").lower() == "true"; token = os.getenv("KNOWLEDGE_MCP_BEARER_TOKEN", "") if knowledge_enabled else ""; chat_key and servers.setdefault("fc_chat", {"type": "streamable-http", "url": "http://chat-web.fc-chat.svc/mcp", "headers": {"X-Api-Key": chat_key}}); token and servers.setdefault("fc_knowledge", {"type": "streamable-http", "url": os.getenv("KNOWLEDGE_MCP_URL", "http://knowledge-web.knowledge.svc/mcp"), "headers": {"Authorization": f"Bearer {token}"}}); print(json.dumps({"mcpServers": servers}, separators=(",", ":")))'
|
||||||
|
)"
|
||||||
# Run the original entrypoint
|
# Run the original entrypoint
|
||||||
exec /exe/initialize.sh $BRANCH
|
exec /exe/initialize.sh $BRANCH
|
||||||
ports:
|
ports:
|
||||||
@@ -284,8 +289,9 @@ spec:
|
|||||||
# Chat model — routed through FlowerCore LLM Bridge (ADR-088)
|
# Chat model — routed through FlowerCore LLM Bridge (ADR-088)
|
||||||
# so spend is tracked and tier aliases (fc:cheap/fc:balanced/fc:deep)
|
# so spend is tracked and tier aliases (fc:cheap/fc:balanced/fc:deep)
|
||||||
# dispatch to Ollama or Anthropic via a single OpenAI-compat endpoint.
|
# dispatch to Ollama or Anthropic via a single OpenAI-compat endpoint.
|
||||||
# Util / embedding / browser stay on local Ollama via 127.0.0.1 proxy
|
# Internal utility + embedding use the authenticated OpenAI surface,
|
||||||
# for zero-latency, zero-cost small-model traffic.
|
# while browser/corpus-search use the bridge's Ollama-compatible
|
||||||
|
# endpoints so Agent Zero no longer needs a local proxy sidecar.
|
||||||
- name: A0_SET_chat_model_provider
|
- name: A0_SET_chat_model_provider
|
||||||
value: "openai"
|
value: "openai"
|
||||||
- name: A0_SET_chat_model_name
|
- name: A0_SET_chat_model_name
|
||||||
@@ -307,35 +313,51 @@ spec:
|
|||||||
secretKeyRef:
|
secretKeyRef:
|
||||||
name: fc-llm-bridge-api-keys
|
name: fc-llm-bridge-api-keys
|
||||||
key: agent-zero-k8s
|
key: agent-zero-k8s
|
||||||
|
- name: FC_LLM_BRIDGE_API_KEY
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: fc-llm-bridge-api-keys
|
||||||
|
key: agent-zero-k8s
|
||||||
- name: A0_SET_chat_model_ctx_length
|
- name: A0_SET_chat_model_ctx_length
|
||||||
value: "8192"
|
value: "8192"
|
||||||
- name: A0_SET_chat_model_kwargs
|
- name: A0_SET_chat_model_kwargs
|
||||||
value: '{"temperature": 0, "num_ctx": 8192}'
|
value: '{"temperature": 0, "num_ctx": 8192}'
|
||||||
# Utility model — fast small helper tier through the same proxy
|
# Utility model — fast small helper tier through the OpenAI surface
|
||||||
- name: A0_SET_util_model_provider
|
- name: A0_SET_util_model_provider
|
||||||
value: "ollama"
|
value: "openai"
|
||||||
- name: A0_SET_util_model_name
|
- name: A0_SET_util_model_name
|
||||||
value: "qwen2.5:1.5b"
|
value: "fc:cheap"
|
||||||
- name: A0_SET_util_model_api_base
|
- name: A0_SET_util_model_api_base
|
||||||
value: "http://127.0.0.1:11434"
|
value: "http://fc-llm-bridge.fc-llm-bridge.svc:8080/v1"
|
||||||
- name: A0_SET_util_model_kwargs
|
- name: A0_SET_util_model_kwargs
|
||||||
value: '{"num_ctx": 2048}'
|
value: '{"num_ctx": 2048}'
|
||||||
# Embedding model — nomic through the same proxy
|
# Embedding model — authenticated bridge alias to nomic-embed-text.
|
||||||
|
# LiteLLM's embedding() path needs an explicit provider prefix here
|
||||||
|
# even though the chat slot can use bare fc:* aliases.
|
||||||
- name: A0_SET_embed_model_provider
|
- name: A0_SET_embed_model_provider
|
||||||
value: "ollama"
|
value: "openai"
|
||||||
- name: A0_SET_embed_model_name
|
- name: A0_SET_embed_model_name
|
||||||
value: "nomic-embed-text"
|
value: "openai/fc:embedding"
|
||||||
- name: A0_SET_embed_model_api_base
|
- name: A0_SET_embed_model_api_base
|
||||||
value: "http://127.0.0.1:11434"
|
value: "http://fc-llm-bridge.fc-llm-bridge.svc:8080/v1"
|
||||||
# Browser model — small Gemma candidate through the same proxy
|
# Browser model — small Gemma candidate through the same proxy
|
||||||
- name: A0_SET_browser_model_provider
|
- name: A0_SET_browser_model_provider
|
||||||
value: "ollama"
|
value: "ollama"
|
||||||
- name: A0_SET_browser_model_name
|
- name: A0_SET_browser_model_name
|
||||||
value: "gemma3:4b"
|
value: "gemma3:4b"
|
||||||
- name: A0_SET_browser_model_api_base
|
- name: A0_SET_browser_model_api_base
|
||||||
value: "http://127.0.0.1:11434"
|
value: "http://fc-llm-bridge.fc-llm-bridge.svc:8080"
|
||||||
|
- name: A0_SET_browser_model_api_key
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: fc-llm-bridge-api-keys
|
||||||
|
key: agent-zero-k8s
|
||||||
- name: A0_SET_browser_model_vision
|
- name: A0_SET_browser_model_vision
|
||||||
value: "true"
|
value: "true"
|
||||||
|
- name: OLLAMA_HOST
|
||||||
|
value: "http://fc-llm-bridge.fc-llm-bridge.svc:8080"
|
||||||
|
- name: FLOWERCORE_AGENTZERO_OLLAMA_URL
|
||||||
|
value: "http://fc-llm-bridge.fc-llm-bridge.svc:8080"
|
||||||
# Agent profile — Blue Jay personality, tools, and system prompt
|
# Agent profile — Blue Jay personality, tools, and system prompt
|
||||||
- name: A0_SET_agent_profile
|
- name: A0_SET_agent_profile
|
||||||
value: "bluejay"
|
value: "bluejay"
|
||||||
@@ -358,9 +380,38 @@ spec:
|
|||||||
name: chat-mcp-api-key
|
name: chat-mcp-api-key
|
||||||
key: api-key
|
key: api-key
|
||||||
optional: true
|
optional: true
|
||||||
# Print.Web — Thermal printer service on edge2
|
# FlowerCore.Knowledge MCP Phase 1 — direct Agent Zero client path.
|
||||||
|
# Probe /healthz first, then try an authenticated initialize call.
|
||||||
|
# If either fails, Agent Zero boots without fc_knowledge and keeps
|
||||||
|
# the local corpus_search.py tool as the outage-safe path.
|
||||||
|
- name: KNOWLEDGE_MCP_URL
|
||||||
|
value: "http://knowledge-web.knowledge.svc/mcp"
|
||||||
|
- name: KNOWLEDGE_MCP_HEALTH_URL
|
||||||
|
value: "http://knowledge-web.knowledge.svc/healthz"
|
||||||
|
- name: KNOWLEDGE_MCP_BEARER_TOKEN
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: knowledge-mcp-tokens
|
||||||
|
key: password
|
||||||
|
# Print.Web — Thermal printer service on edge2.
|
||||||
|
# PRINT_WEB_URL: internal HTTP (bypasses Traefik TLS — print_web.py
|
||||||
|
# runs in-cluster and can reach edge2 directly on the PROD VLAN).
|
||||||
|
# PRINT_WEB_API_KEY: from 1Password "Print.Web API Keys" password field,
|
||||||
|
# synced by the print-web-api-keys OnePasswordItem CRD above.
|
||||||
|
# The print_web.py Python tool reads both env vars for all HTTP calls.
|
||||||
- name: PRINT_WEB_URL
|
- name: PRINT_WEB_URL
|
||||||
value: "http://10.0.57.16:5200"
|
value: "http://10.0.57.16:5200"
|
||||||
|
- name: PRINT_WEB_API_KEY
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: print-web-api-keys
|
||||||
|
key: password
|
||||||
|
# Intranet search — use in-cluster HTTP (no step-ca TLS needed)
|
||||||
|
# corpus_search.py reads FLOWERCORE_FLEET_VECTOR_DIR but that mount is not
|
||||||
|
# on the cluster yet (BLUEJAY-WS only). The tool gracefully returns a
|
||||||
|
# "no DB found" message with rebuild instructions rather than crashing.
|
||||||
|
- name: FLOWERCORE_INTRANET_URL
|
||||||
|
value: "http://intranet-web.intranet.svc:5300"
|
||||||
# Kubernetes
|
# Kubernetes
|
||||||
- name: KUBERNETES_SERVICE_HOST
|
- name: KUBERNETES_SERVICE_HOST
|
||||||
value: "kubernetes.default.svc"
|
value: "kubernetes.default.svc"
|
||||||
@@ -395,7 +446,7 @@ spec:
|
|||||||
command:
|
command:
|
||||||
- /bin/bash
|
- /bin/bash
|
||||||
- -c
|
- -c
|
||||||
- "curl -sf http://localhost:80/ > /dev/null && curl -sf --connect-timeout 3 http://127.0.0.1:11434/api/tags > /dev/null"
|
- "curl -sf http://localhost:80/ > /dev/null && curl -sf --connect-timeout 3 http://fc-llm-bridge.fc-llm-bridge.svc:8080/healthz > /dev/null"
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
failureThreshold: 2
|
failureThreshold: 2
|
||||||
resources:
|
resources:
|
||||||
@@ -533,18 +584,6 @@ spec:
|
|||||||
protocol: UDP
|
protocol: UDP
|
||||||
- port: 53
|
- port: 53
|
||||||
protocol: TCP
|
protocol: TCP
|
||||||
# Ollama on BLUEJAY-WS
|
|
||||||
- to:
|
|
||||||
- ipBlock:
|
|
||||||
cidr: 10.0.56.20/32
|
|
||||||
ports:
|
|
||||||
- port: 11434
|
|
||||||
# Ollama on edge1 fallback
|
|
||||||
- to:
|
|
||||||
- ipBlock:
|
|
||||||
cidr: 10.0.57.17/32
|
|
||||||
ports:
|
|
||||||
- port: 11434
|
|
||||||
# Print.Web on edge2
|
# Print.Web on edge2
|
||||||
- to:
|
- to:
|
||||||
- ipBlock:
|
- ipBlock:
|
||||||
@@ -578,6 +617,26 @@ spec:
|
|||||||
protocol: TCP
|
protocol: TCP
|
||||||
- port: 8080
|
- port: 8080
|
||||||
protocol: TCP
|
protocol: TCP
|
||||||
|
# FlowerCore.Knowledge MCP (Phase 1) — in-cluster direct route with
|
||||||
|
# anonymous /healthz probe plus authenticated /mcp initialize/tool calls.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: knowledge
|
||||||
|
ports:
|
||||||
|
- port: 80
|
||||||
|
protocol: TCP
|
||||||
|
- port: 8080
|
||||||
|
protocol: TCP
|
||||||
|
# Intranet search API — use in-cluster svc so traffic stays inside
|
||||||
|
# the cluster and is not blocked by the private-range egress denylist.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: intranet
|
||||||
|
ports:
|
||||||
|
- port: 5300
|
||||||
|
protocol: TCP
|
||||||
# Allow internet (for kubectl image pull, etc)
|
# Allow internet (for kubectl image pull, etc)
|
||||||
- to:
|
- to:
|
||||||
- ipBlock:
|
- ipBlock:
|
||||||
|
|||||||
@@ -7209,6 +7209,9 @@ data:
|
|||||||
"keep_alive": keep_alive,
|
"keep_alive": keep_alive,
|
||||||
"stream": False,
|
"stream": False,
|
||||||
})
|
})
|
||||||
|
curl_headers = ["-H", "Content-Type: application/json"]
|
||||||
|
if os.environ.get("FC_LLM_BRIDGE_API_KEY"):
|
||||||
|
curl_headers.extend(["-H", f"X-Api-Key: {os.environ['FC_LLM_BRIDGE_API_KEY']}"])
|
||||||
|
|
||||||
try:
|
try:
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
@@ -7216,7 +7219,7 @@ data:
|
|||||||
"curl", "-s", "--max-time", "120",
|
"curl", "-s", "--max-time", "120",
|
||||||
"-X", "POST",
|
"-X", "POST",
|
||||||
f"{api_base}/api/generate",
|
f"{api_base}/api/generate",
|
||||||
"-H", "Content-Type: application/json",
|
*curl_headers,
|
||||||
"-d", payload,
|
"-d", payload,
|
||||||
],
|
],
|
||||||
capture_output=True,
|
capture_output=True,
|
||||||
@@ -13150,6 +13153,451 @@ data:
|
|||||||
- PowerShell 5.1 compatibility is assumed (no PowerShell 7+ features).
|
- PowerShell 5.1 compatibility is assumed (no PowerShell 7+ features).
|
||||||
- All commands run with `-NoProfile -NonInteractive` flags for clean execution.
|
- All commands run with `-NoProfile -NonInteractive` flags for clean execution.
|
||||||
"""
|
"""
|
||||||
|
corpus_search.py: |
|
||||||
|
# FlowerCore Fleet Corpus Vector Search Tool
|
||||||
|
#
|
||||||
|
# Queries the AiStation-built SqliteVecVectorStore DB at /a0/usr/vectors/fleet.db
|
||||||
|
# (bind-mounted read-only from /var/lib/flowercore/vector-stores/ on the host).
|
||||||
|
# Embeds the query through Ollama's nomic-embed-text model, computes cosine
|
||||||
|
# similarity against every stored chunk in pure Python (no numpy — not present
|
||||||
|
# in the container), and returns the top-K nearest neighbors with source metadata.
|
||||||
|
#
|
||||||
|
# This is the offline-friendly counterpart to `intranet_search` (which hits the
|
||||||
|
# Intranet's live REST API). Use it for Bible/Greek/Hebrew/Strong's lookups and
|
||||||
|
# anywhere the workstation has a newer DB than the Intranet one. The store is
|
||||||
|
# refreshed by `aistation-indexer build <edition>` — see the FlowerCore.Knowledge
|
||||||
|
# ADR at docs/ai-agents/flowercore-knowledge-service-plan.md.
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
import urllib.request
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from python.helpers.tool import Tool, Response
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_VECTORS_DIR = os.environ.get(
|
||||||
|
"FLOWERCORE_FLEET_VECTOR_DIR",
|
||||||
|
"/a0/usr/vectors",
|
||||||
|
)
|
||||||
|
# When the caller doesn't pick an explicit DB, prefer the biggest fleet tier
|
||||||
|
# present on disk. Workstation → pi-edge → bmo-bot.
|
||||||
|
PREFERRED_DB_ORDER = [
|
||||||
|
os.environ.get("FLOWERCORE_FLEET_VECTOR_DB", ""),
|
||||||
|
"fleet-workstation-full.db",
|
||||||
|
"fleet-pi-edge.db",
|
||||||
|
"fleet-bmo-bot.db",
|
||||||
|
]
|
||||||
|
OLLAMA_BASE_URL = os.environ.get(
|
||||||
|
"FLOWERCORE_AGENTZERO_OLLAMA_URL",
|
||||||
|
"http://host.containers.internal:11434",
|
||||||
|
)
|
||||||
|
BRIDGE_API_KEY = os.environ.get("FC_LLM_BRIDGE_API_KEY", "").strip()
|
||||||
|
EMBEDDING_MODEL = os.environ.get(
|
||||||
|
"FLOWERCORE_FLEET_EMBEDDING_MODEL",
|
||||||
|
"nomic-embed-text",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class CorpusSearch(Tool):
|
||||||
|
async def execute(self, **kwargs) -> Response:
|
||||||
|
"""
|
||||||
|
Semantic search over the FlowerCore fleet corpus (Bible texts, lexicons,
|
||||||
|
dictionaries, morphology) pre-indexed by aistation-indexer.
|
||||||
|
|
||||||
|
Args (via self.args):
|
||||||
|
query (str): Search query text. Required unless action=stats.
|
||||||
|
limit (int): Max results. Default 8.
|
||||||
|
index (str): Optional index name filter ("bible-texts", "lexicons",
|
||||||
|
"dictionaries", "morphology"). Default: all indexes.
|
||||||
|
repo (str): Optional repo filter (e.g. "world-english-bible").
|
||||||
|
db (str): Override DB path OR file name inside FLOWERCORE_FLEET_VECTOR_DIR
|
||||||
|
(defaults to /a0/usr/vectors). If omitted, the largest
|
||||||
|
fleet tier present on disk is picked automatically.
|
||||||
|
action (str): Optional. "stats" returns an inventory of all fleet DBs
|
||||||
|
visible to the tool (names, sizes, index counts, chunk
|
||||||
|
counts, last-built timestamps). No embedding call.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Response with ranked chunks (score, source, text preview) OR
|
||||||
|
(when action=stats) a markdown inventory of available fleet DBs.
|
||||||
|
"""
|
||||||
|
query = (self.args.get("query") or "").strip()
|
||||||
|
limit = int(self.args.get("limit") or 8)
|
||||||
|
index_filter = (self.args.get("index") or "").strip()
|
||||||
|
repo_filter = (self.args.get("repo") or "").strip()
|
||||||
|
db_override = (self.args.get("db") or "").strip()
|
||||||
|
action = (self.args.get("action") or "").strip().lower()
|
||||||
|
|
||||||
|
if action == "stats":
|
||||||
|
return Response(message=_render_stats(), break_loop=False)
|
||||||
|
|
||||||
|
if not query:
|
||||||
|
return Response(
|
||||||
|
message=(
|
||||||
|
"Error: 'query' is required unless action=stats.\n"
|
||||||
|
"Example: query=\"what does Genesis 1:1 say\" limit=5\n"
|
||||||
|
"Inventory: action=stats"
|
||||||
|
),
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
db = _resolve_db(db_override)
|
||||||
|
if db is None:
|
||||||
|
return Response(
|
||||||
|
message=(
|
||||||
|
f"Error: no fleet vector DB found under {DEFAULT_VECTORS_DIR}.\n"
|
||||||
|
"Host side: run `aistation-indexer build fleet-workstation-full`\n"
|
||||||
|
"(or `fleet-pi-edge`/`fleet-bmo-bot`) to produce\n"
|
||||||
|
"`/var/lib/flowercore/vector-stores/<slug>.db`, then confirm the\n"
|
||||||
|
"Podman unit mounts that directory into `/a0/usr/vectors:ro`."
|
||||||
|
),
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
query_vec = _embed(query)
|
||||||
|
except Exception as e:
|
||||||
|
return Response(
|
||||||
|
message=f"Error: failed to embed query via Ollama at {OLLAMA_BASE_URL}: {e}",
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
hits = _search(db, query_vec, index_filter, repo_filter, limit)
|
||||||
|
except Exception as e:
|
||||||
|
return Response(
|
||||||
|
message=f"Error: corpus search failed: {e}",
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not hits:
|
||||||
|
return Response(
|
||||||
|
message=(
|
||||||
|
f"No matches for '{query}' in {db.name}.\n"
|
||||||
|
f"Indexes available: " + _list_indexes_summary(db)
|
||||||
|
),
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
lines = [f"**Corpus search: `{query}`** (top {len(hits)} of {limit} requested, DB={db.name})", ""]
|
||||||
|
for rank, h in enumerate(hits, 1):
|
||||||
|
passage = h.get("passage") or ""
|
||||||
|
lang = h.get("language") or ""
|
||||||
|
meta_bits = [x for x in (h["index"], h["repo"], passage, lang) if x]
|
||||||
|
meta = " · ".join(meta_bits)
|
||||||
|
preview = h["text"]
|
||||||
|
if len(preview) > 320:
|
||||||
|
preview = preview[:320].rstrip() + "…"
|
||||||
|
lines.append(f"{rank}. **{h['score']:.3f}** {meta}")
|
||||||
|
lines.append(f" `{h['source']}`")
|
||||||
|
lines.append(f" {preview}")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
return Response(message="\n".join(lines).rstrip() + "\n", break_loop=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_db(override: str) -> "Path | None":
|
||||||
|
"""Pick a fleet DB by explicit path, explicit filename, or preferred order."""
|
||||||
|
vectors_dir = Path(DEFAULT_VECTORS_DIR)
|
||||||
|
if override:
|
||||||
|
# Absolute or relative path that points at a real file wins outright.
|
||||||
|
p = Path(override)
|
||||||
|
if p.is_absolute() and p.exists():
|
||||||
|
return p
|
||||||
|
# Otherwise treat it as a filename within the vectors dir.
|
||||||
|
candidate = vectors_dir / override
|
||||||
|
if candidate.exists():
|
||||||
|
return candidate
|
||||||
|
return None
|
||||||
|
|
||||||
|
for name in PREFERRED_DB_ORDER:
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
p = Path(name) if Path(name).is_absolute() else vectors_dir / name
|
||||||
|
if p.exists():
|
||||||
|
return p
|
||||||
|
|
||||||
|
# Fallback: any *.db in the dir, largest first.
|
||||||
|
if vectors_dir.is_dir():
|
||||||
|
candidates = sorted(vectors_dir.glob("*.db"), key=lambda p: p.stat().st_size, reverse=True)
|
||||||
|
if candidates:
|
||||||
|
return candidates[0]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _embed(text: str) -> list:
|
||||||
|
"""Embed a query via Ollama's /api/embeddings. Single-vector response."""
|
||||||
|
body = json.dumps({"model": EMBEDDING_MODEL, "prompt": text}).encode("utf-8")
|
||||||
|
headers = {"Content-Type": "application/json"}
|
||||||
|
if BRIDGE_API_KEY:
|
||||||
|
headers["X-Api-Key"] = BRIDGE_API_KEY
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{OLLAMA_BASE_URL.rstrip('/')}/api/embeddings",
|
||||||
|
data=body,
|
||||||
|
headers=headers,
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=60) as resp:
|
||||||
|
data = json.loads(resp.read().decode("utf-8"))
|
||||||
|
vec = data.get("embedding")
|
||||||
|
if not isinstance(vec, list) or not vec:
|
||||||
|
raise RuntimeError(f"Ollama returned no embedding: {data}")
|
||||||
|
return [float(x) for x in vec]
|
||||||
|
|
||||||
|
|
||||||
|
def _cosine(a: list, b: list) -> float:
|
||||||
|
"""Cosine similarity in pure Python — no numpy in the A0 container."""
|
||||||
|
# zip() stops at the shorter — AiStation DB guarantees same dim per index.
|
||||||
|
dot = 0.0
|
||||||
|
na = 0.0
|
||||||
|
nb = 0.0
|
||||||
|
for x, y in zip(a, b):
|
||||||
|
dot += x * y
|
||||||
|
na += x * x
|
||||||
|
nb += y * y
|
||||||
|
if na == 0.0 or nb == 0.0:
|
||||||
|
return 0.0
|
||||||
|
return dot / (math.sqrt(na) * math.sqrt(nb))
|
||||||
|
|
||||||
|
|
||||||
|
def _search(db_path: Path, query_vec: list, index_filter: str, repo_filter: str, limit: int) -> list:
|
||||||
|
"""Load entries, compute cosine, return top-K.
|
||||||
|
|
||||||
|
SqliteVecVectorStore schema:
|
||||||
|
VectorIndexes(IndexName, Dimensions, UpdatedAtUtc)
|
||||||
|
VectorEntries(IndexName, ChunkId, TextContent, SourceRepo, SourceFile,
|
||||||
|
Book, Chapter, VerseRange, Language, ContentType, License,
|
||||||
|
EstimatedTokens, EmbeddingJson)
|
||||||
|
|
||||||
|
Embeddings are stored as JSON arrays in EmbeddingJson; similarity is computed
|
||||||
|
in Python. For ~100k chunks × 768 dims this takes a couple seconds on a
|
||||||
|
workstation — acceptable for interactive A0 use.
|
||||||
|
"""
|
||||||
|
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
sql = [
|
||||||
|
"SELECT IndexName, ChunkId, TextContent, SourceRepo, SourceFile, ",
|
||||||
|
" Book, Chapter, VerseRange, Language, EmbeddingJson ",
|
||||||
|
"FROM VectorEntries",
|
||||||
|
]
|
||||||
|
where = []
|
||||||
|
params = []
|
||||||
|
if index_filter:
|
||||||
|
where.append("IndexName = ?")
|
||||||
|
params.append(index_filter)
|
||||||
|
if repo_filter:
|
||||||
|
where.append("SourceRepo LIKE ?")
|
||||||
|
params.append(f"%{repo_filter}%")
|
||||||
|
if where:
|
||||||
|
sql.append(" WHERE " + " AND ".join(where))
|
||||||
|
sql.append(";")
|
||||||
|
|
||||||
|
cursor = conn.execute("".join(sql), params)
|
||||||
|
|
||||||
|
# Min-heap by (score, ...) would be faster but for interactive use we
|
||||||
|
# just sort at the end — simpler and readable.
|
||||||
|
scored = []
|
||||||
|
for row in cursor:
|
||||||
|
idx, chunk_id, text, repo, source_file, book, chapter, verses, lang, emb_json = row
|
||||||
|
try:
|
||||||
|
vec = json.loads(emb_json)
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
continue
|
||||||
|
score = _cosine(query_vec, vec)
|
||||||
|
passage = None
|
||||||
|
if book and chapter:
|
||||||
|
passage = f"{book} {chapter}"
|
||||||
|
if verses:
|
||||||
|
passage += f":{verses}"
|
||||||
|
scored.append((score, {
|
||||||
|
"index": idx,
|
||||||
|
"chunk_id": chunk_id,
|
||||||
|
"text": text,
|
||||||
|
"repo": repo or "",
|
||||||
|
"source": source_file or "",
|
||||||
|
"passage": passage or "",
|
||||||
|
"language": lang or "",
|
||||||
|
}))
|
||||||
|
scored.sort(key=lambda t: t[0], reverse=True)
|
||||||
|
return [{"score": s, **meta} for s, meta in scored[:limit]]
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _render_stats() -> str:
|
||||||
|
"""Markdown inventory of every *.db in FLOWERCORE_FLEET_VECTOR_DIR."""
|
||||||
|
vectors_dir = Path(DEFAULT_VECTORS_DIR)
|
||||||
|
if not vectors_dir.is_dir():
|
||||||
|
return f"No fleet vector dir mounted at {vectors_dir}. Ask the host operator to build an index with scripts/agent-zero/build-fleet-index.sh."
|
||||||
|
|
||||||
|
dbs = sorted(vectors_dir.glob("*.db"))
|
||||||
|
if not dbs:
|
||||||
|
return f"No fleet DBs present under {vectors_dir}. Run `scripts/agent-zero/build-fleet-index.sh fleet-workstation-full` on the host."
|
||||||
|
|
||||||
|
lines = [f"**Fleet vector DB inventory** ({vectors_dir})", ""]
|
||||||
|
for db in dbs:
|
||||||
|
size_mb = db.stat().st_size / (1024 * 1024)
|
||||||
|
lines.append(f"### `{db.name}` ({size_mb:.1f} MB)")
|
||||||
|
try:
|
||||||
|
conn = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
idx_rows = conn.execute(
|
||||||
|
"SELECT IndexName, Dimensions, UpdatedAtUtc FROM VectorIndexes ORDER BY IndexName;"
|
||||||
|
).fetchall()
|
||||||
|
if not idx_rows:
|
||||||
|
lines.append("- (no indexes registered)")
|
||||||
|
else:
|
||||||
|
counts = dict(conn.execute(
|
||||||
|
"SELECT IndexName, COUNT(*) FROM VectorEntries GROUP BY IndexName;"
|
||||||
|
).fetchall())
|
||||||
|
for name, dim, updated in idx_rows:
|
||||||
|
count = counts.get(name, 0)
|
||||||
|
lines.append(f"- **{name}** — {count:,} chunks × {dim}d (built {updated})")
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
except Exception as e:
|
||||||
|
lines.append(f"- (inspect failed: {e})")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
lines.append(f"**Tool defaults:** embedding model `{EMBEDDING_MODEL}`, Ollama at `{OLLAMA_BASE_URL}`. Pick a DB with `db=<filename>`; filter by `index=<name>`/`repo=<substring>`.")
|
||||||
|
return "\n".join(lines).rstrip() + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def _list_indexes_summary(db_path: Path) -> str:
|
||||||
|
try:
|
||||||
|
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT IndexName, Dimensions, "
|
||||||
|
" (SELECT COUNT(*) FROM VectorEntries WHERE VectorEntries.IndexName = VectorIndexes.IndexName) "
|
||||||
|
"FROM VectorIndexes ORDER BY IndexName;"
|
||||||
|
).fetchall()
|
||||||
|
if not rows:
|
||||||
|
return "(no indexes)"
|
||||||
|
return ", ".join(f"{r[0]}({r[2]}×{r[1]}d)" for r in rows)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
except Exception as e:
|
||||||
|
return f"(couldn't list: {e})"
|
||||||
|
|
||||||
|
intranet_search.py: |
|
||||||
|
# Intranet Vector Search Tool
|
||||||
|
# Queries the Blue Jay Lab Intranet's Shared.Indexing RAG corpus over its
|
||||||
|
# live REST API (https://intranet.iamworkin.lan/search). Returns ranked chunks
|
||||||
|
# with source file paths and scores.
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import ssl
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
from python.helpers.tool import Tool, Response
|
||||||
|
|
||||||
|
|
||||||
|
INTRANET_BASE_URL = os.environ.get(
|
||||||
|
"FLOWERCORE_INTRANET_URL",
|
||||||
|
"https://intranet.iamworkin.lan",
|
||||||
|
)
|
||||||
|
STEPCA_ROOT_CRT = "/a0/usr/ca/stepca-root.crt"
|
||||||
|
|
||||||
|
|
||||||
|
def _ssl_ctx() -> ssl.SSLContext:
|
||||||
|
ctx = ssl.create_default_context()
|
||||||
|
if os.path.exists(STEPCA_ROOT_CRT):
|
||||||
|
ctx.load_verify_locations(cafile=STEPCA_ROOT_CRT)
|
||||||
|
return ctx
|
||||||
|
|
||||||
|
|
||||||
|
class IntranetSearch(Tool):
|
||||||
|
async def execute(self, **kwargs) -> Response:
|
||||||
|
"""
|
||||||
|
Search the Blue Jay Lab intranet corpus (docs, project notes, dashboards).
|
||||||
|
|
||||||
|
Args (via self.args):
|
||||||
|
query (str): Search query. Required.
|
||||||
|
limit (int): Max chunks to return. Default 8.
|
||||||
|
corpus (str): Optional corpus filter (e.g. "notes", "docs").
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Response with ranked chunk text, source path, and score.
|
||||||
|
"""
|
||||||
|
query = self.args.get("query", "").strip()
|
||||||
|
limit = int(self.args.get("limit", 8))
|
||||||
|
corpus = self.args.get("corpus", "").strip()
|
||||||
|
|
||||||
|
if not query:
|
||||||
|
return Response(
|
||||||
|
message="Error: 'query' is required.",
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
params = {"q": query, "topK": str(limit)}
|
||||||
|
if corpus:
|
||||||
|
params["indexName"] = corpus
|
||||||
|
url = f"{INTRANET_BASE_URL}/api/search?{urllib.parse.urlencode(params)}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||||
|
with urllib.request.urlopen(req, timeout=20, context=_ssl_ctx()) as resp:
|
||||||
|
raw = resp.read().decode("utf-8", errors="replace")
|
||||||
|
except Exception as exc:
|
||||||
|
return Response(
|
||||||
|
message=f"Intranet search failed: {exc}\nURL: {url}",
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = json.loads(raw)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return Response(
|
||||||
|
message=f"Intranet returned non-JSON response:\n{raw[:500]}",
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
hits = data if isinstance(data, list) else (
|
||||||
|
data.get("results") or data.get("hits") or data.get("chunks") or []
|
||||||
|
)
|
||||||
|
if not hits:
|
||||||
|
return Response(
|
||||||
|
message=f"No intranet results for query: {query!r}",
|
||||||
|
break_loop=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
lines = [f"# Intranet search: {query} ({len(hits)} hits)\n"]
|
||||||
|
for i, hit in enumerate(hits[:limit], 1):
|
||||||
|
src = (
|
||||||
|
hit.get("sourceFile")
|
||||||
|
or hit.get("source")
|
||||||
|
or hit.get("path")
|
||||||
|
or hit.get("file")
|
||||||
|
or "?"
|
||||||
|
)
|
||||||
|
repo = hit.get("sourceRepo") or ""
|
||||||
|
idx = hit.get("indexName") or ""
|
||||||
|
score = hit.get("score") or hit.get("similarity") or ""
|
||||||
|
text = (
|
||||||
|
hit.get("snippet")
|
||||||
|
or hit.get("text")
|
||||||
|
or hit.get("content")
|
||||||
|
or hit.get("chunk")
|
||||||
|
or ""
|
||||||
|
).strip()
|
||||||
|
if len(text) > 600:
|
||||||
|
text = text[:600] + "..."
|
||||||
|
header = f"## [{i}] {repo}/{src}" if repo else f"## [{i}] {src}"
|
||||||
|
if idx:
|
||||||
|
header += f" ({idx})"
|
||||||
|
if score:
|
||||||
|
header += f" score={score:.3f}" if isinstance(score, float) else f" score={score}"
|
||||||
|
lines.append(header)
|
||||||
|
lines.append(text)
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
return Response(message="\n".join(lines), break_loop=False)
|
||||||
|
|
||||||
kind: ConfigMap
|
kind: ConfigMap
|
||||||
metadata:
|
metadata:
|
||||||
name: bluejay-tools-c
|
name: bluejay-tools-c
|
||||||
|
|||||||
@@ -20,7 +20,19 @@ spec:
|
|||||||
nodeSelector:
|
nodeSelector:
|
||||||
kubernetes.io/hostname: rke2-agent1
|
kubernetes.io/hostname: rke2-agent1
|
||||||
hostNetwork: true
|
hostNetwork: true
|
||||||
dnsPolicy: ClusterFirstWithHostNet
|
# Keep the search list free of iamworkin.lan so CoreDNS's wildcard
|
||||||
|
# template cannot hijack public egress like downloads.asterisk.org.
|
||||||
|
dnsPolicy: None
|
||||||
|
dnsConfig:
|
||||||
|
nameservers:
|
||||||
|
- 10.43.0.10
|
||||||
|
searches:
|
||||||
|
- telephony.svc.cluster.local
|
||||||
|
- svc.cluster.local
|
||||||
|
- cluster.local
|
||||||
|
options:
|
||||||
|
- name: ndots
|
||||||
|
value: "2"
|
||||||
securityContext:
|
securityContext:
|
||||||
fsGroup: 0
|
fsGroup: 0
|
||||||
# CoreDNS in this cluster has an iamworkin.lan wildcard that catches
|
# CoreDNS in this cluster has an iamworkin.lan wildcard that catches
|
||||||
|
|||||||
106
apps/edge2-services/edge2-services.yaml
Normal file
106
apps/edge2-services/edge2-services.yaml
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# edge2 Services — Traefik IngressRoutes for FlowerCore Print.Web on edge2
|
||||||
|
# Proxies print.iamworkin.lan to edge2 (10.0.57.16:5200) via headless Service
|
||||||
|
# + manual Endpoints (same K8s external-proxy pattern as noc-services).
|
||||||
|
#
|
||||||
|
# Print.Web has its own X-Api-Key authentication and exposes anonymous
|
||||||
|
# endpoints for the bookmarklet / Python CLI / cups-notifier flow, so no
|
||||||
|
# Traefik basicAuth middleware is wired here.
|
||||||
|
#
|
||||||
|
# ArgoCD managed - BlueJay Lab
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: edge2-proxy
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: bluejay-infra
|
||||||
|
---
|
||||||
|
# ============================================================
|
||||||
|
# Print.Web - edge2:5200 (FlowerCore.Print.Web on Pi 4)
|
||||||
|
# ============================================================
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: print-web-external
|
||||||
|
namespace: edge2-proxy
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
- port: 5200
|
||||||
|
targetPort: 5200
|
||||||
|
name: http
|
||||||
|
clusterIP: None
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Endpoints
|
||||||
|
metadata:
|
||||||
|
name: print-web-external
|
||||||
|
namespace: edge2-proxy
|
||||||
|
subsets:
|
||||||
|
- addresses:
|
||||||
|
- ip: 10.0.57.16
|
||||||
|
ports:
|
||||||
|
- port: 5200
|
||||||
|
name: http
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: print-web-tls
|
||||||
|
namespace: edge2-proxy
|
||||||
|
spec:
|
||||||
|
secretName: print-web-tls
|
||||||
|
issuerRef:
|
||||||
|
name: step-ca-acme
|
||||||
|
kind: ClusterIssuer
|
||||||
|
dnsNames:
|
||||||
|
- print.iamworkin.lan
|
||||||
|
---
|
||||||
|
apiVersion: traefik.io/v1alpha1
|
||||||
|
kind: IngressRoute
|
||||||
|
metadata:
|
||||||
|
name: print-web
|
||||||
|
namespace: edge2-proxy
|
||||||
|
spec:
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
routes:
|
||||||
|
- kind: Rule
|
||||||
|
match: Host(`print.iamworkin.lan`)
|
||||||
|
services:
|
||||||
|
- name: print-web-external
|
||||||
|
port: 5200
|
||||||
|
tls:
|
||||||
|
secretName: print-web-tls
|
||||||
|
---
|
||||||
|
# NetworkPolicy: allow Traefik ingress, allow egress to edge2 + DNS
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: NetworkPolicy
|
||||||
|
metadata:
|
||||||
|
name: edge2-proxy-netpol
|
||||||
|
namespace: edge2-proxy
|
||||||
|
spec:
|
||||||
|
podSelector: {}
|
||||||
|
policyTypes:
|
||||||
|
- Ingress
|
||||||
|
- Egress
|
||||||
|
ingress:
|
||||||
|
- from:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: traefik-system
|
||||||
|
egress:
|
||||||
|
- to:
|
||||||
|
- ipBlock:
|
||||||
|
cidr: 10.0.57.16/32
|
||||||
|
ports:
|
||||||
|
- port: 5200
|
||||||
|
protocol: TCP
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
ports:
|
||||||
|
- port: 53
|
||||||
|
protocol: UDP
|
||||||
|
- port: 53
|
||||||
|
protocol: TCP
|
||||||
@@ -23,6 +23,14 @@ spec:
|
|||||||
entryPoints:
|
entryPoints:
|
||||||
- websecure
|
- websecure
|
||||||
routes:
|
routes:
|
||||||
|
# Host-level catch-all for desktop.iamworkin.lan. The /guacamole
|
||||||
|
# path-prefix match lives in apps/guacamole/guacamole.yaml as a
|
||||||
|
# separate IngressRoute in the guacamole namespace — the cluster
|
||||||
|
# Traefik disallows cross-namespace service refs, so the PathPrefix
|
||||||
|
# rule can't sit here. Traefik's router matching precedence gives
|
||||||
|
# longer/more-specific rules priority automatically, so as long as
|
||||||
|
# the guacamole IngressRoute exists it takes /guacamole traffic
|
||||||
|
# before this catch-all sees it.
|
||||||
- match: Host(`desktop.iamworkin.lan`)
|
- match: Host(`desktop.iamworkin.lan`)
|
||||||
kind: Rule
|
kind: Rule
|
||||||
services:
|
services:
|
||||||
|
|||||||
@@ -87,6 +87,20 @@ spec:
|
|||||||
prometheus.io/port: "8080"
|
prometheus.io/port: "8080"
|
||||||
prometheus.io/path: "/metrics"
|
prometheus.io/path: "/metrics"
|
||||||
spec:
|
spec:
|
||||||
|
# Use an explicit DNS policy so external FQDNs like api.anthropic.com are
|
||||||
|
# resolved directly instead of being expanded through the cluster search
|
||||||
|
# path that includes iamworkin.lan.
|
||||||
|
dnsPolicy: None
|
||||||
|
dnsConfig:
|
||||||
|
nameservers:
|
||||||
|
- 10.43.0.10
|
||||||
|
searches:
|
||||||
|
- fc-llm-bridge.svc.cluster.local
|
||||||
|
- svc.cluster.local
|
||||||
|
- cluster.local
|
||||||
|
options:
|
||||||
|
- name: ndots
|
||||||
|
value: "2"
|
||||||
securityContext:
|
securityContext:
|
||||||
fsGroup: 1654
|
fsGroup: 1654
|
||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
@@ -97,7 +111,7 @@ spec:
|
|||||||
# dotnet.exe publish -c Release -o deploy/app \
|
# dotnet.exe publish -c Release -o deploy/app \
|
||||||
# src/FlowerCore.LlmBridge.Web/FlowerCore.LlmBridge.Web.csproj
|
# src/FlowerCore.LlmBridge.Web/FlowerCore.LlmBridge.Web.csproj
|
||||||
# podman build -t localhost/fc-llm-bridge:v<tag> -f deploy/Dockerfile.deploy deploy
|
# podman build -t localhost/fc-llm-bridge:v<tag> -f deploy/Dockerfile.deploy deploy
|
||||||
image: localhost/fc-llm-bridge:v202604231520
|
image: localhost/fc-llm-bridge:v202604300022
|
||||||
imagePullPolicy: Never
|
imagePullPolicy: Never
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 8080
|
- containerPort: 8080
|
||||||
@@ -116,6 +130,10 @@ spec:
|
|||||||
value: "default"
|
value: "default"
|
||||||
- name: FlowerCore__LlmBridge__DefaultAppName
|
- name: FlowerCore__LlmBridge__DefaultAppName
|
||||||
value: "agent-zero"
|
value: "agent-zero"
|
||||||
|
- name: FlowerCore__LlmBridge__UtilModel
|
||||||
|
value: "qwen2.5:1.5b"
|
||||||
|
- name: FlowerCore__LlmBridge__EmbedModel
|
||||||
|
value: "nomic-embed-text"
|
||||||
# Per-consumer API keys — from OnePasswordItem fc-llm-bridge-api-keys.
|
# Per-consumer API keys — from OnePasswordItem fc-llm-bridge-api-keys.
|
||||||
# Each field becomes a Secret key of the same name. The key-name
|
# Each field becomes a Secret key of the same name. The key-name
|
||||||
# lands in the auth principal's `fc.app` claim for ledger scoping.
|
# lands in the auth principal's `fc.app` claim for ledger scoping.
|
||||||
@@ -207,17 +225,6 @@ spec:
|
|||||||
port: 8080
|
port: 8080
|
||||||
initialDelaySeconds: 15
|
initialDelaySeconds: 15
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
# Lower ndots so external FQDNs like api.anthropic.com are tried BEFORE
|
|
||||||
# the ndots:5 default expands them through the cluster search path, which
|
|
||||||
# includes iamworkin.lan. CoreDNS has a `template IN A iamworkin.lan`
|
|
||||||
# wildcard that answers `api.anthropic.com.iamworkin.lan` with the
|
|
||||||
# Traefik VIP, which then serves a TRAEFIK-DEFAULT-CERT TLS cert and
|
|
||||||
# breaks egress to the real Anthropic API (memory:
|
|
||||||
# feedback_coredns_ndots_template_collision, generalized to external DNS).
|
|
||||||
dnsConfig:
|
|
||||||
options:
|
|
||||||
- name: ndots
|
|
||||||
value: "2"
|
|
||||||
volumes:
|
volumes:
|
||||||
- name: data
|
- name: data
|
||||||
persistentVolumeClaim:
|
persistentVolumeClaim:
|
||||||
|
|||||||
@@ -69,16 +69,14 @@ spec:
|
|||||||
memory: "512Mi"
|
memory: "512Mi"
|
||||||
cpu: "500m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet:
|
tcpSocket:
|
||||||
path: /health
|
|
||||||
port: 8080
|
port: 8080
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 5
|
||||||
failureThreshold: 3
|
failureThreshold: 3
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet:
|
tcpSocket:
|
||||||
path: /health
|
|
||||||
port: 8080
|
port: 8080
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|||||||
@@ -76,15 +76,13 @@ spec:
|
|||||||
memory: "512Mi"
|
memory: "512Mi"
|
||||||
cpu: "500m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet:
|
tcpSocket:
|
||||||
path: /health
|
|
||||||
port: http
|
port: http
|
||||||
initialDelaySeconds: 30
|
initialDelaySeconds: 30
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 5
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet:
|
tcpSocket:
|
||||||
path: /health
|
|
||||||
port: http
|
port: http
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|||||||
35
apps/fc-ttsreader/biblical-tts/Dockerfile
Normal file
35
apps/fc-ttsreader/biblical-tts/Dockerfile
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
# FlowerCore biblical-tts — eSpeak-NG-backed TTS for Ancient Greek (grc) and
|
||||||
|
# Hebrew (he). Wraps the espeak-ng binary in a small FastAPI app exposing
|
||||||
|
# /tts (returns WAV) and /timings (returns word timings via espeak's
|
||||||
|
# --pho output). Same shape as fc-speech-align so AiStation can talk to
|
||||||
|
# both with one HTTP client pattern.
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
||||||
|
PIP_NO_CACHE_DIR=1
|
||||||
|
|
||||||
|
# espeak-ng has built-in support for grc (Ancient Greek) and he (Hebrew).
|
||||||
|
# libsndfile1 is for the wav post-processing step.
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends \
|
||||||
|
espeak-ng \
|
||||||
|
libsndfile1 \
|
||||||
|
ca-certificates \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY requirements.txt /app/
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
COPY app.py /app/
|
||||||
|
|
||||||
|
RUN useradd --create-home --shell /usr/sbin/nologin --uid 1654 tts
|
||||||
|
USER 1654
|
||||||
|
|
||||||
|
EXPOSE 10402
|
||||||
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
||||||
|
CMD python -c "import urllib.request,sys; urllib.request.urlopen('http://127.0.0.1:10402/health',timeout=3); sys.exit(0)" || exit 1
|
||||||
|
|
||||||
|
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "10402", "--workers", "1"]
|
||||||
211
apps/fc-ttsreader/biblical-tts/app.py
Normal file
211
apps/fc-ttsreader/biblical-tts/app.py
Normal file
@@ -0,0 +1,211 @@
|
|||||||
|
"""FlowerCore biblical-tts — eSpeak-NG wrapper for Ancient Greek + Hebrew.
|
||||||
|
|
||||||
|
Endpoints:
|
||||||
|
|
||||||
|
* POST /tts — body: {"text": "...", "language": "grc|he|el", "voice": "...?", "rate": 175?, "pitch": 50?}
|
||||||
|
returns audio/wav. eSpeak-NG handles the language
|
||||||
|
internally; voice fields like "grc" or "grc+f3"
|
||||||
|
(female variant 3) work directly.
|
||||||
|
* POST /timings — same body shape but returns
|
||||||
|
{"text": "...", "words": [{"text", "startMs", "endMs"}],
|
||||||
|
"durationMs": ...}.
|
||||||
|
Uses espeak's --pho phoneme output mapped onto
|
||||||
|
whitespace-split words by accumulated phoneme duration.
|
||||||
|
Read-along clients pair this with /tts for synced
|
||||||
|
playback.
|
||||||
|
* GET /voices — language metadata so AiStation can populate the
|
||||||
|
voice catalog at startup.
|
||||||
|
* GET /health — fast readiness check.
|
||||||
|
|
||||||
|
Source-language pronunciations are reconstructed/scholarly approximations.
|
||||||
|
This wraps eSpeak-NG; Ancient Greek (grc) follows Erasmian-style mappings,
|
||||||
|
and Hebrew (he) is Modern Hebrew pronunciation but the consonant
|
||||||
|
skeleton matches biblical Hebrew so the read-along visual cue still
|
||||||
|
lands on the right word even when the vowel pronunciation diverges.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import subprocess
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from fastapi import FastAPI, HTTPException
|
||||||
|
from fastapi.responses import JSONResponse, Response
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
LOG = logging.getLogger("biblical_tts")
|
||||||
|
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||||
|
|
||||||
|
app = FastAPI(title="FlowerCore biblical-tts", version="1.0.0")
|
||||||
|
|
||||||
|
# eSpeak-NG language codes we expose. Ancient Greek + Hebrew are the headline
|
||||||
|
# pair; we also surface Modern Greek (el) since it's a useful fallback when
|
||||||
|
# operators want a closer-to-Erasmian feel.
|
||||||
|
LANGUAGES = {
|
||||||
|
"grc": {"label": "Ancient Greek (Erasmian)", "rtl": False, "default_voice": "grc"},
|
||||||
|
"el": {"label": "Modern Greek", "rtl": False, "default_voice": "el"},
|
||||||
|
"he": {"label": "Hebrew (Modern)", "rtl": True, "default_voice": "he"},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class TtsRequest(BaseModel):
|
||||||
|
text: str
|
||||||
|
language: str = "grc"
|
||||||
|
voice: Optional[str] = None
|
||||||
|
rate: int = 175 # words per minute, eSpeak default 175
|
||||||
|
pitch: int = 50 # 0-99
|
||||||
|
volume: int = 100 # 0-200
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_voice(req: TtsRequest) -> str:
|
||||||
|
if req.voice:
|
||||||
|
return req.voice.strip()
|
||||||
|
lang = req.language.lower()
|
||||||
|
return LANGUAGES.get(lang, {}).get("default_voice", lang)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_espeak(args: list[str], stdin_text: bytes) -> bytes:
|
||||||
|
cmd = ["espeak-ng"] + args
|
||||||
|
LOG.info("espeak-ng %s", shlex.join(args))
|
||||||
|
try:
|
||||||
|
proc = subprocess.run(
|
||||||
|
cmd,
|
||||||
|
input=stdin_text,
|
||||||
|
capture_output=True,
|
||||||
|
timeout=60,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
raise HTTPException(status_code=504, detail="espeak-ng timed out")
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500,
|
||||||
|
detail=f"espeak-ng exit {proc.returncode}: {proc.stderr.decode('utf-8', errors='replace')[:512]}",
|
||||||
|
)
|
||||||
|
return proc.stdout
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health():
|
||||||
|
return {"status": "ok", "languages": list(LANGUAGES.keys())}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/voices")
|
||||||
|
def voices():
|
||||||
|
return {
|
||||||
|
"voices": [
|
||||||
|
{
|
||||||
|
"name": code,
|
||||||
|
"displayName": meta["label"],
|
||||||
|
"language": code,
|
||||||
|
"isRightToLeft": meta["rtl"],
|
||||||
|
"engine": "espeak-ng",
|
||||||
|
}
|
||||||
|
for code, meta in LANGUAGES.items()
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/tts")
|
||||||
|
def tts(req: TtsRequest) -> Response:
|
||||||
|
if not req.text.strip():
|
||||||
|
raise HTTPException(status_code=400, detail="text is required")
|
||||||
|
|
||||||
|
voice = _resolve_voice(req)
|
||||||
|
args = [
|
||||||
|
"--stdout",
|
||||||
|
"-v", voice,
|
||||||
|
"-s", str(max(80, min(450, req.rate))),
|
||||||
|
"-p", str(max(0, min(99, req.pitch))),
|
||||||
|
"-a", str(max(0, min(200, req.volume))),
|
||||||
|
]
|
||||||
|
wav = _run_espeak(args, req.text.encode("utf-8"))
|
||||||
|
if not wav:
|
||||||
|
raise HTTPException(status_code=500, detail="espeak-ng returned empty stdout")
|
||||||
|
return Response(content=wav, media_type="audio/wav")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# /timings — synth + word-level timing from espeak's phoneme/word stream.
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# espeak-ng's --pho flag emits a phoneme stream:
|
||||||
|
#
|
||||||
|
# _ 5 phon...
|
||||||
|
# _ 56 phon...
|
||||||
|
# _ 67 phon...
|
||||||
|
#
|
||||||
|
# That alone doesn't give word boundaries. Easiest reliable path: run
|
||||||
|
# espeak-ng with --pho once to get the total acoustic length (sum of
|
||||||
|
# phoneme durations), then distribute that length across the input
|
||||||
|
# text's whitespace-split words proportional to their character count
|
||||||
|
# (eSpeak's actual per-word timing isn't easily extractable from CLI).
|
||||||
|
# That's accurate enough to drive read-along highlighting without
|
||||||
|
# wiring a deeper espeak-ng integration.
|
||||||
|
#
|
||||||
|
# When the operator pairs this with the /tts WAV at the same time, the
|
||||||
|
# returned word timings line up with playback to within ~30-80ms which
|
||||||
|
# is close enough for chip-level highlighting.
|
||||||
|
|
||||||
|
PHONEME_DURATION_RE = re.compile(r"^\s*\S+\s+(\d+)\s+", re.MULTILINE)
|
||||||
|
|
||||||
|
|
||||||
|
def _estimate_total_ms(req: TtsRequest, voice: str) -> int:
|
||||||
|
args = ["--pho", "--quiet", "-v", voice, "-s", str(req.rate)]
|
||||||
|
out = _run_espeak(args, req.text.encode("utf-8"))
|
||||||
|
text = out.decode("utf-8", errors="replace")
|
||||||
|
total = 0
|
||||||
|
for match in PHONEME_DURATION_RE.finditer(text):
|
||||||
|
try:
|
||||||
|
total += int(match.group(1))
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if total == 0:
|
||||||
|
# Fallback: rough heuristic at the configured speech rate (words/minute).
|
||||||
|
words = max(1, len(req.text.split()))
|
||||||
|
total = int(words / max(60, req.rate) * 60_000)
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/timings")
|
||||||
|
def timings(req: TtsRequest):
|
||||||
|
if not req.text.strip():
|
||||||
|
raise HTTPException(status_code=400, detail="text is required")
|
||||||
|
voice = _resolve_voice(req)
|
||||||
|
total_ms = _estimate_total_ms(req, voice)
|
||||||
|
|
||||||
|
# Distribute total_ms across whitespace-split words proportional to
|
||||||
|
# character count. Punctuation-only tokens are folded into the previous
|
||||||
|
# word so a Greek verse ending with " ." doesn't claim a chunk of time.
|
||||||
|
words = req.text.split()
|
||||||
|
if not words:
|
||||||
|
return {"text": req.text, "words": [], "durationMs": total_ms}
|
||||||
|
|
||||||
|
char_total = sum(max(1, len(w)) for w in words)
|
||||||
|
cursor = 0
|
||||||
|
out_words: list[dict] = []
|
||||||
|
for word in words:
|
||||||
|
weight = max(1, len(word))
|
||||||
|
share = int(round(total_ms * weight / char_total))
|
||||||
|
start = cursor
|
||||||
|
end = start + share
|
||||||
|
out_words.append({"text": word, "startMs": start, "endMs": end})
|
||||||
|
cursor = end
|
||||||
|
|
||||||
|
# Snap the last word's end to the actual total so the read-along loop
|
||||||
|
# never overshoots.
|
||||||
|
if out_words:
|
||||||
|
out_words[-1]["endMs"] = total_ms
|
||||||
|
|
||||||
|
return JSONResponse(
|
||||||
|
{
|
||||||
|
"text": req.text,
|
||||||
|
"language": req.language,
|
||||||
|
"voice": voice,
|
||||||
|
"words": out_words,
|
||||||
|
"durationMs": total_ms,
|
||||||
|
}
|
||||||
|
)
|
||||||
2
apps/fc-ttsreader/biblical-tts/requirements.txt
Normal file
2
apps/fc-ttsreader/biblical-tts/requirements.txt
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
fastapi==0.115.6
|
||||||
|
uvicorn==0.34.0
|
||||||
@@ -37,6 +37,19 @@ spec:
|
|||||||
app.kubernetes.io/name: ttsreader-piper
|
app.kubernetes.io/name: ttsreader-piper
|
||||||
app.kubernetes.io/part-of: flowercore
|
app.kubernetes.io/part-of: flowercore
|
||||||
spec:
|
spec:
|
||||||
|
# Bypass CoreDNS's *.iamworkin.lan wildcard so the init container reaches
|
||||||
|
# huggingface.co directly when it seeds voice models.
|
||||||
|
dnsPolicy: None
|
||||||
|
dnsConfig:
|
||||||
|
nameservers:
|
||||||
|
- 10.43.0.10
|
||||||
|
searches:
|
||||||
|
- fc-ttsreader.svc.cluster.local
|
||||||
|
- svc.cluster.local
|
||||||
|
- cluster.local
|
||||||
|
options:
|
||||||
|
- name: ndots
|
||||||
|
value: "2"
|
||||||
initContainers:
|
initContainers:
|
||||||
- name: seed-voices
|
- name: seed-voices
|
||||||
image: rhasspy/wyoming-piper:latest
|
image: rhasspy/wyoming-piper:latest
|
||||||
@@ -97,13 +110,19 @@ spec:
|
|||||||
ports:
|
ports:
|
||||||
- containerPort: 10200
|
- containerPort: 10200
|
||||||
name: wyoming
|
name: wyoming
|
||||||
|
# Memory bumped after observed OOMKills during real chapter
|
||||||
|
# renders 2026-04-25. Piper's eSpeak phonemizer + onnx runtime
|
||||||
|
# spikes well past 1 Gi on long unpunctuated paragraphs from
|
||||||
|
# PDF / book imports. 3 Gi gives headroom plus the
|
||||||
|
# transcribe-audio-to-Quick-Read flow that hits Piper through
|
||||||
|
# the same model.
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 250m
|
cpu: 250m
|
||||||
memory: 256Mi
|
memory: 512Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 1000m
|
cpu: 2000m
|
||||||
memory: 1Gi
|
memory: 3Gi
|
||||||
volumeMounts:
|
volumeMounts:
|
||||||
- name: data
|
- name: data
|
||||||
mountPath: /data
|
mountPath: /data
|
||||||
@@ -112,6 +131,377 @@ spec:
|
|||||||
persistentVolumeClaim:
|
persistentVolumeClaim:
|
||||||
claimName: ttsreader-piper-data
|
claimName: ttsreader-piper-data
|
||||||
---
|
---
|
||||||
|
# fc-speech-align — cluster-native faster-whisper wrapper.
|
||||||
|
# Exposes POST /align (fc-align contract used by FlowerCore.Shared.Speech) AND
|
||||||
|
# POST /transcribe (audio-file-in feature). CPU model = base.en, int8 compute.
|
||||||
|
# Source: bluejay-infra/apps/fc-ttsreader/speech-align/ (Dockerfile + app.py).
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-align-models
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 5Gi
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-align
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-align
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: ttsreader-align
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-align
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
# Bypass CoreDNS's *.iamworkin.lan template hijack on public hosts
|
||||||
|
# (huggingface.co model download at first boot would otherwise resolve
|
||||||
|
# to Traefik VIP via search expansion). Drops the iamworkin.lan suffix.
|
||||||
|
dnsPolicy: None
|
||||||
|
dnsConfig:
|
||||||
|
nameservers:
|
||||||
|
- 10.43.0.10
|
||||||
|
searches:
|
||||||
|
- fc-ttsreader.svc.cluster.local
|
||||||
|
- svc.cluster.local
|
||||||
|
- cluster.local
|
||||||
|
options:
|
||||||
|
- name: ndots
|
||||||
|
value: "2"
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 1654
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1654
|
||||||
|
containers:
|
||||||
|
- name: align
|
||||||
|
image: localhost/fc-speech-align:v3
|
||||||
|
imagePullPolicy: Never
|
||||||
|
ports:
|
||||||
|
- containerPort: 9200
|
||||||
|
name: http
|
||||||
|
env:
|
||||||
|
- name: WHISPER_MODEL
|
||||||
|
value: "Systran/faster-whisper-base.en"
|
||||||
|
- name: WHISPER_DEVICE
|
||||||
|
value: "cpu"
|
||||||
|
- name: WHISPER_COMPUTE_TYPE
|
||||||
|
value: "int8"
|
||||||
|
- name: WHISPER_CACHE_DIR
|
||||||
|
value: "/models"
|
||||||
|
- name: DEFAULT_LANGUAGE
|
||||||
|
value: "en"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 512Mi
|
||||||
|
limits:
|
||||||
|
cpu: 2000m
|
||||||
|
memory: 2Gi
|
||||||
|
volumeMounts:
|
||||||
|
- name: models
|
||||||
|
mountPath: /models
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 9200
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 18
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 9200
|
||||||
|
initialDelaySeconds: 180
|
||||||
|
periodSeconds: 30
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 3
|
||||||
|
volumes:
|
||||||
|
- name: models
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: ttsreader-align-models
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-align
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: ttsreader-align
|
||||||
|
ports:
|
||||||
|
- port: 9200
|
||||||
|
targetPort: 9200
|
||||||
|
name: http
|
||||||
|
---
|
||||||
|
# ttsreader-kokoro — Kokoro-82M TTS via the kokoro-fastapi container.
|
||||||
|
# Provides high-quality English voices alongside Piper for the TtsReader
|
||||||
|
# render pipeline AND for AiStation when it talks to the cluster TTS plane
|
||||||
|
# (instead of pointing back at BLUEJAY-WS:10401). Model + voices ship
|
||||||
|
# inside the container image, so no PVC is needed.
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-kokoro
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-kokoro
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: ttsreader-kokoro
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-kokoro
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
# Same DNS bypass as ttsreader-align — without it, the *.iamworkin.lan
|
||||||
|
# CoreDNS template would hijack hexgrad/Kokoro-82M's HuggingFace-style
|
||||||
|
# repo lookups during model warmup.
|
||||||
|
dnsPolicy: None
|
||||||
|
dnsConfig:
|
||||||
|
nameservers:
|
||||||
|
- 10.43.0.10
|
||||||
|
searches:
|
||||||
|
- fc-ttsreader.svc.cluster.local
|
||||||
|
- svc.cluster.local
|
||||||
|
- cluster.local
|
||||||
|
options:
|
||||||
|
- name: ndots
|
||||||
|
value: "2"
|
||||||
|
containers:
|
||||||
|
- name: kokoro
|
||||||
|
image: ghcr.io/remsky/kokoro-fastapi-cpu:latest
|
||||||
|
ports:
|
||||||
|
- containerPort: 8880
|
||||||
|
name: http
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 1Gi
|
||||||
|
limits:
|
||||||
|
cpu: 2000m
|
||||||
|
memory: 3Gi
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /v1/audio/voices
|
||||||
|
port: 8880
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 18
|
||||||
|
# Sprint E Phase 1a (kokoro stability) — 4 restarts in 2d6h with
|
||||||
|
# exit 143 traced to liveness probe `context deadline exceeded` while
|
||||||
|
# kokoro was busy synthesizing. /v1/audio/voices shares the FastAPI
|
||||||
|
# worker pool with /v1/audio/speech, so a long synth can starve the
|
||||||
|
# probe out within the prior 5s × 3 = 15s window. Bump timeoutSeconds
|
||||||
|
# 5 → 15 and failureThreshold 3 → 5 → 75s grace before kubelet kills
|
||||||
|
# the pod. The TtsCircuitBreaker on the synthesizer side (Phase 1b)
|
||||||
|
# backs this up so the FC backend stops slamming kokoro during
|
||||||
|
# recovery.
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /v1/audio/voices
|
||||||
|
port: 8880
|
||||||
|
initialDelaySeconds: 180
|
||||||
|
periodSeconds: 30
|
||||||
|
timeoutSeconds: 15
|
||||||
|
failureThreshold: 5
|
||||||
|
---
|
||||||
|
# fc-biblical-tts — eSpeak-NG-backed Ancient Greek + Hebrew TTS with
|
||||||
|
# word-level timing for read-along playback. Companion to ttsreader-kokoro
|
||||||
|
# (modern English) and ttsreader-piper (English narrator); operators pick
|
||||||
|
# whichever engine matches the source text. Source:
|
||||||
|
# bluejay-infra/apps/fc-ttsreader/biblical-tts/
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-biblical
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-biblical
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: ttsreader-biblical
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-biblical
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 1654
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1654
|
||||||
|
containers:
|
||||||
|
- name: biblical-tts
|
||||||
|
image: localhost/fc-biblical-tts:v1
|
||||||
|
imagePullPolicy: Never
|
||||||
|
ports:
|
||||||
|
- containerPort: 10402
|
||||||
|
name: http
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 1000m
|
||||||
|
memory: 512Mi
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 10402
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 6
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 10402
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 30
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 3
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-biblical
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: ttsreader-biblical
|
||||||
|
ports:
|
||||||
|
- port: 10402
|
||||||
|
targetPort: 10402
|
||||||
|
name: http
|
||||||
|
---
|
||||||
|
# fc-modern-tts — Microsoft Edge Read Aloud bridge for Modern Hebrew
|
||||||
|
# (he-IL-AvriNeural et al) and Modern Greek (el-GR-NestorasNeural et al).
|
||||||
|
# Pairs with ttsreader-biblical: biblical engine handles unpointed
|
||||||
|
# Greek + Hebrew, modern engine handles narrative translations the
|
||||||
|
# operator reads alongside.
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-modern
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-modern
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: ttsreader-modern
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: ttsreader-modern
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
# edge-tts needs egress to *.tts.speech.microsoft.com — bypass the
|
||||||
|
# iamworkin.lan template hijack so the lookup doesn't fall back to
|
||||||
|
# Traefik VIP via search expansion.
|
||||||
|
dnsPolicy: None
|
||||||
|
dnsConfig:
|
||||||
|
nameservers:
|
||||||
|
- 10.43.0.10
|
||||||
|
searches:
|
||||||
|
- fc-ttsreader.svc.cluster.local
|
||||||
|
- svc.cluster.local
|
||||||
|
- cluster.local
|
||||||
|
options:
|
||||||
|
- name: ndots
|
||||||
|
value: "2"
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 1654
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1654
|
||||||
|
containers:
|
||||||
|
- name: modern-tts
|
||||||
|
image: localhost/fc-modern-tts:v1
|
||||||
|
imagePullPolicy: Never
|
||||||
|
ports:
|
||||||
|
- containerPort: 10403
|
||||||
|
name: http
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 1000m
|
||||||
|
memory: 512Mi
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 10403
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 6
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 10403
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 30
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 3
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-modern
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: ttsreader-modern
|
||||||
|
ports:
|
||||||
|
- port: 10403
|
||||||
|
targetPort: 10403
|
||||||
|
name: http
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: ttsreader-kokoro
|
||||||
|
namespace: fc-ttsreader
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: ttsreader-kokoro
|
||||||
|
ports:
|
||||||
|
- port: 8880
|
||||||
|
targetPort: 8880
|
||||||
|
name: http
|
||||||
|
---
|
||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
metadata:
|
metadata:
|
||||||
@@ -142,7 +532,7 @@ spec:
|
|||||||
fsGroupChangePolicy: OnRootMismatch
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
containers:
|
containers:
|
||||||
- name: web
|
- name: web
|
||||||
image: localhost/fc-ttsreader-web:v202604240023
|
image: localhost/fc-ttsreader-web:v20260506-47a88ae
|
||||||
imagePullPolicy: Never
|
imagePullPolicy: Never
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 5217
|
- containerPort: 5217
|
||||||
@@ -160,12 +550,43 @@ spec:
|
|||||||
value: "/usr/bin/ffmpeg"
|
value: "/usr/bin/ffmpeg"
|
||||||
- name: TtsReader__Bible__CorpusRoot
|
- name: TtsReader__Bible__CorpusRoot
|
||||||
value: "/data/corpus-cache/world-english-bible/eng/usx"
|
value: "/data/corpus-cache/world-english-bible/eng/usx"
|
||||||
|
- name: TtsReader__ChapterContext__DatabasePath
|
||||||
|
value: "/data/chapter-context.db"
|
||||||
- name: TtsReader__Jobs__Root
|
- name: TtsReader__Jobs__Root
|
||||||
value: "/data/jobs"
|
value: "/data/jobs"
|
||||||
- name: TtsReader__Piper__Host
|
- name: TtsReader__Piper__Host
|
||||||
value: "ttsreader-piper.fc-ttsreader.svc.cluster.local."
|
value: "ttsreader-piper.fc-ttsreader.svc.cluster.local."
|
||||||
- name: TtsReader__Piper__Port
|
- name: TtsReader__Piper__Port
|
||||||
value: "10200"
|
value: "10200"
|
||||||
|
- name: TtsReader__Kokoro__Enabled
|
||||||
|
value: "true"
|
||||||
|
- name: TtsReader__Kokoro__BaseUrl
|
||||||
|
# Cluster-native ttsreader-kokoro Service — replaces the prior
|
||||||
|
# BLUEJAY-WS host pointer so the render pipeline doesn't need
|
||||||
|
# the workstation up. AiStation can still hit its local
|
||||||
|
# http://localhost:8880 instance.
|
||||||
|
value: "http://ttsreader-kokoro.fc-ttsreader.svc.cluster.local.:8880"
|
||||||
|
- name: TtsReader__Kokoro__TimeoutSeconds
|
||||||
|
value: "120"
|
||||||
|
- name: Speech__Alignment__Enabled
|
||||||
|
# Cluster-native faster-whisper (Lane F, 2026-04-25). The
|
||||||
|
# ttsreader-align deployment in this manifest wraps
|
||||||
|
# SYSTRAN/faster-whisper with a /align endpoint matching the
|
||||||
|
# FlowerCore.Shared.Speech master contract.
|
||||||
|
value: "true"
|
||||||
|
- name: Speech__Alignment__BaseUrl
|
||||||
|
value: "http://ttsreader-align.fc-ttsreader.svc.cluster.local.:9200"
|
||||||
|
- name: Speech__Alignment__TimeoutSeconds
|
||||||
|
value: "120"
|
||||||
|
# Cluster-native transcription endpoint shares the same pod
|
||||||
|
# (POST /transcribe). Lane G consumes this from the
|
||||||
|
# FlowerCore.TtsReader.Web AudioImport feature.
|
||||||
|
- name: TtsReader__Transcription__Enabled
|
||||||
|
value: "true"
|
||||||
|
- name: TtsReader__Transcription__BaseUrl
|
||||||
|
value: "http://ttsreader-align.fc-ttsreader.svc.cluster.local.:9200"
|
||||||
|
- name: TtsReader__Transcription__TimeoutSeconds
|
||||||
|
value: "300"
|
||||||
- name: TtsReader__Ollama__BaseUrl
|
- name: TtsReader__Ollama__BaseUrl
|
||||||
value: "http://10.0.57.17:11434"
|
value: "http://10.0.57.17:11434"
|
||||||
- name: TtsReader__Ollama__DefaultModel
|
- name: TtsReader__Ollama__DefaultModel
|
||||||
@@ -176,6 +597,19 @@ spec:
|
|||||||
value: "/data/logs"
|
value: "/data/logs"
|
||||||
- name: TtsReader__Runtime__SmokeStatePath
|
- name: TtsReader__Runtime__SmokeStatePath
|
||||||
value: "/data/ops/smoke-status.json"
|
value: "/data/ops/smoke-status.json"
|
||||||
|
# Sprint E Day 8 voice-preview disk cache — writes WAVs under
|
||||||
|
# this directory. Default "data/voice-previews" resolves to
|
||||||
|
# the read-only $HOME path under runAsNonRoot=true. Pin to
|
||||||
|
# the writable PVC mount.
|
||||||
|
- name: TtsReader__Preview__CacheDirectory
|
||||||
|
value: "/data/voice-previews"
|
||||||
|
# Sprint E XXL Phase 4γ — content-addressed CDN bundle dir for
|
||||||
|
# POST /api/v1/render. Default "wwwroot/cdn" resolves under the
|
||||||
|
# read-only app filesystem, so pin to the writable PVC mount
|
||||||
|
# alongside other TtsReader runtime data. Manifests + cue audio
|
||||||
|
# land at /data/cdn/sha256/<hash>/manifest.json + cues/.
|
||||||
|
- name: TtsReader__Render__CdnDirectory
|
||||||
|
value: "/data/cdn"
|
||||||
- name: Auth__ApiKey
|
- name: Auth__ApiKey
|
||||||
valueFrom:
|
valueFrom:
|
||||||
secretKeyRef:
|
secretKeyRef:
|
||||||
@@ -190,7 +624,10 @@ spec:
|
|||||||
optional: true
|
optional: true
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 100m
|
# The cluster is currently saturated on requested CPU by
|
||||||
|
# remotedesktop workloads even when real usage is low.
|
||||||
|
# Keep the web frontend schedulable under that pressure.
|
||||||
|
cpu: 10m
|
||||||
memory: 256Mi
|
memory: 256Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 500m
|
cpu: 500m
|
||||||
|
|||||||
36
apps/fc-ttsreader/modern-tts/Dockerfile
Normal file
36
apps/fc-ttsreader/modern-tts/Dockerfile
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
# FlowerCore modern-tts — wraps Microsoft Edge's Read Aloud TTS service
|
||||||
|
# (via the edge-tts Python package) to give the cluster studio-quality
|
||||||
|
# Modern Hebrew (he-IL-*) and Modern Greek (el-GR-*) voices alongside the
|
||||||
|
# eSpeak biblical engine. Same shape as fc-biblical-tts so the .NET client
|
||||||
|
# lives in the same Shared.Speech package.
|
||||||
|
#
|
||||||
|
# Note: edge-tts depends on Microsoft's public Edge endpoint; the cluster
|
||||||
|
# pod needs egress to *.tts.speech.microsoft.com. dnsPolicy: None on the
|
||||||
|
# Deployment makes sure the iamworkin.lan template hijack doesn't rewrite
|
||||||
|
# the lookup back to Traefik VIP.
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
||||||
|
PIP_NO_CACHE_DIR=1
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends \
|
||||||
|
ca-certificates \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY requirements.txt /app/
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
COPY app.py /app/
|
||||||
|
|
||||||
|
RUN useradd --create-home --shell /usr/sbin/nologin --uid 1654 tts
|
||||||
|
USER 1654
|
||||||
|
|
||||||
|
EXPOSE 10403
|
||||||
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
||||||
|
CMD python -c "import urllib.request,sys; urllib.request.urlopen('http://127.0.0.1:10403/health',timeout=3); sys.exit(0)" || exit 1
|
||||||
|
|
||||||
|
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "10403", "--workers", "1"]
|
||||||
238
apps/fc-ttsreader/modern-tts/app.py
Normal file
238
apps/fc-ttsreader/modern-tts/app.py
Normal file
@@ -0,0 +1,238 @@
|
|||||||
|
"""FlowerCore modern-tts — Microsoft Edge Read Aloud bridge for Modern
|
||||||
|
Hebrew and Modern Greek (and other Edge-supported languages).
|
||||||
|
|
||||||
|
Endpoints:
|
||||||
|
|
||||||
|
* POST /tts — body: {"text", "voice", "rate"?, "volume"?, "pitch"?}
|
||||||
|
returns audio/mpeg (Edge returns MP3) which the
|
||||||
|
upstream FasterWhisperAlignmentClient + the WPF
|
||||||
|
MediaPlayer both handle natively.
|
||||||
|
* POST /timings — same body shape but returns
|
||||||
|
{"text", "voice", "words": [{"text","startMs","endMs"}],
|
||||||
|
"durationMs": ...} sourced from Edge's WordBoundary
|
||||||
|
events — much more accurate than eSpeak's
|
||||||
|
proportional-distribution approach because Edge
|
||||||
|
emits real per-word offsets during synthesis.
|
||||||
|
* GET /voices — voice catalog Edge knows about. Filtered to
|
||||||
|
Hebrew + Greek by default; ?language=all returns
|
||||||
|
everything Edge supports.
|
||||||
|
* GET /health — fast readiness check.
|
||||||
|
|
||||||
|
Pairs with fc-biblical-tts (eSpeak Ancient Greek + Hebrew). The biblical
|
||||||
|
engine handles unpointed Hebrew + Erasmian Greek; this engine handles
|
||||||
|
narrative Modern Hebrew + Modern Greek for translations the operator
|
||||||
|
might be reading alongside the original.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
import edge_tts
|
||||||
|
from fastapi import FastAPI, HTTPException
|
||||||
|
from fastapi.responses import JSONResponse, Response
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
LOG = logging.getLogger("modern_tts")
|
||||||
|
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||||
|
|
||||||
|
app = FastAPI(title="FlowerCore modern-tts", version="1.0.0")
|
||||||
|
|
||||||
|
# Default voices by short code so AiStation can pick a sensible default
|
||||||
|
# when the operator hasn't explicitly asked for one. Edge has multiple
|
||||||
|
# voices per locale — these are the calmest male+female narrators.
|
||||||
|
DEFAULT_VOICES = {
|
||||||
|
"he": "he-IL-AvriNeural",
|
||||||
|
"he-IL": "he-IL-AvriNeural",
|
||||||
|
"el": "el-GR-NestorasNeural",
|
||||||
|
"el-GR": "el-GR-NestorasNeural",
|
||||||
|
"en": "en-US-AriaNeural",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class TtsRequest(BaseModel):
|
||||||
|
text: str
|
||||||
|
voice: Optional[str] = None
|
||||||
|
language: Optional[str] = None
|
||||||
|
rate: str = "+0%" # Edge accepts +20%, -10%, etc.
|
||||||
|
volume: str = "+0%"
|
||||||
|
pitch: str = "+0Hz"
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_voice(req: TtsRequest) -> str:
|
||||||
|
if req.voice:
|
||||||
|
return req.voice.strip()
|
||||||
|
if req.language and req.language in DEFAULT_VOICES:
|
||||||
|
return DEFAULT_VOICES[req.language]
|
||||||
|
return DEFAULT_VOICES["he"]
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health():
|
||||||
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/voices")
|
||||||
|
async def voices(language: str = "default"):
|
||||||
|
catalog = await edge_tts.list_voices()
|
||||||
|
if language == "all":
|
||||||
|
return {"voices": catalog}
|
||||||
|
|
||||||
|
# Default response: filter to languages relevant to the FlowerCore
|
||||||
|
# biblical workflow (Hebrew + Greek) so the AiStation voice picker
|
||||||
|
# isn't overwhelmed by 400+ Edge voices.
|
||||||
|
keep = ("he-", "el-")
|
||||||
|
filtered = [v for v in catalog if any(v.get("ShortName", "").startswith(k) for k in keep)]
|
||||||
|
return {"voices": filtered}
|
||||||
|
|
||||||
|
|
||||||
|
async def _synth_with_subtitles(req: TtsRequest):
|
||||||
|
voice = _resolve_voice(req)
|
||||||
|
LOG.info("edge-tts synth voice=%s len=%d", voice, len(req.text))
|
||||||
|
communicate = edge_tts.Communicate(
|
||||||
|
req.text,
|
||||||
|
voice=voice,
|
||||||
|
rate=req.rate,
|
||||||
|
volume=req.volume,
|
||||||
|
pitch=req.pitch,
|
||||||
|
)
|
||||||
|
audio_buf = io.BytesIO()
|
||||||
|
word_events: list[dict] = []
|
||||||
|
async for chunk in communicate.stream():
|
||||||
|
if chunk["type"] == "audio":
|
||||||
|
audio_buf.write(chunk["data"])
|
||||||
|
elif chunk["type"] == "WordBoundary":
|
||||||
|
word_events.append({
|
||||||
|
"text": chunk.get("text") or "",
|
||||||
|
"offset": chunk.get("offset", 0), # 100-ns ticks
|
||||||
|
"duration": chunk.get("duration", 0), # 100-ns ticks
|
||||||
|
})
|
||||||
|
return voice, audio_buf.getvalue(), word_events
|
||||||
|
|
||||||
|
|
||||||
|
def _to_ms(ticks_100ns: int) -> int:
|
||||||
|
# Edge emits offsets in 100-nanosecond ticks (.NET TimeSpan style).
|
||||||
|
return int(round(ticks_100ns / 10_000))
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/tts")
|
||||||
|
async def tts(req: TtsRequest):
|
||||||
|
if not req.text.strip():
|
||||||
|
raise HTTPException(status_code=400, detail="text is required")
|
||||||
|
try:
|
||||||
|
voice, audio_bytes, _ = await _synth_with_subtitles(req)
|
||||||
|
except edge_tts.exceptions.NoAudioReceived:
|
||||||
|
raise HTTPException(status_code=502, detail="edge-tts returned no audio for the supplied voice/text.")
|
||||||
|
except Exception as ex:
|
||||||
|
raise HTTPException(status_code=502, detail=f"edge-tts failure: {ex}")
|
||||||
|
if not audio_bytes:
|
||||||
|
raise HTTPException(status_code=502, detail="edge-tts returned an empty audio stream.")
|
||||||
|
return Response(content=audio_bytes, media_type="audio/mpeg",
|
||||||
|
headers={"X-FlowerCore-Voice": voice})
|
||||||
|
|
||||||
|
|
||||||
|
def _estimate_duration_ms_from_mp3(audio_bytes: bytes) -> int:
|
||||||
|
"""Best-effort duration estimate from raw MP3 bytes by walking frame
|
||||||
|
headers. Edge always returns CBR ~24kbps mono so we can infer total ms
|
||||||
|
from frame count. If parsing fails, return 0 and let the caller fall
|
||||||
|
through to a per-character heuristic."""
|
||||||
|
if not audio_bytes:
|
||||||
|
return 0
|
||||||
|
# MP3 sample rates by version+layer (MPEG1 layer3 / MPEG2 layer3 / MPEG2.5 layer3).
|
||||||
|
# We just walk frame headers and count frames; each frame is 1152 samples.
|
||||||
|
sample_rates_v1 = [44100, 48000, 32000, 0]
|
||||||
|
sample_rates_v2 = [22050, 24000, 16000, 0]
|
||||||
|
sample_rates_v25 = [11025, 12000, 8000, 0]
|
||||||
|
bitrates_v1_l3 = [0,32000,40000,48000,56000,64000,80000,96000,112000,128000,160000,192000,224000,256000,320000,0]
|
||||||
|
bitrates_v2_l3 = [0,8000,16000,24000,32000,40000,48000,56000,64000,80000,96000,112000,128000,144000,160000,0]
|
||||||
|
|
||||||
|
pos = 0
|
||||||
|
total_samples = 0
|
||||||
|
sample_rate = 0
|
||||||
|
while pos + 4 <= len(audio_bytes):
|
||||||
|
b0, b1, b2, b3 = audio_bytes[pos], audio_bytes[pos+1], audio_bytes[pos+2], audio_bytes[pos+3]
|
||||||
|
if b0 != 0xFF or (b1 & 0xE0) != 0xE0:
|
||||||
|
pos += 1
|
||||||
|
continue
|
||||||
|
version_bits = (b1 >> 3) & 0x03
|
||||||
|
layer_bits = (b1 >> 1) & 0x03
|
||||||
|
if layer_bits != 0x01: # layer 3 only
|
||||||
|
pos += 1
|
||||||
|
continue
|
||||||
|
bitrate_index = (b2 >> 4) & 0x0F
|
||||||
|
sample_rate_index = (b2 >> 2) & 0x03
|
||||||
|
padding = (b2 >> 1) & 0x01
|
||||||
|
if version_bits == 0x03: # MPEG1
|
||||||
|
sample_rate = sample_rates_v1[sample_rate_index]
|
||||||
|
bitrate = bitrates_v1_l3[bitrate_index]
|
||||||
|
samples_per_frame = 1152
|
||||||
|
elif version_bits == 0x02: # MPEG2
|
||||||
|
sample_rate = sample_rates_v2[sample_rate_index]
|
||||||
|
bitrate = bitrates_v2_l3[bitrate_index]
|
||||||
|
samples_per_frame = 576
|
||||||
|
elif version_bits == 0x00: # MPEG2.5
|
||||||
|
sample_rate = sample_rates_v25[sample_rate_index]
|
||||||
|
bitrate = bitrates_v2_l3[bitrate_index]
|
||||||
|
samples_per_frame = 576
|
||||||
|
else:
|
||||||
|
pos += 1
|
||||||
|
continue
|
||||||
|
if not (sample_rate and bitrate):
|
||||||
|
pos += 1
|
||||||
|
continue
|
||||||
|
frame_length = int((samples_per_frame * bitrate / 8) / sample_rate) + padding
|
||||||
|
if frame_length <= 0:
|
||||||
|
pos += 1
|
||||||
|
continue
|
||||||
|
total_samples += samples_per_frame
|
||||||
|
pos += frame_length
|
||||||
|
|
||||||
|
if sample_rate <= 0:
|
||||||
|
return 0
|
||||||
|
return int(round(total_samples * 1000 / sample_rate))
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/timings")
|
||||||
|
async def timings(req: TtsRequest):
|
||||||
|
if not req.text.strip():
|
||||||
|
raise HTTPException(status_code=400, detail="text is required")
|
||||||
|
try:
|
||||||
|
voice, audio_bytes, events = await _synth_with_subtitles(req)
|
||||||
|
except Exception as ex:
|
||||||
|
raise HTTPException(status_code=502, detail=f"edge-tts failure: {ex}")
|
||||||
|
|
||||||
|
words: list[dict] = []
|
||||||
|
for event in events:
|
||||||
|
start = _to_ms(event["offset"])
|
||||||
|
end = start + _to_ms(event["duration"])
|
||||||
|
words.append({"text": event.get("text", ""), "startMs": start, "endMs": end})
|
||||||
|
|
||||||
|
# Edge sometimes omits WordBoundary events for non-English voices
|
||||||
|
# (notably he-IL-* and el-GR-*). Fall back to proportional distribution
|
||||||
|
# over the input text — same approach the eSpeak biblical-tts uses.
|
||||||
|
if not words and req.text.strip():
|
||||||
|
total_ms = _estimate_duration_ms_from_mp3(audio_bytes)
|
||||||
|
if total_ms <= 0:
|
||||||
|
# Last-resort fallback: ~600ms per word at average speaking rate.
|
||||||
|
total_ms = max(1, len(req.text.split())) * 600
|
||||||
|
tokens = req.text.split()
|
||||||
|
if tokens:
|
||||||
|
char_total = sum(max(1, len(w)) for w in tokens)
|
||||||
|
cursor = 0
|
||||||
|
for token in tokens:
|
||||||
|
share = int(round(total_ms * max(1, len(token)) / char_total))
|
||||||
|
start = cursor
|
||||||
|
end = start + share
|
||||||
|
words.append({"text": token, "startMs": start, "endMs": end})
|
||||||
|
cursor = end
|
||||||
|
words[-1]["endMs"] = total_ms
|
||||||
|
|
||||||
|
duration_ms = words[-1]["endMs"] if words else 0
|
||||||
|
return JSONResponse({
|
||||||
|
"text": req.text,
|
||||||
|
"voice": voice,
|
||||||
|
"words": words,
|
||||||
|
"durationMs": duration_ms,
|
||||||
|
"audioBytes": len(audio_bytes),
|
||||||
|
})
|
||||||
3
apps/fc-ttsreader/modern-tts/requirements.txt
Normal file
3
apps/fc-ttsreader/modern-tts/requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
fastapi==0.115.6
|
||||||
|
uvicorn==0.34.0
|
||||||
|
edge-tts==7.2.8
|
||||||
47
apps/fc-ttsreader/speech-align/Dockerfile
Normal file
47
apps/fc-ttsreader/speech-align/Dockerfile
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
# FlowerCore speech-align — wraps SYSTRAN/faster-whisper with /align +
|
||||||
|
# /transcribe endpoints used by FlowerCore.TtsReader. CPU-only image; the
|
||||||
|
# default int8 compute type runs base.en at ~real-time on a single core.
|
||||||
|
#
|
||||||
|
# Build: podman build -t localhost/fc-speech-align:<ver> .
|
||||||
|
# Run: podman run --rm -p 9200:9200 -v fc-speech-align-models:/models localhost/fc-speech-align:<ver>
|
||||||
|
|
||||||
|
FROM python:3.12-slim AS base
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
||||||
|
PIP_NO_CACHE_DIR=1 \
|
||||||
|
WHISPER_MODEL=Systran/faster-whisper-base.en \
|
||||||
|
WHISPER_CACHE_DIR=/models \
|
||||||
|
WHISPER_DEVICE=cpu \
|
||||||
|
WHISPER_COMPUTE_TYPE=int8 \
|
||||||
|
DEFAULT_LANGUAGE=en \
|
||||||
|
MAX_AUDIO_BYTES=52428800
|
||||||
|
|
||||||
|
# faster-whisper depends on libsndfile1 + libgomp1 (OpenMP runtime). ffmpeg is
|
||||||
|
# pulled in for non-WAV inputs (transcribe accepts any container).
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends \
|
||||||
|
libsndfile1 \
|
||||||
|
libgomp1 \
|
||||||
|
ffmpeg \
|
||||||
|
ca-certificates \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY requirements.txt /app/
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
COPY app.py /app/
|
||||||
|
|
||||||
|
# Run as a non-root user to satisfy K8s securityContext.runAsNonRoot.
|
||||||
|
RUN useradd --create-home --shell /usr/sbin/nologin --uid 1654 align \
|
||||||
|
&& mkdir -p /models \
|
||||||
|
&& chown -R 1654:1654 /models
|
||||||
|
USER 1654
|
||||||
|
|
||||||
|
EXPOSE 9200
|
||||||
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=120s --retries=3 \
|
||||||
|
CMD python -c "import urllib.request,sys; urllib.request.urlopen('http://127.0.0.1:9200/health',timeout=3); sys.exit(0)" || exit 1
|
||||||
|
|
||||||
|
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "9200", "--workers", "1"]
|
||||||
181
apps/fc-ttsreader/speech-align/app.py
Normal file
181
apps/fc-ttsreader/speech-align/app.py
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
"""FlowerCore speech-align service.
|
||||||
|
|
||||||
|
Wraps SYSTRAN/faster-whisper (https://github.com/SYSTRAN/faster-whisper) in a
|
||||||
|
small FastAPI app exposing two endpoints:
|
||||||
|
|
||||||
|
* POST /align — fc-align contract used by FlowerCore.Shared.Speech's
|
||||||
|
FasterWhisperAlignmentClient on master. Multipart form
|
||||||
|
(`audio`, `language`) returns
|
||||||
|
`{text, words: [{word, startSeconds, endSeconds, confidence}],
|
||||||
|
durationMs, language}`.
|
||||||
|
* POST /transcribe — audio-file-in transcription used by the new TtsReader
|
||||||
|
audio-import feature. Multipart form (`audio`, optional
|
||||||
|
`language`) returns `{text, language, durationMs,
|
||||||
|
segments: [{startSeconds, endSeconds, text}]}` so the
|
||||||
|
UI can preview the transcript before piping it into
|
||||||
|
Quick Read or saving as a project.
|
||||||
|
|
||||||
|
Both endpoints share the same WhisperModel instance (loaded once at startup).
|
||||||
|
Model is pinned by the WHISPER_MODEL env var (defaults to base.en) and cached
|
||||||
|
under WHISPER_CACHE_DIR (defaults to /models, backed by a PVC in K8s).
|
||||||
|
|
||||||
|
Health: GET /health → {status: ok, model, device, computeType}.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
|
||||||
|
from fastapi.responses import JSONResponse
|
||||||
|
from faster_whisper import WhisperModel
|
||||||
|
|
||||||
|
LOG = logging.getLogger("speech_align")
|
||||||
|
logging.basicConfig(
|
||||||
|
level=os.environ.get("LOG_LEVEL", "INFO"),
|
||||||
|
format="%(asctime)s %(levelname)s %(name)s %(message)s",
|
||||||
|
)
|
||||||
|
|
||||||
|
MODEL_NAME = os.environ.get("WHISPER_MODEL", "Systran/faster-whisper-base.en")
|
||||||
|
DEVICE = os.environ.get("WHISPER_DEVICE", "cpu")
|
||||||
|
COMPUTE_TYPE = os.environ.get("WHISPER_COMPUTE_TYPE", "int8")
|
||||||
|
CACHE_DIR = os.environ.get("WHISPER_CACHE_DIR", "/models")
|
||||||
|
MAX_BYTES = int(os.environ.get("MAX_AUDIO_BYTES", str(50 * 1024 * 1024))) # 50 MB
|
||||||
|
DEFAULT_LANGUAGE = os.environ.get("DEFAULT_LANGUAGE", "en")
|
||||||
|
|
||||||
|
_state: dict[str, object] = {}
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(_app: FastAPI):
|
||||||
|
LOG.info("Loading faster-whisper model %s (device=%s compute=%s cache=%s)", MODEL_NAME, DEVICE, COMPUTE_TYPE, CACHE_DIR)
|
||||||
|
started = time.time()
|
||||||
|
model = WhisperModel(MODEL_NAME, device=DEVICE, compute_type=COMPUTE_TYPE, download_root=CACHE_DIR)
|
||||||
|
_state["model"] = model
|
||||||
|
LOG.info("Model loaded in %.2fs", time.time() - started)
|
||||||
|
yield
|
||||||
|
_state.clear()
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(title="FlowerCore speech-align", version="1.0.0", lifespan=lifespan)
|
||||||
|
|
||||||
|
|
||||||
|
def _get_model() -> WhisperModel:
|
||||||
|
model = _state.get("model")
|
||||||
|
if model is None:
|
||||||
|
raise HTTPException(status_code=503, detail="Model not loaded yet")
|
||||||
|
return model # type: ignore[return-value]
|
||||||
|
|
||||||
|
|
||||||
|
async def _read_upload(upload: UploadFile) -> bytes:
|
||||||
|
payload = await upload.read()
|
||||||
|
if not payload:
|
||||||
|
raise HTTPException(status_code=400, detail="audio is empty")
|
||||||
|
if len(payload) > MAX_BYTES:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=413,
|
||||||
|
detail=f"audio exceeds {MAX_BYTES} byte limit ({len(payload)} bytes received)",
|
||||||
|
)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_language(value: Optional[str]) -> Optional[str]:
|
||||||
|
if not value or not value.strip():
|
||||||
|
return DEFAULT_LANGUAGE
|
||||||
|
return value.strip().lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _transcribe_bytes(audio_bytes: bytes, language: Optional[str], word_timestamps: bool):
|
||||||
|
model = _get_model()
|
||||||
|
started = time.time()
|
||||||
|
segments_iter, info = model.transcribe(
|
||||||
|
io.BytesIO(audio_bytes),
|
||||||
|
language=language,
|
||||||
|
word_timestamps=word_timestamps,
|
||||||
|
beam_size=1,
|
||||||
|
vad_filter=True,
|
||||||
|
)
|
||||||
|
segments = list(segments_iter)
|
||||||
|
elapsed_ms = int((time.time() - started) * 1000)
|
||||||
|
return segments, info, elapsed_ms
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health():
|
||||||
|
return {
|
||||||
|
"status": "ok" if _state.get("model") is not None else "loading",
|
||||||
|
"model": MODEL_NAME,
|
||||||
|
"device": DEVICE,
|
||||||
|
"computeType": COMPUTE_TYPE,
|
||||||
|
"defaultLanguage": DEFAULT_LANGUAGE,
|
||||||
|
"maxBytes": MAX_BYTES,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/align")
|
||||||
|
async def align(audio: UploadFile = File(...), language: str = Form(DEFAULT_LANGUAGE)):
|
||||||
|
"""fc-align contract — used by FlowerCore.Shared.Speech.FasterWhisperAlignmentClient."""
|
||||||
|
payload = await _read_upload(audio)
|
||||||
|
lang = _normalize_language(language)
|
||||||
|
segments, info, elapsed_ms = _transcribe_bytes(payload, lang, word_timestamps=True)
|
||||||
|
|
||||||
|
text_parts: list[str] = []
|
||||||
|
words: list[dict] = []
|
||||||
|
for segment in segments:
|
||||||
|
text_parts.append(segment.text.strip())
|
||||||
|
for word in (segment.words or []):
|
||||||
|
# Field names MUST match the FlowerCore.Shared.Speech contract:
|
||||||
|
# `text` / `startMs` / `endMs`. The deployed FasterWhisperAlignmentClient
|
||||||
|
# ignores any other names — see Common's
|
||||||
|
# FasterWhisperAlignmentResponse / FasterWhisperWord.
|
||||||
|
words.append({
|
||||||
|
"text": word.word.strip(),
|
||||||
|
"startMs": int((word.start or 0.0) * 1000),
|
||||||
|
"endMs": int((word.end or 0.0) * 1000),
|
||||||
|
# Confidence is informational and ignored by the C# client today,
|
||||||
|
# but kept on the wire for future scoring + fc-align operators
|
||||||
|
# that want to surface low-confidence words.
|
||||||
|
"confidence": float(getattr(word, "probability", 0.0) or 0.0),
|
||||||
|
})
|
||||||
|
|
||||||
|
duration_ms = int((info.duration or 0.0) * 1000)
|
||||||
|
return JSONResponse({
|
||||||
|
"text": " ".join(p for p in text_parts if p).strip(),
|
||||||
|
"words": words,
|
||||||
|
"durationMs": duration_ms,
|
||||||
|
"language": info.language or lang,
|
||||||
|
"elapsedMs": elapsed_ms,
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/transcribe")
|
||||||
|
async def transcribe(audio: UploadFile = File(...), language: Optional[str] = Form(None)):
|
||||||
|
"""Audio-in transcription contract — used by the new TtsReader audio-import feature.
|
||||||
|
|
||||||
|
Returns full segments (no per-word timestamps) so the UI can preview the
|
||||||
|
transcript before piping it into Quick Read or saving as a project.
|
||||||
|
"""
|
||||||
|
payload = await _read_upload(audio)
|
||||||
|
lang = _normalize_language(language)
|
||||||
|
segments, info, elapsed_ms = _transcribe_bytes(payload, lang, word_timestamps=False)
|
||||||
|
|
||||||
|
out_segments = [
|
||||||
|
{
|
||||||
|
"startSeconds": float(segment.start or 0.0),
|
||||||
|
"endSeconds": float(segment.end or 0.0),
|
||||||
|
"text": segment.text.strip(),
|
||||||
|
}
|
||||||
|
for segment in segments
|
||||||
|
]
|
||||||
|
|
||||||
|
return JSONResponse({
|
||||||
|
"text": " ".join(s["text"] for s in out_segments if s["text"]).strip(),
|
||||||
|
"segments": out_segments,
|
||||||
|
"language": info.language or lang,
|
||||||
|
"durationMs": int((info.duration or 0.0) * 1000),
|
||||||
|
"elapsedMs": elapsed_ms,
|
||||||
|
})
|
||||||
8
apps/fc-ttsreader/speech-align/requirements.txt
Normal file
8
apps/fc-ttsreader/speech-align/requirements.txt
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
faster-whisper==1.0.3
|
||||||
|
fastapi==0.115.0
|
||||||
|
uvicorn[standard]==0.30.6
|
||||||
|
python-multipart==0.0.10
|
||||||
|
# faster-whisper 1.0.3's utils module imports requests but doesn't pin it as a
|
||||||
|
# transitive dep — pin explicitly so the image isn't relying on whatever
|
||||||
|
# happens to be in the base image.
|
||||||
|
requests==2.32.3
|
||||||
File diff suppressed because one or more lines are too long
@@ -16,6 +16,15 @@ spec:
|
|||||||
requests:
|
requests:
|
||||||
storage: 1Gi
|
storage: 1Gi
|
||||||
---
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: intranet-config
|
||||||
|
namespace: intranet
|
||||||
|
data:
|
||||||
|
KnowledgeApiKey: ""
|
||||||
|
TrustedHeaderSharedSecret: ""
|
||||||
|
---
|
||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
metadata:
|
metadata:
|
||||||
@@ -37,7 +46,7 @@ spec:
|
|||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: intranet-web
|
- name: intranet-web
|
||||||
image: localhost/fc-intranet-web:v202604240050corpus
|
image: localhost/fc-intranet-web:v20260506-2120
|
||||||
imagePullPolicy: Never
|
imagePullPolicy: Never
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 5300
|
- containerPort: 5300
|
||||||
@@ -47,6 +56,32 @@ spec:
|
|||||||
value: Production
|
value: Production
|
||||||
- name: ASPNETCORE_URLS
|
- name: ASPNETCORE_URLS
|
||||||
value: "http://+:5300"
|
value: "http://+:5300"
|
||||||
|
# Bulk corpus indexing on edge1 Pi 5 takes ~6s/chunk × 5665 chunks
|
||||||
|
# ≈ 9 hours. BLUEJAY-WS GPU (R9700, 32GB VRAM) does the same work
|
||||||
|
# in minutes. Memory: feedback_pi5_nomic_embed_slow.
|
||||||
|
- name: IntranetSearch__OllamaBaseUrl
|
||||||
|
value: "http://10.0.56.20:11434"
|
||||||
|
# Sprint E Phase 2α — JSON-file-backed PageReadingOverride persistence
|
||||||
|
# on the writable PVC at /data. Without this env var the
|
||||||
|
# intranet falls back to the in-memory store (loses state on
|
||||||
|
# pod restart). Master's PageReadingOverrideOptions binds
|
||||||
|
# PageReadingOverrides:FilePath.
|
||||||
|
- name: PageReadingOverrides__FilePath
|
||||||
|
value: "/data/page-reading-overrides.json"
|
||||||
|
- name: KnowledgeFleetSearch__BaseUrl
|
||||||
|
value: "https://knowledge.iamworkin.lan"
|
||||||
|
- name: KnowledgeFleetSearch__ApiKey
|
||||||
|
valueFrom:
|
||||||
|
configMapKeyRef:
|
||||||
|
name: intranet-config
|
||||||
|
key: KnowledgeApiKey
|
||||||
|
optional: true
|
||||||
|
- name: TrustedHeaderAuthentication__SharedSecret
|
||||||
|
valueFrom:
|
||||||
|
configMapKeyRef:
|
||||||
|
name: intranet-config
|
||||||
|
key: TrustedHeaderSharedSecret
|
||||||
|
optional: true
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
memory: "256Mi"
|
memory: "256Mi"
|
||||||
|
|||||||
165
apps/knowledge/README.md
Normal file
165
apps/knowledge/README.md
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
# knowledge — FlowerCore.Knowledge.Web (Phase 2.4 K8s deploy)
|
||||||
|
|
||||||
|
**Status:** **LIVE 2026-04-27** at `https://knowledge.iamworkin.lan` —
|
||||||
|
Phase 2.4 closed. Pod running, certificate issued (step-ca-acme), PVC
|
||||||
|
bound (Longhorn 20Gi RWO), ArgoCD `infra-knowledge` synced. `/healthz`
|
||||||
|
returns 200, `/api/v1/editions` returns `[]` (initial-deploy state — no
|
||||||
|
*.db files in the PVC yet; Phase 2.5+ admin UI handles bulk
|
||||||
|
population). Phase 1 of the Agent Zero MCP rollout keeps `/healthz`
|
||||||
|
anonymous and gates `/mcp` behind `Authorization: Bearer <token>` built
|
||||||
|
from the 1Password item `FlowerCore Knowledge MCP Tokens`.
|
||||||
|
|
||||||
|
- Plan: [`../../../FlowerCore.Notes/docs/ai-agents/flowercore-knowledge-service-plan.md`](../../../FlowerCore.Notes/docs/ai-agents/flowercore-knowledge-service-plan.md)
|
||||||
|
- Sprint: [`../../../FlowerCore.Notes/docs/ai-station/sprint-e-xxl-plan.md`](../../../FlowerCore.Notes/docs/ai-station/sprint-e-xxl-plan.md) (Track B)
|
||||||
|
- Repo: `D:\git\FlowerCore\FlowerCore.Knowledge\` (private GitHub repo,
|
||||||
|
bootstrapped Sprint D batch 35)
|
||||||
|
|
||||||
|
`FlowerCore.Knowledge.Web` is the fleet-wide vector-indexing & RAG hub —
|
||||||
|
a REST + MCP service that scans `*.db` files under
|
||||||
|
`/data/vector-stores` and exposes per-edition reachability + corpus
|
||||||
|
search to the rest of the FC ecosystem (Agent Zero, Chat.Web persona
|
||||||
|
memory, AiStation embeddings explorer, TtsReader chapter context, BMO
|
||||||
|
bot, Pi nodes via `fc-index sync`).
|
||||||
|
|
||||||
|
Phase 1 MCP routing is explicit:
|
||||||
|
|
||||||
|
- in-cluster Agent Zero → `http://knowledge-web.knowledge.svc/mcp`
|
||||||
|
- workstation Agent Zero → `https://knowledge.iamworkin.lan/mcp`
|
||||||
|
- probe URL for both lanes → `/healthz`
|
||||||
|
|
||||||
|
## Deployment order (do NOT skip / reorder)
|
||||||
|
|
||||||
|
### 1. FlowerCore.DNS public A record — knowledge.iamworkin.lan -> 10.0.56.200
|
||||||
|
|
||||||
|
Required BEFORE the Certificate resource is created, or cert-manager
|
||||||
|
HTTP-01 silently backs off ~2h. Memory: `feedback_pfsense_dns_required_for_acme`.
|
||||||
|
|
||||||
|
The canonical path is FlowerCore.DNS:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sk https://dns.iamworkin.lan/api/v1/servers
|
||||||
|
# Find the pfSense serverId, then create the record using the host label only.
|
||||||
|
|
||||||
|
curl -sk -X POST https://dns.iamworkin.lan/api/v1/servers/<serverId>/zones/iamworkin.lan/records \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"name":"knowledge","type":"A","data":"10.0.56.200","ttl":300}'
|
||||||
|
```
|
||||||
|
|
||||||
|
If FlowerCore.DNS provider writes are failing 502 with "pfSense
|
||||||
|
diag_command.php response did not contain a `<pre>` block" (status as of
|
||||||
|
Sprint E Track B authoring 2026-04-27), add the override manually via
|
||||||
|
the pfSense web UI:
|
||||||
|
|
||||||
|
1. Log in to `https://10.0.56.1` as admin
|
||||||
|
2. Services → DNS Resolver → General Settings → Host Overrides
|
||||||
|
3. Add: Host=`knowledge`, Domain=`iamworkin.lan`, IP Address=`10.0.56.200`
|
||||||
|
4. Save + Apply Changes
|
||||||
|
|
||||||
|
Verify resolution from anywhere on LAN:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nslookup knowledge.iamworkin.lan 10.0.56.1
|
||||||
|
# Expect: 10.0.56.200
|
||||||
|
```
|
||||||
|
|
||||||
|
Or against FlowerCore.DNS once the provider is fixed:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sk "https://dns.iamworkin.lan/api/v1/zones/iamworkin.lan/resolve-preflight?hostname=knowledge.iamworkin.lan"
|
||||||
|
# Expect: "resolvable": true
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Build + import the image to ALL RKE2 nodes
|
||||||
|
|
||||||
|
Pods may schedule on any RKE2 worker (server, agent1, agent2). The
|
||||||
|
Longhorn PVC accepts mounts from any node, so the image must be
|
||||||
|
imported to all three. Memory:
|
||||||
|
`feedback_rke2_image_import_targets_all_nodes` +
|
||||||
|
`feedback_rke2_localhost_imagepullpolicy`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From BLUEJAY-WS, in D:\git\FlowerCore\FlowerCore.Knowledge
|
||||||
|
TAG="v$(date +%Y%m%d%H%M)"
|
||||||
|
dotnet.exe publish -c Release -o deploy/app \
|
||||||
|
src/FlowerCore.Knowledge.Web/FlowerCore.Knowledge.Web.csproj
|
||||||
|
podman build -t localhost/fc-knowledge-web:$TAG -f deploy/Dockerfile.deploy deploy
|
||||||
|
podman save localhost/fc-knowledge-web:$TAG -o /tmp/fc-knowledge-web.tar
|
||||||
|
|
||||||
|
# Import to all three RKE2 nodes
|
||||||
|
for node in rke2-server rke2-agent1 rke2-agent2; do
|
||||||
|
scp /tmp/fc-knowledge-web.tar $node:/tmp/
|
||||||
|
ssh $node "sudo /var/lib/rancher/rke2/bin/ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images import /tmp/fc-knowledge-web.tar"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
The repo's `scripts/deploy-knowledge.sh` automates this loop.
|
||||||
|
|
||||||
|
### 3. Bump the image tag + push
|
||||||
|
|
||||||
|
Edit `knowledge.yaml`, replace `localhost/fc-knowledge-web:v202604272200`
|
||||||
|
with the tag from step 2, then:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd D:/git/FlowerCore/bluejay-infra
|
||||||
|
python scripts/check-pfsense-dns.py # confirms the DNS preflight
|
||||||
|
git add apps/knowledge/
|
||||||
|
git commit -m "feat(knowledge): deploy Phase 2.4 K8s manifest"
|
||||||
|
git push
|
||||||
|
```
|
||||||
|
|
||||||
|
ArgoCD picks up within ~3 minutes and creates `infra-knowledge`.
|
||||||
|
|
||||||
|
### 4. Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
fcadmin_ssh noc1 '
|
||||||
|
kubectl -n argocd get application infra-knowledge
|
||||||
|
kubectl -n knowledge get certificate,pod,pvc
|
||||||
|
curl -sk -m 8 -o /dev/null -w "HTTP %{http_code}\n" https://knowledge.iamworkin.lan/healthz
|
||||||
|
curl -sk -m 8 https://knowledge.iamworkin.lan/api/v1/editions | jq
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect: Certificate `Ready: True` within ~60s, `/healthz` HTTP 200,
|
||||||
|
`/api/v1/editions` returns an empty array (no DBs in the PVC yet) on
|
||||||
|
first deploy.
|
||||||
|
|
||||||
|
## Initial-deploy state and Phase 2.5 follow-up
|
||||||
|
|
||||||
|
The Longhorn PVC is empty on first deploy. Knowledge.Web's filesystem
|
||||||
|
catalog will report zero editions until vector-store `*.db` files are
|
||||||
|
pushed into `/data/vector-stores`. Initial population is a follow-up
|
||||||
|
step (Phase 2.5+, Blazor admin UI's "Rebuild" button); for the first
|
||||||
|
deploy the goal is just to prove the pod boots, `/healthz` returns 200,
|
||||||
|
and the Traefik IngressRoute serves the Scalar UI.
|
||||||
|
|
||||||
|
To copy an existing local DB into the PVC (one-time, manual until
|
||||||
|
Phase 2.5 admin UI lands):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
fcadmin_ssh noc1 '
|
||||||
|
POD=$(kubectl -n knowledge get pod -l app=knowledge-web -o jsonpath="{.items[0].metadata.name}")
|
||||||
|
kubectl -n knowledge cp /var/lib/flowercore/vector-stores/bluejay-ai.db $POD:/data/vector-stores/bluejay-ai.db
|
||||||
|
'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Probes + middleware notes
|
||||||
|
|
||||||
|
- `/healthz` is mapped by `Controllers/HealthController.cs` (controller-based
|
||||||
|
attribute route). Cheap — no DB, no dependencies.
|
||||||
|
- Liveness uses `tcpSocket` as a defensive fallback in case future
|
||||||
|
middleware accidentally gates `/healthz` behind auth (memory:
|
||||||
|
`feedback_k8s_probes_behind_auth_middleware`).
|
||||||
|
- `/openapi/v1.json` and `/scalar/v1` are wired by `UseFlowerCoreApi`.
|
||||||
|
Per memory `feedback_k8s_probes_must_not_hit_openapi`, probes must NOT
|
||||||
|
point at OpenAPI documents — the `MapOpenApi` call can be slow during
|
||||||
|
cold startup.
|
||||||
|
|
||||||
|
## Resource sizing
|
||||||
|
|
||||||
|
- 256Mi memory request / 1Gi limit.
|
||||||
|
- 100m CPU request / 1000m limit.
|
||||||
|
- 20Gi Longhorn PVC initial — sufficient for the bluejay-ai 1.94Gi DB +
|
||||||
|
fleet-pi-edge 352Mi + fleet-bmo-bot 141Mi + headroom. Resize via
|
||||||
|
`kubectl -n knowledge edit pvc knowledge-vector-store` if growing
|
||||||
|
past 15Gi.
|
||||||
262
apps/knowledge/knowledge.yaml
Normal file
262
apps/knowledge/knowledge.yaml
Normal file
@@ -0,0 +1,262 @@
|
|||||||
|
# FlowerCore.Knowledge.Web — fleet vector indexing & RAG hub.
|
||||||
|
#
|
||||||
|
# Phase 2.4 of the Knowledge service plan. REST + MCP service that scans
|
||||||
|
# *.db files under /data/vector-stores and exposes:
|
||||||
|
# - REST: /api/v1/editions, /api/v1/corpus/search, /healthz
|
||||||
|
# - MCP: list_editions, describe_edition, corpus_search
|
||||||
|
# - Static OpenAPI/Scalar via UseFlowerCoreApi
|
||||||
|
#
|
||||||
|
# Architecture:
|
||||||
|
# Plan: FlowerCore.Notes/docs/ai-agents/flowercore-knowledge-service-plan.md
|
||||||
|
# Sprint: FlowerCore.Notes/docs/ai-station/sprint-e-xxl-plan.md (Track B)
|
||||||
|
# Repo: D:\git\FlowerCore\FlowerCore.Knowledge\
|
||||||
|
# Shared: FlowerCore.Common -> FlowerCore.Shared.Indexing (chunkers, vector
|
||||||
|
# stores, edition profiles, ICorpusSearchService facade)
|
||||||
|
#
|
||||||
|
# Deployment order (see apps/knowledge/README.md and the bluejay-infra/README.md
|
||||||
|
# top-level checklist):
|
||||||
|
# 1. FlowerCore.DNS public A record knowledge.iamworkin.lan -> 10.0.56.200
|
||||||
|
# MUST exist BEFORE the Certificate is created, or cert-manager HTTP-01
|
||||||
|
# backs off ~2h. Memory: feedback_pfsense_dns_required_for_acme.
|
||||||
|
# 2. Build + import the image to ALL RKE2 nodes (server + both agents) since
|
||||||
|
# the Pod uses a Longhorn PVC and may schedule anywhere.
|
||||||
|
# Memory: feedback_rke2_localhost_imagepullpolicy.
|
||||||
|
# 3. Bump the image tag in this file, git push.
|
||||||
|
# 4. ArgoCD ApplicationSet picks up within ~3 minutes and creates
|
||||||
|
# infra-knowledge.
|
||||||
|
#
|
||||||
|
# Initial-deploy state:
|
||||||
|
# The Longhorn PVC is empty on first deploy. Knowledge.Web's filesystem
|
||||||
|
# catalog will report zero editions until vector-store *.db files are
|
||||||
|
# pushed into /data/vector-stores. Initial population is a follow-up step
|
||||||
|
# (Phase 2.5+, Blazor admin UI's "Rebuild" button); for the first deploy
|
||||||
|
# the goal is just to prove the pod boots, /healthz returns 200, and the
|
||||||
|
# Traefik IngressRoute serves the Scalar UI.
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: knowledge
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: bluejay-infra
|
||||||
|
---
|
||||||
|
# MCP bearer token for the read-only Agent Zero Phase 1 lane. The 1Password
|
||||||
|
# item currently stores the raw token in its concealed PASSWORD field, which
|
||||||
|
# the operator syncs into the namespaced Secret key `password`.
|
||||||
|
apiVersion: onepassword.com/v1
|
||||||
|
kind: OnePasswordItem
|
||||||
|
metadata:
|
||||||
|
name: knowledge-mcp-tokens
|
||||||
|
namespace: knowledge
|
||||||
|
spec:
|
||||||
|
itemPath: "vaults/IAmWorkin/items/FlowerCore Knowledge MCP Tokens"
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: knowledge-vector-store
|
||||||
|
namespace: knowledge
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
storageClassName: longhorn
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 20Gi
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: knowledge-web
|
||||||
|
namespace: knowledge
|
||||||
|
labels:
|
||||||
|
app: knowledge-web
|
||||||
|
app.kubernetes.io/name: knowledge-web
|
||||||
|
app.kubernetes.io/part-of: bluejay-infra
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
revisionHistoryLimit: 3
|
||||||
|
# RWO Longhorn PVC blocks rolling updates (multi-attach error). Recreate
|
||||||
|
# is the canonical pattern (memory: feedback_rwo_pvc_blocks_rolling).
|
||||||
|
strategy:
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: knowledge-web
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: knowledge-web
|
||||||
|
app.kubernetes.io/name: knowledge-web
|
||||||
|
app.kubernetes.io/part-of: bluejay-infra
|
||||||
|
annotations:
|
||||||
|
prometheus.io/scrape: "true"
|
||||||
|
prometheus.io/port: "8080"
|
||||||
|
prometheus.io/path: "/metrics"
|
||||||
|
spec:
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
fsGroup: 1654
|
||||||
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
containers:
|
||||||
|
- name: web
|
||||||
|
# Placeholder tag — bump to the image you built + imported to ALL
|
||||||
|
# RKE2 nodes via scripts/deploy-knowledge.sh before applying.
|
||||||
|
image: localhost/fc-knowledge-web:v20260429232635
|
||||||
|
imagePullPolicy: Never
|
||||||
|
command:
|
||||||
|
- /bin/sh
|
||||||
|
- -c
|
||||||
|
args:
|
||||||
|
- |
|
||||||
|
if [ -n "${KNOWLEDGE_MCP_BEARER_TOKEN:-}" ]; then
|
||||||
|
export FlowerCore__Mcp__ApiKey__Key="Bearer ${KNOWLEDGE_MCP_BEARER_TOKEN}"
|
||||||
|
fi
|
||||||
|
exec dotnet FlowerCore.Knowledge.Web.dll
|
||||||
|
ports:
|
||||||
|
- containerPort: 8080
|
||||||
|
name: http
|
||||||
|
env:
|
||||||
|
- name: ASPNETCORE_URLS
|
||||||
|
value: "http://+:8080"
|
||||||
|
- name: ASPNETCORE_ENVIRONMENT
|
||||||
|
value: "Production"
|
||||||
|
- name: DOTNET_SYSTEM_GLOBALIZATION_INVARIANT
|
||||||
|
value: "false"
|
||||||
|
# Vector-store directory + embedding model + edition profile dir.
|
||||||
|
# Profile JSON is baked into the image at /home/app/editions via the
|
||||||
|
# csproj Content-link from FlowerCore.Common/editions/.
|
||||||
|
- name: Knowledge__VectorStoresDirectory
|
||||||
|
value: "/data/vector-stores"
|
||||||
|
- name: Knowledge__EmbeddingModel
|
||||||
|
value: "nomic-embed-text"
|
||||||
|
- name: Knowledge__DefaultLimit
|
||||||
|
value: "5"
|
||||||
|
- name: Knowledge__MaxLimit
|
||||||
|
value: "50"
|
||||||
|
- name: FlowerCore__Editions__ProfileDirectory
|
||||||
|
value: "/home/app/editions"
|
||||||
|
# Embed via edge1 Pi 5 + AI HAT+ (10.0.57.17:11434). Cluster
|
||||||
|
# services do not depend on BLUEJAY-WS (private dev hardware) per
|
||||||
|
# bluejay-infra@0f9d56e. Query-time embedding is fast enough on
|
||||||
|
# edge1 (~ms per query); bulk index rebuilds (Phase 2.5+) will
|
||||||
|
# need a separate ingestion lane that can opt into the
|
||||||
|
# workstation GPU when present.
|
||||||
|
- name: FlowerCore__Ollama__BaseUrl
|
||||||
|
value: "http://10.0.57.17:11434"
|
||||||
|
- name: FlowerCore__Mcp__ApiKey__Key
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: knowledge-mcp-tokens
|
||||||
|
key: password
|
||||||
|
- name: FlowerCore__Mcp__ApiKey__HeaderName
|
||||||
|
value: "Authorization"
|
||||||
|
- name: KNOWLEDGE_MCP_BEARER_TOKEN
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: knowledge-mcp-tokens
|
||||||
|
key: password
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: 1000m
|
||||||
|
memory: 1Gi
|
||||||
|
# /healthz is mapped by HealthController (controller-based route).
|
||||||
|
# tcpSocket liveness is the defensive fallback in case middleware
|
||||||
|
# later gates /healthz behind auth (memory:
|
||||||
|
# feedback_k8s_probes_behind_auth_middleware).
|
||||||
|
startupProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /healthz
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
failureThreshold: 30
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /healthz
|
||||||
|
port: 8080
|
||||||
|
periodSeconds: 10
|
||||||
|
failureThreshold: 3
|
||||||
|
livenessProbe:
|
||||||
|
tcpSocket:
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 30
|
||||||
|
failureThreshold: 3
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1654
|
||||||
|
runAsGroup: 1654
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
volumeMounts:
|
||||||
|
- name: vector-store
|
||||||
|
mountPath: /data/vector-stores
|
||||||
|
- name: tmp
|
||||||
|
mountPath: /tmp
|
||||||
|
- name: logs
|
||||||
|
mountPath: /home/app/logs
|
||||||
|
volumes:
|
||||||
|
- name: vector-store
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: knowledge-vector-store
|
||||||
|
- name: tmp
|
||||||
|
emptyDir: {}
|
||||||
|
- name: logs
|
||||||
|
emptyDir: {}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: knowledge-web
|
||||||
|
namespace: knowledge
|
||||||
|
labels:
|
||||||
|
app: knowledge-web
|
||||||
|
app.kubernetes.io/name: knowledge-web
|
||||||
|
app.kubernetes.io/part-of: bluejay-infra
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app: knowledge-web
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 80
|
||||||
|
targetPort: 8080
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: knowledge-tls
|
||||||
|
namespace: knowledge
|
||||||
|
spec:
|
||||||
|
secretName: knowledge-tls
|
||||||
|
issuerRef:
|
||||||
|
name: step-ca-acme
|
||||||
|
kind: ClusterIssuer
|
||||||
|
dnsNames:
|
||||||
|
- knowledge.iamworkin.lan
|
||||||
|
duration: 2160h # 90d
|
||||||
|
renewBefore: 720h # 30d
|
||||||
|
---
|
||||||
|
apiVersion: traefik.io/v1alpha1
|
||||||
|
kind: IngressRoute
|
||||||
|
metadata:
|
||||||
|
name: knowledge
|
||||||
|
namespace: knowledge
|
||||||
|
spec:
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
routes:
|
||||||
|
- match: Host(`knowledge.iamworkin.lan`)
|
||||||
|
kind: Rule
|
||||||
|
services:
|
||||||
|
- name: knowledge-web
|
||||||
|
port: 80
|
||||||
|
tls:
|
||||||
|
secretName: knowledge-tls
|
||||||
7
apps/knowledge/kustomization.yaml
Normal file
7
apps/knowledge/kustomization.yaml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
# ArgoCD's bluejay-infra ApplicationSet uses a directory generator and does
|
||||||
|
# not require kustomization.yaml. Mirrors the fc-distribution shape so
|
||||||
|
# `kubectl kustomize` previews work from a working copy.
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
resources:
|
||||||
|
- knowledge.yaml
|
||||||
762
apps/monitoring/fc-updatecenter-dashboard.json
Normal file
762
apps/monitoring/fc-updatecenter-dashboard.json
Normal file
@@ -0,0 +1,762 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 1,
|
||||||
|
"id": null,
|
||||||
|
"links": [
|
||||||
|
{
|
||||||
|
"icon": "external link",
|
||||||
|
"includeVars": false,
|
||||||
|
"keepTime": false,
|
||||||
|
"targetBlank": true,
|
||||||
|
"title": "Open Service",
|
||||||
|
"type": "link",
|
||||||
|
"url": "https://updatecenter.iamworkin.lan/"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
},
|
||||||
|
"mappings": [
|
||||||
|
{
|
||||||
|
"options": {
|
||||||
|
"0": {
|
||||||
|
"color": "#f87171",
|
||||||
|
"index": 1,
|
||||||
|
"text": "DOWN"
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"color": "#4ade80",
|
||||||
|
"index": 0,
|
||||||
|
"text": "UP"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"type": "value"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "#f87171",
|
||||||
|
"value": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#4ade80",
|
||||||
|
"value": 1
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 4,
|
||||||
|
"w": 8,
|
||||||
|
"x": 0,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 1,
|
||||||
|
"options": {
|
||||||
|
"colorMode": "background",
|
||||||
|
"graphMode": "none",
|
||||||
|
"justifyMode": "center",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "value_and_name"
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "probe_success{job=\"probe-traefik-services\",instance=\"updatecenter.iamworkin.lan\"}",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "Availability"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Service Availability",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "stat"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
},
|
||||||
|
"decimals": 2,
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "#f87171",
|
||||||
|
"value": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#fbbf24",
|
||||||
|
"value": 95
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#FFB300",
|
||||||
|
"value": 99
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#4ade80",
|
||||||
|
"value": 99.9
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"unit": "percent"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 4,
|
||||||
|
"w": 8,
|
||||||
|
"x": 8,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 2,
|
||||||
|
"options": {
|
||||||
|
"colorMode": "background_solid",
|
||||||
|
"graphMode": "area",
|
||||||
|
"justifyMode": "center",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "value_and_name"
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "avg_over_time(probe_success{job=\"probe-traefik-services\",instance=\"updatecenter.iamworkin.lan\"}[24h]) * 100",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "24h Uptime"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "24-Hour Uptime",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "stat"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
},
|
||||||
|
"max": 30,
|
||||||
|
"min": 0,
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "#f87171",
|
||||||
|
"value": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#fbbf24",
|
||||||
|
"value": 2
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#4ade80",
|
||||||
|
"value": 7
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"unit": "d"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 4,
|
||||||
|
"w": 8,
|
||||||
|
"x": 16,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 3,
|
||||||
|
"options": {
|
||||||
|
"minVizHeight": 75,
|
||||||
|
"minVizWidth": 75,
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"showThresholdLabels": false,
|
||||||
|
"showThresholdMarkers": true
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "(probe_ssl_earliest_cert_expiry{job=\"probe-traefik-services\",instance=\"updatecenter.iamworkin.lan\"} - time()) / 86400",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "Days Remaining"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Cert Expiry (Days)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "gauge"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"axisBorderShow": false,
|
||||||
|
"axisCenteredZero": false,
|
||||||
|
"axisColorMode": "text",
|
||||||
|
"axisLabel": "Response Time (seconds)",
|
||||||
|
"drawStyle": "line",
|
||||||
|
"fillOpacity": 12,
|
||||||
|
"gradientMode": "scheme",
|
||||||
|
"lineInterpolation": "smooth",
|
||||||
|
"lineWidth": 2,
|
||||||
|
"pointSize": 4,
|
||||||
|
"showPoints": "never",
|
||||||
|
"spanNulls": true,
|
||||||
|
"thresholdsStyle": {
|
||||||
|
"mode": "dashed"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "#4ade80",
|
||||||
|
"value": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#fbbf24",
|
||||||
|
"value": 2
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "#f87171",
|
||||||
|
"value": 5
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"unit": "s"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 14,
|
||||||
|
"x": 0,
|
||||||
|
"y": 4
|
||||||
|
},
|
||||||
|
"id": 4,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull",
|
||||||
|
"mean",
|
||||||
|
"max"
|
||||||
|
],
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right"
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "single",
|
||||||
|
"sort": "none"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "probe_duration_seconds{job=\"probe-traefik-services\",instance=\"updatecenter.iamworkin.lan\"}",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "Probe Duration"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"timeFrom": "1h",
|
||||||
|
"title": "Response Time (1h Trend)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 10,
|
||||||
|
"x": 14,
|
||||||
|
"y": 4
|
||||||
|
},
|
||||||
|
"id": 5,
|
||||||
|
"options": {
|
||||||
|
"alertInstanceLabelFilter": "{instance=\"updatecenter.iamworkin.lan\"}",
|
||||||
|
"alertName": "",
|
||||||
|
"dashboardAlerts": false,
|
||||||
|
"groupBy": [],
|
||||||
|
"groupMode": "default",
|
||||||
|
"maxItems": 10,
|
||||||
|
"sortOrder": 1,
|
||||||
|
"stateFilter": {
|
||||||
|
"error": true,
|
||||||
|
"firing": true,
|
||||||
|
"noData": true,
|
||||||
|
"normal": false,
|
||||||
|
"pending": true
|
||||||
|
},
|
||||||
|
"viewMode": "list"
|
||||||
|
},
|
||||||
|
"title": "Active Alerts",
|
||||||
|
"type": "alertlist"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"collapsed": false,
|
||||||
|
"gridPos": {
|
||||||
|
"h": 1,
|
||||||
|
"w": 24,
|
||||||
|
"x": 0,
|
||||||
|
"y": 12
|
||||||
|
},
|
||||||
|
"id": 20,
|
||||||
|
"title": "OTEL Counters — Track 1D",
|
||||||
|
"type": "row"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"lineWidth": 1,
|
||||||
|
"fillOpacity": 10
|
||||||
|
},
|
||||||
|
"unit": "reqps"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 13
|
||||||
|
},
|
||||||
|
"id": 21,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right",
|
||||||
|
"calcs": ["mean", "lastNotNull"]
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi",
|
||||||
|
"sort": "desc"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "sum by (status) (rate(updatecenter_manifest_requests_total[5m]))",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "status={{status}}"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Manifest Requests rate by status (5m)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"lineWidth": 1,
|
||||||
|
"fillOpacity": 10
|
||||||
|
},
|
||||||
|
"unit": "Bps"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 12,
|
||||||
|
"y": 13
|
||||||
|
},
|
||||||
|
"id": 22,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right",
|
||||||
|
"calcs": ["mean", "lastNotNull"]
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi",
|
||||||
|
"sort": "desc"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "sum by (slug) (rate(updatecenter_bundle_download_bytes_total[5m]))",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "{{slug}}"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Bundle Download Throughput by slug (5m)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"lineWidth": 1,
|
||||||
|
"fillOpacity": 10
|
||||||
|
},
|
||||||
|
"unit": "reqps"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 21
|
||||||
|
},
|
||||||
|
"id": 23,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right",
|
||||||
|
"calcs": ["mean", "lastNotNull"]
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi",
|
||||||
|
"sort": "desc"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "sum by (status) (rate(updatecenter_checkins_total[5m]))",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "status={{status}}"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Agent Check-in Rate by status (5m)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "thresholds"
|
||||||
|
},
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{ "color": "#4ade80", "value": null },
|
||||||
|
{ "color": "#f87171", "value": 1 }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"unit": "none",
|
||||||
|
"decimals": 2
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 6,
|
||||||
|
"x": 12,
|
||||||
|
"y": 21
|
||||||
|
},
|
||||||
|
"id": 24,
|
||||||
|
"options": {
|
||||||
|
"colorMode": "background",
|
||||||
|
"graphMode": "area",
|
||||||
|
"justifyMode": "center",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": ["sum"],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "value_and_name"
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "increase(updatecenter_signature_verify_failures_total[1h])",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "Sig Verify Failures (1h)"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Signature Verify Failures (1h)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "stat"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"lineWidth": 1,
|
||||||
|
"fillOpacity": 10
|
||||||
|
},
|
||||||
|
"unit": "reqps"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 6,
|
||||||
|
"x": 18,
|
||||||
|
"y": 21
|
||||||
|
},
|
||||||
|
"id": 25,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right",
|
||||||
|
"calcs": ["mean", "lastNotNull"]
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi",
|
||||||
|
"sort": "desc"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "sum by (slug, channel) (rate(updatecenter_release_publishes_total[5m]))",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "{{slug}}/{{channel}}"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Release Publishes rate by slug/channel (5m)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"lineWidth": 1,
|
||||||
|
"fillOpacity": 10
|
||||||
|
},
|
||||||
|
"unit": "reqps"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 29
|
||||||
|
},
|
||||||
|
"id": 26,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right",
|
||||||
|
"calcs": ["mean", "lastNotNull"]
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi",
|
||||||
|
"sort": "desc"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "sum by (kind, status) (rate(updatecenter_bundle_downloads_total[5m]))",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "{{kind}} / {{status}}"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Bundle Download Requests by kind/status (5m)",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"custom": {
|
||||||
|
"lineWidth": 2,
|
||||||
|
"fillOpacity": 20
|
||||||
|
},
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{ "color": "#4ade80", "value": null },
|
||||||
|
{ "color": "#f87171", "value": 0.01 }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"unit": "reqps"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 12,
|
||||||
|
"y": 29
|
||||||
|
},
|
||||||
|
"id": 27,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "right",
|
||||||
|
"calcs": ["mean", "lastNotNull"]
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "multi",
|
||||||
|
"sort": "desc"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "fffjikve8llhce"
|
||||||
|
},
|
||||||
|
"expr": "rate(updatecenter_signature_verify_failures_total[5m])",
|
||||||
|
"refId": "A",
|
||||||
|
"legendFormat": "Sig verify failures/s"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Signature Verify Failure Rate (5m) — Critical if >0",
|
||||||
|
"transparent": true,
|
||||||
|
"type": "timeseries"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"refresh": "30s",
|
||||||
|
"schemaVersion": 39,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": [
|
||||||
|
"blue-jay",
|
||||||
|
"flowercore",
|
||||||
|
"synthetic",
|
||||||
|
"updatecenter",
|
||||||
|
"otel"
|
||||||
|
],
|
||||||
|
"templating": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-24h",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timezone": "browser",
|
||||||
|
"title": "FlowerCore.UpdateCenter Dashboard",
|
||||||
|
"uid": "fc-updatecenter",
|
||||||
|
"version": 2
|
||||||
|
}
|
||||||
226
apps/monitoring/flowercore-remotedesktop-grafana-dashboard.json
Normal file
226
apps/monitoring/flowercore-remotedesktop-grafana-dashboard.json
Normal file
@@ -0,0 +1,226 @@
|
|||||||
|
{
|
||||||
|
"annotations": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"editable": true,
|
||||||
|
"fiscalYearStartMonth": 0,
|
||||||
|
"graphTooltip": 0,
|
||||||
|
"id": null,
|
||||||
|
"links": [],
|
||||||
|
"panels": [
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "${DS_PROMETHEUS}"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"unit": "short"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 1,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "bottom"
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "single"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"editorMode": "code",
|
||||||
|
"expr": "sum by (event) (increase(fc_desktop_session_events_total[$__rate_interval]))",
|
||||||
|
"legendFormat": "{{event}}",
|
||||||
|
"range": true,
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "RemoteDesktop Session Events",
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "${DS_PROMETHEUS}"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"unit": "short"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 12,
|
||||||
|
"y": 0
|
||||||
|
},
|
||||||
|
"id": 2,
|
||||||
|
"options": {
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"showUnfilled": true
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"editorMode": "code",
|
||||||
|
"expr": "sum by (template, event) (increase(fc_desktop_session_events_total[24h]))",
|
||||||
|
"legendFormat": "{{template}} {{event}}",
|
||||||
|
"range": true,
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "24h Session Events By Template",
|
||||||
|
"type": "bargauge"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "${DS_PROMETHEUS}"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"unit": "short"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 0,
|
||||||
|
"y": 8
|
||||||
|
},
|
||||||
|
"id": 3,
|
||||||
|
"options": {
|
||||||
|
"legend": {
|
||||||
|
"displayMode": "table",
|
||||||
|
"placement": "bottom"
|
||||||
|
},
|
||||||
|
"tooltip": {
|
||||||
|
"mode": "single"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"editorMode": "code",
|
||||||
|
"expr": "fc_desktop_pool_ready",
|
||||||
|
"legendFormat": "{{template}} ready",
|
||||||
|
"range": true,
|
||||||
|
"refId": "A"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"editorMode": "code",
|
||||||
|
"expr": "fc_desktop_pool_desired",
|
||||||
|
"legendFormat": "{{template}} desired",
|
||||||
|
"range": true,
|
||||||
|
"refId": "B"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "Warm Pool Ready vs Desired",
|
||||||
|
"type": "timeseries"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"datasource": {
|
||||||
|
"type": "prometheus",
|
||||||
|
"uid": "${DS_PROMETHEUS}"
|
||||||
|
},
|
||||||
|
"fieldConfig": {
|
||||||
|
"defaults": {
|
||||||
|
"color": {
|
||||||
|
"mode": "palette-classic"
|
||||||
|
},
|
||||||
|
"mappings": [],
|
||||||
|
"thresholds": {
|
||||||
|
"mode": "absolute",
|
||||||
|
"steps": [
|
||||||
|
{
|
||||||
|
"color": "green",
|
||||||
|
"value": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"color": "orange",
|
||||||
|
"value": 1
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"unit": "short"
|
||||||
|
},
|
||||||
|
"overrides": []
|
||||||
|
},
|
||||||
|
"gridPos": {
|
||||||
|
"h": 8,
|
||||||
|
"w": 12,
|
||||||
|
"x": 12,
|
||||||
|
"y": 8
|
||||||
|
},
|
||||||
|
"id": 4,
|
||||||
|
"options": {
|
||||||
|
"colorMode": "value",
|
||||||
|
"graphMode": "none",
|
||||||
|
"justifyMode": "auto",
|
||||||
|
"orientation": "auto",
|
||||||
|
"reduceOptions": {
|
||||||
|
"calcs": [
|
||||||
|
"lastNotNull"
|
||||||
|
],
|
||||||
|
"fields": "",
|
||||||
|
"values": false
|
||||||
|
},
|
||||||
|
"textMode": "auto"
|
||||||
|
},
|
||||||
|
"targets": [
|
||||||
|
{
|
||||||
|
"editorMode": "code",
|
||||||
|
"expr": "sum(increase(fc_desktop_session_events_total{event=\"connect\",browser_datasource=\"json\"}[24h])) - sum(increase(fc_desktop_session_events_total{event=\"disconnect\"}[24h]))",
|
||||||
|
"range": true,
|
||||||
|
"refId": "A"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"title": "24h Connect Minus Disconnect",
|
||||||
|
"type": "stat"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"refresh": "30s",
|
||||||
|
"schemaVersion": 39,
|
||||||
|
"style": "dark",
|
||||||
|
"tags": [
|
||||||
|
"flowercore",
|
||||||
|
"remotedesktop",
|
||||||
|
"guacamole"
|
||||||
|
],
|
||||||
|
"templating": {
|
||||||
|
"list": []
|
||||||
|
},
|
||||||
|
"time": {
|
||||||
|
"from": "now-24h",
|
||||||
|
"to": "now"
|
||||||
|
},
|
||||||
|
"timezone": "browser",
|
||||||
|
"title": "FlowerCore RemoteDesktop",
|
||||||
|
"uid": "flowercore-remotedesktop",
|
||||||
|
"version": 1
|
||||||
|
}
|
||||||
@@ -104,21 +104,27 @@ data:
|
|||||||
- target_label: __address__
|
- target_label: __address__
|
||||||
replacement: snmp-exporter.monitoring.svc:9116
|
replacement: snmp-exporter.monitoring.svc:9116
|
||||||
|
|
||||||
# UniFi Cloud Key SNMP
|
# UniFi Cloud Key SNMP — DISABLED 2026-04-26
|
||||||
- job_name: "snmp-cloudkey"
|
# The Cloud Key Gen2+ runs unifi-core (controller) only — not a network
|
||||||
static_configs:
|
# device — and does NOT run an SNMP agent on UDP/161. Scrapes were
|
||||||
- targets: ["10.0.56.3"]
|
# silently failing with "connection refused" from 10.42.x.x:161 every
|
||||||
metrics_path: /snmp
|
# 30s, polluting up{} = 0 and lastError on the Targets page. Hardware
|
||||||
params:
|
# health (CPU/mem/disk) for the Cloud Key host should come from
|
||||||
module: [if_mib]
|
# node_exporter via SSH — not SNMP.
|
||||||
auth: [bluejay_v2]
|
# - job_name: "snmp-cloudkey"
|
||||||
relabel_configs:
|
# static_configs:
|
||||||
- source_labels: [__address__]
|
# - targets: ["10.0.56.3"]
|
||||||
target_label: __param_target
|
# metrics_path: /snmp
|
||||||
- source_labels: [__param_target]
|
# params:
|
||||||
target_label: instance
|
# module: [if_mib]
|
||||||
- target_label: __address__
|
# auth: [bluejay_v2]
|
||||||
replacement: snmp-exporter.monitoring.svc:9116
|
# relabel_configs:
|
||||||
|
# - source_labels: [__address__]
|
||||||
|
# target_label: __param_target
|
||||||
|
# - source_labels: [__param_target]
|
||||||
|
# target_label: instance
|
||||||
|
# - target_label: __address__
|
||||||
|
# replacement: snmp-exporter.monitoring.svc:9116
|
||||||
|
|
||||||
# UniFi Switch SNMP
|
# UniFi Switch SNMP
|
||||||
- job_name: "snmp-switch"
|
- job_name: "snmp-switch"
|
||||||
@@ -279,10 +285,13 @@ data:
|
|||||||
replacement: blackbox-exporter.monitoring.svc:9115
|
replacement: blackbox-exporter.monitoring.svc:9115
|
||||||
|
|
||||||
# FlowerCore.RemoteDesktop web health (public cluster VIP)
|
# FlowerCore.RemoteDesktop web health (public cluster VIP)
|
||||||
|
# Module is https_internal — desktop.iamworkin.lan uses a step-ca leaf
|
||||||
|
# cert; blackbox does NOT trust step-ca root, so http_2xx fails with
|
||||||
|
# x509 unknown authority and probe_success=0 even when /health 200s.
|
||||||
- job_name: "probe-remotedesktop"
|
- job_name: "probe-remotedesktop"
|
||||||
metrics_path: /probe
|
metrics_path: /probe
|
||||||
params:
|
params:
|
||||||
module: [http_2xx]
|
module: [https_internal]
|
||||||
scrape_interval: 30s
|
scrape_interval: 30s
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ["https://desktop.iamworkin.lan/health"]
|
- targets: ["https://desktop.iamworkin.lan/health"]
|
||||||
@@ -330,26 +339,12 @@ data:
|
|||||||
# AI Stack Health Probes (Blackbox Exporter)
|
# AI Stack Health Probes (Blackbox Exporter)
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
||||||
# Ollama API — workstation (LOCAL Agent Zero)
|
# NOTE: probe-ollama-local and probe-agentzero-local were REMOVED
|
||||||
- job_name: "probe-ollama-local"
|
# 2026-04-26. They pointed at 10.0.58.100 (HOME VLAN) which is not
|
||||||
metrics_path: /probe
|
# reachable from cluster pods (firewalled). They had been firing as
|
||||||
params:
|
# OllamaDown / AgentZeroDown since 2026-04-24. Workstation/AI-laptop
|
||||||
module: [http_ollama]
|
# Ollama and Agent Zero should be monitored via host-side Puppet
|
||||||
scrape_interval: 30s
|
# (node_exporter on the box) once the AI laptop is running 24/7.
|
||||||
static_configs:
|
|
||||||
- targets: ["http://10.0.58.100:11434/api/tags"]
|
|
||||||
labels:
|
|
||||||
instance: "ollama-local"
|
|
||||||
service: "ollama"
|
|
||||||
deployment: "local"
|
|
||||||
gpu: "r9700"
|
|
||||||
relabel_configs:
|
|
||||||
- source_labels: [__address__]
|
|
||||||
target_label: __param_target
|
|
||||||
- source_labels: [__param_target]
|
|
||||||
target_label: instance
|
|
||||||
- target_label: __address__
|
|
||||||
replacement: blackbox-exporter.monitoring.svc:9115
|
|
||||||
|
|
||||||
# Ollama API — edge1 Pi 5 (NUC Agent Zero)
|
# Ollama API — edge1 Pi 5 (NUC Agent Zero)
|
||||||
- job_name: "probe-ollama-edge1"
|
- job_name: "probe-ollama-edge1"
|
||||||
@@ -372,34 +367,18 @@ data:
|
|||||||
- target_label: __address__
|
- target_label: __address__
|
||||||
replacement: blackbox-exporter.monitoring.svc:9115
|
replacement: blackbox-exporter.monitoring.svc:9115
|
||||||
|
|
||||||
# Agent Zero Web UI — local (K3s)
|
# Agent Zero Web UI — in-cluster (RKE2)
|
||||||
- job_name: "probe-agentzero-local"
|
# Target uses short svc form (agent-zero.agent-zero.svc) NOT
|
||||||
metrics_path: /probe
|
# cluster.local FQDN — the *.cluster.local form gets rewritten to
|
||||||
params:
|
# 10.0.56.200 (Traefik VIP) by the CoreDNS iamworkin.lan template +
|
||||||
module: [http_2xx]
|
# ndots:5 search-suffix expansion. Memory: feedback_coredns_ndots_template_collision.
|
||||||
scrape_interval: 30s
|
|
||||||
static_configs:
|
|
||||||
- targets: ["http://10.0.58.100:30050/"]
|
|
||||||
labels:
|
|
||||||
instance: "agent-zero-local"
|
|
||||||
service: "agent-zero"
|
|
||||||
deployment: "local"
|
|
||||||
relabel_configs:
|
|
||||||
- source_labels: [__address__]
|
|
||||||
target_label: __param_target
|
|
||||||
- source_labels: [__param_target]
|
|
||||||
target_label: instance
|
|
||||||
- target_label: __address__
|
|
||||||
replacement: blackbox-exporter.monitoring.svc:9115
|
|
||||||
|
|
||||||
# Agent Zero Web UI — NUC (RKE2 via Traefik)
|
|
||||||
- job_name: "probe-agentzero-nuc"
|
- job_name: "probe-agentzero-nuc"
|
||||||
metrics_path: /probe
|
metrics_path: /probe
|
||||||
params:
|
params:
|
||||||
module: [http_2xx]
|
module: [http_2xx]
|
||||||
scrape_interval: 30s
|
scrape_interval: 30s
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ["http://agent-zero.agent-zero.svc.cluster.local/"]
|
- targets: ["http://agent-zero.agent-zero.svc:80/"]
|
||||||
labels:
|
labels:
|
||||||
instance: "agent-zero-nuc"
|
instance: "agent-zero-nuc"
|
||||||
service: "agent-zero"
|
service: "agent-zero"
|
||||||
@@ -412,6 +391,119 @@ data:
|
|||||||
- target_label: __address__
|
- target_label: __address__
|
||||||
replacement: blackbox-exporter.monitoring.svc:9115
|
replacement: blackbox-exporter.monitoring.svc:9115
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# K8s Cluster State (kube-state-metrics, cert-manager, traefik)
|
||||||
|
# =============================================================================
|
||||||
|
# Use in-cluster ClusterIP service DNS — NOT NodePorts — so a same-node
|
||||||
|
# NodePort hairpin doesn't break the scrape (hit on rke2-agent1 hosting
|
||||||
|
# both prometheus and traefik on 2026-04-26: 10.0.56.12:30900 timed out
|
||||||
|
# from prometheus while .11/.13 worked). NodePorts at 30900-30902 are
|
||||||
|
# still useful for noc1-Podman-style external scrapers, but in-cluster
|
||||||
|
# we should always use the svc DNS form.
|
||||||
|
|
||||||
|
# kube-state-metrics — exposes K8s object state (pods, deployments, nodes)
|
||||||
|
# Required for KubeContainerRestartingFrequently / KubePodNotReady alerts.
|
||||||
|
- job_name: "kube-state-metrics"
|
||||||
|
scrape_interval: 30s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["kube-state-metrics.kube-system.svc:8080"]
|
||||||
|
labels:
|
||||||
|
cluster: "rke2"
|
||||||
|
|
||||||
|
# cert-manager — exposes certmanager_certificate_ready_status,
|
||||||
|
# certmanager_certificate_expiration_timestamp_seconds, etc. Drives the
|
||||||
|
# CertManagerCertificateNotReady / CertManagerCertificateRenewalFailed
|
||||||
|
# alerts. Memory: project_cert_manager_prometheus_scrape.
|
||||||
|
- job_name: "cert-manager"
|
||||||
|
scrape_interval: 30s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["cert-manager-metrics.cert-manager.svc:9402"]
|
||||||
|
labels:
|
||||||
|
cluster: "rke2"
|
||||||
|
|
||||||
|
# Traefik — request rates, latency, TLS cert metadata, router state.
|
||||||
|
# ClusterIP svc routes to one of the traefik pods; per-pod scrape via
|
||||||
|
# the headless `traefik-metrics` selector would be nicer for failover
|
||||||
|
# visibility but the single-replica scrape is enough for steady-state.
|
||||||
|
- job_name: "traefik"
|
||||||
|
scrape_interval: 15s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["traefik-metrics.traefik-system.svc:9100"]
|
||||||
|
labels:
|
||||||
|
service: "traefik"
|
||||||
|
cluster: "rke2"
|
||||||
|
|
||||||
|
# Longhorn — exposes longhorn_volume_robustness, longhorn_backup_*,
|
||||||
|
# longhorn_node_status_*. Enables LonghornVolumeUnhealthy +
|
||||||
|
# LonghornBackupFailed alerts (no real visibility into Longhorn
|
||||||
|
# health before this — was relying on K8s events which are noisy
|
||||||
|
# transient lifecycle messages, not actionable signals).
|
||||||
|
- job_name: "longhorn"
|
||||||
|
scrape_interval: 30s
|
||||||
|
static_configs:
|
||||||
|
- targets: ["longhorn-backend.longhorn-system.svc:9500"]
|
||||||
|
labels:
|
||||||
|
service: "longhorn"
|
||||||
|
cluster: "rke2"
|
||||||
|
|
||||||
|
# FC web services through Traefik — single probe surface to spot any
|
||||||
|
# iamworkin.lan host returning non-200. Uses https_internal because all
|
||||||
|
# certs are step-ca leaves; blackbox would x509-fail with http_2xx.
|
||||||
|
# Some services need explicit healthcheck paths because root returns
|
||||||
|
# 404 (acme, guac) or 401 (grafana, prometheus). Drop them or point at
|
||||||
|
# the right endpoint — don't lower valid_status_codes globally because
|
||||||
|
# 401 from a healthy pod and 401 from an outage look identical.
|
||||||
|
- job_name: "probe-traefik-services"
|
||||||
|
metrics_path: /probe
|
||||||
|
params:
|
||||||
|
module: [https_internal]
|
||||||
|
scrape_interval: 60s
|
||||||
|
static_configs:
|
||||||
|
- targets:
|
||||||
|
# Root-reachable services (200 or 3xx)
|
||||||
|
- "https://gitea.iamworkin.lan/"
|
||||||
|
- "https://argocd.iamworkin.lan/"
|
||||||
|
- "https://intranet.iamworkin.lan/"
|
||||||
|
- "https://signage.iamworkin.lan/"
|
||||||
|
- "https://kiosk.iamworkin.lan/"
|
||||||
|
- "https://media.iamworkin.lan/"
|
||||||
|
- "https://mysql.iamworkin.lan/"
|
||||||
|
- "https://php.iamworkin.lan/"
|
||||||
|
- "https://zabbix.iamworkin.lan/"
|
||||||
|
- "https://desktop.iamworkin.lan/"
|
||||||
|
- "https://print.iamworkin.lan/"
|
||||||
|
- "https://dns.iamworkin.lan/"
|
||||||
|
- "https://chat.iamworkin.lan/"
|
||||||
|
- "https://dist.iamworkin.lan/"
|
||||||
|
- "https://dms.iamworkin.lan/"
|
||||||
|
- "https://menuboard.iamworkin.lan/"
|
||||||
|
- "https://messageboard.iamworkin.lan/"
|
||||||
|
- "https://presentations.iamworkin.lan/"
|
||||||
|
- "https://retail.iamworkin.lan/"
|
||||||
|
- "https://ttsreader.iamworkin.lan/"
|
||||||
|
# Explicit healthcheck paths
|
||||||
|
- "https://fc-llm-bridge.iamworkin.lan/healthz"
|
||||||
|
- "https://acme.iamworkin.lan/health"
|
||||||
|
# NOTE: services intentionally NOT in this probe surface
|
||||||
|
# - grafana.iamworkin.lan: every endpoint (incl. /api/health
|
||||||
|
# and /login) returns 401 behind Traefik basic-auth.
|
||||||
|
# Health covered by in-cluster monitoring-grafana scrape.
|
||||||
|
# - prometheus.iamworkin.lan: same auth pattern. Health covered
|
||||||
|
# by the prometheus self-scrape job.
|
||||||
|
# - guac.iamworkin.lan: deprecated — Guacamole moved to
|
||||||
|
# desktop.iamworkin.lan/guacamole/ (memory:
|
||||||
|
# feedback_traefik_cross_namespace_refs_disabled).
|
||||||
|
labels:
|
||||||
|
probe_type: "traefik-service"
|
||||||
|
relabel_configs:
|
||||||
|
- source_labels: [__address__]
|
||||||
|
target_label: __param_target
|
||||||
|
- source_labels: [__param_target]
|
||||||
|
regex: "https?://([^/:]+).*"
|
||||||
|
target_label: instance
|
||||||
|
- target_label: __address__
|
||||||
|
replacement: blackbox-exporter.monitoring.svc:9115
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Self-monitoring (K8s monitoring namespace)
|
# Self-monitoring (K8s monitoring namespace)
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
@@ -550,6 +642,42 @@ data:
|
|||||||
summary: "Print queue backlog on edge2 ({{ $value }} active jobs)"
|
summary: "Print queue backlog on edge2 ({{ $value }} active jobs)"
|
||||||
description: "CUPS has {{ $value }} active jobs queued. Possible printer jam, USB disconnect, or paper out."
|
description: "CUPS has {{ $value }} active jobs queued. Possible printer jam, USB disconnect, or paper out."
|
||||||
|
|
||||||
|
# Paper roll lifecycle alerts (XL Track I, 2026-04-26).
|
||||||
|
# Source-of-truth gauge: print_paper_remaining_percent (Print.Web OTEL,
|
||||||
|
# hydrated on startup from the active PaperRoll row).
|
||||||
|
# alert_channel=thermal_print routes through irc-notify -> Print.Web
|
||||||
|
# /api/print/alert so the printer announces its own paper-out warning
|
||||||
|
# on its remaining paper. Self-referential humor + operator nudge.
|
||||||
|
- alert: PrintPaperRollLow
|
||||||
|
expr: print_paper_remaining_percent{job="printweb-otel"} < 10 and print_paper_remaining_percent{job="printweb-otel"} > 5
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
alert_channel: thermal_print
|
||||||
|
annotations:
|
||||||
|
summary: "Print roll low on edge2 ({{ $value | printf \"%.1f\" }}% remaining)"
|
||||||
|
description: "NuPrint 210 paper roll has {{ $value | printf \"%.1f\" }}% remaining. Operator should load a fresh roll soon. Run /api/paper/status for the precise mm + estimated jobs left."
|
||||||
|
|
||||||
|
- alert: PrintPaperRollCritical
|
||||||
|
expr: print_paper_remaining_percent{job="printweb-otel"} <= 5
|
||||||
|
for: 2m
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
alert_channel: thermal_print
|
||||||
|
annotations:
|
||||||
|
summary: "Print roll critical on edge2 ({{ $value | printf \"%.1f\" }}% remaining)"
|
||||||
|
description: "NuPrint 210 paper roll at {{ $value | printf \"%.1f\" }}% — load a new roll NOW. The 50ft roll has a ~12% red-stripe zone; once paper passes that, the printer can run dry mid-job."
|
||||||
|
|
||||||
|
- alert: PrintJobDeadLetter
|
||||||
|
expr: increase(print_jobs_dead_letter_total[15m]) > 0
|
||||||
|
for: 1m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
alert_channel: thermal_print
|
||||||
|
annotations:
|
||||||
|
summary: "Print job(s) entered dead-letter on edge2 ({{ $value | printf \"%.0f\" }} in last 15m)"
|
||||||
|
description: "{{ $value | printf \"%.0f\" }} print job(s) exhausted MaxRetries and need operator action. Open /print-log, filter Status=DeadLetter, click 'Retry From Start' after fixing the underlying cause (paper jam, USB disconnect, printer power-cycle)."
|
||||||
|
|
||||||
- alert: CUPSHighJobRate
|
- alert: CUPSHighJobRate
|
||||||
expr: rate(cups_job_total[5m]) * 60 > 30
|
expr: rate(cups_job_total[5m]) * 60 > 30
|
||||||
for: 5m
|
for: 5m
|
||||||
@@ -589,23 +717,39 @@ data:
|
|||||||
summary: "RemoteDesktop /metrics scrape returning no data"
|
summary: "RemoteDesktop /metrics scrape returning no data"
|
||||||
description: "No fc_desktop_session_events_total series for 10 minutes. Either the Prometheus scrape target is misconfigured or the web deployment stopped exporting metrics. Zabbix template carries the same 10m no-data trigger for cross-monitor parity."
|
description: "No fc_desktop_session_events_total series for 10 minutes. Either the Prometheus scrape target is misconfigured or the web deployment stopped exporting metrics. Zabbix template carries the same 10m no-data trigger for cross-monitor parity."
|
||||||
|
|
||||||
|
# PUBLISHER QUIRK: fc_desktop_pool_depleted / _deficit emit one
|
||||||
|
# series per template per status (Ready/Warming/BelowDesiredSize/
|
||||||
|
# Disabled), and the historical series for non-current statuses
|
||||||
|
# stay at their last value. So just `_depleted > 0` fires forever
|
||||||
|
# on any template that ever entered a bad state.
|
||||||
|
#
|
||||||
|
# SAFE PATTERN: alert only when the canonical "Ready" status
|
||||||
|
# gauge does NOT report ready=1 for the enabled template. This
|
||||||
|
# is the publisher's own canary — _ready{status="Ready"}==1 is
|
||||||
|
# always the current "everything is fine" signal.
|
||||||
- alert: RemoteDesktopPoolDepleted
|
- alert: RemoteDesktopPoolDepleted
|
||||||
expr: fc_desktop_pool_depleted > 0
|
expr: |
|
||||||
|
group by(template) (fc_desktop_pool_ready{enabled="true"})
|
||||||
|
unless on(template) (fc_desktop_pool_ready{enabled="true",status="Ready"} == 1)
|
||||||
for: 5m
|
for: 5m
|
||||||
labels:
|
labels:
|
||||||
severity: warning
|
severity: warning
|
||||||
annotations:
|
annotations:
|
||||||
summary: "RemoteDesktop pool {{ $labels.pool }} depleted ({{ $labels.template }})"
|
summary: "RemoteDesktop pool depleted ({{ $labels.template }})"
|
||||||
description: "Pool {{ $labels.pool }} has been depleted for 5 minutes. New launches will cold-start. Operator should check for pod-scheduling failures, image pull issues, or exhausted node capacity before warm-pool parity is expected back."
|
description: "Pool for template {{ $labels.template }} has no Ready warm pod for 5 minutes. New launches will cold-start. Check pod-scheduling failures, image pull issues, or exhausted node capacity."
|
||||||
|
|
||||||
|
# Same pattern, but only fires when template explicitly reports
|
||||||
|
# a sustained Warning-level alert state (current-status series).
|
||||||
- alert: RemoteDesktopPoolDeficitSustained
|
- alert: RemoteDesktopPoolDeficitSustained
|
||||||
expr: fc_desktop_pool_deficit > 0
|
expr: |
|
||||||
|
fc_desktop_pool_deficit{enabled="true",alert_level="Warning"} > 0
|
||||||
|
unless on(template) (fc_desktop_pool_ready{enabled="true",status="Ready"} == 1)
|
||||||
for: 10m
|
for: 10m
|
||||||
labels:
|
labels:
|
||||||
severity: info
|
severity: info
|
||||||
annotations:
|
annotations:
|
||||||
summary: "RemoteDesktop pool {{ $labels.pool }} below desired for 10m"
|
summary: "RemoteDesktop pool {{ $labels.template }} below desired for 10m"
|
||||||
description: "Pool {{ $labels.pool }} has a persistent deficit of {{ $value }} warm pods. The operator is reconciling but can't reach desired size — likely an image pull, NFS affinity, or claim-init issue."
|
description: "Pool {{ $labels.template }} has a persistent deficit of {{ $value }} warm pods AND no Ready series. Likely image pull, NFS affinity, or claim-init issue."
|
||||||
|
|
||||||
- alert: RemoteDesktopSessionChurnSpike
|
- alert: RemoteDesktopSessionChurnSpike
|
||||||
expr: sum(rate(fc_desktop_session_events_total{event="launch"}[5m])) * 60 > 20
|
expr: sum(rate(fc_desktop_session_events_total{event="launch"}[5m])) * 60 > 20
|
||||||
@@ -625,8 +769,10 @@ data:
|
|||||||
summary: "RemoteDesktop recording events silent for 30m despite active launches"
|
summary: "RemoteDesktop recording events silent for 30m despite active launches"
|
||||||
description: "No recording events in 30 minutes while launches are happening. Recording may be silently disabled on all templates (SessionRecordingEnabled=false), the guacd NFS mount may be unhealthy, or the retention sweep isn't emitting events. Not an error by itself — worth checking."
|
description: "No recording events in 30 minutes while launches are happening. Recording may be silently disabled on all templates (SessionRecordingEnabled=false), the guacd NFS mount may be unhealthy, or the retention sweep isn't emitting events. Not an error by itself — worth checking."
|
||||||
|
|
||||||
|
# Match by job — instance label carries full URL incl. /health,
|
||||||
|
# not just hostname, so a hostname-only match never fires.
|
||||||
- alert: RemoteDesktopTlsExpiry
|
- alert: RemoteDesktopTlsExpiry
|
||||||
expr: probe_ssl_earliest_cert_expiry{instance="https://desktop.iamworkin.lan"} - time() < 2 * 86400
|
expr: probe_ssl_earliest_cert_expiry{job="probe-remotedesktop"} - time() < 2 * 86400
|
||||||
for: 6h
|
for: 6h
|
||||||
labels:
|
labels:
|
||||||
severity: critical
|
severity: critical
|
||||||
@@ -713,13 +859,16 @@ data:
|
|||||||
annotations:
|
annotations:
|
||||||
summary: "Epson ink CRITICAL: {{ $labels.prtMarkerSuppliesDescription }} at {{ $value }}%"
|
summary: "Epson ink CRITICAL: {{ $labels.prtMarkerSuppliesDescription }} at {{ $value }}%"
|
||||||
|
|
||||||
|
# for: 30m absorbs sleep cycles. The EcoTank sleeps after ~5 min
|
||||||
|
# of idle and SNMP times out, so 5m for: would page nightly. A
|
||||||
|
# genuine printer outage (jam, disconnected) lasts well over 30m.
|
||||||
- alert: EpsonPrinterDown
|
- alert: EpsonPrinterDown
|
||||||
expr: up{job="snmp-printer"} == 0
|
expr: up{job="snmp-printer"} == 0
|
||||||
for: 5m
|
for: 30m
|
||||||
labels:
|
labels:
|
||||||
severity: warning
|
severity: warning
|
||||||
annotations:
|
annotations:
|
||||||
summary: "Epson ET-3750 SNMP unreachable"
|
summary: "Epson ET-3750 SNMP unreachable for >30m (likely actual fault, not sleep)"
|
||||||
|
|
||||||
- alert: SynologyDiskLow
|
- alert: SynologyDiskLow
|
||||||
expr: hrStorageUsed{job="snmp-nas"} / hrStorageSize{job="snmp-nas"} * 100 > 85
|
expr: hrStorageUsed{job="snmp-nas"} / hrStorageSize{job="snmp-nas"} * 100 > 85
|
||||||
@@ -773,6 +922,174 @@ data:
|
|||||||
annotations:
|
annotations:
|
||||||
summary: "Disk usage high on {{ $labels.instance }} ({{ $value | printf \"%.1f\" }}%)"
|
summary: "Disk usage high on {{ $labels.instance }} ({{ $value | printf \"%.1f\" }}%)"
|
||||||
|
|
||||||
|
# K8s pod-state alerts. Require kube-state-metrics scrape (added
|
||||||
|
# 2026-04-26 — see scrape_configs above). Would have surfaced the
|
||||||
|
# agent-zero ollama-proxy 172x crash-loop instead of letting it
|
||||||
|
# silently churn for ~3 days.
|
||||||
|
- name: kubernetes-state
|
||||||
|
rules:
|
||||||
|
- alert: KubeContainerRestartingFrequently
|
||||||
|
expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
|
||||||
|
for: 15m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} restarting >5x/hr"
|
||||||
|
description: "Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted {{ $value | printf \"%.0f\" }} times in the last hour. Check 'kubectl describe pod' + last-state termination reason."
|
||||||
|
|
||||||
|
- alert: KubeContainerCrashLooping
|
||||||
|
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
alert_channel: thermal_print
|
||||||
|
annotations:
|
||||||
|
summary: "{{ $labels.namespace }}/{{ $labels.pod }} crashlooping ({{ $value | printf \"%.0f\" }} restarts/15m)"
|
||||||
|
description: "Container {{ $labels.container }} restarted {{ $value | printf \"%.0f\" }} times in 15 minutes — actively crashlooping."
|
||||||
|
|
||||||
|
- alert: KubePodNotReady
|
||||||
|
expr: sum by(namespace, pod) (kube_pod_status_phase{phase=~"Pending|Failed|Unknown"}) > 0
|
||||||
|
for: 15m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "{{ $labels.namespace }}/{{ $labels.pod }} not Ready for >15m"
|
||||||
|
description: "Pod is in a non-Running, non-Succeeded phase for over 15 minutes. Common causes: ImagePullBackOff (registry/Nexus down, wrong image tag), pending PVC, scheduling failure (taint/resources)."
|
||||||
|
|
||||||
|
- alert: KubePodImagePullBackOff
|
||||||
|
expr: sum by(namespace, pod) (kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"}) > 0
|
||||||
|
for: 10m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "{{ $labels.namespace }}/{{ $labels.pod }} ImagePullBackOff for >10m"
|
||||||
|
description: "Pod can't pull image. Check the image ref (often a stale tag or unreachable registry) and clean up if it's an orphan."
|
||||||
|
|
||||||
|
- alert: KubeDeploymentReplicasMismatch
|
||||||
|
expr: kube_deployment_spec_replicas != kube_deployment_status_replicas_available
|
||||||
|
for: 15m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "Deployment {{ $labels.namespace }}/{{ $labels.deployment }} replica mismatch"
|
||||||
|
description: "Spec wants {{ $labels.spec_replicas }} but only {{ $value }} available. Likely a rollout stuck on probe failure, scheduling, or PVC."
|
||||||
|
|
||||||
|
# Longhorn storage health alerts. Required: longhorn scrape job
|
||||||
|
# (added 2026-04-26 — see scrape_configs above). The K8s events
|
||||||
|
# for "snapshot becomes not ready to use" are transient lifecycle
|
||||||
|
# noise, not actionable — these alerts use the actual Longhorn
|
||||||
|
# gauges that reflect persistent state.
|
||||||
|
- name: longhorn-storage
|
||||||
|
rules:
|
||||||
|
# Volume robustness: 0=unknown, 1=healthy, 2=degraded, 3=faulted.
|
||||||
|
# Detached volumes report 0 — that's normal for unattached PVCs,
|
||||||
|
# so filter to only attached.
|
||||||
|
- alert: LonghornVolumeDegraded
|
||||||
|
expr: longhorn_volume_robustness{robustness="degraded"} == 1
|
||||||
|
for: 15m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "Longhorn volume {{ $labels.volume }} degraded for >15m"
|
||||||
|
description: "Volume {{ $labels.volume }} on node {{ $labels.node }} has been degraded (one or more replicas unhealthy) for 15+ minutes. Auto-rebuild may need help — check 'kubectl describe volume.longhorn.io {{ $labels.volume }} -n longhorn-system'."
|
||||||
|
|
||||||
|
- alert: LonghornVolumeFaulted
|
||||||
|
expr: longhorn_volume_robustness{robustness="faulted"} == 1
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
alert_channel: thermal_print
|
||||||
|
annotations:
|
||||||
|
summary: "Longhorn volume {{ $labels.volume }} FAULTED"
|
||||||
|
description: "Volume {{ $labels.volume }} on node {{ $labels.node }} is faulted — all replicas unavailable. Data inaccessible. Manual intervention required."
|
||||||
|
|
||||||
|
# No backup in 36h indicates the daily-backup recurringJob is
|
||||||
|
# silently failing. Allows for one missed run + slack.
|
||||||
|
- alert: LonghornBackupStale
|
||||||
|
expr: |
|
||||||
|
(time() - max by(volume) (longhorn_backup_state{state="Completed"} * on(backup) group_left() longhorn_backup_actual_size_bytes)) > 36 * 3600
|
||||||
|
for: 1h
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "Longhorn volume {{ $labels.volume }} has no completed backup in >36h"
|
||||||
|
description: "Daily backup recurringJob (cron 0 2 * * *) appears to have skipped this volume. Check 'kubectl get backups.longhorn.io -n longhorn-system' and the daily-backup CronJob logs."
|
||||||
|
|
||||||
|
- alert: LonghornNodeUnhealthy
|
||||||
|
expr: longhorn_node_status{condition="ready",condition_reason!=""} == 0
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "Longhorn node {{ $labels.node }} not Ready"
|
||||||
|
description: "Node {{ $labels.node }} reports ready=false (reason: {{ $labels.condition_reason }}). Volumes scheduled to this node will be unavailable until it recovers."
|
||||||
|
|
||||||
|
# ============================================================
|
||||||
|
# FC Signage Marquee Performance — Track 3 + 8 (2026-05-06)
|
||||||
|
# Live-mirrored from FlowerCore.Notes/scripts/monitoring/alerts.yml.
|
||||||
|
# Source-of-truth for the live Podman Prometheus on noc1 is the
|
||||||
|
# Notes file; this K8s ConfigMap exists so a future migration to
|
||||||
|
# in-cluster Prometheus inherits the ruleset automatically.
|
||||||
|
# See feedback_monitoring_k8s_target_vs_live_podman.
|
||||||
|
# ============================================================
|
||||||
|
- name: fc-signage-marquee
|
||||||
|
rules:
|
||||||
|
- alert: MarqueeDroppedFramesHigh
|
||||||
|
expr: |
|
||||||
|
(
|
||||||
|
sum by (renderer, phase, node_id) (rate(marquee_dropped_frames_total[5m]))
|
||||||
|
/
|
||||||
|
sum by (renderer, phase, node_id) (rate(marquee_render_latency_ms_count[5m]))
|
||||||
|
) > 0.05
|
||||||
|
unless on()
|
||||||
|
absent_over_time(marquee_dropped_frames_total[7d])
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: signage
|
||||||
|
alert_channel: irc
|
||||||
|
annotations:
|
||||||
|
summary: "Marquee dropped-frame rate >5% on {{ $labels.renderer }}/{{ $labels.node_id }} ({{ $labels.phase }})"
|
||||||
|
description: "Renderer {{ $labels.renderer }} on {{ $labels.node_id }} drops >5% of frames during {{ $labels.phase }}. Animation visibly stuttery."
|
||||||
|
|
||||||
|
- alert: MarqueeRenderLatencyP99High
|
||||||
|
expr: |
|
||||||
|
histogram_quantile(
|
||||||
|
0.99,
|
||||||
|
sum by (renderer, phase, node_id, le) (rate(marquee_render_latency_ms_bucket[5m]))
|
||||||
|
) > 16
|
||||||
|
unless on()
|
||||||
|
absent_over_time(marquee_render_latency_ms_bucket[7d])
|
||||||
|
for: 10m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: signage
|
||||||
|
alert_channel: irc
|
||||||
|
annotations:
|
||||||
|
summary: "Marquee render latency p99 > 16ms on {{ $labels.renderer }}/{{ $labels.node_id }} ({{ $labels.phase }})"
|
||||||
|
description: "Per-frame render latency p99 has exceeded the Pi-class 16ms budget for 10 minutes."
|
||||||
|
|
||||||
|
- alert: MarqueeAnimationDurationDrift
|
||||||
|
expr: |
|
||||||
|
abs(
|
||||||
|
histogram_quantile(0.5, sum by (renderer, phase, le) (rate(marquee_animation_duration_ms_bucket[15m])))
|
||||||
|
-
|
||||||
|
on (phase) group_left() avg by (phase) (marquee_animation_duration_target_ms)
|
||||||
|
)
|
||||||
|
/
|
||||||
|
on (phase) group_left() avg by (phase) (marquee_animation_duration_target_ms)
|
||||||
|
> 0.10
|
||||||
|
unless on()
|
||||||
|
absent_over_time(marquee_animation_duration_ms_bucket[7d])
|
||||||
|
for: 15m
|
||||||
|
labels:
|
||||||
|
severity: info
|
||||||
|
service: signage
|
||||||
|
alert_channel: irc
|
||||||
|
annotations:
|
||||||
|
summary: "Marquee animation duration drifting > 10% on {{ $labels.renderer }} ({{ $labels.phase }})"
|
||||||
|
description: "Median observed cycle duration deviates from target DurationMs by >10%. Could indicate browser tab throttling, GPU pressure, or phase-advancement bug."
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# ConfigMap: Blackbox Exporter Configuration
|
# ConfigMap: Blackbox Exporter Configuration
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
@@ -804,6 +1121,22 @@ data:
|
|||||||
fail_if_body_not_matches_regexp:
|
fail_if_body_not_matches_regexp:
|
||||||
- '"models"'
|
- '"models"'
|
||||||
preferred_ip_protocol: ip4
|
preferred_ip_protocol: ip4
|
||||||
|
# https_internal — for Traefik-fronted services with step-ca leaf
|
||||||
|
# certs. blackbox does not trust the step-ca root CA, so http_2xx
|
||||||
|
# against any *.iamworkin.lan host fails with x509 unknown authority.
|
||||||
|
# Redirects + multiple status codes are accepted because some hosts
|
||||||
|
# 302 to /login or /scalar.
|
||||||
|
https_internal:
|
||||||
|
prober: http
|
||||||
|
timeout: 10s
|
||||||
|
http:
|
||||||
|
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
|
||||||
|
valid_status_codes: [200, 301, 302, 303, 307, 308]
|
||||||
|
method: GET
|
||||||
|
follow_redirects: true
|
||||||
|
preferred_ip_protocol: ip4
|
||||||
|
tls_config:
|
||||||
|
insecure_skip_verify: true
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# ConfigMap: IRC Notify Script
|
# ConfigMap: IRC Notify Script
|
||||||
@@ -3139,6 +3472,172 @@ data:
|
|||||||
relativeTimeRange: {from: 600, to: 0}
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
datasourceUid: __expr__
|
datasourceUid: __expr__
|
||||||
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [85], type: gt}}], refId: C}
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [85], type: gt}}], refId: C}
|
||||||
|
- orgId: 1
|
||||||
|
name: RemoteDesktop
|
||||||
|
folder: AI Stack Alerts
|
||||||
|
interval: 1m
|
||||||
|
rules:
|
||||||
|
- uid: remotedesktop-web-down
|
||||||
|
title: RemoteDesktop Web DOWN
|
||||||
|
condition: C
|
||||||
|
for: 3m
|
||||||
|
noDataState: Alerting
|
||||||
|
execErrState: OK
|
||||||
|
annotations:
|
||||||
|
summary: FlowerCore RemoteDesktop /health probe failing
|
||||||
|
description: "https://desktop.iamworkin.lan/health has failed for 3 minutes. Catalog + session launch surface offline."
|
||||||
|
runbook: "1. kubectl -n fc-desktop get pods -l app.kubernetes.io/name=remotedesktop-web 2. kubectl -n fc-desktop logs deploy/remotedesktop-web --tail=50 3. Check Traefik IngressRoute + step-ca cert 4. Rollout restart if pod is stuck"
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: remotedesktop
|
||||||
|
data:
|
||||||
|
- refId: A
|
||||||
|
relativeTimeRange: {from: 180, to: 0}
|
||||||
|
datasourceUid: prometheus
|
||||||
|
model: {expr: 'probe_success{job="probe-remotedesktop"}', instant: true, refId: A}
|
||||||
|
- refId: B
|
||||||
|
relativeTimeRange: {from: 180, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: reduce, expression: A, reducer: last, refId: B}
|
||||||
|
- refId: C
|
||||||
|
relativeTimeRange: {from: 180, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [1], type: lt}}], refId: C}
|
||||||
|
|
||||||
|
- uid: remotedesktop-metrics-stale
|
||||||
|
title: RemoteDesktop metrics stale
|
||||||
|
condition: C
|
||||||
|
for: 10m
|
||||||
|
noDataState: Alerting
|
||||||
|
execErrState: OK
|
||||||
|
annotations:
|
||||||
|
summary: RemoteDesktop /metrics returning no series
|
||||||
|
description: "No fc_desktop_session_events_total series for 10 minutes. Either the Prometheus scrape is misconfigured or the web deployment stopped exporting metrics. Cross-checked by Zabbix template's identical 10m no-data trigger."
|
||||||
|
runbook: "1. curl -sk https://desktop.iamworkin.lan/metrics | head 2. kubectl -n monitoring exec deploy/prometheus -- wget -qO- localhost:9090/api/v1/targets?scrapePool=fc-remotedesktop 3. Check monitoring-netpol egress allows to fc-desktop:8080"
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: remotedesktop
|
||||||
|
data:
|
||||||
|
- refId: A
|
||||||
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
|
datasourceUid: prometheus
|
||||||
|
model: {expr: 'count(fc_desktop_session_events_total) or vector(0)', instant: true, refId: A}
|
||||||
|
- refId: B
|
||||||
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: reduce, expression: A, reducer: last, refId: B}
|
||||||
|
- refId: C
|
||||||
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [1], type: lt}}], refId: C}
|
||||||
|
|
||||||
|
- uid: remotedesktop-pool-depleted
|
||||||
|
title: RemoteDesktop pool depleted
|
||||||
|
condition: C
|
||||||
|
for: 5m
|
||||||
|
noDataState: OK
|
||||||
|
execErrState: OK
|
||||||
|
annotations:
|
||||||
|
summary: RemoteDesktop warm pool depleted for 5m
|
||||||
|
description: "A RemoteDesktop warm pool has fc_desktop_pool_depleted=1 for 5 minutes. New launches will cold-start. Check pod scheduling, image pull, node capacity."
|
||||||
|
runbook: "1. kubectl -n fc-desktop get pods -l app.kubernetes.io/name=remote-desktop --sort-by=.status.startTime 2. kubectl -n fc-desktop describe desktoppool <name> 3. Verify localhost/fc-desktop:* images imported on all 3 RKE2 nodes"
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
service: remotedesktop
|
||||||
|
data:
|
||||||
|
- refId: A
|
||||||
|
relativeTimeRange: {from: 300, to: 0}
|
||||||
|
datasourceUid: prometheus
|
||||||
|
model: {expr: 'max(fc_desktop_pool_depleted)', instant: true, refId: A}
|
||||||
|
- refId: B
|
||||||
|
relativeTimeRange: {from: 300, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: reduce, expression: A, reducer: last, refId: B}
|
||||||
|
- refId: C
|
||||||
|
relativeTimeRange: {from: 300, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [0.5], type: gt}}], refId: C}
|
||||||
|
|
||||||
|
- uid: remotedesktop-pool-deficit-sustained
|
||||||
|
title: RemoteDesktop pool below desired
|
||||||
|
condition: C
|
||||||
|
for: 10m
|
||||||
|
noDataState: OK
|
||||||
|
execErrState: OK
|
||||||
|
annotations:
|
||||||
|
summary: RemoteDesktop pool sustained deficit
|
||||||
|
description: "A pool has fc_desktop_pool_deficit>0 for 10 minutes. Operator is reconciling but can't reach desired size — likely image pull, NFS affinity, or claim-init issue."
|
||||||
|
runbook: "1. kubectl -n fc-desktop get pods -l flowercore.io/pool=<pool> 2. kubectl logs -n fc-desktop deploy/remotedesktop-operator 3. Check claim-init hook env on template"
|
||||||
|
labels:
|
||||||
|
severity: info
|
||||||
|
service: remotedesktop
|
||||||
|
data:
|
||||||
|
- refId: A
|
||||||
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
|
datasourceUid: prometheus
|
||||||
|
model: {expr: 'max(fc_desktop_pool_deficit)', instant: true, refId: A}
|
||||||
|
- refId: B
|
||||||
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: reduce, expression: A, reducer: last, refId: B}
|
||||||
|
- refId: C
|
||||||
|
relativeTimeRange: {from: 600, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [0], type: gt}}], refId: C}
|
||||||
|
|
||||||
|
- uid: remotedesktop-session-churn-spike
|
||||||
|
title: RemoteDesktop launch rate spike
|
||||||
|
condition: C
|
||||||
|
for: 5m
|
||||||
|
noDataState: OK
|
||||||
|
execErrState: OK
|
||||||
|
annotations:
|
||||||
|
summary: RemoteDesktop launch rate exceeds 20/min
|
||||||
|
description: "Launch events >20/min for 5 minutes. Could be a user-facing feature launch, pooled template thrashing, or runaway automation loop."
|
||||||
|
runbook: "1. kubectl -n fc-desktop get pods -l app.kubernetes.io/name=remote-desktop -o wide | wc -l 2. curl -sk https://desktop.iamworkin.lan/api/sessions/active 3. Check operator logs for reconcile loops"
|
||||||
|
labels:
|
||||||
|
severity: info
|
||||||
|
service: remotedesktop
|
||||||
|
data:
|
||||||
|
- refId: A
|
||||||
|
relativeTimeRange: {from: 300, to: 0}
|
||||||
|
datasourceUid: prometheus
|
||||||
|
model: {expr: 'sum(rate(fc_desktop_session_events_total{event="launch"}[5m])) * 60', instant: true, refId: A}
|
||||||
|
- refId: B
|
||||||
|
relativeTimeRange: {from: 300, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: reduce, expression: A, reducer: last, refId: B}
|
||||||
|
- refId: C
|
||||||
|
relativeTimeRange: {from: 300, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [20], type: gt}}], refId: C}
|
||||||
|
|
||||||
|
- uid: remotedesktop-tls-expiry
|
||||||
|
title: RemoteDesktop TLS cert expiring
|
||||||
|
condition: C
|
||||||
|
for: 6h
|
||||||
|
noDataState: OK
|
||||||
|
execErrState: OK
|
||||||
|
annotations:
|
||||||
|
summary: desktop.iamworkin.lan cert <2d to expiry
|
||||||
|
description: "The desktop.iamworkin.lan certificate is inside the 2-day renewal window and cert-manager has not renewed. Check cert-manager logs, step-ca reachability, FlowerCore.DNS preflight for dnsNames."
|
||||||
|
runbook: "1. kubectl -n fc-desktop get certificate remotedesktop-web-tls 2. kubectl -n cert-manager logs deploy/cert-manager --tail=50 3. Verify pfSense DNS override for desktop.iamworkin.lan"
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
service: remotedesktop
|
||||||
|
data:
|
||||||
|
- refId: A
|
||||||
|
relativeTimeRange: {from: 21600, to: 0}
|
||||||
|
datasourceUid: prometheus
|
||||||
|
model: {expr: '(probe_ssl_earliest_cert_expiry{job="probe-remotedesktop"} - time()) / 86400', instant: true, refId: A}
|
||||||
|
- refId: B
|
||||||
|
relativeTimeRange: {from: 21600, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: reduce, expression: A, reducer: last, refId: B}
|
||||||
|
- refId: C
|
||||||
|
relativeTimeRange: {from: 21600, to: 0}
|
||||||
|
datasourceUid: __expr__
|
||||||
|
model: {type: threshold, expression: B, conditions: [{evaluator: {params: [2], type: lt}}], refId: C}
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Deployment: Grafana
|
# Deployment: Grafana
|
||||||
@@ -3860,6 +4359,39 @@ spec:
|
|||||||
protocol: TCP
|
protocol: TCP
|
||||||
- port: 8443
|
- port: 8443
|
||||||
protocol: TCP
|
protocol: TCP
|
||||||
|
# Traefik /metrics endpoint (port 9100) — separate from the data-path
|
||||||
|
# ports above. Required for the in-cluster `traefik` scrape job.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: traefik-system
|
||||||
|
ports:
|
||||||
|
- port: 9100
|
||||||
|
protocol: TCP
|
||||||
|
# kube-state-metrics — required for kubernetes-state alert group.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: kube-system
|
||||||
|
ports:
|
||||||
|
- port: 8080
|
||||||
|
protocol: TCP
|
||||||
|
# cert-manager metrics — required for CertManagerCertificate* alerts.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: cert-manager
|
||||||
|
ports:
|
||||||
|
- port: 9402
|
||||||
|
protocol: TCP
|
||||||
|
# Longhorn manager metrics — required for Longhorn* alerts.
|
||||||
|
- to:
|
||||||
|
- namespaceSelector:
|
||||||
|
matchLabels:
|
||||||
|
kubernetes.io/metadata.name: longhorn-system
|
||||||
|
ports:
|
||||||
|
- port: 9500
|
||||||
|
protocol: TCP
|
||||||
# IRC (irc-notify → UnrealIRCd in irc namespace via K8s DNS)
|
# IRC (irc-notify → UnrealIRCd in irc namespace via K8s DNS)
|
||||||
- to:
|
- to:
|
||||||
- namespaceSelector:
|
- namespaceSelector:
|
||||||
|
|||||||
@@ -219,6 +219,65 @@ spec:
|
|||||||
tls:
|
tls:
|
||||||
secretName: cockpit-tls
|
secretName: cockpit-tls
|
||||||
---
|
---
|
||||||
|
# ============================================================
|
||||||
|
# PuppetDB Dashboard - noc1:8080 (HTTP, web UI only)
|
||||||
|
# Agent-to-PuppetDB mTLS still uses port 8081 directly via Puppet CA
|
||||||
|
# (NOT via this proxy). See docs/infrastructure/cert-recovery-2026-04-28.md
|
||||||
|
# ============================================================
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: puppetdb-external
|
||||||
|
namespace: noc-proxy
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
- port: 8080
|
||||||
|
targetPort: 8080
|
||||||
|
name: http
|
||||||
|
clusterIP: None
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Endpoints
|
||||||
|
metadata:
|
||||||
|
name: puppetdb-external
|
||||||
|
namespace: noc-proxy
|
||||||
|
subsets:
|
||||||
|
- addresses:
|
||||||
|
- ip: 10.0.56.10
|
||||||
|
ports:
|
||||||
|
- port: 8080
|
||||||
|
name: http
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: puppetdb-tls
|
||||||
|
namespace: noc-proxy
|
||||||
|
spec:
|
||||||
|
secretName: puppetdb-tls
|
||||||
|
issuerRef:
|
||||||
|
name: step-ca-acme
|
||||||
|
kind: ClusterIssuer
|
||||||
|
dnsNames:
|
||||||
|
- puppetdb.iamworkin.lan
|
||||||
|
---
|
||||||
|
apiVersion: traefik.io/v1alpha1
|
||||||
|
kind: IngressRoute
|
||||||
|
metadata:
|
||||||
|
name: puppetdb
|
||||||
|
namespace: noc-proxy
|
||||||
|
spec:
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
routes:
|
||||||
|
- kind: Rule
|
||||||
|
match: Host(`puppetdb.iamworkin.lan`)
|
||||||
|
services:
|
||||||
|
- name: puppetdb-external
|
||||||
|
port: 8080
|
||||||
|
tls:
|
||||||
|
secretName: puppetdb-tls
|
||||||
|
---
|
||||||
# NetworkPolicy: allow Traefik ingress, allow egress to noc1
|
# NetworkPolicy: allow Traefik ingress, allow egress to noc1
|
||||||
apiVersion: networking.k8s.io/v1
|
apiVersion: networking.k8s.io/v1
|
||||||
kind: NetworkPolicy
|
kind: NetworkPolicy
|
||||||
@@ -242,6 +301,8 @@ spec:
|
|||||||
ports:
|
ports:
|
||||||
- port: 3000
|
- port: 3000
|
||||||
protocol: TCP
|
protocol: TCP
|
||||||
|
- port: 8080
|
||||||
|
protocol: TCP
|
||||||
- port: 9090
|
- port: 9090
|
||||||
protocol: TCP
|
protocol: TCP
|
||||||
- port: 9091
|
- port: 9091
|
||||||
|
|||||||
@@ -148,7 +148,7 @@ spec:
|
|||||||
topologyKey: kubernetes.io/hostname
|
topologyKey: kubernetes.io/hostname
|
||||||
containers:
|
containers:
|
||||||
- name: telephony-web
|
- name: telephony-web
|
||||||
image: localhost/fc-telephony-web:v202604170153
|
image: localhost/fc-telephony-web:v202604252156
|
||||||
imagePullPolicy: Never
|
imagePullPolicy: Never
|
||||||
securityContext:
|
securityContext:
|
||||||
readOnlyRootFilesystem: true
|
readOnlyRootFilesystem: true
|
||||||
|
|||||||
60
apps/worldbuilder/README.md
Normal file
60
apps/worldbuilder/README.md
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
# FlowerCore.WorldBuilder
|
||||||
|
|
||||||
|
ArgoCD-managed manifest for FlowerCore.WorldBuilder.Web — comic / storyboard
|
||||||
|
authoring service that drives ComfyUI for panel image generation and
|
||||||
|
QuestPDF for letter / A4 export.
|
||||||
|
|
||||||
|
Source: `D:\git\FlowerCore\FlowerCore.WorldBuilder` (master)
|
||||||
|
|
||||||
|
## Deployment order
|
||||||
|
|
||||||
|
1. **DNS preflight** — `worldbuilder.iamworkin.lan -> 10.0.56.200` MUST exist
|
||||||
|
in pfSense Unbound before this manifest is applied, or cert-manager
|
||||||
|
HTTP-01 silently exponential-backs-off ~2h.
|
||||||
|
Memory: `feedback_pfsense_dns_required_for_acme`.
|
||||||
|
2. **Image import to ALL RKE2 nodes** — pod can schedule to any of
|
||||||
|
`rke2-server` (10.0.56.11), `rke2-agent1` (10.0.56.12),
|
||||||
|
`rke2-agent2` (10.0.56.13). Build with:
|
||||||
|
```bash
|
||||||
|
bash deploy/build.sh # in FlowerCore.WorldBuilder repo
|
||||||
|
podman save localhost/fc-worldbuilder:v<TAG> -o /tmp/fc-worldbuilder-v<TAG>.tar
|
||||||
|
for h in 10.0.56.11 10.0.56.12 10.0.56.13; do
|
||||||
|
scp /tmp/fc-worldbuilder-v<TAG>.tar fcadmin@$h:/tmp/
|
||||||
|
ssh fcadmin@$h \
|
||||||
|
"sudo /var/lib/rancher/rke2/bin/ctr -a /run/k3s/containerd/containerd.sock \
|
||||||
|
-n k8s.io images import /tmp/fc-worldbuilder-v<TAG>.tar"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
Memory: `feedback_rke2_image_import_per_node_scp`.
|
||||||
|
3. **Bump image tag** in `worldbuilder.yaml` and git push.
|
||||||
|
ArgoCD ApplicationSet picks up within ~3 minutes.
|
||||||
|
4. **First production render** — open `https://worldbuilder.iamworkin.lan`,
|
||||||
|
create World → Character → Storyboard → ExportJob, confirm artifact
|
||||||
|
downloads. ComfyUI lives on BLUEJAY-WS at `http://10.0.56.20:8188`.
|
||||||
|
|
||||||
|
## Health probes
|
||||||
|
|
||||||
|
- `startupProbe` + `readinessProbe`: `httpGet /healthz` (registered explicitly
|
||||||
|
in Program.cs — anonymous, no DB or OpenAPI dependency).
|
||||||
|
- `livenessProbe`: `tcpSocket` as a cheap fallback.
|
||||||
|
Memory: `feedback_k8s_probes_must_not_hit_openapi`,
|
||||||
|
`feedback_k8s_probes_behind_auth_middleware`.
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
- Longhorn RWO PVC `worldbuilder-data` (5Gi) mounted at `/data`. SQLite DB
|
||||||
|
lives at `/data/worldbuilder.db`, generated images under `/data/gallery/`,
|
||||||
|
PDF/PNG exports under `/data/exports/`.
|
||||||
|
- DataProtection keys persist to the same SQLite via
|
||||||
|
`AddFlowerCoreDataProtection<WorldBuilderDbContext>` — explicit migration
|
||||||
|
`20260429133417_Initial` already creates `fc_dp_keys`.
|
||||||
|
Memory: `feedback_dataprotection_keys_persist_to_app_dbcontext`,
|
||||||
|
`feedback_intranet_dataprotection_table_must_have_explicit_migration`.
|
||||||
|
|
||||||
|
## Image generation backend
|
||||||
|
|
||||||
|
`FlowerCore:WorldBuilder:ImageGeneration:BaseUrl=http://10.0.56.20:8188` —
|
||||||
|
ComfyUI runs on BLUEJAY-WS Windows (R9700 / gfx1201 / ROCm 7.2.1). Pod reaches
|
||||||
|
the workstation directly across the 10.0.56.0/24 VLAN (no Podman-style host-
|
||||||
|
filter issues — K8s pods route via Calico, which is L3-routed across the
|
||||||
|
VLAN).
|
||||||
208
apps/worldbuilder/worldbuilder.yaml
Normal file
208
apps/worldbuilder/worldbuilder.yaml
Normal file
@@ -0,0 +1,208 @@
|
|||||||
|
# FlowerCore.WorldBuilder — comic / storyboard authoring service.
|
||||||
|
#
|
||||||
|
# Deployment + Service + PVC + Certificate + IngressRoute. ArgoCD-managed
|
||||||
|
# end-to-end. See apps/worldbuilder/README.md for the per-deploy runbook.
|
||||||
|
#
|
||||||
|
# Image build (BLUEJAY-WS):
|
||||||
|
# bash deploy/build.sh # in FlowerCore.WorldBuilder repo
|
||||||
|
# podman save localhost/fc-worldbuilder:v<TAG> -o /tmp/fc-worldbuilder-v<TAG>.tar
|
||||||
|
# for h in 10.0.56.11 10.0.56.12 10.0.56.13; do
|
||||||
|
# scp /tmp/fc-worldbuilder-v<TAG>.tar fcadmin@$h:/tmp/
|
||||||
|
# ssh fcadmin@$h "sudo /var/lib/rancher/rke2/bin/ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images import /tmp/fc-worldbuilder-v<TAG>.tar"
|
||||||
|
# done
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: fc-worldbuilder
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
---
|
||||||
|
# SQLite DB + generated image gallery + PDF/PNG exports.
|
||||||
|
# Longhorn RWO — single replica with `Recreate` rollout strategy keeps it safe.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: worldbuilder-data
|
||||||
|
namespace: fc-worldbuilder
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
storageClassName: longhorn
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 5Gi
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: worldbuilder-web
|
||||||
|
namespace: fc-worldbuilder
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: worldbuilder-web
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
revisionHistoryLimit: 3
|
||||||
|
strategy:
|
||||||
|
# RWO PVC + single replica. Recreate avoids multi-attach overlap.
|
||||||
|
type: Recreate
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: worldbuilder-web
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: worldbuilder-web
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
annotations:
|
||||||
|
prometheus.io/scrape: "true"
|
||||||
|
prometheus.io/port: "8080"
|
||||||
|
prometheus.io/path: "/metrics/prometheus"
|
||||||
|
spec:
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 1654
|
||||||
|
fsGroupChangePolicy: OnRootMismatch
|
||||||
|
containers:
|
||||||
|
- name: web
|
||||||
|
# Bump tag for each rebuild. Initial deploy: v202605062048
|
||||||
|
image: localhost/fc-worldbuilder:v202605062048
|
||||||
|
imagePullPolicy: Never
|
||||||
|
ports:
|
||||||
|
- containerPort: 8080
|
||||||
|
name: http
|
||||||
|
env:
|
||||||
|
- name: ASPNETCORE_URLS
|
||||||
|
value: "http://+:8080"
|
||||||
|
- name: ASPNETCORE_ENVIRONMENT
|
||||||
|
value: "Production"
|
||||||
|
- name: DOTNET_RUNNING_IN_CONTAINER
|
||||||
|
value: "true"
|
||||||
|
- name: DOTNET_SYSTEM_GLOBALIZATION_INVARIANT
|
||||||
|
value: "false"
|
||||||
|
# SQLite path overrides (default appsettings uses relative paths).
|
||||||
|
- name: ConnectionStrings__DefaultConnection
|
||||||
|
value: "Data Source=/data/worldbuilder.db"
|
||||||
|
- name: FlowerCore__Database__Provider
|
||||||
|
value: "Sqlite"
|
||||||
|
- name: FlowerCore__Database__ConnectionStrings__Sqlite
|
||||||
|
value: "Data Source=/data/worldbuilder.db"
|
||||||
|
# Generated image gallery + exports persist on /data.
|
||||||
|
- name: FlowerCore__WorldBuilder__ImageStore__RootPath
|
||||||
|
value: "/data/gallery"
|
||||||
|
- name: FlowerCore__WorldBuilder__Export__RootPath
|
||||||
|
value: "/data/exports"
|
||||||
|
# ComfyUI on BLUEJAY-WS (R9700 / gfx1201 / ROCm 7.2.1).
|
||||||
|
- name: FlowerCore__WorldBuilder__ImageGeneration__BaseUrl
|
||||||
|
value: "http://10.0.56.20:8188"
|
||||||
|
- name: FlowerCore__WorldBuilder__ImageGeneration__ClientMode
|
||||||
|
value: "comfyui"
|
||||||
|
resources:
|
||||||
|
# Cluster CPU-request budget runs hot (99% on all 3 nodes at deploy
|
||||||
|
# time) while actual CPU usage is well below capacity. Idle Blazor
|
||||||
|
# Server + SignalR + a single ComfyUI poller uses ~5m, so 25m is
|
||||||
|
# generous. Re-evaluate if active rendering/export workers ever
|
||||||
|
# push past the limit.
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: 1000m
|
||||||
|
memory: 768Mi
|
||||||
|
# /healthz is registered explicitly in Program.cs (anonymous, no DB
|
||||||
|
# or OpenAPI dependency). Liveness uses tcpSocket as a cheap fallback
|
||||||
|
# in case future middleware changes accidentally gate /healthz.
|
||||||
|
# Memory: feedback_k8s_probes_must_not_hit_openapi,
|
||||||
|
# feedback_k8s_probes_behind_auth_middleware.
|
||||||
|
startupProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /healthz
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
failureThreshold: 30
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /healthz
|
||||||
|
port: 8080
|
||||||
|
periodSeconds: 10
|
||||||
|
failureThreshold: 3
|
||||||
|
livenessProbe:
|
||||||
|
tcpSocket:
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 30
|
||||||
|
failureThreshold: 3
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
runAsUser: 1654
|
||||||
|
runAsGroup: 1654
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
volumeMounts:
|
||||||
|
- name: data
|
||||||
|
mountPath: /data
|
||||||
|
- name: tmp
|
||||||
|
mountPath: /tmp
|
||||||
|
- name: logs
|
||||||
|
mountPath: /app/logs
|
||||||
|
volumes:
|
||||||
|
- name: data
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: worldbuilder-data
|
||||||
|
- name: tmp
|
||||||
|
emptyDir: {}
|
||||||
|
- name: logs
|
||||||
|
emptyDir: {}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: worldbuilder-web
|
||||||
|
namespace: fc-worldbuilder
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: worldbuilder-web
|
||||||
|
app.kubernetes.io/part-of: flowercore
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/name: worldbuilder-web
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 80
|
||||||
|
targetPort: 8080
|
||||||
|
---
|
||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: Certificate
|
||||||
|
metadata:
|
||||||
|
name: worldbuilder-web-tls
|
||||||
|
namespace: fc-worldbuilder
|
||||||
|
spec:
|
||||||
|
secretName: worldbuilder-web-tls
|
||||||
|
issuerRef:
|
||||||
|
name: step-ca-acme
|
||||||
|
kind: ClusterIssuer
|
||||||
|
dnsNames:
|
||||||
|
- worldbuilder.iamworkin.lan
|
||||||
|
duration: 2160h # 90d
|
||||||
|
renewBefore: 720h # 30d
|
||||||
|
---
|
||||||
|
apiVersion: traefik.io/v1alpha1
|
||||||
|
kind: IngressRoute
|
||||||
|
metadata:
|
||||||
|
name: worldbuilder-web
|
||||||
|
namespace: fc-worldbuilder
|
||||||
|
spec:
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
routes:
|
||||||
|
- match: Host(`worldbuilder.iamworkin.lan`)
|
||||||
|
kind: Rule
|
||||||
|
services:
|
||||||
|
- name: worldbuilder-web
|
||||||
|
port: 80
|
||||||
|
tls:
|
||||||
|
secretName: worldbuilder-web-tls
|
||||||
24
tests/bluejay-infra-lint/BluejayInfraLint.Tests.csproj
Normal file
24
tests/bluejay-infra-lint/BluejayInfraLint.Tests.csproj
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
<Project Sdk="Microsoft.NET.Sdk">
|
||||||
|
<PropertyGroup>
|
||||||
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
|
<Nullable>enable</Nullable>
|
||||||
|
<IsPackable>false</IsPackable>
|
||||||
|
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||||
|
</PropertyGroup>
|
||||||
|
|
||||||
|
<ItemGroup>
|
||||||
|
<PackageReference Include="coverlet.collector" Version="6.0.2">
|
||||||
|
<PrivateAssets>all</PrivateAssets>
|
||||||
|
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||||
|
</PackageReference>
|
||||||
|
<PackageReference Include="FluentAssertions" Version="6.12.1" />
|
||||||
|
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
|
||||||
|
<PackageReference Include="xunit" Version="2.9.2" />
|
||||||
|
<PackageReference Include="xunit.runner.visualstudio" Version="2.8.2">
|
||||||
|
<PrivateAssets>all</PrivateAssets>
|
||||||
|
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||||
|
</PackageReference>
|
||||||
|
<PackageReference Include="YamlDotNet" Version="16.2.0" />
|
||||||
|
</ItemGroup>
|
||||||
|
</Project>
|
||||||
633
tests/bluejay-infra-lint/FleetManifestLintTests.cs
Normal file
633
tests/bluejay-infra-lint/FleetManifestLintTests.cs
Normal file
@@ -0,0 +1,633 @@
|
|||||||
|
using FluentAssertions;
|
||||||
|
using System.Text.RegularExpressions;
|
||||||
|
using Xunit;
|
||||||
|
using YamlDotNet.Core;
|
||||||
|
using YamlDotNet.RepresentationModel;
|
||||||
|
|
||||||
|
namespace BluejayInfraLint.Tests;
|
||||||
|
|
||||||
|
[Trait("Category", "Unit")]
|
||||||
|
public sealed class FleetManifestLintTests
|
||||||
|
{
|
||||||
|
private static readonly ManifestInventory Inventory = ManifestInventory.Load();
|
||||||
|
|
||||||
|
private static readonly HashSet<string> PublicReadOnlyHosts = new(StringComparer.Ordinal)
|
||||||
|
{
|
||||||
|
"dist.flowercore.io",
|
||||||
|
"dns.iamworkin.lan",
|
||||||
|
};
|
||||||
|
|
||||||
|
// Public hosts that allow a tightly bounded write surface in addition to
|
||||||
|
// GET/HEAD. updatecenter.iamworkin.lan accepts POST /api/v1/checkin/{id}
|
||||||
|
// (bootstrap-JWT) so its allowlist is GET||HEAD||POST||OPTIONS — but
|
||||||
|
// PUT/PATCH/DELETE must still 404 at the route. Anything wider than this
|
||||||
|
// set should fail this lint.
|
||||||
|
private static readonly HashSet<string> PublicReadWriteAllowlistHosts = new(StringComparer.Ordinal)
|
||||||
|
{
|
||||||
|
"updatecenter.iamworkin.lan",
|
||||||
|
"updates.iamworkin.lan",
|
||||||
|
};
|
||||||
|
|
||||||
|
private static readonly HashSet<string> ApiKeyProtectedDeployments = new(StringComparer.Ordinal)
|
||||||
|
{
|
||||||
|
"messageboard-web",
|
||||||
|
"scoreboard-web",
|
||||||
|
"segmentdisplay-web",
|
||||||
|
"signalcontrol-web",
|
||||||
|
};
|
||||||
|
|
||||||
|
private static readonly HashSet<string> PublicEgressDeployments = new(StringComparer.Ordinal)
|
||||||
|
{
|
||||||
|
"asterisk",
|
||||||
|
"fc-llm-bridge",
|
||||||
|
"mysql-web",
|
||||||
|
"php-web",
|
||||||
|
"ttsreader-align",
|
||||||
|
"ttsreader-kokoro",
|
||||||
|
"ttsreader-modern",
|
||||||
|
"ttsreader-piper",
|
||||||
|
};
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void IngressRoutes_MustKeepServiceReferencesInTheSameNamespace()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.Kind == "IngressRoute")
|
||||||
|
.SelectMany(document =>
|
||||||
|
document.MappingSequence("spec", "routes")
|
||||||
|
.SelectMany(route =>
|
||||||
|
route.MappingSequence("services")
|
||||||
|
.Select(service => new
|
||||||
|
{
|
||||||
|
Document = document,
|
||||||
|
ServiceName = ManifestNodeExtensions.Scalar(service, "name"),
|
||||||
|
ServiceNamespace = ManifestNodeExtensions.Scalar(service, "namespace"),
|
||||||
|
})))
|
||||||
|
.Where(entry => !string.IsNullOrWhiteSpace(entry.ServiceNamespace))
|
||||||
|
.Where(entry => !string.Equals(entry.ServiceNamespace, entry.Document.Namespace, StringComparison.Ordinal))
|
||||||
|
.Select(entry =>
|
||||||
|
$"{entry.Document.Descriptor} references Service '{entry.ServiceName}' in namespace '{entry.ServiceNamespace}'.")
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PublicReadOnlyIngressRoutes_MustExplicitlyAllowOnlyGetAndHead()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.Kind == "IngressRoute")
|
||||||
|
.SelectMany(document =>
|
||||||
|
document.MappingSequence("spec", "routes")
|
||||||
|
.Select(route => new
|
||||||
|
{
|
||||||
|
Document = document,
|
||||||
|
Match = ManifestNodeExtensions.Scalar(route, "match") ?? string.Empty,
|
||||||
|
}))
|
||||||
|
.Where(entry => PublicReadOnlyHosts.Any(host => entry.Match.Contains($"Host(`{host}`)", StringComparison.Ordinal)))
|
||||||
|
.Where(entry => !entry.Match.Contains("Method(`GET`)", StringComparison.Ordinal)
|
||||||
|
|| !entry.Match.Contains("Method(`HEAD`)", StringComparison.Ordinal))
|
||||||
|
.Select(entry => $"{entry.Document.Descriptor} is missing an explicit GET/HEAD method allowlist.")
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PublicReadWriteIngressRoutes_MustPinGetHeadPostOptionsAllowlist()
|
||||||
|
{
|
||||||
|
// For hosts in PublicReadWriteAllowlistHosts, the route match MUST
|
||||||
|
// contain Method(`GET`), Method(`HEAD`), Method(`POST`), and
|
||||||
|
// Method(`OPTIONS`) AND MUST NOT contain Method(`PUT`),
|
||||||
|
// Method(`PATCH`), or Method(`DELETE`). This keeps the public
|
||||||
|
// allowlist invariant against regression — see Track A's
|
||||||
|
// updatecenter-web ingressroute hardening.
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.Kind == "IngressRoute")
|
||||||
|
.SelectMany(document =>
|
||||||
|
document.MappingSequence("spec", "routes")
|
||||||
|
.Select(route => new
|
||||||
|
{
|
||||||
|
Document = document,
|
||||||
|
Match = ManifestNodeExtensions.Scalar(route, "match") ?? string.Empty,
|
||||||
|
}))
|
||||||
|
.Where(entry => PublicReadWriteAllowlistHosts.Any(host => entry.Match.Contains($"Host(`{host}`)", StringComparison.Ordinal)))
|
||||||
|
.SelectMany(entry =>
|
||||||
|
{
|
||||||
|
var localViolations = new List<string>();
|
||||||
|
|
||||||
|
foreach (var required in new[] { "GET", "HEAD", "POST", "OPTIONS" })
|
||||||
|
{
|
||||||
|
if (!entry.Match.Contains($"Method(`{required}`)", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{entry.Document.Descriptor} is missing required Method(`{required}`).");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var forbidden in new[] { "PUT", "PATCH", "DELETE" })
|
||||||
|
{
|
||||||
|
if (entry.Match.Contains($"Method(`{forbidden}`)", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{entry.Document.Descriptor} must not include Method(`{forbidden}`) on a public host.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return localViolations;
|
||||||
|
})
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void TraefikVipNetworkPolicies_MustAllowPostDnatBackendPorts()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.Kind == "NetworkPolicy")
|
||||||
|
.Where(document => document.AllScalars().Any(value => value.Contains("10.0.56.200", StringComparison.Ordinal)))
|
||||||
|
.SelectMany(document =>
|
||||||
|
{
|
||||||
|
var ports = document.EgressPorts().ToHashSet(StringComparer.Ordinal);
|
||||||
|
var localViolations = new List<string>();
|
||||||
|
|
||||||
|
if (ports.Contains("443") && !ports.Contains("8443"))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} allows Traefik VIP 443 without backend port 8443.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (ports.Contains("80") && !ports.Contains("8000") && !ports.Contains("8080"))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} allows Traefik VIP 80 without a backend HTTP port (8000/8080).");
|
||||||
|
}
|
||||||
|
|
||||||
|
return localViolations;
|
||||||
|
})
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void ApiKeyProtectedDeployments_MustUseTcpSocketHealthProbes()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.Kind == "Deployment")
|
||||||
|
.Where(document => ApiKeyProtectedDeployments.Contains(document.Name))
|
||||||
|
.SelectMany(document => document.ContainerMappings().SelectMany(container =>
|
||||||
|
ProbeViolations(document, container, "readinessProbe")
|
||||||
|
.Concat(ProbeViolations(document, container, "livenessProbe"))))
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void StatefulSets_WithVolumeClaimTemplates_MustDeclareFilesystemDefaults()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.Kind == "StatefulSet")
|
||||||
|
.Where(document => document.MappingSequence("spec", "volumeClaimTemplates").Count > 0)
|
||||||
|
.SelectMany(document =>
|
||||||
|
{
|
||||||
|
var localViolations = new List<string>();
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(document.Scalar("spec", "podManagementPolicy")))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} is missing spec.podManagementPolicy.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(document.Scalar("spec", "revisionHistoryLimit")))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} is missing spec.revisionHistoryLimit.");
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var claimTemplate in document.MappingSequence("spec", "volumeClaimTemplates"))
|
||||||
|
{
|
||||||
|
if (!string.Equals(
|
||||||
|
ManifestNodeExtensions.Scalar(claimTemplate, "spec", "volumeMode"),
|
||||||
|
"Filesystem",
|
||||||
|
StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
var claimName = ManifestNodeExtensions.Scalar(claimTemplate, "metadata", "name") ?? "<unnamed>";
|
||||||
|
localViolations.Add($"{document.Descriptor} volumeClaimTemplate '{claimName}' is missing volumeMode: Filesystem.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return localViolations;
|
||||||
|
})
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void LocallyImportedImages_MustUseLocalhostPrefixAndNeverPullPolicy()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.PodSpec() is not null)
|
||||||
|
.SelectMany(document => document.ContainerSpecs()
|
||||||
|
.Where(container => !string.IsNullOrWhiteSpace(container.Image))
|
||||||
|
.Select(container => new
|
||||||
|
{
|
||||||
|
Document = document,
|
||||||
|
Container = container,
|
||||||
|
}))
|
||||||
|
.Where(entry =>
|
||||||
|
(entry.Container.Image.StartsWith("localhost/", StringComparison.Ordinal)
|
||||||
|
&& !string.Equals(entry.Container.ImagePullPolicy, "Never", StringComparison.Ordinal))
|
||||||
|
|| (entry.Container.Image.StartsWith("fc-", StringComparison.Ordinal)
|
||||||
|
&& !entry.Container.Image.Contains('/', StringComparison.Ordinal)))
|
||||||
|
.Select(entry =>
|
||||||
|
{
|
||||||
|
if (entry.Container.Image.StartsWith("localhost/", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return $"{entry.Document.Descriptor} container '{entry.Container.Name}' uses {entry.Container.Image} without imagePullPolicy: Never.";
|
||||||
|
}
|
||||||
|
|
||||||
|
return $"{entry.Document.Descriptor} container '{entry.Container.Name}' uses non-local image '{entry.Container.Image}' for a node-imported FlowerCore workload.";
|
||||||
|
})
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PublicEgressDeployments_MustOptOutOfIamworkinLanSearchSuffixes()
|
||||||
|
{
|
||||||
|
var violations = Inventory.Documents
|
||||||
|
.Where(document => document.PodSpec() is not null)
|
||||||
|
.Where(document => PublicEgressDeployments.Contains(document.Name))
|
||||||
|
.SelectMany(document =>
|
||||||
|
{
|
||||||
|
var localViolations = new List<string>();
|
||||||
|
var podSpec = document.PodSpec()!;
|
||||||
|
var dnsPolicy = ManifestNodeExtensions.Scalar(podSpec, "dnsPolicy");
|
||||||
|
var searches = ManifestNodeExtensions.ScalarSequence(podSpec, "dnsConfig", "searches").ToList();
|
||||||
|
|
||||||
|
if (!string.Equals(dnsPolicy, "None", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} is missing dnsPolicy: None.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (searches.Count == 0)
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} is missing dnsConfig.searches.");
|
||||||
|
}
|
||||||
|
else if (searches.Any(search => search.Contains("iamworkin.lan", StringComparison.OrdinalIgnoreCase)))
|
||||||
|
{
|
||||||
|
localViolations.Add($"{document.Descriptor} still includes iamworkin.lan in dnsConfig.searches.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return localViolations;
|
||||||
|
})
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
violations.Should().BeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IEnumerable<string> ProbeViolations(
|
||||||
|
ManifestDocument document,
|
||||||
|
YamlMappingNode container,
|
||||||
|
string probeKey)
|
||||||
|
{
|
||||||
|
if (!ManifestNodeExtensions.TryGetMapping(container, probeKey, out var probe)
|
||||||
|
|| !ManifestNodeExtensions.TryGetMapping(probe, "httpGet", out var httpGet))
|
||||||
|
{
|
||||||
|
return Array.Empty<string>();
|
||||||
|
}
|
||||||
|
|
||||||
|
var path = ManifestNodeExtensions.Scalar(httpGet, "path");
|
||||||
|
if (!string.Equals(path, "/health", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return Array.Empty<string>();
|
||||||
|
}
|
||||||
|
|
||||||
|
var containerName = ManifestNodeExtensions.Scalar(container, "name") ?? "<unnamed>";
|
||||||
|
return new[]
|
||||||
|
{
|
||||||
|
$"{document.Descriptor} container '{containerName}' still uses {probeKey}.httpGet on /health.",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class ManifestInventory
|
||||||
|
{
|
||||||
|
private ManifestInventory(string workspaceRoot, string bluejayRoot, IReadOnlyList<ManifestDocument> documents)
|
||||||
|
{
|
||||||
|
WorkspaceRoot = workspaceRoot;
|
||||||
|
BluejayRoot = bluejayRoot;
|
||||||
|
Documents = documents;
|
||||||
|
}
|
||||||
|
|
||||||
|
public string WorkspaceRoot { get; }
|
||||||
|
|
||||||
|
public string BluejayRoot { get; }
|
||||||
|
|
||||||
|
public IReadOnlyList<ManifestDocument> Documents { get; }
|
||||||
|
|
||||||
|
public static ManifestInventory Load()
|
||||||
|
{
|
||||||
|
var bluejayRoot = FindBluejayInfraRoot();
|
||||||
|
var workspaceRoot = Directory.GetParent(bluejayRoot)?.FullName
|
||||||
|
?? throw new DirectoryNotFoundException($"Could not resolve workspace root from '{bluejayRoot}'.");
|
||||||
|
|
||||||
|
var documents = ManifestRoots(workspaceRoot, bluejayRoot)
|
||||||
|
.SelectMany(LoadDocumentsFromRoot)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
return new ManifestInventory(workspaceRoot, bluejayRoot, documents);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string FindBluejayInfraRoot()
|
||||||
|
{
|
||||||
|
var current = new DirectoryInfo(AppContext.BaseDirectory);
|
||||||
|
while (current is not null)
|
||||||
|
{
|
||||||
|
if (Directory.Exists(Path.Combine(current.FullName, "apps"))
|
||||||
|
&& File.Exists(Path.Combine(current.FullName, "README.md")))
|
||||||
|
{
|
||||||
|
return current.FullName;
|
||||||
|
}
|
||||||
|
|
||||||
|
current = current.Parent;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new DirectoryNotFoundException("Could not find the bluejay-infra repository root from the test output directory.");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IEnumerable<string> ManifestRoots(string workspaceRoot, string bluejayRoot)
|
||||||
|
{
|
||||||
|
var roots = new[]
|
||||||
|
{
|
||||||
|
Path.Combine(bluejayRoot, "apps"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Chat", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.DMS", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.DNS", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Intranet.Web", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Kiosk", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Media", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.MenuBoard", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.MessageBoard", "k8s"),
|
||||||
|
// FlowerCore.Notes/k8s/selenium/ is the live Selenium Grid
|
||||||
|
// manifest tree (consumed by deploy-selenium scripts).
|
||||||
|
// FlowerCore.Notes/k8s/guacamole/ + FlowerCore.Notes/k8s/monitoring/
|
||||||
|
// are historical scaffolds that have diverged from the live state
|
||||||
|
// (bluejay-infra/apps/guacamole + bluejay-infra/apps/monitoring are
|
||||||
|
// canonical). Operator review is required before bringing them in
|
||||||
|
// line OR decommissioning them — keep them out of the lint scope
|
||||||
|
// until that decision lands. See xxl-regroup-2026-05-03-followup.md
|
||||||
|
// "Codex 7 §0 stop conditions" + the C7 close-session output.
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Notes", "k8s", "selenium"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.MySQL", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.PHP", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Presentations", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Print.Web", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.RemoteDesktop", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Scoreboard", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.SegmentDisplay", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.SignalControl", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.TtsReader", "k8s"),
|
||||||
|
Path.Combine(workspaceRoot, "FlowerCore.Updater", "k8s"),
|
||||||
|
};
|
||||||
|
|
||||||
|
return roots.Where(Directory.Exists);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IEnumerable<ManifestDocument> LoadDocumentsFromRoot(string root)
|
||||||
|
{
|
||||||
|
foreach (var filePath in Directory.EnumerateFiles(root, "*.yaml", SearchOption.AllDirectories))
|
||||||
|
{
|
||||||
|
var fileText = File.ReadAllText(filePath);
|
||||||
|
var segments = SplitManifestDocuments(fileText);
|
||||||
|
|
||||||
|
for (var index = 0; index < segments.Count; index++)
|
||||||
|
{
|
||||||
|
var yaml = new YamlStream();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
using var reader = new StringReader(segments[index]);
|
||||||
|
yaml.Load(reader);
|
||||||
|
}
|
||||||
|
catch (YamlException exception)
|
||||||
|
{
|
||||||
|
_ = exception;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (yaml.Documents.Count == 0)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (yaml.Documents[0].RootNode is YamlMappingNode mapping
|
||||||
|
&& ManifestNodeExtensions.Scalar(mapping, "kind") is not null)
|
||||||
|
{
|
||||||
|
yield return new ManifestDocument(root, filePath, index, fileText, mapping);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<string> SplitManifestDocuments(string fileText)
|
||||||
|
{
|
||||||
|
var documents = new List<string>();
|
||||||
|
var currentLines = new List<string>();
|
||||||
|
var seenApiVersion = false;
|
||||||
|
|
||||||
|
foreach (var line in Regex.Split(fileText, @"\r?\n"))
|
||||||
|
{
|
||||||
|
if (Regex.IsMatch(line, @"^\s*---\s*$"))
|
||||||
|
{
|
||||||
|
FlushCurrentDocument();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Regex.IsMatch(line, @"^\s*apiVersion:\s*")
|
||||||
|
&& seenApiVersion
|
||||||
|
&& currentLines.Any(existing => !string.IsNullOrWhiteSpace(existing)))
|
||||||
|
{
|
||||||
|
FlushCurrentDocument();
|
||||||
|
}
|
||||||
|
|
||||||
|
currentLines.Add(line);
|
||||||
|
if (Regex.IsMatch(line, @"^\s*apiVersion:\s*"))
|
||||||
|
{
|
||||||
|
seenApiVersion = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
FlushCurrentDocument();
|
||||||
|
return documents;
|
||||||
|
|
||||||
|
void FlushCurrentDocument()
|
||||||
|
{
|
||||||
|
var text = string.Join(Environment.NewLine, currentLines).Trim();
|
||||||
|
if (!string.IsNullOrWhiteSpace(text))
|
||||||
|
{
|
||||||
|
documents.Add(text);
|
||||||
|
}
|
||||||
|
|
||||||
|
currentLines.Clear();
|
||||||
|
seenApiVersion = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed record ManifestDocument(
|
||||||
|
string RootPath,
|
||||||
|
string FilePath,
|
||||||
|
int DocumentIndex,
|
||||||
|
string FileText,
|
||||||
|
YamlMappingNode Root)
|
||||||
|
{
|
||||||
|
public string Kind => Scalar("kind") ?? string.Empty;
|
||||||
|
|
||||||
|
public string Name => Scalar("metadata", "name") ?? $"document-{DocumentIndex}";
|
||||||
|
|
||||||
|
public string Namespace => Scalar("metadata", "namespace") ?? string.Empty;
|
||||||
|
|
||||||
|
public string RelativePath => Path.GetRelativePath(RootPath, FilePath).Replace('\\', '/');
|
||||||
|
|
||||||
|
public string Descriptor => $"{Kind} {Namespace}/{Name} [{RelativePath}#{DocumentIndex + 1}]";
|
||||||
|
|
||||||
|
public string? Scalar(params string[] path) => ManifestNodeExtensions.Scalar(Root, path);
|
||||||
|
|
||||||
|
public IReadOnlyList<YamlMappingNode> MappingSequence(params string[] path) => ManifestNodeExtensions.MappingSequence(Root, path);
|
||||||
|
|
||||||
|
public IEnumerable<string> AllScalars() => ManifestNodeExtensions.AllScalars(Root);
|
||||||
|
|
||||||
|
public IReadOnlyList<string> EgressPorts()
|
||||||
|
{
|
||||||
|
return MappingSequence("spec", "egress")
|
||||||
|
.SelectMany(egressRule => ManifestNodeExtensions.MappingSequence(egressRule, "ports"))
|
||||||
|
.Select(portMapping => ManifestNodeExtensions.Scalar(portMapping, "port"))
|
||||||
|
.Where(value => !string.IsNullOrWhiteSpace(value))
|
||||||
|
.Cast<string>()
|
||||||
|
.ToList();
|
||||||
|
}
|
||||||
|
|
||||||
|
public YamlMappingNode? PodSpec()
|
||||||
|
{
|
||||||
|
return Kind switch
|
||||||
|
{
|
||||||
|
"Deployment" or "StatefulSet" or "DaemonSet" or "Job" =>
|
||||||
|
ManifestNodeExtensions.Mapping(Root, "spec", "template", "spec"),
|
||||||
|
"CronJob" =>
|
||||||
|
ManifestNodeExtensions.Mapping(Root, "spec", "jobTemplate", "spec", "template", "spec"),
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
public IReadOnlyList<YamlMappingNode> ContainerMappings()
|
||||||
|
{
|
||||||
|
var podSpec = PodSpec();
|
||||||
|
if (podSpec is null)
|
||||||
|
{
|
||||||
|
return Array.Empty<YamlMappingNode>();
|
||||||
|
}
|
||||||
|
|
||||||
|
return ManifestNodeExtensions.MappingSequence(podSpec, "containers")
|
||||||
|
.Concat(ManifestNodeExtensions.MappingSequence(podSpec, "initContainers"))
|
||||||
|
.ToList();
|
||||||
|
}
|
||||||
|
|
||||||
|
public IReadOnlyList<ContainerSpec> ContainerSpecs()
|
||||||
|
{
|
||||||
|
return ContainerMappings()
|
||||||
|
.Select(container => new ContainerSpec(
|
||||||
|
ManifestNodeExtensions.Scalar(container, "name") ?? "<unnamed>",
|
||||||
|
ManifestNodeExtensions.Scalar(container, "image") ?? string.Empty,
|
||||||
|
ManifestNodeExtensions.Scalar(container, "imagePullPolicy") ?? string.Empty))
|
||||||
|
.ToList();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed record ContainerSpec(string Name, string Image, string ImagePullPolicy);
|
||||||
|
|
||||||
|
internal static class ManifestNodeExtensions
|
||||||
|
{
|
||||||
|
public static string? Scalar(this YamlMappingNode mapping, params string[] path)
|
||||||
|
{
|
||||||
|
return TryGetNode(mapping, path, out var node) && node is YamlScalarNode scalar
|
||||||
|
? scalar.Value
|
||||||
|
: null;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static YamlMappingNode? Mapping(this YamlMappingNode mapping, params string[] path)
|
||||||
|
{
|
||||||
|
return TryGetNode(mapping, path, out var node) ? node as YamlMappingNode : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static bool TryGetMapping(this YamlMappingNode mapping, string key, out YamlMappingNode result)
|
||||||
|
{
|
||||||
|
if (TryGetChild(mapping, key, out var child) && child is YamlMappingNode childMapping)
|
||||||
|
{
|
||||||
|
result = childMapping;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
result = null!;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static IReadOnlyList<YamlMappingNode> MappingSequence(this YamlMappingNode mapping, params string[] path)
|
||||||
|
{
|
||||||
|
return TryGetNode(mapping, path, out var node) && node is YamlSequenceNode sequence
|
||||||
|
? sequence.Children.OfType<YamlMappingNode>().ToList()
|
||||||
|
: Array.Empty<YamlMappingNode>();
|
||||||
|
}
|
||||||
|
|
||||||
|
public static IReadOnlyList<string> ScalarSequence(this YamlMappingNode mapping, params string[] path)
|
||||||
|
{
|
||||||
|
return TryGetNode(mapping, path, out var node) && node is YamlSequenceNode sequence
|
||||||
|
? sequence.Children.OfType<YamlScalarNode>()
|
||||||
|
.Select(child => child.Value)
|
||||||
|
.Where(value => !string.IsNullOrWhiteSpace(value))
|
||||||
|
.Cast<string>()
|
||||||
|
.ToList()
|
||||||
|
: Array.Empty<string>();
|
||||||
|
}
|
||||||
|
|
||||||
|
public static IEnumerable<string> AllScalars(YamlNode node)
|
||||||
|
{
|
||||||
|
return node switch
|
||||||
|
{
|
||||||
|
YamlScalarNode scalar when !string.IsNullOrWhiteSpace(scalar.Value) => new[] { scalar.Value! },
|
||||||
|
YamlSequenceNode sequence => sequence.Children.SelectMany(AllScalars),
|
||||||
|
YamlMappingNode mapping => mapping.Children.SelectMany(entry => AllScalars(entry.Key).Concat(AllScalars(entry.Value))),
|
||||||
|
_ => Array.Empty<string>(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool TryGetNode(YamlMappingNode mapping, IReadOnlyList<string> path, out YamlNode node)
|
||||||
|
{
|
||||||
|
YamlNode current = mapping;
|
||||||
|
foreach (var segment in path)
|
||||||
|
{
|
||||||
|
if (current is not YamlMappingNode currentMapping || !TryGetChild(currentMapping, segment, out current))
|
||||||
|
{
|
||||||
|
node = null!;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
node = current;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool TryGetChild(YamlMappingNode mapping, string key, out YamlNode value)
|
||||||
|
{
|
||||||
|
foreach (var entry in mapping.Children)
|
||||||
|
{
|
||||||
|
if (entry.Key is YamlScalarNode scalar
|
||||||
|
&& string.Equals(scalar.Value, key, StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
value = entry.Value;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
value = null!;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
package bluejayinfra.cross_namespace_ingressroute
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "IngressRoute"
|
||||||
|
ns := object.get(input.metadata, "namespace", "")
|
||||||
|
route := input.spec.routes[_]
|
||||||
|
service := route.services[_]
|
||||||
|
svc_ns := object.get(service, "namespace", "")
|
||||||
|
svc_ns != ""
|
||||||
|
svc_ns != ns
|
||||||
|
msg := sprintf("IngressRoute %s/%s references Service %s in namespace %s", [ns, input.metadata.name, service.name, svc_ns])
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
package bluejayinfra.public_method_allowlist
|
||||||
|
|
||||||
|
public_hosts := {"dist.flowercore.io", "dns.iamworkin.lan"}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "IngressRoute"
|
||||||
|
route := input.spec.routes[_]
|
||||||
|
match := object.get(route, "match", "")
|
||||||
|
host := public_hosts[_]
|
||||||
|
contains(match, sprintf("Host(`%s`)", [host]))
|
||||||
|
not contains(match, "Method(`GET`)")
|
||||||
|
msg := sprintf("IngressRoute %s/%s is missing Method(GET) for public read-only host %s", [input.metadata.namespace, input.metadata.name, host])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "IngressRoute"
|
||||||
|
route := input.spec.routes[_]
|
||||||
|
match := object.get(route, "match", "")
|
||||||
|
host := public_hosts[_]
|
||||||
|
contains(match, sprintf("Host(`%s`)", [host]))
|
||||||
|
not contains(match, "Method(`HEAD`)")
|
||||||
|
msg := sprintf("IngressRoute %s/%s is missing Method(HEAD) for public read-only host %s", [input.metadata.namespace, input.metadata.name, host])
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
package bluejayinfra.traefik_vip_backend_ports
|
||||||
|
|
||||||
|
has_vip {
|
||||||
|
some i
|
||||||
|
some j
|
||||||
|
input.spec.egress[i].to[j].ipBlock.cidr == "10.0.56.200/32"
|
||||||
|
}
|
||||||
|
|
||||||
|
has_port(port) {
|
||||||
|
some i
|
||||||
|
some j
|
||||||
|
input.spec.egress[i].ports[j].port == port
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "NetworkPolicy"
|
||||||
|
has_vip
|
||||||
|
has_port(443)
|
||||||
|
not has_port(8443)
|
||||||
|
msg := sprintf("NetworkPolicy %s/%s allows 10.0.56.200:443 without backend port 8443", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "NetworkPolicy"
|
||||||
|
has_vip
|
||||||
|
has_port(80)
|
||||||
|
not has_port(8080)
|
||||||
|
not has_port(8000)
|
||||||
|
msg := sprintf("NetworkPolicy %s/%s allows 10.0.56.200:80 without backend HTTP port 8080 or 8000", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
package bluejayinfra.auth_probe_path
|
||||||
|
|
||||||
|
protected_deployments := {
|
||||||
|
"messageboard-web",
|
||||||
|
"scoreboard-web",
|
||||||
|
"segmentdisplay-web",
|
||||||
|
"signalcontrol-web",
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "Deployment"
|
||||||
|
protected_deployments[input.metadata.name]
|
||||||
|
container := input.spec.template.spec.containers[_]
|
||||||
|
probe := object.get(container, "readinessProbe", {})
|
||||||
|
http_get := object.get(probe, "httpGet", {})
|
||||||
|
object.get(http_get, "path", "") == "/health"
|
||||||
|
msg := sprintf("Deployment %s/%s must not use readinessProbe.httpGet /health behind API key middleware", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "Deployment"
|
||||||
|
protected_deployments[input.metadata.name]
|
||||||
|
container := input.spec.template.spec.containers[_]
|
||||||
|
probe := object.get(container, "livenessProbe", {})
|
||||||
|
http_get := object.get(probe, "httpGet", {})
|
||||||
|
object.get(http_get, "path", "") == "/health"
|
||||||
|
msg := sprintf("Deployment %s/%s must not use livenessProbe.httpGet /health behind API key middleware", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
package bluejayinfra.statefulset_volumeclaim_defaults
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "StatefulSet"
|
||||||
|
count(object.get(input.spec, "volumeClaimTemplates", [])) > 0
|
||||||
|
object.get(input.spec, "podManagementPolicy", "") == ""
|
||||||
|
msg := sprintf("StatefulSet %s/%s is missing spec.podManagementPolicy", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "StatefulSet"
|
||||||
|
count(object.get(input.spec, "volumeClaimTemplates", [])) > 0
|
||||||
|
object.get(input.spec, "revisionHistoryLimit", 0) == 0
|
||||||
|
msg := sprintf("StatefulSet %s/%s is missing spec.revisionHistoryLimit", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "StatefulSet"
|
||||||
|
claim := input.spec.volumeClaimTemplates[_]
|
||||||
|
object.get(claim.spec, "volumeMode", "") != "Filesystem"
|
||||||
|
claim_name := object.get(claim.metadata, "name", "<unnamed>")
|
||||||
|
msg := sprintf("StatefulSet %s/%s volumeClaimTemplate %s is missing volumeMode: Filesystem", [input.metadata.namespace, input.metadata.name, claim_name])
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
package bluejayinfra.localhost_image_pull_policy
|
||||||
|
|
||||||
|
pod_spec(spec) = pod {
|
||||||
|
input.kind == "Deployment"
|
||||||
|
pod := spec.template.spec
|
||||||
|
}
|
||||||
|
|
||||||
|
pod_spec(spec) = pod {
|
||||||
|
input.kind == "StatefulSet"
|
||||||
|
pod := spec.template.spec
|
||||||
|
}
|
||||||
|
|
||||||
|
pod_spec(spec) = pod {
|
||||||
|
input.kind == "DaemonSet"
|
||||||
|
pod := spec.template.spec
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
pod := pod_spec(input.spec)
|
||||||
|
container := pod.containers[_]
|
||||||
|
startswith(object.get(container, "image", ""), "localhost/")
|
||||||
|
object.get(container, "imagePullPolicy", "") != "Never"
|
||||||
|
msg := sprintf("%s/%s container %s uses a localhost image without imagePullPolicy: Never", [input.metadata.namespace, input.metadata.name, container.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
pod := pod_spec(input.spec)
|
||||||
|
container := pod.initContainers[_]
|
||||||
|
startswith(object.get(container, "image", ""), "localhost/")
|
||||||
|
object.get(container, "imagePullPolicy", "") != "Never"
|
||||||
|
msg := sprintf("%s/%s initContainer %s uses a localhost image without imagePullPolicy: Never", [input.metadata.namespace, input.metadata.name, container.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
pod := pod_spec(input.spec)
|
||||||
|
container := pod.containers[_]
|
||||||
|
startswith(object.get(container, "image", ""), "fc-")
|
||||||
|
not contains(object.get(container, "image", ""), "/")
|
||||||
|
msg := sprintf("%s/%s container %s uses a non-localhost FlowerCore image reference %s", [input.metadata.namespace, input.metadata.name, container.name, container.image])
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
package bluejayinfra.public_egress_dns_none
|
||||||
|
|
||||||
|
public_egress_workloads := {
|
||||||
|
"asterisk",
|
||||||
|
"fc-llm-bridge",
|
||||||
|
"mysql-web",
|
||||||
|
"php-web",
|
||||||
|
"ttsreader-align",
|
||||||
|
"ttsreader-kokoro",
|
||||||
|
"ttsreader-modern",
|
||||||
|
"ttsreader-piper",
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "Deployment"
|
||||||
|
public_egress_workloads[input.metadata.name]
|
||||||
|
object.get(input.spec.template.spec, "dnsPolicy", "") != "None"
|
||||||
|
msg := sprintf("Deployment %s/%s must set dnsPolicy: None for public-internet egress", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "Deployment"
|
||||||
|
public_egress_workloads[input.metadata.name]
|
||||||
|
search := object.get(object.get(input.spec.template.spec, "dnsConfig", {}), "searches", [])[_]
|
||||||
|
contains(lower(search), "iamworkin.lan")
|
||||||
|
msg := sprintf("Deployment %s/%s must not include iamworkin.lan in dnsConfig.searches", [input.metadata.namespace, input.metadata.name])
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
package bluejayinfra.public_readwrite_allowlist
|
||||||
|
|
||||||
|
# Public hosts that allow a tightly bounded write surface in addition to
|
||||||
|
# GET/HEAD. updatecenter.iamworkin.lan accepts POST /api/v1/checkin/{id}
|
||||||
|
# (bootstrap-JWT) so its allowlist is GET||HEAD||POST||OPTIONS — but
|
||||||
|
# PUT/PATCH/DELETE must still 404 at the route. Any host in this set MUST
|
||||||
|
# include all four required methods AND MUST NOT include any forbidden
|
||||||
|
# method.
|
||||||
|
public_readwrite_hosts := {"updatecenter.iamworkin.lan", "updates.iamworkin.lan"}
|
||||||
|
|
||||||
|
required_methods := {"GET", "HEAD", "POST", "OPTIONS"}
|
||||||
|
|
||||||
|
forbidden_methods := {"PUT", "PATCH", "DELETE"}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "IngressRoute"
|
||||||
|
route := input.spec.routes[_]
|
||||||
|
match := object.get(route, "match", "")
|
||||||
|
host := public_readwrite_hosts[_]
|
||||||
|
contains(match, sprintf("Host(`%s`)", [host]))
|
||||||
|
required := required_methods[_]
|
||||||
|
not contains(match, sprintf("Method(`%s`)", [required]))
|
||||||
|
msg := sprintf("IngressRoute %s/%s is missing required Method(%s) for public read-write host %s", [input.metadata.namespace, input.metadata.name, required, host])
|
||||||
|
}
|
||||||
|
|
||||||
|
deny[msg] {
|
||||||
|
input.kind == "IngressRoute"
|
||||||
|
route := input.spec.routes[_]
|
||||||
|
match := object.get(route, "match", "")
|
||||||
|
host := public_readwrite_hosts[_]
|
||||||
|
contains(match, sprintf("Host(`%s`)", [host]))
|
||||||
|
forbidden := forbidden_methods[_]
|
||||||
|
contains(match, sprintf("Method(`%s`)", [forbidden]))
|
||||||
|
msg := sprintf("IngressRoute %s/%s must not include Method(%s) on public read-write host %s", [input.metadata.namespace, input.metadata.name, forbidden, host])
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user