93 Commits

Author SHA1 Message Date
Andrew Stoltz
f0cb7a5e81 fix(hardening): align probe-path annotations with live health routes 2026-06-04 22:01:04 -05:00
Andrew Stoltz
c4b08f41ab feat(infra): prestage broader app exposure hardening 2026-06-04 18:14:22 -05:00
Andrew Stoltz
e2e93d482c Deploy TtsReader schema repair image
Co-Authored-By: Codex <codex@openai.com>
2026-06-02 22:00:15 -05:00
Andrew Stoltz
2bf339ce51 Deploy TtsReader PR29 live proof image
Co-Authored-By: Codex <codex@openai.com>
2026-06-02 21:47:04 -05:00
Andrew Stoltz
47e2256556 Deploy TtsReader correction bridge images 2026-05-31 12:35:45 -05:00
Andrew Stoltz
200aeab032 ttsreader: deploy study mode repair image 2026-05-18 16:33:08 -05:00
Andrew Stoltz
8182616d4c ttsreader: point render piper to edge1 demo endpoint 2026-05-18 16:06:37 -05:00
Andrew Stoltz
f0862ac03c ttsreader: deploy sprint36 demo audio image 2026-05-18 16:04:59 -05:00
Codex
c4065b15a3 deploy(ttsreader): persist voice reference clips on pvc 2026-05-06 20:48:58 -05:00
Codex
c2eb37dee9 deploy(ttsreader): enable phase6 biblical routing 2026-05-06 19:46:25 -05:00
Codex
6a89a76e39 fc-ttsreader: bump image to v202605061500 (Phase 4delta enrichment pipeline)
Phase 4delta server-side HTML overlay enrichment landed in
FlowerCore.TtsReader@8f23e15 (master @6091618). Adds 9-pass enrichment +
SQLite-backed cache + 4 REST endpoints (/api/v1/enrich/{html,jsonld,both,passes})
+ RenderRequest.sourceJsonLd. Tests 476 -> 522 (+46). Image already imported
to all RKE2 nodes via deploy.sh; this bumps the bluejay-infra-managed tag so
ArgoCD reconciles the live deployment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 16:04:31 -05:00
Codex
c0dceafffd deploy(ttsreader): roll web v20260506-47a88ae 2026-05-06 14:40:57 -05:00
Codex
0b52093b36 K8s manifest hardening + new bluejay-infra-lint test project
Manifest hardening (per documented memories):
- apps/asterisk/deployment.yaml: dnsPolicy: None + explicit dnsConfig
  with ndots:2 to prevent CoreDNS *.iamworkin.lan template from
  hijacking external egress (downloads.asterisk.org).
- apps/fc-llm-bridge/fc-llm-bridge.yaml: same dnsConfig pattern for
  api.anthropic.com egress.
- apps/fc-ttsreader/fc-ttsreader.yaml: same dnsConfig pattern for
  huggingface.co model seeding.
- apps/fc-messageboard/fc-messageboard.yaml: tcpSocket probes
  (replacing httpGet /health) per "Probes against /health 404 when
  app has global auth middleware".
- apps/fc-signalcontrol/fc-signalcontrol.yaml: same tcpSocket probe
  fix.

New lint project:
- tests/bluejay-infra-lint/BluejayInfraLint.Tests.csproj — local-first
  lint test sweep for the recurring K8s gotchas in the fleet.
- tests/bluejay-infra-lint/FleetManifestLintTests.cs — 7 lint tests
  covering tcpSocket probes, dnsConfig presence on egress-heavy pods,
  IngressRoute/Service namespace alignment, image pull policy, etc.
- tests/bluejay-infra-lint/conftest.dev/ — matching conftest policies
  for environments with conftest/opa.
- .gitignore — adds bin/ + obj/ + DS_Store/swp.

README.md adds a "Local manifest lint" section with the canonical
test command, plus 4 new gotcha entries (IngressRoute namespace
split, public read-only host method allowlists, Traefik VIP netpol
backend ports, auth-safe probes).

Tests: 7 / 7 lint tests passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 03:18:04 -05:00
Codex
7a9098d3bd fix(fc-ttsreader): lower web cpu request 2026-05-04 02:28:11 -05:00
Andrew Stoltz
9ec2e2d52e deploy(ttsreader): bump web image to b6ca2d5 2026-04-30 12:43:48 -05:00
Andrew Stoltz
b4d62a8a50 deploy(fc-ttsreader): roll chapter-context image 2026-04-30 02:31:55 -05:00
Andrew Stoltz
c9bf23834b chore(ttsreader): bump image to v202604291817
Per-profile MoodAnnotationModelOverride picker — Profiles page now shows
a model dropdown from IModelRegistry instead of a free-text field; model
override null-falls-back to global TtsReader:Ollama:DefaultModel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 13:21:40 -05:00
Andrew Stoltz
f604ab419e feat(ttsreader): bump image to v202604281923 (SignalR ProgressHub)
Adds ProgressHub endpoint at /hubs/progress with project-scoped
group broadcasting for JobStarted, CueProgress, JobCompleted, and
JobFailed events.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 19:30:41 -05:00
Andrew Stoltz
b2786252b0 chore(ttsreader): bump web image to v202604281831 (ops failed-manifest cleanup)
Deploys fix for stale Failed manifest accumulation in TTS Reader Ops view
and atomic-write guard against empty/corrupt job manifests.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 18:31:53 -05:00
Andrew Stoltz
45ee40920d fix(ttsreader): bump image to v202604281638 (Range support + Ollama timeout 240s) 2026-04-28 16:44:57 -05:00
Andrew Stoltz
8ad7eb714b fix(ttsreader): bump image to v202604281542 (annotation few-shot prompt + UI hint) 2026-04-28 15:46:28 -05:00
Andrew Stoltz
c17af882cc fix(ttsreader): bump image to v202604281444 for UX polish (cross-chapter Bible passage, /profiles dedup, /ops table) 2026-04-28 14:48:13 -05:00
Andrew Stoltz
76b1938afa fix(ttsreader): bump image to v202604281434 for live playback regression patch (study-player + speech override synth) 2026-04-28 14:43:06 -05:00
Andrew Stoltz
f2258b92a2 fc-ttsreader: bump web image to v202604280946 + add Render__CdnDirectory env
Sprint E XXL Phase 4γ MVP deploy — POST /api/v1/render endpoint.

Two changes:
1. Image tag v202604272339 → v202604280946 (TtsReader@d9e0a58 master tip
   includes the new RenderController + RenderService + 9 tests).
2. New TtsReader__Render__CdnDirectory=/data/cdn env var. Default
   wwwroot/cdn resolves under the read-only app filesystem when
   runAsNonRoot=true; pin to the existing writable PVC mount alongside
   other TtsReader runtime data. Manifests + cue audio land at
   /data/cdn/sha256/<hash>/manifest.json + cues/.

Pre-existing PVC mount at /data/ already covers this — no PVC change
needed, just the env var override.

Pairs with TtsReader@d9e0a58 master tip (ready for image build + import).
2026-04-28 09:47:46 -05:00
Andrew Stoltz
0df8f7b936 chore(ttsreader): bump fc-ttsreader-web to v202604272339 (Sprint E Phase C — partial-render UX)
TtsReader@9333480: distinguishes partial-render (yellow Warning, audio
plays, 'Re-render N failed sentences' button) from full-fail (red
Danger, 'Try render again'). New TtsFallbackChainFailedException carries
both voices when Kokoro + Piper both fail; chapter breadcrumb names
the entire chain instead of just the requested voice. +8 tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 23:40:19 -05:00
Andrew Stoltz
38558641c1 fix(ttsreader-kokoro): bump liveness probe timeouts (Sprint E Phase 1a)
Kokoro pod has 4 restarts in 2d6h with exit 143 (SIGTERM from kubelet).
kubectl describe events all show:

  Liveness probe failed: Get "http://10.42.229.109:8880/v1/audio/voices":
    context deadline exceeded

The probe path /v1/audio/voices shares the FastAPI worker pool with
/v1/audio/speech. A long synth (Bible chapter, 30+ sentences) holds the
pool past the prior 5s × 3 = 15s probe window, kubelet kills the pod,
in-flight renders fail. Operator hits "fallback chain failed" toasts +
partial-render breadcrumbs during these windows.

Bump probe timeoutSeconds 5 → 15 and failureThreshold 3 → 5 → 75 s of
grace before kubelet gives up. Combined with the kokoro-side circuit
breaker landing in TtsReader (Sprint E Phase 1b), the FC backend will
also stop slamming kokoro during recovery so it can serve the probe
even faster.

The companion Prometheus alerts (KokoroPodFlapping, PiperPodFlapping)
land in FlowerCore.Notes/scripts/monitoring/alerts.yml.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 23:28:07 -05:00
Andrew Stoltz
63d905b4df chore(ttsreader): bump fc-ttsreader-web to v202604272236 (Thinking + Feedback ALTERs) 2026-04-27 22:37:08 -05:00
Andrew Stoltz
d95f4e0caf chore(ttsreader): bump fc-ttsreader-web to v202604272228 (ChatSessions IsFavorite ALTER hotfix) 2026-04-27 22:28:56 -05:00
Andrew Stoltz
7bc565d17e fix(ttsreader): pin VoicePreview CacheDirectory to /data PVC
Day 8 disk-cache warmer crashes on production with
'Read-only file system : /home/app/data' because the relative default
'data/voice-previews' resolves under runAsNonRoot HOME (read-only with
readOnlyRootFilesystem=true). Pin to /data/voice-previews so the cache
lands on the writable PVC mount alongside ttsreader.db, audio output,
and jobs root.

Image v202604272216 (already on nodes) is unaffected by this — only
the env routing changes. ArgoCD reconciles + rollout restart picks up
the new env without rebuild.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 22:24:04 -05:00
Andrew Stoltz
dfe9c3b67e chore(ttsreader): bump fc-ttsreader-web to v202604272216 (brace-escape fix) 2026-04-27 22:16:19 -05:00
Andrew Stoltz
37f8db89e4 chore(ttsreader): bump fc-ttsreader-web to v202604272208 (Day 10 + VoiceProfiles hotfix)
v202604272157 crash-looped on the production PVC because Database.EnsureCreated()
is a no-op on existing DBs and the VoiceProfiles table was missing. TtsReader@a9f0b73
adds an idempotent CREATE TABLE IF NOT EXISTS to the infra reconciler before
TtsReaderDataSeeder runs. Bumping the manifest to pick up that fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 22:09:08 -05:00
Andrew Stoltz
00c7d8df24 chore(ttsreader): bump fc-ttsreader-web to v202604272157 (Sprint E Day 10 UX polish)
Compact project page (Setup chip strip + chapter inspect-toggle drawer)
+ render feedback (rolling ETA strip + active-chapter pulse) + Bible
Dashboard navigates to /projects/{id} on queue. Source TtsReader@79de78b.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 21:58:12 -05:00
Andrew Stoltz
2d3fd74bab fc-ttsreader: bump web to v202604252002 (alignment Status guard relaxed) 2026-04-25 20:06:26 -05:00
Andrew Stoltz
df4e1f78b0 fc-ttsreader: bump web to v202604251956 (XL: per-chapter annotate + word-level alignment + Study TOC + Resume row) 2026-04-25 19:59:56 -05:00
Andrew Stoltz
2a10b775a8 fc-ttsreader: bump web to v202604251935 (Slice 5: select-to-annotate pronunciation + mood) 2026-04-25 19:39:27 -05:00
Andrew Stoltz
447ddd339d fc-ttsreader: bump web to v202604251917 (Bible passage shorthand: 'Esther 1' etc.) 2026-04-25 19:21:00 -05:00
Andrew Stoltz
7833143c1c fc-ttsreader: bump web to v202604251903 (Slices 2/3/4 + Lane I MCP + Lane J pills) 2026-04-25 19:08:08 -05:00
Andrew Stoltz
8ed77c4627 fc-ttsreader: bump web to v202604251836 (seek-race + auto-scroll + active-cue contrast) 2026-04-25 18:41:17 -05:00
Andrew Stoltz
437f346aee fc-ttsreader: register ttsreader-modern Deployment + Service
Adds the Deployment + Service for the fc-modern-tts container that
landed in the previous commit. Same shape as ttsreader-biblical:
runAsNonRoot uid 1654, dnsPolicy: None to bypass the iamworkin.lan
hijack on Microsoft endpoint lookups, /health probes, modest CPU/mem
since edge-tts is network-bound.

Service surfaces ttsreader-modern.fc-ttsreader.svc:10403 for the web
pod to call when the operator picks a he-IL-* or el-GR-* voice.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 18:39:58 -05:00
Andrew Stoltz
bc32b5ef04 fc-ttsreader: deploy fc-modern-tts (Edge Read Aloud Hebrew/Greek)
Adds a fourth TTS engine alongside Piper / Kokoro / biblical-tts: a
small FastAPI bridge to Microsoft Edge's Read Aloud TTS via the
edge-tts Python package. Provides studio-quality Modern Hebrew (he-IL)
and Modern Greek (el-GR) narrators for the cluster.

modern-tts/Dockerfile + app.py:
- Python 3.12 base + edge-tts==7.2.8 (older versions hit 403 from MS).
- POST /tts -> MP3 audio (audio/mpeg).
- POST /timings -> word-level timings. Edge sometimes omits WordBoundary
  events for non-English voices; fall back to MP3-frame-walking duration
  estimate + proportional distribution across whitespace-split words
  (same approach biblical-tts uses for eSpeak).
- GET /voices?language=all|default — filtered to he-/el- by default so
  the AiStation voice picker isn't overwhelmed by 400+ voices.
- GET /health for probes.
- Body shape mirrors BiblicalTtsRequest so the .NET client lives in the
  same FlowerCore.Shared.Speech package.

K8s deployment in fc-ttsreader namespace:
- ttsreader-modern Deployment + Service on port 10403.
- localhost/fc-modern-tts:v1, imagePullPolicy: Never (built on noc1,
  imported to all 3 RKE2 nodes via ctr).
- runAsNonRoot uid 1654 + fsGroup 1654.
- dnsPolicy: None to bypass the *.iamworkin.lan template hijack on
  Microsoft endpoint lookups.
- Modest resources (100m/128Mi req, 1000m/512Mi limit) — edge-tts is
  network-bound, not compute-bound.
- Probes against /health.

Verified live locally: container handles 'Καλημέρα Ελλάδα Πώς είστε'
in 2496ms, returns el-GR-NestorasNeural voice + 4 word timings.
Hebrew: 'בְּרֵאשִׁית בָּרָא אֱלֹהִים' returns he-IL-AvriNeural,
2472ms, 3 words.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 18:39:21 -05:00
Andrew Stoltz
263d06acb9 fc-ttsreader: bump web to v202604251750 (Lane H Slice 1: inline Study view + chapter notes) 2026-04-25 17:54:09 -05:00
Andrew Stoltz
25dbb2967f fc-ttsreader: bump web to v202604251714 (BuildRenderPlan splits each mood block via SpeechSentenceSegmenter) 2026-04-25 17:18:09 -05:00
Andrew Stoltz
a89a774eaf fc-ttsreader: deploy eSpeak-NG biblical-tts (Ancient Greek + Hebrew)
Adds a third TTS engine alongside Piper (modern English/multi-lang) and
Kokoro (high-quality English): a small FastAPI wrapper around eSpeak-NG
with built-in support for Ancient Greek (grc), Hebrew (he), and Modern
Greek (el). Same shape as fc-speech-align so AiStation talks to all the
TTS/alignment services with one HTTP client pattern.

biblical-tts/Dockerfile + app.py:
- Python 3.12 base + apt-get espeak-ng + libsndfile1 + ffmpeg-free deps.
- POST /tts -> WAV audio bytes (audio/wav).
- POST /timings -> word-level timings derived from espeak's --pho phoneme
  duration stream, distributed across whitespace-split words proportional
  to character count. Accuracy is good enough for chip-level read-along
  highlighting (~30-80ms per-word jitter).
- GET /voices for catalog discovery, GET /health for probes.
- Body shape mirrors AlignmentRequest from FlowerCore.Shared.Speech so
  the .NET BiblicalTtsClient round-trips it cleanly.

K8s deployment in fc-ttsreader namespace:
- ttsreader-biblical Deployment + Service on port 10402.
- localhost/fc-biblical-tts:v1, imagePullPolicy: Never (built on noc1,
  imported to all 3 RKE2 nodes via ctr).
- runAsNonRoot uid 1654 to match the namespace's standard security ctx.
- Modest resources (100m/128Mi req, 1000m/512Mi limit) — eSpeak is
  CPU-cheap.
- Probes hit /health which returns the supported language list.

Verified live: container started, /health returns ok with grc/el/he,
POST /timings on Ἐν ἀρχῇ ἦν ὁ λόγος returned 5 words / 1714ms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 17:17:38 -05:00
Andrew Stoltz
dc39747f3f fc-ttsreader: piper memory 1Gi -> 3Gi to stop OOMKill mid-render 2026-04-25 17:10:20 -05:00
Andrew Stoltz
87050e72a9 fc-ttsreader: deploy Kokoro to the cluster (replaces BLUEJAY-WS host pointer)
The cluster ttsreader-web was reaching across to BLUEJAY-WS:10401 for
Kokoro synthesis, which meant a workstation-down event broke render-
pipeline TTS. Add a cluster-native ttsreader-kokoro Deployment and
Service inside fc-ttsreader so the cluster owns the engine.

- Image: ghcr.io/remsky/kokoro-fastapi-cpu:latest. Model + 67 voices
  ship inside the image, so no PVC is required.
- Port 8880 (the kokoro-fastapi default; the entrypoint hardcodes it).
- Resources: 250m/1Gi request, 2000m/3Gi limit. CPU-only inference
  matches what AiStation runs locally on BLUEJAY-WS.
- dnsPolicy: None to bypass CoreDNS's *.iamworkin.lan template hijack
  on huggingface.co lookups, same shape as ttsreader-align.
- Probes hit /v1/audio/voices since the kokoro server doesn't expose
  /health; that endpoint is cheap (lists configured voice files).

ttsreader-web env var TtsReader__Kokoro__BaseUrl flips from the
workstation pointer to the cluster service:
http://ttsreader-kokoro.fc-ttsreader.svc.cluster.local.:8880.

AiStation keeps its local http://localhost:8880 since the workstation
operator still wants the audio to render on the local sound device
without a network hop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 16:56:39 -05:00
Andrew Stoltz
e8c5d2afd2 fc-ttsreader: bump web to v202604251544 (Unicode sanitize + continue-on-segment-fail) 2026-04-25 15:50:16 -05:00
Andrew Stoltz
eef492125f fc-ttsreader: bump web to v202604251534 (SignalR 8MB + DOM-peek + poll loop refresh fix) 2026-04-25 15:39:18 -05:00
Andrew Stoltz
b51ee35bfa fc-speech-align: v3 — emit FlowerCore.Shared.Speech word contract
The /align endpoint was returning Whisper-native word fields
(word/startSeconds/endSeconds/confidence), but FlowerCore.Shared.Speech's
FasterWhisperAlignmentClient on master deserializes
FasterWhisperWord against [JsonPropertyName("text")/("startMs")/("endMs")].
Result: ttsreader-web reported alignment.source="whisper" with words[]
present but every entry had Text="" and StartMs=EndMs=0 — visible in the
2026-04-25 hello-world smoke against ttsreader.iamworkin.lan.

Match the published Common contract instead of the Python model's native
shape: emit text/startMs/endMs (millisecond ints, not float seconds).
Confidence stays on the wire as informational; the deployed C# client
ignores it but a future fc-align operator UI can surface low-confidence
words. Bump tag to v3 and bump the Deployment image accordingly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 11:52:14 -05:00
Andrew Stoltz
4abc2fa95d fc-speech-align: add dnsPolicy: None to bypass CoreDNS *.iamworkin.lan template hijack on huggingface.co 2026-04-25 11:12:21 -05:00
Andrew Stoltz
d7628a6945 fc-speech-align: bump to v2 with explicit requests dep (faster-whisper 1.0.3 missing transitive) 2026-04-25 10:55:51 -05:00