fix(ttsreader-kokoro): bump liveness probe timeouts (Sprint E Phase 1a)
Kokoro pod has 4 restarts in 2d6h with exit 143 (SIGTERM from kubelet). kubectl describe events all show: Liveness probe failed: Get "http://10.42.229.109:8880/v1/audio/voices": context deadline exceeded The probe path /v1/audio/voices shares the FastAPI worker pool with /v1/audio/speech. A long synth (Bible chapter, 30+ sentences) holds the pool past the prior 5s × 3 = 15s probe window, kubelet kills the pod, in-flight renders fail. Operator hits "fallback chain failed" toasts + partial-render breadcrumbs during these windows. Bump probe timeoutSeconds 5 → 15 and failureThreshold 3 → 5 → 75 s of grace before kubelet gives up. Combined with the kokoro-side circuit breaker landing in TtsReader (Sprint E Phase 1b), the FC backend will also stop slamming kokoro during recovery so it can serve the probe even faster. The companion Prometheus alerts (KokoroPodFlapping, PiperPodFlapping) land in FlowerCore.Notes/scripts/monitoring/alerts.yml. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -296,14 +296,23 @@ spec:
|
|||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 5
|
||||||
failureThreshold: 18
|
failureThreshold: 18
|
||||||
|
# Sprint E Phase 1a (kokoro stability) — 4 restarts in 2d6h with
|
||||||
|
# exit 143 traced to liveness probe `context deadline exceeded` while
|
||||||
|
# kokoro was busy synthesizing. /v1/audio/voices shares the FastAPI
|
||||||
|
# worker pool with /v1/audio/speech, so a long synth can starve the
|
||||||
|
# probe out within the prior 5s × 3 = 15s window. Bump timeoutSeconds
|
||||||
|
# 5 → 15 and failureThreshold 3 → 5 → 75s grace before kubelet kills
|
||||||
|
# the pod. The TtsCircuitBreaker on the synthesizer side (Phase 1b)
|
||||||
|
# backs this up so the FC backend stops slamming kokoro during
|
||||||
|
# recovery.
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet:
|
httpGet:
|
||||||
path: /v1/audio/voices
|
path: /v1/audio/voices
|
||||||
port: 8880
|
port: 8880
|
||||||
initialDelaySeconds: 180
|
initialDelaySeconds: 180
|
||||||
periodSeconds: 30
|
periodSeconds: 30
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 15
|
||||||
failureThreshold: 3
|
failureThreshold: 5
|
||||||
---
|
---
|
||||||
# fc-biblical-tts — eSpeak-NG-backed Ancient Greek + Hebrew TTS with
|
# fc-biblical-tts — eSpeak-NG-backed Ancient Greek + Hebrew TTS with
|
||||||
# word-level timing for read-along playback. Companion to ttsreader-kokoro
|
# word-level timing for read-along playback. Companion to ttsreader-kokoro
|
||||||
|
|||||||
Reference in New Issue
Block a user