infra: export appset and mirror alert polish
This commit is contained in:
@@ -1244,6 +1244,58 @@ data:
|
||||
summary: "Marquee animation duration drifting > 10% on {{ $labels.renderer }} ({{ $labels.phase }})"
|
||||
description: "Median observed cycle duration deviates from target DurationMs by >10%. Could indicate browser tab throttling, GPU pressure, or phase-advancement bug."
|
||||
|
||||
# ============================================================
|
||||
# Update Center public-edge probes
|
||||
# Live-mirrored from FlowerCore.Notes/scripts/monitoring/alerts.yml.
|
||||
# This K8s ConfigMap is the future migration target; live Prometheus
|
||||
# still reads the canonical Notes file from noc1 Podman.
|
||||
# ============================================================
|
||||
- name: update_center
|
||||
rules:
|
||||
# Critical only when the edge is genuinely unreachable. A Cloudflare
|
||||
# HTTP 429 means the prober hit a rate-limit, not that real clients
|
||||
# are down, so the warning rule below owns that signal.
|
||||
- alert: UpdateCenterPublicEdgeDown
|
||||
expr: |
|
||||
(probe_success{job="probe-update-center-public-edge"} == 0)
|
||||
unless on(instance)
|
||||
(probe_http_status_code{job="probe-update-center-public-edge"} == 429)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
service: update-center
|
||||
alert_channel: irc
|
||||
annotations:
|
||||
summary: "Update Center public edge probe failed for {{ $labels.instance }}"
|
||||
description: >-
|
||||
The external probe for {{ $labels.instance }} failed for 10 minutes with a
|
||||
non-2xx status that is not a rate-limit. Public Update Center clients may be
|
||||
unable to fetch manifest schema metadata through Cloudflare.
|
||||
runbook: >-
|
||||
1. curl -sk https://{{ $labels.instance }}/api/v1/manifests/_schema
|
||||
2. Verify Cloudflare DNS record is proxied and targets the current public edge IP
|
||||
3. kubectl -n fc-updater get ingressroute updatecenter-web-public secret cf-origin-flowercore-io
|
||||
4. Check Traefik logs for Method() or TLS secret errors
|
||||
|
||||
- alert: UpdateCenterPublicEdgeRateLimited
|
||||
expr: probe_http_status_code{job="probe-update-center-public-edge"} == 429
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
service: update-center
|
||||
alert_channel: irc
|
||||
annotations:
|
||||
summary: "Cloudflare is rate-limiting (HTTP 429) the public-edge probe for {{ $labels.instance }}"
|
||||
description: >-
|
||||
The blackbox prober receives HTTP 429 from Cloudflare for {{ $labels.instance }}
|
||||
while the origin is healthy. This is a Cloudflare rate-limit / WAF condition on
|
||||
the public hostname, not an outage.
|
||||
runbook: >-
|
||||
1. curl -sk https://{{ $labels.instance }}/api/v1/manifests/_schema (expect 200 from a normal client)
|
||||
2. Review Cloudflare rate-limit / WAF rules for the hostname; the 5m-cadence prober is tripping a 429
|
||||
3. Add a Cloudflare rate-limit exception for the prober source IP or the /api/v1/manifests/_schema path
|
||||
4. Confirm whether the singular host update.flowercore.io is still required, or only updates.flowercore.io
|
||||
|
||||
# =============================================================================
|
||||
# ConfigMap: Blackbox Exporter Configuration
|
||||
# =============================================================================
|
||||
|
||||
Reference in New Issue
Block a user