Skip to main content
Research findings behind infra/ansible/roles/observability, cmd/prober, cmd/sloreport and the alert rules. Everything was read from the pinned sources in the Go module cache or the PyPI sdist on 2026-11-13, and the metric names were confirmed on the local test stack (tests/observability).

Metrics by component

The exporter has no first-message timestamp, so “age of the oldest message” is derived: WEB_EVENTS and API_METRICS are work queues, so their first sequence is the oldest unacknowledged message, and a first sequence that did not move for 10 minutes while the stream was never empty means that message is at least 10 minutes old (rule QueueOldestMessageTooOld).

Discord receivers

  • Alertmanager 0.34.2: discord_configs with webhook_url or webhook_url_file (exactly one), title, message, content, username, avatar_url, send_resolved. Title is cut at 256 runes, message at 4096, content at 2000 (notify/discord). The role uses webhook_url_file, so the secret stays in a 0400 file.
  • Gatus 5.38.0: alerting.discord with webhook-url, title, message-content, default-alert, and per-group overrides (alerting/provider/discord). The URL is read from the environment (${GATUS_DISCORD_WEBHOOK_URL}).
  • Neither validates the host of the URL, so the local test stack points both at a labelled test receiver (tests/observability/receiver) and exercises the real Discord code paths. Real delivery needs input I8.

Gatus

  • External endpoints: external-endpoints[] with name, group, token, alerts, heartbeat.interval (at least 10 s). Results are pushed with POST /api/v1/endpoints/{group}_{name}/external?success=&error=&duration= and a bearer token (api/external_endpoint.go); the key lowercases and replaces / _ . , space # + & with - (config/key/key.go). The result’s timestamp is the time Gatus received it.
  • Retention: storage.maximum-number-of-results per endpoint (default 100) and maximum-number-of-events (default 50). Hourly uptime entries are kept for 30 days (uptimeRetention, cleanup above 32 days in storage/store/sql/sql.go). 30 days is shorter than a calendar month, so sloreport collect stores each day’s raw results in docs/slo/data/ and the monthly report is computed from those files. The role sets maximum-number-of-results: 20160 (one week at 30 s), and sloreport collect -page-size must match it.
  • Uptime API: GET /api/v1/endpoints/{key}/uptimes/{1h|24h|7d|30d}; raw results: GET /api/v1/endpoints/{key}/statuses?page=&pageSize=, newest page first, page 1 capped at the maximum number of results.
  • What Gatus cannot do at 5.38.0, so the prober does it and pushes the result: conditions have no hash function (no SHA-384 of the tracker file), and a check by address cannot present SNI, because the client sets no TLS server name from a Host header and tls:// checks dial without one (client/client.go).

Measures

  • Ingestion (99.99%): one result per 30 s interval, slo_ingestion, available when the gateway answered 202 to the probe’s web event and the collector answered 200 to its OTLP metric.
  • Dashboard (99.90%): slo_dashboard, available when the console, the Umami host and an authenticated machine API call (/web/hourly with the probe’s scoped key) all answered.
  • Freshness, reported separately: slo_freshness-web and slo_freshness-api. Accepted probe events older than the limit (10 minutes in production) must be visible through /web/hourly and /api/hourly. The prober keeps a baseline count and the accepted sends after it, so events in flight or replayed twice can only raise the count (internal/prober).