> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veristar.app/llms.txt
> Use this file to discover all available pages before exploring further.

# The probe outside Azure, with Uptime Kuma and ntfy

> The probe that checks the platform from outside its hosting, with a status page and alerts.

The prober (`cmd/prober`) is the outside view of the platform: every 30 s it sends a real web
event and a real OTLP metric for the internal probe tenant, reads them back through the machine
API, checks the console, the Umami host, the tracker files against the release manifest, and
every node's certificate by address. It must not run inside the system it watches, so it runs on
a host outside Azure. This document runs it on the owner's Home LXC, beside the Uptime Kuma and
ntfy that already alert the owner's phone, and makes Kuma alert on its results.

Nothing here changes the Home LXC by itself. The steps below are for the owner to run there.

## 1. What runs

`infra/ansible/playbooks/probe-bundle.yml` renders a self-contained Docker Compose bundle on the
controller, from the same templates the observability role uses:

| File | Holds |
| - | - |
| `compose.yml` | `gatus` (5.38.0, pinned by digest) and `prober` (the release image, pulled by its verified digest); hardened like the platform's own services (read-only root, no capabilities, `no-new-privileges`) |
| `gatus.yml` | Gatus's own checks of the public hostnames and the external endpoints the prober pushes to |
| `prober.env`, `gatus.env` | the probe tenant's OTLP token, machine API key and site id, the Gatus push token (mode 0600) |
| `releases/` | the tracker release manifest the prober compares the served files with |

Gatus is published only on the address in `PROBE_BIND`; nothing listens publicly. Kuma reads
Gatus's per-check status API, one Kuma monitor per check, and Kuma's existing ntfy notification
delivers the alert.

## 2. Render the bundle (controller)

Needs the environment's inventory (`make <env>-apply`), its SOPS file with the probe tenant
(`platformctl probe-tenant create`, [bootstrap.md](https://docs.veristar.app/deploy/bootstrap.md) step 10) and the release
manifest:

```sh theme={null}
cd infra/ansible
~/.local/share/veristar/ansible-venv/bin/ansible-playbook -i inventories/production/ \
  playbooks/probe-bundle.yml -e sa_release_manifest=<release-images.json>
```

Expected: the release verification, then
`probe bundle for production in .../resources/probe-bundle/production`. The directory is mode
0700 under `resources/`, which Git ignores.

## 3. Run it on the Home LXC (owner)

Copy the directory and start it. The Home LXC's containers reach host services at `172.17.0.1`
(the owner's server notes, Docker networking), so Gatus is published there on port 8095. Check
first that nothing else uses the port: `ssh home 'ss -ltn "( sport = :8095 )"'` must list no
socket; otherwise change `probe_bundle_port` (`-e probe_bundle_port=<port>`).

```sh theme={null}
rsync -a resources/probe-bundle/production/ home:/home/docker/veristar-probe/
ssh home 'cd /home/docker/veristar-probe && PROBE_BIND=172.17.0.1 docker compose up -d'
ssh home 'curl -fsS http://172.17.0.1:8095/api/v1/endpoints/statuses | head -c 300'
```

Expected: both containers `Up`, and a JSON list of endpoints within a minute. The bundle is its
own Compose project (`veristar-probe-production`) beside `/home/docker/docker-compose.yml`, so the
Home stack is not touched. The Home LXC resolves names through its own AdGuard; the platform's
hostnames are under `.app`. On 2026-11-13 AdGuard answered a made-up `.app` name with no
address (not `0.0.0.0`), so the TLD is not blocked as a whole. Check the real names once they
exist: `dig +short in.veristar.app @192.168.1.102` must not answer `0.0.0.0`.

Backups: the bundle holds only configuration and Gatus's history; the Home restic job already
covers `/home/docker` unless an exclude matches.

## 4. Add the Uptime Kuma monitors (owner, Kuma web UI)

One monitor per check. For each, in Uptime Kuma: Add New Monitor.

| Field | Value |
| - | - |
| Monitor Type | HTTP(s) - Json Query |
| Friendly Name | `veristar <key>` |
| URL | `http://172.17.0.1:8095/api/v1/endpoints/<key>/statuses?page=1&pageSize=1` |
| Json Query | `$toMillis(results[0].timestamp) > $millis() - 180000 and results[0].success` |
| Expected Value | `true` |
| Heartbeat Interval | 60 |
| Retries | 2 |
| Notifications | the existing `Server Alerts` (ntfy, topic `server-alerts`) |

The query is true only when the newest result is younger than three minutes and passed, so a
stopped prober alerts too, not only a failed check. Gatus page 1 with `pageSize=1` holds exactly
the newest result (checked with Gatus 5.38.0 on 2026-11-13).

The keys (Gatus key of group and name):

| Key | Alerts when |
| - | - |
| `slo_ingestion` | a web event or OTLP metric is refused on the public path (99.99 % target) |
| `slo_dashboard` | the console, the Umami host or the machine API fails (99.90 % target) |
| `slo_freshness-web`, `slo_freshness-api` | accepted probe data is not visible within 10 minutes |
| `probe_tracker-integrity` | a served tracker file differs from the release manifest |
| `nodes_<node>-<host>` | one node fails for one hostname by address (for example `nodes_production-edge-1-in-veristar-app`); add the ten of the environment |

The full list is at `http://172.17.0.1:8095/api/v1/endpoints/statuses` (field `key`). Kuma's
own status page is behind Cloudflare Access, so these monitors are for the owner, not a public
status page.

## 5. Public status page and the alert heartbeat

The bundle has no public listener. `status.<BASE_DOMAIN>` (the status page) and the
Alertmanager watchdog push are served only when a probe host is in the inventory's `probe`
group, with its own public address and Caddy (`site.yml`, play `Probe`). With the probe on the
Home LXC, the status page can be published by a Cloudflare Tunnel route to Gatus, for example
through the Home LXC's existing `cloudflared`, as a separate decision of the owner: it is a
change to the Home LXC and to the Cloudflare account.

The nightly `slo-report.yml` workflow reads Gatus's results from `STATUS_BASE_URL`
(`https://status.<input I4>`), so the monthly SLO report needs that publication too. Until then
the Gatus history on the Home LXC holds the measurements (about a week of raw results), and
Kuma keeps its own uptime history per monitor.

## 6. What was checked

* `ansible-playbook --syntax-check` and `make ansible-lint` pass for the playbook and template
  (2026-11-13).
* The Gatus API answers used by the Kuma query, with Gatus 5.38.0 and three pushed results:
  `pageSize=1` returned only the newest (`success: false` after `true`, `true`, `false`).
* Not run yet: the bundle against a live environment (there is none before the first apply).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.