Skip to main content
Every node’s Caddy gets its own certificates from Let’s Encrypt with the ACME DNS-01 challenge in the Cloudflare zone (caddy-dns/cloudflare, token cloudflare_api_token from <input I6>), for the hostnames it serves: edge nodes in. and otlp.<input I4>, core nodes app. and web.<input I4>, and a probe host in the inventory status.<input I4>. DNS-01 is used because every node serves the same names, so an HTTP or TLS-ALPN challenge could land on another node (../deploy/ansible.md section 3). Certificates live in the node’s caddydata volume under /data/caddy/certificates/. The zone’s CAA record allows only letsencrypt.org and forbids wildcards. Alerts that lead here: CertificateExpiringSoon (a node serves a certificate that expires within 14 days, read by the prober over a connection to the node’s own address) and ProbeNodeCheckFailing with a TLS error (alerts.md). The port-80 DNS health check does not see TLS, so a node with a broken certificate stays in DNS until you drain it.

Steps

  1. See what the node serves, from the controller or the probe:
    Expected: issuer Let’s Encrypt, notAfter more than 14 days away. Repeat for each name the node serves.
  2. Take the node out of DNS if it serves an expired or wrong certificate: edge-node-failure.md step 2 (the drain flag works the same on core nodes).
  3. Read Caddy’s log on the node:
    Common causes:
    • Cloudflare token rejected or missing: the log names an authorization error from the DNS provider. Check cloudflare_api_token in the SOPS file and that the token is active with Zone:Zone:Read and Zone:DNS:Edit on the zone (curl -s -H "Authorization: Bearer $TOKEN" https://api.cloudflare.com/client/v4/user/tokens/verify). Fix it as in secret-rotation.md, which rewrites /etc/veristar/env/caddy.env.
    • CAA or zone delegation: the domain’s name servers must be Cloudflare’s and the CAA must allow letsencrypt.org. dig +short CAA <input I4> and dig +short NS <input I4>.
    • Let’s Encrypt rate limits: the log says so and when to retry. Do not loop restarts.
    • Clock: ACME fails with a skewed clock. chronyc tracking on the node; the base role checks the offset at every converge.
  4. Retry once the cause is fixed:
    If Caddy does not retry by itself, force a fresh issuance by removing the stored certificates and restarting Caddy (this is what the phase 16 renewal drill does; the node keeps serving nothing on 443 until the new certificate arrives, so drain it first):
  5. Check with step 1, then undrain (edge-node-failure.md step 5). CertificateExpiringSoon resolves at the prober’s next run.
  6. Caddy cannot write its storage (permission denied under /data): the volume was created by an older root image. Fix the ownership as in security-patch.md section 6.

Exercised

Not yet exercised: issuance and renewal through Let’s Encrypt and Cloudflare (first staging converge do not exist; the local converge uses Caddy’s internal CA), and the forced renewal of step 4, whose drill (tests/drills/cert-renewal.sh) was not run because the shared host stopped the phase 16 drills (../reference/drills/2026-11-13.md). The multi-node topology may not run on the shared host, so phase 19 could not run it either.