> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veristar.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Certificate problems

> Diagnose and fix certificate issuance and renewal on edge and core nodes.

Every node's Caddy gets its own certificates from Let's Encrypt with the ACME DNS-01 challenge
in the Cloudflare zone (`caddy-dns/cloudflare`, token `cloudflare_api_token` from `<input I6>`),
for the hostnames it serves: edge nodes `in.` and `otlp.<input I4>`, core nodes `app.` and
`web.<input I4>`, and a probe host in the inventory `status.<input I4>`. DNS-01 is used because every node serves the same names, so an HTTP
or TLS-ALPN challenge could land on another node ([../deploy/ansible.md](https://docs.veristar.app/deploy/ansible.md)
section 3). Certificates live in the node's `caddydata` volume under `/data/caddy/certificates/`.
The zone's CAA record allows only `letsencrypt.org` and forbids wildcards.

Alerts that lead here: `CertificateExpiringSoon` (a node serves a certificate that expires
within 14 days, read by the prober over a connection to the node's own address) and
`ProbeNodeCheckFailing` with a TLS error ([alerts.md](https://docs.veristar.app/runbooks/alerts.md)). The port-80 DNS health check
does not see TLS, so a node with a broken certificate stays in DNS until you drain it.

## Steps

1. **See what the node serves**, from the controller or the probe:

   ```sh theme={null}
   echo | openssl s_client -connect <node public address>:443 -servername in.<input I4> 2>/dev/null \
     | openssl x509 -noout -subject -issuer -dates -serial
   ```

   Expected: issuer Let's Encrypt, `notAfter` more than 14 days away. Repeat for each name the
   node serves.

2. **Take the node out of DNS** if it serves an expired or wrong certificate:
   [edge-node-failure.md](https://docs.veristar.app/runbooks/edge-node-failure.md) step 2 (the drain flag works the same on core
   nodes).

3. **Read Caddy's log** on the node:

   ```sh theme={null}
   cd /opt/veristar/<edge or core>
   sudo docker compose logs --since 24h caddy | grep -i -E 'acme|certificate|obtain|renew|dns|challenge' | tail -50
   ```

   Common causes:

   * **Cloudflare token** rejected or missing: the log names an authorization error from the
     DNS provider. Check `cloudflare_api_token` in the SOPS file and that the token is active
     with `Zone:Zone:Read` and `Zone:DNS:Edit` on the zone
     (`curl -s -H "Authorization: Bearer $TOKEN" https://api.cloudflare.com/client/v4/user/tokens/verify`).
     Fix it as in [secret-rotation.md](https://docs.veristar.app/runbooks/secret-rotation.md), which rewrites
     `/etc/veristar/env/caddy.env`.
   * **CAA or zone delegation**: the domain's name servers must be Cloudflare's and the CAA must
     allow `letsencrypt.org`. `dig +short CAA <input I4>` and `dig +short NS <input I4>`.
   * **Let's Encrypt rate limits**: the log says so and when to retry. Do not loop restarts.
   * **Clock**: ACME fails with a skewed clock. `chronyc tracking` on the node; the base role
     checks the offset at every converge.

4. **Retry** once the cause is fixed:

   ```sh theme={null}
   sudo docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
   ```

   If Caddy does not retry by itself, force a fresh issuance by removing the stored certificates
   and restarting Caddy (this is what the phase 16 renewal drill does; the node keeps serving
   nothing on 443 until the new certificate arrives, so drain it first):

   ```sh theme={null}
   sudo docker compose exec -T caddy sh -c 'rm -rf /data/caddy/certificates/*'
   sudo docker compose restart caddy
   ```

5. **Check** with step 1, then undrain ([edge-node-failure.md](https://docs.veristar.app/runbooks/edge-node-failure.md) step 5).
   `CertificateExpiringSoon` resolves at the prober's next run.

6. **Caddy cannot write its storage** (`permission denied` under `/data`): the volume was created
   by an older root image. Fix the ownership as in [security-patch.md](https://docs.veristar.app/runbooks/security-patch.md)
   section 6.

## Exercised

| Date | What ran | Result | Source |
| - | - | - | - |
| 2026-11-13 | Alert fire-drill: a 7-day certificate served on edge-2 of the observability test stack | CertificateExpiringSoon and the Gatus check for that node fired at 01:34:48 and resolved at 01:39:48 after the normal certificate was back | [alerts.md](https://docs.veristar.app/runbooks/alerts.md), fire-drills |

Not yet exercised: issuance and renewal through Let's Encrypt and Cloudflare (first staging converge
do not exist; the local converge uses Caddy's internal CA), and the forced renewal of step 4,
whose drill (`tests/drills/cert-renewal.sh`) was not run because the shared host stopped the
phase 16 drills ([../reference/drills/2026-11-13.md](../reference/drills/2026-11-13.md)). The
multi-node topology may not run on the shared host, so phase 19 could not run it either.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.