Skip to main content
Status: first version (phase 15, stream b). Pins: SOPS 3.13.3, age 1.3.2 (plan section 5).

1. Layout

Production has a recipient list since 2026-11-13 (recipients/production.txt, the owner’s identity) and therefore a creation rule; its secrets.enc.yaml and tofu.enc.env exist. Staging has no list, so sops encrypt for staging refuses with no matching creation rules found. That refusal is intended; nothing can be encrypted to an unreviewed key. Section 4 asks for at least two recipients per environment; production has one until a second person’s key is added.

2. Who decrypts

Only people. Each recipient is a person’s age public key. The nodes never hold an age identity: the Ansible secrets role decrypts on the controller with the operator’s identity (SOPS_AGE_KEY_FILE, else ~/.config/sops/age/keys.txt) through community.sops.load_vars, and writes per-service env files on each node:
  • /etc/veristar/env/ is root:root 0700, every file in it root:root 0600.
  • Each service gets only its own keys (roles/secrets/templates/<service>.env.j2), read by Docker Compose’s env_file.
  • Config files that must carry a credential (NATS, Patroni) are written by the edge and core roles with mode 0600; Patroni’s is owned by the container’s postgres user.
  • Every task that handles a value runs with no_log.
The secrets role installs SOPS and age on the controller from their GitHub releases, checked against pinned SHA-256 sums (infra/ansible/vars/pins.yml).

3. Contents of <env>/secrets.enc.yaml

Generated (random, by new-secrets.py): PostgreSQL superuser and replication passwords, Patroni REST password, Umami database password, APP_SECRET and admin password, UMAMI_USER_PASSWORD_KEY, the Umami retention role password, OAUTH_STATE_KEY, the five platform role passwords, one NATS password per service user with its bcrypt hash, and the NATS route password with its hash, and for monitoring (phase 17) the Grafana admin password, the postgres_exporter password, the probe metrics token and the Gatus external-endpoint token. The hashes are generated once with the passwords, so the NATS server config does not change on every converge. A file made before the monitoring keys existed is refused by the secrets role, naming the missing key; add each with sops edit and a value from openssl rand -hex 32. Owner inputs (taken from the environment of new-secrets.py, never generated, left out when unset): github_client_id and github_client_secret (I2, as SA_GITHUB_CLIENT_ID, SA_GITHUB_CLIENT_SECRET), cloudflare_api_token (I6, as SA_CLOUDFLARE_API_TOKEN; the zone-scoped token with Zone:Zone:Read and Zone:DNS:Edit, prerequisites.md step 5), alertmanager_discord_webhook_url and gatus_discord_webhook_url (I8, as SA_ALERTMANAGER_DISCORD_WEBHOOK_URL, SA_GATUS_DISCORD_WEBHOOK_URL). The secrets role refuses to converge a core node without I2 and any node using DNS-01 without I6, naming the missing key. The observability role, which site.yml runs on every edge and core node, refuses to run without the Alertmanager webhook (I8); the local converge passes observability_test_webhook_base instead (docs/deploy/local-converge.md).

4. Procedures

Create an environment’s file (after I14 is in place):
Add an input later: sops --config infra/secrets/.sops.yaml edit infra/secrets/staging/secrets.enc.yaml. Adding a recipient. The person generates an identity with age-keygen and sends the public key. Add it to infra/secrets/recipients/<env>.txt with a comment naming the holder, run infra/secrets/sync-recipients.sh (it rewrites .sops.yaml and runs sops updatekeys on every file of that environment), review the diff and commit both. Removing a recipient. Delete the line, run sync-recipients.sh, commit. Re-encryption only stops the person reading future versions; they may have kept the old plaintext. Therefore also rotate every value they could read, and rotate I2 and I6 with their owners. What site.yml applies by itself after a value changes in the file: the five platform role passwords (platformctl db roles), the NATS passwords and hashes (env files and a NATS reload), and every env file. What it does not apply yet, because bootstrap only creates and never alters them: the Umami database password, the Umami admin password (platformctl umami bootstrap replaces only the seeded one), the PostgreSQL superuser, replication and Patroni REST passwords, and UMAMI_USER_PASSWORD_KEY (changing it changes every derived Umami password). Their rotation steps are in ../runbooks/secret-rotation.md; removing a recipient is a complete rotation only once those steps are done too. Losing every identity makes the files unreadable. Each recipient keeps their identity in a password manager; there must be at least two recipients per environment.

5. Setting up an environment, step by step

Staging is shown; production is the same with production in every path. Needs the controller tools of prerequisites.md with sops and age-keygen on PATH, and inputs I14 (recipients) plus whatever owner inputs exist (I2, I5, I6, I8).
  1. Each recipient makes an identity on their own machine and keeps the file private:
    Expected: Public key: age1.... Only that public key is sent.
  2. List the recipients in infra/secrets/recipients/staging.txt, one public key per line, each under a comment line naming the holder (at least two people).
  3. Generate the SOPS rules:
    Expected: wrote .../infra/secrets/.sops.yaml with 1 rule(s) (2 once production has a list too). The rule is path_regex: (^|/)staging/[^/]+\.enc\.(yaml|env)$ with the listed keys.
  4. Create the platform secrets file with the owner inputs that exist, typed into the environment of this one command (the values are never written in plain text to disk):
    Do the same with read -rs for SA_CLOUDFLARE_API_TOKEN (I6), SA_ALERTMANAGER_DISCORD_WEBHOOK_URL and SA_GATUS_DISCORD_WEBHOOK_URL (I8) when you have them. The script names every input it left out on standard error, for example new-secrets.py: SA_CLOUDFLARE_API_TOKEN unset, cloudflare_api_token left out. The venv’s Python is used because the script needs bcrypt and PyYAML.
  5. Check that the file decrypts and holds every key:
    Expected: sa_secrets_environment: first, then every generated key of section 3 and the owner inputs you supplied. Never redirect the decrypted output to a file.
  6. Create the OpenTofu credentials file infra/secrets/staging/tofu.enc.env with the variables of opentofu.md section 1 (“Secrets handed to OpenTofu”):
    SOPS opens an editor on an empty dotenv file; write one NAME=value line per variable (CLOUDFLARE_API_TOKEN and TF_VAR_state_passphrase; Azure needs no secret here, OpenTofu uses the operator’s az login). Make the passphrase with python3 -c 'import secrets; print(secrets.token_urlsafe(32))'; it must be at least 16 characters and different per environment.
  7. Issue the private network’s certificates once the inventory exists (opentofu.md section 6), before the first converge:
    Expected: N certificate(s) issued, ca new, 0 retired ca(s) still trusted, one certificate per edge, core, backup and restore node plus the operators’ one, and two new files, infra/secrets/staging/ca.enc.yaml and tls.enc.yaml (ansible.md section 9). The secrets role refuses to converge a node without them.
  8. Commit infra/secrets/.sops.yaml, the recipients list and the encrypted files.
Steps 1 to 5 were rehearsed on 2026-11-13 in a scratch directory with a throwaway identity and a copy of sync-recipients.sh (never under infra/secrets/): the script wrote one rule, new-secrets.py staging piped into sops encrypt exited 0 and reported the seven owner inputs it left out, and the file decrypted to every generated key listed in section 3 (37 encrypted values). Steps 6 and 7 and every real recipient wait for input I14.