Production readiness checklist

The gate between “production exists” and “production takes its first real tenant” (plan phase 20). Each item has a check or a command, the result that counts as a pass, and an owner. An item is signed off by writing the name of the person who ran it, the UTC date and a link to the evidence (a command log, a pull request, a drill record) in the last column. The sign-off column is filled in a pull request against this file, one pull request per sign-off session, so the history shows who signed what and when. Status on 2026-11-13: creation started. The owner decided on production only, with the local converge as the test environment in place of staging (item 1.4). Inputs I2, I3, I4, I5, I6, I13, I14 and I15 are in place for production and PostgreSQL 18.7 is pinned (items 1.2 and 1.3). OpenTofu created 38 of the 66 resources; the five VMs wait for Azure capacity (opentofu.md section 3.8). The first converge also needs a signed release (item 1.5) and input I8 for the observability role. Nothing from section 2 onwards has run against production. Owners are roles, not people:

Role	Who
Deployer	the person who runs OpenTofu and Ansible for production and holds an age identity of input I14
Operator	an SCF operator listed in `OPERATOR_GITHUB_IDS` (input I3)
Repository owner	the owner of the GitHub repository (input I1)
Legal reviewer	whoever signs off the compliance statements (input I10)

Addresses below follow the conventions of ../runbooks/overview.md: <input I4> is the production domain, node names are production-edge-<n> and production-core-<n> from infra/ansible/inventories/production/hosts.json.

1. Before the environment is created

#	Item	Check or command	Pass	Owner
1.1	Inputs present	Go through the input table in prerequisites.md for production: I2 (a production OAuth app), I3, I4, I5, I6, I8, I13, I14, I15	each input exists and is stored where that page says, none is a placeholder	Deployer
1.2	PostgreSQL 18.7 published	`docker buildx imagetools inspect postgres:18.7-trixie`	the command prints an index digest for `linux/amd64`	Deployer
1.3	PostgreSQL 18.7 pinned	In a pull request: raise `postgres_version` to `18.7` (or newer) in `infra/ansible/vars/pins.yml`, update section 5 of the plan and the image digest, rebuild `postgres-ha`. Then `grep -n '^postgres_version' infra/ansible/vars/pins.yml`	`postgres_version` is at least `postgres_min_version` (`18.7`); the core role’s assertion “Refuse a postgresql below the required minimum” passes during the converge	Deployer
1.4	Staging verified on the same release	verification.md steps 1 to 11 on staging with the release tag that will go to production	every step gives its expected result; the run is linked	Deployer
1.5	Release signed	`release-images.json` of the release is attached to its GitHub release, and `cosign verify` succeeds for each image as in ../security/supply-chain.md	every image verifies against the release workflow’s identity	Deployer
1.6	Compliance statements	The legal reviewer approves `docs/compliance/` (input I10); the drafts’ “pending legal review” notes are removed in the same pull request	a merged pull request with the reviewer’s approval	Legal reviewer

2. Creating production

#	Item	Check or command	Pass	Owner
2.1	Servers, network, DNS	`make production-plan`, review, `make production-apply` (opentofu.md)	apply ends without error; `infra/ansible/inventories/production/hosts.json` lists 3 edge and 2 core nodes	Deployer
2.2	First converge	bootstrap.md for production, then `make production-deploy SA_IMAGE_TAG=<release> SA_RELEASE_MANIFEST=<release-images.json>` a second time	the second run’s `PLAY RECAP` shows `changed=0` and `failed=0` on every node	Deployer
2.3	Environment verified	verification.md steps 1 to 11 with `production` in place of `staging`	every step gives its expected result	Deployer
2.4	Probe	The probe bundle on the owner’s home server with its Uptime Kuma monitors (probe.md, bootstrap.md step 11)	every Kuma monitor is up; `slo_ingestion`, `slo_dashboard`, `slo_freshness-web` and `slo_freshness-api` receive results every 30 s	Deployer
2.5	PostgreSQL version running	On the leader’s node: `cd /opt/veristar/core && sudo docker compose exec postgres psql -U postgres -tAc 'show server_version'`	`18.7` or newer	Deployer

3. Drills repeated on production

Run before any real tenant exists, with only the probe tenant sending traffic. The scripts in tests/drills/ target the local converge only (tests/drills/lib.sh reads the local inventory and SOPS file), so on production each drill follows its runbook by hand, with the load generated by the probe plus examples/otel-go-nethttp and a test site as in verification.md steps 9 and 10. Record each drill in a new docs/reference/drills/<date>.md in the format of ../reference/drills/2026-11-13.md. The pass bounds are the plan’s phase 16 bounds.

#	Item	Check or command	Pass	Owner
3.1	Edge node loss	Stop `production-edge-2` (`az vm stop -g veristar-production -n production-edge-2`) under traffic, wait 5 minutes, start it (`az vm start ...`); follow ../runbooks/edge-node-failure.md. Watch the prober’s ingestion results on the status page	0 acknowledged events lost (every probe event acknowledged in the window is counted in the machine API’s `/web/hourly` for the probe tenant), Traffic Manager removed the node within 120 s, failed ingest requests at most 1% and only within those 120 s; after power-on every container on the node is running again	Deployer
3.2	Database leader loss	Note the leader with `patronictl list`, power off its node, follow ../runbooks/database-failover.md “Automatic failover: check it happened”	promotion without action; dashboard gap at most 60 s on the status page; 0 committed rows lost (the newest `cp.audit_log` row from before the power-off is present); 0 edge errors	Deployer
3.3	Replica back in sync	After 3.2, power the node on and follow “Replica rebuild” in the same runbook	`patronictl list` shows one `Leader` and one `Sync Standby` in `streaming` with lag 0	Deployer
3.4	Certificates	../runbooks/certificates.md step 1 for every node and name	Let’s Encrypt issuer, more than 14 days left, on every node	Deployer

4. A restored backup

#	Item	Check or command	Pass	Owner
4.1	Repository healthy	On the leader’s node: `sudo docker compose exec postgres pgbackrest --stanza=sa info`	`status: ok`, at least one full backup, WAL archived up to the last minute	Deployer
4.2	Restore performed	../runbooks/point-in-time-restore.md steps 1 to 5 against production’s repository on a fresh node, with a target time a few minutes in the past	the playbook ends with `restored to <target> on <address>:5432`; the newest `cp.audit_log` row on the restore node is at or just before the target	Deployer
4.3	Restore timed	From the start of step 4 of that runbook to its last task	the duration is written in the drill record; this is the measured recovery time	Deployer
4.4	Restore node removed	Step 7 of that runbook	the server, its inventory file, container and volume are gone	Deployer

5. Alerts routed

#	Item	Check or command	Pass	Owner
5.1	Rules loaded	Tunnel to Prometheus (../runbooks/overview.md, “Prometheus and Grafana”), then `curl -s http://127.0.0.1:9090/api/v1/rules \| python3 -c 'import json,sys; print(sum(len(g["rules"]) for g in json.load(sys.stdin)["data"]["groups"]))'`	the number of rules (alerting and recording) in `infra/ansible/roles/observability/files/rules` of the release	Deployer
5.2	Targets up	`curl -s http://127.0.0.1:9090/api/v1/targets?state=active \| grep -c '"health":"down"'`	`0`	Deployer
5.3	Alertmanager delivers to Discord	On a core node, post a test alert to Alertmanager on its private address: `curl -fsS -X POST -H 'Content-Type: application/json' http://<private address>:9093/api/v2/alerts -d '[{"labels":{"alertname":"ReadinessTest","severity":"warning"},"annotations":{"summary":"production readiness check 5.3"}}]'`	the message arrives in the Discord channel of input I8 within 2 minutes, and its resolution after the alert expires	Operator
5.4	Gatus delivers to Discord	Stop the gateway on one edge node for 3 minutes (`cd /opt/veristar/edge && sudo docker compose stop gateway`, then `start gateway`)	`EdgeNodeNotReady` and the Gatus alert for that node arrive in Discord and resolve after the start	Operator
5.5	No critical alert open	`curl -s http://127.0.0.1:9090/api/v1/alerts \| grep -c '"severity":"critical"'`	`0`	Deployer
5.6	Console shows alerts	On a core node, post a test alert with root’s certificate: `sudo curl -fsS --cacert /etc/veristar/tls/root/ca.crt --cert /etc/veristar/tls/root/node.crt --key /etc/veristar/tls/root/node.key -X POST -H 'Content-Type: application/json' https://<private address>:9093/api/v2/alerts -d '[{"labels":{"alertname":"ReadinessTest","severity":"ticket"},"annotations":{"summary":"production readiness check 5.6"}}]'`, then open `https://app.<BASE_DOMAIN>/operator/alerts` as an operator (../operators/alerts.md)	within 30 seconds `ReadinessTest` is listed under Ticket, the page shows neither “Alerting pipeline down” nor a member that did not answer, and the alert is gone 5 minutes later (Alertmanager’s `resolve_timeout`)	Operator

6. SLO measurement running

#	Item	Check or command	Pass	Owner
6.1	Probe results flowing	`curl -s https://status.<input I4>/api/v1/endpoints/statuses \| python3 -c 'import json,sys; print([e["key"] for e in json.load(sys.stdin) if e["key"].startswith("slo_")])'`	the four `slo_` endpoints are listed, each with results from the last 5 minutes	Deployer
6.2	Daily collection	Repository variable `STATUS_BASE_URL` set to `https://status.<input I4>`; the next day, the `slo-report.yml` run (../runbooks/slo-report.md)	the workflow succeeds and `docs/slo/data/<yesterday>.json` exists on branch `slo/<YYYY-MM>`	Repository owner
6.3	First report	After the first full month: `go run ./cmd/sloreport report -month <YYYY-MM> -data docs/slo/data -out docs/slo` or the workflow’s pull request	`docs/slo/<YYYY-MM>.md` exists and states coverage for every measure	Operator

7. Isolation gate green on the deployed versions

The gate is make isolation (plan phase 18, ../security/threat-model.md). It builds and runs the code of the working copy, so it proves the deployed versions only when the working copy is the commit the deployed images were built from.

#	Item	Check or command	Pass	Owner
7.1	Deployed commit known	`git rev-list -n 1 v<release>` for the tag in `SA_IMAGE_TAG` of the last `production-deploy`. Item 1.5 ties the images to that tag: `release-images.yml` builds them from the tag, and the keyless signature records the workflow run	one commit, the one the release tag points to	Deployer
7.2	Gate run at that commit	`git checkout <that commit> && make isolation`	every test passes; the log is linked	Deployer
7.3	Umami version matches	The Umami digest in `infra/ansible/vars/pins.yml` at that commit equals the one running: `cd /opt/veristar/core && sudo docker compose images umami`	same digest; the isolation suite tested this Umami	Deployer

8. Repository public

The RFP requires a public open-source repository and a public issue tracker (RFP 3.1, 3.4), and npm provenance needs one too. Making the repository public cannot be undone in practice (forks and caches keep what was public), so it is never done by a session or a deployer on their own.

#	Item	Check or command	Pass	Owner
8.1	History clean	`gitleaks git --redact --no-banner --exit-code 1 .` over the full history, the command of the `gitleaks` job in `.github/workflows/ci.yml` and a review of `git log --stat` for files that must not be public	no finding; nothing from `resources/` or any `.env` file is in the history	Deployer
8.2	Owner confirmation	The repository owner states in writing (an issue or pull request comment) that the repository may be made public	the confirmation is linked here	Repository owner
8.3	Made public	`gh repo edit <owner>/<repo> --visibility public --accept-visibility-change-consequences`, run by the repository owner	`gh repo view <owner>/<repo> --json visibility` answers `PUBLIC`	Repository owner
8.4	Branch protection	../security/branch-protection.md applied	`gh api repos/<owner>/<repo>/branches/main/protection` lists the required checks of that page	Repository owner
8.5	Issue tracker open	../support.md: issue forms visible, `issue-age.yml` scheduled	a test issue can be opened from a signed-out browser view of the forms and the workflow’s next run succeeds	Repository owner

9. Ready for tenants

#	Item	Check or command	Pass	Owner
9.1	Sections 1 to 8 signed off	this file	every row above has a sign-off	Operator
9.2	Probe end to end	verification.md steps 9 and 10 with a test project, then remove the test project (../runbooks/tenant-removal.md)	page view and API requests appear within the 10-minute freshness budget	Operator
9.3	Enrollment mode chosen	`GET /api/operator/settings/enrollment-mode` (../operators/review-and-enrollment.md)	the mode is the one the operators decided for the launch, recorded in the audit log	Operator

When 9.1 to 9.3 are signed off, onboarding of the pilot projects can start (../adoption/overview.md).

Deploy

Runbooks

Service targets