> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veristar.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Production readiness checklist

> The checks production passes and signs off before it takes its first real project.

The gate between "production exists" and "production takes its first real tenant" (plan phase 20).
Each item has a check or a command, the result that counts as a pass, and an owner. An item is
signed off by writing the name of the person who ran it, the UTC date and a link to the evidence
(a command log, a pull request, a drill record) in the last column. The sign-off column is filled
in a pull request against this file, one pull request per sign-off session, so the history shows
who signed what and when.

**Status on 2026-11-13: creation started.** The owner decided on production only, with the
local converge as the test environment in place of staging (item 1.4). Inputs I2, I3, I4, I5,
I6, I13, I14 and I15 are in place for production and PostgreSQL 18.7 is pinned (items 1.2 and
1.3). OpenTofu created 38 of the 66 resources; the five VMs wait for Azure capacity
([opentofu.md](https://docs.veristar.app/deploy/opentofu.md) section 3.8). The first converge also needs a signed release
(item 1.5) and input I8 for the observability role. Nothing from section 2 onwards has run
against production.

Owners are roles, not people:

| Role | Who |
| - | - |
| Deployer | the person who runs OpenTofu and Ansible for production and holds an age identity of input I14 |
| Operator | an SCF operator listed in `OPERATOR_GITHUB_IDS` (input I3) |
| Repository owner | the owner of the GitHub repository (input I1) |
| Legal reviewer | whoever signs off the compliance statements (input I10) |

Addresses below follow the conventions of [../runbooks/overview.md](https://docs.veristar.app/runbooks/overview.md):
`<input I4>` is the production domain, node names are `production-edge-<n>` and
`production-core-<n>` from `infra/ansible/inventories/production/hosts.json`.

## 1. Before the environment is created

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 1.1 | Inputs present | Go through the input table in [prerequisites.md](https://docs.veristar.app/deploy/prerequisites.md) for production: I2 (a production OAuth app), I3, I4, I5, I6, I8, I13, I14, I15 | each input exists and is stored where that page says, none is a placeholder | Deployer | |
| 1.2 | PostgreSQL 18.7 published | `docker buildx imagetools inspect postgres:18.7-trixie` | the command prints an index digest for `linux/amd64` | Deployer | |
| 1.3 | PostgreSQL 18.7 pinned | In a pull request: raise `postgres_version` to `18.7` (or newer) in `infra/ansible/vars/pins.yml`, update section 5 of the plan and the image digest, rebuild `postgres-ha`. Then `grep -n '^postgres_version' infra/ansible/vars/pins.yml` | `postgres_version` is at least `postgres_min_version` (`18.7`); the core role's assertion "Refuse a postgresql below the required minimum" passes during the converge | Deployer | |
| 1.4 | Staging verified on the same release | [verification.md](https://docs.veristar.app/deploy/verification.md) steps 1 to 11 on staging with the release tag that will go to production | every step gives its expected result; the run is linked | Deployer | |
| 1.5 | Release signed | `release-images.json` of the release is attached to its GitHub release, and `cosign verify` succeeds for each image as in [../security/supply-chain.md](https://docs.veristar.app/security/supply-chain.md) | every image verifies against the release workflow's identity | Deployer | |
| 1.6 | Compliance statements | The legal reviewer approves `docs/compliance/` (input I10); the drafts' "pending legal review" notes are removed in the same pull request | a merged pull request with the reviewer's approval | Legal reviewer | |

## 2. Creating production

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 2.1 | Servers, network, DNS | `make production-plan`, review, `make production-apply` ([opentofu.md](https://docs.veristar.app/deploy/opentofu.md)) | apply ends without error; `infra/ansible/inventories/production/hosts.json` lists 3 edge and 2 core nodes | Deployer | |
| 2.2 | First converge | [bootstrap.md](https://docs.veristar.app/deploy/bootstrap.md) for production, then `make production-deploy SA_IMAGE_TAG=<release> SA_RELEASE_MANIFEST=<release-images.json>` a second time | the second run's `PLAY RECAP` shows `changed=0` and `failed=0` on every node | Deployer | |
| 2.3 | Environment verified | [verification.md](https://docs.veristar.app/deploy/verification.md) steps 1 to 11 with `production` in place of `staging` | every step gives its expected result | Deployer | |
| 2.4 | Probe | The probe bundle on the owner's home server with its Uptime Kuma monitors ([probe.md](https://docs.veristar.app/deploy/probe.md), [bootstrap.md](https://docs.veristar.app/deploy/bootstrap.md) step 11) | every Kuma monitor is up; `slo_ingestion`, `slo_dashboard`, `slo_freshness-web` and `slo_freshness-api` receive results every 30 s | Deployer | |
| 2.5 | PostgreSQL version running | On the leader's node: `cd /opt/veristar/core && sudo docker compose exec postgres psql -U postgres -tAc 'show server_version'` | `18.7` or newer | Deployer | |

## 3. Drills repeated on production

Run before any real tenant exists, with only the probe tenant sending traffic. The scripts in
`tests/drills/` target the local converge only (`tests/drills/lib.sh` reads the local inventory and
SOPS file), so on production each drill follows its runbook by hand, with the load generated by
the probe plus `examples/otel-go-nethttp` and a test site as in [verification.md](https://docs.veristar.app/deploy/verification.md)
steps 9 and 10. Record each drill in a new `docs/reference/drills/<date>.md` in the format of
[../reference/drills/2026-11-13.md](../reference/drills/2026-11-13.md). The pass bounds are the
plan's phase 16 bounds.

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 3.1 | Edge node loss | Stop `production-edge-2` (`az vm stop -g veristar-production -n production-edge-2`) under traffic, wait 5 minutes, start it (`az vm start ...`); follow [../runbooks/edge-node-failure.md](https://docs.veristar.app/runbooks/edge-node-failure.md). Watch the prober's ingestion results on the status page | 0 acknowledged events lost (every probe event acknowledged in the window is counted in the machine API's `/web/hourly` for the probe tenant), Traffic Manager removed the node within 120 s, failed ingest requests at most 1% and only within those 120 s; after power-on every container on the node is running again | Deployer | |
| 3.2 | Database leader loss | Note the leader with `patronictl list`, power off its node, follow [../runbooks/database-failover.md](https://docs.veristar.app/runbooks/database-failover.md) "Automatic failover: check it happened" | promotion without action; dashboard gap at most 60 s on the status page; 0 committed rows lost (the newest `cp.audit_log` row from before the power-off is present); 0 edge errors | Deployer | |
| 3.3 | Replica back in sync | After 3.2, power the node on and follow "Replica rebuild" in the same runbook | `patronictl list` shows one `Leader` and one `Sync Standby` in `streaming` with lag 0 | Deployer | |
| 3.4 | Certificates | [../runbooks/certificates.md](https://docs.veristar.app/runbooks/certificates.md) step 1 for every node and name | Let's Encrypt issuer, more than 14 days left, on every node | Deployer | |

## 4. A restored backup

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 4.1 | Repository healthy | On the leader's node: `sudo docker compose exec postgres pgbackrest --stanza=sa info` | `status: ok`, at least one full backup, WAL archived up to the last minute | Deployer | |
| 4.2 | Restore performed | [../runbooks/point-in-time-restore.md](https://docs.veristar.app/runbooks/point-in-time-restore.md) steps 1 to 5 against production's repository on a fresh node, with a target time a few minutes in the past | the playbook ends with `restored to <target> on <address>:5432`; the newest `cp.audit_log` row on the restore node is at or just before the target | Deployer | |
| 4.3 | Restore timed | From the start of step 4 of that runbook to its last task | the duration is written in the drill record; this is the measured recovery time | Deployer | |
| 4.4 | Restore node removed | Step 7 of that runbook | the server, its inventory file, container and volume are gone | Deployer | |

## 5. Alerts routed

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 5.1 | Rules loaded | Tunnel to Prometheus ([../runbooks/overview.md](https://docs.veristar.app/runbooks/overview.md), "Prometheus and Grafana"), then `curl -s http://127.0.0.1:9090/api/v1/rules \| python3 -c 'import json,sys; print(sum(len(g["rules"]) for g in json.load(sys.stdin)["data"]["groups"]))'` | the number of rules (alerting and recording) in `infra/ansible/roles/observability/files/rules` of the release | Deployer | |
| 5.2 | Targets up | `curl -s http://127.0.0.1:9090/api/v1/targets?state=active \| grep -c '"health":"down"'` | `0` | Deployer | |
| 5.3 | Alertmanager delivers to Discord | On a core node, post a test alert to Alertmanager on its private address: `curl -fsS -X POST -H 'Content-Type: application/json' http://<private address>:9093/api/v2/alerts -d '[{"labels":{"alertname":"ReadinessTest","severity":"warning"},"annotations":{"summary":"production readiness check 5.3"}}]'` | the message arrives in the Discord channel of input I8 within 2 minutes, and its resolution after the alert expires | Operator | |
| 5.4 | Gatus delivers to Discord | Stop the gateway on one edge node for 3 minutes (`cd /opt/veristar/edge && sudo docker compose stop gateway`, then `start gateway`) | `EdgeNodeNotReady` and the Gatus alert for that node arrive in Discord and resolve after the start | Operator | |
| 5.5 | No critical alert open | `curl -s http://127.0.0.1:9090/api/v1/alerts \| grep -c '"severity":"critical"'` | `0` | Deployer | |
| 5.6 | Console shows alerts | On a core node, post a test alert with root's certificate: `sudo curl -fsS --cacert /etc/veristar/tls/root/ca.crt --cert /etc/veristar/tls/root/node.crt --key /etc/veristar/tls/root/node.key -X POST -H 'Content-Type: application/json' https://<private address>:9093/api/v2/alerts -d '[{"labels":{"alertname":"ReadinessTest","severity":"ticket"},"annotations":{"summary":"production readiness check 5.6"}}]'`, then open `https://app.<BASE_DOMAIN>/operator/alerts` as an operator ([../operators/alerts.md](https://docs.veristar.app/operators/alerts.md)) | within 30 seconds `ReadinessTest` is listed under Ticket, the page shows neither "Alerting pipeline down" nor a member that did not answer, and the alert is gone 5 minutes later (Alertmanager's `resolve_timeout`) | Operator | |

## 6. SLO measurement running

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 6.1 | Probe results flowing | `curl -s https://status.<input I4>/api/v1/endpoints/statuses \| python3 -c 'import json,sys; print([e["key"] for e in json.load(sys.stdin) if e["key"].startswith("slo_")])'` | the four `slo_` endpoints are listed, each with results from the last 5 minutes | Deployer | |
| 6.2 | Daily collection | Repository variable `STATUS_BASE_URL` set to `https://status.<input I4>`; the next day, the `slo-report.yml` run ([../runbooks/slo-report.md](https://docs.veristar.app/runbooks/slo-report.md)) | the workflow succeeds and `docs/slo/data/<yesterday>.json` exists on branch `slo/<YYYY-MM>` | Repository owner | |
| 6.3 | First report | After the first full month: `go run ./cmd/sloreport report -month <YYYY-MM> -data docs/slo/data -out docs/slo` or the workflow's pull request | `docs/slo/<YYYY-MM>.md` exists and states coverage for every measure | Operator | |

## 7. Isolation gate green on the deployed versions

The gate is `make isolation` (plan phase 18, [../security/threat-model.md](https://docs.veristar.app/security/threat-model.md)).
It builds and runs the code of the working copy, so it proves the deployed versions only when the
working copy is the commit the deployed images were built from.

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 7.1 | Deployed commit known | `git rev-list -n 1 v<release>` for the tag in `SA_IMAGE_TAG` of the last `production-deploy`. Item 1.5 ties the images to that tag: `release-images.yml` builds them from the tag, and the keyless signature records the workflow run | one commit, the one the release tag points to | Deployer | |
| 7.2 | Gate run at that commit | `git checkout <that commit> && make isolation` | every test passes; the log is linked | Deployer | |
| 7.3 | Umami version matches | The Umami digest in `infra/ansible/vars/pins.yml` at that commit equals the one running: `cd /opt/veristar/core && sudo docker compose images umami` | same digest; the isolation suite tested this Umami | Deployer | |

## 8. Repository public

The RFP requires a public open-source repository and a public issue tracker (RFP 3.1, 3.4), and
npm provenance needs one too. Making the repository public cannot be undone in practice (forks and
caches keep what was public), so it is never done by a session or a deployer on their own.

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 8.1 | History clean | `gitleaks git --redact --no-banner --exit-code 1 .` over the full history, the command of the `gitleaks` job in `.github/workflows/ci.yml` and a review of `git log --stat` for files that must not be public | no finding; nothing from `resources/` or any `.env` file is in the history | Deployer | |
| 8.2 | Owner confirmation | The repository owner states in writing (an issue or pull request comment) that the repository may be made public | the confirmation is linked here | Repository owner | |
| 8.3 | Made public | `gh repo edit <owner>/<repo> --visibility public --accept-visibility-change-consequences`, run by the repository owner | `gh repo view <owner>/<repo> --json visibility` answers `PUBLIC` | Repository owner | |
| 8.4 | Branch protection | [../security/branch-protection.md](https://docs.veristar.app/security/branch-protection.md) applied | `gh api repos/<owner>/<repo>/branches/main/protection` lists the required checks of that page | Repository owner | |
| 8.5 | Issue tracker open | [../support.md](https://docs.veristar.app/support.md): issue forms visible, `issue-age.yml` scheduled | a test issue can be opened from a signed-out browser view of the forms and the workflow's next run succeeds | Repository owner | |

## 9. Ready for tenants

| # | Item | Check or command | Pass | Owner | Signed off |
| - | - | - | - | - | - |
| 9.1 | Sections 1 to 8 signed off | this file | every row above has a sign-off | Operator | |
| 9.2 | Probe end to end | [verification.md](https://docs.veristar.app/deploy/verification.md) steps 9 and 10 with a test project, then remove the test project ([../runbooks/tenant-removal.md](https://docs.veristar.app/runbooks/tenant-removal.md)) | page view and API requests appear within the 10-minute freshness budget | Operator | |
| 9.3 | Enrollment mode chosen | `GET /api/operator/settings/enrollment-mode` ([../operators/review-and-enrollment.md](https://docs.veristar.app/operators/review-and-enrollment.md)) | the mode is the one the operators decided for the launch, recorded in the audit log | Operator | |

When 9.1 to 9.3 are signed off, onboarding of the pilot projects can start
([../adoption/overview.md](https://docs.veristar.app/adoption/overview.md)).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.