Skip to main content
One runbook per operational task. Each one says what it is for, the numbered steps, how to tell it worked, and when and where it was last exercised. “Exercised” means the steps (or the exact code path they drive) were run for real and the result is recorded with its date and source. Where nothing ran, the runbook says not yet exercised and why. Plan inputs are written as <input I4> and so on (plan section 3; the list is in ../deploy/prerequisites.md). Values you look up at the time, such as a tenant id, are written as <tenant id>. Nothing in angle brackets is a real value.

Index

No runbook has been run against staging or production: neither exists yet (the first apply waits for review and for inputs I13, I14 and I15). The plan requires every runbook to be executed on staging before phase 19 is done; that requirement is still open, so the documentation phase is not done by its own condition until staging exists.

Conventions used in every runbook

Reaching the nodes

Every server has one login, ops, reachable over SSH from the admin address ranges (<input I13>) with the keys of <input I15>. Node names and addresses are in infra/ansible/inventories/<env>/hosts.json (written by make <env>-apply). Names are <env>-edge-1 to -3 and <env>-core-1, -2. Private addresses follow a fixed scheme (../deploy/opentofu.md section 2):

Compose projects

Every compose command below runs in that directory with sudo, for example:
Env files with credentials are in /etc/veristar/env/ (root only). Never copy them off the node.

Which core node holds the database

The member with role Leader is the primary. Run every psql command of these runbooks on that node:
Use -d umami for the Umami database.

Running platformctl on a core node

This is the same command infra/ansible/playbooks/bootstrap.yml runs. Replace the NATS addresses with the production ones from the table above when working on production, and <release tag> with the deployed SA_IMAGE_TAG:
The image name is <sa_image_registry>/platformctl:<tag>; ghcr.io/justmert/veristar is where release-images.yml publishes for repository <input I1>.

Running a jobs task by hand

jobs run <task> takes the same leader lock as the daemon. While a jobs daemon leads, it refuses with a jobs daemon is leading; it runs <task> itself (internal/jobs/scheduler). So either let the daemon run the task on its schedule (restart it with sudo docker compose restart jobs to run its start-up pass), or stop jobs on both core nodes and then run, on one of them:
Start jobs again on both nodes afterwards (sudo docker compose start jobs).

NATS state without a NATS client

No NATS command-line client is pinned or installed by any role. Stream and consumer counters are read from the NATS monitoring port on the private network, from any core node. The edge relay serves it over TLS and asks for a client certificate of the internal CA (O-26):
The JSON lists every stream with state.messages, state.first_seq and its consumers’ num_pending and num_ack_pending. platformctl nats check reports any difference between the server and the declared topology without changing anything.

Prometheus and Grafana

Then http://127.0.0.1:9090 and http://127.0.0.1:3000. site.yml applies the observability role to the edge and core nodes as its last play. The probe node (status page, Gatus, prober) is not converged by any playbook yet: it has no private address, so the base role refuses it.