> ## Documentation Index > Fetch the complete documentation index at: https://docs.veristar.app/llms.txt > Use this file to discover all available pages before exploring further. # Disaster recovery: both database nodes or the database lost > Bring the platform back when both core nodes or the database cluster are lost. Brings the platform back after both core nodes, or the PostgreSQL cluster on them, are gone: a destroyed location pair, an operator mistake that dropped data on the leader and its synchronous replica alike, or ransomware on both nodes. Finding O-4 of `docs/audit/functional-audit-ops.md`. Ingestion does not stop meanwhile: the edge tier keeps accepting events into JetStream (`WEB_EVENTS` 72 hours, `API_METRICS` 7 days), and they are delivered once the core tier is back. The path chains pieces that exist and are tested on their own: 1. a fresh **restore node** from OpenTofu (`restore_enabled`); 2. `restore.yml`, which restores the pgBackRest repository to a moment on that node; 3. `migrate-export METHOD=dump` from the restored server; 4. new core nodes with an empty, bootstrapped cluster on a **new repository generation**; 5. `migrate-import`, `verify-migration` against the restored server, then the services. Data loss window: what was not yet archived when the database was lost, at most `archive_timeout` (60 s, `roles/core/defaults/main.yml`) of committed transactions ([../reference/availability.md](https://docs.veristar.app/reference/availability.md)). ## Steps 1. **Stop what still runs on the core nodes**, if anything does, so nothing writes into a half-dead cluster: on each reachable core node `cd /opt/veristar/core && sudo docker compose stop`. 2. **Create the restore node.** In `infra/tofu/envs//terraform.tfvars` set `restore_enabled = true` (and `restore_zone` if zone 3 is affected), then `make -plan` and `make -apply`. Expected: `Plan: 1 to add` for `-restore-1` (a core-sized server on the private network at `.41`) and an inventory whose `restore` group holds it (`tests/infra/*.tftest.hcl` run `inventory_with_restore` checks this shape). 3. **Restore** to the moment before the loss (or to the end of the newest backup when the repository itself is the only survivor): ```sh theme={null} cd infra/ansible ansible-playbook -i inventories// playbooks/restore.yml \ -e sa_image_tag= -e sa_release_manifest= \ -e "restore_target='2026-11-20 07:59:00+00'" ``` If someone deleted from the repository too, read it as it was before the deletion by adding `-e '{"restore_extra_args": ["--repo=1", "--repo-target-time=2026-11-20 07:00:00+00"]}'` (the bucket is versioned and object-locked, [../deploy/backups.md](https://docs.veristar.app/deploy/backups.md)). Expected: `restored to ... on :5432`. 4. **Export** from the restored, standalone server (no services write to it) with an env file naming the restore node as in [../handover/rebuild.md](https://docs.veristar.app/handover/rebuild.md) section 4.1 (`PLATFORM_DB_HOST=`, port 5432, the superuser login from the SOPS file): ```sh theme={null} make migrate-export METHOD=dump MIGRATE_ENV= MIGRATE_DIR= ``` 5. **Replace the core nodes**: `tofu ... plan -replace=module.node["-core-1"].azurerm_linux_virtual_machine.this` (and core-2), as in [edge-node-failure.md](https://docs.veristar.app/runbooks/edge-node-failure.md) step 1, then `make -apply`. 6. **Remove the old cluster from etcd**, so the new nodes bootstrap a new cluster instead of waiting to join a leader that no longer exists. On any edge node: ```sh theme={null} cd /opt/veristar/edge sudo docker compose exec etcd etcdctl --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/etcd/tls/ca.crt --cert=/etc/etcd/tls/node.crt --key=/etc/etcd/tls/node.key \ del --prefix /veristar/sa-/ ``` 7. **Raise the repository generation.** The new cluster has a new system id, which the old stanza refuses. Add `backup_repo_generation: ` (it starts at 1) to the environment's `group_vars/all.yml`; the new cluster then archives to `/-g` in the same bucket, and the old generation stays readable and locked until its retention ends. Restores from the old generation pass `-e backup_repo_generation=`. 8. **Converge**: `make -deploy SA_IMAGE_TAG= SA_RELEASE_MANIFEST=<...>`. `site.yml` bootstraps an empty cluster, creates the stanza of the new generation, switches WAL archiving on and starts the services. 9. **Import and verify** with the services on the core nodes stopped (`sudo docker compose stop umami controlplane replayer metricswriter jobs` on both): ```sh theme={null} make migrate-import MIGRATE_ENV= MIGRATE_DIR= make verify-migration SOURCE_ENV= TARGET_ENV= ``` Expected: `verify-migration` exits 0 (row counts of every table, per-tenant daily web and API aggregates, Umami `/stats` for a sample of sites and days). Then start the services again (`make -deploy ...`), and the replayer and metrics writer drain what JetStream held. 10. **Take a full backup** of the new generation at once: `sudo systemctl start sa-pgbackrest-full.service` on the leader, then `pgbackrest --stanza=sa info` shows it. 11. **Remove the restore node**: `restore_enabled = false`, plan and apply. Record the recovery time (from step 2 to step 9) and the data loss window (the restore target against the last archived WAL) in `docs/reference/drills/.md`. ## Exercised | Date | What ran | Result | Source | | - | - | - | - | | 2026-11-13 | the restore node in the OpenTofu inventory (step 2) against mocked providers | `make tofu-check`: `inventory_with_restore` passes for staging and production | `tests/infra/*.tftest.hcl` | Not yet exercised: the whole chain on staging (needs the first apply), and steps 3 to 10 on the local converge. The local point-in-time drill (`tests/drills/pitr.sh`) covers step 3 only and has not been run either (plan phase 16). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.