Umami upgrade and rollback

Upgrade

1. Gate it on the development stack (needs make dev-up, Docker, Node 26 with pnpm and the Playwright browsers, Go 1.27.4 on PATH):
   make umami-upgrade VERSION=<x.y.z>
   
   scripts/umami-upgrade.sh resolves the index digest the registry serves for that version, moves the pin in deploy/compose/dev.yml, infra/ansible/vars/pins.yml, tests/observability/compose.yml and docs/reference/umami-contract.md, recreates the dev stack so Umami migrates the existing database, then runs umami-compat, the isolation gate, console-e2e and webpath, stopping at the first failure. Each step prints step <name>: ok in <n> s. A new version stays red at umami-compat until someone has reviewed its OpenAPI routes against the Caddy deny list and updated tests/umami-compat/testdata/openapi-routes.txt. Commit the diff only when every gate passed.
2. Release. Merge, tag v<release>; release-images.yml builds, scans and signs the platform images and attaches release-images.json. Umami itself is an upstream image and is pulled by its pinned digest.
3. Take a dump of the Umami database before the rollout. On the core node that holds the Patroni leader (overview.md):
   cd /opt/veristar/core
   sudo sh -c 'docker compose exec -T postgres pg_dump -U postgres -Fc -d umami > /root/umami-pre-upgrade.dump'
   sudo ls -l /root/umami-pre-upgrade.dump
   sudo docker compose exec -T postgres psql -U postgres -d umami -tAc \
     "select count(*) || ' migrations, last ' || max(migration_name) from _prisma_migrations where finished_at is not null"
   
   Write down the time and the migration line (for 3.5.0: 27 migrations, last 27_add_error_tracking). The point-in-time backups also cover this moment, but a dump of one database is far quicker to restore.
4. Roll out to staging, then production:
   make staging-deploy ROLLING=1 SA_IMAGE_TAG=<release> SA_RELEASE_MANIFEST=<path to release-images.json>
   
   deploy.yml drains and updates one node at a time and stops at the first unhealthy node. The new Umami migrates the database on its first start on the first core node.
5. Verify: ../deploy/verification.md steps for the core tier, UmamiHeartbeatFailing stays quiet, a tenant dashboard opens through the console’s SSO, and WEB_EVENTS drains (queue-backlog-and-dlq.md).

Rollback

1. Stop delivery into Umami on both core nodes, so nothing more is written to the database you are about to replace. Accepted events wait in WEB_EVENTS (72 hours):
   cd /opt/veristar/core && sudo docker compose stop replayer umami umami-2 umami-3 umami-4
   
   Run it on both core nodes. The list names every Umami process: umami plus umami-2 up to umami-<core_umami_processes> (4 by default; sudo docker compose ps shows them). Tenant web dashboards are down from here until step 5.
2. Restore the dump on the leader’s node:
   cd /opt/veristar/core
   sudo docker compose exec -T postgres psql -U postgres -d postgres -c 'drop database umami with (force)'
   sudo docker compose exec -T postgres psql -U postgres -d postgres -c 'create database umami owner umami_app'
   sudo docker compose exec -T postgres psql -U postgres -d postgres -c 'revoke all on database umami from public'
   sudo docker compose exec -T postgres psql -U postgres -d postgres -c 'grant connect on database umami to umami_app'
   sudo sh -c 'docker compose exec -T postgres pg_restore -U postgres -d umami --no-owner --role=umami_app < /root/umami-pre-upgrade.dump'
   
   Then recreate the retention role’s grants, which live in the Umami database: run platformctl db umami-retention-role as in overview.md.
3. Check the migration level matches what step 3 of the upgrade recorded:
   sudo docker compose exec -T postgres psql -U postgres -d umami -tAc \
     "select count(*) || ' migrations, last ' || max(migration_name) from _prisma_migrations where finished_at is not null"
   
4. Deploy the previous release, whose pins.yml names the previous Umami digest. In your working copy check out that release’s tag, then:
   make staging-deploy ROLLING=1 SA_IMAGE_TAG=<previous release> SA_RELEASE_MANIFEST=<its release-images.json>
   
   The deploy starts Umami and the replayer again.
5. Verify as in upgrade step 5. The replayer delivers what queued since step 1.

Exercised