Handover plan

Status (2026-11-13): tooling built and rehearsed locally; the independent rehearsal is blocked. The migration tooling, rotate-all.yml and the domain cutover playbooks exist (rebuild.md), and the data migration, the cutover by transfer and the credential rotation ran end to end on this repository’s local stand-ins (section 8). The rehearsal the plan requires, in which someone with only this repository rebuilds the system in a second cloud account and domain and transfers a repository to a second GitHub owner, needs input I12 and has not happened. Until it has, this plan is final in its decisions but not proven on real infrastructure. The asset-by-asset list is checklist.md.

• Handover happens between month 6 and month 12 of the 12-month service window, at SCF’s discretion. After 12 months the original operator has no further obligations.
• Handover is complete only when the new operator is independently running a fully working system. Mandatory: transfer of repository ownership, full data migration, and either transfer of domain ownership or a permanent redirect from the old domain.
• Infrastructure can be rebuilt from the open-source monorepo.

1. What is handed over

2. Phases of the handover

2.1 Appointment and preparation (before the shadow period)

1. SCF names the community operator and the date the handover starts.
2. The new operator reads the repository’s documentation: ../architecture.md, ../deploy/overview.md, ../runbooks/overview.md, ../sla.md, ../support.md and this plan.
3. The new operator creates its own accounts: a GitHub owner, a cloud project, a DNS account, a probe account, an alert receiver, and age identities for at least two people.

2.2 Shadow period (at least 4 weeks, proposed)

• The new operator’s people are added as repository collaborators and as age recipients of staging only.
• The new operator answers support issues, with the original operator reviewing, so the 5 and 10 business day targets are kept by the people who will own them.
• The new operator builds a complete environment in its own cloud account from the repository alone (independent rebuild), following ../deploy/overview.md. Every point where it needed knowledge not in the repository is fixed in the docs.
• The new operator runs the runbooks that can run without production write access on its own environment: edge node loss, database failover, point-in-time restore, Umami upgrade.
• The 4-week length is a proposal, long enough to see one monthly SLO report and one daily secret rotation cycle in the new environment; SCF decides the length.

2.3 Cutover: both environments run, ingestion never stops

1. Tenant notice. Every project is told the date, the new processor, and that nothing changes in its snippet when the domain moves (section 5). The processing terms require advance notice of a processor or sub-processor change; the length of that notice is open for the legal review (docs/compliance/dpa-template.md, input I10).
2. New environment built and bootstrapped, empty, in the new operator’s accounts (section 2.2), with sa_tls: cloudflare-delegated and sa_acme_challenge_domain set in its group_vars/all.yml.
3. Certificates before the delegation. The domain still delegates to the old DNS account, so Let’s Encrypt asks the old zone for every DNS-01 answer. The old operator adds one CNAME per name to the old zone (by hand or as extra records in its OpenTofu dns module):
   _acme-challenge.in.<domain>    CNAME  <sa_acme_challenge_domain>
   _acme-challenge.otlp.<domain>  CNAME  <sa_acme_challenge_domain>
   _acme-challenge.app.<domain>   CNAME  <sa_acme_challenge_domain>
   _acme-challenge.web.<domain>   CNAME  <sa_acme_challenge_domain>
   
   <sa_acme_challenge_domain> is a name in a Cloudflare zone the new operator’s token may edit (for example _acme-challenge.<a domain the new operator owns>). The new nodes’ Caddy (sa_tls_cloudflare-delegated in deploy/caddy/assemble.sh, Caddy’s dns_challenge_override_domain) writes the answer there, and Let’s Encrypt follows the CNAME. Expected: each new node serves a Let’s Encrypt certificate for its names when asked by address with SNI (docs/runbooks/certificates.md step 1), before anything moves.
4. Record TTLs. Traffic Manager answers with a 10 s or 30 s TTL behind 300 s CNAMEs (infra/tofu/envs/*/main.tf); the NS TTL at the TLD cannot be lowered and is the length of the parallel phase below.

5. Freeze the old core tier. On both old core nodes stop controlplane, replayer, metricswriter and jobs. The console is unavailable for the freeze (announced). The old edge nodes keep answering 202 after the JetStream acknowledgement, so events and metric batches queue in the old WEB_EVENTS (72 hours), API_METRICS and EDGE_STATS (7 days).
6. Data migration (rebuild.md sections 3 to 5) from the now frozen old databases: make migrate-export (METHOD=pgbackrest from the old repository, or dump), make migrate-import, jobs run reconcile, platformctl verify-migration, which must exit 0.
7. Start the new environment (make <env>-deploy). It now accepts ingestion on its own addresses and serves the console. Nothing points at it yet.
8. Start the queue forwarder on the controller, with an SSH tunnel to one old edge node’s NATS (4222) and one new edge node’s NATS:
   # old environment: the platformctl user of its sops file
   NATS_URL=nats://127.0.0.1:14222 NATS_USER=platformctl NATS_PASSWORD=<old> \
   TARGET_NATS_URL=nats://127.0.0.1:24222 TARGET_NATS_USER=platformctl TARGET_NATS_PASSWORD=<new> \
     dist/bin/platformctl queue forward
   
   It binds the consumers of the stopped old replayer, metrics writer and jobs (and refuses to start while one of them still pulls), publishes every queued message to the new environment’s stream with its subject, headers and Nats-Msg-Id unchanged, and acks it in the old stream only after the new stream acknowledged it, so a crash at any point loses nothing and stores nothing twice (cmd/platformctl/forward.go, TestQueueForward). It keeps running for the whole parallel phase.
9. Permission transfer (section 3) and domain move (section 5): the zone is recreated from OpenTofu in the new DNS account, then the registrar transfer or delegation change. From now on resolvers move to the new nodes as their cached delegation expires. Both stacks take events meanwhile: the new one directly, the old one through the forwarder.
10. Drain the old edge tier only when it is idle: the TLD’s NS TTL has passed since the delegation changed, the old edge Caddy request rate (Grafana “Ingestion” dashboard, or caddy_http_requests_total of the old nodes) has been zero for 24 hours, and the forwarder’s queues are empty. Then stop the old edge services, run platformctl queue forward --until-empty once more, and stop the forwarder.
11. Credential rotation (section 4).
12. The new operator’s probe and status page measure the service; the first SLO report under the new operator covers the month after cutover.

2.4 Removal of the original operator

• The original operator’s age keys are removed from every environment’s recipients, and every value they could read is rotated (section 4).
• Their GitHub access to the repository, their cloud, DNS, probe and registrar access, their alert receiver and any operator API keys they created are removed or revoked.
• Their GitHub id is removed from OPERATOR_GITHUB_IDS if it is there, and the control plane is restarted so the change takes effect at once (../operators/overview.md).
• The old environment is torn down after the advisory tail (section 6) or when SCF says so, and its backups are deleted at the end of their retention. Tenants are told when.

2.5 Advisory tail

3. Permission transfer

1. Repository: transfer to the new owner. Issues, pull requests and releases move with it, and GitHub redirects the old path; set up again what does not move (section 1).
2. Cloud, DNS and probe accounts: the new environment already runs in the new operator’s own accounts, so nothing moves; the old accounts are closed after teardown.
3. OAuth apps: the new environment uses the new operator’s apps from the start.
4. npm scope and trusted publishing: transferred, then a tracker release is published from the new repository to prove it.
5. Alerts: the new receiver is live before the old one is removed.

4. Credential rotation

• Rotating UMAMI_USER_PASSWORD_KEY changes every derived Umami password at once, so every open web dashboard session ends and users re-enter through the console.
• Umami’s APP_SECRET: the new environment uses its own and does not take the old one over. Umami signs its tokens with it and hashes session and visit ids with it; stored data does not depend on it (the local migration imported into an environment with a different value and every /stats answer was equal). On the cutover day a returning visitor is counted again and an open visit splits, which the new environment’s new daily visitor secrets cause anyway.
• OTLP tokens and tracker snippets do not change, so tenants change nothing in their code.
• Operator API keys of machine consumers are revoked and re-issued only if the original operator could have read them; keys are stored only as hashes, so a key the original operator never saw stays valid.

5. Domain: transfer, not redirect

• Transfer. The zone is recreated from OpenTofu in the new DNS account, then the registrar transfer moves the domain; every name, snippet and endpoint stays as it is. cutover-transfer.yml checks the new zone before the transfer and the public delegation, every name, certificates and the tracker bytes after it. If SCF registered the domain from day one, the transfer is a permission change.
• Redirect (cutover-redirect.yml, deploy/caddy/redirect.Caddyfile): kept for a domain that cannot be transferred. The old edge nodes answer 308 to the new names with the CORS header and keep serving the tracker files. Existing snippets and exporters stop delivering; every tenant must change its snippet and OTLP endpoint first, and the playbook refuses to run until that is acknowledged.

6. If no operator is appointed

1. Three months before the end of the window, the original operator opens an issue on the repository and asks SCF in writing whether an operator will be appointed. Without an answer within 30 days, the wind-down below is announced.
2. At least 60 days before the end, every project is notified of the end date (an issue on the repository, SCF’s own channels to awarded projects, and a mention of each project’s members by their GitHub logins, which is the only contact the service stores; the console has no message feature today), that the hosted service stops accepting events and API metrics on that date, and that its data is then deleted. Tenants are told how to remove the snippet and the second OTLP exporter; a page that keeps the snippet only produces failed requests to a name that stops answering.
3. Data export. Until the end date each project can read its numbers through the web dashboards and the console. There is no bulk export feature. If SCF wants one, or wants a copy of the aggregates for itself, it says so with the answer in step 1 and it is built before the notice; what is handed over then follows the processing terms (docs/compliance/dpa-template.md).
4. On the end date: ingestion stops (edge nodes drained and stopped), the dashboards stay readable for 14 more days, then every service stops.
5. Deletion: both databases and every backup are deleted with the servers, the object storage buckets and the repository encryption passphrases, within 30 days of the end date, and the deletion is confirmed on the repository issue. Logs and metrics expire with their retention (at most 30 days).
6. Accounts and domain: cloud, DNS and probe accounts are closed. The domain is offered to SCF; if SCF declines it, it is kept registered and parked for at least 12 months, with no DNS records for in. and otlp., so a third party cannot pick it up and receive the requests of pages and applications that were never cleaned up. Then it lapses.
7. The repository stays public under its licence with this plan, the IaC, the runbooks and a notice in the README that the hosted service ended, so anyone can rebuild the system from it (rebuild.md). If SCF appoints an operator later, the handover starts from a rebuild with no data to migrate.

7. Done when

• The new operator runs the system from its own accounts, built from the repository.
• verify-migration reports identical per-tenant aggregates and matching row counts, and tenants see their history.
• Existing tenant snippets deliver events to the new environment in three browser engines.
• Every old credential is refused.
• The repository is owned by the new operator.
• The domain is transferred (section 5: the redirect path was tested and is not safe).

8. Local rehearsal record (2026-11-13)

Data migration and cutover: make handover-it, passed

compared rows        75 keys
compared web         16 keys
compared web-dim     8 keys
compared api         8 keys
compared umami-stats 3 keys
not compared: platform.cp.leader_fence source 2 target 2 (leader lock fencing revisions ...)
result: source and target are equal

Credential rotation: rotate-all.yml on the minimal local converge, partly passed

• every previous credential refused and every new one accepted for all ten PostgreSQL logins (postgres, replicator over the replication protocol, umami_app, jobs_retention, sa_monitor and the five platform roles), all seven NATS users, the Umami admin and the console user gh-1001 (previous key refused, new key accepted);
• the previous NATS route password refused on all three edge nodes;
• the removed age identity opened neither the new nor the previous secrets file.