Tenant suspension and removal

Suspending a project

1. In the console operator area open the project and choose Suspend, or call POST /api/operator/tenants/{tenant_id}/suspend with {"reason": "..."}. The reason is required and the project’s members see it. Only an active project can be suspended; anything else answers 409 invalid_transition with the current status.
2. In one transaction the status becomes suspended with an audit row (tenant.suspend), then the reconciler switches the project’s entries in the registry_sites and registry_otlp buckets to suspended. From then on the gateways refuse the project’s web events and the collectors refuse its OTLP tokens. Refusals are counted, so the project’s ingest view shows them.
3. The Umami dashboards stay readable for the members. Data already stored stays. Members see the status and reason in the console.
4. Check it took effect:
   select status, status_reason, status_changed_at from cp.tenants where id = '<tenant id>';
   select at, action, detail from cp.audit_log
   where detail ->> 'tenant_id' = '<tenant id>' order by at desc limit 5;
   
   (run on the leader as in overview.md). The gateway’s refusal of a suspended site is rejected_inactive_site in cp.web_ingest_hourly.

Open question for SCF

• Default (the answer assumed until SCF decides): deleted.
• Setting: removal_keep_aggregates in cp.settings, changed by an operator with PUT /api/operator/settings/removal-keep-aggregates ({"keep": true}). Every change is in the audit log as settings.removal_keep_aggregates.
• The value is copied into cp.tenant_removals.keep_aggregates at the moment a project is removed. Changing the setting later never changes a removal that already started.
• keep = true keeps cp.web_ingest_hourly rows and, from phase 11, the agg rows of the project. Umami websites with their raw events, and from phase 9 the api minute rows, are always deleted.

Removing a project

1. In the console operator area, open the project and choose Remove, or call POST /api/operator/tenants/{tenant_id}/remove with {"reason": "..."}. The reason is required and the project’s members see it.
2. In one transaction the control plane sets the project to removed, sets every site to removed (domains stay on the rows), deletes open invites, inserts the cp.tenant_removals row and writes the tenant.remove audit row.
3. The reconciler then purges, store by store, in this order. Ingestion stops first, data goes after:

   Step	Store	What is deleted
   registry	NATS KV registry_sites	the key of every site, so the gateway refuses new events
   otlp_tokens	cp.otlp_tokens, registry_otlp (phase 8)	tokens revoked
   umami	Umami	both websites of every site with their data, the team, and the Umami users of members who belong to no other project that is not removed
   web_ingest	cp.web_ingest_hourly	the project’s rows, unless keep_aggregates
   api	api schema (phase 9)	minute rows, series and resource state, route caps and rollup marks always; hourly, daily and warning rows unless keep_aggregates. ingest_dedup ids stay until they expire (8 days), so a message still queued is not applied twice
   agg	agg schema (phase 11)	aggregate rows, unless keep_aggregates

   Each finished step is stored in cp.tenant_removals.steps and audited as purge.step (actor system:purge). When every step is done, completed_at is set and purge.complete is written.
4. The project row, its member rows and every audit row remain. Former members get 404 on every project route; /api/me lists the project under removed_tenants with the reason.

Following a removal

select tenant_id, requested_at, keep_aggregates, completed_at, steps
from cp.tenant_removals where tenant_id = '<id>';

select at, action, detail from cp.audit_log
where detail ->> 'tenant_id' = '<id>' and action like 'purge.%' order by at;

A step that keeps failing

• umami website ... belongs to team ...: an Umami website with the site’s id belongs to another team. Do not delete it blindly: find out why the ids clash, then delete or move it by hand and let the next reconcile continue.
• Umami or NATS unreachable: nothing to do once they are back.
• removed tenant has no cp.tenant_removals row: the status was set to removed by hand in the database. Removal must go through the operator api. Set the previous status back and remove the project through the api.

Exercised