Control plane HTTP API

• Plan reference: resources/plan.md sections 4.7 to 4.10 and phase 6.
• Database names: tables and columns are those of db/migrations/00002_cp_tables.sql and 00003_cp_security.sql. Where a response field maps to a column, the field has the column’s name.
• Research behind the choices: docs/reference/phase6-research.md.

1. Transport

1.1 Authentication

• Session cookie. Name sa_session. Value: 32 random bytes, base64url, made by scs. Attributes: HttpOnly, Secure, SameSite=Lax, Path=/, no Domain (host-only, so the cookie never reaches web. or in.<BASE_DOMAIN>). Persistent, absolute lifetime 7 days, idle timeout 24 hours. The database holds only the SHA-256 of the value (cp.sessions.token_hash, 32 bytes) and the user (cp.sessions.user_id). See phase6-research section 3 for why pgxstore is not used.
• The token is renewed (RenewToken) at sign-in, so a session id fixed before sign-in is useless afterwards.
• OAuth state cookie. sa_oauth, only during sign-in: HttpOnly, Secure, SameSite=Lax, Path=/api/auth/github/callback, Max-Age=600. Value: AES-256-GCM sealed (key OAUTH_STATE_KEY) JSON {state, verifier, return_to, exp}. No database row is written for a visitor who has not signed in.
• A route marked session answers 401 unauthenticated without a valid session. A route marked public needs none.

1.2 Cross-site request forgery

1. X-Requested-With: sa-console (exact, case-sensitive value). Missing or different: 403 csrf_header_missing. A cross-site form cannot set a custom header, and a cross-site fetch with one needs a CORS preflight, which the control plane never grants (it sends no Access-Control-Allow-* header on any route).
2. Origin equal to CONSOLE_ORIGIN (https://app.<BASE_DOMAIN>; dev http://localhost:8085). Missing or different: 403 origin_mismatch.

1.3 Tenant scoping

• Every route under /api/tenants/{tenant_id} and /api/sites/{site_id} first checks membership under cp_system (cp.tenant_members for the session user). A non-member gets 404 not_found, never 403, for every such route, also for ids that exist. The isolation tests (plan phase 6) assert this.
• The handler then runs under cp_tenant through db.WithTenant(ctx, tenant_id, fn).
• /api/sites/{site_id} resolves the site’s tenant_id under cp_system, then applies the same rule.
• A member whose role is not enough for the action gets 403 owner_required. Members already know the tenant exists, so 403 reveals nothing.
• Tenants with cp.tenants.status = 'removed' answer 404 for every member (phase 7 sets that status; the rule is listed here so phase 6 code already honours it).
• A route that checks the tenant status before a write (site create, member add) checks it again under the tenant row lock (LockTenant), so an operator removal, approval or suspension that commits while the request waits is what the write sees: removed answers 404, suspended 409 tenant_suspended, and a site’s pending or active status follows the locked tenant status (audit finding C-4).
• Every cp_tenant transaction sets lock_timeout = 5s. A request that would wait longer for a row lock (the reconciler holds the tenant row across its Umami calls) answers 503 unavailable with Retry-After instead of holding a database connection (audit finding C-1).
• cp_tenant may update only the cp.sites columns its routes write (name, origins, updated_at, status, collect_utm), only status and revoked_at of cp.otlp_tokens, and only role of cp.tenant_members. The verification and stellar_toml columns of 5.8 and 5.12 are written under cp_system by the verify route, the stellar.toml fetch after site creation and the verify-domains task, next to the check that decides them, so no tenant handler can mark a domain verified. Every tenant transaction carries the session user in app.user_id beside app.tenant_id. Under cp_tenant a site’s status may only change to removed, a token only from active to revoked, audit rows must name exactly the session user (user:<uuid>) as actor, and member rows may be added, changed or removed only when the session user is an owner of the tenant (a member may remove itself). Triggers enforce this (migrations 00024 and 00028, audit finding C-6); operator and system transitions run as cp_system.
• Reconcile after a change. Routes that say “Reconcile runs” ask for a run and answer at once. Requests coalesce per tenant: a tenant is queued once, and a change that arrives while its run is in flight makes exactly one more run follow, so any burst of edits costs at most two runs. At most 4 runs are in flight per control plane instance, on a cp_system pool of their own (4 connections), so session loads never wait behind them. The request pools are sized explicitly: cp_system 16, cp_tenant 16, cp_operator 8 (audit findings C-1 and C-8).

2. Response conventions

2.1 Success

2.2 Error envelope

{
  "error": {
    "code": "validation_failed",
    "message": "origins[1] must be a bare https origin",
    "fields": { "origins[1]": "not_bare_origin" },
    "request_id": "0193a6f2-7c1e-7d40-9f55-2b1c2a5e8a10"
  }
}

• code: stable machine value from 2.4. Clients branch on it, never on message.
• message: English text for people. May change.
• fields: present only for 422; maps a JSON path (name, origins[1], enrollment.stellar_ids[0]) to a field code from 2.5.
• request_id: also sent as response header X-Request-Id, and logged.

2.3 Lists and pagination

{ "items": [ ... ], "next_cursor": "eyJ0IjoiMjAyNi0xMS0xMlQwOTowMDowMFoiLCJpIjoiLi4uIn0" }

• Query limit (1 to 200, default 50) and cursor (the opaque next_cursor of the previous page). next_cursor is null on the last page.
• Order is fixed per route and documented with it; the cursor is a keyset over that order (for example (created_at, id)), so pages do not shift when rows are added.
• A cursor from another route or a tampered one: 400 invalid_cursor.
• Time-series routes (ingest counts) are not paginated; they bound the time range instead.

2.4 Error codes

2.5 Field codes (422)

3. Shared objects

3.1 Objects

{ "id": "uuid", "github_id": 1234567, "github_login": "octocat", "is_operator": false }

{
  "id": "uuid",
  "slug": "stellar-dex",
  "name": "Stellar DEX",
  "description": "",
  "status": "active",
  "status_reason": null,
  "enrollment_answers": {
    "website": "https://example.org",
    "stellar_usage": "Soroban AMM on mainnet",
    "stellar_ids": ["GA...", "CA..."],
    "scf_round_url": null
  },
  "accepted_terms_version": "2026-11-12-draft.1",
  "provisioning": "ready",
  "role": "owner",
  "created_at": "...",
  "updated_at": "...",
  "status_changed_at": "..."
}

• status: one of pending, active, rejected, suspended, removed (column check).
• provisioning: pending while cp.tenants.umami_team_id is null, ready after. The Umami team id itself is never returned.
• is_internal, created_by, approved_by and umami_team_id are not returned to tenants.

{ "user_id": "uuid", "github_id": 1234567, "github_login": "octocat", "role": "member", "created_at": "..." }

{ "id": "uuid", "github_id": 7654321, "github_login": "someone", "role": "member", "created_by": "uuid", "created_at": "...", "expires_at": "..." }

{
  "id": "uuid",
  "tenant_id": "uuid",
  "name": "Marketing site",
  "domain": "example.org",
  "origins": ["https://example.org", "https://www.example.org"],
  "quarantine_website_id": "uuid",
  "status": "active",
  "verify_method": null,
  "verified_at": null,
  "collect_utm": false,
  "created_at": "...",
  "updated_at": "..."
}

• status: pending, active, suspended, removed (column check).
• collect_utm: whether the gateway keeps the utm_source, utm_medium, utm_campaign, utm_content, utm_term and ref query parameters of page URLs for this site. false on every new site and on every site that existed before the field (migration 00027); owners turn it on with 4.3.4 (audit N-7). With false the gateway drops every query parameter. Published to the gateway in the site’s registry_sites entry.
• verify_token, last_verify_* and stellar_toml* are phase 7 fields. Phase 7 adds them to this object; verify_token is shown only through the phase 7 verification route.

3.2 Validation rules

3.3 Audit

• actor: user:<cp.users.id> for console actions, system:<task> for background work (system:reconcile, system:umami-rotate).
• action: dotted name from the list below.
• target: tenant:<id>, site:<id>, user:<id> or invite:<id>.
• detail: JSON with the changed fields (old and new values for updates). Never a token, cookie, password or secret.

4. Phase 6: sign-in, tenants, members, sites, snippet, ingest, SSO

4.1 Auth

4.1.1 GET /api/auth/github/start

4.1.2 GET /api/auth/github/callback

1. Read and open sa_oauth; delete it in the response whatever happens next.
2. state must equal the cookie’s state (constant-time compare) and the cookie must not be expired. Otherwise: failure state_mismatch.
3. error=access_denied: failure access_denied. Any other error: failure github_error.
4. The cookie’s state must not have reached this step before on this control plane instance; a replayed copy of the cookie is failure state_mismatch (states are remembered until their cookie expires; audit finding C-2). Then take a token from the exchange buckets: the client’s (0.5 per second, burst 20, keyed as in 4.1.1) and the instance’s ceiling (10 per second, burst 30), or, for a client whose bucket was full, the reserve (2 per second, burst 10) when the ceiling is empty. Empty: failure rate_limited. Then exchange code with the cookie’s verifier (oauth2.VerifierOption). Failure: exchange_failed.
5. GET https://api.github.com/user with the access token, headers Accept: application/vnd.github+json and X-GitHub-Api-Version: 2022-11-28. Use id (int64) and login only; the token is then discarded, never stored. Failure: github_unavailable.
6. Under cp_system, in one transaction: upsert cp.users (UpsertUser: key github_id, refresh github_login and last_login_at); resolve open, unexpired cp.invites whose github_id equals the user’s GitHub id into cp.tenant_members and delete those invites (audit invite.resolve); audit user.sign_in. The login is never matched: it can be renamed and taken by someone else (audit finding C-5).
7. Renew the session token, store user_id, answer 303 to return_to.

4.1.3 POST /api/auth/logout

4.1.4 GET /api/me

{
  "user": { "id": "uuid", "github_id": 1234567, "github_login": "octocat", "is_operator": false },
  "tenants": [
    { "id": "uuid", "slug": "stellar-dex", "name": "Stellar DEX", "status": "active", "role": "owner",
      "accepted_terms_version": "2026-11-12-draft.1", "terms_current": true }
  ],
  "terms": { "current_version": "2026-11-12-draft.1", "status": "draft", "url": "/terms" }
}

4.1.5 GET /api/terms

{ "version": "2026-11-12-draft.1", "status": "draft", "markdown": "# Terms ..." }

4.2 Tenants

4.2.1 POST /api/tenants

{
  "name": "Stellar DEX",
  "slug": "stellar-dex",
  "description": "",
  "enrollment": {
    "website": "https://example.org",
    "stellar_usage": "Soroban AMM on mainnet",
    "stellar_ids": ["GA..."],
    "scf_round_url": "https://communityfund.stellar.org/..."
  },
  "accepted_terms_version": "2026-11-12-draft.1"
}

• slug and description optional, enrollment.stellar_ids and enrollment.scf_round_url optional; the rest required (3.2).
• accepted_terms_version must equal the current version: else 409 terms_version_mismatch.
• Under cp_system, one transaction: insert cp.tenants (enrollment_answers = the enrollment object, accepted_terms_version, created_by, status from cp.settings enrollment_mode: open gives active; approval, added in phase 7, gives pending), insert the creator into cp.tenant_members with role owner, audit tenant.create. Then trigger Reconcile(tenant_id) asynchronously.
• 201 Tenant (3.1), provisioning: "pending" until the reconciler finishes.
• Errors: 409 slug_taken, 409 terms_version_mismatch, 422, 429 (at most 10 tenants created per user per day).

4.2.2 GET /api/tenants

4.2.3 GET /api/tenants/{tenant_id}

4.2.4 PATCH /api/tenants/{tenant_id}

4.2.5 Members and invites

{ "members": [Member, ...], "invites": [Invite, ...] }

{ "github_login": "someone", "role": "member" }

• The login is first resolved to the GitHub account that holds it now: GET https://api.github.com/users/{login} with the OAuth app’s client id and secret as basic auth (GitHub’s public-data rate limit for an OAuth app, instead of the 60 per hour of an anonymous caller), headers as in 4.1.2 step 5. Everything below uses that account’s numeric id; the login is kept for display only (audit findings C-5 and C-10). A login renamed later, and then taken by someone else, never moves a membership or an invite.
• If a cp.users row with that GitHub id exists, the user is added to cp.tenant_members at once and the row’s github_login is refreshed to GitHub’s current login: 201 {"kind": "member", "member": Member}, audit member.add.
• Otherwise an invite bound to that GitHub id is created with expires_at = now() + 14 days: 201 {"kind": "invite", "invite": Invite}, audit invite.create (detail includes github_id). It is resolved when that account next signs in, under whatever login it has then (4.1.2 step 6).
• A login GitHub does not know, or one that names an organization (which cannot sign in), is 422 unknown_github_login on github_login. GitHub unreachable or answering anything else: 503 unavailable.
• Invites made before migration 00025 carried no GitHub id; that migration revoked them (one invite.revoke audit row each, actor system:migration), and owners send them again.
• Errors: 409 already_member, 409 invite_exists, 422, 422 unknown_github_login, 503 unavailable, and 422 too_many on field github_login when the tenant would exceed 50 members plus open invites.
• Reconcile runs after the change (Umami membership).

4.2.6 POST /api/tenants/{tenant_id}/terms

4.3 Sites

4.3.1 POST /api/tenants/{tenant_id}/sites

{ "name": "Marketing site", "domain": "example.org", "origins": ["https://example.org"] }

• The control plane makes id and quarantine_website_id (UUIDv7 each) and verify_token (32 random bytes, base64url, used from phase 7).
• status is active when the tenant is active, else pending, with the tenant status read under the tenant row lock (1.3): a tenant removed while the request waited answers 404, one suspended meanwhile 409 tenant_suspended.
• Under cp_tenant: insert cp.sites (CreateSite), audit site.create. Reconcile then creates the two Umami websites and publishes registry_sites.
• 201 Site.
• Errors: 422, 422 too_many on field name above 50 non-removed sites per tenant.

4.3.2 GET /api/tenants/{tenant_id}/sites

4.3.3 GET /api/tenants/{tenant_id}/sites/{site_id}

4.3.4 PATCH /api/tenants/{tenant_id}/sites/{site_id}

4.3.5 DELETE /api/tenants/{tenant_id}/sites/{site_id}

4.3.6 GET /api/sites/{site_id}/snippet

{
  "site_id": "uuid",
  "version": "1.0.0",
  "src": "https://in.<BASE_DOMAIN>/v/1.0.0/sdk.js",
  "integrity": "sha384-G7jAlHsp/0HhreHeIKFZIr1Udyb8lXfTJPPZcLC8YcVCAQ9Mr/OS3JOMW4n4OT6j",
  "tag": "<script defer src=\"https://in.<BASE_DOMAIN>/v/1.0.0/sdk.js\" integrity=\"sha384-...\" crossorigin=\"anonymous\" data-website-id=\"uuid\"></script>",
  "npm": null,
  "versions": [ { "version": "1.0.0", "released": "2026-11-12", "bytes": 6734, "gzip_bytes": 3273 } ]
}

• tag is built by internal/controlplane/snippet.go and is byte-identical to snippet({siteId, version, origin}) in packages/tracker/src/snippet.ts with origin = INGEST_ORIGIN (attribute order and quoting are part of that contract).
• integrity equals the manifest’s integrity for the version.
• versions: every non-revoked manifest entry, newest first.
• npm: null until the tracker package is published (phase 18, input I9). Then {"package": "<name>", "version": "1.0.0", "install": "pnpm add <name>@1.0.0", "import": ...}. The import line is not yet defined: phase 18 defines it with the published package. Until then the control plane refuses to start when TRACKER_NPM_PACKAGE is set, rather than answer with a guessed import. The console hides the npm tab while npm is null.
• Errors: 404 not_found (site), 422 unknown_tracker_version on version, 410 tracker_version_revoked, 409 site_removed.

4.3.7 GET /api/sites/{site_id}/ingest

{
  "site_id": "uuid",
  "from": "2026-11-11T09:00:00Z",
  "to": "2026-11-12T09:00:00Z",
  "bucket": "hour",
  "series": [
    { "at": "2026-11-11T09:00:00Z", "counts": { "accepted": 120, "quarantined": 3, "rejected_origin": 2 } }
  ],
  "totals": { "accepted": 2810, "quarantined": 41, "bot": 9, "dlq": 0, "rejected_origin": 17 }
}

• Outcomes are the values stored in cp.web_ingest_hourly.outcome: accepted, quarantined, bot, dlq and rejected_<reason> (internal/events outcomeRe). Unknown future outcomes are passed through; the console shows them under “other rejections”.
• Buckets with no rows are omitted from series; counts holds only non-zero outcomes.
• Rows are written by the jobs task edgestats and lag by up to one fold interval; the console says “updated every few minutes”.

4.4 SSO to Umami

4.4.1 GET /api/sso/umami

• Requires Sec-Fetch-Site same-origin or none when the header is present; otherwise failure cross_site.
• The site must not be removed or pending (in review, 5.6: the reconciler creates no Umami website for it); the tenant’s Umami user and team membership must exist (reconciler done). Otherwise failure provisioning_pending (audit finding C-7).
• Logs in to Umami on the private network (POST /api/auth/login) as gh-<github_id> with that user’s derived password for the current UTC day (plan 4.8). It never sets a password.
• 302 to https://web.<BASE_DOMAIN>/sso?token=<token>&url=/websites/<id> where <id> is the site id for primary and quarantine_website_id for quarantine. Response headers: Cache-Control: no-store, Referrer-Policy: no-referrer.
• The token is never logged. Caddy on app. cuts the query from its access log; the response Location header is removed from logs as well.
• Failure: 303 to /sso-error?code=<code>&site=<site_id>, code one of not_found, cross_site, provisioning_pending, umami_unavailable, internal.
• Rate limit: 30 per user per minute; over it the failure code is rate_limited.
• Removed members. Reconcile removes the Umami team membership of a removed member at once, but it does not change that user’s password, so an Umami token the person already holds still passes /api/auth/verify until the next daily umami-rotate. Every request for the tenant’s websites with that token is refused with 401 from the moment the membership is gone, so no data is reachable; only the token itself stays syntactically valid for at most about a day.

4.5 Configuration keys (control plane, phase 6)

5. Phase 7: operator governance

5.1 Operator identity

• cp.users.is_operator is set from OPERATOR_GITHUB_IDS at every sign-in (4.1.2, unchanged) and once at control plane start: one statement under cp_system sets it to true for listed github_id values and false for every other row. Each changed user gets one audit row user.operator_sync (actor system:startup, target user:<id>, detail {"is_operator": {"old", "new"}}).
• Every route under /api/operator/ is session and operator: the session user must have cp.users.is_operator = true (read per request) and a github_id in the loaded OPERATOR_GITHUB_IDS. Otherwise 403 operator_required, for every method and for unknown ids alike. Removing an id takes effect at that person’s next sign-in or the next control plane start, whichever comes first.
• CSRF rules (1.2) apply unchanged. The tenant-scoping rules of 1.3 do not apply to operator routes: an operator sees every tenant except is_internal ones, which answer 404 on operator routes.
• Database roles: reads of cp.tenants, cp.operator_sites, cp.web_ingest_hourly, cp.settings and cp.tenant_removals run as cp_operator; actions, setting changes, the member list of 5.4 and the audit list of 5.9 run as cp_system.
• Operators are not made members of tenants and get no Umami access (plan 4.8).

5.2 Operator objects

{
  "id": "uuid",
  "slug": "stellar-dex",
  "name": "Stellar DEX",
  "status": "pending",
  "status_reason": null,
  "website": "https://example.org",
  "created_at": "...",
  "status_changed_at": "...",
  "site_count": 2,
  "sites_in_review": 0,
  "sites_verified": 1,
  "sites_with_stellar_toml": 1
}

• website is enrollment_answers.website.
• site_count counts sites with status <> 'removed'. sites_in_review counts pending sites of an active tenant (5.6). sites_verified uses the rule of 5.8. sites_with_stellar_toml counts sites whose stellar_toml.status is found.

{
  "id": "uuid",
  "name": "Marketing site",
  "domain": "example.org",
  "origins": ["https://example.org"],
  "status": "active",
  "status_reason": null,
  "in_review": false,
  "domain_previously_removed": false,
  "verification": "verified",
  "verify_method": "dns",
  "verified_at": "...",
  "last_verify_at": "...",
  "last_verify_error": null,
  "stellar_toml": { "...": "StellarToml of 5.12" },
  "stellar_toml_checked_at": "...",
  "created_at": "...",
  "updated_at": "..."
}

• in_review: status = 'pending' and the tenant is active.
• domain_previously_removed: the domain equals, is a subdomain of, or is a parent of the domain of a site whose tenant is removed (5.6).
• verification: verified, unverified or unchecked (5.8).

5.3 GET /api/operator/tenants

• Keyset over (<sort column>, id) in the requested order; the cursor encodes the sort and order, and a cursor used with another sort or order is 400 invalid_cursor. name sorts by lower(name).
• Tenants with is_internal = true are never listed.
• One query serves every sort: the keyset runs over (sort key, id), where the sort key of a timestamp is its UTC ISO form with microseconds and the key of name is lower(name).
• 400 invalid_query for an unknown status, sort or order, or q out of range.

5.4 GET /api/operator/tenants/{tenant_id}

{
  "tenant": {
    "id": "uuid", "slug": "...", "name": "...", "description": "...",
    "status": "pending", "status_reason": null,
    "enrollment_answers": { "website": "...", "stellar_usage": "...", "stellar_ids": ["G..."], "scf_round_url": null },
    "accepted_terms_version": "...",
    "provisioning": "pending",
    "created_by": { "id": "uuid", "github_id": 1, "github_login": "octocat" },
    "approved_by": null,
    "created_at": "...", "updated_at": "...", "status_changed_at": "..."
  },
  "stellar_ids": [ { "id": "G...", "type": "account", "valid": true } ],
  "members": [ { "github_id": 1, "github_login": "octocat", "role": "owner", "created_at": "..." } ],
  "sites": [ "OperatorSite", "..." ],
  "ingest": {
    "last_24h": { "accepted": 2810, "quarantined": 41, "rejected_origin": 17 },
    "last_30d": { "accepted": 90210 }
  },
  "removal": null,
  "recent_audit": [ "AuditEntry of 5.9, newest 20 with detail.tenant_id = this tenant" ],
  "allowed_actions": ["approve", "reject", "remove"]
}

• stellar_ids: the enrollment ids re-checked with strkey (type is account, muxed_account or contract).
• members excludes nothing; operators see logins to contact a project. Read under cp_system.
• ingest sums cp.web_ingest_hourly over all the tenant’s sites for the last 24 full hours and 30 full days (UTC), non-zero outcomes only.
• removal: null, or for a removed tenant {"requested_at", "requested_by", "keep_aggregates", "completed_at", "steps": {"registry": {"done_at": "..."}, ...}} from cp.tenant_removals.
• allowed_actions: the actions of 5.5 valid from the current status, so the console shows only those buttons.
• 404 not_found for unknown, malformed or internal ids.

5.5 Tenant lifecycle actions

1. cp.tenants.status = 'removed', status_reason = the reason.
2. Every site of the tenant: status = 'removed', updated_at = now(). Domains stay on the rows, which drives the review rule of 5.6.
3. Open cp.invites of the tenant are deleted, so none resolves later.
4. Insert cp.tenant_removals (tenant_id, requested_at, requested_by, reason, keep_aggregates, completed_at = null, steps = '{}'), keep_aggregates copied from cp.settings.removal_keep_aggregates at this moment.

5.6 Sites in review

5.7 Enrollment mode and pending tenants

• The mode is read at each enrollment (4.2.1), so a switch takes effect for the next enrollment without a restart and never changes existing tenants.
• pending and rejected tenants: the reconciler creates nothing in Umami and publishes nothing to registry_sites, so the gateway answers rejected_unknown_site for their sites. SSO answers provisioning_pending. Owners can still add, edit and delete sites and manage members while pending or rejected.

5.8 Site domain verification

{
  "site_id": "uuid",
  "domain": "example.org",
  "verify_token": "43 characters base64url",
  "dns": {
    "name": "_veristar.example.org",
    "type": "TXT",
    "value": "veristar-verify=<verify_token>"
  },
  "file": {
    "url": "https://example.org/.well-known/veristar.txt",
    "body": "<verify_token>"
  },
  "verification": "unverified",
  "verify_method": null,
  "verified_at": null,
  "last_verify_at": "...",
  "last_verify_error": "dns_no_record"
}

• DNS check: a TXT record at dns.name whose value (strings of one record joined) equals dns.value exactly.
• File check: GET file.url through the request-forgery guard (phase7-research section 3), status 200, body at most 1,024 bytes, equal to the token after trimming trailing whitespace.
• On success: verify_method, verified_at = now(), last_verify_at = now(), last_verify_error = null. On failure: last_verify_at = now(), last_verify_error = <code>; verified_at and verify_method keep the last success.
• verification: verified when verified_at is not null and last_verify_error is null; unchecked when last_verify_at is null; unverified otherwise.
• last_verify_error codes: dns_no_record, dns_mismatch, dns_error, file_not_found (any status other than 200), file_mismatch, file_too_large, refused_address, redirect_refused, tls_error, timeout, fetch_error. When both methods were tried, the file method’s code is stored and the DNS code is in the audit detail.
• The jobs task verify-domains (actor system:verify-domains) re-runs both checks for every non-removed site of a non-removed tenant whose last_verify_at is older than 24 hours, plus the stellar.toml fetch (5.12). Its site.verify audit row is written only when the result changes (verification value or error code), so the daily run does not add one row per site per day; POST .../verify always writes one.

5.9 GET /api/operator/audit

{ "id": 1234, "at": "...", "actor": "user:uuid", "actor_login": "octocat",
  "action": "tenant.suspend", "target": "tenant:uuid",
  "detail": { "tenant_id": "uuid", "status": { "old": "active", "new": "suspended" }, "reason": "..." } }

• id is the cp.audit_log identity (a number). actor_login is the github_login for user: actors, else null.
• Query filters, all optional and combined with AND: tenant_id (rows whose target is tenant:<id> or whose detail.tenant_id is <id>), target (exact), action (exact, or a prefix ending in ., such as tenant.), actor (exact), from and to (RFC 3339, from < to, at most 400 days apart), limit, cursor.
• 400 invalid_query for a malformed value.

5.10 Tenant-visible status and reason

• Tenant (3.1) already carries status, status_reason and status_changed_at. The console shows a banner on every project page for pending (“waiting for review”), rejected (reason plus the resubmit form of 5.7), suspended (reason; data stays readable, new events are refused). No banner for active.
• GET /api/me (4.1.4) gains removed_tenants: projects the user is a member of whose status is removed, each {"id", "name", "status_reason", "status_changed_at"}, newest first, at most 20. Every other route still answers 404 for them (1.3). The console shows each once as a notice with the reason.
• Site (3.1) gains status_reason, in_review, verification, last_verify_at, last_verify_error, stellar_toml and stellar_toml_checked_at. verify_token stays out of Site (5.8 only).

5.11 Effects of status on tenant routes

5.12 stellar.toml signal

{
  "status": "found",
  "error": null,
  "http_status": 200,
  "bytes": 2048,
  "version": "2.7.0",
  "network_passphrase": "Public Global Stellar Network ; September 2015",
  "network": "public",
  "accounts_count": 3,
  "accounts_valid": 3,
  "signing_key_present": true,
  "signing_key_valid": true,
  "org_url": "https://example.org",
  "org_url_same_domain": true,
  "currencies": [
    { "code": "USDX", "issuer": "G...", "contract": null, "valid": true },
    { "code": "TKN", "issuer": null, "contract": "C...", "valid": true }
  ],
  "currencies_total": 2
}

• status: found (parsed), not_found (any status other than 200), invalid (not TOML, or wrong type for a read field), too_large, refused (address or redirect refused by the guard), error (timeout, TLS, network). Every field except status, error and http_status is null (or 0, false, empty list) unless status is found.
• error: the guard or parser code for non-found statuses (refused_address, redirect_refused, tls_error, timeout, fetch_error, toml_syntax, toml_type), else null.
• network: public, testnet, futurenet (the SDK network constants), other for any other passphrase, null when absent.
• org_url is [DOCUMENTATION].ORG_URL; org_url_same_domain compares its host with domain (SEP-1: the file “must be hosted on the same domain”).
• currencies: at most 50 entries kept, currencies_total is the full count. valid is true when exactly one of issuer (valid G strkey) or contract (valid C strkey) is set. code is cut to 12 characters. No other text from the file is stored, so tenant-controlled prose never reaches the console.
• Tenants see the same object on their Site; operators on OperatorSite.
• Audit site.stellar_toml only when status or network changes from the previous value, so the daily run does not write one row per site per day.

5.13 removal_keep_aggregates

• The value is copied into cp.tenant_removals.keep_aggregates when a tenant is removed (5.5), so changing it later never changes a removal in progress.
• true keeps cp.web_ingest_hourly rows, api.metrics_hourly, api.metrics_daily and api.ingest_warnings rows and, from phase 11, agg rows of the removed tenant. Umami websites, their raw data and api minute rows are always deleted. The open SCF question behind the default is recorded in docs/runbooks/tenant-removal.md.

5.14 Configuration keys (phase 7)

6. Phase 8: OTLP tokens

6.1 Token object

{
  "id": "uuid",
  "tenant_id": "uuid",
  "name": "Production API",
  "environment": "production",
  "prefix": "sa_otlp_Ab3x",
  "status": "active",
  "created_by": "uuid",
  "created_at": "...",
  "revoked_at": null,
  "last_seen_at": null
}

• Format (plan 4.10): sa_otlp_ followed by 40 characters drawn uniformly from 0-9A-Za-z with crypto/rand.
• Storage: hash is the SHA-256 of the whole token (32 bytes); prefix is the first 12 characters (sa_otlp_ plus 4 random ones), for display only. The token itself is stored nowhere: not in the database, the audit log, the registry or a log line.
• status: active or revoked (column check). last_seen_at is stamped by the metrics writer (phase 9).

6.2 POST /api/tenants/{tenant_id}/otlp-tokens

{ "name": "Production API", "environment": "production" }

• name: 1 to 100 characters (text rules of 3.2). environment: optional, empty or ^[a-z0-9]([a-z0-9._-]{0,31})?$, else 422 invalid_format.
• The tenant must be active: 409 tenant_suspended while suspended, 409 tenant_not_active while pending or rejected. The status is read again under the tenant row lock in the insert transaction, so a concurrent suspension waits for the insert and then reconciles the new token.
• At most 50 active tokens per tenant: 422 too_many on name.
• Under cp_tenant: insert, audit otlp_token.create (detail: tenant_id, name, environment, prefix), then write the registry_otlp entry {tenant_id, token_id, status: "active"} under the hex SHA-256 before the commit. A registry failure rolls the insert back: 503 unavailable.
• 201 Token object plus "token": "sa_otlp_...", with Location set to the token’s URL. This answer is the only place the token ever appears.

6.3 GET /api/tenants/{tenant_id}/otlp-tokens and GET .../otlp-tokens/{token_id}

6.4 DELETE /api/tenants/{tenant_id}/otlp-tokens/{token_id}

• Under cp_tenant with the row locked: status = 'revoked', revoked_at = now(), audit otlp_token.revoke (detail tenant_id, status old and new), then the registry_otlp entry is set to status: "revoked" before the commit. A registry failure rolls back: 503 unavailable, so a revocation never looks done while the key still says active.
• 204. Repeating the call: 204, no new audit row, the registry entry is written again (which heals a lost write).

6.5 Registry state and reconcile

• Keys are never deleted, so an inactive token is refused as such, not as unknown.
• The reconciler (provision, after every lifecycle action and every 10 minutes in jobs) writes each entry that differs, first in its run and before any Umami call, so suspension and resume reach the collector even while Umami is unreachable. Each write is audited as provision.otlp_registry_put (actor system:reconcile, target otlp_token:<id>, detail tenant_id, status).
• Removal (5.5): the otlp_tokens purge step runs right after registry. It sets every active row to revoked and flips every key to revoked; its step detail lists tokens_revoked and keys_revoked. Revocation by removal is final.
• NATS users: controlplane and jobs may write registry_otlp (jobs runs the reconcile task); collector only watches it.

6.6 Effects of tenant status (extends 5.11)

7. Phase 10: API metrics

7.1 Query parameters

• A parameter not listed for the route, or given twice, is 400 invalid_query.
• Filter values are compared for equality and always sent to PostgreSQL as bound parameters, never spliced into SQL, so quotes, %, _ and SQL keywords are literal characters that simply match nothing unless a stored value has them.
• A filter present with an empty value (service=) matches the empty value, which is how a point without that attribute is stored. An absent parameter means no filter.
• Filter values must be valid UTF-8 without control characters (U+0000 to U+001F, U+007F) and at most the column limit in bytes: service 128, method 16, route 1,024. Anything else is 400 invalid_query.

7.2 Range and resolution

1. Resolution minute and from before the minute horizon (start of the current UTC day minus API_MINUTE_RETENTION_DAYS days, the first day whose minute partition the retention job keeps): the resolution becomes hour, from and to are aligned to hours, resolution_note is older_than_minute_retention.
2. Resolution hour and from before the hourly horizon (first day of the current UTC month minus API_HOURLY_RETENTION_MONTHS months): the resolution becomes day, aligned to days, resolution_note is older_than_hourly_retention.

{
  "preset": "24h",
  "from": "2026-11-11T10:00:00Z",
  "to": "2026-11-12T10:00:00Z",
  "resolution": "hour",
  "bucket_seconds": 3600,
  "resolution_note": null
}

• preset: the requested preset, custom, or previous for the comparison range of 7.5.
• bucket_seconds: 60, 3600 or 86400 (UTC days).
• resolution_note: null, older_than_minute_retention or older_than_hourly_retention.

7.3 Source selection

• metric rows when at least one metric row of that service exists in the range,
• otherwise span rows.

{ "service": "payments-api", "source": "span", "span_data_ignored": false }

• source: metric or span. The console shows a “from traces” badge for span with one line: “Derived from sampled traces; counts are lower than the real request count when traces are sampled.”
• span_data_ignored: true when the service has span rows in the range besides its metric rows, which is the normal case for a service that exports both metrics and traces. Only the metric rows are used. The console shows “metric data used; span-derived data ignored for this service”. It does not imply the service switched sources; a service that moved from spans to metrics during the range shows the same flag.

7.4 Numbers

• requests: sum(count) of the selected rows.
• errors_4xx: requests with status_code 400 to 499; errors_5xx: 500 to 599. status_code 0 means the point had no status attribute; it counts in requests and in class unknown, never as an error.
• rate_4xx, rate_5xx: errors divided by requests, a fraction between 0 and 1; null when requests is 0.
• p50, p95, p99: api.histogram_quantile(q, api.sum_buckets(counts)) over exactly the rows the number describes (plan 4.11): bucket arrays are summed first, then interpolated linearly inside the bucket. Percentiles are never averaged, neither across rows, buckets, services nor time. Value: a JSON number in seconds; the string ">10 s" when the percentile falls in the open bucket above 10 s (never a number); null when there are no requests.
• lossy: true when any selected row was re-bucketed inexactly from a non-canonical histogram layout (phase 9), so percentiles are approximate. The console shows “approximate” next to such percentiles.

{
  "requests": 1200, "errors_4xx": 30, "errors_5xx": 2,
  "rate_4xx": 0.025, "rate_5xx": 0.0016666666666666668,
  "p50": 0.042, "p95": 0.31, "p99": ">10 s", "lossy": false
}

7.5 GET .../api-metrics/overview

{
  "current":  { "range": { ... }, "sources": [ ... ], "summary": { ... } },
  "previous": { "range": { ... }, "sources": [ ... ], "summary": { ... } }
}

7.6 GET .../api-metrics/timeseries

{
  "range": { ... },
  "sources": [ ... ],
  "points": [
    { "at": "2026-11-12T09:00:00Z", "requests": 52, "errors_4xx": 1, "errors_5xx": 0,
      "rate_4xx": 0.019230769230769232, "rate_5xx": 0, "p50": 0.031, "p95": 0.2, "p99": 0.48, "lossy": false }
  ]
}

7.7 GET .../api-metrics/routes

• sort: requests (default), errors_4xx, errors_5xx, rate_4xx, rate_5xx, p50, p95, p99, route (orders by service, method, route).
• order: asc or desc; default asc for route, desc otherwise.
• Ties are broken by service, method and route ascending. null values sort last in both orders. ">10 s" sorts above every number.
• limit 1 to 200, default 50. cursor is the opaque next_cursor of the previous page. Pages are offsets into the sorted list: the cursor is bound to the route, sort, order, range, from, to and the filters, and is 400 invalid_cursor with any other combination. Because the numbers keep changing as data arrives, an item can move between pages; total is the item count at the time of the request.

{
  "range": { ... },
  "sources": [ ... ],
  "total": 134,
  "items": [
    { "service": "payments-api", "method": "GET", "route": "/v1/accounts/{id}", "source": "metric",
      "requests": 930, "errors_4xx": 12, "errors_5xx": 0, "rate_4xx": 0.012903225806451613, "rate_5xx": 0,
      "p50": 0.04, "p95": 0.3, "p99": 0.9, "lossy": false }
  ],
  "next_cursor": null
}

7.8 GET .../api-metrics/status

{
  "range": { ... },
  "sources": [ ... ],
  "requests": 1200,
  "classes": [
    { "class": "1xx", "requests": 0 }, { "class": "2xx", "requests": 1150 },
    { "class": "3xx", "requests": 18 }, { "class": "4xx", "requests": 30 },
    { "class": "5xx", "requests": 2 }, { "class": "unknown", "requests": 0 }
  ],
  "codes": [
    { "status_code": 200, "class": "2xx", "requests": 1100 }
  ]
}

• classes: always these six, in this order; unknown is status_code 0.
• codes: every status code with requests, ascending.

7.9 GET .../api-metrics/services

{
  "range": { ... },
  "items": [
    { "service": "payments-api", "source": "metric", "span_data_ignored": false,
      "requests": 1200, "lossy": false }
  ]
}

7.10 GET .../api-metrics/warnings

{
  "range": { ... },
  "totals": [
    { "kind": "span_derived", "count": 0, "last_day": null },
    { "kind": "lossy_histogram", "count": 4, "last_day": "2026-11-12" },
    { "kind": "cardinality_overflow", "count": 0, "last_day": null },
    { "kind": "secret_in_path", "count": 1, "last_day": "2026-11-11" },
    { "kind": "coarse_buckets", "count": 0, "last_day": null }
  ],
  "days": [
    { "day": "2026-11-11", "kind": "secret_in_path", "count": 1 }
  ]
}

• totals: always these five kinds, in this order. count is the sum over the days, last_day the latest day with a count.
• days: every non-zero day and kind, ordered by day, then kind.
• Kinds (phase 9 writer): span_derived (points derived from spans were stored), lossy_histogram (a histogram was re-bucketed inexactly), cardinality_overflow (routes or services above the daily cap were folded into {other}, or a new stream was refused by the writer’s stream cap), secret_in_path (a Stellar secret seed appeared in a request path and was redacted), coarse_buckets (a histogram’s buckets were much wider than the canonical ones, so its percentiles are not reliable; added by the functional audit, A-12). Counts are requests, not exports (audit A-8); see docs/reference/api-storage.md. The console shows a banner per kind with a non-zero count.

7.11 Configuration keys (phase 10)

8. Phase 11: global web and API views

8.1 Range

8.2 Project filters, sort and pages

• sort is any column the route lists, order is asc or desc (default desc). Projects without a value for the sort column (a rate without a denominator, a percentile without requests) come last in both orders. Ties are broken by id in the same direction. name sorts by lower(name).
• limit and cursor as 2.3. The envelope adds from, to and total (projects matching the filters): {"from", "to", "total", "items", "next_cursor"}.
• Sort values are range sums that the hourly task changes, so no keyset over them stays stable. The cursor is a position in the sorted list, bound to the route, range, filters, sort and order it was made for; any other combination is 400 invalid_cursor. A page fetched after the task committed can repeat or skip a project whose numbers changed.
• 400 invalid_query for an unknown status, sort or order, a boolean other than true or false, a malformed date, enrolled_to before enrolled_from, or q out of range.

8.3 GET /api/operator/web/projects

{
  "tenant": { "id": "uuid", "slug": "stellar-dex", "name": "Stellar DEX", "status": "active", "created_at": "..." },
  "sites": 2, "sites_verified": 1, "sites_with_stellar_toml": 1,
  "has_web_data": true, "has_api_data": false,
  "pageviews": 1200, "visitors": 410, "visits": 530, "bounces": 260, "totaltime": 48200,
  "bounce_rate": 0.4905660377358491, "avg_visit_duration": 90.94339622641509,
  "quarantine_pageviews": 40, "quarantine_visitors": 12, "quarantine_share": 0.03225806451612903,
  "ingest_requests": 1890, "ingest_rejected": 75, "rejected_share": 0.03968253968253968,
  "top": {
    "referrer": [ { "key": "github.com", "value": 81 } ],
    "country": [ { "key": "DE", "value": 120 } ],
    "device": [ { "key": "laptop", "value": 300 } ],
    "browser": [ { "key": "chrome", "value": 250 } ]
  }
}

• pageviews, visitors, visits, bounces, totaltime (seconds): sums of the primary band over the range.
• bounce_rate = bounces / visits, avg_visit_duration = totaltime / visits (seconds), both null when visits is 0. These are Umami’s own dashboard formulas.
• quarantine_pageviews, quarantine_visitors: the quarantine band. quarantine_share = quarantine_pageviews / (pageviews + quarantine_pageviews), null when both are 0.
• ingest_requests: gateway outcomes in cp.web_ingest_hourly over the range, accepted + quarantined + rejected_*; ingest_rejected the rejected_* outcomes (including rejected_gpc, a privacy drop); rejected_share = ingest_rejected / ingest_requests, null when there were none. The replayer’s bot and dlq outcomes describe accepted events again and are not requests.
• top: per dimension the 5 keys with the largest sum of daily values over the range (ties by key); a value is Umami’s y, distinct visitors with a page view carrying that key on a day.
• Sorts: name, status, created_at, sites, sites_verified, sites_with_stellar_toml, pageviews, visitors, visits, bounces, totaltime, bounce_rate, avg_visit_duration, quarantine_pageviews, quarantine_visitors, quarantine_share, ingest_requests, ingest_rejected, rejected_share.

8.4 GET /api/operator/web/compare

{
  "from": "2026-11-01", "to": "2026-11-12",
  "days": ["2026-11-01", "...", "2026-11-12"],
  "projects": [
    {
      "tenant": { "TenantRef": "..." },
      "totals": {
        "pageviews": 0, "visitors": 0, "visits": 0, "bounces": 0, "totaltime": 0,
        "bounce_rate": null, "avg_visit_duration": null,
        "quarantine_pageviews": 0, "quarantine_visitors": 0, "quarantine_share": null
      },
      "series": [
        { "day": "2026-11-01", "aggregated": true, "pageviews": 31, "visitors": 12, "visits": 14,
          "bounces": 6, "totaltime": 1300, "bounce_rate": 0.42857142857142855,
          "avg_visit_duration": 92.85714285714286, "quarantine_pageviews": 2 }
      ],
      "breakdowns": {
        "referrer": { "items": [ { "key": "github.com", "value": 9 } ], "total_keys": 14 },
        "country": { "items": [], "total_keys": 0 },
        "device": { "items": [], "total_keys": 0 },
        "browser": { "items": [], "total_keys": 0 },
        "os": { "items": [], "total_keys": 0 }
      }
    }
  ]
}

• projects come in the order of tenants. Every series has one point per entry of days, so the charts align.
• A point is aggregated: false, with every number null, when no site of the project has a row for that day yet (the day was never read, for example before a backfill). A day the task read without traffic is aggregated: true with zeros.
• totals sum the aggregated points; derived values follow 8.3.
• breakdowns: referrer, country, device, browser and operating system of the primary band over the range, the breakdown_limit largest keys (ties by key), and total_keys, the number of distinct keys. For a project with one site, each item equals Umami’s /metrics for the same range.
• 400 invalid_query when tenants is missing, empty, longer than 5, repeats an id or holds anything but lowercase UUIDs, and for a bad breakdown_limit. 404 not_found when an id is unknown or internal.

8.5 GET /api/operator/api/projects

{
  "tenant": { "TenantRef": "..." },
  "sites": 1, "sites_verified": 1, "sites_with_stellar_toml": 0,
  "has_web_data": false, "has_api_data": true,
  "requests": 182000, "requests_4xx": 2100, "requests_5xx": 90,
  "rate_4xx": 0.011538461538461539, "rate_5xx": 0.0004945054945054945,
  "p50_seconds": 0.041, "p50_over_10s": false,
  "p95_seconds": 0.62, "p95_over_10s": false,
  "p99_seconds": null, "p99_over_10s": true,
  "services": 3, "routes": 41, "span_services": 1, "source": "mixed", "lossy": false
}

• requests is the sum of count; requests_4xx and requests_5xx count status codes 400 to 499 and 500 to 599 (rows without a status code count only in requests). Rates are null without requests.
• Percentiles use api.histogram_quantile over the summed buckets of the chosen rows (plan 4.11), never averaged. A percentile in the open bucket above 10 s has *_seconds: null and *_over_10s: true; without requests both are null and false. Sorting puts over-10 s above every number.
• services: distinct services; routes: distinct (service, route) pairs with a non-empty route; span_services: services whose numbers are span derived; source: metric or span when every service uses that source, mixed when they differ, null without data; lossy: some chosen row was re-bucketed inexactly.
• Sorts: name, status, created_at, sites, requests, requests_4xx, requests_5xx, rate_4xx, rate_5xx, p50, p95, p99, services, routes. Sorting by requests in each order finds the lowest and highest usage projects.

8.6 GET /api/operator/api/compare

{
  "from": "...", "to": "...", "days": ["..."],
  "projects": [
    {
      "tenant": { "TenantRef": "..." },
      "totals": { "requests": 0, "requests_4xx": 0, "requests_5xx": 0, "rate_4xx": null, "rate_5xx": null,
                  "p50_seconds": null, "p50_over_10s": false, "p95_seconds": null, "p95_over_10s": false,
                  "p99_seconds": null, "p99_over_10s": false,
                  "services": 0, "routes": 0, "span_services": 0, "source": null, "lossy": false },
      "series": [ { "day": "...", "requests": 0, "requests_4xx": 0, "requests_5xx": 0, "rate_4xx": null,
                    "rate_5xx": null, "p50_seconds": null, "p50_over_10s": false, "...": "..." } ],
      "services": [ { "service": "payments-api", "source": "metric", "requests": 0, "...": "APINumbers",
                      "routes": 12, "lossy": false } ]
    }
  ]
}

• totals equal the project’s row of 8.5 for the same range.
• series: one point per day, the same source choice for every day (made over the whole range); a day without requests has 0 requests and null rates and percentiles.
• services: every service with its source, numbers and routes, ordered by requests (service_order), then by name, so the first and last entries are the highest and lowest usage services.
• Errors as 8.4.

8.7 Removal and removal_keep_aggregates

8.8 Configuration (phase 11)

9. Phase 12: operator API keys and machine metrics API

9.1 Two doors that never open each other

• A key is sa_op_ plus 40 random base62 characters (plan 4.10). The database holds its SHA-256 (cp.operator_api_keys.hash) and the first 10 characters (prefix) for display. The key is shown once, in the create answer.
• The machine API is mounted beside /api, not inside it (controlplane.Options.Mounts): no session middleware, CSRF check or cookie handling runs there. A console session cookie on /api/metrics/v1/ is 401.
• Every other route ignores Authorization: a key on any route outside /api/metrics/v1/ authenticates nothing (401 or 404, as without it).
• Every machine route is GET; other methods are 405.

9.2 Key management (operator)

{
  "id": "uuid", "name": "warehouse", "prefix": "sa_op_Ab3x",
  "scope": "metrics:read", "tenant_id": null, "status": "active",
  "created_by": "uuid", "created_at": "...", "expires_at": "...",
  "revoked_at": null, "last_used_at": null
}

• scope is always metrics:read; tenant_id is null for a key that reads every tenant, or the one tenant it reads (cp.operator_api_keys.scope is global or tenant).
• status: revoked when revoked_at is set, else expired when expires_at has passed, else active.
• last_used_at: the last request the key was accepted for.

9.3 Machine authentication, scope and limits

1. Missing, repeated or malformed header, or no key with that SHA-256: 401 unauthenticated with WWW-Authenticate: Bearer realm="metrics". The lookup is by hash (an index on the hash of a 238-bit random value tells a timing observer nothing), and the stored hash is compared again in constant time.
2. revoked_at set or expires_at reached: 401 unauthenticated, the message says which. Nothing is cached: a revocation or expiry applies to the very next request.
3. Token bucket per key and control plane instance: 10 requests per second, burst 20. Empty: 429 rate_limited with Retry-After in whole seconds.
4. A key scoped to one tenant with a tenant parameter naming another tenant: 403 scope_forbidden.
5. The request is counted, then served.

9.4 GET /tenants

9.5 GET /web/daily

9.6 GET /web/dimensions

9.7 GET /web/hourly

9.8 GET /api/daily

9.9 GET /api/routes/daily

9.10 GET /api/hourly

9.11 GET /api/operator/api-keys/{key_id}/usage

9.12 Configuration (phase 12)

10. Phase 13: erasure

10.1 Erasure request object

{
  "id": "uuid",
  "tenant_id": "uuid",
  "site_id": "uuid",
  "kind": "visitor_today",
  "status": "done",
  "outcome": "erased",
  "message": "Deleted 2 session(s) with 4 event(s) of this visitor on 2026-11-12 (UTC) and 0 undeliverable queued message(s).",
  "day": "2026-11-12",
  "session_id": null,
  "requested_by": "uuid",
  "requested_at": "...",
  "completed_at": "...",
  "sessions_deleted": 2,
  "events_deleted": 4,
  "dlq_messages_deleted": 0,
  "affected_from": "2026-11-12",
  "affected_to": "2026-11-12",
  "reaggregated_at": null,
  "error": null
}

• kind: visitor_today, session or site_reset.
• status: running, then done or failed (pending exists in the column check but is not used).
• outcome (only when done): erased, or nothing_found when the request was valid but matched no data. message explains the result in English; clients show it as is.
• day: visitor_today only, the UTC day whose secret produced the token (the gateways’ day, not the client’s).
• session_id: session only, the Umami session id supplied.
• affected_from, affected_to: UTC days whose aggregates the jobs task reads again; null when nothing was deleted. reaggregated_at is set when that is done.
• error (only when failed): why; counts still show what was deleted before the failure.
• No field ever holds the IP address, the user agent or the visitor token.

10.2 POST /api/tenants/{tenant_id}/erasure

{ "kind": "visitor_today", "site_id": "uuid", "ip": "203.0.113.7", "user_agent": "Mozilla/5.0 ..." }
{ "kind": "session", "site_id": "uuid", "session_id": "uuid" }
{ "kind": "site_reset", "site_id": "uuid", "confirm_domain": "example.org" }

10.3 GET /api/tenants/{tenant_id}/erasure

10.4 GET /api/tenants/{tenant_id}/erasure/{request_id}

10.5 What a run does

1. visitor_today: before the row is written, the control plane sends {site, addr, agent} on edge.visitor-token; a gateway answers {vt, day} (internal/gateway/visitor.go). The address and user agent are dropped after this request.
2. The control plane records the last sequence of WEB_EVENTS, inserts the row (running, heartbeat_at), writes erasure.request and answers 202.
3. The run renews heartbeat_at every 10 seconds and waits until WEB_EVENTS holds no message on web.events.<site> at or below the recorded sequence. If that takes longer than ERASURE_QUEUE_WAIT, it fails with nothing deleted.
4. visitor_today: reads visit.<site>.<vt> in visit_state, purges it, and deletes each Umami session it names (both bands); deletes WEB_DLQ messages on dlq.web.<site> that contain the token (secure delete). session: deletes the session from whichever band has it, then finds visit records naming it, purges them and deletes their WEB_DLQ messages. site_reset: reads Umami’s session stats for the counts, resets both Umami websites, deletes every WEB_DLQ message and visit record of the site.
5. The row becomes done (or failed) with counts and affected days, and erasure.complete (or erasure.fail) is written.
6. The jobs task erasure-followup (every minute) fails running rows whose heartbeat is older than 2 minutes, then for each finished row with affected days reads those days of agg.web_daily and agg.web_daily_dim again from Umami; for site_reset it deletes every aggregate row of the site and reads the last 4 days again. It sets reaggregated_at.

10.6 Configuration (phase 13)

11. Operator alerts

11.1 GET /api/operator/alerts

{
  "checked_at": "2026-11-13T12:00:05Z",
  "watchdog_firing": true,
  "sources": [
    { "address": "10.0.1.4:9093", "reachable": true, "watchdog_firing": true, "error": null },
    { "address": "10.0.1.5:9093", "reachable": false, "watchdog_firing": false,
      "error": "dial tcp 10.0.1.5:9093: connect: connection refused" }
  ],
  "alerts": [
    { "fingerprint": "5f1c0e7d2a9b3c41", "name": "ServiceDown", "severity": "page",
      "summary": "controlplane on core-2 does not answer scrapes", "description": null,
      "node": "core-2", "instance": "10.0.1.5:9303", "service": "controlplane",
      "starts_at": "2026-11-13T09:55:00Z", "state": "active", "inhibited_by": [],
      "runbook": "docs/runbooks/alerts.md#servicedown",
      "labels": { "alertname": "ServiceDown", "severity": "page", "node": "core-2",
                  "job": "controlplane", "instance": "10.0.1.5:9303" } }
  ]
}

• checked_at: when the control plane asked.
• sources: one entry per member of ALERTMANAGER_URLS, in that order. error is the reason a member gave no usable answer (connection, TLS, an HTTP status other than 200, a body that is not a list of alerts carrying the fields the v2 API documents as required), else null.
• alerts: every alert that has not resolved, without silenced alerts (a non-empty silencedBy) and without Watchdog. Prometheus sends every alert to every member, so one fingerprint seen by several members is one entry: the first member in configured order gives its fields, and starts_at is the earliest of them. Ordered by severity (page, then ticket, then any other severity by name, then none), then starts_at ascending (longest firing first), then name and fingerprint.
• name is the alertname label. severity, node, instance are labels of those names, service is the label job (the scrape job, which names the service in every rule), summary, description and runbook are the annotations of those names; each is null when absent. labels holds every label.
• state is Alertmanager’s: active; suppressed (inhibited by another firing alert, named in inhibited_by by fingerprint, or muted); unprocessed (received, not yet processed).
• watchdog_firing is true when at least one member that answered holds Watchdog (infra/ansible/roles/observability/files/rules/platform.yml), silenced or not; per member in sources. Watchdog always fires, so false means the pipeline from Prometheus to Alertmanager is down (Prometheus not evaluating rules, or unable to reach Alertmanager) and alerts can be missing alerts. The console shows that as “alerting pipeline down” and never as all clear.

11.2 Errors

• 503 alert_source_unreachable when no member answered. The message names each member and its reason. There is no list in this case: an empty list would read as all clear.
• 503 alert_source_not_configured when ALERTMANAGER_URLS is unset.
• As every operator route: 401 without a session, 403 operator_required for everyone else, and the route then never asks Alertmanager.

11.3 Configuration