Drill record, 2026-11-13 (phase 16)

Where: the local converge (docs/deploy/local-converge.md section 6): 3 edge, 2 core and 1 backup Debian 13 systemd node on one Proxmox LXC (16 GB, memory.high 12 GB), three location networks joined by a router with tc netem (about 3 ms fsn1 to nbg1, 23 ms to hel1, assumed values). Not staging: real servers, Hetzner Object Storage and bunny.net DNS do not exist (inputs I4, I5, I6). Result: phase 16 is not done. Only the first load run produced numbers that can be set against the pass bounds. Every other drill is either blocked on an input or was not run to a valid result, because the host machine stopped being usable as a test bed (details below).

Tooling

tests/drills/drillctl: setup (24 tenants, 40 sites, 24 OTLP tokens through the console API), load with a ledger of every acknowledged event, verify (every 202 looked up by path in Umami, OTLP sums in api.metrics_minute), lag, probe (dashboard gap on both core nodes), commits (acknowledged-commit writer), snapshot and checksum (table checksums at time T under write locks), clock (NATS /varz time and HTTP Date).
Load client: one Go process on the host. Web events solve the real proof; a connection failure moves to the next edge address (what a browser does with several A records); the tracker never resends an answered event. OTLP exports retry connection errors and 429, 502, 503, 504 up to 6 times. A DNS stand-in checks port 80 every 30 s and answers the healthy addresses; it has no resolver cache, no TTL and no multi-region checks, so its removal times are not bunny.net’s.
Drill scripts: tests/drills/{edge-loss,nats-member,db-leader,core-outage,partition,nats-restart,pitr,cert-renewal,clock-skew,rolling-deploy}.sh, load profile tests/load/run.sh with tests/load/profiles.env (web 50/s and API 116/s expected peak, factor 10).

Topology checks (pass)

site.yml from fresh nodes, then a second run: changed=0 on all six nodes.
make local-converge-e2e: TestConvergedStack passes.
Patroni: core 2 leader, core 1 Sync Standby, streaming, lag 0; live DCS config shows synchronous_mode: true, synchronous_mode_strict: false, ttl 30, loop_wait 10.
pgBackRest: stanza-create and check succeed against the posix repository on the backup node over TLS (WAL archived). The schedules exist as timers (full Sunday 01:00, differential Monday to Saturday 01:00, verify plus check daily 05:30), the retain-web timer on both core nodes (03:17 and 03:47 UTC).

Drills

Drill	Pass bound (plan)	Measured	Verdict
Load at 10x (run A, 02:18 to 02:23 UTC)	gateway p99 < 250 ms, collector p99 < 500 ms, 0 acknowledged lost, lag 0 within 600 s	web 149,995 sent, 149,995 acknowledged, p99 126 ms; OTLP 347,988 acknowledged, p99 81 ms; acknowledged lost 0 and 0; 11 duplicate rows; backlog empty 1,101 s after the run	fail (drain)
Load at 10x (run B, 02:46)	same	host load average 624 to 880 during the run: TLS handshake timeouts, 18,289 web and 48,707 OTLP requests the generator itself could not send, p99 177 s; acknowledged lost still 0	invalid (host)
Edge node loss	0 acknowledged lost, DNS removal within 120 s, failures at most 1% and only within 120 s	run at expected peak (factor 1): the host froze for about 104 s right at the power-off (NATS logged `Readloop processing time: 1m44s` on every connection, all three edges failed the stand-in check at once). Finding: after power-on, edge 2’s NATS container stayed `Exited (255)` and Docker never retried it	invalid (host); design fix made, not re-run
NATS member loss	publishes continue on 2 of 3, 0 lost, rejoin catches up	started while edge 2’s NATS was still down from the previous drill, so it was stopped before injecting anything	not run
Database leader loss	promotion without action, dashboard gap at most 60 s, 0 committed lost, 0 edge errors	not run (host)	not run
Whole core tier down 2 h	0 ingest errors, backlog drains, exact counts	not run (host). Plan for it in `core-outage.sh`: 12 min at 10x queues the same 360,000 web events and 835,200 exports as 2 h at expected peak	not run
Location partition	NATS and etcd quorum on the other two, 0 lost	not run (host)	not run
Full NATS restart	0 lost, visitors counted again and visits split as documented	not run (host)	not run
Point-in-time restore	checksums at T equal, recovery time and loss window recorded	not run (host); `restore.yml` and `pitr.sh` are written, not executed	not run
Certificate renewal	forced renewal succeeds on every node	not run (host). Real ACME is impossible locally (inputs I4, I6); the script only proves Caddy’s local CA re-issues	not run
Rolling deploy under load	0 failed ingest requests	not run (host)	not run
Clock skew detection	detector fires; clocks within 15 s	not run (host). Nodes share one kernel clock, so a skew cannot be injected here at all	not run

Blocked on inputs regardless of the host: bunny.net health-check removal timing (I6), the S3 repository on Hetzner Object Storage including repo1-s3-uri-style=host (I5), real cross-location latency (I5), ACME issuance (I4, I6). The local substitutes (port-80 stand-in, posix repository node, netem delays, Caddy’s local CA) are not equivalent.

What the load runs showed

Consumers were the bottleneck, not the edge. Run A’s edge answered every request within bounds. Draining is what failed. Measured drain rates while the backlog was being worked off:
- first configuration: web about 77/s, OTLP about 150/s. The replayer’s 16 lanes each spend several confirmed NATS round trips per event (duplicate guard, visit state, ack). The metricswriter confirmed its 256 acks one after another, each a round trip to a consumer leader that can sit in another location (150 a second means about 6.7 ms each).
- Changes: REPLAYER_WORKERS 16 to 128 and REPLAYER_MAX_IN_FLIGHT 512 to 900 (core role defaults), metricswriter acks sent 64 at a time after the batch commits (internal/metricswriter/writer.go).
- after the change: OTLP about 1,417/s (above the 1,160/s load); web about 280/s, with Umami’s single Node process at about 65% of a core. At 10x the web load (500/s) outruns one Umami process. The backlog grows by about 220 events/s while the load runs, and drains at about 280/s once it stops. A 5-minute run then needs about 4 minutes to drain, inside the 600 s bound, but this was never measured cleanly, and longer peaks do not drain within the bound. The real fix is more Umami processes behind the replayer (Umami is stateless apart from the database). That is a design change for the next run.
The 11 duplicate rows in run A come from restarting the replayer during the drain to apply the new settings. That is the documented at-least-once window (plan 4.4 step 16).

Web drain with several Umami processes (follow-up, 2026-11-13 05:09 to 05:14 UTC)

Change: each core node runs core_umami_processes Umami processes (default 4) on the one database with one APP_SECRET (roles/core/templates/compose.yml.j2: umami plus umami-2 to umami-N). UMAMI_URL of the replayer lists all of them, and the replayer sends each visitor to Umami[visitorHash % N] with the same fnv-1a hash that picks the visitor’s lane, so a visitor’s events go to one process, one at a time, in stream order. Why the process does not matter: docs/reference/umami-contract.md item 14 (source and TestUmamiProcessesAgreeOnSessionAndVisit). Measurement: TestDrainRateByUmamiProcesses (internal/replayer, REPLAYER_DRAIN_MEASURE=<events>). Not the 6-node topology: one NATS server, one postgres:18.6 container, N Umami 3.5.0 containers each capped at one core (NanoCPUs), one replayer in the test process with the core role’s 128 lanes and 900 in flight. A backlog of 12,000 pageviews (2,000 visitors, 6 each) is published first, then the replayer starts and the time until the stream is empty is taken; every run checks that Umami holds exactly 12,000 views. All processes are warmed with 600 events each first. The host was shared (load average 3 to 5).

Umami processes	run 1 (cold)	run 2 (warmed)	run 3 (warmed, 6,000 events)	CPU per Umami process (cores, run 2)
1	454/s	502/s	442/s	1.00
2	665/s	755/s	681/s	0.85, 0.99
3	842/s	927/s		0.77, 0.74, 0.99
4		1,162/s		0.75, 0.71, 0.78, 0.99

Sends split evenly (6,000 events over 2 processes: 3,018 and 2,982). One process per run always sat at the one-core cap while the others stayed below it; the cause was not found. Because the drain ends only when the slowest process’s visitors are done, that process sets the rate. Reading: one process here (about 450 to 500/s at one full core) is faster than in run A (280/s at 65% of a core), where every event paid cross-location NATS round trips on a saturated host. The local numbers are therefore an upper bound for one process. Scaling is 1.5x for 2, 1.85x for 3 and 2.3x for 4. Applied to run A’s 280/s that is about 520/s for 3 and 650/s for 4, so the default is 4: the smallest count clearly above the 500/s of the 10x profile. Each idle process holds about 115 MB. The 10x drill has not been rerun on the topology, so the 600 s drain bound is still not shown to pass.

Restart after power-off (follow-up, 2026-11-13)

tests/drills/start-unit.sh: one Debian 13 systemd node from infra/local (no other node), converged with the base role and the edge role’s start_unit.yml (shared word for word with the core role but for the project directory), running a test-only compose project in /opt/veristar/edge with the pinned nats and etcd images and the edge project’s restart policy. A power-off is docker kill of the node container (no shutdown), followed by docker start.

Step	What	Measured (05:39 to 05:47 UTC)	Verdict
1	power-off and power-on	`nats`, `etcd`, `slow`, `client` all running again; unit `Result=success`, started in the same second docker became active	pass
2	`nats` and `client` set to restart policy `no` (docker leaves them down at boot, as it left edge 2’s nats in the edge loss drill), power-off	the unit started `nats` at once and `client` 100 s later, once `slow` (healthy only 100 s after start, like umami’s 120 s start period on core) was healthy; all running, `Result=success`	pass
3	control: unit disabled, same as step 2	`nats` and `client` stay `exited`, so step 2 measured the unit, not docker	pass

Finding, fixed: the unit as phase 16 wrote it had systemd’s 90 s default timeout for a oneshot. docker compose start waits for each dependency’s health condition before it starts a dependent, so on a core node (the replayer, controlplane and jobs wait for umami’s health, 120 s start period) systemd killed it. Run with START_UNIT_TIMEOUT=90s (the old behaviour put back by a drop-in) at 05:24 to 05:32: start operation timed out. Terminating. at 90 s, Result=timeout, client left exited. Both units now set TimeoutStartSec=15min. Local artifact, not a design finding: after a power-on of the node container, docker attached its two networks in the other order in one of three boots (eth0 became the private address). The base role’s firewall names interfaces, so ssh to the node then timed out and a converge after that boot failed; the drill disables the unit over docker exec for that reason. Cloud servers keep their interface names across reboots; the local converge’s drills that converge after a power-on can hit this.

Why the host stopped the drills

The LXC is shared with other agents’ stacks and builds. During the runs the load average reached 1,129, available memory fell to about 600 MB with swap 78% used, and the LXC’s 12 GB memory.high throttled every process in it. The 6-node topology needed about 5.7 GB. Freezes of more than 100 s hit every process, the load client included. Numbers taken in that state measure the machine, not the design, so they are not reported as drill results. Failure-mode drills at expected peak (factor 1) were attempted, and the first one (edge loss) hit such a freeze. The topology was then removed to give the memory back. Rerunning needs a host where the topology has about 8 GB to itself, or staging.

Fixes made from what ran

deploy/images/go-service.Dockerfile copies docs/compliance and docs/reference/openapi. Without them the platformctl and jobs images no longer built after phase 12.
Edge and core roles install sa-{edge,core}-start.service, which runs docker compose start after Docker comes up. A start that fails during daemon startup is not retried by the restart policy. Verified by power-off on one node on 2026-11-13 (section “restart after power-off”), which also found that it needs TimeoutStartSec.
bootstrap.yml runs on the current Patroni leader (it ran on core[0], which is a replica after a failover).
Core compose: the pgbackrest server container is created after postgres (two containers creating one new volume raced). Repository tls-server-auth uses one line per client CN (the value after = lists stanzas).

Deploy

Runbooks

Service targets