> ## Documentation Index > Fetch the complete documentation index at: https://docs.veristar.app/llms.txt > Use this file to discover all available pages before exploring further. # Drill record, 2026-11-13 (phase 16) > Results of the recovery and load drills run on the local converge. Where: the local converge (`docs/deploy/local-converge.md` section 6): 3 edge, 2 core and 1 backup Debian 13 systemd node on one Proxmox LXC (16 GB, `memory.high` 12 GB), three location networks joined by a router with `tc netem` (about 3 ms fsn1 to nbg1, 23 ms to hel1, assumed values). Not staging: real servers, Hetzner Object Storage and bunny.net DNS do not exist (inputs I4, I5, I6). **Result: phase 16 is not done. Only the first load run produced numbers that can be set against the pass bounds. Every other drill is either blocked on an input or was not run to a valid result, because the host machine stopped being usable as a test bed (details below).** ## Tooling * `tests/drills/drillctl`: setup (24 tenants, 40 sites, 24 OTLP tokens through the console API), `load` with a ledger of every acknowledged event, `verify` (every 202 looked up by path in Umami, OTLP sums in `api.metrics_minute`), `lag`, `probe` (dashboard gap on both core nodes), `commits` (acknowledged-commit writer), `snapshot` and `checksum` (table checksums at time T under write locks), `clock` (NATS `/varz` time and HTTP `Date`). * Load client: one Go process on the host. Web events solve the real proof; a connection failure moves to the next edge address (what a browser does with several A records); the tracker never resends an answered event. OTLP exports retry connection errors and 429, 502, 503, 504 up to 6 times. A DNS stand-in checks port 80 every 30 s and answers the healthy addresses; it has no resolver cache, no TTL and no multi-region checks, so its removal times are **not** bunny.net's. * Drill scripts: `tests/drills/{edge-loss,nats-member,db-leader,core-outage,partition,nats-restart,pitr,cert-renewal,clock-skew,rolling-deploy}.sh`, load profile `tests/load/run.sh` with `tests/load/profiles.env` (web 50/s and API 116/s expected peak, factor 10). ## Topology checks (pass) * `site.yml` from fresh nodes, then a second run: `changed=0` on all six nodes. * `make local-converge-e2e`: `TestConvergedStack` passes. * Patroni: core 2 leader, core 1 `Sync Standby`, `streaming`, lag 0; live DCS config shows `synchronous_mode: true`, `synchronous_mode_strict: false`, ttl 30, loop\_wait 10. * pgBackRest: `stanza-create` and `check` succeed against the posix repository on the backup node over TLS (WAL archived). The schedules exist as timers (full Sunday 01:00, differential Monday to Saturday 01:00, verify plus check daily 05:30), the retain-web timer on both core nodes (03:17 and 03:47 UTC). ## Drills | Drill | Pass bound (plan) | Measured | Verdict | | - | - | - | - | | Load at 10x (run A, 02:18 to 02:23 UTC) | gateway p99 \< 250 ms, collector p99 \< 500 ms, 0 acknowledged lost, lag 0 within 600 s | web 149,995 sent, 149,995 acknowledged, p99 126 ms; OTLP 347,988 acknowledged, p99 81 ms; acknowledged lost 0 and 0; 11 duplicate rows; backlog empty **1,101 s** after the run | **fail** (drain) | | Load at 10x (run B, 02:46) | same | host load average 624 to 880 during the run: TLS handshake timeouts, 18,289 web and 48,707 OTLP requests the generator itself could not send, p99 177 s; acknowledged lost still 0 | invalid (host) | | Edge node loss | 0 acknowledged lost, DNS removal within 120 s, failures at most 1% and only within 120 s | run at expected peak (factor 1): the host froze for about 104 s right at the power-off (NATS logged `Readloop processing time: 1m44s` on every connection, all three edges failed the stand-in check at once). Finding: after power-on, edge 2's NATS container stayed `Exited (255)` and Docker never retried it | invalid (host); design fix made, not re-run | | NATS member loss | publishes continue on 2 of 3, 0 lost, rejoin catches up | started while edge 2's NATS was still down from the previous drill, so it was stopped before injecting anything | not run | | Database leader loss | promotion without action, dashboard gap at most 60 s, 0 committed lost, 0 edge errors | not run (host) | not run | | Whole core tier down 2 h | 0 ingest errors, backlog drains, exact counts | not run (host). Plan for it in `core-outage.sh`: 12 min at 10x queues the same 360,000 web events and 835,200 exports as 2 h at expected peak | not run | | Location partition | NATS and etcd quorum on the other two, 0 lost | not run (host) | not run | | Full NATS restart | 0 lost, visitors counted again and visits split as documented | not run (host) | not run | | Point-in-time restore | checksums at T equal, recovery time and loss window recorded | not run (host); `restore.yml` and `pitr.sh` are written, not executed | not run | | Certificate renewal | forced renewal succeeds on every node | not run (host). Real ACME is impossible locally (inputs I4, I6); the script only proves Caddy's local CA re-issues | not run | | Rolling deploy under load | 0 failed ingest requests | not run (host) | not run | | Clock skew detection | detector fires; clocks within 15 s | not run (host). Nodes share one kernel clock, so a skew cannot be injected here at all | not run | Blocked on inputs regardless of the host: bunny.net health-check removal timing (I6), the S3 repository on Hetzner Object Storage including `repo1-s3-uri-style=host` (I5), real cross-location latency (I5), ACME issuance (I4, I6). The local substitutes (port-80 stand-in, posix repository node, netem delays, Caddy's local CA) are not equivalent. ## What the load runs showed 1. **Consumers were the bottleneck, not the edge.** Run A's edge answered every request within bounds. Draining is what failed. Measured drain rates while the backlog was being worked off: * first configuration: web about 77/s, OTLP about 150/s. The replayer's 16 lanes each spend several confirmed NATS round trips per event (duplicate guard, visit state, ack). The metricswriter confirmed its 256 acks one after another, each a round trip to a consumer leader that can sit in another location (150 a second means about 6.7 ms each). * Changes: `REPLAYER_WORKERS` 16 to 128 and `REPLAYER_MAX_IN_FLIGHT` 512 to 900 (core role defaults), metricswriter acks sent 64 at a time after the batch commits (`internal/metricswriter/writer.go`). * after the change: OTLP about 1,417/s (above the 1,160/s load); web about 280/s, with Umami's single Node process at about 65% of a core. **At 10x the web load (500/s) outruns one Umami process.** The backlog grows by about 220 events/s while the load runs, and drains at about 280/s once it stops. A 5-minute run then needs about 4 minutes to drain, inside the 600 s bound, but this was never measured cleanly, and longer peaks do not drain within the bound. The real fix is more Umami processes behind the replayer (Umami is stateless apart from the database). That is a design change for the next run. 2. The 11 duplicate rows in run A come from restarting the replayer during the drain to apply the new settings. That is the documented at-least-once window (plan 4.4 step 16). ## Web drain with several Umami processes (follow-up, 2026-11-13 05:09 to 05:14 UTC) Change: each core node runs `core_umami_processes` Umami processes (default 4) on the one database with one `APP_SECRET` (`roles/core/templates/compose.yml.j2`: `umami` plus `umami-2` to `umami-N`). `UMAMI_URL` of the replayer lists all of them, and the replayer sends each visitor to `Umami[visitorHash % N]` with the same fnv-1a hash that picks the visitor's lane, so a visitor's events go to one process, one at a time, in stream order. Why the process does not matter: `docs/reference/umami-contract.md` item 14 (source and `TestUmamiProcessesAgreeOnSessionAndVisit`). Measurement: `TestDrainRateByUmamiProcesses` (`internal/replayer`, `REPLAYER_DRAIN_MEASURE=`). Not the 6-node topology: one NATS server, one postgres:18.6 container, N Umami 3.5.0 containers each capped at **one core** (`NanoCPUs`), one replayer in the test process with the core role's 128 lanes and 900 in flight. A backlog of 12,000 pageviews (2,000 visitors, 6 each) is published first, then the replayer starts and the time until the stream is empty is taken; every run checks that Umami holds exactly 12,000 views. All processes are warmed with 600 events each first. The host was shared (load average 3 to 5). | Umami processes | run 1 (cold) | run 2 (warmed) | run 3 (warmed, 6,000 events) | CPU per Umami process (cores, run 2) | | - | - | - | - | - | | 1 | 454/s | 502/s | 442/s | 1.00 | | 2 | 665/s | 755/s | 681/s | 0.85, 0.99 | | 3 | 842/s | 927/s | | 0.77, 0.74, 0.99 | | 4 | | 1,162/s | | 0.75, 0.71, 0.78, 0.99 | Sends split evenly (6,000 events over 2 processes: 3,018 and 2,982). One process per run always sat at the one-core cap while the others stayed below it; the cause was not found. Because the drain ends only when the slowest process's visitors are done, that process sets the rate. Reading: one process here (about 450 to 500/s at one full core) is faster than in run A (280/s at 65% of a core), where every event paid cross-location NATS round trips on a saturated host. The local numbers are therefore an upper bound for one process. Scaling is 1.5x for 2, 1.85x for 3 and 2.3x for 4. Applied to run A's 280/s that is about 520/s for 3 and 650/s for 4, so the default is **4**: the smallest count clearly above the 500/s of the 10x profile. Each idle process holds about 115 MB. The 10x drill has not been rerun on the topology, so the 600 s drain bound is still not shown to pass. ## Restart after power-off (follow-up, 2026-11-13) `tests/drills/start-unit.sh`: one Debian 13 systemd node from `infra/local` (no other node), converged with the base role and the edge role's `start_unit.yml` (shared word for word with the core role but for the project directory), running a test-only compose project in `/opt/veristar/edge` with the pinned nats and etcd images and the edge project's restart policy. A power-off is `docker kill` of the node container (no shutdown), followed by `docker start`. | Step | What | Measured (05:39 to 05:47 UTC) | Verdict | | - | - | - | - | | 1 | power-off and power-on | `nats`, `etcd`, `slow`, `client` all running again; unit `Result=success`, started in the same second docker became active | pass | | 2 | `nats` and `client` set to restart policy `no` (docker leaves them down at boot, as it left edge 2's nats in the edge loss drill), power-off | the unit started `nats` at once and `client` 100 s later, once `slow` (healthy only 100 s after start, like umami's 120 s start period on core) was healthy; all running, `Result=success` | pass | | 3 | control: unit disabled, same as step 2 | `nats` and `client` stay `exited`, so step 2 measured the unit, not docker | pass | Finding, fixed: the unit as phase 16 wrote it had systemd's 90 s default timeout for a oneshot. `docker compose start` waits for each dependency's health condition before it starts a dependent, so on a core node (the replayer, controlplane and jobs wait for umami's health, 120 s start period) systemd killed it. Run with `START_UNIT_TIMEOUT=90s` (the old behaviour put back by a drop-in) at 05:24 to 05:32: `start operation timed out. Terminating.` at 90 s, `Result=timeout`, `client` left `exited`. Both units now set `TimeoutStartSec=15min`. Local artifact, not a design finding: after a power-on of the node container, docker attached its two networks in the other order in one of three boots (`eth0` became the private address). The base role's firewall names interfaces, so ssh to the node then timed out and a converge after that boot failed; the drill disables the unit over `docker exec` for that reason. Cloud servers keep their interface names across reboots; the local converge's drills that converge after a power-on can hit this. ## Why the host stopped the drills The LXC is shared with other agents' stacks and builds. During the runs the load average reached 1,129, available memory fell to about 600 MB with swap 78% used, and the LXC's 12 GB `memory.high` throttled every process in it. The 6-node topology needed about 5.7 GB. Freezes of more than 100 s hit every process, the load client included. Numbers taken in that state measure the machine, not the design, so they are not reported as drill results. Failure-mode drills at expected peak (factor 1) were attempted, and the first one (edge loss) hit such a freeze. The topology was then removed to give the memory back. Rerunning needs a host where the topology has about 8 GB to itself, or staging. ## Fixes made from what ran * `deploy/images/go-service.Dockerfile` copies `docs/compliance` and `docs/reference/openapi`. Without them the platformctl and jobs images no longer built after phase 12. * Edge and core roles install `sa-{edge,core}-start.service`, which runs `docker compose start` after Docker comes up. A start that fails during daemon startup is not retried by the restart policy. Verified by power-off on one node on 2026-11-13 (section "restart after power-off"), which also found that it needs `TimeoutStartSec`. * `bootstrap.yml` runs on the current Patroni leader (it ran on `core[0]`, which is a replica after a failover). * Core compose: the pgbackrest server container is created after postgres (two containers creating one new volume raced). Repository `tls-server-auth` uses one line per client CN (the value after `=` lists stanzas). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.