> ## Documentation Index > Fetch the complete documentation index at: https://docs.veristar.app/llms.txt > Use this file to discover all available pages before exploring further. # Local converge: the Ansible roles against real Debian 13 systems > Run the same Ansible roles against Debian 13 containers on one machine. Status: first version (phase 15, stream b). The real servers, domain and credentials (inputs I2 to I6, I13, I14) do not exist yet, so the roles are verified against a local stand-in for the staging topology: 3 edge nodes and 1 core node. This page records the choice, how to run it, and what it does and does not prove. ## 1. The choice The machine that runs it is an unprivileged Proxmox LXC with Docker 29.3.1 and no nested virtualisation. Options considered: | Option | Verdict | | - | - | | VMs (QEMU, Vagrant, Multipass) | Not available: no `/dev/kvm` inside the LXC. | | Nested LXC or systemd-nspawn | Needs privileges the LXC does not grant. | | Ansible `community.docker.docker` connection into plain containers | Skips SSH, so SSH hardening, `AllowUsers` and the firewall's SSH rule are never exercised. Rejected as less faithful. | | **Debian 13 systemd containers, privileged inside the LXC, reached over SSH** | **Chosen.** | Each node is `infra/local/node.Dockerfile`: `debian:trixie` (pinned by digest) with systemd as PID 1, OpenSSH, sudo and Python, plus an `ops` user whose key is installed at start, which is what cloud-init does on an Azure Debian 13 VM (docs/deploy/opentofu.md section 1). Nothing the roles configure is preinstalled: Docker Engine and Compose come from Docker's apt repository at their pins, chrony, nftables and unattended-upgrades from Debian. * `--privileged` gives the node the capabilities Docker, nftables and systemd need. They stay inside the LXC's user namespace, so the node cannot change the host clock or anything outside the LXC. * `/var/lib/docker` and `/var/lib/containerd` are volumes per node, because overlayfs cannot be stacked on the container's own overlay root (the first attempt failed with `mount ... overlay ... invalid argument`). * Two Docker networks stand in for the cloud networks: `sa-lc-public` (172.31.99.0/24, default route, `gw-priority=1`) and `sa-lc-private` (10.99.0.0/16, addresses `.11+index` edge and `.21+index` core per location subnet, as in opentofu.md section 2). * A `registry:3.0.0` container (pinned by digest) on the public network plays the repository host's registry. `infra/local/build-images.sh` builds every image from `deploy/images` and pushes it there; the nodes pull from it. * Ansible connects as `ops` over SSH with a throwaway key. The controller's address (172.31.99.1) is the only admin CIDR. ## 2. Running it ```sh theme={null} make ansible-venv # uv venv with requirements.txt, then sops and age make local-converge-up # networks, registry, 4 nodes, secrets, inventory make local-converge-images # build and push every platform image make local-converge # site.yml; run it twice, the second run changes nothing make local-converge-e2e # phase 5 and 8 paths through the converged stack make local-converge-down # removes containers, volumes, networks, images, state ``` `make local-converge-up` also issues the internal CA and node certificates (`make internal-ca ENV=local`, [ansible.md](https://docs.veristar.app/deploy/ansible.md) section 9). After a converge, `tests/localconverge/tls-audit.sh` captures the private network with tcpdump sidecars while the end-to-end suite runs, checks that no flow carries readable protocol data or any secret of the environment, and tries every private endpoint from a host without a certificate (finding O-26). On 2026-11-13 the full six-node topology exhausted this machine's memory alongside other work (load above 900); the audit ran on `LC_TOPOLOGY=minimal` with `-e core_umami_processes=1`. `LC_TOPOLOGY=minimal` before `local-converge-up` leaves out the second core node (3 edge, 1 core, 1 backup node), the smallest set the roles converge; `make rotate-local` uses it for the credential rotation run (plan phase 21). Since 2026-11-13 `site.yml` ends with the `observability` play (phase 17) on every edge and core node. It needs somewhere to send alerts: the Discord webhook (input I8) does not exist locally, so pass the address of the test receiver instead, and run that receiver on the controller, where its log shows every Alertmanager message: ```sh theme={null} go run ./tests/observability/receiver -listen 172.31.99.1:8099 & make local-converge ANSIBLE_EXTRA='-e observability_test_webhook_base=http://172.31.99.1:8099' ``` Without one of the two the play stops at its input check and names both. Everything generated stays under `resources/scratch/local-converge/` (SSH key, known hosts, age identity, logs), `infra/secrets/local/` and `infra/ansible/inventories/local/hosts.json`, all gitignored. The local secrets are random values from `infra/secrets/new-secrets.py`, encrypted with SOPS to a throwaway age identity; they are local-only and are deleted by `local-converge-down`. The inventory `up.sh` writes has exactly the shape of the OpenTofu output (opentofu.md section 1) with `sa_environment: local`, `sa_provider: local` and the base domain `sa-local.test` (the reserved `.test` TLD). Local-only settings are in `infra/ansible/inventories/local/group_vars/all.yml`, each with its reason. ## 3. What was verified (2026-11-12) * `ansible-lint` (production profile) and `yamllint -s` pass. `deploy/caddy/assemble_test.go` validates the assembled node configs with `SA_TLS` cloudflare, cloudflare-delegated and internal on the custom Caddy build and refuses an unset or unknown value. * `site.yml` converged the four nodes from fresh containers, and the next run reported `changed=0` on every node (section 5 has the recap). * Every service ran on every node: on each edge node Caddy, gateway, collector, NATS and etcd; on the core node postgres-ha (Patroni leader, PostgreSQL 18.6), HAProxy, Umami, control plane, replayer, metricswriter, jobs, Caddy with the console, and the oauth fixture. * NATS: three members, one meta leader, the 10 streams and 6 consumers of `platformctl nats apply` at R3. etcd: three members at 3.6.16, one leader. * Port 80 on every node answered 308 (healthy) for any Host header. Stopping the collector on edge 1 turned edge 1 to 503 and back to 308 once it was restarted. Stopping NATS on edges 2 and 3 turned edge 1 to 503 (its NATS reported `stream ... is not current`). Stopping Umami turned the core node to 503. * From an address on the public network that is not an admin CIDR: 80 and 443 open on every node; 22, 4222, 6222, 8222, 2379, 9090, 5432, 8008, 3000, 5000 and 8085 closed. * Clock check: every node within 5 ms of NTP (`chronyc tracking`). * `deploy.yml` with a new image tag recreated the services one node at a time (edge 1, 2, 3, then core 1), each gated on its port-80 answer before the next. * `tests/localconverge` (build tag `localconverge`): sign-in through the core Caddy and the fixture as an operator, tenant and site created through the console API, one pageview through each edge node's Caddy and gateway, all three stored by Umami (`website_event` paths `/converge/sa-lc-edge-1..3`), an OTLP token created through the API and one delta histogram of 5 requests sent through each edge node's collector, 15 requests from 3 services in `api.metrics_minute`. ## 4. Limits: what this does not prove * **Not real servers, network or DNS.** No network security group, no Azure NAT of the public address (the local nodes hold theirs on an interface), no latency between zones, no Traffic Manager health checks. Hostnames are mapped by the test client. * **No public TLS.** Caddy uses its local CA (`sa_tls: internal`); DNS-01 through `caddy-dns/cloudflare` is configured (`sa_tls: cloudflare`), compiles and validates, but has not issued a certificate yet: that happens at the first staging converge. * **Clock.** The nodes share this machine's kernel clock and may not set it, so chronyd runs with `-x` (measure, do not steer). The offset check runs for real; clock steering does not. * **No GitHub.** Sign-in goes to the test-only oauth fixture (`tests/fixtures/githuboauth`), through a control plane image built with `TAGS=oauthfixture` that exists only in the local registry. The fixture listens on port 8099 of the core node, which the local inventory opens; no other environment may set `core_oauth_fixture` (the core role refuses). Real sign-in is blocked on I2, operator ids on I3 (the local operator is fixture user 1001). * **PostgreSQL 18.7**, the same image as every other environment (plan section 5). * **Two core nodes since phase 16** (section 6); phase 15 ran one. * **Scripted events, not browsers.** The web path check uses `tests/gateway/gwclient`, which solves the proof in Go. The browser matrices stay with the phase 3 to 5 suites, which start their own stacks and were not re-pointed at this one. * **Mutable local tag.** The local images are tagged `lc`; a rebuild under the same tag is not pulled again (`pull: missing`). Real releases use immutable tags. ## 5. Converge record Fresh nodes (images already in the local registry), 2026-11-13. First converge 177 s, second 47 s, then `make local-converge-e2e`: ``` run 1: sa-lc-core-1 ok=91 changed=40 failed=0 | sa-lc-edge-1 ok=74 changed=32 | edge-2, edge-3 ok=67 changed=32 run 2: sa-lc-core-1 ok=82 changed=0 failed=0 | sa-lc-edge-1 ok=69 changed=0 | edge-2, edge-3 ok=62 changed=0 --- PASS: TestConvergedStack (5.31s) --- PASS: TestConvergedStack/web_path_through_every_edge_node_into_umami (3.15s) --- PASS: TestConvergedStack/otlp_metrics_through_every_edge_node_into_the_platform_database (2.11s) ``` An earlier fresh run failed once on edge 3 when quay.io answered one pull of the pinned etcd digest with `not found` (the same digest pulled on the other nodes). Image pulls are now a separate task retried five times; every other compose failure still fails the node. ## 6. Phase 16 topology: locations, latency, backups Phase 16 (high availability and drills) extends the topology to the shape of plan section 4.1 plus a backup node: | Node | Location | Private | Runs | | - | - | - | - | | `sa-lc-edge-1` | fsn1 | 10.99.1.12 | caddy, gateway, collector, nats, etcd | | `sa-lc-edge-2` | nbg1 | 10.99.2.13 | same | | `sa-lc-edge-3` | hel1 | 10.99.3.14 | same | | `sa-lc-core-1` | fsn1 | 10.99.1.22 | patroni member, haproxy, umami, control plane, replayer, metricswriter, jobs, pgbackrest server, caddy | | `sa-lc-core-2` | nbg1 | 10.99.2.23 | same | | `sa-lc-backup-1` | hel1 | 10.99.3.31 | pgbackrest repository (posix) | | `sa-lc-restore-1` | fsn1 | 10.99.1.41 | created only by the restore drill, removed after it | * **Locations are separate Docker networks** (`sa-lc-fsn1`, `sa-lc-nbg1`, `sa-lc-hel1`). A router container (`sa-lc-router`, `infra/local/lib.sh`) joins them; each node gets the route `10.99.0.0/16 via 10.99..254` at every start (`node-init.sh`), as a cloud private network would hand it out. Docker drops packets routed between its bridges unless the source bridge is trusted, so each location network trusts the other two (`trusted_host_interfaces`). * **Latency.** `tc netem` on the router delays what it sends into each location: 1.5 ms toward fsn1 and nbg1, 11.5 ms toward hel1, 0.3 ms jitter. Round trips are therefore about 3 ms fsn1 to nbg1 and 23 ms to hel1 (measured with TCP connects: 4.8 ms and 24.3 ms including the handshake). The local network names keep the labels of the first hosting plan; they stand for three separate zones. These numbers are assumptions, not measurements of Azure's network between availability zones, which the first staging drill measures. * **Partition.** `lc_partition ` takes the router's interface into that location down, cutting it off the private network in both directions while its public addresses stay reachable; `lc_heal` restores it. * **Backups.** No Azure storage exists locally, so the pgbackrest repository is a posix repository on `sa-lc-backup-1` in hel1, a location with no database node, reached over pgbackrest's TLS protocol with the node certificates of the internal CA (`make internal-ca ENV=local`, which `make local-converge-up` runs). This is a stand-in, not an equivalent: object storage is not a server pgbackrest talks to, has its own durability and latency. Staging and production use `backup_repo_mode: azure` (the default), which this machine cannot run: it needs the Azure instance metadata service for the managed identity. * **Power off** in the drills is `docker kill` of a node container: no clean shutdown of anything inside. Power on is `docker start`, which boots systemd and Docker again; compose restarts the services (`restart: unless-stopped`). ## 7. Object storage stand-in, probe node and `make ops-local` Two switches of `infra/local/env.sh`, read by `up.sh` and `down.sh`: * `LC_BACKUP=s3` replaces `sa-lc-backup-1` with `sa-lc-s3` (hel1, `10.99.3.35`), SeaweedFS 4.52 serving S3 over TLS with a private CA, host-style addressing under `s3.sa-local.test`. The inventory then carries `backup_repo_mode: s3` and an `sa_object_storage` with the endpoint, region and bucket plus two local-only keys, `storage_host` and `ca_file`. The roles accept the `s3` mode in the local converge only. It exercises the stanza, archiving and backup flow over an object store protocol, not Azure itself ([backups.md](https://docs.veristar.app/deploy/backups.md) section 5). * `LC_PROBE=1` adds `sa-lc-probe-1` on the public network only (`172.31.99.51`), as the a probe host outside the platform's network would. `site.yml`'s probe play converges it. `make ops-local` (`tests/localconverge/ops-local.sh`) runs the operations checks of `docs/audit/functional-audit-ops.md` on `LC_TOPOLOGY=minimal LC_BACKUP=s3 LC_PROBE=1`: backup bucket, stanza and WAL archiving in `s3` mode, a full backup, delete refusal and `--repo-target-time`, idempotency, Patroni failsafe with etcd stopped, the etcd rejoin of a rebuilt edge node, the probe and the watchdog push, then tears down. It refuses to start at a load average of 20 or more and waits before each step until the load is below 20 (ten minutes at most). `LC_SRC` builds the images from another checkout (for example `git archive HEAD` unpacked elsewhere) when the working tree is mid-change. Run record, 2026-11-13: steps `up`, `images`, `converge` (247 s, `failed=0` on every node), `bucket` and `archiving` passed. The run stopped during `backup` when the shared host's load average rose above 600, and was torn down. Three later attempts lost their node containers within seconds to `docker rm -f` from outside the run (Docker events show `kill` signal 9 and `destroy` of every `sa-lc` container, newest first, which no script in this repository does), so the remaining steps have not completed yet. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.