Skip to main content
RFP section 3.2 asks for “a concrete strategy for detecting/mitigating bot traffic and artificial metric inflation (at minimum: distinguish real browser JS execution from pure server-side posts, and treat multiple distinct fingerprints from the same IP with skepticism)” that builds on the base project’s existing bot detection. This document is that strategy, as built. Parameter names and values are the real ones from the code (internal/config/gateway.go, internal/gateway/) and the deployed defaults (infra/ansible/roles/edge/defaults/main.yml, deploy/compose/dev.yml), recorded in docs/reference/gateway-limits.md. The short version: the layers below raise the cost of forged traffic, catch naive server-side posts and keep suspicious bursts out of the primary numbers. They do not prove that a browser ran the page, and they do not stop a determined actor. Section 9 says exactly where they stop.

1. Where the checks run

The gateway (internal/gateway/server.go, handleEvent) checks every POST /v1/e in this order, and stops at the first refusal: Checks 5 and 6 are one call (ProofVerifier.Verify, internal/gateway/proof.go). The replay counter moves only for proofs that passed every other part of the check, so forged proofs cannot use up a real visitor’s allowance. GET /v1/c checks the site, Origin and fetch metadata before it issues a challenge. After the gateway, the replayer delivers the event to Umami with the visitor’s original User-Agent, and Umami’s own isbot check runs (section 2).

2. Umami’s isbot on the original user agent

The RFP asks to build on the base project’s bot detection. Umami 3.5.0 runs isbot on the User-Agent of POST /api/send; a bot answers 200 {"beep":"boop"} and stores nothing. The replayer forwards the visitor’s original User-Agent header unchanged (plan 4.4 step 12), and Umami is deployed without DISABLE_BOT_CHECK (infra/ansible/roles/core/templates/compose.yml.j2, deploy/compose/dev.yml). A bot answer is counted as bot on EDGE_STATS and in replayer_outcomes_total{outcome="bot"}, so tenants and operators see how many events were dropped this way. Proved by: TestBotUserAgentStoresNothing (tests/umami-compat/send_test.go: Googlebot and bingbot store no session and no pageview) and TestWebPath subtest bots (tests/webpath/webpath_test.go: a crawler user agent through the whole chain, events answered 202 by the gateway, counted twice as bot, nothing in Umami). Limit: isbot matches declared bots. A headless browser or script that sends an ordinary browser user agent is not caught by it.

3. Server-issued challenge and proof

Full detail and measurements: docs/reference/browser-proof.md and docs/reference/wire-v1.md.
  • Once per page load the tracker calls GET /v1/c?w=<site id>. The gateway answers {w, iat, salt, bits, tag}: iat is server time, salt 16 random bytes, bits the current PROOF_BITS, and tag = HMAC-SHA256(proof_key, w|iat|salt|bits) with a daily key held in the memory KV bucket edge_secrets.
  • The tracker finds a nonce such that SHA-256(w|iat|salt|nonce) has at least bits leading zero bits (WebCrypto, batches of 64) and attaches the proof to every event of that page load. The challenge stays in page memory: no cookie, no browser storage.
  • The gateway refuses a proof older than ChallengeTTL (6 hours of server time), more than MaxClockSkew (30 seconds) in the future, with bits above MaxProofBits (24), with a wrong tag (which also covers a proof issued for another site, since the tag covers w), or with too little work. The previous day’s key is still accepted inside the 6 hour window.
  • PROOF_BITS is 12. It is the highest tested value whose median solve time stays under 150 ms at a 6x CPU slowdown in Chromium, Firefox and WebKit. Because difficulty and time come from the server, difficulty can be raised without a tracker release.
What this distinguishes: a request without a valid proof did not go through the challenge flow, so a plain HTTP client posting a JSON body is refused. That is the RFP’s minimum, “distinguish real browser JS execution from pure server-side posts”, met for naive posts. Section 9 says why it is not proof of browser execution. Proved by: TestChallengeRoundTrip, TestKeyRotationOverlap, TestKeyLost, TestKeyTTL (internal/gateway/challenge_test.go); TestTrackerVectorBytes, TestTrackerVectorsVerify, TestLeadingZeroBits (internal/gateway/proof_test.go, shared vectors with the TypeScript tracker); TestEventChecks subtests naive post without proof is 403 and forged tag is 403 (tests/gateway/checks_test.go); TestClocks (tests/gateway/clock_test.go).

4. Origin and fetch-metadata checks

  • Origin. The Origin header must equal one of the site’s registered origins (1 to 20 bare origins under the site’s domain, https only except localhost and 127.0.0.1 for testing). Proved by TestCheckOrigin, TestValidOriginPattern (internal/gateway/origin_test.go) and TestEventChecks subtest wrong origin is 403.
  • Fetch metadata. The tracker’s fetch (keepalive, credentials: 'omit', text/plain, no custom headers) produces Sec-Fetch-Site: cross-site (or same-site when the tenant shares the base domain), Sec-Fetch-Mode: cors and Sec-Fetch-Dest: empty in Chromium 156, Firefox 157 and WebKit 27.2 (tests/e2e/specs/headers.spec.ts, recorded in browser-proof.md). The SignalsChecker (internal/gateway/signals.go) refuses any of these headers that is present with another value or repeated. It refuses missing headers only when the user agent claims an engine known to send all three (Chromium 80 or later, Firefox 90 or later, Safari 16.4 or later) and GATEWAY_SECURE_TRANSPORT is true. Android WebViews and non-Safari iOS browsers (; wv), CriOS/, FxiOS/, EdgiOS/, OPiOS/) and EdgeHTML were not measured, so they are never refused for missing headers. Proved by TestSignals and TestSendsFetchMetadata (internal/gateway/signals_test.go).
These checks stop requests that come from the wrong page and naive clients that send nothing. They are headers, and any non-browser client can set them (section 9).

5. Replay limit

A captured proof (one page load) may carry at most PROOF_REPLAY_LIMIT events on one node. The default is 50. The count is kept per proof tag in a bounded least recently used table of 200,000 entries per node (replayEntries, internal/gateway/node.go).
  • The limit is per node and in memory. With three edge nodes, one proof can be used up to 3 x PROOF_REPLAY_LIMIT (150) times in total.
  • A proof pushed out of the table by 200,000 newer ones starts counting again (internal/gateway/proof.go, NewProofVerifier).
  • A gateway restart forgets the table.
Proved by: TestReplayLimit (internal/gateway/proof_test.go) and TestEventChecks subtest replay past the limit is 403.

6. Same-IP skepticism and quarantine bands

This is the RFP’s “treat multiple distinct fingerprints from the same IP with skepticism”.
  • Fingerprint. The visitor token is vt = hex(HMAC-SHA256(daily_secret, site_id || 0x00 || ip || 0x00 || user_agent)). Distinct user agents from one address, or distinct addresses in one prefix, give distinct tokens.
  • Key. Per site and per keyed hash of the truncated address (IPv4 /24, IPv6 /48) under a second daily secret. The network itself is never sent or stored.
  • Window and threshold. IPClusterWindow is one hour. When the set for a key reaches IPCLUSTER_THRESHOLD distinct tokens (default 20), the event that brings it to 20 and every later event for that key go to the quarantine band, until one window has passed since the threshold was last reached (internal/gateway/ipcluster.go).
  • Across nodes. Each gateway publishes every token that is new to it on the NATS core subject edge.ipcluster (at most IPCLUSTER_THRESHOLD messages per key and window) and merges what peers publish, so all three nodes count the same set. At most 200,000 keys per node (clusterKeys), least recently used evicted. Nothing is persisted.
  • Bands. Every site has two Umami websites: the primary (id equals the site id) and a quarantine website. Quarantined events go to the quarantine website. They are never deleted and never mixed into primary numbers. Tenants open the quarantine view through GET /api/sso/umami?site=<id>&view=quarantine, and the per-site counters include quarantined.
Proved by: TestEdgeSecretsCluster subtests quarantine starts at exactly the threshold across three gateways, a node publishes at most threshold tokens per key and window and the key bound evicts the least recently used set (internal/gateway/visitor_test.go, implemented in internal/gateway/ipcluster_test.go); TestWebPath subtest quarantine (tests/webpath/webpath_test.go, real browsers through the whole chain).

False positives on shared addresses

Many real people share one address or one /24:
  • Carrier-grade NAT. Mobile carriers and some home ISPs put thousands of subscribers behind a few addresses.
  • VPN exits and privacy relays.
  • Offices and campuses behind one NAT.
When more than 20 distinct visitor tokens reach one site from one such prefix in an hour, real visitors are quarantined too. This is expected and accepted: quarantine is a separate band shown next to the primary data, never deletion, so the tenant and operators can still see and judge those visits. A popular site with much mobile traffic should expect a non-zero quarantine band. Change IPCLUSTER_THRESHOLD only with a recorded reason in docs/reference/gateway-limits.md.

One visitor’s volume

Same-IP skepticism counts distinct visitor tokens, so one scripted client with one address and one user agent stays one token and is never caught by it. A second count closes that gap: each gateway counts queued events per site and visitor token in a window of one hour from the visitor’s first event. Past VISITOR_EVENTS_PER_HOUR (default 600, ten a minute on average) every further event of that visitor in the window goes to the quarantine band and is counted as quarantined (internal/gateway/volume.go). The count is per node, in a bounded table of 200,000 visitors, and nothing is persisted; a visitor spread over three nodes can send three times the limit into the primary band. Proved by: TestOneVisitorsVolumeIsQuarantined (internal/gateway/node_test.go).

7. Rate limits

Token buckets in each gateway, per node (internal/gateway/ratelimit.go): Over either: 429. A refusal by the prefix bucket does not take a site token. Each table holds at most 200,000 buckets. Limits are per node, so a site can send three times the site rate across three edge nodes. GET /v1/c has its own buckets at the same rates, checked after the origin and fetch metadata checks, so fetching challenges never uses up the event budget and refused challenge requests take no token (docs/reference/gateway-limits.md, change of 2026-11-13; TestChallengeRateLimits in tests/gateway). Proved by: TestRateLimitPrefixBurstAndRefill, TestRateLimitSite, TestRateLimitPrefixRefusalKeepsSiteTokens, TestRateLimitClockBackwards, TestRateLimitBoundedKeys, TestRateLimitConfig, TestRateLimitConcurrent (internal/gateway/ratelimit_test.go). Acceptance: no event of the phase 3 and phase 4 browser suites was refused, rate limited or quarantined (TestBrowserMatrix, tests/gateway).

8. Visibility

Every outcome is counted per site on EDGE_STATS and stored hourly (cp.web_ingest_hourly): accepted, quarantined, bot, dlq and rejected_<reason> (for example rejected_origin, rejected_challenge_tag, rejected_proof_work, rejected_replay, rejected_signals, rejected_rate_limited; full list in internal/gateway/reasons.go). Tenants see them under GET /api/sites/{site_id}/ingest; operators see them in the tenant detail. A sudden rise in quarantined or rejected_replay is the signal that someone is forging traffic. A Sec-GPC drop is counted as rejected_gpc; it is a privacy choice, not a sign of forgery.

9. Limits, stated plainly

  • The proof does not prove browser execution. A scripted non-browser client can fetch a challenge, solve 12 bits in milliseconds, and forge Origin, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest and User-Agent. The gateway then accepts its events. This is tested on purpose: TestEventChecks subtest scripted client with forged headers is accepted (tests/gateway/checks_test.go) posts from a Go client and expects 202. The proof raises the cost of forged traffic and catches naive server posts. It cannot stop a determined actor.
  • Real browsers on residential proxies pass every layer. A headless or real browser with an ordinary user agent runs the real tracker, so the proof, Origin and fetch metadata are genuine; isbot does not match; and spreading the traffic over many residential addresses in different prefixes keeps each prefix under IPCLUSTER_THRESHOLD and the prefix rate limit.
  • Replay is per node and in memory: up to 3 x PROOF_REPLAY_LIMIT events per proof across three nodes, more if the table evicts the entry or a gateway restarts. Only queued events count; an event refused by a later check gives its use back.
  • Volume per visitor is bounded, not stopped. One address and one user agent can send VISITOR_EVENTS_PER_HOUR (600) events an hour per node into the primary band, three times that across three nodes, before the rest go to quarantine (section 6). Page views, custom events and session duration can therefore still be raised by one client within that bound, while visitors stay at 1. A client that rotates its user agent becomes several visitors and then meets same-IP skepticism instead.
  • Same-IP skepticism has false positives on shared addresses (section 6) and false negatives for traffic spread over many prefixes.
  • API metrics are self-reported. OTLP data comes from the tenant’s own servers; the collector can rate limit and cap cardinality but cannot verify that requests happened (threat-model.md section 3.3).
  • No first-party proxy. A tenant must not proxy events through its own server: every event would then be a server-side post from one address, which the proof and same-IP checks treat as suspicious (plan 4.4 step 17).
Measures considered and left out by the project owner: on-chain corroboration through Hubble and traffic plausibility flags (plan section 2). The result meets the RFP’s minimum by raising cost and making manipulation visible. It is not a guarantee that every counted visit was a person, and operators comparing projects for funding should read the numbers with the quarantine band and the outcome counters beside them.