Picture a bouncer working a rope line outside a busy nightclub who never once asks anyone for ID. He does not need to. He watches how each person approaches: pace, hesitation, whether they glance back at a car idling at the curb, whether they drift toward the door in the loose, correcting way people actually walk while scanning a crowd for a friend, or whether they walk in a mechanically straight line from the parking garage door like they were dropped in on rails. By the time someone reaches the rope, he already knows whether to wave them through, watch them a little closer once they are inside, or stop them cold. He never interrupted anyone’s night to check a card. A bot detection system built on behavioral signals is trying to be that bouncer at web scale: telling a browsing human apart from a scripted client by how a session behaves, not by demanding a credential nobody legitimate wants to produce.

Consider a mid-size e-commerce and ticketing platform serving around 800 million page views a month, spiking to 50,000 requests a second during a flash sale or a ticket drop. Independent bot-traffic studies routinely find that somewhere between 30% and 40% of that kind of traffic is not a person at all: scalper bots snapping up concert tickets before a human can click “buy”, credential-stuffing scripts testing stolen password lists against the login form, and scraper fleets hammering the product catalog for a competitor’s pricing feed. Every one of those has to be told apart from the other 60-70% of traffic that is a real customer trying to check out before the sale ends.

The naive approach is to put a CAPTCHA in front of anything that looks suspicious. That fails for two reasons that compound each other. First, CAPTCHAs cost real money: every checkbox a genuine buyer has to solve during a flash sale is a buyer who might abandon the cart, and conversion-sensitive businesses measure that cost in dollars, not inconvenience. Second, CAPTCHAs are a solved problem for anyone motivated enough to solve them: human CAPTCHA-solving farms clear a few cents per solve and modern computer-vision models handle the rest, so a challenge that annoys your real customers barely slows a well-funded scalper operation down.

A second failure mode is scoring on a single signal. Block by IP address alone and you punish an entire corporate office or mobile carrier sharing one NAT’d address while missing bots that rotate through a fresh residential proxy on every request. Rate-limit by request count alone and a patient bot that stays just under the threshold sails through, while a legitimate flash-sale crowd refreshing a product page gets throttled. Detection that actually works has to fuse many weak, individually gameable signals - how someone moves a mouse, how fast requests arrive, what an IP address has done at other sites, what a device’s hardware and software configuration looks like - into a single score, and it has to compute that score inline, on the request path, in the time it takes to render a page, without ever showing a real human a challenge they would resent. We need to solve for three things at once: collecting enough behavioral and network signal to make a confident call without adding perceptible latency, gating access on a continuous score rather than a binary allow-or-deny so genuine users never see a challenge, and building a feedback loop that keeps the model current as bot operators study our defenses and adjust their scripts to look more human, because a model trained once and left alone degrades the day a bot author notices it.

Requirements and Constraints

Functional Requirements

• Collect behavioral telemetry (mouse movement, scroll velocity, touch events, keystroke timing) client-side without adding perceptible page-load latency
• Compute a device fingerprint that fuses canvas, WebGL, font, and hardware-configuration signals into a stable per-device identifier that survives cookie clearing
• Compute request velocity signals per IP, per session, and per device fingerprint over sliding time windows
• Maintain an IP reputation store combining ASN and datacenter/proxy/Tor classification with locally observed abuse history
• Score every request or session in real time and route it to an allow, monitor, challenge, or block decision using score-based gating, with no visible challenge shown to low-risk traffic
• Provide a feedback loop that ingests confirmed bot and human labels (honeypots, challenge outcomes, chargebacks, manual review) and retrains the scoring model on a schedule
• Expose a query API so other services can fetch a session’s current score without duplicating the scoring pipeline
• Support emergency policy overrides that tighten or loosen thresholds instantly during an active attack, without a full redeploy

Non-Functional Requirements

• Decision latency: p99 under 15ms for the inline scoring call, since it sits directly on the request path
• Throughput: sustain 50,000 requests/second at peak, 18,000/second sustained average, each scored individually
• Scale: 800 million page views/month, roughly 12 million daily active sessions, around 150 million behavioral telemetry events ingested per day
• Accuracy: catch at least 95% of known automated traffic patterns while keeping the false-positive challenge rate under 0.05% of genuine sessions
• Availability: 99.99% for the decision path; the system must fail open (allow traffic) rather than fail closed if the scoring engine degrades, since blocking all traffic during an infrastructure incident is worse than briefly under-detecting bots
• Model freshness: incremental retrain daily, full retrain weekly, and an out-of-band emergency retrain within 4 hours of detecting a new evasion pattern
• Retention: 90 days of raw behavioral telemetry for retraining; aggregated reputation and score history retained longer

Constraints and Assumptions

• No CAPTCHA-first design. Visible challenges are a last resort for the highest-risk score band only, never the primary detection mechanism
• We do not design the challenge UI itself in depth (a standard JS proof-of-work or interactive check is assumed) - the focus is the detection and scoring pipeline that decides when to invoke it
• TLS everywhere and a same-origin client SDK are assumed
• We do not attempt to enumerate every individual anti-evasion heuristic in the bot-vs-detector arms race; the goal is an architecture that lets the model keep adapting, not a fixed list of today’s tricks
• A single global deployment is described in depth, with multi-region replication discussed only where reputation and feature data synchronization matters

High-Level Architecture

The system has six major components: the Signal Collection SDK, the Edge Ingestion and Velocity Service, the Reputation Store, the Scoring Engine, the Decision Gateway, and the Adversarial Feedback Loop.

The Signal Collection SDK runs in the browser or mobile app and captures pointer movement, touch events, keystroke timing, and a device fingerprint, batching them for delivery without blocking page rendering. The Edge Ingestion and Velocity Service receives those batches at the network edge, computes request-rate signals keyed on IP, fingerprint, and session, and forwards everything toward scoring. The Reputation Store is a fast, sharded key-value layer holding IP intelligence (datacenter and proxy classification, ASN, locally observed abuse history) that gets looked up on nearly every request. The Scoring Engine joins behavioral, velocity, reputation, and device features into a single feature vector and runs model inference to produce a continuous bot score. The Decision Gateway maps that score to a policy tier and executes it (serve normally, log more closely, invoke an invisible step-up check, or block), and the Adversarial Feedback Loop collects ground-truth labels from that outcome and periodically retrains the model that feeds the Scoring Engine.

A request flows like this: as a session interacts with a page, the SDK batches telemetry and flushes it every few hundred milliseconds. The edge layer increments velocity counters and looks up reputation for the request’s IP, then hands a feature vector to the Scoring Engine, which returns a score in low single-digit milliseconds. The Decision Gateway acts on that score inline, before the response is served, so a legitimate user never notices any of this happened. Separately, decision outcomes, honeypot triggers, and confirmed-fraud events stream into a label store that a scheduled retraining job consumes, producing a new model version that is shadow-evaluated against a held-out labeled set before it ever replaces the production model.

Component Deep Dives

The Signal Collection SDK and Behavioral Fingerprinting

This component’s job is to capture how a session behaves, not just what it claims to be, and to do it without the user noticing any of the collection is happening.

A bot’s mouse path looks like it was drawn with a ruler: dead-straight lines between two points at close to constant velocity. A human path wanders, overshoots the target, and corrects, and it speeds up and slows down non-linearly on the way there, a pattern HCI research calls Fitts’s law. Behavioral fingerprinting is the practice of turning that difference into a numeric feature rather than eyeballing it: sampling pointer position over time and computing statistics like path straightness and velocity variance that separate the two populations even when no single sample looks obviously wrong.

// Client-side telemetry collector - samples pointer movement and batches it
// Demonstrates: throttled event capture, canvas/WebGL fingerprint hash, beacon flush
const MAX_BUFFER = 400;
const FLUSH_INTERVAL_MS = 500;

class SignalCollector {
  constructor(endpoint, sessionId) {
    this.endpoint = endpoint;
    this.sessionId = sessionId;
    this.buffer = [];
    this.lastSample = 0;
    window.addEventListener('pointermove', (e) => this.onPointerMove(e), { passive: true });
    window.addEventListener('touchstart', (e) => this.onTouch(e), { passive: true });
    setInterval(() => this.flush(), FLUSH_INTERVAL_MS);
  }

  onPointerMove(e) {
    const now = performance.now();
    // Sample at roughly 60 events/sec max - enough resolution for curvature
    // analysis without flooding the buffer on high-refresh-rate displays
    if (now - this.lastSample < 16) return;
    this.lastSample = now;
    this.buffer.push({ t: Math.round(now), x: e.clientX, y: e.clientY, type: 'move' });
    if (this.buffer.length >= MAX_BUFFER) this.flush();
  }

  onTouch(e) {
    const touch = e.touches[0];
    if (!touch) return;
    this.buffer.push({ t: Math.round(performance.now()), x: touch.clientX, y: touch.clientY, type: 'touch' });
  }

  deviceFingerprint() {
    const canvas = document.createElement('canvas');
    const ctx = canvas.getContext('2d');
    ctx.textBaseline = 'top';
    ctx.font = '14px Arial';
    ctx.fillText('scalablesystem-fp', 2, 2);
    const canvasHash = canvas.toDataURL();
    const gl = document.createElement('canvas').getContext('webgl');
    const renderer = gl ? gl.getParameter(gl.RENDERER) : 'no-webgl';
    return {
      canvasHash,
      renderer,
      hwConcurrency: navigator.hardwareConcurrency,
      tz: Intl.DateTimeFormat().resolvedOptions().timeZone,
    };
  }

  flush() {
    if (this.buffer.length === 0) return;
    const payload = JSON.stringify({ sessionId: this.sessionId, events: this.buffer, fp: this.deviceFingerprint() });
    this.buffer = [];
    // sendBeacon survives page navigation/unload, unlike a regular fetch call
    navigator.sendBeacon(this.endpoint, payload);
  }
}

Device fingerprinting takes several individually weak entropy sources (the exact pixels a canvas renders, the reported GPU renderer string, timezone, hardware concurrency) and hashes them into a single identifier that is stable across sessions even after cookies are cleared, unlike a session cookie or a localStorage value a bot can trivially wipe. The server side of mouse movement analysis turns the raw batch into features a model can consume:

# Server-side behavioral feature extraction from a raw pointer-event batch
# Demonstrates: path straightness and velocity variance as bot vs human signals
import math

def extract_mouse_features(events: list[dict]) -> dict:
    if len(events) < 3:
        return {"straightness": None, "velocity_cv": None, "sample_count": len(events)}

    total_arc_length = 0.0
    velocities = []
    for i in range(1, len(events)):
        dx = events[i]["x"] - events[i - 1]["x"]
        dy = events[i]["y"] - events[i - 1]["y"]
        dt = max(events[i]["t"] - events[i - 1]["t"], 1)
        dist = math.hypot(dx, dy)
        total_arc_length += dist
        velocities.append(dist / dt)

    displacement = math.hypot(
        events[-1]["x"] - events[0]["x"],
        events[-1]["y"] - events[0]["y"],
    )
    # A bot moving point-to-point in a straight line has straightness near 1.0.
    # A human path wanders and corrects, pulling this well below 1.0.
    straightness = displacement / total_arc_length if total_arc_length > 0 else 1.0

    mean_v = sum(velocities) / len(velocities)
    variance = sum((v - mean_v) ** 2 for v in velocities) / len(velocities)
    # Coefficient of variation: humans show high velocity variance (accelerate,
    # decelerate near the target per Fitts's law); scripted paths stay near constant speed.
    velocity_cv = (variance ** 0.5) / mean_v if mean_v > 0 else 0.0

    return {
        "straightness": round(straightness, 4),
        "velocity_cv": round(velocity_cv, 4),
        "sample_count": len(events),
    }

Think of the canvas fingerprint like an engine’s exhaust note: two cars off the same assembly line still sound slightly different once you listen closely enough, because tiny manufacturing and configuration variance leaks through. A canvas renders through a specific combination of GPU driver, font rasterizer, and operating system, and that combination leaves a measurable fingerprint even across two otherwise-identical machines. If you simplified this component to just a session cookie and a user-agent string, every bot would trivially evade tracking by wiping cookies between requests and rotating a list of user-agent strings, which is exactly what unsophisticated scraping frameworks already do by default.

The Edge Ingestion and Request Velocity Layer

This component’s job is to answer “how fast is this identity making requests, and does that rate look human” cheaply enough to run on every single request.

Request velocity signals are the oldest bot-detection tool in the book and the easiest to get wrong. The naive version keys purely on IP address, which breaks in both directions: it over-blocks a corporate office or a mobile carrier’s NAT gateway sharing one address across thousands of real users, and it under-blocks a bot that simply rotates through a fresh residential proxy IP on every request while reusing the same device fingerprint. The fix is keying velocity on a compound identity - IP plus device fingerprint plus session - so a bot that rotates one dimension is still caught by the others staying constant.

// Edge ingestion: per-key request velocity using fixed 1-second buckets in Redis
// Demonstrates: O(1) increment, bucketed sliding window, keyed on IP+fingerprint (not IP alone)
package velocity

import (
	"context"
	"fmt"
	"time"

	"github.com/redis/go-redis/v9"
)

const bucketWindowSeconds = 10

// RecordAndScore increments the current second's bucket for a compound key
// and returns the request count over the trailing bucketWindowSeconds, so a
// burst is visible even if the bot spreads requests across rotating source
// IPs while reusing the same device fingerprint.
func RecordAndScore(ctx context.Context, rdb *redis.Client, ip, fingerprint string) (int64, error) {
	now := time.Now().Unix()
	key := fmt.Sprintf("velocity:%s:%s:%d", ip, fingerprint, now)

	pipe := rdb.TxPipeline()
	pipe.Incr(ctx, key)
	pipe.Expire(ctx, key, bucketWindowSeconds*time.Second)
	if _, err := pipe.Exec(ctx); err != nil {
		return 0, err
	}

	keys := make([]string, 0, bucketWindowSeconds)
	for i := int64(0); i < bucketWindowSeconds; i++ {
		keys = append(keys, fmt.Sprintf("velocity:%s:%s:%d", ip, fingerprint, now-i))
	}
	counts, err := rdb.MGet(ctx, keys...).Result()
	if err != nil {
		return 0, err
	}
	var total int64
	for _, c := range counts {
		if c == nil {
			continue
		}
		var n int64
		fmt.Sscanf(c.(string), "%d", &n)
		total += n
	}
	return total, nil
}

Beyond raw count, a bot’s request timing is often suspiciously regular: a script firing every 300ms plus or minus a few milliseconds of jitter has a much lower interval variance than a human clicking through pages, who naturally pauses to read, gets distracted, and comes back at irregular intervals. That interval-regularity feature is computed alongside the raw count and fed into scoring as a separate signal, because a high request rate alone is not damning (a real user might be rapidly refreshing a ticket page) while a low-variance interval pattern combined with any elevated rate is a much stronger tell.

IP Reputation and the Reputation Store

This component’s job is to answer “has this network address behaved badly before, here or elsewhere” without a synchronous call to a third party on every request.

IP reputation works like a credit score for network addresses: it aggregates a history of prior behavior into a single number that follows the address around, decaying over time because IP addresses get reassigned and yesterday’s abuser might be today’s residential customer on a dynamic-IP ISP plan. We combine a static threat-intelligence feed (known datacenter ranges, VPN and proxy exit lists, Tor exit nodes) with a locally observed abuse history that decays with a half-life, so a burst of bad behavior a month ago carries much less weight than one from an hour ago.

# Reputation score lookup with time-decay weighting
# Demonstrates: blending a static IP intelligence feed with locally observed abuse history
import time

DECAY_HALF_LIFE_SECONDS = 7 * 24 * 3600  # local abuse signal loses half its weight every 7 days

def reputation_score(static_feed: dict, local_abuse_events: list[dict]) -> float:
    score = 0.0
    if static_feed.get("is_datacenter"):
        score += 25.0
    if static_feed.get("is_known_proxy") or static_feed.get("is_vpn"):
        score += 20.0
    if static_feed.get("is_tor_exit"):
        score += 35.0

    now = time.time()
    local_score = 0.0
    for event in local_abuse_events:
        age = now - event["ts"]
        decay = 0.5 ** (age / DECAY_HALF_LIFE_SECONDS)
        local_score += event["severity"] * decay
    score += min(local_score, 40.0)  # cap so one noisy IP can't dominate the feature alone

    return min(score, 100.0)

We deliberately do not block on is_datacenter or is_known_proxy alone, since privacy-conscious real users route through commercial VPNs constantly and a growing share of legitimate mobile traffic exits through carrier-operated proxy infrastructure that looks identical to a malicious proxy at the IP level. Reputation is one input feature among several, not a standalone gate.

The Scoring Engine and Score-Based Gating

This component’s job is to turn a pile of individually weak features into one number confident enough to act on, and to act on that number without interrupting anyone who did not earn a second look.

Score-based gating means the system never makes a binary bot-or-human call directly from raw signals. It computes a continuous score, typically 0 to 100, and maps score ranges to graduated responses: silently allow, allow but log more closely, invoke an invisible step-up check, or block outright. This is the mechanism that lets the system work without a user-visible challenge for the overwhelming majority of traffic, since only the highest-risk band ever triggers anything the user could notice.

# Scoring engine: assembles the feature vector, runs model inference, applies gating
# Demonstrates: score-based gating with hysteresis to avoid flapping between tiers
import lightgbm as lgb

MODEL = lgb.Booster(model_file="bot_score_v42.txt")

FEATURE_ORDER = [
    "mouse_straightness", "mouse_velocity_cv", "keystroke_dwell_cv",
    "request_velocity_10s", "request_interval_cv", "ip_reputation",
    "fingerprint_age_days", "fingerprint_session_fanout", "is_headless_hint",
]

def build_feature_vector(behavioral, velocity, reputation, device) -> list[float]:
    return [
        behavioral.get("straightness", 0.5),
        behavioral.get("velocity_cv", 0.0),
        behavioral.get("keystroke_dwell_cv", 0.0),
        velocity["count_10s"],
        velocity["interval_cv"],
        reputation["score"],
        device["fingerprint_age_days"],
        device["session_fanout_24h"],
        1.0 if device.get("headless_signals", 0) > 0 else 0.0,
    ]

def score_request(behavioral, velocity, reputation, device) -> float:
    features = build_feature_vector(behavioral, velocity, reputation, device)
    prob_bot = MODEL.predict([features])[0]
    return round(prob_bot * 100, 2)

def gate(score: float, previous_tier: str | None) -> str:
    # Hysteresis: once a session escalates to "challenge" or "block", it takes a
    # materially lower score to fall back down than the score that triggered the
    # escalation. This stops a score sitting near a threshold from flapping the
    # decision on every single request.
    if previous_tier == "block" and score > 60:
        return "block"
    if previous_tier == "challenge" and score > 45:
        return "challenge"
    if score >= 90:
        return "block"
    if score >= 70:
        return "challenge"
    if score >= 30:
        return "monitor"
    return "allow"

A gradient-boosted tree ensemble is the workhorse model here rather than a deep neural network, for a practical reason: the feature vector is tabular and low-dimensional, tree ensembles run inference in well under a millisecond on commodity hardware without a GPU, and their feature-importance output is directly interpretable, which matters when a security engineer has to explain why a specific session got blocked. The one thing a smart engineer usually assumes wrong here is that a fixed score threshold is enough; without the hysteresis shown above, a session whose true risk sits right at a boundary flaps between tiers on every request, which both wastes challenge invocations and produces a confusing audit trail.

The Adversarial Feedback Loop

This component’s job is to keep the model honest as bot operators study its behavior and adjust their scripts to look more human, which they will, because a static defense degrades the moment an attacker notices it.

Adversarial model updates are what separate a bot-detection system from a one-time classifier. The moment a scoring model ships, sophisticated bot operators start probing it: replaying recorded human mouse paths with randomized noise, adding artificial jitter to request timing, or renting residential proxy pools to defeat IP reputation. A model trained once and never refreshed degrades on a predictable timeline as those countermeasures spread. The fix is a continuous label-collection and retraining pipeline rather than a single training run.

Labels come from several sources with different trust levels. A honeypot is the cheapest and most reliable: a form field or link that is invisible to a real user but present in the DOM for any script that blindly fills every input or follows every link.

<!-- Honeypot field - invisible to a real user, present for any script that
     blindly fills every form field. A non-empty value here is a strong bot signal. -->
<input type="text" name="middle_name_confirm" style="position:absolute;left:-9999px" tabindex="-1" autocomplete="off" />

Other label sources include challenge outcomes (a session forced into the challenge tier that fails is a strong bot label; one that passes is a weaker but still useful human label), confirmed chargebacks and account-takeover reports from the fraud team, and periodic manual review of a sampled slice of medium-confidence sessions to calibrate the model against edge cases automated labeling misses. Before a retrained model ever replaces the production one, it runs in shadow mode against the same live traffic and is compared against a held-out labeled set.

# Shadow evaluation - score traffic with the candidate model without acting on
# it, then compare against the production model before promoting
# Demonstrates: canary-safe model promotion gated on precision/recall against a labeled holdout
from dataclasses import dataclass

@dataclass
class EvalResult:
    precision: float
    recall: float
    false_positive_rate: float

def evaluate_candidate(candidate_model, labeled_holdout: list[dict]) -> EvalResult:
    tp = fp = fn = tn = 0
    for row in labeled_holdout:
        predicted_bot = candidate_model.predict([row["features"]])[0] >= 0.5
        actual_bot = row["label"] == "bot"
        if predicted_bot and actual_bot:
            tp += 1
        elif predicted_bot and not actual_bot:
            fp += 1
        elif not predicted_bot and actual_bot:
            fn += 1
        else:
            tn += 1

    precision = tp / (tp + fp) if (tp + fp) else 0.0
    recall = tp / (tp + fn) if (tp + fn) else 0.0
    fpr = fp / (fp + tn) if (fp + tn) else 0.0
    return EvalResult(precision, recall, fpr)

def should_promote(candidate: EvalResult, production: EvalResult) -> bool:
    # Never promote a model that trades detection gains for materially more
    # false positives than production already has - false positives cost real
    # revenue, and a small regression compounds across millions of sessions.
    return (
        candidate.recall >= production.recall
        and candidate.false_positive_rate <= production.false_positive_rate * 1.05
    )

Data Model

The data model spans four entities: device fingerprints, IP reputation, append-only score decisions, and training labels.

-- Device fingerprints: stable identity that survives cookie clearing
CREATE TABLE device_fingerprints (
    fingerprint_id        BYTEA       PRIMARY KEY,  -- SHA-256 of combined entropy sources
    canvas_hash           TEXT        NOT NULL,
    webgl_renderer        TEXT,
    hardware_concurrency  SMALLINT,
    first_seen            TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    last_seen             TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    session_fanout_24h    INTEGER     NOT NULL DEFAULT 0  -- distinct sessions on this device today
);

CREATE INDEX ON device_fingerprints (last_seen);

-- IP reputation: refreshed from threat-intel feeds plus locally observed abuse
CREATE TABLE ip_reputation (
    ip_bucket        INET        PRIMARY KEY,   -- /24 for IPv4, /64 for IPv6 - buckets rotating addresses
    asn              INTEGER,
    is_datacenter    BOOLEAN     NOT NULL DEFAULT FALSE,
    is_known_proxy   BOOLEAN     NOT NULL DEFAULT FALSE,
    is_tor_exit      BOOLEAN     NOT NULL DEFAULT FALSE,
    abuse_score      REAL        NOT NULL DEFAULT 0,
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX ON ip_reputation (asn);

-- Score decisions: one row per scored request, append-only and very high volume
CREATE TABLE score_decisions (
    id              BIGINT      GENERATED ALWAYS AS IDENTITY,
    session_id      UUID        NOT NULL,
    fingerprint_id  BYTEA       REFERENCES device_fingerprints(fingerprint_id),
    ip_bucket       INET        NOT NULL,
    score           REAL        NOT NULL,
    decision        TEXT        NOT NULL CHECK (decision IN ('allow','monitor','challenge','block')),
    model_version   TEXT        NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (created_at, id)
) PARTITION BY RANGE (created_at);

CREATE TABLE score_decisions_2026_07_03 PARTITION OF score_decisions
    FOR VALUES FROM ('2026-07-03') TO ('2026-07-04');

-- BRIN index instead of btree: rows arrive in near-timestamp order on an
-- append-only table, so a BRIN index gives fast range scans at a fraction of
-- a btree's storage cost at this volume
CREATE INDEX ON score_decisions USING BRIN (created_at);
CREATE INDEX ON score_decisions (session_id);

-- Training labels: ground truth fed back from honeypots, challenge outcomes,
-- and confirmed fraud, each tagged with a provenance source and confidence
CREATE TABLE training_labels (
    id           BIGSERIAL   PRIMARY KEY,
    session_id   UUID        NOT NULL,
    label        TEXT        NOT NULL CHECK (label IN ('bot','human')),
    source       TEXT        NOT NULL CHECK (source IN ('honeypot','challenge_outcome','chargeback','manual_review')),
    confidence   REAL        NOT NULL DEFAULT 1.0,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX ON training_labels (source, created_at);

score_decisions is partitioned by day because it is by far the highest-volume table (tens of millions of rows daily at peak) and the query patterns (recent-window lookups, rolling off data past the 30-day hot retention window) both align naturally with time-range partitioning. ip_reputation is sharded by hashing ip_bucket across the Reputation Store’s cluster, since reputation lookups happen on nearly every request and need to stay off any single node. We bucket individual IPs into /24 or /64 ranges rather than tracking single addresses, both because reputation intelligence is typically published at that granularity and because it caps the table’s cardinality at a manageable size instead of growing unbounded with every rotating residential-proxy address a bot ever touches.

A record’s lifecycle, traced in the diagram above: a batch of behavioral events is born at the SDK, lives for at most FLUSH_INTERVAL_MS, and is consumed once by the Edge Ingestion layer. A score_decisions row is born at the moment the Scoring Engine returns a value, is immutable from then on, and eventually rolls off to cold storage after its 30-day hot retention window. A training_labels row is born much later, sometimes hours or days after the original request, when a challenge outcome resolves or a chargeback lands, and it is that delayed arrival that makes the Adversarial Feedback Loop fundamentally asynchronous from the scoring path.

Key Algorithms and Protocols

Exact vs Bucketed Sliding-Window Request Counting

The fixed-bucket counter shown in the Edge Ingestion component is an approximation: it can overcount slightly near bucket boundaries. For lower-volume, higher-stakes keys, like per-user login-attempt counting, we want an exact window instead, computed with a Redis sorted set.

-- Exact sliding-window request counter using a Redis sorted set
-- Demonstrates: O(log n) insert/trim per call, exact (not bucketed) window count
-- KEYS[1] = velocity key, ARGV[1] = now (ms), ARGV[2] = window_ms, ARGV[3] = member id
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local member = ARGV[3]

redis.call('ZADD', key, now, member)
redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
redis.call('EXPIRE', key, math.ceil(window / 1000) + 1)
return redis.call('ZCARD', key)

# Python wrapper around the sliding-window-log Lua script
# Demonstrates: exact request count over a trailing window, no bucket-boundary error
import time
import uuid

def exact_request_count(redis_client, script, key: str, window_ms: int = 10_000) -> int:
    now_ms = int(time.time() * 1000)
    member = f"{now_ms}-{uuid.uuid4().hex[:6]}"  # disambiguate same-millisecond requests
    return script(keys=[key], args=[now_ms, window_ms, member])

The exact log costs O(log n) per ZADD and ZREMRANGEBYSCORE, where n is the number of entries currently inside the window, and O(n) space per key, versus the fixed-bucket approach’s O(1) time and space with a small approximation error at bucket edges. At 50,000 requests/second across millions of distinct keys, that O(n) space cost is unaffordable on the hot path, so we reserve the exact log for keys where n stays naturally small, like per-user login attempts, and use fixed buckets everywhere the request volume per key could grow large.

Mouse Path Jitter via Path Simplification

Straightness and velocity variance, computed in the Signal Collection component, capture the overall shape of a path. A complementary signal is how many small corrective direction changes a path contains, independent of its length or duration, computed with a simplification algorithm borrowed from cartography.

# Path simplification to count corrective micro-movements (jitter events)
# Demonstrates: O(n log n) average-case Douglas-Peucker simplification, edge-case filtering
import math

def _perpendicular_distance(point, start, end):
    if start == end:
        return math.hypot(point[0] - start[0], point[1] - start[1])
    num = abs(
        (end[1] - start[1]) * point[0]
        - (end[0] - start[0]) * point[1]
        + end[0] * start[1]
        - end[1] * start[0]
    )
    den = math.hypot(end[0] - start[0], end[1] - start[1])
    return num / den

def simplify_path(points: list[tuple[float, float]], epsilon: float = 2.0) -> list[tuple[float, float]]:
    if len(points) < 3:
        return points
    max_dist, index = 0.0, 0
    for i in range(1, len(points) - 1):
        dist = _perpendicular_distance(points[i], points[0], points[-1])
        if dist > max_dist:
            max_dist, index = dist, i
    if max_dist > epsilon:
        left = simplify_path(points[: index + 1], epsilon)
        right = simplify_path(points[index:], epsilon)
        return left[:-1] + right
    return [points[0], points[-1]]

def jitter_score(raw_points: list[tuple[float, float]]) -> float:
    # Filter out multi-monitor "teleports" - a single-frame jump over 800px is
    # a display switch, not a real hand movement, and would otherwise register
    # as a huge, spurious corrective jitter event
    filtered = [raw_points[0]]
    for p in raw_points[1:]:
        if math.hypot(p[0] - filtered[-1][0], p[1] - filtered[-1][1]) < 800:
            filtered.append(p)
    if len(filtered) < 3:
        return 0.0
    simplified = simplify_path(filtered)
    # More simplification points relative to raw points means more direction
    # changes the algorithm had to preserve - i.e. more corrective jitter
    return len(simplified) / len(filtered)

Douglas-Peucker simplification runs in O(n log n) on average when the recursive splits stay balanced, degrading to O(n squared) in the worst case of a near-straight line where every recursive call only trims one point at a time, which happens to be a rare case in practice for real pointer data. The edge cases worth handling explicitly: paths with fewer than three points return no signal rather than a misleading default, and any single-frame jump larger than a plausible hand movement is filtered before simplification, since a multi-monitor setup or a resized window can otherwise register as an enormous, spurious jitter event.

Consistent Hashing for Reputation Store Sharding

The Reputation Store is queried on nearly every request, so it has to be sharded across many nodes, and the shard set needs to change (scale up during a traffic spike, replace a failed node) without invalidating most of the cache at once.

# Consistent hash ring for routing reputation lookups across shards
# Demonstrates: O(log n) shard lookup via bisect, virtual nodes for load balance
import bisect
import hashlib

class ConsistentHashRing:
    def __init__(self, nodes: list[str], virtual_nodes: int = 150):
        self.virtual_nodes = virtual_nodes
        self.ring: dict[int, str] = {}
        self.sorted_keys: list[int] = []
        for node in nodes:
            self.add_node(node)

    def _hash(self, key: str) -> int:
        return int(hashlib.sha256(key.encode()).hexdigest(), 16)

    def add_node(self, node: str):
        for i in range(self.virtual_nodes):
            h = self._hash(f"{node}#{i}")
            self.ring[h] = node
            bisect.insort(self.sorted_keys, h)

    def remove_node(self, node: str):
        for i in range(self.virtual_nodes):
            h = self._hash(f"{node}#{i}")
            del self.ring[h]
            self.sorted_keys.remove(h)

    def get_node(self, key: str) -> str:
        if not self.ring:
            raise RuntimeError("no shards registered")
        h = self._hash(key)
        idx = bisect.bisect(self.sorted_keys, h) % len(self.sorted_keys)
        return self.ring[self.sorted_keys[idx]]

Lookup is O(log n) via binary search over the ring, where n is nodes * virtual_nodes. Adding or removing a shard remaps roughly 1/N of keys instead of the full keyspace, which is the entire point: a naive hash(key) % node_count scheme would remap almost every key the instant the node count changes, momentarily blinding the Scoring Engine to reputation data for nearly every request during a scale-up event.

Scaling and Performance

The Signal Collection ingestion path and the Scoring Engine are both stateless and scale horizontally behind a load balancer with no sticky sessions, since every request carries its own compound key and all shared state lives in the Reputation Store or the feature aggregation layer, not in process memory.

Capacity Estimation:

Given:
  - 50,000 requests/second at peak (flash sale / ticket drop)
  - 18,000 requests/second sustained average
  - Raw behavioral event: ~150 bytes; ~35 events per active session per minute
  - 12M daily sessions, 90-day raw telemetry retention
  - score_decisions row: ~140 bytes

Scoring throughput (inline, per request):
  50,000/s * (1 feature-vector assembly + 1 GBDT inference)
  GBDT inference (compiled tree ensemble): ~0.3-0.8ms per call on one core
  Cores needed at peak: 50,000/s / (~2,000 inferences/sec/core) = ~25 cores
  -> ~7 pods of 4 vCPU each, autoscaled with headroom to 12 pods

Raw telemetry storage (90-day retention):
  12M sessions/day * 35 events/session (avg active minutes) * 150 bytes
  = ~63 GB/day raw -> ~5.7 TB over 90 days
  Compressed columnar storage (Parquet, ~4x) -> ~1.4 TB retained

Score decision log (append-only, BRIN-indexed):
  18,000/s average * 86,400s * 140 bytes = ~218 GB/day
  Retained hot for 30 days (~6.5 TB), rolled to cold storage after

Reputation store (sharded, in-memory):
  ~40M distinct IP /24 buckets tracked globally * ~120 bytes/entry = ~4.8 GB
  Fits comfortably in a sharded Redis cluster, replicated per region

Decision cache (short-TTL, cuts repeat inference during bursts):
  Caching a session's score for 3s during a burst absorbs an estimated
  70-80% of repeat requests from the same session without a fresh inference call

The dominant cost is CPU on the Scoring Engine’s inference path, not storage or bandwidth, which is why we compile the tree ensemble ahead of time rather than interpreting it and cache a session’s score for a few seconds so a burst of repeat requests from the same session does not trigger redundant inference calls. Read/write ratio favors reads heavily on the Reputation Store (looked up on nearly every request, updated only when a threat-intel feed refreshes or new local abuse is observed) and writes concentrate on score_decisions, which is why that table is optimized purely for append throughput rather than update-in-place performance. The hottest keys in the Reputation Store are, unsurprisingly, well-known datacenter and cloud-provider IP ranges, which we pin to a small number of frequently-accessed shards with extra replicas rather than letting the consistent-hash ring spread that concentrated load thinly.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Scoring Engine pod overload	p99 decision latency alert on the request path	Requests queue past latency budget	Fail open to allow, or fall back to a lightweight rule-based heuristic on request velocity and reputation alone, while pods autoscale
Reputation Store shard unavailable	Connection timeouts trip a circuit breaker	Reputation feature treated as unknown/neutral for the affected IP range	Circuit breaker skips the reputation feature (small accuracy dip), shard reprovisioned from a replica
Adversarial evasion (bots mimic human signals closely enough to slip under threshold)	Gradual precision/recall drift caught in scheduled shadow evaluation, or a conversion-metric anomaly (checkout success rate)	Silent rise in false negatives	Emergency retrain triggered, thresholds temporarily tightened, honeypot and challenge-outcome labels reweighted higher
Telemetry ingestion pipeline backlog	Consumer lag metric on the ingestion queue	Scoring uses stale aggregated velocity/behavioral features	Backpressure alert, autoscale consumers, degrade gracefully to the last-known feature snapshot with a freshness flag that lowers model confidence
Label poisoning / feedback loop attack	Anomaly in label-source distribution (spike in labels from one source or IP range)	A retrained model could learn an attacker-favorable pattern	Label-provenance weighting, quarantine suspicious label batches, require human review before a retrained model is promoted
Clock skew between edge nodes and the velocity service	Velocity computations show impossible request rates or negative time deltas	Incorrect burstiness scoring	NTP-sync all edge nodes; use server receipt time as canonical whenever a client-reported timestamp is untrusted

Comparison of Approaches

Approach	Latency per request	Complexity	Failure mode	Best fit
CAPTCHA-first (challenge every suspicious session upfront)	Adds seconds of user friction	Low	Solvable by CAPTCHA-solving farms for a few cents per solve; alienates legitimate users	Low-traffic sites without engineering resources for a behavioral pipeline
Static IP/user-agent blocklist only	Near-zero	Low	Trivially bypassed via residential proxy rotation and user-agent spoofing; high false positives on shared NAT IPs	First line of defense only, never sufficient alone
Pure rate limiting (request velocity only, no behavioral signal)	Near-zero	Low-medium	Low-and-slow bots that stay under threshold evade detection entirely; punishes legitimate bursty traffic like flash sales	Baseline abuse protection layered under a richer system
Behavioral + network score-based gating (this design)	Sub-15ms, fully inline	High (SDK, feature pipeline, ML model, retraining loop)	Adversarial evasion as bots mimic human signals over time; requires continuous retraining	High-value, high-traffic targets where CAPTCHA friction has a real, measurable revenue cost
Managed third-party bot service (Cloudflare, DataDome, HUMAN)	Near-zero when integrated at the edge	Low for the integrating team, high behind the scenes	Vendor lock-in, less visibility into scoring logic, cost scales directly with traffic volume	Teams without dedicated ML/security engineering capacity to build and maintain this in-house

We would pick behavioral plus network score-based gating for any platform where CAPTCHA friction has a measurable cost, like a checkout or a ticket-drop flow, because the alternative approaches either fail to catch anything beyond unsophisticated bots or impose a tax on every real user to do it. In practice, many teams start with a managed vendor for the baseline layer and build proprietary behavioral signal on top once traffic and abuse patterns justify the investment, since the two approaches are not mutually exclusive: a managed service can sit in front of a system like this one, absorbing the bulk of unsophisticated traffic while the in-house model specializes on the patterns unique to the business.

Key Takeaways

• Behavioral fingerprinting turns how a session interacts with a page into a numeric signal, catching automation that a static credential check would never see.
• Mouse movement analysis exploits the gap between Fitts’s-law human motion and a script’s straight, constant-velocity path, without ever asking the user to prove anything.
• Device fingerprinting produces a stable identity that survives cookie clearing, closing the trivial evasion of wiping session state between requests.
• Request velocity signals must be keyed on a compound identity, IP plus device plus session, not IP alone, or the system either over-blocks shared networks or under-catches rotating proxies.
• IP reputation behaves like a credit score: most valuable when pooled across many sites and decayed over time, not computed fresh from one site’s own limited traffic.
• Score-based gating replaces a binary allow-or-deny with a continuum, which is what makes it possible to protect the site without ever showing most users a challenge.
• Fail-open design on the decision path matters more than almost any single detection heuristic, since a scoring outage that blocks all traffic is strictly worse than one that briefly under-detects bots.
• Adversarial model updates are not optional maintenance; a model trained once and left alone degrades on a predictable timeline the moment attackers start probing it.

The counter-intuitive lesson is that no individual signal in this system is particularly hard for a determined attacker to spoof in isolation. Mouse-path randomization, proxy rotation, and canvas-fingerprint spoofing are all documented, achievable techniques. What is hard is spoofing all of them simultaneously, consistently, across every request, while staying under a score threshold tuned against exactly that behavior, and the system’s real defense is architectural: fusing many individually weak signals into one score and continuously retraining against fresh adversarial behavior, rather than any single clever heuristic being unbeatable on its own.

Frequently Asked Questions

Q: Why not just use CAPTCHA for every suspicious request instead of building this entire pipeline?

A: CAPTCHA solves a much narrower problem and at a real cost: it interrupts users, and a meaningful share of legitimate visitors abandon a flow rather than solve one, which is measurable revenue loss during a high-value flow like checkout. It is also a solved problem for motivated attackers, since human CAPTCHA-solving services clear a few cents per solve and computer-vision models handle simple challenges outright. This design reserves a visible challenge for the small minority of highest-risk traffic instead of the default response to any uncertainty.

Q: Why not just block all datacenter and proxy IPs outright instead of treating reputation as one signal among several?

A: A large and growing share of legitimate traffic now routes through commercial VPNs and carrier-operated proxy infrastructure that is indistinguishable from malicious proxy traffic at the IP level alone. Blocking on that signal alone would turn away privacy-conscious real customers and entire mobile carrier user bases. Reputation is deliberately weighted as one input feature the model combines with behavioral and velocity signals, not a standalone gate.

Q: How do you avoid false positives against legitimate automation, like QA test scripts or assistive-technology users who do not move a mouse the way the model expects?

A: Known internal QA and monitoring traffic is allowlisted by API key or service identity before it ever reaches the scoring path, since it should never be scored as anonymous traffic in the first place. For accessibility tooling, the model is deliberately trained with labeled examples of keyboard-only and screen-reader navigation patterns as legitimate human behavior, and the false-positive-rate target in this design (under 0.05%) is monitored specifically against known assistive-technology sessions in the shadow evaluation step, not just against the general population.

Q: Why score every individual request instead of just scoring the session once and caching the verdict?

A: A session’s risk can change mid-flight, most notably when a bot successfully establishes a session and then automates the rest of the flow, or when a compromised account starts behaving abnormally partway through. The short-TTL decision cache described in the Scaling section captures most of the performance benefit of session-level caching (avoiding redundant inference on rapid repeat requests) while still allowing the score to shift within seconds if behavior changes.

Q: How is training data labeled reliably when you fundamentally cannot ask a session “are you a bot”?

A: Labels come from indirect but reliable signals rather than a direct question: honeypot interactions that only automation would trigger, outcomes of the invisible step-up challenge reserved for medium-risk scores, confirmed fraud and chargeback events from the payments team, and periodic manual review of a sampled slice of medium-confidence sessions. Each label carries a source and a confidence weight precisely because these signals differ in reliability, and the retraining pipeline accounts for that rather than treating every label as equally trustworthy.

Q: Isn’t failing open on a scoring engine outage a security hole an attacker could deliberately trigger?

A: It is a real tradeoff, not a free lunch. An attacker who can reliably take down the Scoring Engine could, in principle, force a window of degraded detection. The mitigation is a layered fallback rather than a single cliff edge: on scoring engine failure, the system does not disable protection entirely, it drops to a cheaper rule-based heuristic on request velocity and reputation alone, which still catches unsophisticated volumetric attacks even without full behavioral scoring, while avoiding the far larger blast radius of blocking every legitimate user during an infrastructure incident.

Interview Questions

Q: Design the client-side signal collection SDK so it degrades gracefully when JavaScript is disabled or blocked by a privacy extension. What signal remains available server-side alone?

Expected depth: Discuss falling back to velocity and reputation-only scoring when no behavioral or device-fingerprint payload arrives, and why the score should shift toward a more conservative default rather than a score of zero when signal is simply missing, since ad blockers and privacy extensions strip fingerprinting scripts for real users too. Cover server-observable fallback signals like TLS/JA3 fingerprinting, HTTP header ordering, and request timing patterns that do not depend on any client-side script running at all.

Q: Walk through how you would detect a new bot family that has learned to replicate human-like mouse jitter by replaying recorded human sessions with randomization.

Expected depth: Discuss detecting near-duplicate mouse paths across many distinct sessions as a sign of scripted replay, correlating device-fingerprint reuse across sessions that claim to be distinct users, and evolving the feature set from raw path shape toward population-level uniqueness (how similar is this path to every other path seen recently) rather than only per-session statistics. Tie this back to the retraining cadence: this kind of evasion typically shows up first as a slow precision drift in shadow evaluation before it is obvious in any single session.

Q: How would you design the shadow evaluation and canary process for promoting a newly retrained model into the live scoring path without risking a regression during a peak sales event?

Expected depth: Cover running the candidate model in shadow mode against live traffic without it affecting any decision, comparing precision, recall, and false-positive rate against both a held-out labeled set and the production model’s historical decisions, then a staged canary rollout to a small percentage of traffic with automatic rollback triggers tied to false-positive-rate or conversion-rate anomalies. Mention deliberately avoiding promotion during a known high-traffic window regardless of how clean the shadow metrics look.

Q: A large enterprise customer reports that employees behind a corporate NAT are getting blocked because request-velocity limits treat thousands of employees as a single IP. How do you fix this without weakening bot detection?

Expected depth: Discuss keying velocity on a compound identity (IP plus device fingerprint plus session) instead of raw IP alone, ASN-aware thresholds that scale up for known large-NAT ranges, and an enterprise-integration path like a verified corporate IP range allowlist. Cover the tradeoff explicitly: any relaxation for a known-good range is itself a target, so it needs its own monitoring rather than becoming a permanent blind spot.

Q: How would you extend this system to catch bots that never render a real page at all, doing pure headless HTTP scraping with no browser and no JavaScript execution?

Expected depth: Discuss that the total absence of any behavioral or fingerprint telemetry is itself a strong signal, distinct from a privacy-conscious user who simply blocks fingerprinting scripts, and how to tell the two apart using secondary signals like TLS/JA3 fingerprinting, HTTP header ordering, and the pattern of which sub-resources get requested (a real browser fetches CSS, JS, and images in a characteristic parallel pattern that a raw HTTP scraper fetching only the HTML endpoint does not replicate).