Picture a hospital ward’s bedside monitors. Every patient gets the same default alarm: heart rate over 100 beats per minute triggers a page to the nurse’s station. A marathoner recovering from surgery runs a resting heart rate of 48, so 90 already means something is badly wrong for her, but the monitor stays silent until 100. Meanwhile a patient on a standard anxiety medication sits at 95 all day as a matter of course, and the monitor screams every few hours for a condition that is not an emergency at all. The fixed threshold is wrong for almost everyone, because it was never actually about the number 100, it was about what is normal for this particular patient at this particular time of day. A good ward instead learns each patient’s personal baseline and its normal daily rhythm, sleeping heart rate lower, post-meal heart rate higher, and pages a nurse only when a reading breaks that personal pattern by more than chance would explain.

Infrastructure metrics have the exact same problem, at a much larger scale. A checkout service’s request rate is not one number, it is a curve that rises every weekday morning, dips every night, spikes on paydays, and doubles during a flash sale that was planned two weeks in advance. A fixed alert threshold, “page if CPU exceeds 80%,” either fires constantly during expected peak traffic, training every engineer on the team to ignore pages, or sits so loose that it misses a real degradation that happens to occur during a quiet period. At a scale of 2 million actively monitored time series across 50,000 hosts and containers, with metrics arriving at roughly 200,000 data points per second sustained, hand-tuning a static threshold per metric is not a staffing problem, it is a mathematical impossibility, and even if it were possible, a threshold tuned for Tuesday’s traffic is already wrong by Saturday.

The naive fix, alert whenever a metric moves more than some fixed percentage from its value an hour ago, fails for a related but sharper reason: it cannot distinguish a genuine anomaly from an expected seasonal swing, and it has no concept of correlation. When a database’s connection pool actually exhausts during a real incident, that single root cause ripples outward and trips independent alerts on API latency, queue depth, error rate, and downstream service timeouts, all within the same sixty seconds. A naive system pages on-call twenty separate times for one incident, an alert storm that buries the one signal that actually matters under nineteen symptoms of it, and by the third storm in a month, engineers start silencing the entire alerting channel rather than triaging it. The forces in tension are sensitivity against noise, and per-metric precision against the operational cost of maintaining millions of individually tuned models.

We need to solve for three things simultaneously: a baseline that adapts to each metric’s own seasonal rhythm and trend rather than a single fixed number, a correlation layer that recognizes when many simultaneous anomalies are one incident rather than many separate problems and suppresses the redundant noise, and a feedback mechanism that gets measurably better at not paging engineers for the same false positive twice.

Requirements and Constraints

Functional Requirements

• Learn a dynamic baseline for every actively monitored metric, capturing daily and weekly seasonality, trend, and normal variance, without requiring an engineer to hand-configure a threshold per metric
• Score every incoming data point for how anomalous it is relative to that metric’s current baseline, in near real time
• Correlate anomalies that fire close together in time and across topologically related services into a single incident, rather than raising one alert per metric
• Suppress duplicate and downstream alerts once an incident is open, while continuing to record every underlying anomaly for the postmortem timeline
• Onboard a brand-new metric with no history by borrowing a baseline shape from statistically similar metrics until it accumulates enough history of its own
• Capture on-call feedback, acknowledged as real, marked as false positive, or snoozed, and use it to adjust future sensitivity for that metric without a manual re-tuning cycle
• Expose an API and dashboard view showing a metric’s actual value plotted against its learned expected band, so an engineer can see why an alert did or did not fire

Non-Functional Requirements

• Scale: 2,000,000 actively scored time series across roughly 50,000 hosts and containers, ingesting at approximately 200,000 data points per second sustained, with bursts to 400,000 points per second during large deploys or scaling events
• Scoring latency: p99 under 5 seconds from the time a data point is ingested to the time an anomaly score is available for it
• Alert latency: end-to-end latency from a genuine anomaly’s onset to a page reaching on-call under 60 seconds
• Noise reduction: fewer than 5% of pages sent to a human should be later labeled false positive, and the correlated, suppressed alert volume should be at least 90% lower than naive per-metric static threshold alerting would produce at this scale
• Availability: 99.9% for the real-time scoring and correlation path; a brief scoring gap should degrade to “no new anomalies detected” rather than a crash, since a missed page is worse than a delayed one but a false page trains engineers to ignore the system
• Retention: 13 months of raw and rolled-up metric history to support seasonal model training across a full year-over-year cycle, with anomaly events and incident records retained indefinitely for audit and trend analysis
• Model freshness: per-metric baseline parameters updated continuously via streaming statistics, with a full seasonal re-decomposition at least once daily per metric cluster

Constraints and Assumptions

• We are not building the metric collection agents or exporters; we assume Prometheus-compatible scrapers, OpenTelemetry SDKs, and cloud provider metric APIs already deliver labeled time series into our ingestion layer
• We are not building a general-purpose long-term time-series database from scratch; we assume an existing TSDB such as a Prometheus-compatible remote-write target already durably stores raw metrics, and this design owns a purpose-built rollup and feature store that sits alongside it for anomaly scoring
• We are not building the paging mechanics themselves, integrations with an on-call scheduling tool are a downstream consumer; this design owns the decision of whether an anomaly is worth paging a human, not how the page is delivered
• We assume a service topology graph, which service calls which, already exists in a service catalog or CMDB and is available to the correlation engine as a read-only input
• We assume metric label cardinality is bounded by an existing ingestion-time cardinality limiter; unbounded label explosion is a separate problem this design does not attempt to solve

High-Level Architecture

The system has five major components: the Metrics Ingestion Pipeline, the Baseline Modeling Engine, the Real-Time Anomaly Scorer, the Alert Correlation and Suppression Engine, and the Feedback Loop Service.

A metric data point, a host’s CPU utilization sample, a service’s request latency, arrives from an exporter or SDK and reaches the Metrics Ingestion Pipeline, which normalizes labels, deduplicates retried scrapes, and fans the point out to two destinations in the same write: a long-term time-series database for durable raw storage, and a purpose-built Rollup and Feature Store that maintains rolling seasonal windows optimized for anomaly scoring rather than dashboarding. The Baseline Modeling Engine reads from the feature store to continuously maintain, per metric, a decomposed trend, seasonal pattern, and expected variance band. The Real-Time Anomaly Scorer reads every new point alongside the current baseline for that metric and emits a score; scores that cross a metric’s dynamic threshold become anomaly events. The Alert Correlation and Suppression Engine consumes the stream of anomaly events, groups related ones into incidents using time-window and service-topology correlation, opens a suppression window so downstream symptoms of an already-known incident do not each generate a separate page, and forwards only the deduplicated incident to the paging gateway. When an engineer acknowledges, dismisses, or snoozes an alert, that action flows into the Feedback Loop Service, which adjusts the sensitivity the Baseline Modeling Engine and Anomaly Scorer apply to that metric going forward.

The full loop, from a raw metric point through scoring, correlation, paging, and feedback back into the model, is what turns this from a one-shot statistical threshold into a system that gets quieter over time without getting blind to real incidents.

Component Deep Dives

The Metrics Ingestion Pipeline

This component’s job is to take raw metric points from thousands of heterogeneous sources, normalize them into a consistent shape, and durably fan them out to both long-term storage and the anomaly detection feature store within milliseconds.

The obvious shortcut is to let the Baseline Modeling Engine and Anomaly Scorer read directly from the long-term TSDB. That fails because a TSDB optimized for dashboard queries, arbitrary label filters, long time ranges, ad hoc aggregation, is a poor fit for the access pattern anomaly scoring actually needs: the last N seasonal cycles of one specific metric, pre-aggregated into rolling windows, fetched in single-digit milliseconds for millions of metrics continuously. The ingestion pipeline instead writes once and fans out twice, so the TSDB stays focused on durable, queryable history and the feature store stays focused on low-latency rolling statistics.

# Ingestion pipeline stage definition for the metrics gateway
pipeline: metrics-ingestion-gateway
sources:
  - type: prometheus_remote_write
    listen_port: 9201
  - type: otlp_metrics
    listen_port: 4317
  - type: cloud_provider_poll
    interval_seconds: 60
normalize:
  label_order: canonical
  dedupe_window_seconds: 5
  drop_labels: ["pod_ip", "trace_id"]
  cardinality_guard:
    max_series_per_metric_name: 500000
    action_on_breach: drop_and_alert
fanout:
  - sink: tsdb_remote_write
    target: long-term-tsdb:9090
    mode: raw
  - sink: rollup_feature_store
    target: feature-store-cluster:7000
    mode: windowed
    windows_seconds: [10, 60, 300]
partitioning:
  key: metric_id
  partitions: 256
  consumer_group: ingestion-workers

Partitioning by metric_id across 256 partitions is what lets every downstream stage, rollup computation, baseline modeling, scoring, process a given metric’s points in strict arrival order without a distributed lock, since one metric’s entire history always lands on the same partition. Deduplication within a 5 second window absorbs the retried scrapes that happen when a Prometheus exporter times out and a scrape client retries, which would otherwise register as a burst of identical values and quietly bias the seasonal model’s variance estimate.