Every piece of content uploaded to a large platform - a comment, a photo, a four-hour livestream - needs a decision: does it violate policy? At a platform receiving 10 million items per day, no human team can review everything before it goes live. The alternative, letting everything through and acting reactively, means harmful content spreads for hours before removal. The engineering challenge is building a system that makes accurate, policy-consistent decisions at machine speed while keeping a human in the loop for the cases where machine confidence is genuinely uncertain.

Think of it like a postal inspection system sorting millions of packages. The vast majority go straight through automated scanners and leave the facility in seconds. A package that triggers a scan alert gets routed to a secondary inspection lane where a human examiner makes the final call. A tiny fraction gets escalated to a senior inspector. The challenge is calibrating the scanner’s sensitivity: too loose and prohibited items get through, too tight and you paralyze the entire facility with false alarms.

The hard part in content moderation is that the “package” is multi-modal. A post might combine benign text with a harmful image, or require cultural context to disambiguate satire from incitement. A text-only classifier misses coordinated image-based abuse campaigns. An image-only classifier misreads ironic memes. The system needs a classifier ensemble that fuses signals across modalities, a confidence thresholding layer that routes each item to the right decision lane, and a human review queue that handles the uncertain middle band - all while tracking false positive rates obsessively to avoid silencing legitimate users.

We need to solve for classification accuracy across three modalities, sub-200ms latency on the automated path, human reviewer throughput at scale, a fair and auditable appeal workflow, and policy consistency across classifier versions - simultaneously.

Requirements and Constraints

Functional Requirements

• Evaluate text, image, and video content against a versioned policy ruleset with per-category confidence thresholds
• Route each item to one of three outcomes: auto-approve, auto-remove, or human review based on ensemble confidence scores
• Provide an appeal workflow where users can contest removals, with senior reviewer assignment and 3-day SLA
• Support policy version updates that take effect within 30 seconds of publish and can be applied retroactively to borderline items from the last 7 days
• Expose a real-time false positive rate dashboard segmented by category, modality, and policy version

Non-Functional Requirements

• Throughput: 10 million items per day (approximately 116 items per second sustained, with 5x burst capacity)
• Latency: text classification P99 under 150ms, image classification P99 under 250ms, video moderation SLA 10 minutes
• Human review SLA: items must receive a reviewer decision within 4 hours of queue entry
• Availability: 99.9% uptime for the moderation pipeline; classifier outages must not block content uploads
• False positive rate: under 1% of auto-removed items should be reinstated on appeal
• Policy propagation: new policy versions must reach all classifier nodes within 30 seconds of publish
• Compliance: GDPR data retention controls, DMCA takedown workflow, no public storage of CSAM perceptual hashes

Constraints

• The upload critical path must not block on moderation decisions - content goes live optimistically
• Human reviewers see the content plus the policy excerpt, not raw classifier scores
• Appeals are handled by a separate reviewer pool from initial review to prevent anchoring bias
• Retroactive policy enforcement applies only to currently live content, never to already-removed items

High-Level Architecture

The system separates the upload path from the moderation path entirely. Content is accepted at upload time and a moderation event is placed on a Kafka topic. The pipeline picks up the event, dispatches it to the classifier ensemble, applies confidence thresholds to determine routing, and either auto-approves, auto-removes, or enqueues the item for human review. Six major components carry out this flow.

The Content Ingestion API accepts uploads, acknowledges immediately, and publishes a content.submitted event to Kafka partitioned by content type. The upload path never waits for a moderation decision.

The Moderation Router consumes from Kafka and dispatches content to the appropriate classifier workers. Text and image content goes to synchronous classifier pools. Video content goes to an async video analysis queue with a separate consumer group and longer timeout budget.

The Classifier Ensemble runs text, image, and video specialist classifiers in parallel, producing confidence scores per policy category per modality. Each classifier is a dedicated ML model trained on policy-specific labeled data.

The Decision Engine receives ensemble outputs, applies per-category confidence thresholds, and determines the routing outcome. It writes an immutable moderation event record to the audit store and executes the action.