Picture a library system before any of its branches shared a computer. Each branch keeps its own card catalog, and the same patron might hold a card at the downtown branch under “J. Doe” and a card at the uptown branch under “Jane A. Doe”, registered years apart with a different apartment number on file. A librarian who wants to combine those two borrowing histories into one record cannot just check whether the names are spelled identically. She has to weigh several imperfect clues at once, the name, the birth year on file, the phone number, and decide whether the balance of evidence says “same person” or “two different people who happen to look similar on paper.”

A medical record deduplication system is that librarian’s job, done automatically, at a scale where guessing wrong is not a returned-book fine but a patient safety incident. A regional hospital network with 400 participating source systems and 50 million tracked patient identities ingests roughly 200,000 new or updated demographic records a day, and for every one of them the system has to answer “is this an existing patient we already know, or someone new” without a universal identifier it can simply look up. Two records for the same person almost never look identical: a maiden name changes, a transposed digit corrupts a date of birth, a self-pay patient never provides a Social Security number, a family shares one home phone line across three different people.

The naive approach is to require an exact match on some identifier, SSN plus name plus date of birth, and treat anything less as a different person. That collapses immediately in real hospital data. SSNs are frequently missing, self-reported incorrectly, or withheld for privacy reasons, especially for newborns and undocumented patients. Exact string matching on names misses the ordinary variation of nicknames, transliteration, and marriage. Fuzzy matching alone, done as an ad hoc weighted similarity score, has no principled way to say how much a matching SSN should count compared to a matching ZIP code, and no natural way to translate that fuzziness into a numeric threshold anyone can defend to an auditor. Comparing every incoming record against all 50 million existing patients is also simply not computationally viable at the latency this system needs.

The forces in tension are recall, precision, throughput, and reversibility, and healthcare weighs them unlike almost any other domain. Missing a true duplicate (recall) means a fragmented medical history, redundant tests, and a clinician working from an incomplete allergy or medication list. Incorrectly merging two different patients (precision) means one person’s chart shows another person’s blood type, drug allergies, and diagnoses, an error with direct clinical consequences. Comparing every record against the full population (throughput) does not scale past a small hospital. And because even a well-tuned model will occasionally be wrong, every merge decision has to be reversible years later without losing a single field of data. We need to solve for a bounded-cost candidate search that scales sublinearly with population size, a scoring model with mathematically principled confidence bands rather than an arbitrary similarity number, a conflict resolution layer that produces one coherent golden record from disagreeing source data, and a fully reversible, auditable trail of every decision, because in this domain an incorrect merge is not an inconvenience, it is an incident report.

Requirements and Constraints

Functional Requirements

• Ingest patient demographic records from heterogeneous hospital systems via HL7 v2 ADT feeds, FHIR Patient resources, and periodic batch CSV extracts from legacy systems
• Generate blocking keys for each incoming record and retrieve a bounded candidate set of potentially matching existing patients, without scanning the full population
• Score every candidate pair using a probabilistic record linkage model and classify each pair as match, non_match, or review
• Route review pairs to a human review queue along with the underlying field-by-field comparison evidence
• On a confirmed match (automatic or reviewer-approved), resolve field-level conflicts between source records using survivorship rules and materialize a single golden record
• Maintain a durable, append-only audit trail of every merge decision, including a full snapshot of both source records and the reasoning that produced the decision
• Support unmerging a previously merged pair, restoring both source records as independent identities without losing any data recorded since the merge

Non-Functional Requirements

• Scale: 50 million golden patient identities, 400 participating hospital source systems, roughly 200,000 new or updated records ingested per day
• Match latency: p95 under 2 seconds for a single incoming record on the real-time admission feed, from arrival to an auto-merge, auto-non-match, or queued-for-review decision
• Batch throughput: able to absorb a nightly flat-file batch of up to 50,000 records from legacy hospital feeds within a 2-hour processing window
• Accuracy: false-merge rate (two distinct patients incorrectly combined) under 1 in 100,000 merges; missed-duplicate rate under 2%, both measured against a labeled clerical-review sample
• Availability: 99.9% for the real-time matching path; an outage must fail toward queuing records for later matching, never toward guessing a merge decision
• Auditability: every merge and unmerge decision must be reconstructable years later, meeting a multi-year, HIPAA-aligned retention requirement
• Reversibility: any merge must be fully undoable at any point after it happens, without permanent loss of either source record’s data

Constraints and Assumptions

• We are not designing the clinical data model (labs, medications, encounters) itself, only the patient identity layer and the stable golden identifier that clinical records reference
• Source systems are assumed to emit demographic changes reliably through existing HL7 or FHIR interfaces; the interface engines themselves are out of scope
• We assume a single legal entity (a hospital network or health information exchange) with data-sharing agreements already in place. Cross-organization matching without such agreements is a legal and policy problem, not an engineering one, and is out of scope
• Downstream clinical decision support that consumes the golden record (allergy checks, drug interaction warnings) is a consumer of this system, not part of it
• Field-level provenance (which source system supplied which value, and when) is assumed to be available on every source record

High-Level Architecture

The system has six major components: the Ingestion and Normalization Gateway, the Blocking Service, the Matching Engine, the Merge Resolution and Golden Record Service, the Master Patient Index (the identity graph store), and the Audit and Unmerge Service.

A record arrives from a hospital system, HL7 v2 for a live admission, FHIR for an API-integrated system, or a batch CSV drop for a legacy feed, and the Ingestion and Normalization Gateway standardizes its fields into one internal shape before anything downstream ever sees it. The Blocking Service then generates several independent blocking keys for the record and retrieves the union of candidate patients that share any of those keys, cutting the comparison universe from 50 million down to a bounded set, typically 20 to 200 candidates. The Matching Engine scores every candidate pair using the Fellegi-Sunter probabilistic linkage model, producing a log-likelihood ratio for each pair. Pairs scoring above the upper threshold flow directly to the Merge Resolution and Golden Record Service; pairs in the gray zone between thresholds go to a human review queue, and an approved review feeds into the same Merge Resolution path.

The Merge Resolution and Golden Record Service applies field-level survivorship rules to reconcile any conflicting data between the two source records and writes a single golden record to the Master Patient Index, which is the durable identity graph every clinical system references by its stable mpi_id. In parallel, the Audit and Unmerge Service records an immutable snapshot of both source records and the exact rule trace that produced the merge, which is the only mechanism that ever allows a merge to be reversed later.

Component Deep Dives

The Ingestion and Normalization Gateway

This component’s job is to turn wildly inconsistent demographic data from 400 different hospital systems into one consistent internal shape before any matching logic ever runs.

A smart engineer’s first instinct is to treat normalization as a minor preprocessing step, trimming whitespace and uppercasing strings. In practice, a large share of dedup quality problems trace back to normalization inconsistency rather than the matching algorithm itself. A hospital that silently changes its date-of-birth format, or a feed that sends ZIP+4 codes where the rest of the system expects five digits, corrupts every blocking key and comparator built on top of that field, for every record from that source, until someone notices.

# Normalizes an inbound HL7/FHIR patient record into the internal comparison shape
# Demonstrates: multi-format DOB parsing, phone digit stripping, deterministic PII tokenization
import re
import hmac
import hashlib
from datetime import datetime

DOB_FORMATS = ("%Y-%m-%d", "%Y%m%d", "%m/%d/%Y", "%d-%b-%Y")

def parse_dob(raw: str) -> str:
    raw = raw.strip()
    for fmt in DOB_FORMATS:
        try:
            return datetime.strptime(raw, fmt).strftime("%Y-%m-%d")
        except ValueError:
            continue
    raise ValueError(f"unrecognized DOB format: {raw!r}")

def normalize_phone(raw: str) -> str:
    digits = re.sub(r"\D", "", raw or "")
    return digits[-10:] if len(digits) >= 10 else digits

def tokenize_ssn(raw_ssn: str, hmac_key: bytes) -> str:
    digits = re.sub(r"\D", "", raw_ssn or "")
    if len(digits) != 9:
        return ""
    return hmac.new(hmac_key, digits.encode(), hashlib.sha256).hexdigest()

def normalize_record(raw: dict, hmac_key: bytes) -> dict:
    return {
        "source_system_id": raw["source_system_id"],
        "external_mrn": raw["mrn"].strip(),
        "first_name": raw["first_name"].strip().upper(),
        "last_name": raw["last_name"].strip().upper(),
        "dob": parse_dob(raw["dob"]),
        "sex": raw.get("sex", "U").strip().upper()[:1],
        "zip5": re.sub(r"\D", "", raw.get("zip", ""))[:5],
        "phone": normalize_phone(raw.get("phone", "")),
        "ssn_token": tokenize_ssn(raw.get("ssn", ""), hmac_key),
    }

The SSN is deterministically tokenized with an HMAC rather than stored or compared in plaintext. This keeps the field usable as a strong comparator (two tokens are equal if and only if the original SSNs were equal) without the system ever persisting raw SSNs longer than the ingestion request itself.

The Blocking Service