A URL shortener looks deceptively simple: take a long URL, generate a short code, return a redirect. Any junior developer can build a working version in an afternoon with a single table and a random string generator. But then a viral tweet drops a short link that gets clicked 50,000 times in the first 60 seconds. The database melts. The service returns 503. The tweet goes cold, and with it, your product’s reputation.

The problem is not the write path - most URL shorteners handle far more reads than writes, often at a 1000:1 ratio. A link shared by a celebrity may receive 100,000 clicks in minutes, each one hitting your redirect endpoint. Think of it like a highway toll plaza: the challenge is not issuing passes (writes), it is processing millions of cars passing through every hour without every car stopping to verify its pass with the central DMV office. The toll reader needs to work at the edge, with a cached copy of valid passes, not a round-trip to headquarters.

At 100,000 redirects per second with a sub-10ms p99 latency target, no database on earth can keep up if it is consulted for every redirect. Even a highly-tuned Postgres instance with index-only scans tops out around 50,000 simple reads per second per node before latency climbs. The math simply does not work for direct-to-database reads at this scale. We need to push the read path off the database entirely, catching 95%+ of requests in Redis or CDN edge cache, and only falling through to Postgres for cold-cache misses.

But the read side is only half the problem. The other half is collision-free short code generation, which must work under concurrent write load without central coordination, and analytics tracking, which must capture click metadata without adding latency to the redirect itself. We need to solve for collision-free code generation, sub-10ms redirect latency, and decoupled analytics pipeline simultaneously.

Requirements and Constraints

Functional Requirements

• Create short URLs from long URLs, returning a 7-character short code
• Redirect users visiting a short URL to the original long URL
• Support custom slugs (user-specified short codes)
• Support optional link expiry with a TTL
• Track click analytics: timestamp, IP, user agent, referrer, country
• Provide per-link analytics dashboards for creators
• Support link deletion and updating the destination URL

Non-Functional Requirements

• Redirect throughput: 100,000 redirects per second sustained, 500,000 peak
• Write throughput: 1,000 new links per second at peak
• Redirect latency: p99 under 10ms (CDN edge), p99 under 30ms (Redis cache hit), p99 under 100ms (DB fallback)
• Availability: 99.99% (52 minutes downtime per year)
• Durability: Zero link loss after successful 201 response
• Short code uniqueness: Guaranteed - no collision in the URL namespace
• Analytics freshness: Click counts visible within 60 seconds of click
• Storage: 10 billion stored links, roughly 200 bytes per link = 2 TB

Constraints and Assumptions

• Short codes are 7 characters from [0-9A-Za-z] (Base62), giving 62^7 = 3.5 trillion possible codes
• Analytics are eventually consistent - exact counts may lag by up to 60 seconds
• Links do not expire by default; expiry is an optional feature
• We do not crawl or validate destination URLs for safety (separate concerns)
• Users can be anonymous (no auth required to shorten a URL)

High-Level Architecture

The system has two clean paths that share nothing except the URL storage database. The write path - POST requests to create a new short URL - goes through the Shortener Service, which generates a code, checks for collisions, writes to Postgres, and populates Redis. The read path - GET requests to follow a short link - goes through CDN first, then the Redirect Service backed by Redis, only falling back to Postgres on a cold cache miss.

Four major components drive this architecture. The API Gateway handles rate limiting, authentication (for premium features), and routes POST /shorten to the write path and GET /{code} to the CDN-fronted read path. The Shortener Service encapsulates all code generation logic including Base62 encoding, collision detection via Bloom filter, and transactional writes to Postgres and Redis. The Redirect Service is a thin, stateless Go process that calls HGET on Redis with the short code and returns an HTTP 302 response in microseconds. The Analytics Pipeline is fully async - the Redirect Service fires a Kafka event on each click and returns the redirect immediately, with no latency added to the hot path.

The Shortener Service

The Shortener Service’s job is to take a long URL and produce a guaranteed-unique 7-character short code, then durably store the mapping.

A naive approach generates a random string and checks if it exists in the database. This works until you have billions of URLs, at which point collision probability grows meaningfully and each insert may require multiple database round-trips. A smarter approach uses a hash-then-encode strategy: compute an MD5 of the long URL, take the first 43 bits of the hash, encode it in Base62 to get 7 characters, then verify uniqueness. MD5 is not cryptographically secure, but for this purpose we only need fast, uniformly distributed bits - security is not the concern.

The 43-bit extraction is deliberate. 62^7 = 3,521,614,606,208 distinct codes, and 2^43 = 8,796,093,022,208, so the Base62 encoding of a 43-bit integer will always produce exactly 7 characters. This makes the mapping deterministic and reversible.

For collision avoidance, a Bloom filter is the key optimization. A Bloom filter can answer “is this code definitely NOT in the database?” in O(1) with no disk access. We maintain a Bloom filter in Redis containing all allocated short codes. Before writing to Postgres, we check the Bloom filter: if it says the code is new (no membership), we proceed directly to the INSERT. The Bloom filter has a configurable false positive rate - we set it at 0.1%, meaning 0.1% of “new” codes will trigger an unnecessary DB lookup, but the DB lookup confirms the code is actually free. The Bloom filter never produces false negatives, so we never skip a required collision check.

# Shortener Service - core code generation logic
import hashlib
import base64
import string
import redis

BASE62_CHARS = string.digits + string.ascii_uppercase + string.ascii_lowercase

def int_to_base62(n: int, length: int = 7) -> str:
    """Convert integer to base62 string of fixed length."""
    result = []
    while n > 0:
        result.append(BASE62_CHARS[n % 62])
        n //= 62
    # Pad to desired length
    while len(result) < length:
        result.append(BASE62_CHARS[0])
    return ''.join(reversed(result))

def url_to_short_code(long_url: str) -> str:
    """Deterministic hash-based short code generation."""
    # MD5 is fast and distributes uniformly - not used for security
    digest = hashlib.md5(long_url.encode()).digest()
    # Take first 43 bits (5 bytes + 3 extra bits)
    value = int.from_bytes(digest[:6], 'big') >> 5  # 48-5 = 43 bits
    return int_to_base62(value & ((1 << 43) - 1))

def generate_short_code(long_url: str, redis_client: redis.Redis, max_retries: int = 5) -> str:
    """Generate a unique short code, handling collisions with salted rehash."""
    for attempt in range(max_retries):
        # Salt the URL with attempt number on collisions
        candidate_url = long_url if attempt == 0 else f"{long_url}:{attempt}"
        code = url_to_short_code(candidate_url)

        # Bloom filter check first - avoids most DB round-trips
        if not redis_client.execute_command('BF.EXISTS', 'url:bloom', code):
            return code  # Bloom says definitely not taken

        # Bloom false positive - confirm with DB (rare, ~0.1% of cases)
        existing = redis_client.get(f"url:{code}")
        if existing is None:
            return code  # DB confirms it's free despite Bloom hit

        # True collision - try next salted hash
        continue

    raise RuntimeError(f"Could not generate unique code after {max_retries} attempts")

-- Shortener Service write transaction
-- Insert the URL mapping atomically
BEGIN;

INSERT INTO urls (short_code, long_url, user_id, created_at, expires_at, custom_slug)
VALUES ($1, $2, $3, NOW(), $4, $5)
ON CONFLICT (short_code) DO NOTHING
RETURNING id, short_code;

-- If INSERT returned a row, we own it. Populate Redis and Bloom filter.
-- These run in application code after the INSERT confirms.
COMMIT;

After the Postgres write succeeds, the service calls:

# Populate Redis cache for the new URL
SET url:{short_code} {long_url} EX 86400

# Register in Bloom filter to prevent future collision re-checks
BF.ADD url:bloom {short_code}

The Redirect Service

The Redirect Service has exactly one job: look up a short code in Redis and return an HTTP redirect response as fast as possible. It is stateless, runs as dozens of identical Go processes behind a load balancer, and should never touch Postgres on the hot path.

// Redirect Service - core handler (Go)
package main

import (
    "context"
    "fmt"
    "net/http"
    "time"

    "github.com/redis/go-redis/v9"
)

type RedirectHandler struct {
    redis    *redis.ClusterClient
    db       URLRepository
    producer KafkaProducer
}

func (h *RedirectHandler) Handle(w http.ResponseWriter, r *http.Request) {
    shortCode := r.URL.Path[1:] // strip leading /

    // Primary lookup: Redis cache (sub-millisecond)
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Millisecond)
    defer cancel()

    longURL, err := h.redis.Get(ctx, "url:"+shortCode).Result()
    if err == redis.Nil {
        // Cache miss - fall back to DB and repopulate cache
        longURL, err = h.db.GetByShortCode(r.Context(), shortCode)
        if err != nil {
            http.NotFound(w, r)
            return
        }
        // Async cache repopulation - don't block the redirect
        go h.redis.Set(context.Background(), "url:"+shortCode, longURL, 24*time.Hour)
    } else if err != nil {
        // Redis error - degrade gracefully to DB
        longURL, err = h.db.GetByShortCode(r.Context(), shortCode)
        if err != nil {
            http.Error(w, "Service temporarily unavailable", http.StatusServiceUnavailable)
            return
        }
    }

    // Emit click event asynchronously - never blocks redirect
    go h.emitClickEvent(shortCode, r)

    // 302 for trackable links, 301 for permanent/SEO-safe
    // We use 302 by default so browsers don't cache the redirect
    http.Redirect(w, r, longURL, http.StatusFound)
}

func (h *RedirectHandler) emitClickEvent(shortCode string, r *http.Request) {
    event := ClickEvent{
        ShortCode: shortCode,
        Timestamp: time.Now().UnixMilli(),
        IP:        extractIP(r),
        UserAgent: r.UserAgent(),
        Referrer:  r.Referer(),
    }
    // Fire-and-forget to Kafka
    h.producer.Publish("click.events", event)
}

Hot URL Detection

URLs that go viral create a pathological case: a single short code may generate 50,000 requests per second, all hitting the same Redis key. A standard 16-shard Redis cluster routes all requests for that key to a single shard - potentially overwhelming it.

The solution is local in-process caching for hot URLs. The Redirect Service maintains a small LRU cache in process memory (not Redis) for the 1,000 most-recently accessed short codes. This cache has a 1-second TTL. For a URL getting 50,000 RPS across 50 redirect service instances, each instance sees 1,000 RPS for that code and serves 999 of them from process memory, with only 1 per second going to Redis. Redis traffic for hot URLs drops by 99.9%.

// Hot URL in-process cache with TinyLFU eviction
type HotURLCache struct {
    cache *ristretto.Cache
}

func NewHotURLCache() *HotURLCache {
    c, _ := ristretto.NewCache(&ristretto.Config{
        NumCounters: 1e6,      // Track frequency for 1M items
        MaxCost:     10 << 20, // 10 MB max memory
        BufferItems: 64,
    })
    return &HotURLCache{cache: c}
}

func (h *HotURLCache) Get(code string) (string, bool) {
    val, found := h.cache.Get(code)
    if !found {
        return "", false
    }
    return val.(string), true
}

func (h *HotURLCache) Set(code string, url string) {
    // Cost is proportional to URL length; TTL is 1 second
    h.cache.SetWithTTL(code, url, int64(len(url)), time.Second)
}

The Analytics Pipeline

The Analytics Pipeline is the system that lets URL creators answer “how many times was my link clicked today, and from where?” without the analytics ever touching the redirect hot path.

The pipeline is a textbook example of write-path decoupling: the Redirect Service publishes a click.events Kafka message and immediately returns the HTTP redirect. The message flows asynchronously through a stream processor (Apache Flink or Spark Structured Streaming) that enriches the raw click event with geo-IP data (country/city from IP), device type parsed from the user agent string, and bot detection signals. Enriched events are written in micro-batches to ClickHouse, a columnar OLAP database optimized for high-write-throughput time-series aggregation queries.

-- ClickHouse schema for click events
-- Optimized for time-range and per-short-code aggregation
CREATE TABLE click_events
(
    short_code   LowCardinality(String),
    clicked_at   DateTime64(3),         -- millisecond precision
    country_code LowCardinality(String),
    device_type  LowCardinality(String), -- mobile, desktop, tablet, bot
    referrer     String,
    ip_hash      UInt64                  -- hashed for privacy
)
ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(clicked_at)
ORDER BY (short_code, clicked_at)
SETTINGS index_granularity = 8192;

-- Materialized view for hourly aggregates (pre-computed)
CREATE MATERIALIZED VIEW click_hourly_mv
ENGINE = SummingMergeTree()
PARTITION BY toYYYYMMDD(hour)
ORDER BY (short_code, hour, country_code, device_type)
AS SELECT
    short_code,
    toStartOfHour(clicked_at) AS hour,
    country_code,
    device_type,
    count() AS click_count
FROM click_events
GROUP BY short_code, hour, country_code, device_type;

For the analytics API, queries against ClickHouse with proper partitioning and the materialized view return in milliseconds even over billions of rows, because ClickHouse only scans the relevant date partitions and aggregates from pre-rolled-up data.

Data Model

-- Primary URL mapping table
CREATE TABLE urls (
    id           BIGSERIAL PRIMARY KEY,
    short_code   VARCHAR(12) NOT NULL,     -- 7 chars typical, up to 12 for custom slugs
    long_url     TEXT NOT NULL,
    user_id      BIGINT REFERENCES users(id) ON DELETE SET NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at   TIMESTAMPTZ,               -- NULL means never expires
    custom_slug  BOOLEAN NOT NULL DEFAULT FALSE,
    click_count  BIGINT NOT NULL DEFAULT 0, -- approximate, updated asynchronously
    is_active    BOOLEAN NOT NULL DEFAULT TRUE,
    CONSTRAINT urls_short_code_unique UNIQUE (short_code)
);

-- Index for fast lookups by short code (primary access pattern)
CREATE INDEX CONCURRENTLY idx_urls_short_code ON urls (short_code)
    WHERE is_active = TRUE;

-- Index for user's links dashboard
CREATE INDEX CONCURRENTLY idx_urls_user_id ON urls (user_id, created_at DESC)
    WHERE user_id IS NOT NULL;

-- Index for expiry sweep job
CREATE INDEX CONCURRENTLY idx_urls_expires_at ON urls (expires_at)
    WHERE expires_at IS NOT NULL AND is_active = TRUE;

-- Users table for authenticated creators
CREATE TABLE users (
    id           BIGSERIAL PRIMARY KEY,
    email        VARCHAR(255) UNIQUE NOT NULL,
    plan         VARCHAR(20) NOT NULL DEFAULT 'free', -- free, pro, enterprise
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    api_key_hash VARCHAR(64)  -- SHA-256 of their API key
);

CREATE INDEX idx_users_api_key ON users (api_key_hash);

The partitioning story for the urls table: at 10 billion rows, a single Postgres table with a B-tree index on short_code remains fast because the index is essentially a lookup against a uniformly distributed key - the B-tree depth stays at about 5 levels regardless of row count for this key distribution. We do not need to shard the URL store until we exceed approximately 5 TB, which at 200 bytes per URL means about 25 billion URLs.

When sharding becomes necessary, the natural shard key is short_code itself - a consistent hash of the code routes each URL to a specific shard, and redirects always go to the correct shard without cross-shard joins.

Key Algorithms and Protocols

Base62 Encoding

Base62 encoding maps an integer to a string using 62 characters: digits 0-9, uppercase A-Z, and lowercase a-z. It is to URLs what hexadecimal is to memory addresses - a compact human-readable representation of a number.

# Base62 encode/decode - complete implementation
BASE62 = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'

def encode(n: int) -> str:
    """Encode non-negative integer to base62 string."""
    if n == 0:
        return BASE62[0]
    chars = []
    while n:
        chars.append(BASE62[n % 62])
        n //= 62
    return ''.join(reversed(chars))

def decode(s: str) -> int:
    """Decode base62 string to integer."""
    result = 0
    for char in s:
        result = result * 62 + BASE62.index(char)
    return result

# Verify: encode(3521614606207) == last valid 7-char code
assert len(encode(3521614606207)) == 7
assert decode(encode(42_000_000)) == 42_000_000

Time complexity: O(log_62(n)) for both encode and decode - constant for 7-character codes. Space complexity: O(k) where k is the code length.

Collision Avoidance with Bloom Filter

A Bloom filter is like the “did you try looking in the index card catalog?” check at a library - much faster than searching the full stacks, with the small possibility of a false lead. It is a probabilistic data structure using k hash functions and a bit array: to add an element, set k bits; to check membership, verify all k bits are set.

# Redis Bloom Filter usage (requires RedisBloom module)
import redis

def setup_bloom_filter(redis_client: redis.Redis, capacity: int = 10_000_000_000, error_rate: float = 0.001):
    """
    Initialize Bloom filter for 10 billion URLs with 0.1% false positive rate.
    Memory: ~17.9 GB (14.4 bits per element at 0.1% FPR)
    """
    redis_client.execute_command(
        'BF.RESERVE', 'url:bloom',
        error_rate,  # 0.001 = 0.1% false positive rate
        capacity,    # 10 billion expected elements
        'EXPANSION', 2  # expand when full
    )

def check_and_register(redis_client: redis.Redis, short_code: str) -> bool:
    """
    Returns True if code was newly registered (not seen before).
    Returns False if code likely already exists (may be false positive).
    Uses a pipeline to minimize round-trips.
    """
    pipe = redis_client.pipeline()
    pipe.execute_command('BF.EXISTS', 'url:bloom', short_code)
    pipe.execute_command('BF.ADD', 'url:bloom', short_code)
    exists, _ = pipe.execute()
    return not exists

The key property: a Bloom filter has zero false negatives. If it says a code is new, it is guaranteed new. The 0.1% false positive rate means 0.1% of codes that appear to be taken will require a DB confirmation check - at 1,000 writes/second, that is just 1 unnecessary DB lookup per second.

301 vs 302 Redirects

This protocol decision has significant operational consequences. An HTTP 301 response is “permanent” - browsers and CDNs cache it indefinitely. Once a user’s browser sees a 301 for /abc123 -> https://example.com/long, it will redirect directly to the destination forever without checking your service again. That means:

• Analytics tracking is broken for returning users from the same browser
• You cannot change the destination URL - the browser ignores your service
• You cannot disable a link for users who have cached the 301

An HTTP 302 response is “temporary” - browsers do not cache it by default. Every click hits your service, enabling analytics and mutability. The cost is a network round-trip to your service on every click - which is exactly what we have engineered the system to handle efficiently.

Use 301 only for permanent links where SEO authority transfer matters more than analytics. Use 302 (or 307 for method-preserving) for everything else.

Scaling and Performance

Capacity Estimation - 100K RPS Redirect System:

Given:
  - 100,000 redirects per second (sustained)
  - 1,000 new links per second (writes)
  - 200 bytes per URL record
  - 100 bytes per click event
  - 90-day analytics retention

Storage (URLs):
  10 billion URLs * 200 bytes = 2 TB total URL storage

Storage (Analytics - ClickHouse):
  100,000 RPS * 100 bytes * 90 days * 86400 sec/day
  = 100,000 * 100 * 7,776,000
  = 77.76 TB raw (before ClickHouse compression ~5x)
  = ~15 TB compressed

Redis Cache:
  Assuming 100 million hot URLs in cache
  100M * (12 bytes key + 200 bytes URL + 20 bytes overhead) = ~23 GB
  With Redis cluster at 25 GB per node: 2 nodes (plus replicas)

Bandwidth (Outbound):
  100,000 RPS * ~500 bytes (302 redirect response) = 50 MB/s
  CDN absorbs ~80% -> 10 MB/s origin bandwidth

Redirect Service Compute:
  Each instance handles ~2,000 RPS (Go, mostly IO-bound)
  100,000 RPS / 2,000 RPS per instance = 50 instances
  With 2x headroom: 100 instances of c5.xlarge

The scaling story has three distinct phases. At 10,000 RPS, a single Postgres read replica and a Redis primary handle everything. At 100,000 RPS, CDN edge caching becomes mandatory - the CDN absorbs 70-80% of traffic for popular links at the PoP closest to the user, with sub-millisecond responses. At 1,000,000 RPS, we add local in-process caching in each Redirect Service instance (using Ristretto or Caffeine-inspired eviction), reducing Redis calls by an order of magnitude for hot links.

The write path never becomes the bottleneck because 1,000 writes/second is trivially handled by a single Postgres primary even at high concurrency, and the Bloom filter eliminates most collision-checking round-trips.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Redis primary crash	Redis Sentinel health check (2s)	Cache miss surge, DB load spike	Automatic failover to replica; Sentinel promotes in ~10s
Postgres primary crash	PgBouncer health probe (1s)	Write path down; read path degrades to replica	Promote read replica; writes resume after ~30s
Kafka broker failure	Consumer lag alert (threshold: 10K events)	Click events buffered, analytics delayed	Kafka replication auto-recovers; events process once broker restores
ClickHouse slow	Analytics query timeout (5s)	Dashboard shows stale data	Query hits pre-aggregated materialized view; full scan as fallback
CDN PoP failure	Edge health probe, BGP re-routing	Traffic reroutes to next PoP	Automatic BGP failover; increased latency from farther PoP
Short code collision storm	Alert on retry_count > 1 in Shortener	Write latency spikes	Bloom filter expansion; investigation of hash function distribution

Comparison of Approaches

Approach	Redirect Latency	Consistency	Complexity	Best Fit
Direct Postgres (no cache)	20-100ms	Strong	Low	Under 1,000 RPS
Redis cache-aside	1-5ms (hit), 20ms (miss)	Eventual (seconds)	Medium	1K - 100K RPS
CDN edge caching	0.5-2ms (hit), 5ms (miss)	Eventual (minutes)	Medium-High	100K+ RPS, immutable links
In-process LRU + Redis	0.1-1ms (hit), 3ms (Redis), 20ms (DB)	Eventual (seconds)	High	1M+ RPS, hot URL dominated
Cloudflare Workers + KV	0.1-1ms	Eventual (5-15s propagation)	Low-Medium	Global edge-native apps
Pre-computed redirect rules	Sub-millisecond	Strong (deploy-time)	Very High	Static marketing links only

The right choice for a system targeting 100K RPS with mutable links and analytics is Redis cache-aside backed by CDN caching for truly hot links. We use CDN caching for links that have already proven popular (automatically promoted by request frequency), and Redis for everything else. This gives us the latency of CDN on the hot path with the flexibility of server-side logic (analytics, link updates, expiry) on the cold path.

In-process caching is worth adding once a link regularly receives more than 5,000 RPS on a single service instance - at that point Redis itself becomes the bottleneck, and the process-level cache pays for itself immediately.

Key Takeaways

• Base62 encoding maps a 43-bit integer to a 7-character string, providing 3.5 trillion unique short codes from a deterministic, collision-rare hash of the original URL.
• Bloom filters eliminate 99.9% of database round-trips during collision detection by storing all allocated codes in a probabilistic membership structure that never produces false negatives.
• Cache-aside pattern means the application is responsible for populating the cache on DB reads, giving explicit control over TTL and eviction without complex cache-invalidation machinery.
• 301 vs 302 redirects is a product decision masquerading as a technical one: 301 breaks analytics and link mutability forever in the user’s browser, while 302 preserves server-side control at the cost of one network hop per click.
• Analytics pipeline decoupling is non-negotiable - synchronous analytics writes during redirect would make your latency proportional to your analytics database’s write performance, not your cache’s read performance.
• Hot URL detection via in-process LRU cache prevents a single viral link from overwhelming a Redis shard by serving the majority of its traffic from process memory with a 1-second TTL.
• Geo-routing at the CDN layer means a user in Tokyo hitting a link shared on US Twitter gets a redirect from a Tokyo PoP, not from a US data center - saving 150ms of round-trip latency.
• Expiry logic belongs in application code, not Redis TTL - Redis TTL is for cache eviction, not business-logic expiration, and conflating them causes links to silently stop working without proper error handling.

The counter-intuitive lesson from this design is that the technical core - generating a short code - is the trivial part. The entire system complexity exists to answer a single question at 100,000 times per second: “what is the long URL for this code?” Everything else, from Bloom filters to CDN caching to analytics pipeline decoupling, is in service of making that lookup fast and cheap at scale.

Frequently Asked Questions

Q: Why not just use auto-increment IDs as short codes instead of hashing?

A: Auto-increment IDs converted to Base62 work fine, but they are sequential and therefore guessable - an attacker can enumerate all your links by incrementing the ID. They also require a centralized sequence generator, which becomes a coordination bottleneck in a multi-region setup. Hash-based codes are non-sequential and deterministic: the same URL always produces the same code, enabling deduplication without a DB lookup.

Q: Why not store links in a distributed key-value store like DynamoDB instead of Postgres?

A: DynamoDB works well for this access pattern (single-key lookup), and many URL shorteners use it. The tradeoff is cost at scale and the loss of relational features. Postgres with proper indexing handles 50,000 read QPS per instance comfortably, and with a Redis cache in front, DB load is minimal. DynamoDB makes more sense when you need multi-region active-active writes or are starting with DynamoDB operational expertise.

Q: How do you handle malicious URLs (phishing, malware)?

A: URL safety checking is a separate concern from shortening and should be done asynchronously after creation. At creation time, you can do a fast lookup against a locally cached Google Safe Browsing blocklist. For comprehensive protection, run an async crawler that checks each new URL against full Safe Browsing and VirusTotal APIs within 30 seconds of creation. If a URL is found malicious, mark it inactive in Postgres and set a tombstone in Redis. This keeps the creation path fast while still preventing malicious links from going live.

Q: Why 302 instead of 307 for redirects?

A: 307 is “Temporary Redirect” that preserves the HTTP method - a POST to a 307 URL will POST to the destination. For URL shorteners, you almost never want to forward POST bodies to the destination, so 302 (which degrades POST to GET in practice across browsers) is the correct choice. 307 is appropriate for API proxies where method preservation matters.

Q: How do you handle custom slugs that conflict with system routes?

A: Maintain a blocklist of reserved paths (/api, /health, /admin, /login, etc.) and validate custom slugs against it at creation time. Return a clear 409 Conflict error with the message “this slug is reserved.” The blocklist should also include common typosquatting targets and offensive terms, managed as a database table updated by a content moderation process.

Q: What is the right Redis eviction policy for URL cache?

A: Use allkeys-lru as the Redis eviction policy for URL caches. This evicts the least-recently-used keys across all stored keys when memory is full, which naturally preserves hot links (recently accessed) and evicts cold links. Avoid volatile-lru (only evicts keys with TTL set) because it can fill memory with TTL-less keys that never evict. With allkeys-lru, Redis effectively acts as a self-managing hot URL cache.

Interview Questions

Q: Design a URL shortener that handles 100,000 redirects per second with sub-10ms latency.

Expected depth: Discuss the read-to-write ratio (1000:1) and why this drives a cache-first architecture. Explain Base62 encoding and the Bloom filter for collision avoidance. Detail the CDN-to-Redis-to-Postgres fallback chain. Address the 301 vs 302 decision and its analytics implications. Explain how analytics are decoupled from the redirect path using Kafka.

Q: How would you ensure short codes are unique across a distributed system without a central coordinator?

Expected depth: Compare hash-based generation (deterministic, possible collisions) vs pre-allocated ID blocks (no collisions, requires coordination) vs UUID-to-Base62 (no collisions, no coordination, longer codes). Discuss Bloom filter efficiency for collision pre-screening. Explain the ON CONFLICT DO NOTHING Postgres pattern as the atomic collision resolver of last resort.

Q: A single short URL is getting 500,000 requests per second. How does your system handle this?

Expected depth: Identify the hot URL as a Redis single-key hotspot. Explain local in-process caching with 1-second TTL as the solution. Discuss CDN edge caching as an upstream layer. Note that with 100 Redirect Service instances each caching locally, Redis traffic for the hot URL drops to 100 requests per second total. Discuss the 1-second staleness window for link updates as an acceptable tradeoff.

Q: How would you add support for link expiry with 5-second precision?

Expected depth: Discuss why Redis TTL is insufficient (it evicts from cache, not logically expires the link). Explain storing expires_at timestamp in Postgres and checking it in the Redirect Service. For precise expiry, use a cron job or Redis ZSET expiry queue that updates the is_active flag and deletes the Redis key at expiry time. Note that cache entries may serve briefly after expiry - acceptable with 5-10 second TTL refresh.

Q: Walk through the analytics pipeline from click to dashboard query.

Expected depth: Trace the event: click -> Redirect Service emits Kafka message fire-and-forget -> Kafka persists event -> Flink consumer enriches with geo-IP and user-agent parsing -> micro-batch write to ClickHouse -> ClickHouse materialized view pre-aggregates by hour/day -> dashboard API queries materialized view. Discuss Kafka as durability buffer, ClickHouse partition pruning, and the ~60-second staleness acceptable for analytics.