Object vs Block Storage: Choosing the Right Storage for Your Data

Your team stores user-uploaded images directly on the application server’s filesystem. It works fine with one server. You add a second server. Now half the image requests fail because the image is on the other server. You move images to a shared NFS mount. It works until the NFS server becomes a bottleneck. You move to S3. Problem solved.

But then you try to use S3 as a database. You store JSON files and update them in place. Concurrent updates corrupt data. S3 is not a filesystem. It is object storage. Understanding the difference between storage types prevents these mistakes.

The three storage types

Block storage

Block storage presents raw storage as a device with fixed-size blocks (typically 512 bytes or 4KB). The operating system formats it with a filesystem (ext4, NTFS, XFS) and treats it like a local disk.

Characteristics:

• Low latency (microseconds to milliseconds)
• Random read/write access
• Requires a filesystem on top
• Attached to a single instance at a time (usually)
• Supports in-place updates

Examples: AWS EBS, Google Persistent Disk, Azure Managed Disks, physical SSDs and HDDs

Use for: Databases (PostgreSQL, MySQL data files), operating system volumes, applications that need a local filesystem

File storage (network file system)

File storage presents a filesystem over a network. Multiple instances can mount the same filesystem simultaneously. Files are organized in directories.

Characteristics:

• Higher latency than block storage (network overhead)
• Shared access from multiple instances
• Standard filesystem semantics (directories, permissions)
• Supports in-place updates

Examples: AWS EFS, Google Filestore, Azure Files, NFS, SMB/CIFS

Use for: Shared configuration files, content management systems that need shared file access, legacy applications that expect a filesystem

Object storage

Object storage stores data as objects (blobs) with a flat namespace (no directories, just key-value). Each object has a key (path-like string), data, and metadata. Objects are accessed via HTTP APIs.

Characteristics:

• Higher latency than block storage (HTTP overhead, 10-100ms)
• Massively scalable (exabytes)
• Accessible from anywhere via HTTP
• No in-place updates (must replace the entire object)
• Eventual consistency (though S3 now offers strong consistency)
• Cheap at scale

Examples: AWS S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, MinIO

Use for: Images, videos, backups, logs, static website assets, data lake storage, ML training data

graph TB
subgraph block["Block Storage"]
  B1["Raw blocks
512B or 4KB"]
  B2["Filesystem
ext4, NTFS, XFS"]
  B3["Application
Database, OS"]
  B1 --> B2 --> B3
end

subgraph file["File Storage"]
  F1["Network filesystem
NFS, SMB"]
  F2["Multiple instances
shared access"]
  F1 --> F2
end

subgraph object["Object Storage"]
  O1["HTTP API
GET PUT DELETE"]
  O2["Flat namespace
bucket/key"]
  O3["Any client
anywhere"]
  O1 --> O2
  O2 --> O3
end

style B1 fill:#EEEDFE,stroke:#534AB7,color:#3C3489
style B2 fill:#EEEDFE,stroke:#534AB7,color:#3C3489
style B3 fill:#EEEDFE,stroke:#534AB7,color:#3C3489
style F1 fill:#FAEEDA,stroke:#854F0B,color:#633806
style F2 fill:#FAEEDA,stroke:#854F0B,color:#633806
style O1 fill:#E1F5EE,stroke:#0F6E56,color:#085041
style O2 fill:#E1F5EE,stroke:#0F6E56,color:#085041
style O3 fill:#E1F5EE,stroke:#0F6E56,color:#085041

Object storage deep dive: S3

S3 is the canonical object storage service. Understanding its semantics prevents common mistakes.

The flat namespace

S3 has no real directories. s3://my-bucket/images/users/123/avatar.jpg looks like a path, but images/users/123/ is just a prefix of the key. There are no directory objects. The console shows a folder-like UI, but it is an illusion.

This matters for operations: listing all objects with a prefix is efficient. Listing all objects in a “directory” is just listing by prefix. But there is no atomic “rename directory” operation - you must copy all objects and delete the originals.

No in-place updates

You cannot update part of an S3 object. To change a file, you must upload the entire new version. This makes S3 unsuitable for databases or any data that needs partial updates.

For large files that change frequently, consider whether object storage is the right choice. For append-only data (logs), you can use S3 Select or partition logs into time-based objects.

Multipart upload

For large files (over 100MB), use multipart upload. Split the file into parts, upload them in parallel, then complete the upload. Benefits: parallel upload (faster), resume on failure (only re-upload failed parts), no size limit.

Presigned URLs

Instead of proxying file downloads through your application server, generate a presigned URL that gives the client temporary direct access to S3. The client downloads directly from S3, bypassing your server entirely. This reduces your bandwidth costs and server load.

GET https://my-bucket.s3.amazonaws.com/images/avatar.jpg
  ?X-Amz-Algorithm=AWS4-HMAC-SHA256
  &X-Amz-Expires=3600
  &X-Amz-Signature=...

The URL is valid for a specified duration (e.g., 1 hour). After that, it returns 403.

S3 storage classes

S3 has multiple storage classes with different cost and retrieval time tradeoffs:

• S3 Standard - Frequent access. Highest cost, lowest latency.
• S3 Intelligent-Tiering - Automatically moves objects between tiers based on access patterns.
• S3 Standard-IA - Infrequent access. Lower storage cost, retrieval fee.
• S3 Glacier - Archival. Very low storage cost, retrieval takes minutes to hours.
• S3 Glacier Deep Archive - Cheapest. Retrieval takes 12 hours.

Use lifecycle policies to automatically transition objects to cheaper tiers as they age.

Where it breaks or gets interesting

The consistency model

S3 now offers strong read-after-write consistency for all operations (as of December 2020). Before that, it was eventually consistent for overwrite PUTs and DELETEs. If you have old code that works around eventual consistency (retry loops, delays), you can simplify it.

The small file problem

S3 charges per request (PUT, GET, LIST). Storing millions of tiny files (1KB each) is expensive in request costs and slow to list. For small files, consider aggregating them into larger objects (e.g., store a day’s worth of log lines in one gzipped file) or using a different storage system.

Block storage for databases: IOPS matter

Databases are IOPS-intensive. AWS EBS has different volume types with different IOPS characteristics:

• gp3 - General purpose SSD. Up to 16,000 IOPS. Good for most databases.
• io2 - Provisioned IOPS SSD. Up to 64,000 IOPS. For high-performance databases.
• st1 - Throughput-optimized HDD. High throughput, low IOPS. For sequential workloads (data warehouses, log processing).

Choosing the wrong EBS type is a common cause of database performance problems.

Object storage as a data lake

S3 is the foundation of most data lakes. Raw data is stored in S3 in open formats (Parquet, ORC, JSON). Query engines (Athena, Spark, Presto) read directly from S3. This separates storage from compute: you pay for storage separately from query processing, and you can use different query engines on the same data.

graph LR
subgraph use_cases["Storage Type by Use Case"]
  DB["Database files
PostgreSQL, MySQL"] --> BLOCK["Block Storage
EBS, Persistent Disk"]
  OS["Operating system
Boot volume"] --> BLOCK
  IMG["User images
videos, files"] --> OBJ["Object Storage
S3, GCS"]
  LOG["Logs, backups
ML training data"] --> OBJ
  SHARE["Shared config
CMS files"] --> FILE["File Storage
EFS, NFS"]
  LEGACY["Legacy apps
need filesystem"] --> FILE
end

style BLOCK fill:#EEEDFE,stroke:#534AB7,color:#3C3489
style OBJ fill:#E1F5EE,stroke:#0F6E56,color:#085041
style FILE fill:#FAEEDA,stroke:#854F0B,color:#633806

Real-world systems

Netflix - Stores all video content in S3. Uses S3 as the source of truth for the content pipeline. Transcoding jobs read from S3 and write back to S3. CDN pulls from S3 for delivery.

Dropbox - Originally stored files on S3. Later built their own object storage (Magic Pocket) for cost reasons. The interface is still S3-compatible.

Airbnb - User photos stored in S3. Served via CloudFront CDN. Presigned URLs for direct upload from the browser.

Stripe - Uses S3 for log storage and data lake. Logs are written to S3 in Parquet format and queried with Athena.

GitHub - Git objects (blobs, trees, commits) stored in a custom object storage system. Large file storage (Git LFS) uses S3.

How to apply it in practice

The decision matrix

Direct upload pattern

For user file uploads, never route through your application server:

1. Client requests a presigned upload URL from your API
2. API generates a presigned S3 PUT URL (valid for 15 minutes)
3. Client uploads directly to S3 using the presigned URL
4. S3 notifies your API via S3 Event Notification or the client notifies your API after upload
5. Your API records the S3 key in the database

This eliminates your server as a bottleneck for file uploads.

FAQ

Q: Can you use S3 as a database?

Not directly. S3 lacks the features databases need: transactions, indexes, partial updates, and low-latency random access. But S3 is used as the storage layer for databases like DynamoDB (internally) and as the foundation for data lake query engines (Athena, Spark). For structured data that needs SQL queries, use S3 + Athena or S3 + Redshift Spectrum. For key-value access, use DynamoDB. For relational data, use RDS.

Q: When should you use EFS instead of S3?

Use EFS when your application needs filesystem semantics: directory structure, file locking, POSIX permissions, or in-place file updates. EFS is also necessary when multiple EC2 instances need to read and write the same files simultaneously. Use S3 when you need massive scale, cheap storage, HTTP access, or CDN integration. EFS is significantly more expensive than S3 for the same amount of data.

Q: How do you handle large file uploads to S3 from a browser?

Use presigned URLs for direct browser-to-S3 uploads. For large files (over 5MB), use the S3 multipart upload API: the browser requests presigned URLs for each part, uploads parts in parallel, then calls your API to complete the multipart upload. Libraries like aws-sdk handle this automatically. Set appropriate CORS headers on the S3 bucket to allow browser uploads. Use S3 Transfer Acceleration for users far from the S3 region.

Interview questions

Q1: You are building a photo sharing app. Users upload photos, which are displayed to other users. Design the storage architecture.

Strong answer: Store photos in S3. Never store them on application servers (not scalable, not durable). For uploads: generate a presigned S3 PUT URL from your API, have the client upload directly to S3. After upload, store the S3 key in your database. For display: serve photos via CloudFront CDN in front of S3. Use presigned GET URLs if photos are private (only visible to certain users). For thumbnails: use S3 Event Notifications to trigger a Lambda function that generates thumbnails and stores them back in S3. Use S3 lifecycle policies to move old photos to S3 Intelligent-Tiering to reduce costs. The result: scalable to billions of photos, cheap storage, fast delivery via CDN, no application server bottleneck.

Q2: Your PostgreSQL database is slow. Investigation shows high I/O wait. What storage-related changes would you make?

Strong answer: Check the EBS volume type and IOPS. If you are on gp2 (older general purpose), migrate to gp3 which has better baseline IOPS and allows independent IOPS and throughput configuration. Check if you are hitting the IOPS limit with CloudWatch metrics. If so, either provision more IOPS (io2 volume) or optimize the database to reduce I/O (better indexes, query optimization, connection pooling to reduce concurrent I/O). Check if the database is doing sequential scans that would benefit from a higher-throughput volume. Also check if the EBS volume is on the same physical host as the instance (EBS-optimized instances have dedicated bandwidth for EBS). For very high IOPS requirements, consider instance store (NVMe SSD directly attached to the instance) for the database data directory, with regular snapshots to EBS for durability.

Q3: How would you design a system to store and query 10TB of application logs per day?

Strong answer: Write logs to S3 in compressed, columnar format (Parquet or ORC). Partition by date and service: s3://logs/year=2024/month=01/day=15/service=api/. Use a log aggregation service (Fluentd, Logstash, Kinesis Firehose) to buffer and batch logs before writing to S3 - writing individual log lines to S3 is expensive in request costs. For querying: use Athena (serverless SQL on S3) for ad-hoc queries. Create an Athena table pointing to the S3 prefix. Queries scan only the partitions they need. For real-time alerting: stream logs to Kinesis Data Streams and process with Lambda or Kinesis Analytics. For long-term retention: use S3 lifecycle policies to move logs older than 90 days to S3 Glacier. Total cost: S3 Standard for recent logs, Glacier for archives, Athena for queries (pay per TB scanned).