Sharding & Partitioning
The problem
Eventually a dataset or workload grows too large for any single machine to hold or serve, no matter how powerful — sharding splits data across many machines so each holds only a slice, letting total capacity grow by adding machines instead of upgrading one.
Why now
Modular arithmetic already introduced the exact mechanism (hashing a key into one of N buckets, and consistent hashing's ring for minimizing remapping) that determines which shard a piece of data belongs to; NoSQL and data modeling tradeoffs already established that access-pattern-driven design matters, which sharding strategy directly depends on.
Mental model
Sharding is hashing, made physical: instead of a key mapping to a bucket in a hash table, it maps to a specific machine holding that slice of data — and consistent hashing (already covered) is exactly the technique used so that adding or removing a shard remaps only a small fraction of keys instead of nearly all of them.
Requires
- Modular ArithmeticMathematical Thinking
Basic sharding is literally hash(key) mod N applied to select a machine instead of an array bucket, and consistent hashing — the refinement that avoids massive remapping on resize — is the exact ring-based technique already introduced there.
- NoSQL & Data Modeling TradeoffsDatabase Engineering
A good sharding key depends entirely on the system's access patterns, the same access-pattern-first thinking nosql-and-data-modeling-tradeoffs already established as central to non-relational data design.
Used in
- Horizontally scaled databases (Cassandra, DynamoDB, sharded MongoDB) distributing data across many nodes
- Choosing a shard key that avoids hot spots (one shard receiving disproportionate traffic)
- Resharding — the operationally difficult process of changing shard count as data grows
- Consistent hashing in distributed caches, directly reusing the modular-arithmetic mechanism
Projects
- Implement basic hash-based sharding (hash(key) mod N) across simulated shards, then implement consistent hashing and compare how many keys remap when a shard is added
- Design a shard key for a real dataset (e.g. user data) and identify a naive choice that would create a hot spot, versus a better one
Examples
- Sharding users by hash(user_id) mod 4 spreads them across 4 machines, but adding a 5th machine with plain mod remaps almost every user — consistent hashing avoids this
- Sharding by signup_date instead of a well-distributed key can create a hot spot where all of today's active users hit one shard
Resources
- Consistent hashing — Toptalarticle
Mastery checklist
- I can implement basic hash-based sharding and explain its resizing problem
- I can implement or explain consistent hashing and why it minimizes remapping on resize
- I can identify a shard key choice likely to create a hot spot
- I can explain the operational challenge of resharding a live system