Case Study — Discord's Message Store

The workload shape

Writes never stop, and nothing ages out. January 2017: past 120 million messages a day. By March 2023: trillions of messages stored. There is no archival tier — all of it stays queryable.

Reads are overwhelmingly recent. Opening a channel fetches the last page. But mostly recent is not only recent: permalinks, search hits and determined scrolling reach far back, and those reads must not be pathological.

Access is always per-channel, in time order. Nothing asks this store for one user's messages across all channels. Every query is one channel, a position in time, and a direction.

That last property is the gift. When a read names one entity and a contiguous slice of time, you can lay the data out so the common read is one short sequential scan on one node.

Why a wide-column store

A wide-column store offers exactly two knobs: a partition key (which node, which physical row group) and a clustering key (sort order inside it). Per-channel and time-ordered maps onto them with nothing left over. Underneath sits an LSM tree — see storage-engines — so a write is a memtable append plus a sequential flush, with merge work deferred off the request path.

You can model this relationally with an index on (channel_id, message_id). What you inherit is a B-tree taking random page writes on every insert, and manual re-sharding as the scaling story. Discord left MongoDB in 2015 once 100 million messages meant the index no longer fit in RAM; their requirements were linear scalability without re-sharding, automatic failover, low maintenance and predictable latency — they alert when API p95 crosses 80 ms.

The bucket: the whole lesson in one key

The primary key is ((channel_id, bucket), message_id). The inner parentheses matter: they make (channel_id, bucket) a compound partition key, with message_id clustering inside it.

message_id is a Snowflake — a time-ordered 64-bit id — so clustering by it is clustering by time, with no separate timestamp column and no ties.

So why is bucket there? Partition by channel_id alone and the model is still correct: sorted, single node, one query per read. It fails anyway, because the partition is the unit of everything — placement, repair, compaction, and read. A busy channel's partition grows for the life of the channel. Importing their history, Discord saw warnings for partitions over 100 MB; Cassandra advertises support for 2 GB partitions, but large ones bring GC pressure and expensive maintenance.

bucket is a static time window, computed arithmetically from the message's timestamp. Discord looked at the largest channels on Discord and found that about 10 days of messages per bucket kept even those partitions comfortably under 100 MB.

The cost is real. A read derives the bucket range from now back to the channel's creation and queries partitions sequentially until it has enough messages, so a quiet channel can burn several reads to fill one page. Discord tracks empty buckets per channel and skips them next time.

This is the composite-key move from partitioning at its most instructive: key design fixes distribution and query cost simultaneously, and the window size is a genuine sizing exercise — too wide and the busiest channels blow the limit, too narrow and every read gets chatty.

What Cassandra cost them

By early 2022 the cluster was 177 nodes at roughly 4 TB each, and Discord described it as needing rising effort just to maintain rather than improve.

Hot partitions. A very high-traffic channel concentrates load on one partition and its replicas. Discord's phrasing is worth borrowing: traffic to a hot partition produced unbounded concurrency, and cascading latency in which each subsequent query was slower than the last. Consistent hashing does not help — as partitioning argues, it fixes rebalancing, not skew.

Garbage collection. Cassandra runs on the JVM, and at this heap size collection pauses land straight on user latency. The documented worst case, from 2017: a channel with millions of deleted messages and one survivor. Reading it made Cassandra scan the tombstones and triggered a 10-second stop-the-world pause.

Compaction. An LSM defers merging to a background process competing with live traffic for the same disk and CPU. Nodes fell behind, and the remedy was manual: the gossip dance — pulling a node out of rotation so it could compact untroubled, returning it to process hints, repeating until the backlog cleared. Not a bug, but the LSM tradeoff arriving as an on-call rotation.

All three produce a node that is up, passing health checks, and slow. That is gray failure from failure-models, and it is worse than a dead node because nothing evicts it.

The data service in front

Before changing databases, Discord put a tier of Rust services between the API and storage.

Its defining feature is request coalescing: when many users request the same row at the same moment, one query reaches the database and the single result fans back to every waiter. Paired with consistent hash-based routing, so every request for a channel lands on the same instance — concurrent requests must meet inside one process to be merged.

This is the thundering herd from caching, solved with single-flight, aimed at a database rather than a cache. A thousand simultaneous reads of one hot row become one read. The partition is still hot in the data model; the database never finds out.

The costs: an extra hop on every request, a new tier that can fail, and database skew converted into service skew.

The migration: same model, different engine

By 2020 every Discord database except this one ran on ScyllaDB. Messages cut over in May 2022.

ScyllaDB is a Cassandra-compatible database written in C++. No JVM, so no collector to pause. And a shard-per-core architecture: each core owns a slice of the data with its own memory and IO, so work is isolated per core rather than contending on one shared heap.

What stayed the same is the data model — partitioned by channel and bucket, clustered by Snowflake id, queried the same way. What changed is the implementation beneath it. That is the point of the case: the expensive, thought-intensive part was correct and survived a total replacement of the storage layer. What got replaced was an operational cost profile.

The migration reused the new tier. ScyllaDB's Spark-based migrator projected about three months, so Discord rewrote it in Rust on their data service library, reached up to 3.2 million messages per second, and finished in nine days.

Read the p99 column carefully. 40–125 ms is not a number, it is a spread — and the spread is GC and compaction. The variance collapsing matters as much as the median falling. Note also that the coalescing tier landed first, and ScyllaDB's team did specific work on reverse-query performance for this read path, so this is not purely an engine swap.

Not public: the current node count, the current write rate, and whether the ten-day window has been retuned since 2017. Say so rather than guessing.

What to take away

• Derive the partition key from the read pattern: one channel, time-ordered, mostly recent.
• Wide-column plus LSM suits append-heavy ordered scans; a B-tree pays random writes for the same shape.
• Bucketing turns an unbounded partition into a bounded one, at the price of multi-partition reads.
• Hot partitions are a key-design problem; coalescing requests above the database is the practical fix.
• Compaction and GC surface as p99 variance, not errors — and variance is what users feel.
• The data model outlived the database. Get the key right and the engine stays replaceable.