Netflix AI Team Cuts Wide-Partition Read Latency from Seconds to Milliseconds by Splitting Cassandra Partitions Per ID

Netflix’s engineering team published a method for handling wide partitions in Apache Cassandra. The research work targets Netflix’s TimeSeries Abstraction, a platform for temporal event data.

TL;DR

Dynamic partitioning splits wide Cassandra partitions per TimeSeries ID, asynchronously and transparently, with no application changes.

Detection runs on the read path via byte counting and a Kafka event; splitting targets immutable partitions first.

Bloom filters (single-digit microseconds) plus a cached wide_row metadata lookup route reads to the smaller child partitions.

Checksums, retained original partitions, Data Bridge Spark checks, and shadow comparison guard correctness.

Reads improved from seconds to low double-digit milliseconds; tail latency fell to ~200 ms; 500MB+ partitions stayed available.

What is Dynamic Repartitioning?

Netflix’s TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency. It uses Apache Cassandra 4.x as the underlying storage. Cassandra was chosen for throughput, latency, cost, and operational maturity.

Time-series data is organized into partitions that group events by identifier and time range. These partitions can grow ‘wide’ as events accumulate over time. Dynamic repartitioning splits an oversized partition into smaller child partitions asynchronously. Applications keep querying the same logical partition while the storage layout evolves transparently.

Why Wide Partitions Hurt Reads

For most datasets, average read latency stays in single-digit milliseconds. When partitions grow too wide, tail read latencies rise into seconds. These slow reads can produce read timeouts. In extreme cases, clusters see Garbage Collection pauses, high CPU utilization, and thread queueing.

TimeSeries servers also handle very high read throughput, which compounds the problem. Scaling up the Cassandra cluster is always an option. But Netflix team wanted smarter alternatives than just throwing more money at the problem.

The Partitioning Strategy Behind TimeSeries

TimeSeries breaks datasets into discrete time chunks: Time Slices, time buckets, and event buckets. This lets Netflix team query and drop data by time without creating tombstones. When a namespace (dataset) is created, users specify anticipated workload characteristics. A provisioning pipeline runs Monte Carlo simulations to pick infrastructure and partition configuration.

This up-front approach falls short in three situations. Workload can be unknown or inaccurately estimated early in a project. Workload can also evolve as traffic and product requirements change. Finally, data outliers exist, where a few IDs receive far more events.

Discrete Time Slices provide a natural escape hatch for the first two cases. Each new Time Slice can use a different partitioning strategy. But manually tuning thousands of datasets is not sustainable, so automation is required.

Solution 1: Time Slice Re-Partitioning

Cassandra exposes introspection APIs such as nodetool tablehistograms for partition-size percentiles. A background worker monitors these histograms and exposes them through a Cassandra virtual table. It computes an adjustment factor when partition sizes miss a configured density. That density is often set between 2 MiB and 10 MiB, depending on workload.

For example, provisioning once selected 60-second time buckets, producing partitions under 10 KB. That over-partitioning caused high read amplification and thread queueing. The worker updates future Time Slices with a corrected strategy:

This reduced read latencies and timeouts caused by thread queueing. But it only helps when most of the table warrants re-partitioning. It does not help when only a percentage of IDs are wide. For those partial cases, Netflix team offers three options:

Do Nothing: the right approach when top-level metrics show no impact.

Partial Returns: aborts an in-flight request breaching a latency SLO, returning data collected so far.

Block IDs: an extreme step for test or spam IDs that destabilize the system.

Block IDs is applied through configuration, listing the offending TimeSeries IDs:

These options do not help when valid, important IDs grow wide. Those callers still need every event, which is where Solution 2 applies.

Solution 2: Dynamic Partitioning per ID

Dynamic partitioning is an asynchronous pipeline that splits wide partitions per TimeSeries ID. It operates at the ID level, not the table level. It has three stages: Detection, Planning & Splitting, and Serving Reads.

Detection happens on the read path. Every read tracks bytes read for a partition. When bytes exceed a configured threshold, the server emits an event to Kafka:

Here, immutable marks a partition no longer receiving writes. The version field is reserved for future use, such as invalidation. Netflix team detects on reads, not writes, because most data never needs splitting. Some reads on wide partitions stay slow for seconds until the pipeline catches up. The initial implementation targets immutable partitions to reduce complexity.

Planning reads the entire partition once to compute an accurate split plan. Checkpointing lets failed planning reads resume from the last saved point. The wide_row metadata table stores split states, checkpoints, and routing information.

Splitting delegates to a strategy such as EventBucketPartitionSplitStrategy. It assigns more event buckets to the same time bucket. For ultra-wide partitions, it caps event buckets to control read amplification. Spreading across partitions still distributes reads over Cassandra replicas.

Validating compares a pre-split checksum against a post-split checksum. A split is marked COMPLETED only when both checksums match. Netflix also tracks pre- and post-split partition sizes to confirm splits are sized well.

TimeSeries servers periodically load completed split partition-keys into in-memory Bloom filters. Every read checks the Bloom filter, which responds in single-digit microseconds. That check is small enough to be practically invisible to callers. On a hit, the server reads wide_row metadata to route the query:

That metadata read is backed by a read-through cache. The existing PartitionReader then serves reads from the smaller split partitions. Reusing the same schema minimizes code changes, and results are merged before returning. The original wide partition is never deleted, providing a safe fallback for partial failures and eventual consistency.

Netflix team also verified splits offline using Data Bridge Spark jobs. A phased rollout advanced through read modes as confidence grew. Its Comparison phase compared bytes served by the old and new read paths in shadow mode.

Solution 1 vs Solution 2: Comparison

Use Cases With Examples

Long-lived user activity logs: A profileId:123 accumulating years of playback events becomes wide. Solution 2 spreads it across event buckets, so pagination stays available. Netflix paginated 500MB+ partitions while remaining available using this path.

Device or IoT telemetry: A small set of chatty devices dominate event volume. Solution 2 isolates those hot IDs without repartitioning the whole table.

Over-provisioned new dataset: A team guesses 60-second buckets and gets sub-10 KB partitions. Solution 1 widens the interval for future Time Slices, cutting read amplification.

Latency-sensitive dashboards: A caller needs a fast response more than complete data. Partial Returns caps latency at the SLO and returns what was gathered.

Try It: Interactive Splitter Demo

The embedded simulator below models the read path end to end. Set a partition size, a detection threshold, and a target event-bucket count. Running a read walks through Detection, Planning, Splitting, Validation, and Serving. It shows the Kafka detection event and a modeled latency drop; the numbers are illustrative, not measured.

Results

Average read latency for wide partitions dropped from seconds to low double-digit milliseconds. Tail latencies fell from several seconds to around 200 ms or better. Read timeouts dropped, and Netflix team reported lower CPU utilization and minimal thread queueing. Overall, the Cassandra clusters became more stable.

For extreme wide rows, the service could paginate and query 500MB+ partitions while remaining available. Such partitions previously caused constant timeouts and unavailability blips. A paginated query returns successfully, trading elevated latency for availability:

Netflix team lists future work, including splitting mutable wide partitions and reprocessing previously failed splits. Two lessons stand out. Reducing the surface area of a complex change and deploying incrementally pays off operationally. Investing in confidence-building mechanisms is justified by the feature’s complexity, blast radius, and impact.

Check out the Technical details here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us