2

I understand that Cassandra is massively scalable, but it currently has a limitation for storing 2 billion individual pieces of information.

Now, say I want to store information in a table and I have 20 billion data points. An example might be storing multiple devices (desktop PC, mobile devices, etc.) per user, where there are over 7 billion individuals (possible users) on the planet. With multiple devices per person, it is conceivable that the data set could reach 20+ billion records.

  1. Can Cassandra handle this scenario? If possible, then how?
  2. If not, how can this scenario be handled?
2
  • World population is only 7 billion. And how would you enter the 20 billion records? Your requirement makes no sense.
    – Raedwald
    Commented Nov 25, 2014 at 11:57
  • @Raedwald Edit made. Commented Nov 26, 2014 at 10:01

1 Answer 1

7

Yes, Cassandra can store 20 billion or more individual pieces of data.

The maximum number of cells (rows x columns) in a single partition is 2 billion.

This is the limitation you alluded to, but it is more specific than your interpretation. Specifically that limit is for a single partition. Were you to insert the maximum 2 x 109 records into a partition, it would require a minimum of 10 separate partitions to, collectively, store the hypothetical 20B records. Creating 10 partitions is easy.

This is the answer to the "how" in the original question: Cassandra scales beyond this limitation when you, the application developer, split the data across multiple partitions.

In fact, a well-designed, healthy Cassandra cluster will consist of thousands or millions (or more) individual partitions. While each partition can theoretically contain a unique set of two billion data points, in practice you are unlikely to see partitions grow to be that large, and you should not design your schema with the intent to reach that limit. (After all, it is a limit and should be avoided.)

A single node (separate machine) in a Cassandra cluster can store multiple partitions, but the data for each partition must be able to reside completely within one node. That node must also perform sort operations on the partition when making changes to its data. You can probably imagine that sorting anywhere close to a billion data points will take measurable amounts of time. Instead, Cassandra intends you to scale "massively" by distributing the work by distributing the data across multiple nodes. Production clusters can easily consist of dozens, hundreds, or even thousands of individual nodes.

  1. Avoid getting anywhere close to the 2B/partition limit by splitting data across many partitions.
  2. Each node will be able to hold a finite number of partitions, based on the capacity of its disk.
  3. Avoid being limited by disk space by adding more nodes to your cluster, thus distributing the same data across more disks.
7
  • Thumb up for 'In fact, a well-designed, healthy Cassandra cluster will consist of thousands or millions (or more) individual partitions' Commented Nov 28, 2014 at 20:26
  • Thanks William for the answer but it seems still confusing as there are more other documents regarding the cassandra limitation and interpretation. So far what I understood- 1. Using partition key we can create any number of partition 2. In each partition(Each row), it can have 2B column values. Commented Nov 30, 2014 at 3:38
  • Partition != row. Columns are stored in rows. Rows are stored in partitions. Partitions are stored in nodes. One or more nodes make up a cluster. All of those terms are different things. The count of all columns of all rows in one partition must be <= 2 billion. (Tip: make sure you're reading documents that pertain to recent versions of Cassandra; some of the terminology changed meaning a little bit when they introduced CQL.) Commented Nov 30, 2014 at 3:42
  • @William would you check the following link ?<br> link. Actually a straight forward example would be great to understand and interpret. Commented Nov 30, 2014 at 3:43
  • See slide #48 of this presentation Commented Nov 30, 2014 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.