85

Googling for a definition either returns results for a column oriented DB or gives very vague definitions.

My understanding is that wide column stores consist of column families which consist of rows and columns. Each row within said family is stored together on disk. This sounds like how row oriented databases store their data. Which brings me to my first question:

How are wide column stores different from a regular relational DB table? This is the way I see it:

* column family        -> table
* column family column -> table column
* column family row    -> table row

This image from Database Internals simply looks like two regular tables:

Two column families, contents, and anchors

The guess I have as to what is different comes from the fact that "multi-dimensional map" is mentioned along side wide column stores. So here is my second question:

Are wide column stores sorted from left to right? Meaning, in the above example, are the rows sorted first by Row Key, then by Timestamp, and finally by Qualifier?

3
  • 1
    What is a Wide Column Database? dataversity.net/wide-column-database/# Commented May 26, 2020 at 0:33
  • 8
    I think that this suffers from the same issue as other online definitions, it's short and vague: "Its architecture uses persistent, sparse matrix, multi-dimensional mapping (row-value, column-value, and timestamp) in a tabular format meant for massive scalability (over and above the petabyte scale). Column Family stores do not follow the relational model, and they aren’t optimized for joins." I realize that this is probably enough if you already get what wide column stores are, but if you haven't, this doesn't help build the mental model out. The BigTable paper is 14 pages long, after all.
    – Moo
    Commented May 26, 2020 at 14:39
  • 3
    BTW I read the first 2 pages of the BigTable paper and it actually has one of the best explainers of a wide-column store. Would recommend anyone try it.
    – Moo
    Commented Feb 1, 2021 at 15:35

3 Answers 3

81

Let's start with the definition of a wide column database.

Its architecture uses (a) persistent, sparse matrix, multi-dimensional mapping (row-value, column-value, and timestamp) in a tabular format meant for massive scalability (over and above the petabyte scale).

A relational database is designed to maintain the relationship between the entity and the columns that describe the entity. A good example is a Customer table. The columns hold values describing the Customer's name, address, and contact information. All of this information is the same for each and every customer.

A wide column database is one type of NoSQL database.

Maybe this is a better image of four wide column databases.

Wide column databases

My understanding is that the first image at the top, the Column model, is what we called an entity/attribute/value table. It's an attribute/value table within a particular entity (column).

For Customer information, the first wide-area database example might look like this.

Customer ID    Attribute    Value
-----------    ---------    ---------------
     100001    name         John Smith
     100001    address 1    10 Victory Lane
     100001    address 3    Pittsburgh, PA  15120

Yes, we could have modeled this for a relational database. The power of the attribute/value table comes with the more unusual attributes.

Customer ID    Attribute    Value
-----------    ---------    ---------------
     100001    fav color    blue
     100001    fav shirt    golf shirt

Any attribute that a marketer can dream up can be captured and stored in an attribute/value table. Different customers can have different attributes.

The Super Column model keeps the same information in a different format.

Customer ID: 100001
Attribute    Value
---------    --------------
fav color    blue
fav shirt    golf shirt

You can have as many Super Column models as you have entities. They can be in separate NoSQL tables or put together as a Super Column family.

The Column Family and Super Column family simply gives a row id to the first two models in the picture for quicker retrieval of information.

7
  • 2
    Thanks for the comprehensive answer; I found this to be a helpful, supplemental resource to this answer.
    – sam2426679
    Commented Jun 20, 2022 at 20:03
  • Can we say that wide column is similar to how excel stores data? You can make super columns there too. Besides this feature of using smaller tables/columns as modules in larger tables I do not see any other unique feature of "wide column". Am I correct in my understanding please?
    – A B
    Commented Mar 3, 2023 at 9:59
  • 2
    I'm shocked this was accepted, it doesn't seem to address any of the questions asked. It merely described with text what is in the embedded picture, and in a very confusing way with inconsistent terminology. It seems, that in a round about way, you're saying that the way "wide column stores different from a regular relational DB table" is that... they're noSQL? That's it? "No schema, different entities can have different attributes"? "It's an attribute/value table within a particular entity (column)" does this mean the column is equivalent to an entity (row) in a rdbms? Commented Jun 20 at 20:28
  • 1
    (Continued) Are these "entities" the same as the entities in "You can have as many Super Column models as you have entities"? How is "the first image at the top" a wide column db model, and not just a regular relational db column? "The Column Family and Super Column family simply gives a row id to the first two models in the picture for quicker retrieval of information" how is information retrieved in the first two models, by column name? Commented Jun 20 at 20:32
  • 1
    (Continued) What's the practical purpose of an additional grouping? And what are the practical differences between these, rdbms's, and document stores? Is there something to the way this is organized or stored that affects how it's queried or how quickly it can be written to or read from in an advantageous way, compared to RBDMSs (and other nosql modalities)? ...These are the kind of things one might expect in an actual answer to "What exactly is a wide column store?" Commented Jun 20 at 20:35
11

Most (if not all) Wide-column stores are indeed row-oriented stores in that every parts of a record are stored together. You can see that as a 2-dimensional key-value store. The first part of the key is used to distribute the data across servers, the second part of the key lets you quickly find the data on the target server.

Wide-column stores will have different features and behaviors. However, Apache Cassandra, for example, allows you to define how the data will be sorted. Take this table for example:

| id | country | timestamp  | message |
|----+---------+------------+---------|
| 1  | US      | 2020-10-01 | "a..."  |
| 1  | JP      | 2020-11-01 | "b..."  |
| 1  | US      | 2020-09-01 | "c..."  |
| 2  | CA      | 2020-10-01 | "d..."  |
| 2  | CA      | 2019-10-01 | "e..."  |
| 2  | CA      | 2020-11-01 | "f..."  |
| 3  | GB      | 2020-09-01 | "g..."  |
| 3  | GB      | 2020-09-02 | "h..."  |
|----+---------+------------+---------|

If your partitioning key is (id) and your clustering key is (country, timestamp), the data will be stored like this:

[Key 1]
1:JP,2020-11-01,"b..." | 1:US,2020-09-01,"c..." | 1:US,2020-10-01,"a..."
[Key2]
2:CA,2019-10-01,"e..." | 2:CA,2020-10-01,"d..." | 2:CA,2020-11-01,"f..."
[Key3]
3:GB,2020-09-01,"g..." | 3:GB,2020-09-02,"h..."

Or in table form:

| id | country | timestamp  | message |
|----+---------+------------+---------|
| 1  | JP      | 2020-11-01 | "b..."  |
| 1  | US      | 2020-09-01 | "c..."  |
| 1  | US      | 2020-10-01 | "a..."  |
| 2  | CA      | 2019-10-01 | "e..."  |
| 2  | CA      | 2020-10-01 | "d..."  |
| 2  | CA      | 2020-11-01 | "f..."  |
| 3  | GB      | 2020-09-01 | "g..."  |
| 3  | GB      | 2020-09-02 | "h..."  |
|----+---------+------------+---------|

If you change the primary key (composite of partitioning and clustering key) to (id, timestamp) WITH CLUSTERING ORDER BY (timestamp DESC) (id is the partitioning key, timestamp is the clustering key in descending order), the result would be:

[Key 1]
1:US,2020-09-01,"c..." | 1:US,2020-10-01,"a..." | 1:JP,2020-11-01,"b..." 
[Key2]
2:CA,2019-10-01,"e..." | 2:CA,2020-10-01,"d..." | 2:CA,2020-11-01,"f..."
[Key3]
3:GB,2020-09-01,"g..." | 3:GB,2020-09-02,"h..."

Or in table form:

| id | country | timestamp  | message |
|----+---------+------------+---------|
| 1  | US      | 2020-09-01 | "c..."  |
| 1  | US      | 2020-10-01 | "a..."  |
| 1  | JP      | 2020-11-01 | "b..."  |
| 2  | CA      | 2019-10-01 | "e..."  |
| 2  | CA      | 2020-10-01 | "d..."  |
| 2  | CA      | 2020-11-01 | "f..."  |
| 3  | GB      | 2020-09-01 | "g..."  |
| 3  | GB      | 2020-09-02 | "h..."  |
|----+---------+------------+---------|
0

Say you have a column family A which contains column X, Y, Z. If you flatten out the schema, is it not the same as a traditional relational database with three columns A.X, A.Y, A.Z?

Cassandra: The Definitive Guide says the main difference is in the storage.

The code below defines a table with 4 columns in Cassandra, in which each row is uniquely identified by the primary key (last_name, first_name):

CREATE TABLE my_keyspace.user (
last_name text,
first_name text,
middle_initial text,
title text,
PRIMARY KEY (last_name, first_name)
) 

Insert two rows into the table:

INSERT INTO user (first_name, middle_initial, last_name, title)
VALUES ('Bill', 'A', 'A', 'Mr.')

INSERT INTO user (first_name, last_name)
VALUES ('Tom', 'B')

On disk, the storage in Cassandra will look like:

(first_name='Bill', middle_initial='A', last_name='A', title='Mr.')
(first_name='Tom', last_name='B')

The two records are of different sizes - the null columns are not stored on disk. This is not the case with relational databases, in which every record has the same size and fits nicely on a page. In the relational database world, the storage will look like the following where the nulls take up disk space so that the two records have the same size:

(first_name='Bill', middle_initial='A', last_name='A', title='Mr.')
(first_name='Tom', middle_initial=null, last_name='b', title=null)

So for wide column databases you could define 1000 columns in your schema without worrying whether each read is reading tons of nulls because your data is sparse.

1
  • Not all RDBMSs store nulls.
    – philipxy
    Commented Mar 15 at 3:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.