Is there technical documentation describing how replication between two Couches works?
What is the basic overview of CouchDB replication? What are some noteworthy characteristics about it?
Is there technical documentation describing how replication between two Couches works?
What is the basic overview of CouchDB replication? What are some noteworthy characteristics about it?
Unfortunately there is no detailed documentation describing the replication protocol. There is only the reference implementation built into CouchDB, and Filipe Manana's rewrite of the same which will probably become the new implmentation in the future.
However, this is the general idea:
If you know Git, then you know how Couch replication works. Replicating is very similar to pushing or pulling with distributed source managers like Git.
CouchDB replication does not have its own protocol. A replicator simply connects to two DBs as a client, then reads from one and writes to the other. Push replication is reading the local data and updating the remote DB; pull replication is vice versa.
The replication algorithm is trivial, uninteresting. A trained monkey could design it. It's simple because the cleverness is the data model, which has these useful characteristics:
JOIN
or a transaction, but it's awesome if you want to write a replicator. Just figure out how to replicate one record, and then repeat that for each record.In addition to application data ({"name": "Jason", "awesome": true}
), every record stores the evolutionary timeline of all previous revision IDs leading up to itself.
Git isn't really a linear list. It has forks, when one parent has multiple children. CouchDB has that too.
Exercise: Compare two different records, A and B. A's revision ID does not appear in B's timeline; however, one revision ID, C, is in both A's and B's timeline. Thus A didn't evolve from B. B didn't evolve from A. But rather, A and B have a common ancestor C. In Git, that is a "fork." In CouchDB, it's a "conflict."
In Git, if both children go on to develop their timelines independently, that's cool. Forks totally support that.
Git also has merges, when one child has multiple parents. CouchDB sort of has that too.
git merge
.At least one sentence in this writeup (possibly this one) is complete BS.
Thanks Jason for the excellent overview! Jens Alfke, who is working on TouchDB and its replication for Couchbase, has (unofficially) described the CouchDB replication algorithm itself if you're interested in the technical details of how a "standard" CouchDB replicator protocol tends to work.
To summarize the steps he's outlined:
_changes
since that pointrevs_diff
on a batch of changes to see which are needed on the targetbulk_docs
both for optimization and so as to store the docs differently than the usual higher-level MVCC handling does on PUT
.I've glossed over many details here, and would recommend reading through the original explanation as well.
The documentation for CouchDB v2.0.0 covers the replication algorithm much more extensively. They have diagrams, example intermediate responses, and example errors. They use the "MUST", "SHALL", etc. language of IETF RFCs.
The specifics for 2.0.0 (still unreleased as of January 2016) are a bit different from 1.x, but the basics are still as @natevw described.
At Apache CouchDB Conf 2013, Benjamin Young introduced replication.io in his Replication, FTW! talk. It's an ongoing effort to define, and eventually mint, the spec for HTTP-based master-master replication.
it is also documented here: http://www.dataprotocols.org/en/latest/couchdb_replication.html, well, sort of.