2

After hearing about NoSQL for a couple years I finally started playing with RavenDB today in an .Net MVC app (simple blog). Getting the embedded database up and running was pretty quick and painless.

However, I've found that after inserting objects into the document store, they are not always there when the subsequent page refreshes. When I refresh the page they do show up. I read somewhere that this is due to stale indexes.

My question is, how are you supposed to use this in production on a site with inserts happening all the time (example: e-commerce). Isn't this always going to result in stale indexes and unreliable query results?

5
  • Wow, maybe I'm missing something, but after using it some more I don't see how this can be used as a primary data store. Sometimes it takes several requests, spanning 15+ seconds, for the query to contain the correct results. This is in a super small document store with 1 doc type, no hierarchy, and 3 documents?!? That's fine if you are mostly read-only and don't need data consistency (e.g. displaying web pages), but I couldn't use this as a real replacement for an RDBMS for something like ecommerce/customer/order data. I tried WaitForNonStaleResultsAsOfLastWrite but then my app just hangs.
    – Sam
    Sep 12, 2013 at 18:00
  • 1
    Eventual Consistency allows many applications to function better. Developers from a SQL background often assume it to be a bigger challenge than it really is from a business perspective. Sep 12, 2013 at 19:57
  • When a customer adds something to their account, they don't expect it to be there right away? My employer puts an emphasis on customer experience, not ease of development. In today's day and age most people want things now. That's one of the driving forces behind the whole JavaScript/Asynchronous web...you send smaller bits of data for a faster and more responsive user experience. Can you give me examples where unpredictable Eventual Consistency is acceptable for a business and/or where it is the norm?
    – Sam
    Sep 12, 2013 at 20:11
  • I'm also curious how RavenDB claims to be ACID when it clearly lacks the Consistency part? Isn't "Eventual Consistency" just another way of saying "Currently Inconsistent"?
    – Sam
    Sep 12, 2013 at 20:18
  • 2
    Eventual consistency in RavenDB applies only to queries against indexes. Loading and retrieving documents by ID is always ACID. Sep 15, 2013 at 17:15

2 Answers 2

8

Think of what actually happens with a traditional database like SQL Server.

  • When an item is created, updated, or deleted from a table, any indexes associated with table also have to be updated.
  • The more indexes you have on a table, the slower your write operations will be.
  • If you create a new index on an existing table, it isn't used at all until it is fully built. If no other index can answer a query, then a slow table scan occurs.
  • If others attempt to query from an existing index while it is being modified, the reader will block until the modification is complete, because of the requirement for Consistency being higher priority than Availability.
  • This can often lead to slow reads, timeouts, and deadlocks.

The NoSQL concept of "Eventual Consistency" is designed to alleviate these concerns. It is optimized reads by prioritizing Availability higher than Consistency. RavenDB is not unique in this regard, but it is somewhat special in that it still has the ability to be consistent. If you are retrieving single document, such as reviewing an order or an end user viewing their profile, these operations are ACID compliant, and are not affected by the "eventual consistency" design.

To understand "eventual consistency", think about a typical user looking at a list of products on your web site. At the same time, the sales staff of your company is modifying the catalog, adding new products, changing prices, etc. One could argue that it's probably not super important that the list be fully consistent with these changes. After all, a user visiting the site a couple of seconds earlier would have received data without the changes anyway. The most important thing is to deliver product results quickly. Blocking the query because a write was in progress would mean a slower response time to the customer, and thus a poorer experience on your web site, and perhaps a lost sale.

So, in RavenDB:

  • Writes occur against the document store.
  • Single Load operations go directly to the document store.
  • Queries occur against the index store
  • As documents are being written, data is being copied from the document store to the index store, for those indexes that are already defined.
  • At any time you query an index, you will get whatever is already in that index, regardless of the state of the copying that's going on in the background. This is why sometimes indexes are "stale".
  • If you query without specifying an index, and Raven needs a new index to answer your query, it will start building an index on the fly and return you some of those results right away. It only blocks long enough to give you one page of results. It then continues building the index in the background so next time you query you will have more data available.

So now lets give an example that shows the down side to this approach.

  • A sales person goes to a "products list" page that is sorted alphabetically.
  • On the first page, they see that "Apples" aren't currently being sold.
  • So they click "add product", and go to a new page where they enter "Apples".
  • They are then returned to the "products list" page and they still don't see any Apples because the index is stale. WTF - right?

Addressing this problem requires the understanding that not all viewers of data should be considered equal. That particular sales person might demand to see the newly added product, but a customer isn't going to know or care about it with the same level of urgency.

So on the "products list" page that the sales person is viewing, you might do something like:

var results = session.Query<Product>()
                     .Customize(x => x.WaitForNonStaleResultsAsOfLastWrite())
                     .OrderBy(x=> x.Name)
                     .Skip((pageNumber-1) * pageSize).Take(pageSize);

While on the customer's view of the catalog, you would not want to add that customization line.

If you wanted to get super precise, you could use a slightly more optimized strategy:

  • When going back from the "add product" page to the "list products" page, pass along the ProductID that was just added.
  • Just before you query on that page, if the ProductID was passed in then change your query code to:

    var product = session.Load(productId);
    var etag = session.Advanced.GetEtagFor(product);
    
    var results = session.Query<Product>()
                     .Customize(x => x.WaitForNonStaleResultsAsOf(etag))
                     .OrderBy(x=> x.Name)
                     .Skip((pageNumber-1) * pageSize).Take(pageSize);
    
  • This will ensure that you only wait as long as absolutely necessary to get just that one product's changes included in the results list along with the other results from the index.

  • You could optimize this slightly by passing the etag back instead of the ProductId, but that might be less reusable from other places in your application.

But do keep in mind that if the list is sorted alphabetically, and we added "Plums" instead of "Apples", then you might not have seen these results instantly anyway. By the time the user had skipped to the page that includes that product, it would likely have been there already.

6
  • I had tried the WaitForNonStaleResultsAsOfLastWrite and my MVC page timed out. This was in a database containing 3 "documents" with 3 fields each (ID, Name, Description). Are there known performance issues with using the Embedded version with it's data store in the App_Data directory? It might be great for web, category, product type data, but I can't see doing anything like inventory, carts, orders, users, or roles in RavenDB.
    – Sam
    Sep 15, 2013 at 23:54
  • No, there shouldn't be any performance issues with that. There must be something else going on. You may want to ask for help over on the RavenDB Google Group or raise a support issue. Plenty of people use RavenDB for exactly what you are describing and it works perfectly. Those type of operations (inventory, carts, etc.) would be done with Load, not Query. If you are using Query to pull inventory items, then you need to rethink your design. Sep 16, 2013 at 0:18
  • I haven't even been that far. I stopped when it was taking tens of seconds to refresh the index for the small document set I described. I do have a question though. Let's say on a shopping cart page, how do you load all the items. In RDBMS you hold on to the cart id and query off that. To use Load, don't I need to know the id for every line item to do what you are prescribing?
    – Sam
    Sep 16, 2013 at 13:51
  • 3
    A shopping cart would exist in entirety in a single document. You would just load it. It seems you have a common mental block that comes from working with relational databases. RavenDB works best with structured data. A cart and its line items, or an order and its line items, are held together in a single document. In DDD terms, the document represents the entire "Aggregate Entity", inclusive of all non-aggregate entities and value objects. Sep 16, 2013 at 14:59
  • Matt, that makes a lot more sense.
    – Sam
    Sep 16, 2013 at 15:45
3

You are running into stale queries. That is a by design part of RavenDB. You need to make distinction between queries (BASE) and loading by id (ACID).

2
  • So how is RavenDB commonly used for development? I cant reliably save a document to the repository and then immediately return it back as part of a list (common in CRUD scaffold for MVC; create then return to list view). So are we supposed to use RDBMS for the important stuff like users, carts, orders, then use Raven for searching site content, basically as a lucene replacement?
    – Sam
    Sep 13, 2013 at 5:32
  • @Sam You can use RavenDB for all the above. For the "important" stuff you can use WaitForNonStaleResuls if you hit an index. However, for users and carts you could probably load by Id in most cases. Sep 13, 2013 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.