6

Background

I am receiving CSV data files from clients that contain a large amount of data that I don't need and a small amount of data that I do. In the future I may need to access that data and although I'm archiving the original data files I was hoping for something a bit easier to query. I was hoping for a solution that didn't mean the data files stayed in the same format - i.e. the client may add/remove columns and I don't want my implementation to bail on missing data or fail to archive additional data.

As I'm building the application in Azure, Azure table storage looked correct to me - I could read the data files and then store whatever key/value pairs I read into the data store.

Upshot

I would like to know how to store a Dictionary<K, V> or Hashtable or some other key/value pairs in Azure.

1 Answer 1

11

By overriding the ReadEntity and WriteEntity methods on classes derived from TableEntity, additional properties can be stored.

Here's my naive implementation

public class TestEntity : TableEntity {

    public TestEntity(string a, string b) {
        PartitionKey = a;
        RowKey = b;
        DataItems = new Dictionary<string, string>();
    }

    public Dictionary<string, string> DataItems { get; set; }

    public override IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext) {
        var results = base.WriteEntity(operationContext);
        foreach (var item in DataItems) {
            results.Add("D_" + item.Key, new EntityProperty(item.Value));
        }
        return results;
    }

    public override void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext) {
        base.ReadEntity(properties, operationContext);

        DataItems = new Dictionary<string, string>();

        foreach (var item in properties) {
            if (item.Key.StartsWith("D_")) {
                string realKey = item.Key.Substring(2);
                ItemData[realKey] = item.Value.StringValue;
            }
        }
    }
}

I've noted that Azure table storage can only store a total of 255 key/value pairs – or 252 custom ones once PartitionKey, etc. is taken into account, so this has to be handled somewhere as well.

5
  • I've blogged about this and would welcome any additional information anyone has, or just a note if you're using it!
    – Rob Church
    Commented Jan 30, 2013 at 2:02
  • 1
    How about if you just serialize your Dictionary<string, string>() using binaryformatter (or XML formatter) a write the entire object to a single entity property of type byte[] (or string)? I would use that approach instead.
    – astaykov
    Commented Jan 30, 2013 at 6:45
  • Leaving them as additional properties would (in theory, not tested) allow me to use TableQuery.GenerateFilterConditionFor* to query on that data (which could be useful if I suddenly needed to use it rather than just archive it). Serialized, I imagine that would be quite tricky. And while you do sidestep the 252 property limit, you still have to worry about bumping into the 64kb-per-property limit (so you'd have to serialize over several properties).
    – Rob Church
    Commented Jan 31, 2013 at 0:28
  • If I had a very large number of known-to-be-small key/value pairs I may have ended up using that solution though, my data probably leans me towards mine (for now).
    – Rob Church
    Commented Jan 31, 2013 at 0:30
  • true for byte[] limitation, but still quering on anything other than PL&RK will be slow. Have you looked at Lokad.Cloud's Fat Entities - github.com/Lokad/lokad-cloud/wiki/FatEntities
    – astaykov
    Commented Jan 31, 2013 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.