2

I have file consisting of thousands of lines(each containing 3 fields, first is a k length string, then a number, third another string) of form:-

k|1|r1
k|1|r2
k|2|r2
k1|1|r3

I load it using redis-py, by using:-

sadd('k:1', 'r1')
sadd('k:1', 'r2')
sadd('k:2', 'r2')
sadd('k1:1', 'r3')

to form a mapping like

{
     "k:1"  : ("r1", "r2"),
     "k:2"  : ("r2"),
     "k1:1" : ("r3")
}

I intend to store the values of the form, by removing the repetitive information of k(which is a k length string common for the first 3 records):

{
     "k": {
         "1"  : ("r1", "r2"),
        "2"  : ("r2")
      }
     "k1": {
       "1" : ("r3")
     }
}

I have the idea of storing the value of storing the set under a different key, which can act as value for k in the hash. Is there a better way than that?

2
  • Is this data more or less static, or will you be adding/removing a lot of data from the sets? May 28, 2015 at 14:14
  • @Zim-ZamO'Pootertoot the data can be considered static
    – Mohit
    May 28, 2015 at 17:32

1 Answer 1

1

Storing the sets under a different key will work, however if the sets are static then you don't need the set functionality - instead you can save yourself a lookup and store the set directly in the map, e.g. as a comma separated value (or whatever separator is appropriate for your data) if you've only got a few values, or as a gzipped comma separated value if you've got a lot of values (in my experience, if the string is large enough then the cost of g(un)zip is offset by the reduced network cost).

4
  • My use case addresses a lot of values and I process a line at a time, if I gzip the string(the seperator seperated), then I think I have to, while loading the values, read the value from redis, convert it into a set in the python engine I have written, add the new value if it does not exist and then dump the gzipped string back. Is this what your solution suggests?
    – Mohit
    May 29, 2015 at 6:50
  • 1
    @ahujamoh Yes, the gzip approach is really only appropriate for static data - if you're able to build the sets in a local variable before converting them to a gzipped string then this approach is viable, but if you're building the sets in Redis then the gzip approach has too much overhead, in which case it's preferable to store the sets as Redis Sets. Once the data is entered you can then optionally switch to the gzip approach to improve read access. May 29, 2015 at 12:00
  • I thought of using a different database for each of different top level key(i.e. k and k1 in the question I asked) or starting up a new instance for each top level key. How does this direction sound?
    – Mohit
    Jun 1, 2015 at 6:49
  • @ahujamoh That is probably overkill unless you're trying to spread the data across multiple servers - from a correctness standpoint that approach is fine, but it adds to the complexity of database management. Jun 1, 2015 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.