3

I am new to Python and have been studying its fundementals for 3 months now, learning types, functions and algorithms. Now I started practiciging web app development with GAE framework.

Goal: have a very large dictionary, which can be accessed from all .py files throughout the web app without having it stored more than once or re-created each time when someone visits a URL of the app.

I want to render a simple DB table to a dictionary, with hopes of speed gain as it will be in memory.

Also I am planing on creating an in memory DAWG - TRIE

I don't want this dictionary to be created each time a page is called, I want it to be stored in memory once, kept there and used and accessed by all sessions and if possible modified too.

How can I achieve this? Like a simple in memory DB but actually a Python dictionary?

Thank you.

3 Answers 3

7

Use the memcache. You can store a pickled dict in the memcache, but you could also just store the keys/values directly in memcache. Write a wrapper class that ensures loading of the values from the data store if they are not already in memcache.

Or even better, just use ndb, which automatically caches values in memcache for you. This way you just query the values from the data store, and ndb will automatically cache the values in memory for you (across multiple requests).

5
  • +1: But what about persistence? The OP kind of hinted at that as a possibility
    – mhawke
    Jun 13, 2012 at 11:42
  • Hello! Thank you very much for your input. I looked at ndb however it would mean a lock-down of me to GAE which I don't want to. I am using them for practicing purposes right now, to gain better knowledge of Python. I also looked at redis and memcache and it looks as if either would be the solution but I would much prefer a pure Python hack here.
    – Phil
    Jun 13, 2012 at 11:52
  • 1
    @Phil, every platform solves problems differently. When using GAE you should take advantage of the features it provides. If you are afraid of locking yourself down to a given platform, in my opinion, the correct way to overcome this is by writing your code in a modularized manner such that you have a "service" that provides data in the way that is natural for the given platform. Your .py files use this "service" in a generic manner, and when you change platform you just write a new implementation of the service that fits the new platform. Jun 13, 2012 at 12:00
  • 2
    @david: is anything possible for you?
    – mhawke
    Jun 13, 2012 at 12:02
  • 1
    @KlausByskovHoffmann, your suggestion and advise is very sound. I shall definitely take it into account now and in future for all the projects I work on. I will go with Memcache now and modularize whatever I can. Thank you!
    – Phil
    Jun 13, 2012 at 12:23
0

I'll have a stab at this and put forward the standard python shelve module. This provides a simple persistent dictionary that is backed by an dbm file.

There are a few caveats particularly relating to concurrency. It does provide some caching if writeback is enabled, but again there might be concerns re memory consumption. One other limitation is that keys must be strings.

Still, it might be worth a look. It definitely fulfils the description of "pure Python hack".

Simple example:

import shelve

d = shelve.open('my_shelf')
for i in range(100000):
    d[str(i)] = 'Item %s' % i
d.close()
d = shelve.open('my_shelf')
>>> d['50000']
'Item 50000'
4
  • 1
    You can't write to the filesystem on App Engine, so this won't work.
    – Wooble
    Jun 13, 2012 at 12:19
  • Hello and thank you! This, although not suitable for GAE (and I probably will go with memcache or redis), is the thing I was wondering, I believe! I just read the shelve's docs. If I want to store not strings but objects, which type of file shall I use, if not dbm which is string-only?
    – Phil
    Jun 13, 2012 at 12:30
  • You can store objects. It is only the keys that need to be strings. Internally shelve will store pickled object, but there's no need for you to worry about that.
    – mhawke
    Jun 13, 2012 at 14:10
  • @Wooble: Shame, I'm not that familiar with GAE.
    – mhawke
    Jun 13, 2012 at 14:11
-2

This is impossible (without an external service). DBs are made for this to store data longer than one request. What you could do is to safe the dict "in" the users session, but I don't recommend that. Unless you have millions of entries every DB is fast enough even sqlite.

8
  • 1
    It is possible to keep something in memory that outlives a request, why would you claim that's not the case? Jun 13, 2012 at 11:31
  • It is not. Not on serious webservers which may stop the thread and start a new one on a new request.
    – dav1d
    Jun 13, 2012 at 11:33
  • It is: think of memcached or other services running on a different port. You can simply access those services and they keep the data in memory instead. Jun 13, 2012 at 11:34
  • 1
    But the question was: is it possible to keep a Python dictionary in memory in a way that outlives a request? The answer to that is yes. Jun 13, 2012 at 11:37
  • 1
    Your desire to be right seems a bit stronger than needed. He doesn't demand that services may not be used and furthermore, the serialization to/from such a service is part of any wrapper code. That is not an unfair requirement as all data needs to be stored in memory somehow. Jun 13, 2012 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.