2

I need to communicate update events to all running instances of my python script, and i would like to keep the code as simple as possible. I have zero experience with communicating between running processes. Up until now, i have been reading/writing configuration files, which each instance will read and/or update.

Here is some pseudo code i have written (sort of a simple template) to wrap my head around how to solve this problem. Please try to help me fill in the blanks. And remember, i have no experience with sockets, threads, etc...

import process # imaginary module

class AppA():
    def __init__(self):
        # Every instance that opens will need to attach
        # itself to the "Instance Manager". If no manager
        # exists, then we need to spawn it. Only one manager
        # will ever exist no matter how many instances are
        # running.
        try:
            hm = process.get_handle(AppA_InstanceManager)
        except NoSuchProgError:
            hm.spawn_instance(AppA_InstanceManager)
        finally:
            hm.register(self)
        self.instance_manager = hm

    def state_update(self):
        # This method won't exist in the real code, however,
        # it emulates internal state changes for the sake of
        # explaination.
        #
        # When any internal state changes happen, we will then
        # propagate the changes outward by calling the
        # appropriate method of "self.instance_manager".
        self.instance_manager.propagate_state()

    def cb_state_update(self):
        # Called from the "Instance Manager" only!
        #
        # This may be as simple as reading a known
        # config file. Or could simply pass data
        # to this method.


class AppA_InstanceManager():
    def __init__(self):
        self.instances = []

    def register_instance(self, instance):
        self.instances.append(instance)

    def unregister_instance(self, instance):
        # nieve example for now.
        self.instances.remove(instance)

    def propagate_state(self):
        for instance in self.instances:
            instance.cb_state_update(data)

if __name__ == '__main__':
    app = AppA()

Any Suggestions?

3
  • When you say "instances", are you launching multiple concurrent interpreters from a shell script or some such, or are you spawning multiple instances from a single python interpreter instance? It makes a difference in terms of what approaches will work.
    – Silas Ray
    Apr 16, 2012 at 17:02
  • Hmm, not sure exactly (apologies for my ignorance). I believe the first because i can see one instance of "python.exe" (in task manager) for each instance of my script application. Does that help?
    – Joe
    Apr 16, 2012 at 17:15
  • Yes, it does. I'm pretty sure that precludes using multithread/multiprocess without more significantly rearchitecting your system. I think you have to use an external source like a DB or a file to control inter-process data flow. If you have the flexibility to rearchitect though, mutliprocess/mutlithread may be preferable.
    – Silas Ray
    Apr 16, 2012 at 17:22

1 Answer 1

1

There are a few options for this kind of design.

You could use a message queue, its made for this kind of stuff, e.g. AMQP or some ZeroMQ or something like it.

Or you could use something like Redis or some other (in-memory) database for synchronization.

If you don't want to use something like that, you could use the multiprocessing modules synchronization stuff.

Or use a platform specific IPC system, e.g. shared memory via mmap, sysv sockets, etc.

If you want to do things the way you explained, you could have a look at Twisteds perspective broker.

5
  • I was about to type out something almost exactly like this. Was going to make a specific point though about choosing between a dedicated server process, or a leader-election type situation like the OPs example is suggesting. Redis would be the dedicated device approach, and a message queue like ZeroMQ would give you the option for either.
    – jdi
    Apr 16, 2012 at 16:57
  • Redis provides publish/subscribe message queuing.
    – Marcin
    Apr 16, 2012 at 16:58
  • Python has a Queue module in the stdlib. However. I am wondering if i even need something like a "session manager". If it is possible to find all running instances of an application by "process identifier", or something :/. If so, i could just find all the instances, and then "notify" each instance to go read the new updated config file by passing a message or something. Hmm. I'm just riffing here. I'm a complete idiot of how these communications work.
    – Joe
    Apr 16, 2012 at 17:02
  • The Queue module doesn't help (well, multiprocessing's queue might), as its for threads and not processes.
    – schlenk
    Apr 17, 2012 at 19:34
  • If you have a bunch of processes and just need to find all running instances you can of course use system specific methods (e.g. parsing files in /proc or ps efx on unix, or the appropriate Win32 APIs on windows). Its the same as the solution above, just that you use the OS as the coordinating instance instead of a database/message queue.
    – schlenk
    Apr 17, 2012 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.