3

for a django application I'm working on, I need to implement a two ways RPC so :

  • the clients can call RPC methods from the platform and
  • the platform can call RPC methods from each client.

As the clients will mostly be behind NATs (which means no public IPs, and unpredictable weird firewalling policies), the platform to client way has to be initiated by the client.

I have a pretty good idea on how I can write this from scratch, I also think I can work something out of the publisher/subscriber model of twisted, but I've learned that there is always a best way to do it in python.

So I'm wondering what would be the best way to do it, that would also integrate the best to django. The code will have to be able to scope with hundreds of clients in short term, and (we hope) with thousands of clients in medium/long term.

So what library/implementation would you advice me to use ? I'm mostly looking to starting points for RTFM !

7

4 Answers 4

7
+50

websocket is a moving target, with new specifications from time to time. Brave developpers implements server side library, but few implements client side. The client for web socket is a web browser.

websocket is not the only way for a server to talk to a client, event source is a simple and pragmatic way to push information to a client. It's just a never ending page. Twitter fire hose use this tricks before its specification. The client open a http connection and waits for event. The connection is kept open, and reopen if there is some troubles (connection cut, something like that). No timeout, you can send many events in one connection.

The difference between websocket and eventsource is simple. Websocket is bidirectionnal and hard to implement. Eventsource is unidirectionnal and simple to implement, both client and server side.

You can use eventsource as a zombie controller. Each client connects and reconnect to the master and wait for instruction. When instruction is received, the zombie acts and if needed can talk to its master, with a classical http connection, targeting the django app.

Eventsource keep the connection open, so you need an async server, like tornado. Django need a sync server, so, you need both, with a dispatcher, like nginx. Django or a cron like action talks to the async server, wich talks to the right zombie. Zombie talks to django, so, the async server doesn't need any peristance, it's just a hub with plugged zombies.

Gevent is able to handle such http server but there is no decent doc and examples for this point. It's a shame. I want a car, you give me a screw.

2

You can also use Tornado + Tornadio + Socket.io. That's what we are using right now for notifications, and the amount of code that you should write is not that much.

from tornadio2 import SocketConnection, TornadioRouter, SocketServer
class RouterConnection(SocketConnection):
__endpoints__ = {'/chat': ChatConnection,
                 '/ping': PingConnection,
                 '/notification' : NotificationConnection
                 }

def on_open(self, info):
    print 'Router', repr(info)

MyRouter = TornadioRouter(RouterConnection)

# Create socket application
application = web.Application(
MyRouter.apply_routes([(r"/", IndexHandler),
    (r"/socket.io.js", SocketIOHandler)]),
        flash_policy_port = 843,
        flash_policy_file = op.join(ROOT, 'flashpolicy.xml'),
        socket_io_port = 3001,
        template_path=os.path.join(os.path.dirname(__file__), "templates/notification")
)

class PingConnection(SocketConnection):
     def on_open(self, info):
       print 'Ping', repr(info)

     def on_message(self, message):
       now = dt.utcnow()
       message['server'] = [now.hour, now.minute, now.second, now.microsecond / 1000]
       self.send(message)
class ChatConnection(SocketConnection):
   participants = set()
   unique_id = 0

   @classmethod
   def get_username(cls):
       cls.unique_id += 1
       return 'User%d' % cls.unique_id

   def on_open(self, info):
       print 'Chat', repr(info)

       # Give user unique ID
       self.user_name = self.get_username()
       self.participants.add(self)

   def on_message(self, message):
       pass

   def on_close(self):
       self.participants.remove(self)

   def broadcast(self, msg):
       for p in self.participants:
            p.send(msg)
2
  • damn... I see your comment just after I made a full event_source implementation in tornado... and it had more locs ! (and though your answer seems cool, I don't have the nerves to try out yet another solution, and as I based the solution I coded on athoune's suggestions, I'll let him keep the 'accepted answer').
    – zmo
    May 25, 2012 at 13:03
  • too bad :) no problem. Somebody else will benefit as well. May 29, 2012 at 22:51
1

here is a really simple solution I could came up with :

import tornado.ioloop
import tornado.web
import time

class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous 
    def get(self):
        self.set_header("Content-Type", "text/event-stream")
        self.set_header("Cache-Control", "no-cache")
        self.write("Hello, world")
        self.flush()
        for i in range(0, 5):
            msg = "%d<br>" % i
            self.write("%s\r\n" % msg) # content
            self.flush()
            time.sleep(5)


application = tornado.web.Application([
    (r"/", MainHandler),
])

if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

and

curl http://localhost:8888

gives output when it comes !

Now, I'll just have to implement the full event-source spec and some kind of data serialization between the server and the clients, but that's trivial. I'll post an URL to the lib I'll write here when it'll be done.

2
1

I've recently played with Django, Server-Sent Events and WebSocket, and I've wrote an article about it at http://curella.org/blog/2012/jul/17/django-push-using-server-sent-events-and-websocket/

Of course, this comes with the usual caveats that Django probably isn't the best fit for evented stuff, and both protocols are still drafts.

1
  • interesting article, though until wsgi supports multithreading, it's not imaginable to go that way in production.
    – zmo
    Aug 8, 2012 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.