-3

I have created redis pool instance with singleton design pattern in python.

class RedisPool:
    pool = None
    @classmethod
    def get_connection_pool(cls, startup_nodes):
        if cls.pool == None:
            cls.pool = ConnectionPool(host = startup_nodes['host'], port = startup_nodes['port'], db = 0, password = startup_nodes['password'])
        return cls.pool

As I understand from redis documentation we should have single pool instance which is why we use singleton . However one of the flow in python script executes scala script with spark-submit which uses redis connection to dump some data. How can I use same pool instance i have created in python script to use that in scala.

9
  • it would be better if somebody down voting the question mentions the reason to so. It would help me learn as i am new to this.
    – dilkash
    Commented Dec 19, 2018 at 10:02
  • You may be wondering - "why the downvotes?". The reason is that this question is driven by very basic level mis-conceptions. In simple analogous terms, if a program is the script of Romeo and Juliet then an "instance" in a running program can be thought of as "instance" of Romeo in a running Romeo and Juliet play. Now, this "particular Romeo" can not be moved to some other play as this Romeo does not exist without a running play and outside that play. And as you should have been able to figure out, you don't want more than one Romeo in a running Romeo and Juliet play. Commented Dec 19, 2018 at 10:04
  • Now, think of another drama company as a different language, they will have functionally the same script but with some structural and aesthetic differences. They will have different actors to play the same role as compared to your drama company. And it goes without saying that the Romeo from one of the running plays by your drama company can not do anything in the a running play by this company. Commented Dec 19, 2018 at 10:07
  • @SarveshKumarSingh Thanks for writing the basics of OOPS to me but i already know this. Also your example is not valid as single Actor can play Romeo in French and English movie i.e. movies of different language or no matter if it is a different company or with "structural and aesthetic differences" provided he is unique person which in our case if there is some kind of id which is unique which i can get from redis that i can use in python and scala is what i meant.
    – dilkash
    Commented Dec 19, 2018 at 10:26
  • 1
    There is no other way. And there is no need for the other way. As for "why?"; you should look into why it it advised to have a single "Pool" and just what exactly is this "Pool". Commented Dec 19, 2018 at 11:11

1 Answer 1

1

No, your scala script cannot reuse the connection pool defined in python.

Python's connection pool is only valid within a single process. When you call a scala script from python, you are essentially launching a new process. This new process does not have access to the connection pool.

You will have to pass redis host/port/password from python to scala, and your scala script would have to create another connection to redis.

To improve this, you could make scala process itself long lived with it's own connection pool. Then expose some "api" for the python process to call.

1
  • Thanks for your reply. Just to clarify more on the approach i was thinking , can i destroy the pool each time i am done with redis processing and create a new pool each time i need a connection, will it get inefficient?
    – dilkash
    Commented Dec 20, 2018 at 6:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.