0

I have a large app, that has multiple databases, with identical schema. What is the best solution to implement, so I can have users dynamically, switch between databases to make queries. Creating an engine for each database is not a solution, as I receive a ' too many connections' postgres error. The main issue here is creating too many engines, so is there a way to remove an engine after using it? The number of databases will be thousands, with several hundred users at the same time.

Thanks.

Edit: Here is the connection code

engine = create_engine(datbase_uri)  
session = sessionmaker(bind=engine)

^This is done every time a connection to particular database is required. And is done dynamically (when ever user requests within the app). The issue is when this is done multiple times, the 'too many connections' error is given.

What is the best way to go about closing the engine?

5
  • Use multiple Postgres schemas (i.e. the namespace feature in Postgres, not the table schema) and either make your queries all fully-qualify the schemas, or set the search_path dynamically.
    – rkrzr
    Aug 24, 2015 at 22:02
  • The current implementation uses many databases with one schema, is there a solution based on this?
    – navy_green
    Aug 24, 2015 at 23:38
  • Snippet of code added.
    – navy_green
    Aug 25, 2015 at 0:31
  • 1
    "This is done every time a connection to particular database is required" <-- don't do that. An engine should be created once and then keep being used during the lifetime of the process. The engine handles the connection threadpool for you, so it is really inefficient to create a new engine for every single db query. You should really consider reorganizing your db to use many schemas instead of many dbs.
    – rkrzr
    Aug 25, 2015 at 9:32
  • This may be the solution to the problem.
    – navy_green
    Aug 25, 2015 at 14:38

1 Answer 1

0

We had the same issue before. In our multi-tenant app we use set search_path to <schemaName> psql query to serve DMLs from paticular client. You can check detailed implementation for SQLAlchemy here

Regarding too many connections issue it's better to use connection pool. Answer for SQLAlchemy is here

2
  • The current implementation uses many databases with one schema, is there a solution based on this?
    – navy_green
    Aug 24, 2015 at 23:38
  • unless you have a legitimate reason to have multiple databases with one schema each, I would go with one db - multiple schemas in postgres. Aug 24, 2015 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.