176

Using SQLAlchemy, an Engine object is created like this:

from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost/mydb")

Accessing engine fails if the database specified in the argument to create_engine (in this case, mydb) does not exist. Is it possible to tell SQLAlchemy to create a new database if the specified database doesn't exist?

5

7 Answers 7

209

SQLAlchemy-Utils provides custom data types and various utility functions for SQLAlchemy. You can install the most recent official version using pip:

pip install sqlalchemy-utils

The database helpers include a create_database function:

from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database

engine = create_engine("postgresql://localhost/mydb")
if not database_exists(engine.url):
    create_database(engine.url)

print(database_exists(engine.url))
6
  • 4
    I get this error when trying this exact codeblock: psycopg2.OperationalError: fe_sendauth: no password supplied. When using "postgres://test:abc123@localhost:5432/test" I get psycopg2.OperationalError: FATAL: password authentication failed for user "test"
    – Guus
    Commented Feb 7, 2019 at 15:23
  • 1
    Sorry for the spam, but I tried changing the port to 9000 and now I get this: "postgres://test:abc123@localhost:9000/test" psycopg2.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
    – Guus
    Commented Feb 7, 2019 at 15:34
  • 1
    psql -U postgres, and then you can create a password with \password. Then you create the engine via engine = create_engine("postgres://postgres:yourpassword@localhost/mydb") Commented Nov 22, 2020 at 22:14
  • 2
    I had no idea this package existed - thank you so much!
    – mattyb
    Commented Mar 4, 2021 at 23:23
  • 1
    this didn't work for me on python 3.8.2, sqlalchemy 1.3.2, sqlalchemy-utils 0.37. a recent similar comment was made for the database_exists() function here
    – arturomp
    Commented Apr 22, 2021 at 20:06
132

On postgres, three databases are normally present by default. If you are able to connect as a superuser (eg, the postgres role), then you can connect to the postgres or template1 databases. The default pg_hba.conf permits only the unix user named postgres to use the postgres role, so the simplest thing is to just become that user. At any rate, create an engine as usual with a user that has the permissions to create a database:

>>> engine = sqlalchemy.create_engine("postgresql://postgres@/postgres")

You cannot use engine.execute() however, because postgres does not allow you to create databases inside transactions, and sqlalchemy always tries to run queries in a transaction. To get around this, get the underlying connection from the engine:

>>> conn = engine.connect()

But the connection will still be inside a transaction, so you have to end the open transaction with a commit:

>>> conn.execute("commit")

And you can then proceed to create the database using the proper PostgreSQL command for it.

>>> conn.execute("create database test")
>>> conn.close()
3
  • 3
    This worked well for me. As a side note, when I did conn.execute('drop database DBWithCaps') I had problems with it not recognizing the caps. conn.execute('drop database "DBWithCaps"') (with the quotes) worked fine.
    – KobeJohn
    Commented Oct 6, 2013 at 14:49
  • I know that PostgreSQL expects all entities in lower case, unless quoted. So if you created a field using MyColumn some DBs will take it as mycolumn. In other words, not sure how you created your table, but if it was created using quotes, it will case-sensitive, so when you access it in an SQL statement you'll need the quotes as well.
    – Guy
    Commented Aug 3, 2016 at 12:00
  • If you use sql alchemy with clickhouse remove conn.execute("commit") Commented Oct 31, 2022 at 10:15
17

It's possible to avoid manual transaction management while creating database by providing isolation_level='AUTOCOMMIT' to create_engine function:

import sqlalchemy

with sqlalchemy.create_engine(
    'postgresql:///postgres',
    isolation_level='AUTOCOMMIT'
).connect() as connection:
    connection.execute('CREATE DATABASE my_database')

Also if you are not sure that database doesn't exist there is a way to ignore database creation error due to existence by suppressing sqlalchemy.exc.ProgrammingError exception:

import contextlib
import sqlalchemy.exc

with contextlib.suppress(sqlalchemy.exc.ProgrammingError):
    # creating database as above
3
  • It seems that you can't connect to a progres server without specifying a database, so you'll probably want to connect to the default "postgres" database to execute the db creation commands, otherwise it will try to connect to the default "user" database and complain if it doesn't exist.
    – Acorn
    Commented May 13, 2015 at 16:21
  • This is the only solution that works for me with SA 2.
    – auxsvr
    Commented Oct 10, 2023 at 20:37
  • 2
    I had to wrap the 'CREATE DATABASE my_database' in a sqlalchemy.sql.text(…), then this worked perfectly fine.
    – Hermann
    Commented Nov 22, 2023 at 10:33
9

Extending the accepted answer using with yields:

from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost")

NEW_DB_NAME = 'database_name'

with engine.connect() as conn:
    conn.execute("commit")
    # Do not substitute user-supplied database names here.
    conn.execute(f"CREATE DATABASE {NEW_DB_NAME}")
6
  • You forgot the line conn.execute("commit") before creating the database
    – AugBar
    Commented May 29, 2021 at 20:13
  • I don't think we need a commit, as CREATE DATABASE issues an implicit commit. Reference for MySQL: dev.mysql.com/doc/refman/8.0/en/implicit-commit.html Reference for other RDMS: datasunrise.com/blog/professional-info/…
    – ATH
    Commented Jun 14, 2021 at 12:52
  • The default isolation level is READ COMMITED when invoking create_connection, so you do need to commit before executing the query - or explicitly pass the argument isolation_level=AUTOCOMMIT
    – AugBar
    Commented Jun 15, 2021 at 11:57
  • Thanks for clarifying, added your suggestion in the answer:)
    – ATH
    Commented Jun 15, 2021 at 20:26
  • isn't the f-string an unsafe practice?
    – baggiponte
    Commented Oct 12, 2022 at 18:50
3

Please note that I couldn't get the above suggestions with database_exists because whenever I check if the database exists using if not database_exists(engine.url): I get this error:

InterfaceError('(pyodbc.InterfaceError) (\'28000\', u\'[28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user \\'myUser\\'. (18456) (SQLDriverConnect); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open database "MY_DATABASE" requested by the login. The login failed. (4060); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user \\'myUser\\'. (18456); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open database "MY_DATABASE" requested by the login. The login failed. (4060)\')',)

Also contextlib/suppress was not working and I'm not using postgres so I ended up doing this to ignore the exception if the database happens to already exist with SQL Server:

import logging
import sqlalchemy

logging.basicConfig(filename='app.log', format='%(asctime)s-%(levelname)s-%(message)s', level=logging.DEBUG)
engine = create_engine('mssql+pyodbc://myUser:mypwd@localhost:1234/MY_DATABASE?driver=SQL+Server+Native+Client+11.0?trusted_connection=yes', isolation_level = "AUTOCOMMIT")

try: 
    engine.execute('CREATE DATABASE ' + a_database_name)
except Exception as db_exc:
    logging.exception("Exception creating database: " + str(db_exc))  
3

If someone like me don't want to take whole sqlalchemy_utils to your project just for database creation, you can use script like this. I've come with it, based on SingleNegationElimination's answer. I'm using pydantic here (it's FastAPI project) and my imported settings for reference, but you can easily change this:

from sqlalchemy import create_engine
from sqlalchemy.exc import OperationalError
from pydantic import PostgresDsn

from src.conf import settings


def build_db_connection_url(custom_db: Optional[str] = None):
    db_name = f"/{settings.POSTGRES_DB or ''}" if custom_db is None else "/" + custom_db
    return PostgresDsn.build(
        scheme='postgresql+psycopg2',
        user=settings.POSTGRES_USER,
        password=settings.POSTGRES_PASSWORD,
        host=settings.POSTGRES_HOST,
        path=db_name,
    )


def create_database(db_name: str):
    try:
        eng = create_engine(build_db_connection_url(custom_db=db_name))
        conn = eng.connect()
        conn.close()
    except OperationalError as exc:
        if "does not exist" in exc.__str__():
            eng = create_engine(build_db_connection_url(custom_db="postgres"))
            conn = eng.connect()
            conn.execute("commit")
            conn.execute(f"create database {db_name}")
            conn.close()
            print(f"Database {db_name} created")
        else:
            raise exc
    eng.dispose()

create_database("test_database")
0

heres a working sniplet

base_url = 'postgresql://postgres:admin@postgres:5432'
engine = create_engine(base_url, isolation_level="AUTOCOMMIT")
# Create database if not exists
dbname = f"{namespace}_{name}"
with engine.connect() as conn:
    cusor = conn.execute(text(f"SELECT 1 FROM pg_database WHERE datname = '{dbname}'"))
    if not cusor.fetchone():
        conn.execute(text(f"CREATE DATABASE {dbname}"))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.