25

It looks like Psycopg has a custom command for executing a COPY:

psycopg2 COPY using cursor.copy_from() freezes with large inputs

Is there a way to access this functionality from with SQLAlchemy?

6 Answers 6

41

accepted answer is correct but if you want more than just the EoghanM's comment to go on the following worked for me in COPYing a table out to CSV...

from sqlalchemy import sessionmaker, create_engine

eng = create_engine("postgresql://user:pwd@host:5432/db")
ses = sessionmaker(bind=engine)

dbcopy_f = open('/tmp/some_table_copy.csv','wb')

copy_sql = 'COPY some_table TO STDOUT WITH CSV HEADER'

fake_conn = eng.raw_connection()
fake_cur = fake_conn.cursor()
fake_cur.copy_expert(copy_sql, dbcopy_f)

The sessionmaker isn't necessary but if you're in the habit of creating the engine and the session at the same time to use raw_connection you'll need separate them (unless there is some way to access the engine through the session object that I don't know). The sql string provided to copy_expert is also not the only way to it, there is a basic copy_to function that you can use with subset of the parameters that you could past to a normal COPY TO query. Overall performance of the command seems fast for me, copying out a table of ~20000 rows.

http://initd.org/psycopg/docs/cursor.html#cursor.copy_to http://docs.sqlalchemy.org/en/latest/core/connections.html#sqlalchemy.engine.Engine.raw_connection

2
  • 1
    This was an awesome find. This reduced my time saving data from 8 hours+ overnight to just 4 minutes or so. Oh my!
    – trench
    Jul 5, 2016 at 14:12
  • 4
    Worked for me, but I had to do fake_conn.commit() at the end
    – Michael
    Jan 6, 2017 at 15:59
31

If your engine is configured with a psycopg2 connection string (which is the default, so either "postgresql://..." or "postgresql+psycopg2://..."), you can create a psycopg2 cursor from an SQL Alchemy session using

cursor = session.connection().connection.cursor()

which you can use to execute

cursor.copy_from(...)

The cursor will be active in the same transaction as your session currently is. If a commit or rollback happens, any further use of the cursor with throw a psycopg2.InterfaceError, you would have to create a new one.

1
  • 1
    Is there a way to do this with the asyncpg dialect? The driver itself has a connection.copy_from_table method but that is not available on the cursor. If I get a raw connection as in the other answers it won't be in the same transaction.
    – tommyip
    Jan 20, 2021 at 7:47
17

You can use:

def to_sql(engine, df, table, if_exists='fail', sep='\t', encoding='utf8'):
    # Create Table
    df[:0].to_sql(table, engine, if_exists=if_exists)

    # Prepare data
    output = cStringIO.StringIO()
    df.to_csv(output, sep=sep, header=False, encoding=encoding)
    output.seek(0)

    # Insert data
    connection = engine.raw_connection()
    cursor = connection.cursor()
    cursor.copy_from(output, table, sep=sep, null='')
    connection.commit()
    cursor.close()

I insert 200000 lines in 5 seconds instead of 4 minutes

9
  • Could you give some details on what the df object is?
    – EoghanM
    Jun 13, 2017 at 22:08
  • df is a pandas dataframe Jun 18, 2017 at 19:01
  • 1
    This is gold! (using pandas and sqlalchemy and postgres) Apr 5, 2018 at 14:07
  • I know its been a while, but could you explain why you are using pandas to_sql AND sqlalchemy copy_from? Doesn't to_sql create the table AND write the contents of df to the table? if so, why then insert the data after again? I'm using this to bulk insert data, but the destination table has a serial id as it's index, and which ever way I try I cant seem to be able to just insert the data without an index (without getting an error saying there's missing column data) Jul 5, 2018 at 16:45
  • @JustinMoser As it happens, pandas.DataFrame has a to_sql method, but that method is not used here. Instead, the to_csv method is used to store df in an in-memory text stream, output, which is then passed to psycopg2 using raw_connection. Jul 23, 2018 at 21:11
12

It doesn't look like it.

You may have to just use psycopg2 to expose this functionality and forego the ORM capabilities. I guess I don't really see the benefit of ORM in such an operation anyway since it's a straight bulk insert and dealing with individual objects a la an ORM would not really make a whole lot of sense.

1
  • 6
    super - can get to psycopg via engine.raw_connection()
    – EoghanM
    Oct 29, 2012 at 16:21
6

If you're starting from SQLAlchemy, you need to first get to the connection engine (also known by the property name bind on some SQLAlchemy objects):

engine = create_engine('postgresql+psycopg2://myuser:password@localhost/mydb')
# or 
engine = session.engine
# or any other way you know to get to the engine

From the engine you can isolate a psycopg2 connection:

# get a psycopg2 connection
connection = engine.connect().connection

# get a cursor on that connection
cursor = connection.cursor()

Here are some templates for the COPY statement to use with cursor.copy_expert(), a more complete and flexible option than copy_from() or copy_to() as it is indicated here: https://www.psycopg.org/docs/cursor.html#cursor.copy_expert.

# to dump to a file
dump_to = """
COPY mytable 
TO STDOUT
WITH (
    FORMAT CSV,
    DELIMITER ',',
    HEADER
);
"""

# to copy from a file:
copy_from = """
COPY mytable 
FROM STDIN
WITH (
    FORMAT CSV,
    DELIMITER ',',
    HEADER
);
"""

Check out what the options above mean and others that may be of interest to your specific situation https://www.postgresql.org/docs/current/static/sql-copy.html.

IMPORTANT NOTE: The link to the documentation of cursor.copy_expert() indicates to use STDOUT to write out to a file and STDIN to copy from a file. But if you look at the syntax on the PostgreSQL manual, you'll notice that you can also specify the file to write to or from directly in the COPY statement. Don't do that, you're likely just wasting your time if you're not running as root (who runs Python as root during development?) Just do what's indicated in the psycopg2's docs and specify STDIN or STDOUT in your statement with cursor.copy_expert(), it should be fine.

# running the copy statement
with open('/path/to/your/data/file.csv') as f:
     cursor.copy_expert(copy_from, file=f)

# don't forget to commit the changes.
connection.commit()
5

You don't need to drop down to psycopg2, use raw_connection nor a cursor.

Just execute the sql as usual, you can even use bind parameters with text():

engine.execute(text('''copy some_table from :csv
                       delimiter ',' csv'''
                   ).execution_options(autocommit=True),
               csv='/tmp/a.csv')

You can drop the execution_options(autocommit=True) if this PR will be accepted

3
  • Just disclose that it is your repository of code, which it appears to be
    – Drew
    Dec 19, 2015 at 20:37
  • 2
    No, the one linked as "this PR" is not my repository, it's the normal/official repository for sqlalchemy. OTOH the PR is obviously mine, that's not a secret: I use the same username on both stackoverflow and bitbucket. Anyhow, I stumbled upon this question while researching examples for that change, and everything I've written is factually correct. I could've avoided linking the PR, but (in case it'll be accepted) then this answer will suggest an outdated snippet, unless me or someone will remember to update it after the fact
    – berdario
    Dec 20, 2015 at 22:25
  • 1
    Adding a note to an old answer, but COPY naming a file requires database superuser privileges, which might not be very desirable. Aug 13, 2018 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.