57

I have a working_df in pandas I'd like to output to sqlite database.

from sqlalchemy import create_engine

sql_engine = create_engine('sqlite:///test.db', echo=False)
working_df.to_sql('data', sql_engine,index=False, if_exists='append')

returns: AttributeError: 'Engine' object has no attribute 'cursor'

Any thoughts?

Pandas version '0.18.1'

Edit: Added full trace

AttributeError                            Traceback (most recent call last)
<ipython-input-41-4f64fc939721> in <module>()
----> 1 working_df.to_sql('data', engine, index=False, if_exists='append')

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/core/generic.py in to_sql(self, name, con, flavor, schema, if_exists, index, index_label, chunksize, dtype)
   1163         sql.to_sql(self, name, con, flavor=flavor, schema=schema,
   1164                    if_exists=if_exists, index=index, index_label=index_label,
-> 1165                    chunksize=chunksize, dtype=dtype)
   1166 
   1167     def to_pickle(self, path):

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/io/sql.py in to_sql(frame, name, con, flavor, schema, if_exists, index, index_label, chunksize, dtype)
    569     pandas_sql.to_sql(frame, name, if_exists=if_exists, index=index,
    570                       index_label=index_label, schema=schema,
--> 571                       chunksize=chunksize, dtype=dtype)
    572 
    573 

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/io/sql.py in to_sql(self, frame, name, if_exists, index, index_label, schema, chunksize, dtype)
   1659                             if_exists=if_exists, index_label=index_label,
   1660                             dtype=dtype)
-> 1661         table.create()
   1662         table.insert(chunksize)
   1663 

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/io/sql.py in create(self)
    688 
    689     def create(self):
--> 690         if self.exists():
    691             if self.if_exists == 'fail':
    692                 raise ValueError("Table '%s' already exists." % self.name)

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/io/sql.py in exists(self)
    676 
    677     def exists(self):
--> 678         return self.pd_sql.has_table(self.name, self.schema)
    679 
    680     def sql_schema(self):

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/io/sql.py in has_table(self, name, schema)
   1674         query = flavor_map.get(self.flavor)
   1675 
-> 1676         return len(self.execute(query, [name, ]).fetchall()) > 0
   1677 
   1678     def get_table(self, table_name, schema=None):

/Users/tom/anaconda/envs/data_science/lib/python3.5/site-packages/pandas/io/sql.py in execute(self, *args, **kwargs)
   1557             cur = self.con
   1558         else:
-> 1559             cur = self.con.cursor()
   1560         try:
   1561             if kwargs:

AttributeError: 'Engine' object has no attribute 'cursor'
10
  • Are you sure that is is picking up the correct pandas version? Can you show the output of pd.__version__ in the same session as where you get that error? As you get typically this error if you have an older pandas version (< 0.14)
    – joris
    Commented Jul 12, 2016 at 15:19
  • pd.__version__ returns '0.18.1', I had seen it was an older pandas issue which is why it was strange
    – nahata5
    Commented Jul 12, 2016 at 15:21
  • going on the older issue stackoverflow.com/questions/30631325/…, I did a engine.raw_connection() which when applied to the to_sql, it seemed to work
    – nahata5
    Commented Jul 12, 2016 at 15:22
  • 3
    For some reason, it thinks that it is not a sqlalchemy engine (so falls back to sqlite connection). Can you show the output of pd.io.sql._is_sqlalchemy_connectable(sql_engine)?
    – joris
    Commented Jul 12, 2016 at 18:07
  • 1
    And isinstance(sql_engine, sqlalchemy.engine.Connectable)?
    – joris
    Commented Jul 12, 2016 at 18:10

9 Answers 9

76

Solution for 2024

I know this question is old, but a recent change to Pandas warrants a new solution

Pandas 2.2.0 appears to have modified how a SQLAlchemy Engine object operates when passed to the con argument for pd.read_sql, pd.to_sql, etc. Unsure if this is intentional.

My project worked perfectly fine with Pandas 2.1.4. When updated to 2.2.0, I receive the AttributeError due to the cursor not being present.

  • Solution: Use the connection attribute of a SQLAlchemy Connection object instead of the Engine object directly.

Peep this example:

from sqlalchemy import create_engine
import pandas as pd


# Engine for MS SQL Server using pymssql
conn_url = f"mssql+pymssql://{username}:{password}@{host}/{database}"
engine = create_engine(conn_url)

# Example query
query = 'select id from users'

# Works with pandas<2.2.0
df = pd.read_sql(
    sql=query,
    con=engine,
)

# Works with pandas==2.2.0
with engine.connect() as conn:
    df = pd.read_sql(
        sql=query,
        con=conn.connection
    )

As per comments below, this works with pandas 2.2 for .read_sql(), .read_sql_query(), etc., but does not work for .to_sql() with backends other than SQLite.

10
  • 2
    You, sir, are a hero! Thank you so much! I run into this error after updating pandas today..
    – tturbo
    Commented Feb 8 at 15:03
  • 6
    This should be the first answer now
    – linSESH
    Commented Feb 13 at 15:30
  • 7
    I suspect this is this pandas issue, relating to Pandas 2.2.0 setting the minimum supported version of SQLAlchemy to 2.0. Commented Feb 19 at 19:11
  • 5
    works for pd.read_sql but introduces a new error for df.to_sql`: "Invalid object name 'sqlite_master'."
    – Lawrence
    Commented Jun 13 at 2:15
  • 1
    @jasko yes - my solution was to update sqlalchemy to v2.0. More details are available in the link form snakecharmerb above
    – Lawrence
    Commented Jun 24 at 5:55
53

adding in a raw_connection() worked for me

from sqlalchemy import create_engine

sql_engine = create_engine('sqlite:///test.db', echo=False)
connection = sql_engine.raw_connection()
working_df.to_sql('data', connection, index=False, if_exists='append')

I had conda install sqlalchemy during my notebook session, so while it was accessible, since I already initiated pandas, it appeared as if there was no sqlalchemy. Restarting the session allowed me to remove the raw_connection().

from sqlalchemy import create_engine

sql_engine = create_engine('sqlite:///test.db', echo=False)
connection = sql_engine
working_df.to_sql('data', connection, index=False, if_exists='append')
5
  • 31
    I was using Jupyter. After pip install sqlalchemy, I just needed to restart the Jupyter kernel for pandas to be able to use sqlalchemy.
    – jjmontes
    Commented Apr 11, 2017 at 11:57
  • I was in a REPL and installed SQLAlchemy in another terminal tab without restarting the REPL. The overall takeaway seems to be that SQLAlchemy needs to be available when Pandas is loaded for this to work. Commented Oct 30, 2017 at 19:26
  • 1
    Exactly true, @jjmontes. Resart before trying anything.
    – Axis
    Commented Nov 28, 2017 at 11:03
  • For me, no restart issue, this change from connect() to raw_connection() made it. On top of that, with statement on the connection was also not possible, I have to manually conn.close() it in the end, see [Python Error: AttributeError: enter [without the markup: __enter__])](stackoverflow.com/questions/51427729/…). I thought sqlalchemy was a standard? Commented Jan 27, 2022 at 11:03
  • Please check Solution for 2024 for pandas >2.2.0 (and help voting up)
    – tturbo
    Commented Feb 8 at 15:05
7

The following code was giving me the same error while I was working to push data to MS Sql Server:

engine_string = 'mssql+pyodbc://'+username_dsdb+':'+password_dsdb+'@'+server_dsdb+':1433/'+database_dsdb+'?driver=ODBC Driver 13 for SQL Server'
engine = sqlalchemy.create_engine(engine_string,deprecate_large_types=True)

conn_dsdb = engine.connect()

df_year_week_contract_wise.to_sql("KPIEngine_FACT_Historical", conn_dsdb, index=False, if_exists='append')

conn_dsdb.close()

But, then I restarted the session and it worked fine.

1
  • 1
    For me similarly i needed to restart session (for jupyter restart kernel) . I think main problem is on the same python session previously there was a connection (likely with pyodbc) but it hanged and was not closed properly (happens not using with context).
    – Gorkem
    Commented Aug 26, 2021 at 9:31
4
from sqlalchemy import create_engine
sql_engine = create_engine('sqlite:///test.db', echo=False)
working_df.to_sql('data', sql_engine,index=False, if_exists='append')

Instead of above code, can you try to use connect method of the engine object instead of engine object itself as below.,

sql_engine = create_engine('sqlite:///test.db', echo=False)
conn = sql_engine.connect()
working_df.to_sql('data', conn,index=False, if_exists='append')

Try above steps and let me know if you still facing the issue.

2
  • 4
    I'm having the same issue of this post. I tried your suggestion but the error now is AttributeError: 'Connection' object has no attribute 'cursor'. Commented Jan 2, 2020 at 16:00
  • 1
    @LucasAimaretto Neither connection nor engine do not have cursor method. I assume the cursor purpose for you is to fetch the results stored. You can use execute method of connection for this purpose. You can refer link. Commented Jun 24, 2020 at 6:00
2

Austin is a lifesaver.

To make this work until pandas is updated and works nicely with sqlalchemy, make sure you pandas version is <2.2 and sqlalchemy<2.0

My setting to get this to stop erroring: pandas==2.1.* sqlalchemy==1.4.*

To supress sqlalchemy warning, add this: from sqlalchemy.util import deprecations deprecations.SILENCE_UBER_WARNING = True

3
  • I'm honestly puzzled by this: SQLA 2.0.27 and Pandas 2.2.0 execute df.to_sql without any issues for me. Commented Feb 19 at 19:00
  • Ah, never mind, see this comment Commented Feb 19 at 19:12
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Feb 20 at 15:04
1

This is an old question but I had the same error this week and it turned out that there was a problem with MySQLdb on my Mac. As a test I tried to ‘import MySQLdb as mysql’ and discovered an ImportError: “Library not loaded: libmysqlclient.18.dylib”

The solution in my case was to add a symbolic link to the missing library as suggested in Python mysqldb: Library not loaded: libmysqlclient.18.dylib

1

As an alternative to raw_connection, using the connection attribute on a connection seems to work with pandas>2, sqlalchemy<2.

with sql_engine.connect() as connection:
    working_df.to_sql("data", connection.connection, index=False, if_exists='append')

In this way you can still use the context manager and have the connection closed.

0

From my experience, it's because just pip install the sqlalchemy and didn't restart the jupyter.

So what just need to do is restart the kernel...

0

I uninstalled sqlalchemy and pandas then reinstalled and it fixed the problem

2
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented May 31 at 13:44
  • 1
    What modules versions were you operating before and after the update? Pip or conda?
    – OCa
    Commented Jun 2 at 5:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.