I am trying to update a larger amount of objects at once (around 1000-1500) using SQLAlchemy's bulk_save_objects
method.
Below is a simplified version of the code. The real use case also includes conditional creation of objects, which is why I am not using bulk_update_mappings
.
For each object, multiple columns may be modified. In order to batch all updates into one query, I am using update_changed_only=False
. I figured this has no drawbacks since all objects are modified anyway.
Code:
import random
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, scoped_session
import logging
logging.basicConfig()
logger = logging.getLogger('sqlalchemy.engine')
logger.setLevel(logging.DEBUG)
# init sqlalchemy
engine = create_engine('sqlite://')
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
session = Session()
Base = declarative_base()
# simple test model
class Foo(Base):
__tablename__ = 'test_foo'
id = Column(Integer, primary_key=True, autoincrement=True)
a = Column(String, default='')
b = Column(String, default='')
c = Column(String, default='')
# recreate database and sample data
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)
items = [
Foo() for _ in range(1000)
]
session.add_all(items)
session.commit()
# query all items
items = session.query(Foo).all()
# update some items
for item in items:
if random.random() < 0.5:
item.a += 'a'
if random.random() < 0.5:
item.b += 'b'
if random.random() < 0.5:
item.c += 'c'
# save the objects
session.bulk_save_objects(items, update_changed_only=False)
# ... do something more
# commit the transaction
session.commit()
Expected behaviour:
The update is performed in one single query.
INFO:sqlalchemy.engine.base.Engine:UPDATE test_foo SET a=?, b=?, c=? WHERE test_foo.id = ?
INFO:sqlalchemy.engine.base.Engine:(('a', 'b', 'c', 1), ('a', 'b', 'c', 2), ('a', '', '', 3), ('a', '', '', 4), ('a', '', '', 5), ('', '', 'c', 6), ('a', 'b', '', 7), ('', '', 'c', 8) ... displaying 10 of 1000 total bound parameter sets ... ('', 'b', '', 999), ('a', 'b', 'c', 1000))
Actual behaviour:
The update is performed in a single statement and then as only somewhat-grouped update statements again.
SQLAlchemy version:
SQLAlchemy==1.2.2
Question:
What can I do to avoid the query being run twice?
expunge
was a good hint, thank you for that. If you'd like, you could submit your commit as an answer so I can accept it.