5

I am trying to update a larger amount of objects at once (around 1000-1500) using SQLAlchemy's bulk_save_objects method.

Below is a simplified version of the code. The real use case also includes conditional creation of objects, which is why I am not using bulk_update_mappings.

For each object, multiple columns may be modified. In order to batch all updates into one query, I am using update_changed_only=False. I figured this has no drawbacks since all objects are modified anyway.

Code:

import random

from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, scoped_session
import logging

logging.basicConfig()
logger = logging.getLogger('sqlalchemy.engine')
logger.setLevel(logging.DEBUG)

# init sqlalchemy
engine = create_engine('sqlite://')
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
session = Session()

Base = declarative_base()


# simple test model
class Foo(Base):
    __tablename__ = 'test_foo'

    id = Column(Integer, primary_key=True, autoincrement=True)
    a = Column(String, default='')
    b = Column(String, default='')
    c = Column(String, default='')


# recreate database and sample data
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)

items = [
    Foo() for _ in range(1000)
]
session.add_all(items)
session.commit()

# query all items
items = session.query(Foo).all()

# update some items
for item in items:
    if random.random() < 0.5:
        item.a += 'a'
    if random.random() < 0.5:
        item.b += 'b'
    if random.random() < 0.5:
        item.c += 'c'

# save the objects
session.bulk_save_objects(items, update_changed_only=False)

# ... do something more

# commit the transaction
session.commit()

Expected behaviour:

The update is performed in one single query.

INFO:sqlalchemy.engine.base.Engine:UPDATE test_foo SET a=?, b=?, c=? WHERE test_foo.id = ?
INFO:sqlalchemy.engine.base.Engine:(('a', 'b', 'c', 1), ('a', 'b', 'c', 2), ('a', '', '', 3), ('a', '', '', 4), ('a', '', '', 5), ('', '', 'c', 6), ('a', 'b', '', 7), ('', '', 'c', 8)  ... displaying 10 of 1000 total bound parameter sets ...  ('', 'b', '', 999), ('a', 'b', 'c', 1000))

Actual behaviour:

The update is performed in a single statement and then as only somewhat-grouped update statements again.

bulk_update_mappings.log

SQLAlchemy version:

SQLAlchemy==1.2.2

Question:

What can I do to avoid the query being run twice?

2
  • 2
    Don't have a proper answer yet, but since the bulk operations skip the normal unit of work book keeping, I'm pretty sure the reason is that the session still sees those objects as changed, even after the bulk operation. This is backed up by the fact that if you first expunge them all, then perform the bulk save, you get a single UPDATE only. Feb 13, 2018 at 18:23
  • @IljaEverilä expunge was a good hint, thank you for that. If you'd like, you could submit your commit as an answer so I can accept it.
    – Birne94
    Feb 14, 2018 at 9:32

0

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.