Use case and background:
I want to use SELECT GET_LOCK
against a fixed replica, so all of my servers can see the same locks which are against a subset of my data, but distribute the load against mysql to multiple replicas, so that my servers can be working on different subsets of the data. Therefore I have mysql_connector_locks
and mysql_connector_data
.
Each of these connector
objects is a wrapper around a sqlalchemy engine and sessionmaker. They each have
def get_mysql_session(self, isolation_level=None):
if not self.session_maker:
self.session_maker = sessionmaker()
self.session_maker.configure(bind=self.engine)
session = self.session_maker()
if isolation_level is not None:
session.connection(execution_options={'isolation_level': isolation_level.value})
try:
yield session
session.commit()
except Exception:
session.rollback()
raise
finally:
session.close()
Now, I have my code
for data_subset_id in partitioned_data:
with mysql_connector_locks.get_mysql_session() as session_locks:
try:
with get_lock(session_locks, data_subset_id):
with mysql_connector_data.get_mysql_session(
isolation_level=IsolationLevel.READ_UNCOMMITTED
) as session_data:
data = get_data(session_data, data_subset_id)
process_data(data)
except LockNotAcquired:
continue
where get_lock
follows a standard recipe for acquiring a lock.
What is going wrong:
Each server goes through one iteration of the loop, acquires a lock for the second iteration, and errors with MySQLdb._exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')
on the set transaction isolation level
query during the call to mysql_connector_data.get_mysql_session
.
Again: it succeeds on acquiring a lock, getting a subset of data, releasing the lock, acquiring the next one, and then fails to get the data the second time.
Other background
Previously, I was using the same connector for both the lock and the data. This has worked for years with no issues. I'm trying to distribute load for get_data
, and need to keep the locks common to all the servers. If I can't get this to work I'll switch to holding the locks in a common redis server but I prefer not to go this route if possible.
Thanks for any insight!