I'm using Python to populate some tables in a MySQL 5.7.26. Python is unrelated, though. The issue occurs when running the queries by hand on the database as well.
I am adding data to some temporary table, say tmp_source
and, due to the lack of a MERGE
statement, I then do an INNER JOIN UPDATE
on the destination table destination
followed by a LEFT JOIN INSERT
. The queries would look something like this:
UPDATE destination D
INNER JOIN tmp_source S
ON D.col1 = S.col1
AND D.col2 = S.col2
SET D.col3 = S.col3,
D.col4 = S.col4;
INSERT INTO destination
SELECT * FROM tmp_source S
LEFT JOIN destination D
ON D.col1 = S.col1
AND D.col2 = S.col2
WHERE D.col1 IS NULL;
The intention is to update existing rows and insert missing ones, like a MERGE. But while the inserts work fine, updates sometimes fail due to:
1071, 'Specified key was too long; max key length is 1024 bytes'
I'm not allowed to modify the destination
table structurally. It has some poorly made indexes (like VARCHAR(256)
for columns that really never get over 50 characters) and I know this poses a problem.
I tried creating tmp_source
with and without indexes - no dice.
Is there any way I can overcome this without altering the destination
table? I have an "it's stupid but it works so it's not stupid" idea of generating static updates and just running those, such as:
UPDATE desination
SET col3 = val3,
col4 = val4
WHERE col1 = val1
AND col2 = val2
The idea is that I'm not handling an obscene amount of data. I could just catch this particular exception and only do this for tables where this occurs. Problematic tables will generally have less than 200 rows. And speed is not really of the essence in this case.
But it feels like such a workaround (probably because it is). Does anyone have a better idea?