2

We all know that InnoDB updates are row-locking and MyISAM ones are table-locking, but how do you explain this? Here are the results of a SHOW FULL PROCESSLIST when the problem is in the middle of occurring:

╔═════════╦════════╦═════════╦════════════════════════════════╗
║ Command ║ Time   ║ State   ║ Info                           ║
╠═════════╬════════╬═════════╬════════════════════════════════╣
║ Query   ║ 0      ║ NULL    ║ SHOW FULL PROCESSLIST          ║
║ Query   ║ 121    ║ end     ║ UPDATE [InnoDB table]          ║
║ Query   ║ 121    ║ update  ║ INSERT INTO [MyISAM table]     ║
║ Query   ║ 121    ║ Locked  ║ INSERT INTO [MyISAM table]     ║
║ Query   ║ 120    ║ Locked  ║ INSERT INTO [MyISAM table]     ║
║ Query   ║ 120    ║ Locked  ║ INSERT INTO [MyISAM table]     ║
╚═════════╩════════╩═════════╩════════════════════════════════╝

I've left some of the columns out to save space. The above all occurs in one database, and refers to a single InnoDB table and a single MyISAM table. There were many other rows below this similar to the bottom three, but again I've left them out.

The problem is started by the UPDATE on the InnoDB table, which then seems to lock the MyISAM table. The database as a whole isn't locked since there were hundreds of SELECT queries occurring each second and they weren't getting stuck. Note that the state of the first INSERT is 'update', which is presumably what is then causing the subsequent inserts to be locked, but that then begs the question: what is the INSERT updating? The auto-increment? Table indexes? Presumably. But why would such an update be blocked by the UPDATE of a different table? See here for details on what the states mean:

http://dev.mysql.com/doc/refman/5.1/en/general-thread-states.html

Here's what I've figured out so far:

  • There's nothing wrong with the queries - the UPDATE is appropriately indexed, and both the INSERT and UPDATE queries normally take < 100 ms to execute, but as you can see this one is taking over 2 minutes.
  • Almost all the queries executed on the database are SELECTs and INSERTs - UPDATEs are the exception.
  • It's definitely the UPDATE that causes the problem, in that it's semi-replicable - if I execute a similar query, sometimes it recreates the problem, sometimes it executes quickly as normal.
  • It's only this particular InnoDB table that seems to be a problem, in that the other InnoDB tables in the database update without a problem, but then it is largest - it contains ~ 1 million rows and is ~ 40 MB in size.
  • The UPDATE is replicated over several servers, but the long execution time only seems to occur on those with significant load.
  • I've checked the CPU load and free RAM on the server while the problem is occurring and they're both fine.

Server specs:

  • Debian Squeeze
  • MySQL 5.1
  • 8 cores
  • 32 GB RAM

Before you ask, here's some of the [mysqld] config settings:

skip-external-locking
innodb_file_per_table
innodb_flush_method     = O_DIRECT
innodb_buffer_pool_size = 512M
key_buffer              = 1500M
read_rnd_buffer_size    = 512k
table_open_cache        = 4096
tmp_table_size          = 256M
max_heap_table_size     = 256M
concurrent_insert       = 2
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
max_connections         = 2000
open_files_limit        = 60000
query_cache_limit       = 1M
query_cache_size        = 512M

After extensive googling I'm still at loss, so any help would be gratefully received!

Added

Create syntax for the InnoDB table:

CREATE TABLE `tablename` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `column1` bigint(20) unsigned NOT NULL,
  `column2` int(10) unsigned NOT NULL,
  `column3` int(10) unsigned NOT NULL,
  `column4` int(10) unsigned NOT NULL,
  `column5` int(10) unsigned DEFAULT NULL,
  `column6` enum('yes','no') NOT NULL DEFAULT 'no',
  PRIMARY KEY (`id`),
  UNIQUE KEY (`column1`),
  KEY (`column2`),
  KEY (`column5`)
) ENGINE=InnoDB AUTO_INCREMENT=993266 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT;
12
  • Are the updates in a transaction or pure update statement? Whats you autocommit setting? Nov 6, 2013 at 10:55
  • You should also increase your innodb_buffer_pool_size significantlly Nov 6, 2013 at 10:57
  • If the queries are all using the same connection this could be the problem. Since each query has to finish before the connection can accept a new one..... Nov 6, 2013 at 10:58
  • @KayNelson The updates are 'pure', i.e. not in a transaction. The autocommit is default so 1. All the queries use different connections. I could increase the buffer pool size, but the total InnoDB data size is ~ 70 MB, so there didn't seem much point - unless you know otherwise?
    – Philip
    Nov 6, 2013 at 11:19
  • Can you please put the table def and the update syntax? Nov 6, 2013 at 11:39

1 Answer 1

0

Since your status is on state 'end'. These things can occur

For the end state, the following operations could be happening according to documentation:

◾ Removing query cache entries after data in a table is changed

◾ Writing an event to the binary log

◾ Freeing memory buffers, including for blobs

I would advise to disable your query cache or set it a lower size. Now its as big as your entire buffer pool.

5
  • Thanks, that sounds plausible - I'll give it a try. I missed the above information as it's under the 'init' state rather than the 'end' state for some reason!
    – Philip
    Nov 7, 2013 at 10:08
  • @Philip, Any progess? Nov 8, 2013 at 12:33
  • 1
    I've implemented the solution, but am going to wait a week or two before I confirm the problem as solved.
    – Philip
    Nov 8, 2013 at 16:16
  • @Philip cool, did you disable completley or did you lower the size of the cache? Good luck i hope it solves your issue! Nov 8, 2013 at 16:34
  • I decided to disable it completely as there's little repetition in the SELECT queries that are executed and they're all v. quick and only return one field anyway. Not being a DBA I hadn't thought it through before!
    – Philip
    Nov 11, 2013 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.