0

We've had a master/slave setup for well over 6 months. Replication has never been an issue. The slave is not utilized for anything other than an "insurance policy" if the master goes down. The only activity it has on it is at 2:30 every morning the slave is stopped an a full backup is done, then the slave is restarted. The backup usually takes about 30 minutes, the slave catches back up within 10 minutes.

The slave is a much beefier machine (24 cores v.s 8) and we were just considering to switch it to be the master and reverse the replication this coming weekend.

Yesterday at 9am suddenly the slave began to fall behind. There was no great load on the master. What's really unusual is the load average is around 3 on the slave, there's about a 2% wait-time (shown in top) and around 1/10% of CPU utilization, yet the slave isn't catching up. It looks like it's practically at a standstill. It takes about 10 minutes to process 1 second of replication log (subtracting the seconds behind from actual time). The slave IO thread is keeping up with the master's bin log, it's just the sql thread that is crawling through the queries. And yet, the queries are being processed, doing a watch on slave status shows a continual update of the exec master log pos.

We've tried stopping the slave io thread to see if it would help, it doesn't have any impact. It's as if suddenly every query has just become super expensive.

we've disk-checked the underlying raid, there is nothing in the system or mysql log that indicates any errors at all. We've rebooted and restarted mysql multiple times, cleared the system cache, etc...

This is on a production system that hadn't had any code changes for a week and no abnormal operational issues before this event.

We're at a total loss as to why a system that is no where near peak load can't seem to keep up with the master.

What else should we be looking into? I'll be happy to post system stats, etc here if it will help someone help us determine what's "wrong".

2 Answers 2

0

The last I checked, replication was single-threaded, so I would expect you to find a slow query that's constipating the system. I had a client whose replication was OK, but 10M seconds behind! (oops)

What query shows in a SHOW FULL PROCESSLIST? It is always the same query? If so, maybe that query has become more expensive. Try EXPLAINing it (or a variant, if it's an UPDATE, etc).

If you don't see it right away, try turning on the slow query log and see what you get.

0

It turns out one inodb table had grown rather large and inserts into it were becoming more and more expensive. We switched the table to myisam on the master (and the slave actually) and the slave it caught up. When the "alter table" to convert it to myisam came down to the slave it basically became a no-op.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.