4

Our stack is nodejs with MySQL we're using MySQL connections pooling our MySQL database is managed on AWS aurora . in case of auto failover the master DB is changed the hostname stays the same but the connections inside the pool stays connected to the wrong DB. The only why we found in order to reset the connection is to roll our servers.

this is a demonstration of a solution I think could solve this issue but I prefer a solution without the set interval

const mysql = require('mysql');


class MysqlAdapter {
    constructor() {
        this.connectionType = 'MASTER';
        this.waitingForAutoFaileOverSwitch = false;
        this.poolCluster = mysql.createPoolCluster();
        this.poolCluster.add(this.connectionType, {
            host: 'localhost',
            user: 'root',
            password: 'root',
            database: 'app'
        });

        this.intervalID = setInterval(() => {
            if(this.waitingForAutoFaileOverSwitch) return;
            this.excute('SHOW VARIABLES LIKE \'read_only\';').then(res => {
                // if MASTER is set to read only is on then its mean a fail over is accoure and swe need to switch all connection in poll to secondry database
                if (res[0].Value === 'ON') {
                    this.waitingForAutoFaileOverSwitch = true
                    this.poolCluster.end(() => {
                        this. waitingForAutoFaileOverSwitch = false
                    });
                };
            });
        }, 5000);

    }
    async excute(query) {
        // delay all incoming request until pool kill all connection to read only database
        if (this.waitingForAutoFaileOverSwitch) {
            return new Promise((resolve, reject) => {
                setTimeout(() => {
                    this.excute(query).then(res => {
                        resolve(res);
                    });
                }, 1000);
            });
        }
        return new Promise((resolve, reject) => {

            this.poolCluster.getConnection(this.connectionType, (err, connection) => {
                if (err) {
                    reject(err);
                }
                connection.query(query, (err, rows) => {
                    connection.release();
                    if (err) {
                        reject(err);
                    }
                    resolve(rows);
                });
            });
        });
    }
}



const adapter = new MysqlAdapter();

Is there any other programmable way to reset the connection inside the pool?

Is there any notification we can listing to In case of auto-failover?

4
  • 1
    We had to implement a non opensource package to test whether we were connected to the write or read-only nodes of the cluster, if it detected a failover - it would remove all connections from the pool - iirc correctly the issue was a connection that would try to write to the readonly instance would be evicted from the pool, but once it got down to the last connection, it wouldn't terminate that conn, but subsequent new connections would be directed to the new correct master. Another option is to propagate fatal write errors (if you can tolerate the loss) and allow your app to crash.
    – jeeves
    Jan 26, 2022 at 12:50
  • 1
    depending on the frequency/concurrency of requests for a new connection from the pool, i.e. whether a request for a connection retrieved the old connection or caused a new connection to be recreated, would indicate the frequency of the errors being observed
    – jeeves
    Jan 26, 2022 at 12:56
  • 1
    Why not use Amazon RDS Proxy? Jan 30, 2022 at 17:53
  • i am using haproxy already
    – Naor Tedgi
    Jan 30, 2022 at 19:56

1 Answer 1

3
+100

Instead of manually monitoring the DB health, as you have also hinted, ideally we subscribe to failover events published by AWS RDS Aurora.

There are multiple failover events listed here for the DB cluster: Amazon RDS event categories and event messages

You can use and test to see which one of them is the most reliable in your use case for triggering poolCluster.end() though.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.