We have a PostgreSQL setup with 3 node in each availability of zone and each availability zone is replicated across Availability zone. All replication are hot standby with async replication. We leverage replication slots for each slave AZ cluster.
We often do automated rolling upgrades of DB containers. After sometime, slaves stops replicating from master cluster with messages like following:
2019-08-06 16:57:11.061 UTC [2987482] LOG: replication terminated by primary
server
2019-08-06 16:57:11.061 UTC [2987482] DETAIL: End of WAL reached on timeline 3
at 0/293D7AE0.
2019-08-06 16:57:11.136 UTC [2987477] LOG: new timeline 4 forked off current
database system timeline 3 before current recovery point 0/293D7D28
I am looking for some pointers to fix this problem.
We tried to scale down each cluster to single node in each AZ and we dont see this problem there. My understanding is that since replication across AZ happens with a proxy in front, during rolling upgrade once the master changes, the slaves are not sure how to continue replication.
I wanted to understand if there is something in protcol during replication which can handle something like this.
recovery.conf on slaves looks like:
standby_mode = 'on'
primary_conninfo = 'host=foo-bar.com port=5432 user=repluser password=
sslmode=verify-ca sslcert=/etc/db_repl_secrets/tls.crt
sslkey=/etc/db_repl_secrets/tls.key sslrootcert=/etc/db_repl_secrets/ca.crt'
primary_slot_name = 'stolon_us_east_1'
recovery_target_timeline = 'latest'