Main question:
archive_cleanup_command in the postgresql.conf file does not clear the archived wal files. How can I get it to clear the archived wal files?
Relevant information:
- My OS is Linux, Ubuntu v18.04 LTS.
- Database is Postgresql version 13
My current settings:
/etc/postgresql/13/main/postgresql.conf file:
wal_level = replica
wal_compression = on
wal_recycle = on
checkpoint_timeout = 5min
max_wal_size = 1GB
min_wal_size = 80MB
archive_mode = on
archive_command = 'pxz --compress --keep --force -6 --to-stdout --quiet %p > /datadrive/postgresql/13/wal_aerchives/%f.xz'
archive_timeout = 10min
restore_command = 'pxz --decompress --keep --force -6 --to-std-out --quiet /datadrive/postgresql/13/wal_archives/%f.xz > %p'
archive_cleanup_command = 'pg_archivecleanup -d -x .xz /datadrive/postgresql/13/wal_archives %r >> /datadrive/postgresql/13/wal_archives/archive_cleanup_command.log 2>&1'
archive_cleanup_command.log has 777 permissions.
I have a master database doing logical replication with a publication and a slave database subscribing to that publication. It is on the slave that I am intending to do the archiving and restore points.
What am I expecting to happen?
The checkpoint timeout setting in the postgresql.conf file means that a restart point is created atleast every 5 mins. And the archive_timeout setting of 10 mins means that postgresql forces a logfile segment switch after every 10 mins. Therefore, atleast every 10 mins, a restart point is created. Whenever a restart point is created, the archive cleanup command is run. When this command is run it will clear all the .xz files older than this restart point. Therefore the wal_archives directory should not really have .xz files older than 20mins or even 2hours....
What is actually happening?
The
/datadrive/postgresql/13/wal_archives
directory piles up with lots of .xz files that never get cleared.cat archive_cleanup_command.log
shows an empty file. Nothing is ever writing to it.When I run the pg_archivecleanup command manually via bash, it works (i.e. clears all the archive files before the one specified and cat archive_cleanup_command shows the files that were cleared.
Example:pg_archivecleanup -d -x .xz /datadrive/postgresql/13/wal_archives 000000010000045E000000E5 >> /datadrive/postgresql/13/wal_archives/archive_cleanup_command.log 2>&1
Then running
cat archive_cleanup_command.log
gives this:pg_archivecleanup: keeping WAL file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E5" and later pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000DE.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000DF.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E0.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E1.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E2.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E3.xz" pg_archivecleanup: removing file "/datadrive/postgresql/13/wal_archives/000000010000045E000000E4.xz"
What have I tried?
I have tried various permission settings (examples: chmod 777 the wal_archive directory, add other users to the postgres group, etc...)
Extensively and thoroughly read the postgresql documentation and looked atleast 20 different related stackoverflow posts.
Initially tried 7zip cmd line tool to do the zipping instead of pxz.
Successfully restarted the database multiple times using the following commands:
sudo systemctl stop postgresql@13-main sudo systemctl start postgresql@13-main
Dropped the logical replication and re-created the publication on the master and subscription on the slave.
Enabled checkpoints on the master itself.
Looked at
/var/log/postgresql/postgresql-13-main.log
. Unfortunately no relevant errors show up in this log.