So this has been baffling me for days.
I have a Postgres database with the Timescale extension active. I have a table table_name
which is partitioned by week on field created_at
(date time with timezone) using Postgres data partitioning feature. It had about 244M rows (~300GB). I wanted to take advantage of the Timescaledb extension and move all the data from the data partitions to a hypertable, so I followed the same db migration steps: https://docs.timescale.com/latest/getting-started/migrating-data#same-db :
- I created a table,
table_name_timescale
like thetable_name
but without the primary key onid
as the hypertable creation would not work. RanSELECT create_hypertable('table_name_timescale', 'created_at');
and created the hypertable successfully. - Then ran
INSERT INTO table_name_timescale SELECT * FROM table_name;
, which failed after a few hours because server ran out of disk space. The initial db was ~ 350GB, Server had about 1TB in total, about half of which was free,table_name
occupied about ~ 300GBs.
After that the database size went up to ~750GB and I cannot recover that space. table_name_timescale
did not contain any rows, neither did all the Timescale-specific tables.
Among the things I've tried:
- A
VACUUM FULL VERBOSE
on the table VACUUM FULL VERBOSE
on each partition
Example output of VACUUM FULL ANALYZE
VACUUM FULL ANALYZE VERBOSE table_name;
INFO: vacuuming "public.table_name"
INFO: "request_sets": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL: 0 dead row versions cannot be removed yet.
CPU: user: 0.09 s, system: 0.00 s, elapsed: 0.10 s.
INFO: analyzing "public.table_name"
INFO: "table_name": scanned 0 of 0 pages, containing 0 live rows and 0 dead rows; 0 rows in sample, 0 estimated total rows
INFO: analyzing "public.table_name" inheritance tree
INFO: "table_name_y2020_w23": scanned 19639 of 2653099 pages, containing 163525 live rows and 0 dead rows; 19639 rows in sample, 22091146 estimated total rows
INFO: "table_name_y2020_w24": scanned 24264 of 3277907 pages, containing 201591 live rows and 0 dead rows; 24264 rows in sample, 27233620 estimated total rows
INFO: "table_name_y2020_w25": scanned 24970 of 3373276 pages, containing 205052 live rows and 0 dead rows; 24970 rows in sample, 27701121 estimated total rows
INFO: "table_name_y2020_w26": scanned 21279 of 2874745 pages, containing 170232 live rows and 0 dead rows; 21279 rows in sample, 22997960 estimated total rows
INFO: "table_name_y2020_w27": scanned 21227 of 2867687 pages, containing 169816 live rows and 0 dead rows; 21227 rows in sample, 22941496 estimated total rows
INFO: "table_name_y2020_w28": scanned 19487 of 2632630 pages, containing 155896 live rows and 0 dead rows; 19487 rows in sample, 21061040 estimated total rows
INFO: "table_name_y2020_w29": scanned 19134 of 2584880 pages, containing 153072 live rows and 0 dead rows; 19134 rows in sample, 20679040 estimated total rows
VACUUM
In general, VACUUM FULL does not come up with dead tuples or release any space, especially since the table_name
is only used for inserts and no deletes or updates at all.
- Ran a manual
CHECKPOINT
to release space from wal - I understand that this is not the correct way but... - Retried the insert, but per partition this time
INSERT INTO table_name_timescale SELECT * FROM table_name_y2020_w1;
, in case this would force the server to re-write/ release space (some of the partitions were inserted successfully but when we reached e.g. week 28, it once more failed withpanic
and out of space). I didn't want to delete the data in partitions that were successfully inserted because I want to be sure Timescale works as it is supposed to before deleting anything. - Discarded some old data that wasn't in use, table size went down to 155GB, 159GB with index and toast.
- Ran
SELECT drop_chunks(interval '1 days', 'table_name_timescale');
to drop anything that could have been inserted into timescale, which of course did nothing. - Dropped
table_name_timescale
, which again did nothing as it did not contain any data, but it did take a lot of time to complete. - Checked all timescale related tables to see if there's anything hanging, checked also with
VACUUM ANALYZE
, nothing. - Checked for locks
- Checked for long running transactions
- Stopped - restarted the server
I will eventually to back-up - drop db - restore, (I've tried restoring on a different server and the db size was back to normal) but before I do that I would really like to understand what happened here. What I cannot explain or understand is that the table size is 159GB* in total (checking with https://wiki.postgresql.org/wiki/Disk_Usage), the database size is 541GB (SELECT pg_size_pretty( pg_database_size('database_name') );
). And of course I can see that under /var/lib/postgresql/data/base/24834
the files occupy the 541GB*.
(* after I deleted some old rows as mentioned above)
The output of:
SELECT nspname || '.' || relname AS "relation",
pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
AND C.relkind <> 'i'
AND nspname !~ '^pg_toast'
ORDER BY pg_total_relation_size(C.oid) DESC
LIMIT 10;
"public.table_name_y2020_w25" "26 GB"
"public.table_name_y2020_w24" "26 GB"
"public.table_name_y2020_w26" "22 GB"
"public.table_name_y2020_w27" "22 GB"
"public.table_name_y2020_w23" "21 GB"
"public.table_name_y2020_w28" "21 GB"
"public.table_name_y2020_w29" "20 GB"
"public.models" "1564 MB"
"public.table_name_y2020_w20" "567 MB"
"public.table_name_y2020_w18" "97 MB"
Digging into the directory as suggested:
select oid, * from pg_database where oid = 24834;
-- output:
oid datname datdba encoding datcollate datctype datistemplate datallowconn datconnlimit datlastsysoid datfrozenxid datminmxid dattablespace datacl
24834 database_name 10 6 en_US.utf8 en_US.utf8 FALSE TRUE -1 13043 30663846 293 1663
And
select relname, relfilenode from pg_class where oid='24834';
outputs nothing.
SELECT pg_relation_filepath('table_name');
pg_relation_filepath
base/24834/343621
select * from pg_class where relfilenode='343621';
relname relnamespace reltype reloftype relowner relam relfilenode reltablespace relpages reltuples relallvisible reltoastrelid relhasindex relisshared relpersistence relkind relnatts relchecks relhasoids relhaspkey relhasrules relhastriggers relhassubclass relrowsecurity relforcerowsecurity relispopulated relreplident relispartition relfrozenxid relminmxid relacl reloptions relpartbound
table_name 2200 24940 0 10 0 343621 0 0 0 0 24943 TRUE FALSE p r 26 0 FALSE TRUE FALSE TRUE TRUE FALSE FALSE TRUE d FALSE 30664717 293
Disclaimer: Postgres internals noob here, still reading upon how things work. Also, yes, space is tight on the server and ~400GB free space to do this transition is almost ok, but it is what it is unfortunately.. :)
analyze reclaim
. Releasing space back to OS is whatVACUUM FULL
does.pg_class.relfilenode
using functions hereDocsTable 9.90. Database Object Location Functions. See if there are extraneous files in/base/24834
.ERROR: could not extend file "base/24834/271430.5": wrote only 4096 of 8192 bytes at block 658830
errors. I manually moved271430, 271430.1, 271430.2 ... 271430.5
out of the database dir and the database size dropped about 5GB to 536GB and db is fully functional after restart too. The errors though are not that many to explain all of the additional GB but I will be doing a drop/ restore in a few hours, so I'm relatively ok with the explanation.