1

So this has been baffling me for days. I have a Postgres database with the Timescale extension active. I have a table table_name which is partitioned by week on field created_at (date time with timezone) using Postgres data partitioning feature. It had about 244M rows (~300GB). I wanted to take advantage of the Timescaledb extension and move all the data from the data partitions to a hypertable, so I followed the same db migration steps: https://docs.timescale.com/latest/getting-started/migrating-data#same-db :

  • I created a table, table_name_timescale like the table_name but without the primary key on id as the hypertable creation would not work. Ran SELECT create_hypertable('table_name_timescale', 'created_at'); and created the hypertable successfully.
  • Then ran INSERT INTO table_name_timescale SELECT * FROM table_name;, which failed after a few hours because server ran out of disk space. The initial db was ~ 350GB, Server had about 1TB in total, about half of which was free, table_name occupied about ~ 300GBs.

After that the database size went up to ~750GB and I cannot recover that space. table_name_timescale did not contain any rows, neither did all the Timescale-specific tables.

Among the things I've tried:

  • A VACUUM FULL VERBOSE on the table
  • VACUUM FULL VERBOSE on each partition

Example output of VACUUM FULL ANALYZE

VACUUM FULL ANALYZE VERBOSE table_name;
INFO:  vacuuming "public.table_name"
INFO:  "request_sets": found 0 removable, 0 nonremovable row versions in 0 pages
DETAIL:  0 dead row versions cannot be removed yet.
CPU: user: 0.09 s, system: 0.00 s, elapsed: 0.10 s.
INFO:  analyzing "public.table_name"
INFO:  "table_name": scanned 0 of 0 pages, containing 0 live rows and 0 dead rows; 0 rows in sample, 0 estimated total rows
INFO:  analyzing "public.table_name" inheritance tree
INFO:  "table_name_y2020_w23": scanned 19639 of 2653099 pages, containing 163525 live rows and 0 dead rows; 19639 rows in sample, 22091146 estimated total rows
INFO:  "table_name_y2020_w24": scanned 24264 of 3277907 pages, containing 201591 live rows and 0 dead rows; 24264 rows in sample, 27233620 estimated total rows
INFO:  "table_name_y2020_w25": scanned 24970 of 3373276 pages, containing 205052 live rows and 0 dead rows; 24970 rows in sample, 27701121 estimated total rows
INFO:  "table_name_y2020_w26": scanned 21279 of 2874745 pages, containing 170232 live rows and 0 dead rows; 21279 rows in sample, 22997960 estimated total rows
INFO:  "table_name_y2020_w27": scanned 21227 of 2867687 pages, containing 169816 live rows and 0 dead rows; 21227 rows in sample, 22941496 estimated total rows
INFO:  "table_name_y2020_w28": scanned 19487 of 2632630 pages, containing 155896 live rows and 0 dead rows; 19487 rows in sample, 21061040 estimated total rows
INFO:  "table_name_y2020_w29": scanned 19134 of 2584880 pages, containing 153072 live rows and 0 dead rows; 19134 rows in sample, 20679040 estimated total rows
VACUUM

In general, VACUUM FULL does not come up with dead tuples or release any space, especially since the table_name is only used for inserts and no deletes or updates at all.

  • Ran a manual CHECKPOINT to release space from wal - I understand that this is not the correct way but...
  • Retried the insert, but per partition this time INSERT INTO table_name_timescale SELECT * FROM table_name_y2020_w1;, in case this would force the server to re-write/ release space (some of the partitions were inserted successfully but when we reached e.g. week 28, it once more failed with panic and out of space). I didn't want to delete the data in partitions that were successfully inserted because I want to be sure Timescale works as it is supposed to before deleting anything.
  • Discarded some old data that wasn't in use, table size went down to 155GB, 159GB with index and toast.
  • Ran SELECT drop_chunks(interval '1 days', 'table_name_timescale'); to drop anything that could have been inserted into timescale, which of course did nothing.
  • Dropped table_name_timescale, which again did nothing as it did not contain any data, but it did take a lot of time to complete.
  • Checked all timescale related tables to see if there's anything hanging, checked also with VACUUM ANALYZE, nothing.
  • Checked for locks
  • Checked for long running transactions
  • Stopped - restarted the server

I will eventually to back-up - drop db - restore, (I've tried restoring on a different server and the db size was back to normal) but before I do that I would really like to understand what happened here. What I cannot explain or understand is that the table size is 159GB* in total (checking with https://wiki.postgresql.org/wiki/Disk_Usage), the database size is 541GB (SELECT pg_size_pretty( pg_database_size('database_name') );). And of course I can see that under /var/lib/postgresql/data/base/24834 the files occupy the 541GB*.

(* after I deleted some old rows as mentioned above)

The output of:

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
FROM pg_class C
LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
  AND C.relkind <> 'i'
  AND nspname !~ '^pg_toast'
ORDER BY pg_total_relation_size(C.oid) DESC
LIMIT 10;
"public.table_name_y2020_w25"   "26 GB"
"public.table_name_y2020_w24"   "26 GB"
"public.table_name_y2020_w26"   "22 GB"
"public.table_name_y2020_w27"   "22 GB"
"public.table_name_y2020_w23"   "21 GB"
"public.table_name_y2020_w28"   "21 GB"
"public.table_name_y2020_w29"   "20 GB"
"public.models" "1564 MB"
"public.table_name_y2020_w20"   "567 MB"
"public.table_name_y2020_w18"   "97 MB"

Digging into the directory as suggested:

select oid, * from pg_database where oid = 24834;

-- output:
oid datname datdba  encoding    datcollate  datctype    datistemplate   datallowconn    datconnlimit    datlastsysoid   datfrozenxid    datminmxid  dattablespace   datacl
24834   database_name   10  6   en_US.utf8  en_US.utf8  FALSE   TRUE    -1  13043   30663846    293 1663    

And

select relname, relfilenode from pg_class where oid='24834';

outputs nothing.

SELECT pg_relation_filepath('table_name');

pg_relation_filepath
base/24834/343621
select * from pg_class where relfilenode='343621';

relname relnamespace    reltype reloftype   relowner    relam   relfilenode reltablespace   relpages    reltuples   relallvisible   reltoastrelid   relhasindex relisshared relpersistence  relkind relnatts    relchecks   relhasoids  relhaspkey  relhasrules relhastriggers  relhassubclass  relrowsecurity  relforcerowsecurity relispopulated  relreplident    relispartition  relfrozenxid    relminmxid  relacl  reloptions  relpartbound
table_name  2200    24940   0   10  0   343621  0   0   0   0   24943   TRUE    FALSE   p   r   26  0   FALSE   TRUE    FALSE   TRUE    TRUE    FALSE   FALSE   TRUE    d   FALSE   30664717    293         

Disclaimer: Postgres internals noob here, still reading upon how things work. Also, yes, space is tight on the server and ~400GB free space to do this transition is almost ok, but it is what it is unfortunately.. :)

11
  • Have you done this Tablespace?: SELECT * FROM show_tablespaces('table_name_timescale'); Jul 26, 2020 at 19:20
  • 1
    I started at around Postgres 7.2 and I have never seen analyze reclaim. Releasing space back to OS is what VACUUM FULL does. Jul 26, 2020 at 19:48
  • 1
    All I can think of is use pg_ls_dir()Docs9.26.9. Generic File Access Functions to fetch the filenodes and compare them to pg_class.relfilenode using functions hereDocsTable 9.90. Database Object Location Functions. See if there are extraneous files in /base/24834. Jul 26, 2020 at 20:01
  • 1
    " And of course I can see that under /var/lib/postgresql/data/base/24834 the files occupy the 541GB" Dig into that further. Are there files in there not following the usual naming convention? Ones that are not referenced by pg_class.relfilenode?
    – jjanes
    Jul 26, 2020 at 21:31
  • 1
    Thanks @AdrianKlaver and @jjanes for the suggestions, I dug deeper into the dir and the logs and there were a few ERROR: could not extend file "base/24834/271430.5": wrote only 4096 of 8192 bytes at block 658830 errors. I manually moved 271430, 271430.1, 271430.2 ... 271430.5 out of the database dir and the database size dropped about 5GB to 536GB and db is fully functional after restart too. The errors though are not that many to explain all of the additional GB but I will be doing a drop/ restore in a few hours, so I'm relatively ok with the explanation.
    – mkaran
    Jul 29, 2020 at 12:21

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.