We have a 12 node cassandra cluster in Production. In the recent past, almost all the nodes are using higher than 85% disc space. We tried to add default_time_to_live, gc_grace_seconds for few tables. But there seems to be no effect on the count of records or the disc space. There are suggestions to perform nodetool compact and cleanup. But this also mentions that it is not suggested to be run on production environment.
Some specific questions,
- Tried setting the TTL as 100 days and gc as 3hours. Expectation was records older than 90 days should get deleted after 3 hours. But it was still intact. Is there anything else to be taken care to delete records older than 100 days using TTL settings? Disc space is also expected to be freed up. Again what else should be done to free up disc space after deleting records.
ALTER TABLE my_keyspace.my_item WITH default_time_to_live=8640000
ALTER TABLE my_keyspace.my_item WITH gc_grace_seconds=10800
- Is it ok to run nodetool compact followed by nodetool cleanup on a prod environment with all instances over 85% disc space utilized?
Please share other suggestions as well to free up disc space utilized by Cassandra.
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.1.x.x 997.26 GiB 256 24.7% erff8abf-16a1-4a72-b63e-5c4rg2c8d003 rack1
UN 10.2.x.x 1.22 TiB 256 26.1% a8auuj76-f635-450f-a2fd-7sdfg0ss713e rack1
UN 10.3.x.x 1.21 TiB 256 25.4% 8ebas25c-4c0b-4be9-81e3-013fasdas255 rack1
UN 10.4.x.x 1.27 TiB 256 25.1% wwwdba15-16f3-41a8-b3d1-2d2b6e35715d rack1
UN 10.5.x.x 975.67 GiB 256 24.7% 72ed4df7-fb65-4332-b8ac-e7461699f633 rack1
UN 10.6.x.x 1.01 TiB 256 24.8% 39803f58-127f-453b-b102-ed7bdfb8afb2 rack1
UN 10.7.x.x 1.18 TiB 256 25.9% b6e692a6-249f-433d-8b54-1d20d4bc4962 rack1
UN 10.8.x.x 1.12 TiB 256 24.5% 8ed8c306-9ac9-4130-bff1-97f7d5d9a02f rack1
UN 10.9.x.x 973.26 GiB 256 24.4% f7489923-3cc3-43ec-83ca-42bbdeb0cbb7 rack1
UN 10.10.x.x 1.13 TiB 256 26.0% ea694224-ds0b-42f5-9acf-ff4ddfb450e0 rack1
UN 10.11.x.x 1.22 TiB 256 24.0% ddde4bce-553e-4246-9920-47sdfdf324ed rack1
UN 10.12.x.x 1.28 TiB 256 24.4% 0222d40f-edb8-4710-9bae-39dsfd87e18db rack1