2

Below is the code that I am trying in PySpark

from delta import DeltaTable

delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.logRetentionDuration = "interval 1 days"

After this do we need to save this config or it will be applicable automatically. How we can check what is current logRetentionDuration set for table. I tried below to get properties info

delta_table.detail()

But it return empty {}

0

1 Answer 1

3

Use Spark SQL:

spark.sql("ALTER TABLE delta.`path\to\delta\table` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')")
spark.sql("DESCRIBE DETAIL delta.`path\to\delta\table\path`").show(truncate=False)

Under the "properties" column you can see the log retention duration of the specified delta table.

Note: If the retention period was not set properly, the above SQL command returns {} in the properties column.

3
  • I did that it return {} in properties column.
    – YOGESH
    Mar 15, 2023 at 10:31
  • Try this spark.sql("ALTER TABLE delta.`path\to\delta\table` SET TBLPROPERTIES ('delta.logRetentionDuration'='1 days')") and then use the decribe command. Mar 15, 2023 at 10:36
  • Thanks it set property. I accept answer. Is there any API option to do the same using DeltaTable object.
    – YOGESH
    Mar 15, 2023 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.