7

In SQL there is:

REFRESH TABLE table_name

and in pyspark the equivalent is:

spark.refreshTable(table_name)

So, what would be the Pyspark's equivalent to:

MSCK REPAIR TABLE table_name
2
  • 1
    spark.sql('MSCK REPAIR TABLE table_name')
    – Steven
    Commented May 16, 2019 at 14:34
  • I know you can execute sql through spark.sql(), but that was not my question. I want to know a pyspark function exclusive for that. Commented May 16, 2019 at 15:15

1 Answer 1

9

refreshTable is integrated with spark session catalog.

as steven suggested, you can go with spark.sql('MSCK REPAIR TABLE table_name')

There is something called recoverPartitions (Only works with a partitioned table, and not a view) in the above link. I believe this is aliased version of msck repair table. you can go ahead and try this..

see this ddl.scala seems like its equalent by documentation.
example usage :

spark.catalog.recoverPartitions(tableName) 
4
  • 2
    I think that recoverPartitions is what I am looking for. Will test it and come back later to confirm. Commented May 16, 2019 at 19:01
  • @Ram Ghadiyaram could you please throw some more light on recoverPartitions.. how does it works and is it available with pyspark Commented May 17, 2019 at 0:57
  • 1
    @vikrantrana yes it is available with pyspark. Just follow the link of the catalog doc in the answer.
    – Steven
    Commented May 17, 2019 at 9:17
  • 1
    @vikrantrana you just need to get your context and its catalog, In my case I used Spark Context: spark.catalog.recoverPartitions() Commented May 17, 2019 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.