0

I have requirement to read the multiple-smaller parquet files of a dataframe(s3-folder) and re-write them into same location with single/more files of size range in b/w min:128mb and Max:900mb.

Appreciate suggestions or solution on this use case.

1 Answer 1

0

You can do that by repartitioning the dataframe and save it s3 as

If you have the count of the dataframe then you can create multiple files as

recordsRequiredPerFIle = 100000
numOfFiles = count/recordsRequiredPerFIle
df.repartition(numOfFiles).write.parquet....
1
  • Requirement is to re partitioning based on the size of the S3 file. what ever no of files it should be reside on single file after re-partitioning. While doing the partitioning the minimum size of the target dataframe would be 128mb and maximum is 900MB. If the file size increased to 900mb then it should move to another file.
    – Govinda
    Jul 23, 2020 at 2:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.