6

I have searched through every documentation and still didn't find why there is a prefix and what is c000 in the below file naming convention:

file:/Users/stephen/p/spark/f1/part-00000-445036f9-7a40-4333-8405-8451faa44319- c000.snappy.parquet

1
  • Maybe Combiner? It would be useful to see any code that outputs information Mar 8, 2018 at 6:58

1 Answer 1

11

You should use "Talk is cheap, show me the code." methodology. Everything is not documented and one way to go is just the code.

Consider part-1-2_3-4.parquet :

  1. Split/Partition number.

  2. Random UUID to prevent collision between different (appending) write jobs.

  3. Unique Job/Task ID (sometimes it will not be included).
  4. The "c" stands for count. This is file counter which means the number of files that have been written in the past for this specific partition. This is used to limit the max number of records written for a single file. The value should start from 0.

I found it based on this code and this code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.