2

I will be using a large amount of files structured as follows:

/day/hour-min.txt.gz

with a total of 14 days. I will use a cluster of 90 nodes/workers.

I am reading everything with wholeTextFiles() as it is the only way that allows me to split the data appropriately. All the computations will be done on a per-minute basis (so basically per file) with a few reduce steps at the end. There are roughly 20.000 files; How to efficiently partition them? Do I let spark decide?

Ideally, I think each node should receive entire files; does spark do that by default? Can I enforce it? How?

2
  • Where do your input files reside? HDFS/S3/..? Oct 3, 2016 at 8:12
  • HDFS <lengthen comment>
    – Dimebag
    Oct 3, 2016 at 8:20

2 Answers 2

2

I think each node should receive entire files; does spark do that by default?

Yes, given that WholeTextFileRDD(what you get after sc.wholeTextFiles) has its own WholeTextFileInputFormat to read the whole files as a single record, you're covered. If your Spark executors and datanodes are co-located, you can also expect Node-local data locality. (You can check this in Spark UI once your application is running.)

A word of caution from note withing the Spark documentation for sc.wholeTextFiles:

Small files are preferred, large file is also allowable, but may cause bad performance.

2
  • Just to be clear, do you mean I should implement my own wholeTextFileInputFormat? Or is the default implementation good?
    – Dimebag
    Oct 3, 2016 at 12:32
  • 1
    No, you don't need to implement your own input format for this. It's already provided. Just keep in mind that if your file is large then it will probably be split across many hdfs blocks and your RDD will have to read these blocks (perhaps from non local data nodes) to construct a single record for a file. Also for large files the memory needed to process a single file will be large. That is the basic idea behind the caution against using wholeTextFiles for large RDDs. Other than that you're pretty much covered. Oct 3, 2016 at 13:16
1

You could use the rule of thumb for your partitions:

data = data.coalesce(total_cores * 3) 

Ideally, I think each node should receive entire files; does spark do that by default? Can I enforce it? How?

It all depends on your RDD, not of your files. If you build an RDD that contains all the contents of the files for example, then Spark will distribute that RDD, and whether a whole file lies in a node is affected by many parameters (#partitions, size of every file, etc.).

I do not think you can enforce something like that, so focus on the number of partitions; which is critical.


As for the number of files, I had written in my pseudosite, that too few files, will result in huge files and may be just too big, too many files and you will have HDFS maintaining a huge amount of metadata, thus putting a lot of pressure to it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.