I am currently working with some files to parse with a Scala app. The problem is that the files are too large so they always end up throwing an exception in the heap size (and I've tried with the max heap size I can and still no use).

Now, the files looks like this:

This is
one paragraph
for Scala
to parse

This is
another paragraph
for Scala
to parse

Yet another
paragraph

And so on. Basically I would like to take all this files and split them in 10 or 20 pieces each, but I have to be sure a paragraph is not splitted in half in the results. Is there any way of doing this?

Thank you!

share|improve this question
    
See this: stackoverflow.com/questions/4255021/… – Brian Mar 26 '14 at 22:21
    
That's not use for me :/ I already do it. Still, I have to have all the parsed paragraph in the app, it is simpler to have smaller files than to change all my program logic (plus I don't have much time to do it) – crscardellino Mar 26 '14 at 22:27
    
The question is tagged with bash: So what about splitting it in a separate Scala program? – Beryllium Mar 27 '14 at 8:18
1  
I finally solved the way Brian said, with a lazy iterator. Had to change some of the logic of the program but wasn't as much as I originally thought. – crscardellino Mar 27 '14 at 11:32
up vote 1 down vote accepted

Here's an awk script that will break up input files into batch_size blocks ( with a garbage trailing record separating newline ). Put this into a file and change it into an executable:

#!/usr/bin/awk -f

BEGIN {RS=""; ORS="\n\n"; last_f=""; batch_size=20}

# perform setup whenever the filename changes
FILENAME!=last_f {r_per_f=calc_r_per_f(); incr_out(); last_f=FILENAME; fnum=1}

# write a record to an output file
{print $0 > out}

# after a batch, change the file name
(FNR%r_per_f)==0 {incr_out()}

# function to roll the file name
function incr_out() {close(out); fnum++; out=FILENAME"_"fnum".out"}

# function to get the number of records per file
function calc_r_per_f() {
    cmd=sprintf( "grep \"^$\" %s | wc -l", FILENAME )
    cmd | getline rcnt
    close(cmd)
    return( sprintf( "%d", rcnt/batch_size ) )
    }

You would change the batch_size element in the begin block to adjust the number of output files per input file and the output file name itself can be altered by changing the out= assignment in incr_out().

If you put it into a file called awko, you would run it like awko data1 data2 and get files like data2_7.out for example. Of course the output names are more horrible than that if your input file names have extensions etc.

share|improve this answer
    
Very complicated to make what we (me,fredtantini...) purposed in our comments. That does exactly the same... isn't it ? - sorry, I just understood that you allow to split in "n" files, as we set "n" paragraphs in our solutions – Metal3d Mar 28 '14 at 13:32
    
It's not so complicated (only 3 actions) and is indeed similar. It allows for ~n individual files from each input file like the other solutions. It differs in that it close()s each file when done with them, the output files are related to the input file names and each input file is broken up into roughly the same number of sub-files as the others(calc_r_per_f()). It should work the same when called like awko data1 data2 data3 data4 ... as awko data1 + awko data2 +... . The functions reduce copied code and keep the action blocks short. Let me know if I can improve the comments. – n0741337 Mar 28 '14 at 18:05

csplit file.txt /^$/ {*}

csplit splits a file separated by the specified pattern.

/^$/ matches empty lines.

{*} repeats the previous pattern indefinitely.

share|improve this answer

to split every 3 paragraphs:

awk 'BEGIN{nParMax=3;npar=0;nFile=0}
     /^$/{npar++;if(npar==nParMax){nFile++;npar=0;next}}
     {print $0 > "foo."nFile}' foo.orig

to split every 10 lines:

awk 'BEGIN{nLineMax=10;nline=0;nFile=0}
    /^$/{if(nline>=nLineMax){nFile++;nline=0;next}}
    {nline++;print $0 > "foo."nFile}' foo.orig
share|improve this answer

You can use "split" command, but as you want to split paragraphs, you can use this kind of script:

awk -v RS="\n\n" 'BEGIN {n=1}{print $0 > "file"n++".txt"}' yourfile.txt

That split each paragraph in file named "file1.txt", "file2.txt", and so on...

To set "n++" each "N" paragraph, you can do:

awk -v RS="\n\n" 'BEGIN{n=1; i=0; nbp=100}{if (i++ == nbp) {i=0; n++} print $0 > "file"n".txt"}' yourfile.txt

Just change "nbp" value to setup the paragraph numbers

share|improve this answer
    
The problem with this way is that paragraphs can become too many (a file can hold oveer 100k paragraph and the files are over 50) :/ – crscardellino Mar 26 '14 at 22:51
    
You can update "n" each N paragraph, I can edit my command to set it up – Metal3d Mar 28 '14 at 13:20
    
csplit allows you to split by pattern instead of size. – Marco Roy Jan 18 '16 at 21:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.