-2

Could somebody help me to get some logic to following in perl I am using windows 7.

C:\script>perl split_concatenate.pl large_file a or b (Input would be large file and value a or b to process it later).

  1. Check the file if it is greater than 40KB (some size), and choice is "a" , if not run a command

command -i large_file.txt -o large_file_new -a

else if the choice is b

command -i large_file.txt -o large_file_new -b

else

if it is greater say 40KB+, split the file for each 40KB arround (will be part1,) and append a first "particular string" which will be in the file_part2 to the part1 save it for processing, if there are multiple "particular string" then create subsequent files which should end with next "particular string" in the following part. ("Particular String" starts with some String but ends in different value). So script should search if there are more "Particular string", in the part2 or so and append first available one, if there is only one available no need to anything just split. As file always should end with a particular string.

Then process same command

command -i filepart1.txt -o filepart1.dat -a command -i filepart2.txt - o filepart2.dat (if needed) -a

or

command -i filepart1.txt -o filepart1.dat -b command -i filepart2.txt - o filepart2.dat (if needed) -b

After this needs to be concatenated.

Concatenate filepart1.dat + filepart2.dat + filepartN =large_file.dat

I started to find the size first using below code,

#!/usr/bin/perl
    use strict;
    use warnings;
    use File::stat;

    my $filesize = stat("Full_File.txt")->size;

    print "Size: $filesize\n";

    exit 0;

It will be great if some one help so I can learn. If this is not possible, then @ each 500th line the file reaches to 40KB, so I think this would be easier, every 500th line append the Next available "Particular String", and split and process above command, if file is less than 1000 lines then only 2 split and no need to append in the part2 as already it has one in its eof. May be easier?

Better explanation:

large_file.txt

xxx
xxxx
xxxx
xxxxx
var:value_var_v(1234)
xxxxxx
xxxxx
xxxxx
var:value_var_v(4567)
xxxxxxx
xxxxxx
var:value_var_v(abcd)
xxxxxxx  // first split happens here as here assume it is 40kb
xxxxxx
xxxxxxx
var:value_var_v(efgh)

If this is too big then split at line 5, say large_files_part1. Its end should contain var:value_var_v(1234). After the 5th line it should split again at line 9 and will become large_files_part2 and have var:value_var_v(4567) at the end. part3 wil go till line 12 and include var:value_var_v(abcd) at the end, and so on. If there is only one var:value_var_v? after the first split then only two parts is fine as long as the lines in both parts is arround 500. If there are say 1300 line in the main file then three splits are needed. The end of each file should have the next available "string", so the 1001st line will be the first var:value_var_v(1234) available after line 1000. String always starts with var:value_var_v, end with any thing. Hope this is clear.

Output Expected: First case: So, out put will be, _part1.txt will be arround 40,000 if it had only one occurrence of string

xxx
xxxx
xxxx
xxxxx
var:value_var_v(1234)
xxxxxx
xxxxx
xxxxx
var:value_var_v(4567)
xxxxxxx
xxxxxx
var:value_var_v(abcd)
xxxxxxx  // split happened here
var:value_var_v(efgh)

_part2.txt
xxxxxxx
xxxxx
var:value_var_v(efgh)

After I do some process on these files (part1 and par2) I again concatenate

_part1+_part2=large+file

Final large_file after concatenation:

    xxx
    xxxx
    xxxx
    xxxxx
    var:value_var_v(1234)
    xxxxxx
    xxxxx
    xxxxx
    var:value_var_v(4567)
    xxxxxxx
    xxxxxx
    var:value_var_v(abcd)
    xxxxxxx  // split happened here
    var:value_var_v(efgh)
    ****
  xxxxxxx
  xxxxx
  var:value_var_v(efgh)

2nd splitting and concatenate case:

If that file is too big say 80KB and has many strings "var:value_var() after a first split @40KB,do subsequent splits where it sees a next string which will be again "var:value_var_v()" and do a split, based on the string else based on the size. Everytime the file pat shoudl contain next available var:value_var_v().

Orginal file:

    xxx
    xxxx
    xxxx
    xxxxx
    var:value_var_v(1234)
    xxxxxx
    xxxxx
    xxxxx
    var:value_var_v(4567)  
    xxxxxxx 
- - // assume now split happens here as here assume it is 40kb there are two more strings starting with var:value_var_v, split after var:value_var_v(abcd) and print this string in previous parts eof. Then final part will be ending with var:value_var_v(efgh). keep as it is.
    xxxxxx
    xxxxxx
    xxxxx
    var:value_var_v(abcd)
    xxxxxxx  
    xxxxxx
    xxxxxxx
    var:value_var_v(efgh)



part1.txt 
   xxx
    xxxx
    xxxx
    xxxxx
    var:value_var_v(1234)
    xxxxxx
    xxxxx
    xxxxx
    var:value_var_v(4567)  
    xxxxxxx - - // split happens here as here assume it is 40kb
    var:value_var_v(abcd) - //prints next available string which is var:value_var_v(abcd)
    _part2.txt
    xxxxxx
    xxxxxx
    xxxxx
    var:value_var_v(abcd)  

// Here part1 and part2 ends with same string.

    _part3.txt
    xxxxxxx  
    xxxxxx
    xxxxxxx
    var:value_var_v(efgh) - This is last part and size should be below 40KB 

process all of these part1,part2,part3 then concatenate to a big file.

Final file after concatenating

Full fille would look like.

xxx
    xxxx
    xxxx
    xxxxx
    var:value_var_v(1234)
    xxxxxx
    xxxxx
    xxxxx
    var:value_var_v(4567)  
    xxxxxxx // split happened here in the first split assumed 40KB 
    var:value_var_v(abcd) 
      ******
    xxxxxx
   xxxxxx
    xxxxx
    var:value_var_v(abcd)  
     ******
   xxxxxxx  
    xxxxxx
    xxxxxxx
    var:value_var_v(efgh)
   ******

PS: Once I get processed end of in each part I get **** a unique value and retain it as it during concatenation.

10
  • What is the "particular string"? You said it will be in file_part2, but how does the script find it?
    – Barmar
    Feb 2, 2013 at 7:48
  • This is a string that's hard-coded into the script, a pattern that it looks for in the file, or a parameter to the script? It's already in the file, or it's something that the script adds to the file?
    – Barmar
    Feb 2, 2013 at 8:06
  • Its hard coded in the file, and needs to go to the end of the file if it is split. Basically the script should take the first available pattern if any after the split. In case if there are many similar strings after 500th line and before 1000th line it should split and append first available "var:value_var_v1" @ the end of split file.
    – sharp
    Feb 2, 2013 at 8:09
  • I still don't understand. You say "append a first particular string", which means the script adds it. But you also say "script should search if there are more particular string", which means that it's already in the file. I think it would help if you showed some sample data. Show what the original file would look like, and what you want the result to be like.
    – Barmar
    Feb 2, 2013 at 8:17
  • Sorry, Let me put this way. @ the end of each file that "string" should be there.
    – sharp
    Feb 2, 2013 at 8:30

1 Answer 1

1

I think this is pretty close to what you want.

my $part_string = 'var:value_var_v';
my $file_count = 1;
my $total_length = 0;
my $max_length = 40000;

open (my $input, '<', $input_filename);
open (my $output, '>', "${output_filename}_part_${file_count}");
while (my $line = <$input>) {
  print $output $line;
  $total_length += length($line);
  if ($total_length > $max_length) {
    print $output "$part_string$file_count\n";
    close $output;
    $file_count++;
    open ($output, '>', "${output_filename}_part_${file_count}");
    $total_length = 0;
  }
}
if ($total_length > 0) {
  print $output "$part_string$file_count\n";
}
close $output;
16
  • Thanks, but how would I run this? I tried nothing happened.
    – sharp
    Feb 2, 2013 at 18:39
  • I changed the value $input_filename, to my test.txt it created _part_1 but nothing in it.
    – sharp
    Feb 2, 2013 at 18:46
  • When I enabled warnings, Global symbol "$full" requires explicit package name at split.pl line 9. Global symbol "$output_filename" requires explicit package name at split.pl line 10. Global symbol "$output_filename" requires explicit package name at split.pl line 18. Bareword "test" not allowed while "strict subs" in use at split.pl line 9. Execution of split.pl aborted due to compilation errors. I am using windows.
    – sharp
    Feb 2, 2013 at 18:52
  • This is just an excerpt from the program, you need to write the rest. I expect you to initialize the ifilename variables from the arguments or however you want. I don't have a variable $full, so I don't know what you did to cause that -- but the error usually means that you forgot to use my when initializing it. I assume you know how to program, and are just having trouble with this algorithm.
    – Barmar
    Feb 2, 2013 at 19:50
  • I can hardly understand what you're trying to do, I'm not going to write the whole thing for you. If you don't know perl that well, use another language that you know better. My script should help you with the general algorithm, which is the same in any language.
    – Barmar
    Feb 2, 2013 at 20:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.