13

So I've read through this question on SO but it does not quite help me any. I want to import a Gmail generated mbox file into another webmail service, but the problem is it only allows 40 MB huge files per import.

So I somehow have to split the mbox file into max. 40 MB big files and import them one after another. How would you do this?

My initial thought was to use the other script (formail) to save each mail as a single file and afterwards run a script to combine them to 40 MB huge files, but still I wouldnt know how to do this using the terminal.

I also looked at the split command, but Im afraid it would cutoff mails. Thanks for any help!

4 Answers 4

18

I just improved a script from Mark Sechell's answer. As We can see, that script can parse the mbox file based on the amount of email per chunk. This improved script can parse the mbox file based on the defined-maximum-size for each chunk.
So, if you have size limitation in uploading or importing the mbox file, you can try the script below to split the mbox file into chunks with specified size*.
Save the script below to a text file, e.g. mboxsplit.txt, in the directory that contains the mbox file (e.g. named mbox):

BEGIN{chunk=0;filesize=0;}
    /^From /{
    if(filesize>=40000000){#file size per chunk in byte
        close("chunk_" chunk ".txt");
        filesize=0;
        chunk++;
    }
  }
  {filesize+=length()}
  {print > ("chunk_" chunk ".txt")}

And then run/type this line in that directory (contains the mboxsplit.txt and the mbox file):

  awk -f mboxsplit.txt mbox

Please note:

  • The size of the result may be larger than the defined size. It depends on the last email size inserted into the buffer/chunk before checking the chunk size.
  • It will not split the email body
  • One chunk may contain only one email if the email size is larger than the specified chunk size

I suggest you to specify the chunk size less or lower than the maximum upload/import size.

7
  • If I use this, is there a way to re-combine the split files? Thanks
    – Ycon
    Jun 26, 2017 at 3:22
  • of course, you can combine them! remember that the split files are text files which can be simply combined. Jul 4, 2017 at 8:32
  • It did work perfectly with the huge mbox files generated by Google Takeout. I use them to import mail in Horde (GoDaddy email accounts). I just changed the size of the chunk mbox files from 40 MB to 100 MB.
    – abiyi
    May 18, 2018 at 12:14
  • 1
    Excellent answer. So fast as well.
    – jangeador
    Feb 12, 2019 at 15:25
  • 3
    an improvement to consider: use sprintf to zero-pad the file name indexes. Something like: {print > ("chunk_" sprintf("%03d",chunk) ".txt");}
    – Sojoodi
    Nov 21, 2019 at 20:51
15

If your mbox is in standard format, each message will begin with From and a space:

From someone@somewhere.com

So, you could COPY YOUR MBOX TO A TEMPORARY DIRECTORY and try using awk to process it, on a message-by-message basis, only splitting at the start of any message. Let's say we went for 1,000 messages per output file:

awk 'BEGIN{chunk=0} /^From /{msgs++;if(msgs==1000){msgs=0;chunk++}}{print > "chunk_" chunk ".txt"}' mbox

then you will get output files called chunk_1.txt to chunk_n.txt each containing up to 1,000 messages.

If you are unfortunate enough to be on Windows (which is incapable of understanding single quotes), you will need to save the following in a file called awk.txt

BEGIN{chunk=0} /^From /{msgs++;if(msgs==1000){msgs=0;chunk++}}{print > "chunk_" chunk ".txt"}

and then type

awk -f awk.txt mbox
17
  • how to make sure theyre about < 40 mb each?
    – Alex
    Jan 23, 2015 at 15:27
  • Try with 10,000 messages and if the files are too big, remove the chunk files and increase the 10,000 to 20,000 before running again. It is not scientific but I guess you don't have to do it every day, so you may need to experiment a bit. Jan 23, 2015 at 15:37
  • can i run this right in the console? and mbox is the file url?
    – Alex
    Jan 23, 2015 at 15:41
  • 1
    I have simplified it - just copy the one line and paste it into the Terminal and press Enter Jan 23, 2015 at 16:18
  • 2
    nope, still the same error: awk: illegal statement at source line 1 i even started to edit your answer to make sure that i didnt copy unwanted chars. there just seems to be something wrong. i have no clue, but to me print > "chunk_" chunk ".txt" looks somehow weird, is this correct syntax?
    – Alex
    Jan 23, 2015 at 17:04
3

formail is perfectly suited for this task. You may look at formail's +skip and -total options

Options
...
+skip
Skip the first skip messages while splitting.
-total
Output at most total messages while splitting.

Depending on the size of your mailbox and mails, you may try

formail -100 -s <google.mbox >import-01.mbox
formail +100 -100 -s <google.mbox >import-02.mbox
formail +200 -100 -s <google.mbox >import-03.mbox

etc.

The parts need not be of equal size, of course. If there's one large e-mail, you may have only formail +100 -60 -s <google.mbox >import-02.mbox, or if there are many small messages, maybe formail +100 -500 -s <google.mbox >import-02.mbox.

To look for an initial number of mails per chunk, try

formail -100 -s <google.mbox | wc
formail -500 -s <google.mbox | wc
formail -1000 -s <google.mbox | wc

You may need to experiment a bit, in order to accommodate to your mailbox size. On the other hand, since this seems to be a one time task, you may not want to spend too much time on this.

0

My initial thought was to use the other script (formail) to save each mail as a single file and afterwards run a script to combine them to 40 MB huge files, but still I wouldnt know how to do this using the terminal.

If I understand you correctly, you want to split the files up, then combine them into a big file before importing them. That sounds like what split and cat were meant to do. Split splits the files based upon your size specification whether based upon line or bytes. It then adds a suffix to these files to keep them in order, You then use cat to put the files back together:

$ split -b40m -a5 mbox  # this makes mbox.aaaaa, mbox.aaab, etc.

Once you get the files on the other system:

$ cat mbox.* > mbox

You wouldn't do this if you want to break the files so messages aren't split between files because you are going to import each file into the new mail system one at a time.

2
  • not quite, i thought formail was a good idea to export each email in an own textfile and from those, create chunks that are about < 40 mb so that i can import them. because split might split the file maybe right in the middle of an email so that i would not get imported correctly
    – Alex
    Jan 23, 2015 at 15:29
  • Split will split an email into two separate files. But, you made it sound like you're recombining the files before importing. If that's the case, it doesn't matter that split split up an individual email because cat would have put patched them back together. split always splits on lines.
    – David W.
    Jan 23, 2015 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.