1

I made a java agent that generates an xml code containing texts per slide from powerpoint files attached in lotus notes documents. However, when it tries a 66 MB attachment the error is thrown.

Here's the full code in the agent

import java.io.*;
import org.apache.poi.hslf.HSLFSlideShow;
import org.apache.poi.hslf.model.Slide;
import org.apache.poi.hslf.model.TextRun;
import org.apache.poi.hslf.usermodel.SlideShow;
import org.apache.poi.poifs.filesystem.POIFSFileSystem;
import org.apache.poi.xslf.usermodel.XMLSlideShow;
import org.apache.poi.xslf.usermodel.XSLFShape;
import org.apache.poi.xslf.usermodel.XSLFSlide;
import org.apache.poi.xslf.usermodel.XSLFTextParagraph;
import org.apache.poi.xslf.usermodel.XSLFTextShape;

import lotus.domino.*;

public class JavaAgent extends AgentBase {

    public void NotesMain() {
        System.out.println("Start");
        try {
            Session session = getSession();
            AgentContext agentContext = session.getAgentContext();
            DocumentCollection dc = agentContext.getUnprocessedDocuments();
            Document doc = dc.getFirstDocument();
            Database db = agentContext.getCurrentDatabase();
            Document profdoc = db.getProfileDocument("f.SomeProfileDoc", "");
            String siteurl = profdoc.getItemValueString("SomeField");
            //System.out.println(siteurl);
            //ArrayList listFilenames = new ArrayList();
            //int indexnum = 0;
            String extension = "";
            String buildXMLStr = "";
            InputStream fis = null;
            POIFSFileSystem fs = null;
            while (doc != null) {
                RichTextItem body = (RichTextItem)doc.getFirstItem("Attachments");
                if(body != null) {
                    RichTextNavigator rtnav = body.createNavigator();
                    String filename = "";
                    if (rtnav.findFirstElement(RichTextItem.RTELEM_TYPE_FILEATTACHMENT)) {
                        buildXMLStr = "<data>";
                        do {
                            EmbeddedObject att = (EmbeddedObject) rtnav.getElement();
                            filename = att.getSource();

                            int z = filename.lastIndexOf('.');
                            if (z > 0) {
                                extension = filename.substring(z+1);
                            }

                            if(extension.equalsIgnoreCase("ppt") || extension.equalsIgnoreCase("pptx")) {

                                buildXMLStr += "\n<sfile>";
                                buildXMLStr += "<sfextension>" + extension+ "</sfextension>";
                                buildXMLStr += "\n<surl>" + siteurl + "/ContentDocs/_" + doc.getUniversalID() + "/$FILE/" + filename + "</surl>";

                                try {                                   
                                    if(extension.equalsIgnoreCase("ppt")) { // If file is a 2003 powerpoint file
                                        System.out.println("File has ppt extension");
                                        fs = new POIFSFileSystem(att.getInputStream());
                                        HSLFSlideShow show = new HSLFSlideShow(fs);
                                        SlideShow ss = new SlideShow(show);
                                        Slide[] slides = ss.getSlides();
                                        for (int x = 0; x < slides.length; x++) {

                                            buildXMLStr += "\n<slide>";
                                            buildXMLStr += "\n<snum>" + (x + 1) + "</snum>";

                                            if (slides[x].getTitle() == null) {
                                                //System.out.println("Slide " + (x + 1) + " | " + "No Title");
                                                buildXMLStr += "\n<stitle>No Title</stitle>";
                                            } else {
                                                //System.out.println("Slide " + (x + 1) + " | " + slides[x].getTitle());
                                                buildXMLStr += "\n<stitle>" + slides[x].getTitle() + "</stitle>";
                                            }

                                            buildXMLStr += "\n<stexts>";

                                            TextRun[] runs = slides[x].getTextRuns();
                                            for (int i = 0; i < runs.length; i++) {
                                                TextRun run = runs[i];
                                                buildXMLStr += " " + run.getText().replaceAll("[^\\w\\s-]", " ").replaceAll("(?m)^\\s+$", "");
                                            }

                                            buildXMLStr += "\n</stexts>";
                                            buildXMLStr += "\n</slide>";
                                        }
                                        att.getInputStream().close();
                                    }
                                    if(extension.equalsIgnoreCase("pptx")) { // If file is 2007+ powerpoint file
                                        System.out.println("File has pptx extension");
                                        try {
                                            XMLSlideShow pptxshow = new XMLSlideShow(att.getInputStream());
                                            XSLFSlide[] slides = pptxshow.getSlides();
                                            for (int x = 0; x < slides.length; x++) {

                                                buildXMLStr += "\n<slide>";
                                                buildXMLStr += "\n<snum>" + (x + 1) + "</snum>";

                                                if (slides[x].getTitle() == null) {
                                                    // System.out.println("No Title");
                                                    buildXMLStr += "\n<stitle>No Title</stitle>";
                                                } else {
                                                    // System.out.println((x + 1) + slides[x].getTitle());
                                                    buildXMLStr += "\n<stitle>" + slides[x].getTitle() + "</stitle>";
                                                }

                                                buildXMLStr += "\n<stexts>";

                                                for (XSLFShape shape : slides[x]) {
                                                    if (shape instanceof XSLFTextShape) {
                                                        XSLFTextShape txShape = (XSLFTextShape) shape;
                                                        for (XSLFTextParagraph xslfParagraph : txShape.getTextParagraphs()) {
                                                            buildXMLStr += " " + xslfParagraph.getText().replaceAll("[^\\w\\s-]", " ").replaceAll("(?m)^\\s+$", "");
                                                        }
                                                    }
                                                }

                                                buildXMLStr += "\n</stexts>";
                                                buildXMLStr += "\n</slide>";

                                            }
                                            att.getInputStream().close();
                                        } catch(Exception epptx) {
                                            epptx.printStackTrace();
                                        }

                                    }

                                } catch(Exception estart) {
                                    System.out.println("IOE Exception");
                                    estart.printStackTrace();
                                }

                                //System.out.println(filename);

                                buildXMLStr += "\n</sfile>";
                                buildXMLStr += "\n</data>";
                                System.out.println(buildXMLStr);

                            }
                            //indexnum++;
                        } while (rtnav.findNextElement());
                    }
                }
                else {
                    System.out.println("There are no attachments on the current document");
                }
                System.out.println("Next Document");
                doc = dc.getNextDocument();
            }

        } catch(Exception e) {
            System.out.println("General Exception");
            e.printStackTrace();
        }
        System.out.println("Finished scanning reports");
    }
}
7
  • Where do you run out of memory? Your code or POI?
    – Thilo
    Jun 10, 2014 at 23:45
  • You should use a StringBuilder instead of string concatenation to assemble the output. Or, stream the output as you create it, instead of building it all up in memory. Jun 10, 2014 at 23:53
  • Domino version> Is this agent running under the AMGR on a Domino server? Or is it running locally on a client? What is the Notes/Domino version? Also, what platform are you running on, and is it 32 or 64 bit? Is there a JavaMaxHeapSize setting in notes.ini? If so, what is it? Jun 11, 2014 at 4:34
  • I asked the above questions because you can control the amount of memory allocated to heap by the JVM with JavaMaxHeapSize, but on a server you must take into account the fact that the address space that you allocate to the JVM becomes unavailable for anything else. Although a Domino server consists of many processes in separate address spaces, there is a large chunk of shared memory, which means that a large Java heap in the AMGR can cause a shortage of memory across an entire server. Jun 11, 2014 at 4:35
  • BTW, I believe the default for JavaMaxHeapSize is 64 MB, although that may not be true in all versions, plkatforms and 32/64 bit variations. But that would explain why you hit the wall on a 66 MB file. Jun 11, 2014 at 4:37

4 Answers 4

2

Two things I would suggest:

  1. Use a NPOIFSFileSystem instead of a POIFSFileSystem. It uses NIO and should consume less memory.

  2. Don't concatenate your output into strings. If you have to keep it in memory, use a StringBuilder, otherwise immediately write it to an output stream.

If all else fails, increase the memory allocated to your JVM.

2

As Richard Schwartz points out you can increase the amount of memory available to the agent manager by setting JavaMaxHeapSize to something larger than the default value (which is a max of 64 MB as detailed here).

(Please notice that JavaMaxHeapSize does not impact the HTTP JVM. To set the amount of memory available to the HTTP task you need to set HTTPJVMMaxHeapSize).

1
  • It does not impacvt the HTTP JVM, but as I alluded to in my comment, it does impact the HTTP task (and all other server tasks) because reserving that heap size in in AMGR means there is less heap space available for shared memory in AMGR, which means that shared memory in all tasks is limited by the max availble in AMGR. Jun 11, 2014 at 13:17
2

“java.lang.OutOfMemoryError: Java heap space” error is be triggered when you try to add more data into the heap space area in memory, but the size of this data is larger than the JVM can accommodate in the Java heap space.

Java applications are allowed to use only limited amount of memory. This limit is specified during the application startup. To make things more complex, the Java memory is separated to two different regions. These regions are called heap space and permgen

Size of those regions is set during the JVM launch by specifying JVM parameters such as -Xmx and -XX:MaxPermSize. If you do not explicitly set the sizes, platform-specific defaults will be used.

In many cases, the easiest way to get rid of the java.lang.OutOfMemoryError: Java heap space error if you just increase the heap space, specified in the -Xmx parameter, similar to the following

java -Xmx1024m com.yourcompany.YourClass
1

First of all - you should decide how much memory you can give when starting your program (if you don't have enough memory(RAM), no point running the program if you expect the attachments to be big (66mb shouldn't be an issue). If you have to read the whole document in memory and the size is more than memory allocated to your program, then you can't do anything. It will run out of memory.

But - if you can process part of the document and then remove it from memory, then you can load the next part and continue processing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.