6

I have a tomcat 6.0.20, apr 1.2, jdk 1.6.0_15 with mysql 5.1.38 running on a rhel box with 4 GB ram. There is one simple jsp/servlet application on it with 5 users, one struts 1.2.0.9 with 64 users on it and one struts 2.0 application with 35 users on it. The struts 2.0 users make an entry every second, about 900 entries in a day. I am also using toplink for persistence in the last two applications. I have declared all non referenced objects to null, in the code, have applied production values for config files from the struts 2 site and from the tomcat site. Applied thread caching in mysql, reduce wait_timeout and interactive_timeout to be equivalent to session timeout of tomcat.Increased file descriptors in linux. Reworked queries. Examined the thread dump, watched gc stats, applied above changes on basis of this,

YET still facing "java.lang.OutOfMemoryError" error.

at different times its for different things, sometimes its Servlet.service(), sometimes its image.servlet, sometimes it jasper that causes it.

extremely frustrtating, as the errors are not constant but keep changing over time

Any help please would be greatly appreciated!!!

JAVA_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+CMSParallelRemarkEnabled (tomcat manager reports 34 mb empty so have not used permsize, mx and mn etc.)

persistence.xml

 <property name="toplink.jdbc.url" value="jdbc:mysql://localhost:3306/dbname?autoReconnect=false"/>

server.xml

<Connector port="80" protocol="HTTP/1.1" connectionTimeout="2000" redirectPort="8443" compression="on" compressableMimeType="application/octet-stream,text/html,text/xml,text/plain,application/x-javascript,image/gif,text/css,image/gif,application/vnd.ms-excel,application/pdf" enableLookups="false"/>

context.xml

<Context reloadable="false" delegate="false" privileged="false">
3
  • Are you changing max heap size from the default? I don't see a -Xmx argument.
    – matt b
    Oct 9, 2009 at 12:43
  • Is it a PermGen problem? PermGen has a limit different from the max heap setting. So it is entirely possible to run out of PermGen space and still have 34MiB unused.
    – Stroboskop
    Oct 9, 2009 at 12:58
  • And from my experience you need to completely restart JBoss/Tomcat about every third application reload or face a PermGen problem. So now we restart JBoss every time we deploy just to be sure.
    – Stroboskop
    Oct 9, 2009 at 13:01

4 Answers 4

5

First, you must make sure which process is eating all the CPU. Is it really the java program? Try top(1) to figure this out if you haven't already.

If you have and you're sure that it's the Java program, enable remote debugging. When the CPU boils over next time, connect and make sure that no thread is in an infinite loop.

If that's not the case, you're out of memory (no matter what the tomcat manager says). Start a JMX console and check the various memory spaces. My guess is that the permgen is pretty full (are there many big JARs on your classpath? Or are you using cglib somewhere?) When that happens (and since you have perm gen GC enabled), the Java VM will try to free memory in the perm gen space all the time.

Unless you are using something which creates class files at runtime (a scripting language like Groovy, for example), that will not work: In normal Java programs, classes can never be GC'd. If you still do it, the GC will run and run and achieve nothing but burn all CPU power.

1

Perhaps it's the same problem addressed by 1 and 2. Try to create only small transactions.

0

I'd recommend using jstat to monitor tomcat's memory and garbage collection patterns.

You say you're facing perm gen errors, but haven't increased the amount of perm gen. Reloading tomcat apps can cause perm gen leakage, but you're probably observing your usage before you open all your pages, resulting in loading more classes into perm gen.

6
  • Thanks for the surprisingly quick replies! Yes i am using top, i am also using jstat to monitor the gc, old space reaches about 90% and then empties in about 10 secs to reach 67%. The young generations (sq,s2,e) is a continous process where in they fill and empty at regular intervals. have not increased permgen, because i thought that eventually that would fill too in time and cause even a larger old space collection pause. Is this an incorrect way of thinking? Could be locked thread am running jstack to check the thread dump, as runing jmap to see memory dump is not showing any obvious errors
    – sam
    Oct 9, 2009 at 10:11
  • Like to specify this is the error i get when java does its collection thing "java.lang.OutOfMemoryError" so i guess jstat is reporting incorrect figures?
    – sam
    Oct 9, 2009 at 10:33
  • Objects do not migrate from old space to perm gen. Perm gen is primarily for compiled classes. See blogs.sun.com/jonthecollector/entry/… Oct 9, 2009 at 12:03
  • ... and classes get created by deploying the application, Tomcat's JSP compiler and some persistence tools like Hibernate that use magically extended classes. Don't know about toplink or struts...
    – Stroboskop
    Oct 9, 2009 at 13:05
  • you sir are absolutely correct, perm gen is not a part of the heap, what i meant to say is that even if i do increase the heap size, the memory leak would just fill the available space and then cause the out of memory error to occur. restarting the tomcat every three to five days, is causing a lot of stress at the users and management level, any suggestions would be welcome, will try to increase the xmx xms perm tomorrow and observe performance
    – sam
    Oct 9, 2009 at 13:16
0

your problem is that you are having in code something like this:

       new Thread(){
          public void run(){
             //something
          }
       }.start();

which in turn creates each time new class definition file in Tomcat's cache - and that eats space, so you have PermGenSpace exception...

rewrite some of your critical classes not to create custom classes 'on the fly', make separate inner classes and things will change...

      class MyJobDoer extends Thread{
          public void run(){
             //something
          }
       }

      MyJobDoer mjd=new MyJobDoer();
      mjd.start();
5
  • This will not change anything: the compiler does just that. No class is created 'on the fly'. Oct 9, 2009 at 11:04
  • we faced the same problem with permGenSpace, and it disappeared when we rewrited code this way.
    – ante.sabo
    Oct 9, 2009 at 11:08
  • 2
    Well your problem disappeared for some other reason. Oct 9, 2009 at 11:26
  • 1
    This doesn't create a new class on each execution, it creates a new anonymous class once at compile time.
    – matt b
    Oct 9, 2009 at 12:43
  • well, mine answer was written in good faith to share my experience and 'pull' out some similar experiences. Can't be sure of anything. Our Tomcat has to be restarted on every second redeploy. I think if you have class written this way tomcat might overload its perm gen space, and this is some kind of bug even people in Apache are not aware of...
    – ante.sabo
    Oct 10, 2009 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.