14

EDIT 1

2013/06/07 - While I'm still having this problem, once again, it is still only affecting redeploys. Since the original question was posted, we've upgraded a few things. Here are our new version (which still exhibit the issue at hand):

<properties>
    <!-- Persistence and Validation-->
    <hibernate.version>4.1.0.Final</hibernate.version>
    <hibernate.jpa.version>1.0.1.Final</hibernate.jpa.version>
    <javax.validation.version>1.0.0.GA</javax.validation.version>
    <querydsl.version>2.2.5</querydsl.version>
    <spring.jpa.version>1.2.0.RELEASE</spring.jpa.version>
    <spring.ldap.version>1.3.1.RELEASE</spring.ldap.version>

    <!-- Spring and Logging -->
    <spring.version>3.1.3.RELEASE</spring.version>
    <spring.security.version>3.1.3.RELEASE</spring.security.version>
    <slf4j.version>1.6.4</slf4j.version>
    <jackson.version>1.9.9</jackson.version>

    <cglib.version>3.0</cglib.version>
</properties>

As you can see, it's basically just a Spring Framework bump and Spring (Data) Jpa bump. We also moved up to Tomcat 7.0.39. CGLIB (which wasn't mentioned before but was included) has also been bumped to 3.0

Here are some of the things I've tried to remedy the problem at hand with no luck:

  1. Changed up our POM (using Maven) to set our database driver to have its scope as provided
  2. Added dependency for SLF4J to included jul-to-slf4j as it was mentioned that there could be a potential leak here
  3. Tried out a Classload Leak Protector (https://github.com/mjiderhamn/classloader-leak-prevention). This appeared to have worked (as it spammed my logs during undeploy/shutdown by listing a ton of things it was cleaning up) but after using VisualVM, my perm gen space was not reclaimed. This may be useful for others...
  4. Tried to list all the exclusions in my POM by using Maven's dependency analysis tool (In IntelliJ in the Maven Projects window, you can right click the dependencies and select 'Show Dependencies' and Shift-Delete all the red dependencies). This didn't help, but it decrease my WAR file size by about a MB or so.
  5. Refactored the JPA Persistence configuration as follows (notice the comments) based upon an unresolved bug report from Spring (https://jira.springsource.org/browse/SPR-9274):

    @Bean
    public PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) { // Important line (notice entityManagerFactory is 'provided/autowired'
        return new JpaTransactionManager(entityManagerFactory);
    }
    
    @Bean
    public EntityManagerFactory getEntityManagerFactory(DataSource dataSource) { // Important line (notice dataSource is 'provided/autowired'
    
        LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean();
        factoryBean.setDataSource(dataSource);
        factoryBean.setPackagesToScan("my.scanned.domain");
    
        AbstractJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
        vendorAdapter.setGenerateDdl(true);
        vendorAdapter.setShowSql(false);
        vendorAdapter.setDatabase(Database.POSTGRESQL);
        vendorAdapter.setDatabasePlatform("org.hibernate.dialect.PostgreSQL82Dialect");
    
        factoryBean.setJpaVendorAdapter(vendorAdapter);
    
        Map<String, Object> properties = new HashMap<>();
        properties.put("hibernate.ejb.naming_strategy", "org.hibernate.cfg.ImprovedNamingStrategy");
        properties.put( "hibernate.bytecode.provider", "cglib" );   // Suppose to help java pergem space issues with hibernate
    
        factoryBean.setPersistenceProvider(new HibernatePersistence());
        factoryBean.setJpaPropertyMap(properties);
        factoryBean.setPersistenceUnitName("myPersistenace");
        factoryBean.afterPropertiesSet();
    
        return factoryBean.getObject(); // Important line
    }
    
    @Bean
    public PersistenceExceptionTranslator getHibernateExceptionTranslator() { // Required
        return new HibernateExceptionTranslator();
    }
    
    @Bean
    public DataSource getDataSource() {
        JndiDataSourceLookup lookup = new JndiDataSourceLookup();
        DataSource dataSource = lookup.getDataSource("java:comp/env/jdbc/myLookup");
    
        lookup = null;
    
        return dataSource;
    }
    
  6. Created a 'ThreadImmolator' as per https://stackoverflow.com/a/15710827/941187 in the following SO Question: 'Is this very likely to create a memory leak in Tomcat?'. This appeared to do nearly the same thing as the TreadLocal Leak Detection above -- I didn't have the Tomcat Leak detector complaining; however, perm gen space was still not recovered after redeploy.
    • My first attempt was to add a @PreDestory in my WebConfig (same bean that has @EnableWebMvc) to attempt to trigger it on close. Perm. Gen remained.
    • My second attempt was to subclass the ContextLoaderListener, override the ContextDestoryed() and inline the functions. Perm. Gen remained.
  7. Since the ThreadImmolator didn't help (or appeared not to help), I tried the solution suggested in https://stackoverflow.com/a/16644476 on the following SO Question: 'How to clean up threadlocals'. This lead me to try: 'https://weblogs.java.net/blog/jjviana/archive/2010/06/09/dealing-glassfish-301-memory-leak-or-threadlocal-thread-pool-bad-ide' and 'http://blog.igorminar.com/2009/03/identifying-threadlocal-memory-leaks-in.html'.

At this point, I have run out of ideas.

I have also tried learning about Heap analysis using this as a starting point (http://cdivilly.wordpress.com/2012/04/23/permgen-memory-leak/). I can find the class loader that was not cleaned up and I can see that it is still referencing all of the classes related to Spring. I've also tried searching for org.springframework.core.NamedThreadLocal and I can still see them showing up in the Heap after taking a dump after running the ThreadImmolator, Thread Leak Preventor and the other 'heavy-handed-solutions' that I tried above.

Perhaps the above information may help someone, but I will continue to revisit this SO with new information or if I solve the issue.

Problem

The application has no issues running for days on end on the production server, but when I perform a deploy for updates, the Tomcat Manager program will complain about leaks (if I click find leaks). If I perform 6-10 deploys, eventually Tomcat runs out of memory with a PermGen memory error and I need to restart the Tomcat service and everything returns to normal.

When I run/debug the application locally and perform some actions that require access through Jpa/Hibernate (ie, I login or request a List from a JpaRepository) and then shutdown the application, I receive the following messages in my debug output from Tomcat:

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract java.lang.Object org.springframework.data.repository.CrudRepository.findOne(java.io.Serializable)=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract java.util.List org.springframework.data.jpa.repository.JpaRepository.findAll()=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract java.lang.Iterable org.springframework.data.querydsl.QueryDslPredicateExecutor.findAll(com.mysema.query.types.Predicate)=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Oct 03, 2012 2:55:13 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{public abstract data.domain.UserAccount UserAccountRepository.findByUserName(java.lang.String)=java.lang.Object@842e211}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

etc, etc.

Configuration

Spring is configured via annotations and I'm also using Postgres 8.4 as the database backend.

JPA is configured via annotations (jpa-repository-context.xml just says to look for this class):

@Configuration
@EnableTransactionManagement
@ImportResource( "classpath*:*jpa-repository-context.xml" )
@ComponentScan( basePackages = { "data.repository" } )
public class PersistenceJpaConfig
{
    @Bean
        public LocalContainerEntityManagerFactoryBean entityManagerFactory()
        {
            LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean();
            factoryBean.setDataSource( dataSource() );
            factoryBean.setPackagesToScan( new String[] { "data.domain" } );

            // Setup vendor specific information. This will depend on the chosen DatabaseType
            HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
            vendorAdapter.setGenerateDdl( true );
            vendorAdapter.setShowSql( false );
            vendorAdapter.setDatabasePlatform( "org.hibernate.dialect.PostgreSQL82Dialect" );

            factoryBean.setJpaVendorAdapter( vendorAdapter );

            Map<String, Object> properties = new HashMap<String, Object>();
            properties.put( "hibernate.ejb.naming_strategy", "org.hibernate.cfg.ImprovedNamingStrategy" );

            factoryBean.setJpaPropertyMap( properties );

            return  factoryBean;
        }

        @Bean
        public DataSource dataSource()
        {
            JndiDataSourceLookup lookup = new JndiDataSourceLookup();
            DataSource dataSource;

            dataSource = lookup.getDataSource( "java:comp/env/jdbc/postgres" );


            return dataSource;
        }

        @Bean
        public PlatformTransactionManager transactionManager()
        {
            JpaTransactionManager transactionManager = new JpaTransactionManager();
            transactionManager.setEntityManagerFactory( entityManagerFactory().getObject() );

            return transactionManager;
        }
}

Example Repository:

public interface UserAccountRepository extends JpaRepository<UserAccount, Long>, QueryDslPredicateExecutor<UserAccount> {
}

All of my Repositories are accessed through a Service class which is registered as an @Component in Spring. This is done to remove repository access from Spring controllers:

@Component
public class UserAccountService {

    @Autowired
    private UserAccountRepository userAccountRepository;

    public List<UserAccount> getUserAccounts() {
        return userAccountRepository.findAll();
    }
    ...
}

And here are the versions for the various components in use in Maven's pom.xml:

<properties>
        <!-- Persistence and Validation-->
        <hibernate.version>4.1.0.Final</hibernate.version>
        <hibernate.jpa.version>1.0.1.Final</hibernate.jpa.version>
        <javax.validation.version>1.0.0.GA</javax.validation.version>
        <querydsl.version>2.2.5</querydsl.version>
        <spring.jpa.version>1.1.0.RELEASE</spring.jpa.version>

        <!-- Spring and Logging -->
        <spring.version>3.1.2.RELEASE</spring.version>
        <spring.security.version>3.1.2.RELEASE</spring.security.version>
        <slf4j.version>1.6.4</slf4j.version>
        <jackson.version>1.9.9</jackson.version>

        <!-- Testing Suites -->
        <selenium.version>2.24.1</selenium.version>
</properties>

Questions

  1. What is causing the memory leak and how do I fix it?
  2. How would I go about debugging this particular issue?
  3. Is there anything in my configuration set that could be improved?

I have really run out of ideas on solving this problem.

2
  • 1
    FYI - I think you may have two kinds of leaks going on at the same time. Spring is warning you about a normal "heap" memory leak. This has not caused you a problem yet, because ... your redeployment is also causing excessive PermGen usage, and this problem is hitting you first. For info on the second kind of leak see stackoverflow.com/questions/88235/… Oct 3, 2012 at 23:12
  • Worth making this comment an answer. I'd vote it up.
    – duffymo
    Oct 3, 2012 at 23:15

4 Answers 4

4

I think you may have two kinds of leaks going on at the same time. Spring is warning you about a normal "heap" memory leak. This has not caused you a problem yet, because ... your redeployment is also causing excessive PermGen usage, and this problem is hitting you first. For info on the second kind of leak see Dealing with "java.lang.OutOfMemoryError: PermGen space" error [Thanks duffymo]

[update]

Since you say the suggestions in the above link did not help, the only other suggestions I can think of are:

  • make sure your spring beans are cleaning themselves up when spring shuts down
  • every bean that allocates a resource (anything unlikely to be garbage collected) in the constructor or init method should have a destroy method to deallocate it
  • this is especially true for any beans that reference any classes in the spring-data module, catalina is complaining about a class in this module
  • increase your permgen space. Even if this suggestion did not fix the problem adding something like the following should make these failures occur less frequently.

Try -XX:MaxPermSize=256m and if it persists, try -XX:MaxPermSize=512m

The ultimate uber-brute-force approach is to gradually strip things out of your app until the problem goes away. That will help you narrow it down to the point where you can identify whether it is an issue with your code or a bug in Spring, Hibernate etc

4
  • Thanks for the help! Unfortunately, trying the various suggestions (VM switches) did not solve my problem. It seems this is a Spring problem? Oct 4, 2012 at 17:18
  • Thank for the guidance. I'll try pouring over the code carefully to see if I'm holding onto references that I shouldn't. Perhaps I should use some @PreDestroy methods in some of the beans? Oct 4, 2012 at 19:34
  • I think I found it! It appears that Spring data's proxying to the repositories are sticking around after unloads. $Proxyxxxx (xxxx is some number) gets attached to the class which appears to have a link to the classloader. I am still unsure how to fix Oct 4, 2012 at 19:53
  • I have added additional information to document my progress with this problem (EDIT 1) Jun 7, 2013 at 21:02
3

slash3tc - first of all thanks for the great writeup, it helped me plug a few leaks in my application. I had the exact same problem which turned out to be invocations of spring-data-jpa DAOs without an active transaction. This can easily happen with a scheduler or a @PostConstruct annotation (formerly InitializingBean). Make sure you have configured transaction management correctly and that any JPA invocations happen under an active transaction (i.e. by annotating service methods with @Transactional).

Edit: Seems to be (also?) a bug with older spring-data-jpa versions that was fixed in 1.4.3

More details: http://georgovassilis.blogspot.nl/2014/01/tomcat-spring-and-memory-leaks-when.html

1
  • Thank you for adding new information to this thread! I recently updated Spring-data-jpa across all projects to fix a bug where it was using a hashmap to cache query string information which resulted in a spin-lock of cpu cores. I'm wondering if I should place an @Transactional on my base service layer and if our transaction management has even been configured correctly. I'll play around with this and see it it fixes these issues. Jan 16, 2014 at 16:40
1

I had a similar problem, actually 3 problems:

  1. JDBC driver not being released
  2. Connection not being relased
  3. Heaps not being released

Each of them has a different solution:

  1. Moving mysql.jar to Tomcat/lib folder
  2. Manually killing the threads withholding connection in a listener
  3. Using SPRING-JPA 1.3.4, cause 1.3.4 is the version where the bug was fixed, but it does not change the behavior of SPRING-JPA module (the next version 1.4.0 introduces updates to queries and "breaks" existing code)
0
0

I had a similar problem and I've just fixed it.

[org.springframework.core.NamedThreadLocal] (value [Transactional resources]

It may be created by TransactionSynchronizationManager.

private static final ThreadLocal<Map<Object, Object>> resources =
        new NamedThreadLocal<Map<Object, Object>>("Transactional resources");

private static final ThreadLocal<Set<TransactionSynchronization>> synchronizations =
        new NamedThreadLocal<Set<TransactionSynchronization>>("Transaction synchronizations");

private static final ThreadLocal<String> currentTransactionName =
        new NamedThreadLocal<String>("Current transaction name");

private static final ThreadLocal<Boolean> currentTransactionReadOnly =
        new NamedThreadLocal<Boolean>("Current transaction read-only status");

private static final ThreadLocal<Integer> currentTransactionIsolationLevel =
        new NamedThreadLocal<Integer>("Current transaction isolation level");

private static final ThreadLocal<Boolean> actualTransactionActive =
        new NamedThreadLocal<Boolean>("Actual transaction active");

In my case, NamedThreadLocals except "Transactional resources" were leaked. So I added the following code into the last line of contextDestroyed.

TransactionSynchronizationManager.clear();

I think you can also release resources with adding:

Map<Object, Object> ojb = TransactionSynchronizationManager.getResourceMap();
for (Object key : ojb.keySet()) {
    TransactionSynchronizationManager.unbindResource(key);
}

Additional to it, I needed to do the following:

  1. Removed PostgreSQL driver from tomcat/libs. I had already added dependency in build.gradle so they are duplicated.
  2. Updated hibernate-validator library to the latest version

    compile ('org.springframework.boot:spring-boot-starter-thymeleaf:' + bootVersion + '.RELEASE') {
        exclude module: 'spring-boot-starter-tomcat'
        exclude module: 'hibernate-validator'
    }
    // https://mvnrepository.com/artifact/org.hibernate/hibernate-validator
    compile group: 'org.hibernate', name: 'hibernate-validator', version: '5.2.4.Final'
    

The version of Spring Boot in my app is 1.2.7.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.