23

I have an application running on tomcat and sometimes I have the error below:

SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:722)

....

SEVERE: Error processed default web.xml named conf/web.xml at /local/myApp/apache-tomcat/conf/web.xml
java.io.FileNotFoundException: /local/myApp/apache-tomcat/conf/web.xml (Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.apache.catalina.startup.ContextConfig.getWebXmlSource(ContextConfig.java:1838)
        at org.apache.catalina.startup.ContextConfig.getGlobalWebXmlSource(ContextConfig.java:1745)
        at org.apache.catalina.startup.ContextConfig.getDefaultWebXmlFragment(ContextConfig.java:1418)
        at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1253)
        at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5269)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3926)
        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:426)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1345)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1530)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
        at java.lang.Thread.run(Thread.java:722)

I check the limits of the open files and it's 1024 but when I check the number of open files of the application with lsof it's nearly 200, why this happen if it does not reach the limit? Should I increase the limit? Is there any other reason to get this error? The only way to get the service running correctly again, it's restarting the tomcat, is there another way of getting back to normal?

Thanks in advance.

EDIT: Here is the servlet which process the doPost method, at the beginning I didn't close every stream, could it be that? I added the finally statement for doing that:

    InputStream is = null;
    DataInputStream dis = null;
    OutputStream os = null;
    DataOutputStream dos = null;
    String paramName = "";
    try {
        os = response.getOutputStream();
        is = request.getInputStream();
        dis = new DataInputStream(is);
        dos = new DataOutputStream(os);
        .....
        }catch (Throwable e) {
        LOGGER.error(e.getMessage());
        } finally {
          if (dis != null) {
             dis.close();
           }
           else if(is != null) {
             is.close();
           }                
           if (dos != null) {
             dos.close();
           }
           else if( os != null) {
             os.close();
           }
        }

EDIT2: After making some testing I realize that if I close first the DataInputStream and then the InputStream, I get in the other part of the communication a number before the message(I don't know why). I changed the order of closing the stream and it seems that everything it's ok. But I still have the problem. Any idea?

  finally {

    if(is != null) {
        try {
            is.close();
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
        }
    }
    if (dis != null) {
        try {
            dis.close();
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
        }
    }
    if(os != null) {
        try {
            os.close();
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
        }
    }
    if (dos != null) {
        try {
            dos.close();
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
        }
    }
}
8
  • Yes I checked this web, but as the service it's nor using more than 200 as maximum, why to increase to 4096 the limit?
    – alicia
    Oct 28, 2013 at 13:59
  • You should search your error on the site first: stackoverflow.com/questions/5656458/… Oct 28, 2013 at 22:54
  • Share your code! we can't help without any code snippet.
    – tokhi
    Oct 29, 2013 at 7:12
  • @developerwjk first I check the site and incremeting the limit of the open files could be an idea but as I do not reach the top of the limit, I think it will not solve my problem.
    – alicia
    Oct 29, 2013 at 10:15
  • Is the tomcat running on a Windows or on a Linux machine? Oct 29, 2013 at 11:43

5 Answers 5

19

Do the following to get the pid, say 1234, of tomcat7

ps aux |grep tomcat7

and then do

cat /proc/1234/limits to read a line like the following

Max open files 16384 16384 files

These are the maximum number of open files that are allowed by Tomcat. To increase it, follow the instructions below

Too many open files Tomcat.

Edit: Link 1 is broken, see latest snapshot on web-archive.

5
  • 1
    I have already check and increase the number of open files to unreachable number. I think it is more a bad managament of the threads inside the application, could it be?
    – alicia
    Mar 17, 2015 at 13:36
  • 1
    Yes, I guess that you are probably leaking file handlers / connections in your applications.
    – Zouzias
    Mar 17, 2015 at 14:00
  • Is there a way to see a list of those open files? And would be useful to see what thread had opened them.
    – Miklos
    Aug 24, 2017 at 9:16
  • The link no longer works
    – mwarren
    Mar 28 at 14:14
  • Indeed, I added a link from the web-archive.
    – Zouzias
    Mar 29 at 15:46
12

Follow the below instructions to have a quick analysis of current configuration with your server and tune the tomcat hard and soft limits to fix this issue.

This will display all open files of that process.

ls -l /proc/tomcatPID/fd 

This will show the count of open files.

ls -l /proc/tomcatPID/fd | wc -l 

To increase open files limit update /etc/security/limits.conf.

To check no of open files specific to tomcat:

Hard limit: su - tomcat -c 'ulimit -Hn' -s '/bin/bash'

Soft limit: su - tomcat -c 'ulimit -Sn' -s '/bin/bash'

You can run below script with a corn job to know the details of open files.

=============================
#!/bin/bash

PID=$(ps -ef|grep tomcat6|grep -v grep |awk '{print $2}')
value=$(ls -l /proc/$PID/fd | wc -l)
echo `date`@$PID:$value >> /usr/local/filecount.txt
if [ $value -gt 2000 ];
then
printf "\n\n\n\n\n" >> /usr/local/files_report.txt
echo "-------------------------------`date`--Starting Session----------------------" >> /usr/local/files_report.txt
openfiles=$(ls -l /proc/$PID/fd | awk '{print NR,$11 "" >> "/usr/local/files_report.txt"}')
echo "--------------------`date`---Ending  Session ------------------------------" >> /usr/local/files_report.txt
fi
================= 
1
10

It may be useful to know that you can change the limit of opened files by adding the following to /etc/security/limits.conf:

* soft nofile 2048 # Set the limit according to your needs
* hard nofile 2048

Then you can reload the configuration using sysctl -p on the shell. Check this article.

Just for completeness you can verify what is the current limit for opened files using: ulimit -n

4

The answer @gaboroncancio posted is basically correct, but his advice on how to make the settings take effect is not quite right. sysctl -p will reload /etc/sysctl.conf, or whatever file you pass in as an argument. However, the sysctl command will not recognize the format of /etc/security/limits.conf.

To reload /etc/security/limits.conf, you just need to log out and log back in.

1
  • Yes value reload needs log back Jan 23, 2018 at 6:07
1
  1. If this code is from net operations (sockets), I'm not sure Java XxxxxStrem has 1:1 relation to OS file limits (or has relation at all). Maybe need some research, exception message has wrong text? Quite often in software.

  2. My intuition say, we don't understand Exception 2, nothing in code (or configuration absent in question) has relation.

  3. Extending OS files limit when software error (leak) is main problem is bad policy, as You good understand

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.