5

I was working on a project, and I want to contribute with the solution I found:

The code is of the kind:

while true
do
     while read VAR
     do
        ......
     done < <(find ........ | sort)
     sleep 3
done

The errors in the log where:

/dir/script.bin: redirection error: cannot duplicate fd: Too many open files
/dir/script.bin: cannot make pipe for process substitution: Too many open files
/dir/script.bin: line 26: <(find "${DIRVAR}" -type f -name '*.pdf' | sort): Too many open files
find: `/somedirtofind': Too many open files
/dir/script.bin: cannot make pipe for process substitution: Too many open files
/dir/script.bin: cannot make pipe for process substitution: Too many open files
/dir/script.bin: line 26: <(find "${DIRVAR}" -type f -name '*.pdf' | sort): ambiguous redirect

I noticed with command:

ls -l /proc/3657(pid here)/fd that the file descriptors were constantly increasing.

Using Debian 7, GNU bash, version 4.2.37(1)-release (i486-pc-linux-gnu)

2 Answers 2

4

The solution that worked for me is:

while true
do
     find ........ | sort | while read VAR
     do
     done
     sleep 3
done

Which is, avoiding the subshell at the end, there must be some kind of leak.

Now I don't see file descriptors in the process directory when doing ls

Mail to bugtracker:

The minimum reproduceable code is:

#!/bin/bash
function something() {
  while true
  do
    while read VAR
    do
      dummyvar="a"
    done < <(find "/run/shm/directory" -type f | sort)
    sleep 3
  done
}
something &

Which fails with many pipes fd open.

Changing the While feed to this:

#!/bin/bash
function something() {
  find "/run/shm/directory" -type f | sort | while true
  do
    while read VAR
    do
      dummyvar="a"
    done
    sleep 3
  done
}
something &

Works completely normal.

However, removing the call as function in background:

#!/bin/bash
while true
do
  while read VAR
  do
    dummyvar="a"
  done < <(find "/run/shm/debora" -type f | sort)
  sleep 3
done

But executing the script with ./test.sh & (in background), works
without problems too.
9
  • Kind of just curious. What commands do you run in the loop? Do you break the loop before reading all data?
    – konsolebox
    Jun 12, 2014 at 19:35
  • @konsolebox The most important part if the cat of PDFs into a single one. There are conditionals, sed, echos to a log, and a cat and echo to the same log. Then I delete the original PDFs
    – JorgeeFG
    Jun 12, 2014 at 19:38
  • Do you run a process in the background?
    – konsolebox
    Jun 12, 2014 at 19:41
  • @konsolebox yes, the while is contained in a function. The function is called in the same script in background, with &
    – JorgeeFG
    Jun 12, 2014 at 19:41
  • 3
    @Warbo I know this is old, but they replied me that it was fixed in Bash 4.3
    – JorgeeFG
    May 6, 2017 at 5:16
0

I was having the same issue and played around with the things you suggested. You said:

But executing the script with ./test.sh & (in background), works without problems too.

So what worked for me is to run in the background and just wait for it to finish each time:

while true
do
     while read VAR
     do
        ......
     done < <(find ........ | sort) &
     wait
done

Another thing that worked was to put the code creating the descriptor into a function without running on background:

function fd_code(){
     while read VAR
     do
        ......
     done < <(find ........ | sort)
}

while true
do
     fd_code
done

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.