4

This question is somehow similar to Bad file descriptor but it's not the same at all. I know this is "bad question" ("too localized" maybe), but I can't figure it out and I'm now out of any ideas.


Introduction

I have a manager thread, that starts 75 other threads. Each of these threads do a lot of things, so I'll describe only the relevant ones.

Please note: if I start only a few threads - for example 3 or 5, or 10, this error does not appear! This makes me think, that this is some multithreading issue, but it doesn't seem to be such.. And you'll see why in the next sections.

So, in the following 2 cases, SOMETIMES I receive this error Bad file descriptor:


case 1

The error appears in TinyXML

There's an xml file, that's needed by all threads. All of these threads use TinyXML to parse the file. ALL of these threads use this file READ-ONLY! (I know this can be optimized, but whatever).

So, the code, that causes the Bad file descriptor error is this:

// ...
// NOTE: this is LOCAL, other threads do NOT have access to it
TiXmlDocument   doc;
doc.LoadFile( filename );

// and here's the LoadFile:
bool TiXmlDocument::LoadFile( const char* _filename, TiXmlEncoding encoding )
{
    //...
    FILE* file = fopen( value.c_str (), "rb" ); 
    if ( file )
    {
        // this IS executed, so file is NOT NULL for sure
        bool result = LoadFile( file, encoding );
        //...
    }
    //...
}

bool TiXmlDocument::LoadFile( FILE* file, TiXmlEncoding encoding )
{
    // ...
    long length = 0;
    fseek( file, 0, SEEK_END );
    // from the code above, we are SURE that file is NOT NULL, it's valid, but
    length = ftell( file ); // RETURNS -1 with errno: 9 (BAD FILE DESCRIPTOR)
    // how is this possible, as "file" is not NULL and it appears to be valid?
    // ...
}

case 2

This is a bit more complicated. I've removed the checking of the return values, but I have them in my real code, so this is not a problem

int hFileR = open( sAlarmFileName.c_str(), O_CREAT | O_RDONLY, 
                   S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH );
// hFileR is > 0 for sure, so success 

flock( hFileR, LOCK_EX ) /* the result is > 0 for sure, so success*/ 

// read the file into a string
while( (nRes = read(hFileR, BUFF, MAX_RW_BUFF_SIZE)) > 0 ) // ...

//Write new data to file: reopen/create file - write and truncate mode
int hFileW = open( sAlarmFileName.c_str(), 
                   O_CREAT | O_WRONLY | O_TRUNC, S_IRUSR | 
                   S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH );
// hFileW is > 0 for sure, so success

do
{
    int nWrtRes = write( hFileW, 
                        szText + nBytesWritten, nSize - nBytesWritten ); 
    // nWrtRes is always >= 0, so success
    nBytesWritten +=  nWrtRes;
}
while( nSize > nBytesWritten );

close( hFileW );    // this one is successful too

if( flock(hFileR, LOCK_UN) == -1 )
{
    // THIS FAILS and executes _Exit( FAILURE );
}

if( close( hFileR ) < 0 )
{
    // if the previous one do not fail, this one is successful too
}

Sorry for the long question. Any ideas?

3
  • I'm surprised TinyXML doesn't check the ret-val of that fseek() before running off to ftell().
    – WhozCraig
    Dec 7, 2012 at 17:50
  • It's almost certainly a multi-threading issue. Between the two uses of the FILE *, some other thread has closed the underlying file descriptor. The most likely cause of this is a double close somewhere else in the code. Dec 7, 2012 at 19:26
  • 2
    By the way, the most common cause of this hard-to-find bug is code that intentionally calls close on another thread's file descriptor as a way to get that other thread to stop. For example, TCP code that has a "read thread" and a "write thread" may idiotically have some other thread call close on the descriptor as a way to make the read and write threads fail. Among the many horrible things this causes, if another thread gets the same descriptor just as the write thread is about to write, the write thread may write sensitive data to the wrong connection. Use shutdown. Dec 7, 2012 at 19:33

3 Answers 3

6

One thing to look for is code that closes the same file descriptor twice.

In a single-threaded program this is a harmless programming mistake, because the second close() doesn't do anything except return EBADF, and a lot of code doesn't bother to check the close() return value anyway. In a multithreaded program, however, the descriptor number of the closed descriptor can be allocated in another thread between the two calls to close(), so the second close() will close an unrelated socket from another thread. Further reads and writes of the other thread's descriptor will then cause the "bad file descriptor" error.

3
  • @MooingDuck: How do you figure? It seems quite likely that some other piece of code is closing the underlying file descriptor. Dec 7, 2012 at 19:25
  • @MooingDuck: The other thread calls close on its own file descriptor. Then this code runs and allocates the very same file descriptor since it's now free. Then the other thread calls close again because it's broken. Then this code runs again and finds its file descriptor closed. As far as the other thread is concerned, everything is fine. Dec 7, 2012 at 19:27
  • @DavidSchwartz: And this answer even clearly says so. I'm a bad reader :( Dec 7, 2012 at 19:28
3

Some words on the understanding of file descriptors:

Files are global resources. To handle such, a (process) global indexing is used: Integer values, called file descriptors. If a thread opens a file this opened file is referred by an index. This index is unique to a process (not to a thread). If a file is closed, the file descriptor (integer index) is not used any more and could be reused by the process (and any of its threads).

Example:

By any thread in a process the 1st call to open() might return 3, the 2nd might return 4.

If then 3 is closed, the 3rd call to open() may return 3 again.

If the 1st call is done by thread 1, the 2nd by thread 2 and the 3rd by thread 3, it is easy to understand that thread 1 shall not close its file descriptor again, as the value of 3 might already have been recycled and in use by thread 3, which would try to access an invalid file descriptor as it might have been closed by the 2nd (errorous) call to close() by thread 1. Ok? ;-)

Try setting up some example code, and inspect/log the integer values returned by calls to open() and assigned as file descriptors, to get an impression on how it works.

Note:

This also might refer to stdin, stdout and stderr, the "predefined" file descriptors 0, 1 and 2. Under recent Linux closing stdin followed by a call to int fd = open("myfoofile.bar", ...) might very well return 0 as file descriptor fd. Anyhow, either the Kernel or the glibc is not able to handle such a 0 as expected. Obscure errors might occur using lseek(fd, ...) for example. Try it! ;->>

1
  • 1
    The note at the end is wrong. Not only may open return 0 when stdin is closed; it must return 0, since it's the lowest-available file descriptor. In single-threaded programs this is a valid way to replace stdin, though of course it has race conditions that preclude doing it in multi-threaded programs. Neither the kernel nor glibc has any problem using file descriptor 0; they both use it all the time as stdin. Aug 14, 2013 at 15:04
2

If the application is multi-threaded, it might happen if some thread closed the file, while another still tries to access it.

(because file descriptors, like address space, are global & common to all threads of a process)

You could use strace to understand what syscalls are done.

11
  • You might want to explain how another thread could close a file descriptor that's local to another thread's functions. Dec 7, 2012 at 19:29
  • 1
    There's really no such thing as a local file descriptor. They're process-level resources. Dec 7, 2012 at 19:31
  • 1
    @MooingDuck My answer provides one path for that to happen, but there are others. File descriptors are sometimes explicitly shared among threads, stored in global tables, etc. Mistakes happen. Dec 7, 2012 at 19:39
  • 1
    @KirilKirov: You don't need to sync the threads with a mutex. You just need to make sure another thread never calls a function on a file descriptor that it isn't supposed to be touching. For example, if a thread calls close twice on the same file descriptor, it can wreak havoc on a thread that called open or fopen in-between those two close calls (if it got assigned the now-free file descriptor). Dec 10, 2012 at 4:40
  • 1
    @KirilKirov: This code is the victim, likely not the perpetrator. It can be completely different code that's closing this code's file descriptor by mistake. Dec 10, 2012 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.