2

I am trying to write a program, in C++, which runs on a cluster of machines, and all machines are talking to each other over TCP sockets. Program crashes randomly at one of the machines. I did an analysis of core-dump with gdb. Following are the output:

$ gdb executable dump

  Core was generated by `/home/user/experiments/files/executable 2 /home/user/'.
  Program terminated with signal SIGABRT, Aborted.
  0 0x00007fb76a084c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
  56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

  (gdb) backtrace
  0 0x00007fb76a084c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
  1 0x00007fb76a088028 in __GI_abort () at abort.c:89
  2 0x00007fb76a0c12a4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fb76a1cd113 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
  3 0x00007fb76a158bbc in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7fb76a1cd0aa "buffer overflow detected") at fortify_fail.c:38
  4 0x00007fb76a157a90 in __GI___chk_fail () at chk_fail.c:28
  5 0x00007fb76a158b07 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
  6 0x000000000040a918 in LocalSenderPort::run() ()
  7 0x000000000040ae70 in LocalSenderPort::LocalSenderPort(unsigned int, std::string, std::vector<std::string, std::allocator<std::string> >, char*) ()
  8 0x00000000004033d5 in main ()

Any suggestions for what should I look? How should I proceed? Any help is really appreciated.

I am not sharing code right now, as its a large code spread across files. But I can share if needed.

7
  • Assuming that you're using G++/Clang pass the -g option to the compiler. That will generate debug symbol information and should allow you to see the lines of code where the crash was caused from in gdb.
    – tambre
    Commented Dec 18, 2016 at 20:01
  • @tambre I have added -g during compilation. And it does shows following : #0 0x00007fb76a084c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
    – PHcoDer
    Commented Dec 18, 2016 at 20:04
  • 1
    Don't even try to post the stack trace in a comment. Instead edit the new (and useful) the stack trace into the question. Though the stack trace indicating the location in your own code should usually be enough for you to debug the rest yourself.
    – tambre
    Commented Dec 18, 2016 at 20:07
  • The right tool to solve such problems is your debugger. You should step through your code line-by-line before asking on Stack Overflow. For more help, please read How to debug small programs (by Eric Lippert). At a minimum, you should [edit] your question to include a Minimal, Complete, and Verifiable example that reproduces your problem, along with the observations you made in the debugger. Commented Dec 18, 2016 at 20:07
  • The raise entry is not interesting. LocalSenderPort::run entry is. It should show the line number idf compiled with -g. Commented Dec 18, 2016 at 20:16

1 Answer 1

4

This error: __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25 means that your program violated precondition of one of the FD_* macros.

The source of fdelt_chk is quite simple, and there are only two conditions under which it fails: you pass in negative file descriptor, or you pass in a file descriptor greater than 1023.

In this day and age, using select and/or FD_SET in any program that can have more than 1024 simultaneous connections (which Linux easily allows) can only end in tears. Use epoll instead.

4
  • Thanks for the answer. The problem was something on same lines. But it's not with select. function socket is returning -1 when it has already returned 1023 sockets. And all of them are alive yet. How can I increase the permissible number of total alive sockets at a time?
    – PHcoDer
    Commented Dec 19, 2016 at 23:46
  • 1
    If socket returns error, it's likely because you are running into the limit on total number of file descriptors (man setrlimit or man ulimit). Note: if you increase that limit and are using select or FD_* macros, then you'll immediately run into the problem in my answer. Commented Dec 20, 2016 at 1:07
  • Thanks. Yes that's correct, now I am in the problem mentioned in your answer. Will go with epoll now.
    – PHcoDer
    Commented Dec 20, 2016 at 15:22
  • This answer is still one of the top results on google, and you have saved me a pile of work with this summary. I bumped the problem code from select to epoll and everything was resolved. Many thanks!
    – Buffoonism
    Commented Aug 2, 2022 at 5:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.