9

Without keeping a list of current threads, I'm trying to see that a realtime signal gets delivered to all threads in my process. My idea is to go about it like this:

  • Initially the signal handler is installed and the signal is unblocked in all threads.
  • When one thread wants to send the 'broadcast' signal, it acquires a mutex and sets a global flag that the broadcast is taking place.
  • The sender blocks the signal (using pthread_sigmask) for itself, and enters a loop repeatedly calling raise(sig) until sigpending indicates that the signal is pending (there were no threads remaining with the signal blocked).
  • As threads receive the signal, they act on it but wait in the signal handler for the broadcast flag to be cleared, so that the signal will remain masked.
  • The sender finishes the loop by unblocking the signal (in order to get its own delivery).
  • When the sender handles its own signal, it clears the global flag so that all the other threads can continue with their business.

The problem I'm running into is that pthread_sigmask is not being respected. Everything works right if I run the test program under strace (presumably due to different scheduling timing), but as soon as I run it alone, the sender receives its own signal (despite having blocked it..?) and none of the other threads ever get scheduled.

Any ideas what might be wrong? I've tried using sigqueue instead of raise, probing the signal mask, adding sleep all over the place to make sure the threads are patiently waiting for their signals, etc. and now I'm at a loss.

Edit: Thanks to psmears' answer, I think I understand the problem. Here's a potential solution. Feedback would be great:

  • At any given time, I can know the number of threads running, and I can prevent all thread creation and exiting during the broadcast signal if I need to.
  • The thread that wants to do the broadcast signal acquires a lock (so no other thread can do it at the same time), then blocks the signal for itself, and sends num_threads signals to the process, then unblocks the signal for itself.
  • The signal handler atomically increments a counter, and each instance of the signal handler waits until that counter is equal to num_threads to return.
  • The thread that did the broadcast also waits for the counter to reach num_threads, then it releases the lock.

One possible concern is that the signals will not get queued if the kernel is out of memory (Linux seems to have that issue). Do you know if sigqueue reliably informs the caller when it's unable to queue the signal (in which case I would loop until it succeeds), or could signals possibly be silently lost?

Edit 2: It seems to be working now. According to the documentation for sigqueue, it returns EAGAIN if it fails to queue the signal. But for robustness, I decided to just keep calling sigqueue until num_threads-1 signal handlers are running, interleaving calls to sched_yield after I've sent num_threads-1 signals.

There was a race condition at thread creation time, counting new threads, but I solved it with a strange (ab)use of read-write locks. Thread creation is "reading" and the broadcast signal is "writing", so unless there's a thread trying to broadcast, it doesn't create any contention at thread-creation.

6
  • OK, if all of the updates to the data touched by the signal handler are atomic then this is getting close to working :) I think there's still a race condition if all threads wait for the counter to equal num_threads: if another broadcast event happens immediately after the first, then it's possible that the counter will go back to 0 before all threads have noticed that it's equal to num_threads. I think this can be solved with a condition variable. I'd also advise against busy-waiting (even with sched_yield) - this can lead to high CPU load (and poor battery life if that's relevant!)
    – psmears
    Nov 18, 2010 at 14:36
  • Another possible issue you may want to test for is that it is possible for a signal to be delivered to a thread after it has been created in the kernel, but before its thread-local data is fully set up (i.e. in the time between when the thread starts running, and when it calls the function passed to pthread_create())...
    – psmears
    Nov 18, 2010 at 14:37
  • The updates are all atomic because they all happen in kernel-space via syscalls which wrongly affect only the calling thread and not the process; this code is a workaround. :-) Only one broadcast event can happen at a time; the whole function is protected by a global mutex. As for busy waiting, perhaps a timed wait on a conditional variable with a very short timeout would work, but I've since found myself having trouble with too few signals being delivered unless I keep calling sigqueue until it works. Nov 18, 2010 at 20:16
  • The mutex doesn't help with the problem I'm describing - the issue occurs because (as far as I understand it) although the broadcasting thread waits for all other threads to receive the signal before it releases the mutex, it doesn't wait for all the other threads to notice that the broadcast is complete. So it's possible that another thread will then obtain the mutex and reset the counter to zero, before all the threads have decided to stop waiting (which I think would cause a deadlock?).
    – psmears
    Nov 19, 2010 at 19:43
  • I'm sorry, I thought I mentioned that. The signal handler waits to return until the broadcaster detects that all threads have received the signal, and the broadcaster waits to return until the count reaches 0 again (the signal handler atomically decrements the count before returning). Nov 19, 2010 at 21:19

4 Answers 4

5

raise() sends the signal to the current thread (only), so other threads won't receive it. I suspect that the fact that strace makes things work is a bug in strace (due to the way it works it ends up intercepting all signals sent to the process and re-raising them, so it may be re-raising them in the wrong way...).

You can probably get round that using kill(getpid(), <signal>) to send the signal to the current process as a whole.

However, another potential issue you might see is that sigpending() can indicate that the signal is pending on the process before all threads have received it - all that means is that there is at least one such signal pending for the process, and no CPU has yet become available to run a thread to deliver it...

Can you describe more details of what you're aiming to achieve? And how portable you want it to be? There's almost certainly a better way of doing it (signals are almost always a major headache, especially when mixed with threads...)

4
  • 1
    I was under the impression that the signal would not show up as pending unless all threads were blocking it. But I suspect your answer here is correct - the first time around, the loop sends a signal, and then since no other thread gets immediately scheduled to handle it, it shows up as pending, and the sending thread unblocks it and handles it itself... :-( Nov 17, 2010 at 15:34
  • As for what I'm trying to accomplish - I want to be able to run a piece of code in each thread (making use of thread-local state) without having to keep track of a list of all running threads. The idea behind that requirement is that this code that needs to be run in all threads is very rarely (almost never) used, while thread creation and exiting could be very frequent, and I don't want to impose the cost of synchronizing thread bookkeeping just for a single almost-never-used operation. Nov 17, 2010 at 15:37
  • Is there a chance that the thread-local state is being updated by the thread at the time the signal handler is called? If so the signal handler may see it (and update it?) while it's inconsistent, which may cause issues (and you can't safely call malloc(), free(), *printf() etc for the same reason!)... what are your threads actually doing? Do they wait loop waiting for events (eg calling select()), or do bulk processing, or a variety of different things?
    – psmears
    Nov 17, 2010 at 17:20
  • Threads could be doing anything at the time they're interrupted; it's outside of my control as this is library code. But the state is always consistent (only updates to it are atomic). Signals are really the correct and only way to solve the problem. Take a look at my updates to the question and see if you think it'll work. Nov 17, 2010 at 18:50
3

In multithreaded program raise(sig) is equivalent to pthread_kill(pthread_self(), sig). Try kill(getpid(), sig)

0

Given that you can apparently lock thread creation and destruction, could you not just have the "broadcasting" thread post the required updates to thread-local-state in a per-thread queue, which each thread checks whenever it goes to use the thread-local-state? If there's outstanding update(s), it first applies them.

1
  • Nope, for two reasons. One is a matter of cost. The whole idea here is to minimize the cost of something that needs to be able to happen, but almost never happens in reality (most programs won't even use the functionality at all). The other reason is that I don't have a list of threads to get to their local state. I surely could add a list, but that would again add cost (additional locking at thread creation and termination). Nov 18, 2010 at 5:56
0

You are trying to synchronize a set of threads. From a design pattern point of view the pthread native solution for your problem would be a pthread barrier.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.