10

I'm running a multithreaded socket program with valgrind. The client will send out a request to the server over TCP, and then busy wait on a boolean. The boolean will be set when the callback function which services the response from the server is called. Once the response is received (and the boolean flag is set), the server will again send out a request, and do this repeatedly in a loop.

I realise that unsychronised access to shared variables (the boolean) can cause threading issues, but I've tried using pthread mutexes, and the program slows down by about 20% (speed is of importance here). I'm confident that writing to the shared boolean variable is fine as it can be done in a single cycle.

The program runs fine outside of valgrind, but will often stall when run with valgrind. I left the program to run overnight.. usually it takes a few seconds to complete, so I don't think it's a case of not waiting long enough for the program to finish. The threading is managed by the open source engine framework (quick fix), so I don't think it's a problem with how the threads are created/managed.

Does anyone know of any problems with valgrind around multi threaded programs/busy wait loops/socket communications (or a combination of these)?

3
  • 1
    busy-waiting on a shared boolean variable is not "done in a single cycle", it's done in several cycles per loop iteration, and if your busy-loop is waiting on a TCP round-trip across the network, then the loop is likely going to iterate several billion times (and therefore waste several billion CPU cycles that might have been better used elsewhere). A better solution than either of the things you mentioned would be to wait on a condition variable, and have the callback function signal the condition variable to wake your thread up when the data is ready. Dec 29, 2011 at 1:48
  • I was stating that writing to the boolean variable is done in a single cycle (not the entire busy wait process). Having said that I should have stated that writing to the boolean variable is done atomically (as cache misses etc can push a write of a single byte past a single CPU cycle)
    – Trent
    Dec 29, 2011 at 5:18
  • What Jeremy said -- busy wait is a bad idea, condition variable is better, and unlikely to be slower...
    – BillT
    Mar 31, 2014 at 17:22

4 Answers 4

12
+100

I just had a similar problem. Like the OP I had one thread doing a busy wait. In my case the problem was that the busy wait was taking almost all the CPU cycles and causing the other threads to run many thousands of times slower. At first I fixed this by putting a usleep(1) in the busy wait loop (only for Valgrind builds). Then I read the Valgrind manual and learned of the --fair-sched=yes option, which also fixed the problem and allowed me to remove the usleep(1).

10

While other answers focus on insisting that you take the standard synchronization approach (something I fully agree with), I thought instead I should answer your question regarding Valgrind.

As far as I know there are no issues with Valgrind running in multi-threaded environment. I believe Valgrind forces the application to run on a single core, but other than that it should not affect your threads.

What Valgrind is probably doing to your application is altering the timings and interactions between your threads in ways that might be exposing bugs and race conditions in your code that you don't normally see while running stand-alone.

The same logic you applied to decide that the bug could not be in the open source threading framework you are using also applies to Valgrind in my opinion. I recommend that you consider these hangs as bugs in your code and debug them as such, because that is most likely what they are.

As a side note, using a mutex is probably overkill for the problem you described. You should investigate semaphores or condition variables instead.

Good luck.

2
  • Not entirely convinced that there's bugs in my code, but your post is noted. Side note - I looked into condition variables using pthread library, and these require a mutex to stop race conditions on the actual condition itself.
    – Trent
    Dec 29, 2011 at 22:21
  • 2
    @Taras: correct on the condition variable. Semaphores would be my preference in this case. Much lighter than mutexes. On the hang problem, why don't you inspect the hung process with gdb to find out what is it doing? Dec 30, 2011 at 0:30
3

Reading/writing a boolean is not an atomic operation on x86.

See my question here: Is volatile a proper way to make a single byte atomic in C/C++?

1
  • The answer does state that for most writing a boolean is atomic: "CPUs typically read and write single bytes atomically." From the question's answers the problem is not atomicity, it is the time when all CPUs see the new value.
    – Trent
    Dec 29, 2011 at 5:20
2

Even if writing your boolean is an atomic operation, the compiler and the CPU are free to re-order the update around other memory accesses. Your busy-waiting thread may awake from the busy loop and discover that the shared data structure has not actually been updated yet.

I strongly recommend sticking to the threading primitives available to you to write consistent programs that execute exactly as you want them to, every single time.

2
  • The only thing shared between the two threads is the boolean flag (it is being used as a signal). The busy waiting thread only wakes up if the flag has been set, which means that the change in the flag must have propagated through to the CPU that is running the busy waiting thread. Given this, will the code as described still not function correctly?
    – Trent
    Dec 29, 2011 at 5:23
  • Heh, have you tried signals yet? That might do the trick. :)
    – sarnold
    Dec 29, 2011 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.