2

Running into a very strange issue with 2+ threads waiting on an interprocess_condition variable.

Boost 1.60.0

  • With 1 thread calling wait() and a 2nd calling notify_all(), everything works as expected.
  • When there are 2+ calling wait(), I get an assertion failure on do_wait() and the process exits.

Test.cpp:


#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/sync/interprocess_mutex.hpp>
#include <boost/interprocess/sync/interprocess_condition.hpp>
#include <iostream>

using namespace boost::interprocess;

struct Data {
    interprocess_mutex mux_;
    interprocess_condition cond_;
};

int main(int argc, char *argv[]) {
    if (argc > 1 && atoi(argv[1]) == 0) {
        struct shm_remove {
            shm_remove() { shared_memory_object::remove("MySharedMemory"); }
            ~shm_remove() { shared_memory_object::remove("MySharedMemory"); }
        } remover;

        managed_shared_memory seg(create_only, "MySharedMemory", 65536);
        Data *const d = seg.construct<Data>(unique_instance)();
        scoped_lock<interprocess_mutex> lock(d->mux_);
        std::cout << "Waiting" << std::endl;
        d->cond_.wait(lock);
    } else if (argc > 1 && atoi(argv[1]) == 1) {
        managed_shared_memory seg(open_only, "MySharedMemory");
        std::pair<Data *, std::size_t> res = seg.find<Data>(unique_instance);
        scoped_lock<interprocess_mutex> lock(res.first->mux_);
        std::cout << "Waiting" << std::endl;
        res.first->cond_.wait(lock);
    } else {
        managed_shared_memory seg(open_only, "MySharedMemory");
        std::pair<Data *, std::size_t> res = seg.find<Data>(unique_instance);
        scoped_lock<interprocess_mutex> lock(res.first->mux_);
        std::cout << "Notifying" << std::endl;
        res.first->cond_.notify_all();
    }
}

Compiled as:

$ clang++ -I/usr/local/include test.cpp

Running with 1 wait() and 1 notify():

$ ./a.out 0&
[8] 25889
Waiting

$ ./a.out 2&
[9] 25901
Notifying
[8]-  Done                    ./a.out 0
[9]+  Done                    ./a.out 2

Running with 2 waits:

$ ./a.out 0&
[8] 25986
Waiting
$ ./a.out 1&
[9] 25998
Waiting
Assertion failed: (res == 0), function do_wait, file /usr/local/include/boost/interprocess/sync/posix/condition.hpp, line 175.

Tested on OSX El Capitan

$ uname -a
Darwin LUS-JOHUGHES2 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64

I also tried the above example on an Ubuntu Trusty machine and all examples work as expected, leading me to believe there is an issue with the OSX implementation. I have not tried it on Windows.

1
  • Agree this looks like a platform specific quirk. Everything looks good on my Ubuntu box. Here it is Live On Coliru (using managed_mapped_file due to Coliru limitations)
    – sehe
    Feb 10, 2016 at 21:05

2 Answers 2

2

Did some digging and found a definite answer to the problem.

  1. The boost assertion error above is failing when the second process calls do_wait(), which calls pthread_wait() which returns immediately with EINVAL (instead of a successful 0).

  2. In OSX's pthread implementation, the condition variable stores a raw pointer to the mutex variable. The first call to pthread_wait() in the first process sets this pointer. The second call to pthread_wait() checks this stored mutex pointer against the mutex pointer passed into pthread_wait(). {Can be found in the source here: https://opensource.apple.com/source/libpthread/libpthread-137.1.1/src/pthread_cond.c}

  3. Since the two processes have mapped the shared mutex and condition variables in different address spaces, the second call to pthread_wait() will never work, since it compares raw pointers.

Thus the 2 options to make this work are as follows:

  1. Use Fixed Address Mapping: http://www.boost.org/doc/libs/1_60_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.mapped_region.mapped_region_fixed_address_mapping which guarantees that the mapped regions will be at the same address, and thus raw pointers will work, or

  2. Instead of exec()'ing new processes, use fork() which means that the child process will have a copy of the original segment manager, mapped to the same address, and thus raw pointers will work.

I didn't dig much into the glibc pthreads code to see what they do differently than Apple, so I'm not sure why the original example works on Linux but not OSX.

I think the Boost docs would definitely benefit with a paragraph discussing this pitfall.

1
  • 1
    Excellent find. Instead of adding this as some kind of quirk to the boost docs, I'd consider this a bug in Boost Interprocess. The library aims to provide portable interprocess communication (and synchronization) and in this spot, the implementation isn't portable. Please report the bug at Trac since a report doesn't appear to exist
    – sehe
    Feb 14, 2016 at 19:58
1

This is a bug both on Darwin's C library and Boost.Interprocess. First, that C library claims it's posix-compliant and supports process shared memory condition variables, which is false (as it uses a raw pointer to store the address of the mutex). Second, Boost.Interprocess should detect this platform as a buggy one, should disable the use of pthreads and fallback to emulation.

In boost/interprocess/detail/workaround.hpp, you will find a comment saying:

//Mac Os X < Lion (10.7) might define _POSIX_THREAD_PROCESS_SHARED but there is no real support.

Some old reports claimed that newer macos versions really support process shared condition variables, but this claim is false, so the __APPLE__ section should be just:

#define BOOST_INTERPROCESS_BUGGY_POSIX_PROCESS_SHARED

1
  • Cool additions. Thanks for adding those
    – sehe
    May 3, 2018 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.