4

Before coroutines we used callbacks to run asynchronous operations. Callbacks was normal functions and could have thread_local variables.

Let see this example:

void StartTcpConnection(void) 
{
    using namespace std;
    thread_local int my_thread_local = 1;
    cout << "my_thread_local = " << my_thread_local << endl;
    auto tcp_connection = tcp_connect("127.0.0.1", 8080);
    tcp_connection.async_wait(TcpConnected);
}

void TcpConnected(void)
{
    using namespace std;
    thread_local int my_thread_local = 2;
    cout << "my_thread_local = " << my_thread_local << endl;
}

As we see from code, I have some (undocumented here) tcp_connect function that connects to TCP endpoint and returns tcp_connection object. This object can wait until TCP connection will really occur and call TcpConnected function. Because we don't know specific implementation of tcp_connect and tcp_connection, we don't know will it call TcpConnected on the same or on different thread, both implementations are possible. But we know for sure that my_thread_local is different for 2 different functions, because each function has its own scope.

If we need this variable to be the same (as soon as thread is the same), we can create 3rd function that will return reference to thread_local variable:

int& GetThreadLocalInt(void)
{
    thread_local int my_variable = 1;
    return my_variable;
}

So, we have full control and predictability: we know for sure that variables will be different if TcpConnected and StartTcpConnection will run on different threads, and we know that we can have them different or the same depending on our choice when these functions will run on the same thread.

Now let see coroutine version of the same operation:

void Tcp(void)
{
    thread_local int my_thread_local = 1;
    auto tcp_connection = co_await tcp_connect("127.0.0.1", 8080);
    cout << "my_thread_local = " << my_thread_local << endl;
}

This situation is a bit questionable for me. I still need thread local storage, it is important language feature that I don't want to abandon. However, we here have 2 cases:

  1. Thread before co_await is the same one as after co_await. What will happen with my_thread_local? Will it be the same variable before and after co_await, especially if we'll use GetThreadLocalInt function to get its reference instead of value?
  2. Thread changes after co_await. Will C++ runtime reinitialize my_thread_local to value from new thread, or make a copy of previous thread value, or may be use reference to the same data? And similar question for GetThreadLocalInt function, it returns reference to thread_local object, but the reference storage itself is auto, will coroutine reinitialize it to new thread, or we'll get (dangerous!!!) race condition, because thread 2 will strangely get reference to thread 1 thread local data and potentially use it in parallel?

Even it is easy to debug and test what will happen on any specific compiler, the important question is whether standard says us something about that, otherwise even if we'll test it on VC++ or gcc an see that it behaves somehow on these 2 popular compilers, the code may loose portability and compile differently on some exotic compilers.

7
  • Thread local variable, are as said - thread local. If after co_await, a different thread opens the coroutine, it will access its instance of thread local variable.
    – ALX23z
    Feb 14 at 14:01
  • If you make local reference to thread local variable, then new thread will access via the reference to a thread local variable of the old thread - which can unsafe for various reasons.
    – ALX23z
    Feb 14 at 14:03
  • @ALX23z Are you sure that each time when we access thread_local variable in C++ the compiler emits code that calls OS level API to access thread local storage instead of doing it one time and using previously received pointer to thread local data? I. e. that code thread_local o = new object(); o.method1();o.method2();o.method3() will emit code that calls TlsGetValue 4 times? Is it guarantee of standard? Or any optimization may occur that insignificant for standard function but may change behavior of coroutine?
    – Vitalii
    Feb 14 at 14:14
  • 1
    @Vitalii: If it would change behavior, then that implementation would be faulty. The compiler can see you doing co_await, so it knows that afterwards, it needs to access the memory again instead of using a cached copy. This is no different from accessing a global and calling a function the compiler can't see the definition of; the compiler must assume that the global was changed by the function call, so later accesses to the function have to be real memory fetches. Feb 14 at 14:32
  • 1
    There is MSVC bug - which states GCC and Clang compile this correctly. The specification does not allow compilers to cache thread-local variables in this situation. (without references). So, it seems, thread_local is required be bound to the current thread of execution, but MSVC currently does not handle this case correctly.
    – dewaffled
    Feb 14 at 14:45

1 Answer 1

1

For global thread_local variables, the coroutine behavior ought to be as expected (and MSVC seems to have a bug with this). But function-local thread_local variables in coroutines, there seems to be a hole in the specification. Indeed, I'm not sure the wording makes sense even without coroutines.

[stmt.dcl]/3 says:

Dynamic initialization of a block variable with static storage duration or thread storage duration is performed the first time control passes through its declaration; such a variable is considered initialized upon the completion of its initialization.

The problem is that, while by C++'s rules there is only one thread_local variable, there are multiple objects represented by that one variable. And those objects need to be initialized. So... how does that happen?

The only sane interpretation of this is that the object for a particular thread gets initialized the first time control flow passes through the declaration on that thread. And that's the problem.

If using co_await the thread a function executes on changes, then how could you pass through the declaration on the new thread? Which means that the thread_local for that thread should be zero-initialized.

Ultimately, I would say that you should never use thread_local in a coroutine function. It's just not clear what value the thread_local ought to have. And the only logical value for the new thread_local to have is the one from the previous thread. But that's not how a thread_local is supposed to work. Overall, the idea feels inherently nonsensical (and I would say that the standard should have explicitly forbid the declaration of thread_locals in a coroutine, just as they did for using co_await in a thread_local's initializer).

Just use a namespace-scoped thread_local variable. Outside of the aforementioned MSVC bug, it ought to work and make sense.

5
  • > I would say that the standard should have explicitly forbid the declaration of thread_locals in a coroutine -- The problem is that they can't reasonably forbid it because the thread_local variable may be created in a normal function that happens to be called from a coroutine. Something like this: 1. A coroutine calls a normal function and passes a callback pointer. 2. The normal function initializes a thread_local and calls the callback. 3. The callback yields, suspending the coroutine. 4. In another thread, the coroutine resumes and returns from the callback. Apr 7 at 21:49
  • I think, the best they could do is to say that accessing a thread_local variable from a coroutine is undefined behavior. Apr 7 at 21:51
  • @AndreySemashev: "The problem is that they can't reasonably forbid it because the thread_local variable may be created in a normal function that happens to be called from a coroutine." But that wouldn't cause this problem. If the coroutine's thread changes between two separate calls to this "normal function", that's fine. The "normal function" will do its normal behavior for each thread, just as if a non-coroutine called it from the other thread. The problem is that a coroutine happens to be able to switch threads in the middle of execution. That's where things go sideways. Apr 7 at 22:49
  • @AndreySemashev: "The callback yields, suspending the coroutine." The callback cannot suspend the coroutine higher up on the stack. The callback can only suspend itself, returning control to the "normal function" who called it. That's part and parcel of being a "stackless" coroutine system. Apr 7 at 22:50
  • Yes, you're right. I was thinking that yielding in the callback would return from the original coroutine, but this wouldn't be the case. Apr 8 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.