I'm testing how tracing works in ASP.NET Core 3.1 and something is not working. I have 3 ASP.NET Core apps (Node1, Node2, Node3) with ActivityIdFormat.W3C enabled and call hierarchy is Node1->Node2->Node3 using httpclient. When Node2 is called it has proper ParentId that equals SpanId from Node1. But when Node3 is callsd from Node2 ParentId is wrong. Below logs from each node - you can see that I dump also headers and it looks like headers sent from Node2 are wrong. I see same behaviour in .NET 5 but when I checked in .NET 3.1 without ActivityIdFormat.W3C parent is propagated on all nodes.
{
"@t": "2021-03-03T14:14:28.1125178Z",
"@mt": "Test on Node 1 with {@Headers}",
"@l": "Error",
"Headers": "[Connection, keep-alive], [Accept, */*], [Accept-Encoding, gzip, deflate, br], [Host, localhost:5001], [Referer, http://localhost:5000/WeatherForecast?test=three], [User-Agent, PostmanRuntime/7.26.10], [Postman-Token, 566e895a-efc9-440d-9cc2-d3272a59f08e]",
"SourceContext": "TraceTest.Controllers.WeatherForecastController",
"ActionId": "8322cc4b-f499-4292-ad92-95edf9932274",
"ActionName": "TraceTest.Controllers.WeatherForecastController.Get (TraceTest)",
"RequestId": "0HM6U9QJT75QA:00000001",
"RequestPath": "/WeatherForecast",
"SpanId": "c0e4e78f0ab2684f",
"TraceId": "532c5503c7def848953038f49ba81918",
"ParentId": "0000000000000000",
"ConnectionId": "0HM6U9QJT75QA"
}
{
"@t": "2021-03-03T14:14:28.5933479Z",
"@mt": "Test on Node 2 with {@Headers}",
"@l": "Error",
"Headers": "[Host, localhost:5011], [traceparent, 00-532c5503c7def848953038f49ba81918-c0e4e78f0ab2684f-00]",
"SourceContext": "TraceTest.Controllers.WeatherForecastController",
"ActionId": "47ec867a-b69b-49bf-91d2-4aeb089fe7d6",
"ActionName": "TraceTest.Controllers.WeatherForecastController.Get (TraceTestNode2)",
"RequestId": "0HM6U9QK2GI54:00000001",
"RequestPath": "/WeatherForecast",
"SpanId": "65583073c1fa3644",
"TraceId": "532c5503c7def848953038f49ba81918",
"ParentId": "c0e4e78f0ab2684f",
"ConnectionId": "0HM6U9QK2GI54"
}
{
"@t": "2021-03-03T14:14:29.0369082Z",
"@mt": "Test on Node 3 with {@Headers}",
"@l": "Error",
"Headers": "[Host, localhost:5021], [traceparent, 00-532c5503c7def848953038f49ba81918-ffcefa1bb1710340-00]",
"SourceContext": "TraceTest.Controllers.WeatherForecastController",
"ActionId": "c475354c-ba74-4f45-b9d4-57362a3b8899",
"ActionName": "TraceTest.Controllers.WeatherForecastController.Get (TraceTestNode3)",
"RequestId": "0HM6U9QK6QAPH:00000001",
"RequestPath": "/WeatherForecast",
"SpanId": "262cbf3f566d9044",
"TraceId": "532c5503c7def848953038f49ba81918",
"ParentId": "ffcefa1bb1710340",
"ConnectionId": "0HM6U9QK6QAPH"
}
Source codes can be found here https://github.com/dominikjeske/Samples/tree/main/TraceTest