2

I've got a process that is showing ~4,294,965,900 "current logical threads" (according to the performance counters) and ~400 physical threads.

I've created a memory dump using ADPlus (-hang), and windbg (!threads) only shows me the physical threads.

How do I find out where all these logical threads are coming from?

5
  • The first number looks like a bug, but even 400 threads is an awful lot unless this is running on a huge multi-core server. May 19, 2011 at 5:59
  • I've seen this when I include MS SQL Express 2005 into the project, do you have any MS SQL access in your code and could you remove them to see if the logical threads are due to them?
    – Martin
    May 19, 2011 at 7:46
  • It is running on quite an impressive server.
    – Mark
    May 20, 2011 at 14:01
  • @Mark Did you ever figure out why this was happening to you? Aug 2, 2018 at 12:39
  • @StephanRyer I did not, sorry.
    – Mark
    Aug 7, 2018 at 4:54

4 Answers 4

4

That looks like a suspiciously high number to me.

The number -1396 represented as an unsigned 32-bit integer is 4,294,965,900, and 1396 looks more reasonable.

A bug somewhere, perhaps?

4
  • And, to reiterate, yes, of course it's a suspiciously high number. That's why I'm trying to figure out where they are. Remember, WinDbg only showed the ~400 physical threads, so no, I don't think there's actually ~1400 threads.
    – Mark
    May 18, 2011 at 18:00
  • I don't get the upvotes? How do you have a negative number of threads? This number is coming from the Windows performance counter, and it's a 64-bit number.
    – Mark
    May 18, 2011 at 19:40
  • @Mark: because the smallish negative number is a reasonable explanation of why this ridiculously high number is being reported. -1396 could be an error code, for example.
    – jalf
    May 19, 2011 at 7:42
  • @jalf: Now you mention it, an error code does sound like a reasonable explanation.
    – MRAB
    May 19, 2011 at 16:35
0

How do I find out where all these logical threads are coming from?

they aren't. They don't exist. You simply can't have 4 billion threads of any kind, unless you're running on a 64-bit machine with, oh, say a couple of petabyte of RAM at the very least.

Every thread, whether it is a "physical" OS thread or is provided by some framework, need at the very least, some kind of identifier. If that's a 32-bit number then just storing these identifiers will take up nearly 16GB of RAM. (And, of course, you'll have around 1600 unused identifiers left). If the identifiers are 64 bits wide, you need 32GB RAM. On top of that, every thread needs some stack space (a common default is 1MB, which brings us up to 4 petabytes of memory).

It is a bug. The threads don't exist, and the performance counters are reporting a garbage value to you for some reason or other.

For example, it could be a negative error code which, when converted an unsigned integer, becomes this huge number.

Or it could be some other error condition.

0

Since your process is running managed code, chances are the logical thread count refers to CLR threads. .Net does mapping between CLR logical threads and physical threads. To investigate this further, you can use !threads command in Windbg. This is example of output from this command:


0:028> !threads
ThreadCount:      25
UnstartedThread:  0
BackgroundThread: 22
PendingThread:    0
DeadThread:       3
Hosted Runtime:   yes
                                   PreEmptive   GC Alloc                Lock
       ID  OSID ThreadOBJ    State GC           Context       Domain   Count APT Exception
   0    1  12b0 007b69d0      4220 Enabled  120337b4:12034a3c 007afef8     0 STA
   6    2  1f70 007c2688      b220 Enabled  11ed2a84:11ed4a3c 007afef8     0 MTA (Finalizer)
   7    3  2340 007c8ac8      1220 Enabled  00000000:00000000 007afef8     0 Ukn
  11    4  1c4c 0aaf3380      7220 Enabled  00000000:00000000 007afef8     0 STA
  13    8  2414 0d4932f0       220 Enabled  00000000:00000000 007afef8     0 Ukn
   3    a  2780 0d4d08e8   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  15    7   970 0d4d0df0   1009220 Enabled  11ed4ad8:11ed6a3c 007afef8     0 MTA (Threadpool Worker)
  19    9  2510 0d4d12f8   200b220 Enabled  00000000:00000000 007afef8     0 MTA
  20    b   80c 0d4d1800   200b220 Enabled  00000000:00000000 007afef8     0 MTA
  21    c  2490 0d4d1d08   200b220 Enabled  00000000:00000000 007afef8     0 MTA
  23    d  2724 0d4d2210   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  24    e  2200 0d4d2718   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  26    f  1f3c 0d4d2c20   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  25   10  200c 0d4d3128   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  27   11  2708 0d4d3630   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  17    6  21b4 0d4d3b38   1009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  18    5  2148 0d4d4548       220 Enabled  00000000:00000000 007afef8     0 MTA
XXXX   16       0d4d6378     19820 Enabled  00000000:00000000 007afef8     0 MTA
XXXX   15       0d4d5e70     19820 Enabled  00000000:00000000 007afef8     0 MTA
  30   14  112c 0d4d5968   200b220 Enabled  00000000:00000000 007afef8     0 MTA
  32   13  2734 0d4d5460      b220 Enabled  00000000:00000000 007afef8     0 MTA
  33   12  11ec 0d4d4a50   100a220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Worker)
  34   17  166c 0d4d6880   8009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Completion Port)
  35   18  24f4 0d4d6d88   8009220 Enabled  00000000:00000000 007afef8     0 MTA (Threadpool Completion Port)
XXXX   19       0d4d7798     19820 Enabled  00000000:00000000 007afef8     0 Ukn

Note at the top of output it prints out statistics. If you find exessively large number of dead threads, that might indicate resource leaks. Check out one example of this type of resource leak.

In the !threads output the left column is unmanaged thread ID (same as displayed by ~ command), second column is CLR thread ID and third column is OS thread ID.

1
  • Right, youll notice that I did that, and that only the physical threads were shown.
    – Mark
    May 20, 2011 at 14:00
0

I had the same problem this week, same symptoms. It was real. Yes my server is impressive, 128G Ram and 24 cores.

The problem here was logical threads indeed. CLR doesn't create a real thread if it can avoid it. I had a Timer with periodic reactivation like timer.Change(10000, 10000) and inside the timer callback my code hung on network, which let CLR runtime know this 'physical thread' could be reused. Then 10 seconds later the timer is triggered again, and a new logical thread is created, and so on. The next issue is that the rest of my code uses Tasks thoroughly, and those pull also logicalthreads. Combine it all, and you have a ripple effect of billions of logical threads in a week or two.

My fix was easy: make the timer not recurring, but reschedule the next timer trigger only after the previous one is finished: timer.Change(10000,Timeout.Infinite) and make the timer callback so it cancels the io upon some reasonable timeout.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.