3

I have a fairly busy site which does around 10m views a month.

One of my app pools seemed to jam up for a few hours and I'm looking for some ideas on how to troubleshoot it..? I suspect that it somehow ran out of threads but I'm not sure how to determine this retroactively..? Here's what I know:

  • The site never went 'down', but around 90% of requests started timing out.
  • I can see a high number of "HttpException - Request timed out." in the log during the outage
  • I can't find any SQL errors or code errors that would have caused the timeouts.
  • The timeouts seem to have been site wide on all pages.
  • There was one page with a bug on it which would have caused errors on that specific page.
  • The site had to be restarted.

The site is ASP.NET C# 3.5 WebForms..

Possibilities:

  1. Thread depletion: My thought is that the page causing the error may have somehow started jamming up the available threads?
  2. Global code error: Another possibility is that one of my static classes has an undiscovered bug in it somewhere. This is unlikely as the this has never happened before, and I can't find any log errors for these classes, but it is a possibility.

UPDATE

I've managed to trace the issue now while it's occurring. The pages are being loaded normally but for some reason WebResource.axd and ScriptResource.axd are both taking a minute to load. In the performance counters I can see ASP.NET Requests Queued spikes at this point.

3
  • 2
    btw; there is no such thing as C# 3.5; I assume you mean C# 3.0 targeting .NET 3.5 Commented Apr 26, 2011 at 9:21
  • Do you use threads/locks/mutex and more than one working pool ? Search for mutex lock.
    – Aristos
    Commented Apr 26, 2011 at 9:36
  • Hi Aristos, yes this is definately an area I should investigate further. All my central thread locks have a 25ms bypass on them, eg. Monitor.TryEnter(_mylock, 25); But I might have missed something there.. Commented Apr 26, 2011 at 9:56

1 Answer 1

3

The first thing I'd try is Sam Saffron's CPU analyzer tool, which should give an indication if there is something common that is happening too much / too long. In part because it doesn't involve any changes; just run it at the server.

After that, there are various other debugging tools available; we've found that some very ghetto approaches can be insanely effective at seeing where time is spent (of course, it'll only work on the 10% of successful results).

You can of course just open the server profiling tools and drag in various .NET / IIS counters, which may help you spot some things.

Between these three options, you should be covered for:

  • code dropping into a black hole and never coming out (typically threading related)
  • code running, but too slowly (typically data access related)
6
  • Thanks Marc, appreciate the info. I've loaded up some performance counters for this now, but as you would guess, all counters are looking pretty normal now. Will keep them loaded if it occurs again.. Also, forgot to mention, CPU dropped to nothing during the outage.. Commented Apr 26, 2011 at 9:47
  • @Dave if CPU is nothing, that could be mass-blocking (thread related errors), or could be data-access issues; both involve no CPU Commented Apr 26, 2011 at 9:49
  • Thanks Marc, yeah the low CPU does make it look that way. I think I can rule out the data-access side. I've encountered many of those through the years and know how to spot them quite well. Also can't pick up any SQL timeouts in the log, and static pages with no SQL where also timing out. So I think thread blocking is looking like the culprit here.. Commented Apr 26, 2011 at 10:01
  • 1
    @Dave - thread blocking tends to be easy to spot with a polling-based profiler, as every time it asks what the thread is doing it'll be stuck waiting Commented Apr 26, 2011 at 10:04
  • 1
    I've located the main symptom now, for some reason my two .axd files are taking a minute to load.. Commented Apr 26, 2011 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.