2

Situation: ASP.NET live website that's occasionally too busy.

Adding full profiling to the code will be too heavy an impact on the performance. Using performance monitor, we've quickly find a saw-teeth figure in the "bytes in all heaps" counter, which pair with the GC counters. We consider certain pieces of code as being the culprit.

Is there a way or possibility to temporarily "inject" profiling, either for certain pages, libs, functions or whatever? Preferably as lightweight as possible, as any additional overhead might bring down this fragile system. I'm aware that .NET does not support method callback hooks (as is common with AOP).

4 Answers 4

2

A few ideas:

  1. Use custom Windows performance counters. They are very lightweight (1 to 2%), and you can use them not just to count, but also for timing measurements, to look at how often certain operations are slower than a threshold, etc.
  2. If you're using IIS 7, you can enable Failed Request Tracing. To limit the perf impact, be careful to not enable it for too many pages. Those traces can provide lots of detail, and you can inject more info into them programmatically if you need to.
  3. Use the Windows Event log to write custom details under exceptional conditions. Perf impact is minimal as long as you don't over-do it.
  4. One cause of sawtooth memory behavior can be not calling Dispose() when you should (or wrapping IDisposable objects in using statements, which will call it for you); you might want to review your code to look for that.

In case it's helpful, you might also be interested in the performance tips from my book: Ultra-Fast ASP.NET.

Edit: you might also try using .NET Memory Profiler (free trial available) to attach to the running process. It's fairly invasive compared to counters, but if you need to capture a snapshot of the current state of memory to debug your problem, there may not be a better choice.

6
  • Thanks for the advises. I'm aware of all of those and indeed, the pieces of the code we deemed the possible causes are not using using blocks around Disposable objects. But my gut feeling says, there's something else up here (ran smooth for a long time with 10x as much users, same code). Nothing odd in event log, no odd failed requests. Performance counters are on fulltime with us (see original question).
    – Abel
    Nov 27, 2009 at 14:50
  • 1
    From your question, I thought you were just using the existing Windows perf counters, not adding your own custom ones.... Threading / locking issues are another common cause of perf hiccups.
    – RickNZ
    Nov 27, 2009 at 14:53
  • Yes, perhaps my question wasn't very clear (sorry, tried to be brief, not my best asset). But I should do more with them (custom perf counters), but can't do a rebuild now, only monitor (important period for this shop). Threading and locks, or other type of wait loops should certainly be on our monitor, thanks.
    – Abel
    Nov 27, 2009 at 15:12
  • Sorry, I was trying to suggest that threading / locking issues combined with improper handling of IDisposables might explain why your code was working fine before and isn't any more. Custom instrumentation is often the least invasive (and sometimes the only) way to find problems like that (other than doing some detailed code walkthroughs).
    – RickNZ
    Nov 28, 2009 at 11:57
  • Another possibility is to use .NET Memory Profiler (see memprofiler.com) -- I updated my answer with that info too.
    – RickNZ
    Nov 28, 2009 at 12:03
1

Yes, you can take a memory dump of the application as it's running and see what it's holding in memory. This should reinforce or deny your assumptions.

3
  • Looking into it, seems the path to take indeed.
    – Abel
    Nov 27, 2009 at 14:33
  • The most interesting thing is missing from that post: making the memory dump. How to?...
    – Abel
    Nov 27, 2009 at 14:46
  • Just did a memory dump, but it reset the app pool, stopped serving requests for a while (during the dump, but it took too long) and the actual dump is 200MB compared to 800MB for the process... Hmm... Is there something even less evasive available?
    – Abel
    Nov 27, 2009 at 15:15
1

I recently posted a possible solution to a similar challenge:

Profiling a multi-tiered, distributed, web application (server side) shows a high level approach (profiling on the URL level) that is:

  • platform and language independent
  • completely non invasive
  • gives a high-level picture of where your application is spending most of its time

The idea is to use existing web logs and convert them into a "one-picture is worth a 1000 words" kind of chart.

This approach is not sufficient for use-cases requiring more fine-level detail, but it helped me personally, and may be worth exploring in your case.

0

Can you do this on ASP .NET? It's a good quick way to find out what's going on.

2
  • Not really. However, you can attach a debugger like windbg and basically get most of the same information -- it's a pretty invasive solution, though, particularly for a production environment.
    – RickNZ
    Nov 28, 2009 at 11:59
  • @RickNZ: It is invasive. Some folks make gadgets that glom onto a running process and take stackshots. I'll see if I can find the link. Nov 28, 2009 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.