4

In the old (single-threaded) days we instructed our testing team to always report the CPU time and not the real-time of an application. That way, if they said that in version 1 an action took 5 CPU seconds, and in version 2 it took 10 CPU seconds, that we had a problem.

Now, with more and more multi-threading, this doesn't seem to make sense anymore. It could be that the version 1 of an application takes 5 CPU seconds, and version 2 10 CPU seconds, but that version 2 is still faster if version 1 is single-threaded, and version 2 uses 4 threads (each consuming 2.5 CPU seconds).

On the other hand, using real-time to compare performance isn't reliable either since it can be influenced by lots of other elements (other applications running, network congestion, very busy database server, fragmented disk, ...).

What is in your opinion the best way to 'numerate' performance? Hopefully it's not intuition since that is not an objective 'value' and probably leads to conflicts between the development team and the testing team.

3
  • Can't you just measure wall-clock time in a system not loaded with unrelated programs? Feb 26, 2010 at 23:41
  • @Mike, for long-running actions, yes. But sometimes the test team also reports performance decreases if the time for certain actions increases from 1 to 1.2 seconds. This can hardly be measured using the wall-clock (but in those cases I also have doubts about the actual performance problem).
    – Patrick
    Feb 27, 2010 at 11:58
  • 1
    I'm sure you understand I don't mean a physical plastic electric clock hanging on a wall, and that I do mean the ending time-of-day minus the starting time-of-day. This is in contrast to things that try to measure in-process CPU time distinct from I/O time, as some profilers do. IMO the latter may satisfy some curiosity but is little help in locating problems. Mar 1, 2010 at 14:08

1 Answer 1

1

Performance needs to be defined before it is measured.

Is it:

  • memory consumption?
  • task completion times?
  • disk space allocation?

Once defined, you can decide on metrics.

1
  • Good point. Problem is that testing people tend to report anything that increases: CPU Time, resource usage, memory, ... Time to discuss this with the testers. Thanks.
    – Patrick
    Feb 26, 2010 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.